Skip to contentSkip to conversation

drm/msm: Add MSM_SUBMIT_BO_NO_IMPLICIT

PATCH In review Rob Clark 3 participants 6 messages Dec 6, 2022 – Jan 5, 2023

Subsystems (5)
lore
Share & tools
lore

Rob Clark

author
#0 lore raw

PATCH 1/1 drm/msm: Add MSM_SUBMIT_BO_NO_IMPLICIT

From: Rob Clark <[email protected]>

In cases where implicit sync is used, it is still useful (for things like sub-allocation, etc) to allow userspace to opt-out of implicit sync on per-BO basis.

---
 drivers/gpu/drm/msm/msm_drv.c        |  3 ++-
 drivers/gpu/drm/msm/msm_gem_submit.c | 11 +++++++++++
 include/uapi/drm/msm_drm.h           |  4 +++-
 3 files changed, 16 insertions(+), 2 deletions(-)
drivers/gpu/drm/msm/msm_drv.c +2 -1
index 017a512982a2..e0e1199a822f 100644
@@ -45,9 +45,10 @@
  * - 1.7.0 - Add MSM_PARAM_SUSPENDS to access suspend count
  * - 1.8.0 - Add MSM_BO_CACHED_COHERENT for supported GPUs (a6xx)
  * - 1.9.0 - Add MSM_SUBMIT_FENCE_SN_IN
+ * - 1.10.0 - Add MSM_SUBMIT_BO_NO_IMPLICIT
  */
 #define MSM_VERSION_MAJOR	1
-#define MSM_VERSION_MINOR	9
+#define MSM_VERSION_MINOR	10
 #define MSM_VERSION_PATCHLEVEL	0
 
 static const struct drm_mode_config_funcs mode_config_funcs = {
drivers/gpu/drm/msm/msm_gem_submit.c +11 -0
index eb3536e3d66a..8bad07a04f85 100644
@@ -334,9 +334,20 @@ static int submit_fence_sync(struct msm_gem_submit *submit, bool no_implicit)
 		if (ret)
 			return ret;
 
+		/* If userspace has determined that explicit fencing is
+		 * used, it can disable implicit sync on the entire
+		 * submit:
+		 */
 		if (no_implicit)
 			continue;
 
+		/* Otherwise userspace can ask for implicit sync to be
+		 * disabled on specific buffers.  This is useful for internal
+		 * usermode driver managed buffers, suballocation, etc.
+		 */
+		if (submit->bos[i].flags & MSM_SUBMIT_BO_NO_IMPLICIT)
+			continue;
+
 		ret = drm_sched_job_add_implicit_dependencies(&submit->base,
 							      obj,
 							      write);
include/uapi/drm/msm_drm.h +3 -1
index f54b48ef6a2d..329100016e7c 100644
@@ -222,10 +222,12 @@ struct drm_msm_gem_submit_cmd {
 #define MSM_SUBMIT_BO_READ             0x0001
 #define MSM_SUBMIT_BO_WRITE            0x0002
 #define MSM_SUBMIT_BO_DUMP             0x0004
+#define MSM_SUBMIT_BO_NO_IMPLICIT      0x0008
 
 #define MSM_SUBMIT_BO_FLAGS            (MSM_SUBMIT_BO_READ | \
 					MSM_SUBMIT_BO_WRITE | \
-					MSM_SUBMIT_BO_DUMP)
+					MSM_SUBMIT_BO_DUMP | \
+					MSM_SUBMIT_BO_NO_IMPLICIT)
 
 struct drm_msm_gem_submit_bo {
 	__u32 flags;          /* in, mask of MSM_SUBMIT_BO_x */
Signature · 1 line
2.38.1
#1 lore raw

Hi Rob,

Am Dienstag, dem 06.12.2022 um 11:21 -0800 schrieb Rob Clark:

Quoted 6 lines

From: Rob Clark <[email protected]>

In cases where implicit sync is used, it is still useful (for things like sub-allocation, etc) to allow userspace to opt-out of implicit sync on per-BO basis.

Out of curiosity and because I have been thinking about something like that for etnaviv for while: do you only use this for immutable buffers or do you have some kind of userspace fencing in place for the suballocated buffers?

Regards,
Lucas

Quoted 66 lines
---
 drivers/gpu/drm/msm/msm_drv.c        |  3 ++-
 drivers/gpu/drm/msm/msm_gem_submit.c | 11 +++++++++++
 include/uapi/drm/msm_drm.h           |  4 +++-
 3 files changed, 16 insertions(+), 2 deletions(-)
drivers/gpu/drm/msm/msm_drv.c +2 -1
index 017a512982a2..e0e1199a822f 100644
@@ -45,9 +45,10 @@
  * - 1.7.0 - Add MSM_PARAM_SUSPENDS to access suspend count
  * - 1.8.0 - Add MSM_BO_CACHED_COHERENT for supported GPUs (a6xx)
  * - 1.9.0 - Add MSM_SUBMIT_FENCE_SN_IN
+ * - 1.10.0 - Add MSM_SUBMIT_BO_NO_IMPLICIT
  */
 #define MSM_VERSION_MAJOR	1
-#define MSM_VERSION_MINOR	9
+#define MSM_VERSION_MINOR	10
 #define MSM_VERSION_PATCHLEVEL	0
 
 static const struct drm_mode_config_funcs mode_config_funcs = {
drivers/gpu/drm/msm/msm_gem_submit.c +11 -0
index eb3536e3d66a..8bad07a04f85 100644
@@ -334,9 +334,20 @@ static int submit_fence_sync(struct msm_gem_submit *submit, bool no_implicit)
 		if (ret)
 			return ret;
 
+		/* If userspace has determined that explicit fencing is
+		 * used, it can disable implicit sync on the entire
+		 * submit:
+		 */
 		if (no_implicit)
 			continue;
 
+		/* Otherwise userspace can ask for implicit sync to be
+		 * disabled on specific buffers.  This is useful for internal
+		 * usermode driver managed buffers, suballocation, etc.
+		 */
+		if (submit->bos[i].flags & MSM_SUBMIT_BO_NO_IMPLICIT)
+			continue;
+
 		ret = drm_sched_job_add_implicit_dependencies(&submit->base,
 							      obj,
 							      write);
include/uapi/drm/msm_drm.h +3 -1
index f54b48ef6a2d..329100016e7c 100644
@@ -222,10 +222,12 @@ struct drm_msm_gem_submit_cmd {
 #define MSM_SUBMIT_BO_READ             0x0001
 #define MSM_SUBMIT_BO_WRITE            0x0002
 #define MSM_SUBMIT_BO_DUMP             0x0004
+#define MSM_SUBMIT_BO_NO_IMPLICIT      0x0008
 
 #define MSM_SUBMIT_BO_FLAGS            (MSM_SUBMIT_BO_READ | \
 					MSM_SUBMIT_BO_WRITE | \
-					MSM_SUBMIT_BO_DUMP)
+					MSM_SUBMIT_BO_DUMP | \
+					MSM_SUBMIT_BO_NO_IMPLICIT)
 
 struct drm_msm_gem_submit_bo {
 	__u32 flags;          /* in, mask of MSM_SUBMIT_BO_x */
author
#2 lore raw
Quoted 6 lines · Lucas Stach

Hi Rob,

Am Dienstag, dem 06.12.2022 um 11:21 -0800 schrieb Rob Clark:

Quoted 2 lines

From: Rob Clark <[email protected]>

Quoted 8 lines
Quoted 4 lines

In cases where implicit sync is used, it is still useful (for things like sub-allocation, etc) to allow userspace to opt-out of implicit sync on per-BO basis.

Out of curiosity and because I have been thinking about something like that for etnaviv for while: do you only use this for immutable buffers or do you have some kind of userspace fencing in place for the suballocated buffers?

yup, userspace fences.. which is a thing you'd really want for doing suballocation

BR,
-R

Quoted 72 lines

Regards,
Lucas

Quoted 66 lines
---
 drivers/gpu/drm/msm/msm_drv.c        |  3 ++-
 drivers/gpu/drm/msm/msm_gem_submit.c | 11 +++++++++++
 include/uapi/drm/msm_drm.h           |  4 +++-
 3 files changed, 16 insertions(+), 2 deletions(-)
drivers/gpu/drm/msm/msm_drv.c +2 -1
index 017a512982a2..e0e1199a822f 100644
@@ -45,9 +45,10 @@
  * - 1.7.0 - Add MSM_PARAM_SUSPENDS to access suspend count
  * - 1.8.0 - Add MSM_BO_CACHED_COHERENT for supported GPUs (a6xx)
  * - 1.9.0 - Add MSM_SUBMIT_FENCE_SN_IN
+ * - 1.10.0 - Add MSM_SUBMIT_BO_NO_IMPLICIT
  */
 #define MSM_VERSION_MAJOR    1
-#define MSM_VERSION_MINOR    9
+#define MSM_VERSION_MINOR    10
 #define MSM_VERSION_PATCHLEVEL       0
 
 static const struct drm_mode_config_funcs mode_config_funcs = {
drivers/gpu/drm/msm/msm_gem_submit.c +11 -0
index eb3536e3d66a..8bad07a04f85 100644
@@ -334,9 +334,20 @@ static int submit_fence_sync(struct msm_gem_submit *submit, bool no_implicit)
              if (ret)
                      return ret;
 
+             /* If userspace has determined that explicit fencing is
+              * used, it can disable implicit sync on the entire
+              * submit:
+              */
              if (no_implicit)
                      continue;
 
+             /* Otherwise userspace can ask for implicit sync to be
+              * disabled on specific buffers.  This is useful for internal
+              * usermode driver managed buffers, suballocation, etc.
+              */
+             if (submit->bos[i].flags & MSM_SUBMIT_BO_NO_IMPLICIT)
+                     continue;
+
              ret = drm_sched_job_add_implicit_dependencies(&submit->base,
                                                            obj,
                                                            write);
include/uapi/drm/msm_drm.h +3 -1
index f54b48ef6a2d..329100016e7c 100644
@@ -222,10 +222,12 @@ struct drm_msm_gem_submit_cmd {
 #define MSM_SUBMIT_BO_READ             0x0001
 #define MSM_SUBMIT_BO_WRITE            0x0002
 #define MSM_SUBMIT_BO_DUMP             0x0004
+#define MSM_SUBMIT_BO_NO_IMPLICIT      0x0008
 
 #define MSM_SUBMIT_BO_FLAGS            (MSM_SUBMIT_BO_READ | \
                                      MSM_SUBMIT_BO_WRITE | \
-                                     MSM_SUBMIT_BO_DUMP)
+                                     MSM_SUBMIT_BO_DUMP | \
+                                     MSM_SUBMIT_BO_NO_IMPLICIT)
 
 struct drm_msm_gem_submit_bo {
      __u32 flags;          /* in, mask of MSM_SUBMIT_BO_x */
#3 lore raw
In reply to Rob Clark
Quoted 42 lines · Rob Clark

From: Rob Clark <[email protected]>

In cases where implicit sync is used, it is still useful (for things like sub-allocation, etc) to allow userspace to opt-out of implicit sync on per-BO basis.

---
 drivers/gpu/drm/msm/msm_drv.c        |  3 ++-
 drivers/gpu/drm/msm/msm_gem_submit.c | 11 +++++++++++
 include/uapi/drm/msm_drm.h           |  4 +++-
 3 files changed, 16 insertions(+), 2 deletions(-)
drivers/gpu/drm/msm/msm_drv.c +2 -1
index 017a512982a2..e0e1199a822f 100644
@@ -45,9 +45,10 @@
  * - 1.7.0 - Add MSM_PARAM_SUSPENDS to access suspend count
  * - 1.8.0 - Add MSM_BO_CACHED_COHERENT for supported GPUs (a6xx)
  * - 1.9.0 - Add MSM_SUBMIT_FENCE_SN_IN
+ * - 1.10.0 - Add MSM_SUBMIT_BO_NO_IMPLICIT
  */
 #define MSM_VERSION_MAJOR	1
-#define MSM_VERSION_MINOR	9
+#define MSM_VERSION_MINOR	10
 #define MSM_VERSION_PATCHLEVEL	0
 
 static const struct drm_mode_config_funcs mode_config_funcs = {
drivers/gpu/drm/msm/msm_gem_submit.c +4 -0
index eb3536e3d66a..8bad07a04f85 100644
@@ -334,9 +334,20 @@ static int submit_fence_sync(struct msm_gem_submit *submit, bool no_implicit)
 		if (ret)
 			return ret;
 
+		/* If userspace has determined that explicit fencing is
+		 * used, it can disable implicit sync on the entire
+		 * submit:
+		 */
 		if (no_implicit)

drivers/gpu/drm/msm/msm_gem_submit.c static int submit_fence_sync(struct msm_gem_submit *submit, bool no_implicit) · line 351

Quoted 10 lines
drivers/gpu/drm/msm/msm_gem_submit.c +7 -0
 			continue;
 
+		/* Otherwise userspace can ask for implicit sync to be
+		 * disabled on specific buffers.  This is useful for internal
+		 * usermode driver managed buffers, suballocation, etc.
+		 */
+		if (submit->bos[i].flags & MSM_SUBMIT_BO_NO_IMPLICIT)
+			continue;
+
 		ret = drm_sched_job_add_implicit_dependencies(&submit->base,

Won't this break shrinkers and fun stuff like that? It's why we added the new USAGE_OTHER fence slot at least, and also why I wonder whether we shouldn't push this into the helper to make the right call. Every driver kinda needs the same wheel.
-Daniel

Quoted 23 lines
drivers/gpu/drm/msm/msm_gem_submit.c +0 -0
 							      obj,
 							      write);
include/uapi/drm/msm_drm.h +3 -1
index f54b48ef6a2d..329100016e7c 100644
@@ -222,10 +222,12 @@ struct drm_msm_gem_submit_cmd {
 #define MSM_SUBMIT_BO_READ             0x0001
 #define MSM_SUBMIT_BO_WRITE            0x0002
 #define MSM_SUBMIT_BO_DUMP             0x0004
+#define MSM_SUBMIT_BO_NO_IMPLICIT      0x0008
 
 #define MSM_SUBMIT_BO_FLAGS            (MSM_SUBMIT_BO_READ | \
 					MSM_SUBMIT_BO_WRITE | \
-					MSM_SUBMIT_BO_DUMP)
+					MSM_SUBMIT_BO_DUMP | \
+					MSM_SUBMIT_BO_NO_IMPLICIT)
 
 struct drm_msm_gem_submit_bo {
 	__u32 flags;          /* in, mask of MSM_SUBMIT_BO_x */
Signature · 1 line
2.38.1
Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author
#4 lore raw
Quoted 46 lines · Daniel Vetter
Quoted 44 lines · Rob Clark

From: Rob Clark <[email protected]>

In cases where implicit sync is used, it is still useful (for things like sub-allocation, etc) to allow userspace to opt-out of implicit sync on per-BO basis.

---
 drivers/gpu/drm/msm/msm_drv.c        |  3 ++-
 drivers/gpu/drm/msm/msm_gem_submit.c | 11 +++++++++++
 include/uapi/drm/msm_drm.h           |  4 +++-
 3 files changed, 16 insertions(+), 2 deletions(-)
drivers/gpu/drm/msm/msm_drv.c +2 -1
index 017a512982a2..e0e1199a822f 100644
@@ -45,9 +45,10 @@
  * - 1.7.0 - Add MSM_PARAM_SUSPENDS to access suspend count
  * - 1.8.0 - Add MSM_BO_CACHED_COHERENT for supported GPUs (a6xx)
  * - 1.9.0 - Add MSM_SUBMIT_FENCE_SN_IN
+ * - 1.10.0 - Add MSM_SUBMIT_BO_NO_IMPLICIT
  */
 #define MSM_VERSION_MAJOR    1
-#define MSM_VERSION_MINOR    9
+#define MSM_VERSION_MINOR    10
 #define MSM_VERSION_PATCHLEVEL       0
 
 static const struct drm_mode_config_funcs mode_config_funcs = {
drivers/gpu/drm/msm/msm_gem_submit.c +4 -0
index eb3536e3d66a..8bad07a04f85 100644
@@ -334,9 +334,20 @@ static int submit_fence_sync(struct msm_gem_submit *submit, bool no_implicit)
              if (ret)
                      return ret;
 
+             /* If userspace has determined that explicit fencing is
+              * used, it can disable implicit sync on the entire
+              * submit:
+              */
              if (no_implicit)
                      continue;
 
Quoted 11 lines

drivers/gpu/drm/msm/msm_gem_submit.c static int submit_fence_sync(struct msm_gem_submit *submit, bool no_implicit) · line 351

Quoted 8 lines
drivers/gpu/drm/msm/msm_gem_submit.c +7 -0
+             /* Otherwise userspace can ask for implicit sync to be
+              * disabled on specific buffers.  This is useful for internal
+              * usermode driver managed buffers, suballocation, etc.
+              */
+             if (submit->bos[i].flags & MSM_SUBMIT_BO_NO_IMPLICIT)
+                     continue;
+
              ret = drm_sched_job_add_implicit_dependencies(&submit->base,

Won't this break shrinkers and fun stuff like that? It's why we added the new USAGE_OTHER fence slot at least, and also why I wonder whether we

Only if the entire explicit sync path was busted.. My daily driver for email/docs/meet/chat/corpstuff is a 4G device and CrOS is all explicit sync.. I would have found out rapidly and dramatically if it was busted :-P

But seriously, this doesn't change what fences we attach to buffers, only what the sched job waits on

Quoted 2 lines

shouldn't push this into the helper to make the right call. Every driver kinda needs the same wheel.

We kinda already have moved everything we can (with the current driver-specific-uabi model) to helpers, what is left is driver specific ioctl parsing. We absolutely should take a step back and re-evaluate this before anyone else adds a new submit/execbuf ioctl. For example, the driver specific ioctl could just have a pointer to a drm_gem_submit_bo_table type structure, and then we could move the whole thing to a helper. Short of breaking the submit ioctl up (which a uring type uabi would let us do), I think the next best thing is to split out common cross-driver structs for common parts of submit/execbuf.

BR,
-R

Quoted 30 lines

-Daniel

Quoted 21 lines
drivers/gpu/drm/msm/msm_gem_submit.c +0 -0
                                                            obj,
                                                            write);
include/uapi/drm/msm_drm.h +3 -2
index f54b48ef6a2d..329100016e7c 100644
@@ -222,10 +222,12 @@ struct drm_msm_gem_submit_cmd {
 #define MSM_SUBMIT_BO_READ             0x0001
 #define MSM_SUBMIT_BO_WRITE            0x0002
 #define MSM_SUBMIT_BO_DUMP             0x0004
+#define MSM_SUBMIT_BO_NO_IMPLICIT      0x0008
 
 #define MSM_SUBMIT_BO_FLAGS            (MSM_SUBMIT_BO_READ | \
                                      MSM_SUBMIT_BO_WRITE | \
-                                     MSM_SUBMIT_BO_DUMP)
+                                     MSM_SUBMIT_BO_DUMP | \
+                                     MSM_SUBMIT_BO_NO_IMPLICIT)
 
 struct drm_msm_gem_submit_bo {
      __u32 flags;          /* in, mask of MSM_SUBMIT_BO_x */
--
Quoted 2 lines

2.38.1

--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

#5 lore raw
Quoted 60 lines · Rob Clark
Quoted 46 lines · Daniel Vetter
Quoted 44 lines · Rob Clark

From: Rob Clark <[email protected]>

In cases where implicit sync is used, it is still useful (for things like sub-allocation, etc) to allow userspace to opt-out of implicit sync on per-BO basis.

---
 drivers/gpu/drm/msm/msm_drv.c        |  3 ++-
 drivers/gpu/drm/msm/msm_gem_submit.c | 11 +++++++++++
 include/uapi/drm/msm_drm.h           |  4 +++-
 3 files changed, 16 insertions(+), 2 deletions(-)
drivers/gpu/drm/msm/msm_drv.c +2 -1
index 017a512982a2..e0e1199a822f 100644
@@ -45,9 +45,10 @@
  * - 1.7.0 - Add MSM_PARAM_SUSPENDS to access suspend count
  * - 1.8.0 - Add MSM_BO_CACHED_COHERENT for supported GPUs (a6xx)
  * - 1.9.0 - Add MSM_SUBMIT_FENCE_SN_IN
+ * - 1.10.0 - Add MSM_SUBMIT_BO_NO_IMPLICIT
  */
 #define MSM_VERSION_MAJOR    1
-#define MSM_VERSION_MINOR    9
+#define MSM_VERSION_MINOR    10
 #define MSM_VERSION_PATCHLEVEL       0
 
 static const struct drm_mode_config_funcs mode_config_funcs = {
drivers/gpu/drm/msm/msm_gem_submit.c +4 -0
index eb3536e3d66a..8bad07a04f85 100644
@@ -334,9 +334,20 @@ static int submit_fence_sync(struct msm_gem_submit *submit, bool no_implicit)
              if (ret)
                      return ret;
 
+             /* If userspace has determined that explicit fencing is
+              * used, it can disable implicit sync on the entire
+              * submit:
+              */
              if (no_implicit)
                      continue;
 
Quoted 11 lines

drivers/gpu/drm/msm/msm_gem_submit.c static int submit_fence_sync(struct msm_gem_submit *submit, bool no_implicit) · line 351

Quoted 8 lines
drivers/gpu/drm/msm/msm_gem_submit.c +7 -0
+             /* Otherwise userspace can ask for implicit sync to be
+              * disabled on specific buffers.  This is useful for internal
+              * usermode driver managed buffers, suballocation, etc.
+              */
+             if (submit->bos[i].flags & MSM_SUBMIT_BO_NO_IMPLICIT)
+                     continue;
+
              ret = drm_sched_job_add_implicit_dependencies(&submit->base,

Won't this break shrinkers and fun stuff like that? It's why we added the new USAGE_OTHER fence slot at least, and also why I wonder whether we

Quoted 7 lines

Only if the entire explicit sync path was busted.. My daily driver for email/docs/meet/chat/corpstuff is a 4G device and CrOS is all explicit sync.. I would have found out rapidly and dramatically if it was busted :-P

But seriously, this doesn't change what fences we attach to buffers, only what the sched job waits on

Oh I'm a silly one :-)

Quoted 13 lines
Quoted 2 lines

shouldn't push this into the helper to make the right call. Every driver kinda needs the same wheel.

We kinda already have moved everything we can (with the current driver-specific-uabi model) to helpers, what is left is driver specific ioctl parsing. We absolutely should take a step back and re-evaluate this before anyone else adds a new submit/execbuf ioctl. For example, the driver specific ioctl could just have a pointer to a drm_gem_submit_bo_table type structure, and then we could move the whole thing to a helper. Short of breaking the submit ioctl up (which a uring type uabi would let us do), I think the next best thing is to split out common cross-driver structs for common parts of submit/execbuf.

Yeah I thought this was the "attach fence to dma_resv" side of things. This here is I think fine as a pattern, unless we build some kind of outright submit ioctl helpers that take care of everything. -Daniel

Quoted 34 lines

BR,
-R

Quoted 30 lines

-Daniel

Quoted 21 lines
drivers/gpu/drm/msm/msm_gem_submit.c +0 -0
                                                            obj,
                                                            write);
include/uapi/drm/msm_drm.h +3 -2
index f54b48ef6a2d..329100016e7c 100644
@@ -222,10 +222,12 @@ struct drm_msm_gem_submit_cmd {
 #define MSM_SUBMIT_BO_READ             0x0001
 #define MSM_SUBMIT_BO_WRITE            0x0002
 #define MSM_SUBMIT_BO_DUMP             0x0004
+#define MSM_SUBMIT_BO_NO_IMPLICIT      0x0008
 
 #define MSM_SUBMIT_BO_FLAGS            (MSM_SUBMIT_BO_READ | \
                                      MSM_SUBMIT_BO_WRITE | \
-                                     MSM_SUBMIT_BO_DUMP)
+                                     MSM_SUBMIT_BO_DUMP | \
+                                     MSM_SUBMIT_BO_NO_IMPLICIT)
 
 struct drm_msm_gem_submit_bo {
      __u32 flags;          /* in, mask of MSM_SUBMIT_BO_x */
--
Quoted 2 lines

2.38.1

--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch