Skip to contentSkip to conversation

drm/amd/amdgpu: locking fixes

PATCH Superseded Nicolai Hähnle 3 participants 9 messages Jan 11, 2017 5 messages not in the archive

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover drm/amd/amdgpu: locking fixes

Hi all,

two fixes for locking issues that I noticed.

The first one is something that I actually encountered live; it probably only matters when lock debugging is enabled, but obviously needs to be fixed anyway.

The second one I only noticed upon reading the code -- I haven't seen it fail live yet, at least not with lock debugging enabled (who knows what its effects could be without lock debugging).

Please review!

I noticed two more locking-related problems, but since I'm not sure whether I'll get to those this week, I'm sending out these two patches already. The other two problems are (both of them probably mostly benign, but I did encounter them in live tests):

1. amdgpu_bo_create_restricted(resv == NULL) --> ttm_bo_init will free a reservation object while it is locked. Fixing this soundly probably requires mild changes to the ttm_bo_init API (perhaps a separate ttm_bo_init_locked?).

2. ttm_bo_pipeline_move will ttm_bo_unref(&ghost_obj) leading to mutex_destroy(&ghost_obj->wu_mutex) while that mutex "looks" locked (due to the dumb copy).

Thanks,
Nicolai

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1484119865-11390-1-git-send-email-nhaehnle-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

#2 lore raw

PATCH 1/2 drm/amd/amdgpu: lock reservation object while creating shadow bo

From: Nicolai Hähnle <[email protected]>

ttm_bo_init checks that the reservation object is locked. This is the caller's responsibility when resv != NULL. Otherwise, the inline reservation object of the newly allocated buffer is used and must explicitly be locked. Using a trylock is fine, since nobody else has a reference to the reservation lock.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +11 -0
index 11c12c4d..357eed9 100644
@@ -465,21 +465,32 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
 	amdgpu_ttm_placement_init(adev, &placement,
 				  placements, domain, flags);
 
 	r = amdgpu_bo_create_restricted(adev, size, byte_align, kernel,
 					domain, flags, sg, &placement,
 					resv, bo_ptr);
 	if (r)
 		return r;
 
 	if (amdgpu_need_backup(adev) && (flags & AMDGPU_GEM_CREATE_SHADOW)) {
+		if (!resv) {
+			bool locked;
+
+			locked = ww_mutex_trylock(&(*bo_ptr)->tbo.resv->lock);
+			WARN_ON(!locked);
+		}
+
 		r = amdgpu_bo_create_shadow(adev, size, byte_align, (*bo_ptr));
+
+		if (!resv)
+			ww_mutex_unlock(&(*bo_ptr)->tbo.resv->lock);
+
 		if (r)
 			amdgpu_bo_unref(bo_ptr);
 	}
 
 	return r;
 }
 
 int amdgpu_bo_backup_to_shadow(struct amdgpu_device *adev,
 			       struct amdgpu_ring *ring,
 			       struct amdgpu_bo *bo,
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1484119865-11390-2-git-send-email-nhaehnle-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

#4 lore raw

Re: [PATCH 1/2] drm/amd/amdgpu: lock reservation object while creating shadow bo

Am 11.01.2017 um 08:31 schrieb Nicolai Hähnle:

Quoted 7 lines

From: Nicolai Hähnle <[email protected]>

ttm_bo_init checks that the reservation object is locked. This is the caller's responsibility when resv != NULL. Otherwise, the inline reservation object of the newly allocated buffer is used and must explicitly be locked. Using a trylock is fine, since nobody else has a reference to the reservation lock.

Good, catch. But while using trylock is fine, why do you need to use it in the first place?

Regards,
Christian.

Quoted 33 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +11 -0
index 11c12c4d..357eed9 100644
@@ -465,21 +465,32 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
  	amdgpu_ttm_placement_init(adev, &placement,
  				  placements, domain, flags);
 
  	r = amdgpu_bo_create_restricted(adev, size, byte_align, kernel,
  					domain, flags, sg, &placement,
  					resv, bo_ptr);
  	if (r)
  		return r;
 
  	if (amdgpu_need_backup(adev) && (flags & AMDGPU_GEM_CREATE_SHADOW)) {
+		if (!resv) {
+			bool locked;
+
+			locked = ww_mutex_trylock(&(*bo_ptr)->tbo.resv->lock);
+			WARN_ON(!locked);
+		}
+
  		r = amdgpu_bo_create_shadow(adev, size, byte_align, (*bo_ptr));
+
+		if (!resv)
+			ww_mutex_unlock(&(*bo_ptr)->tbo.resv->lock);
+

drivers/gpu/drm/amd/amdgpu/amdgpu_object.c int amdgpu_bo_create(struct amdgpu_device *adev, · line 496

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +0 -0
  		if (r)
  			amdgpu_bo_unref(bo_ptr);
  	}
 
  	return r;
  }
 
  int amdgpu_bo_backup_to_shadow(struct amdgpu_device *adev,
  			       struct amdgpu_ring *ring,
  			       struct amdgpu_bo *bo,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 621bf6ab-ad3b-7655-c633-d18c051f4667-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org

#6 lore raw

Re: [PATCH 1/2] drm/amd/amdgpu: lock reservation object while creating shadow bo

Quoted 11 lines · Christian König

Am 11.01.2017 um 08:31 schrieb Nicolai Hähnle:

Quoted 7 lines

From: Nicolai Hähnle <[email protected]>

ttm_bo_init checks that the reservation object is locked. This is the caller's responsibility when resv != NULL. Otherwise, the inline reservation object of the newly allocated buffer is used and must explicitly be locked. Using a trylock is fine, since nobody else has a reference to the reservation lock.

Good, catch. But while using trylock is fine, why do you need to use it in the first place?

(*bo_ptr)->tbo.resv is passed as the reservation object to ttm_bo_init via amdgpu_bo_create_shadow, and ttm_bo_init has a lockdep check for it.

I haven't dug deeper whether that's _really_ necessary, but since among other things ttm_bo_validate is called by ttm_bo_init, I'd assume that it is (ttm_bo_validate also has a lockdep check). If it isn't, then the internal locking of bo->resv->lock done by ttm_bo_init in the !resv case would also be unnecessary...

Cheers,
Nicolai

Quoted 37 lines

Regards,
Christian.

Quoted 33 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +11 -0
index 11c12c4d..357eed9 100644
@@ -465,21 +465,32 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
      amdgpu_ttm_placement_init(adev, &placement,
                    placements, domain, flags);
        r = amdgpu_bo_create_restricted(adev, size, byte_align, kernel,
                      domain, flags, sg, &placement,
                      resv, bo_ptr);
      if (r)
          return r;
        if (amdgpu_need_backup(adev) && (flags &
 AMDGPU_GEM_CREATE_SHADOW)) {
+        if (!resv) {
+            bool locked;
+
+            locked = ww_mutex_trylock(&(*bo_ptr)->tbo.resv->lock);
+            WARN_ON(!locked);
+        }
+
          r = amdgpu_bo_create_shadow(adev, size, byte_align, (*bo_ptr));
+
+        if (!resv)
+            ww_mutex_unlock(&(*bo_ptr)->tbo.resv->lock);
+
Quoted 10 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_object.c int amdgpu_bo_create(struct amdgpu_device *adev, · line 496

Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +0 -0
          if (r)
              amdgpu_bo_unref(bo_ptr);
      }
        return r;
  }
    int amdgpu_bo_backup_to_shadow(struct amdgpu_device *adev,
                     struct amdgpu_ring *ring,
                     struct amdgpu_bo *bo,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#7 lore raw

Re: [PATCH 1/2] drm/amd/amdgpu: lock reservation object while creating shadow bo

In reply to a message not in the archive
Quoted 11 lines · Christian König

Am 11.01.2017 um 08:31 schrieb Nicolai Hähnle:

Quoted 7 lines

From: Nicolai Hähnle <[email protected]>

ttm_bo_init checks that the reservation object is locked. This is the caller's responsibility when resv != NULL. Otherwise, the inline reservation object of the newly allocated buffer is used and must explicitly be locked. Using a trylock is fine, since nobody else has a reference to the reservation lock.

Good, catch. But while using trylock is fine, why do you need to use it in the first place?

Or is this a question about trylock vs. lock?

Actually, now that I think about this again, perhaps the following sequence would be possible:

1. Create the main bo in amdgpu_bo_create.
2. Other thread, for whatever reason, tries to make space and evict the bo, and probably locks the reservation for that?
3. amdgpu_bo_create continues, calls the trylock which now fails.

Nicolai

Quoted 37 lines

Regards,
Christian.

Quoted 33 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +11 -0
index 11c12c4d..357eed9 100644
@@ -465,21 +465,32 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
      amdgpu_ttm_placement_init(adev, &placement,
                    placements, domain, flags);
        r = amdgpu_bo_create_restricted(adev, size, byte_align, kernel,
                      domain, flags, sg, &placement,
                      resv, bo_ptr);
      if (r)
          return r;
        if (amdgpu_need_backup(adev) && (flags &
 AMDGPU_GEM_CREATE_SHADOW)) {
+        if (!resv) {
+            bool locked;
+
+            locked = ww_mutex_trylock(&(*bo_ptr)->tbo.resv->lock);
+            WARN_ON(!locked);
+        }
+
          r = amdgpu_bo_create_shadow(adev, size, byte_align, (*bo_ptr));
+
+        if (!resv)
+            ww_mutex_unlock(&(*bo_ptr)->tbo.resv->lock);
+
Quoted 10 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_object.c int amdgpu_bo_create(struct amdgpu_device *adev, · line 496

Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +0 -0
          if (r)
              amdgpu_bo_unref(bo_ptr);
      }
        return r;
  }
    int amdgpu_bo_backup_to_shadow(struct amdgpu_device *adev,
                     struct amdgpu_ring *ring,
                     struct amdgpu_bo *bo,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 56e05f07-2b21-5fc9-7d69-9acfe6d0b519-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

#9 lore raw

Re: [PATCH 1/2] drm/amd/amdgpu: lock reservation object while creating shadow bo

Am 11.01.2017 um 15:09 schrieb Nicolai Hähnle:

Quoted 13 lines · Christian König
Quoted 11 lines

Am 11.01.2017 um 08:31 schrieb Nicolai Hähnle:

Quoted 7 lines

From: Nicolai Hähnle <[email protected]>

ttm_bo_init checks that the reservation object is locked. This is the caller's responsibility when resv != NULL. Otherwise, the inline reservation object of the newly allocated buffer is used and must explicitly be locked. Using a trylock is fine, since nobody else has a reference to the reservation lock.

Good, catch. But while using trylock is fine, why do you need to use it in the first place?

Quoted 7 lines

Or is this a question about trylock vs. lock?

Actually, now that I think about this again, perhaps the following sequence would be possible:

1. Create the main bo in amdgpu_bo_create.
2. Other thread, for whatever reason, tries to make space and evict the bo, and probably locks the reservation for that?
3. amdgpu_bo_create continues, calls the trylock which now fails.

Yes, exactly that was my thinking as well.

Please just use the normal uninterruptible locking variant if there isn't a good reason for using trylock.

Christian.

Quoted 51 lines

Nicolai

Quoted 38 lines

Regards,
Christian.

Quoted 34 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +11 -0
index 11c12c4d..357eed9 100644
@@ -465,21 +465,32 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
      amdgpu_ttm_placement_init(adev, &placement,
                    placements, domain, flags);
        r = amdgpu_bo_create_restricted(adev, size, byte_align, kernel,
                      domain, flags, sg, &placement,
                      resv, bo_ptr);
      if (r)
          return r;
        if (amdgpu_need_backup(adev) && (flags &
 AMDGPU_GEM_CREATE_SHADOW)) {
+        if (!resv) {
+            bool locked;
+
+            locked = ww_mutex_trylock(&(*bo_ptr)->tbo.resv->lock);
+            WARN_ON(!locked);
+        }
+
          r = amdgpu_bo_create_shadow(adev, size, byte_align, 
 (*bo_ptr));
+
+        if (!resv)
+ ww_mutex_unlock(&(*bo_ptr)->tbo.resv->lock);
+
Quoted 10 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_object.c int amdgpu_bo_create(struct amdgpu_device *adev, · line 496

Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +0 -0
          if (r)
              amdgpu_bo_unref(bo_ptr);
      }
        return r;
  }
    int amdgpu_bo_backup_to_shadow(struct amdgpu_device *adev,
                     struct amdgpu_ring *ring,
                     struct amdgpu_bo *bo,
Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#10 lore raw

PATCH 2/2 drm/amd/amdgpu: fix locking in bo creation error path

In reply to a message not in the archive

From: Nicolai Hähnle <[email protected]>

Unlock the resv lock only if we were the ones to lock it in the first place.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +2 -1
index 357eed9..d9d6793 100644
@@ -401,21 +401,22 @@ int amdgpu_bo_create_restricted(struct amdgpu_device *adev,
 	}
 	if (!resv)
 		ww_mutex_unlock(&bo->tbo.resv->lock);
 	*bo_ptr = bo;
 
 	trace_amdgpu_bo_create(bo);
 
 	return 0;
 
 fail_unreserve:
-	ww_mutex_unlock(&bo->tbo.resv->lock);
+	if (!resv)
+		ww_mutex_unlock(&bo->tbo.resv->lock);
 	amdgpu_bo_unref(&bo);
 	return r;
 }
 
 static int amdgpu_bo_create_shadow(struct amdgpu_device *adev,
 				   unsigned long size, int byte_align,
 				   struct amdgpu_bo *bo)
 {
 	struct ttm_placement placement = {0};
 	struct ttm_place placements[AMDGPU_GEM_DOMAIN_MAX + 1];
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1484119865-11390-3-git-send-email-nhaehnle-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

Reviewed-by
#12 lore raw

Re: [PATCH 2/2] drm/amd/amdgpu: fix locking in bo creation error path

Am 11.01.2017 um 08:31 schrieb Nicolai Hähnle:

Quoted 6 lines

From: Nicolai Hähnle <[email protected]>

Unlock the resv lock only if we were the ones to lock it in the first place.

Quoted 22 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +2 -1
index 357eed9..d9d6793 100644
@@ -401,21 +401,22 @@ int amdgpu_bo_create_restricted(struct amdgpu_device *adev,
  	}
  	if (!resv)
  		ww_mutex_unlock(&bo->tbo.resv->lock);
  	*bo_ptr = bo;
 
  	trace_amdgpu_bo_create(bo);
 
  	return 0;
 
  fail_unreserve:
-	ww_mutex_unlock(&bo->tbo.resv->lock);
+	if (!resv)
+		ww_mutex_unlock(&bo->tbo.resv->lock);

drivers/gpu/drm/amd/amdgpu/amdgpu_object.c int amdgpu_bo_create_restricted(struct amdgpu_device *adev, · line 422

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +0 -0
  	amdgpu_bo_unref(&bo);
  	return r;
  }
 
  static int amdgpu_bo_create_shadow(struct amdgpu_device *adev,
  				   unsigned long size, int byte_align,
  				   struct amdgpu_bo *bo)
  {
  	struct ttm_placement placement = {0};
  	struct ttm_place placements[AMDGPU_GEM_DOMAIN_MAX + 1];

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Reviewed-by
#13 lore raw
In reply to a message not in the archive

this series is,

Quoted 28 lines · Nicolai Hähnle

Hi all,

two fixes for locking issues that I noticed.

The first one is something that I actually encountered live; it probably only matters when lock debugging is enabled, but obviously needs to be fixed anyway.

The second one I only noticed upon reading the code -- I haven't seen it fail live yet, at least not with lock debugging enabled (who knows what its effects could be without lock debugging).

Please review!

I noticed two more locking-related problems, but since I'm not sure whether I'll get to those this week, I'm sending out these two patches already. The other two problems are (both of them probably mostly benign, but I did encounter them in live tests):

1. amdgpu_bo_create_restricted(resv == NULL) --> ttm_bo_init will free a reservation object while it is locked. Fixing this soundly probably requires mild changes to the ttm_bo_init API (perhaps a separate ttm_bo_init_locked?).

2. ttm_bo_pipeline_move will ttm_bo_unref(&ghost_obj) leading to mutex_destroy(&ghost_obj->wu_mutex) while that mutex "looks" locked (due to the dumb copy).

Quoted 8 lines

Thanks,
Nicolai

_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx