Skip to contentSkip to conversation

drm/ttm: set ttm_buffer_object pointer as null after it's freed

PATCH In review Huang Rui 5 participants 13 messages Sep 10, 2018 – Sep 11, 2018 7 messages not in the archive

Subsystems (4)
lore
Share & tools
lore

Huang Rui

author
#0 lore raw

PATCH 1/2 drm/ttm: set ttm_buffer_object pointer as null after it's freed

It avoids to be refered again after freed.

Cc Christian König <[email protected]>
Cc Tom StDenis <[email protected]>
---
 drivers/gpu/drm/ttm/ttm_bo.c | 1 +
 1 file changed, 1 insertion(+)
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
index 138c989..d3ef5f8 100644
@@ -54,6 +54,7 @@ static struct attribute ttm_bo_count = {
 static void ttm_bo_default_destroy(struct ttm_buffer_object *bo)
 {
 	kfree(bo);
+	bo = NULL;
 }
 
 static inline int ttm_mem_type_from_place(const struct ttm_place *place,
Signature · 1 line
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1536569876-27262-1-git-send-email-ray.huang-5C7GfCeVMHo@public.gmane.org

author
#2 lore raw

PATCH 2/2 drm/amdgpu: set tbo pointer as null after amdgpu bo is freed

The tbo pointer will still have value even the amdgpu bo is freed. It doesn't make sense that it still points a freed memory. It could be refered mistakenly, so set it as null.

Cc Christian König <[email protected]>
Cc Tom StDenis <[email protected]>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 1 +
 1 file changed, 1 insertion(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +1 -0
index de990bd..ae84c08 100644
@@ -108,6 +108,7 @@ static void amdgpu_bo_destroy(struct ttm_buffer_object *tbo)
 	}
 	kfree(bo->metadata);
 	kfree(bo);
+	tbo = NULL;
 }
 
 /**
Signature · 1 line
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#3 lore raw
In reply to a message not in the archive

Hi Ray,

well those patches doesn't make sense, the pointer is only local to the function.

Regards,
Christian.

Am 10.09.2018 um 10:57 schrieb Huang Rui:

Quoted 14 lines

It avoids to be refered again after freed.

Cc Christian König <[email protected]>
Cc Tom StDenis <[email protected]>
---
  drivers/gpu/drm/ttm/ttm_bo.c | 1 +
  1 file changed, 1 insertion(+)
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
index 138c989..d3ef5f8 100644

drivers/gpu/drm/ttm/ttm_bo.c static struct attribute ttm_bo_count = { · line 60

Quoted 7 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
@@ -54,6 +54,7 @@ static struct attribute ttm_bo_count = {
  static void ttm_bo_default_destroy(struct ttm_buffer_object *bo)
  {
  	kfree(bo);
+	bo = NULL;
  }
 
  static inline int ttm_mem_type_from_place(const struct ttm_place *place,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): d40314d2-f1c4-b353-bda6-50aed5dda8db-5C7GfCeVMHo@public.gmane.org

author
#5 lore raw
Quoted 4 lines · Christian König

Hi Ray,

well those patches doesn't make sense, the pointer is only local to the function.

You're right.
I narrowed it with gdb dump from ttm_bo_bulk_move_lru_tail+0x2b, the use-after-free should be in below codes:

man = &bulk->tt[i].first->bdev->man[TTM_PL_TT]; ttm_bo_bulk_move_helper(&bulk->tt[i], &man->lru[i], false);

Is there a case, when orignal bo is destroyed in the bulk pos, but it doesn't update pos->first pointer, then we still use it during the bulk moving?

Thanks,
Ray

Quoted 14 lines

Regards,
Christian.

Am 10.09.2018 um 10:57 schrieb Huang Rui:

Quoted 9 lines

It avoids to be refered again after freed.

Cc Christian König <[email protected]>
Cc Tom StDenis <[email protected]>
---
 drivers/gpu/drm/ttm/ttm_bo.c | 1 +
 1 file changed, 1 insertion(+)
Quoted 16 lines

drivers/gpu/drm/ttm/ttm_bo.c static struct attribute ttm_bo_count = { · line 59

Quoted 11 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
index 138c989..d3ef5f8 100644
@@ -54,6 +54,7 @@ static struct attribute ttm_bo_count = {
 static void ttm_bo_default_destroy(struct ttm_buffer_object *bo)
 {
 	kfree(bo);
+	bo = NULL;
 }
 static inline int ttm_mem_type_from_place(const struct ttm_place *place,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#6 lore raw

Am 10.09.2018 um 11:23 schrieb Huang Rui:

Quoted 9 lines · Christian König
Quoted 4 lines

Hi Ray,

well those patches doesn't make sense, the pointer is only local to the function.

You're right.
I narrowed it with gdb dump from ttm_bo_bulk_move_lru_tail+0x2b, the use-after-free should be in below codes:

Quoted 6 lines

man = &bulk->tt[i].first->bdev->man[TTM_PL_TT]; ttm_bo_bulk_move_helper(&bulk->tt[i], &man->lru[i], false);

Is there a case, when orignal bo is destroyed in the bulk pos, but it doesn't update pos->first pointer, then we still use it during the bulk moving?

Only when a per VM BO is freed or the VM destroyed.

The first case should now be handled by "drm/amdgpu: set bulk_moveable to false when a per VM is released" and when we use a destroyed VM we would see other problems as well.

BTW: Just pushed this commit to the repository, should show up any second.

Christian.

Quoted 32 lines

Thanks,
Ray

Quoted 28 lines

Regards,
Christian.

Am 10.09.2018 um 10:57 schrieb Huang Rui:

Quoted 14 lines

It avoids to be refered again after freed.

Cc Christian König <[email protected]>
Cc Tom StDenis <[email protected]>
---
  drivers/gpu/drm/ttm/ttm_bo.c | 1 +
  1 file changed, 1 insertion(+)
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
index 138c989..d3ef5f8 100644

drivers/gpu/drm/ttm/ttm_bo.c static struct attribute ttm_bo_count = { · line 59

Quoted 6 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
@@ -54,6 +54,7 @@ static struct attribute ttm_bo_count = {
  static void ttm_bo_default_destroy(struct ttm_buffer_object *bo)
  {
  	kfree(bo);
+	bo = NULL;
  }
  static inline int ttm_mem_type_from_place(const struct ttm_place *place,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#7 lore raw
Quoted 17 lines · Christian

Am 10.09.2018 um 11:23 schrieb Huang Rui:

Quoted 9 lines · Christian König
Quoted 4 lines

Hi Ray,

well those patches doesn't make sense, the pointer is only local to the function.

You're right.
I narrowed it with gdb dump from ttm_bo_bulk_move_lru_tail+0x2b, the use-after-free should be in below codes:

Quoted 6 lines

man = &bulk->tt[i].first->bdev->man[TTM_PL_TT]; ttm_bo_bulk_move_helper(&bulk->tt[i], &man->lru[i], false);

Is there a case, when orignal bo is destroyed in the bulk pos, but it doesn't update pos->first pointer, then we still use it during the bulk moving?

Quoted 5 lines

Only when a per VM BO is freed or the VM destroyed.

The first case should now be handled by "drm/amdgpu: set bulk_moveable to false when a per VM is released" and when we use a destroyed VM we would see other problems as well.

When a VM instance is teared down, all BOs which belong to that VM should be removed from LRU list. How can we use a destroyed VM when we submmit a command? You know, we do the bulk moving at the last step of submission.

Quoted 2 lines

BTW: Just pushed this commit to the repository, should show up any second.

I see, thanks.

Thanks,
Ray

Quoted 36 lines

Christian.

Quoted 32 lines

Thanks,
Ray

Quoted 28 lines

Regards,
Christian.

Am 10.09.2018 um 10:57 schrieb Huang Rui:

Quoted 14 lines

It avoids to be refered again after freed.

Cc Christian König <[email protected]>
Cc Tom StDenis <[email protected]>
---
  drivers/gpu/drm/ttm/ttm_bo.c | 1 +
  1 file changed, 1 insertion(+)
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
index 138c989..d3ef5f8 100644

drivers/gpu/drm/ttm/ttm_bo.c static struct attribute ttm_bo_count = { · line 59

Quoted 6 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
@@ -54,6 +54,7 @@ static struct attribute ttm_bo_count = {
  static void ttm_bo_default_destroy(struct ttm_buffer_object *bo)
  {
  	kfree(bo);
+	bo = NULL;
  }
  static inline int ttm_mem_type_from_place(const struct ttm_place *place,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#8 lore raw
In reply to Christian König
Quoted 17 lines · Christian

Am 10.09.2018 um 11:23 schrieb Huang Rui:

Quoted 9 lines · Christian König
Quoted 4 lines

Hi Ray,

well those patches doesn't make sense, the pointer is only local to the function.

You're right.
I narrowed it with gdb dump from ttm_bo_bulk_move_lru_tail+0x2b, the use-after-free should be in below codes:

Quoted 6 lines

man = &bulk->tt[i].first->bdev->man[TTM_PL_TT]; ttm_bo_bulk_move_helper(&bulk->tt[i], &man->lru[i], false);

Is there a case, when orignal bo is destroyed in the bulk pos, but it doesn't update pos->first pointer, then we still use it during the bulk moving?

Quoted 6 lines

Only when a per VM BO is freed or the VM destroyed.

The first case should now be handled by "drm/amdgpu: set bulk_moveable to false when a per VM is released" and when we use a destroyed VM we would see other problems as well.

If a VM instance is teared down, all BOs which belong that VM should be removed from LRU. But how can we submit cmd based on a destroyed VM? You know, we do the bulk move at last step of submission.

Thanks,
Ray

Quoted 37 lines

BTW: Just pushed this commit to the repository, should show up any second.

Christian.

Quoted 32 lines

Thanks,
Ray

Quoted 28 lines

Regards,
Christian.

Am 10.09.2018 um 10:57 schrieb Huang Rui:

Quoted 14 lines

It avoids to be refered again after freed.

Cc Christian König <[email protected]>
Cc Tom StDenis <[email protected]>
---
  drivers/gpu/drm/ttm/ttm_bo.c | 1 +
  1 file changed, 1 insertion(+)
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
index 138c989..d3ef5f8 100644

drivers/gpu/drm/ttm/ttm_bo.c static struct attribute ttm_bo_count = { · line 59

Quoted 6 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
@@ -54,6 +54,7 @@ static struct attribute ttm_bo_count = {
  static void ttm_bo_default_destroy(struct ttm_buffer_object *bo)
  {
  	kfree(bo);
+	bo = NULL;
  }
  static inline int ttm_mem_type_from_place(const struct ttm_place *place,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#9 lore raw

Am 10.09.2018 um 14:05 schrieb Huang Rui:

Quoted 19 lines · Christian
Quoted 18 lines

Am 10.09.2018 um 11:23 schrieb Huang Rui:

Quoted 9 lines · Christian König
Quoted 4 lines

Hi Ray,

well those patches doesn't make sense, the pointer is only local to the function.

You're right.
I narrowed it with gdb dump from ttm_bo_bulk_move_lru_tail+0x2b, the use-after-free should be in below codes:

Quoted 6 lines

man = &bulk->tt[i].first->bdev->man[TTM_PL_TT]; ttm_bo_bulk_move_helper(&bulk->tt[i], &man->lru[i], false);

Is there a case, when orignal bo is destroyed in the bulk pos, but it doesn't update pos->first pointer, then we still use it during the bulk moving?

Only when a per VM BO is freed or the VM destroyed.

Quoted 7 lines
Quoted 4 lines

The first case should now be handled by "drm/amdgpu: set bulk_moveable to false when a per VM is released" and when we use a destroyed VM we would see other problems as well.

If a VM instance is teared down, all BOs which belong that VM should be removed from LRU. But how can we submit cmd based on a destroyed VM? You know, we do the bulk move at last step of submission.

Well exactly that's the point this can't happen :)

Otherwise we would crash because of using freed up memory much earlier in the command submission.

The best idea I have to track this down further is to add some trace_printk in ttm_bo_bulk_move_helper and amdgpu_bo_destroy and see why and when we are actually using a destroyed BO.

Christian.

Quoted 40 lines

Thanks,
Ray

Quoted 35 lines

BTW: Just pushed this commit to the repository, should show up any second.

Christian.

Quoted 31 lines

Thanks,
Ray

Quoted 28 lines

Regards,
Christian.

Am 10.09.2018 um 10:57 schrieb Huang Rui:

Quoted 14 lines

It avoids to be refered again after freed.

Cc Christian König <[email protected]>
Cc Tom StDenis <[email protected]>
---
   drivers/gpu/drm/ttm/ttm_bo.c | 1 +
   1 file changed, 1 insertion(+)
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
index 138c989..d3ef5f8 100644

drivers/gpu/drm/ttm/ttm_bo.c static struct attribute ttm_bo_count = { · line 59

Quoted 6 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
@@ -54,6 +54,7 @@ static struct attribute ttm_bo_count = {
   static void ttm_bo_default_destroy(struct ttm_buffer_object *bo)
   {
   	kfree(bo);
+	bo = NULL;
   }
   static inline int ttm_mem_type_from_place(const struct ttm_place *place,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Quoted 4 lines

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 746c5afd-9264-7dfa-4e3c-425ea82453ff-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

#11 lore raw

Hi Christian,

Are you adding new traces or turning on existing ones? Would you like me to try them out in my setup?

Tom

Quoted 28 lines · Christian König

Am 10.09.2018 um 14:05 schrieb Huang Rui:

Quoted 19 lines · Christian
Quoted 18 lines

Am 10.09.2018 um 11:23 schrieb Huang Rui:

Quoted 9 lines · Christian König
Quoted 4 lines

Hi Ray,

well those patches doesn't make sense, the pointer is only local to the function.

You're right.
I narrowed it with gdb dump from ttm_bo_bulk_move_lru_tail+0x2b, the use-after-free should be in below codes:

Quoted 6 lines

man = &bulk->tt[i].first->bdev->man[TTM_PL_TT]; ttm_bo_bulk_move_helper(&bulk->tt[i], &man->lru[i], false);

Is there a case, when orignal bo is destroyed in the bulk pos, but it doesn't update pos->first pointer, then we still use it during the bulk moving?

Only when a per VM BO is freed or the VM destroyed.

Quoted 7 lines
Quoted 4 lines

The first case should now be handled by "drm/amdgpu: set bulk_moveable to false when a per VM is released" and when we use a destroyed VM we would see other problems as well.

If a VM instance is teared down, all BOs which belong that VM should be removed from LRU. But how can we submit cmd based on a destroyed VM? You know, we do the bulk move at last step of submission.

Quoted 53 lines

Well exactly that's the point this can't happen :)

Otherwise we would crash because of using freed up memory much earlier in the command submission.

The best idea I have to track this down further is to add some trace_printk in ttm_bo_bulk_move_helper and amdgpu_bo_destroy and see why and when we are actually using a destroyed BO.

Christian.

Quoted 44 lines

Thanks,
Ray

Quoted 35 lines

BTW: Just pushed this commit to the repository, should show up any second.

Christian.

Quoted 31 lines

Thanks,
Ray

Quoted 28 lines

Regards,
Christian.

Am 10.09.2018 um 10:57 schrieb Huang Rui:

Quoted 14 lines

It avoids to be refered again after freed.

Cc Christian König <[email protected]>
Cc Tom StDenis <[email protected]>

---
   drivers/gpu/drm/ttm/ttm_bo.c | 1 +

   1 file changed, 1 insertion(+)
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
index 138c989..d3ef5f8 100644

drivers/gpu/drm/ttm/ttm_bo.c static struct attribute ttm_bo_count = { · line 59

Quoted 6 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
@@ -54,6 +54,7 @@ static struct attribute ttm_bo_count = {
    static void ttm_bo_default_destroy(struct ttm_buffer_object *bo)
    {
        kfree(bo);
+    bo = NULL;
    }
    static inline int ttm_mem_type_from_place(const struct ttm_place *place,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 382aab77-b9e2-1d18-ecd8-cd763227e5d0-5C7GfCeVMHo@public.gmane.org

#13 lore raw

Hi Tom,

I'm talking about adding new printks to figure out what the heck is going wrong here.

Thanks,
Christian.

Am 10.09.2018 um 14:59 schrieb Tom St Denis:

Quoted 89 lines

Hi Christian,

Are you adding new traces or turning on existing ones?  Would you like me to try them out in my setup?

Tom

Quoted 28 lines · Christian König

Am 10.09.2018 um 14:05 schrieb Huang Rui:

Quoted 19 lines · Christian
Quoted 18 lines

Am 10.09.2018 um 11:23 schrieb Huang Rui:

Quoted 9 lines · Christian König
Quoted 4 lines

Hi Ray,

well those patches doesn't make sense, the pointer is only local to the function.

You're right.
I narrowed it with gdb dump from ttm_bo_bulk_move_lru_tail+0x2b, the use-after-free should be in below codes:

Quoted 6 lines

man = &bulk->tt[i].first->bdev->man[TTM_PL_TT]; ttm_bo_bulk_move_helper(&bulk->tt[i], &man->lru[i], false);

Is there a case, when orignal bo is destroyed in the bulk pos, but it doesn't update pos->first pointer, then we still use it during the bulk moving?

Only when a per VM BO is freed or the VM destroyed.

Quoted 7 lines
Quoted 4 lines

The first case should now be handled by "drm/amdgpu: set bulk_moveable to false when a per VM is released" and when we use a destroyed VM we would see other problems as well.

If a VM instance is teared down, all BOs which belong that VM should be removed from LRU. But how can we submit cmd based on a destroyed VM? You know, we do the bulk move at last step of submission.

Quoted 53 lines

Well exactly that's the point this can't happen :)

Otherwise we would crash because of using freed up memory much earlier in the command submission.

The best idea I have to track this down further is to add some trace_printk in ttm_bo_bulk_move_helper and amdgpu_bo_destroy and see why and when we are actually using a destroyed BO.

Christian.

Quoted 44 lines

Thanks,
Ray

Quoted 35 lines

BTW: Just pushed this commit to the repository, should show up any second.

Christian.

Quoted 31 lines

Thanks,
Ray

Quoted 28 lines

Regards,
Christian.

Am 10.09.2018 um 10:57 schrieb Huang Rui:

Quoted 14 lines

It avoids to be refered again after freed.

Cc Christian König <[email protected]>
Cc Tom StDenis <[email protected]>

---
   drivers/gpu/drm/ttm/ttm_bo.c | 1 +

   1 file changed, 1 insertion(+)
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
index 138c989..d3ef5f8 100644

drivers/gpu/drm/ttm/ttm_bo.c static struct attribute ttm_bo_count = { · line 59

Quoted 6 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
@@ -54,6 +54,7 @@ static struct attribute ttm_bo_count = {
    static void ttm_bo_default_destroy(struct ttm_buffer_object *bo)
    {
        kfree(bo);
+    bo = NULL;
    }
    static inline int ttm_mem_type_from_place(const struct ttm_place *place,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): c26b96d6-a9b2-8609-4779-bf1f25402fe7-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

#15 lore raw
Quoted 6 lines · Christian König

Hi Tom,

I'm talking about adding new printks to figure out what the heck is going wrong here.

Thanks,
Christian.

Hi Christian,

Sure, if you want to send me a simple patch that adds more printk I'll gladly give it a try (doubly so since my workstation depends on our staging tree to work properly...).

Tom

Quoted 96 lines

Am 10.09.2018 um 14:59 schrieb Tom St Denis:

Quoted 93 lines

Hi Christian,

Are you adding new traces or turning on existing ones?  Would you like me to try them out in my setup?

Tom

Quoted 28 lines · Christian König

Am 10.09.2018 um 14:05 schrieb Huang Rui:

Quoted 19 lines · Christian
Quoted 18 lines

Am 10.09.2018 um 11:23 schrieb Huang Rui:

Quoted 9 lines · Christian König
Quoted 4 lines

Hi Ray,

well those patches doesn't make sense, the pointer is only local to the function.

You're right.
I narrowed it with gdb dump from ttm_bo_bulk_move_lru_tail+0x2b, the use-after-free should be in below codes:

Quoted 6 lines

man = &bulk->tt[i].first->bdev->man[TTM_PL_TT]; ttm_bo_bulk_move_helper(&bulk->tt[i], &man->lru[i], false);

Is there a case, when orignal bo is destroyed in the bulk pos, but it doesn't update pos->first pointer, then we still use it during the bulk moving?

Only when a per VM BO is freed or the VM destroyed.

Quoted 7 lines
Quoted 4 lines

The first case should now be handled by "drm/amdgpu: set bulk_moveable to false when a per VM is released" and when we use a destroyed VM we would see other problems as well.

If a VM instance is teared down, all BOs which belong that VM should be removed from LRU. But how can we submit cmd based on a destroyed VM? You know, we do the bulk move at last step of submission.

Quoted 53 lines

Well exactly that's the point this can't happen :)

Otherwise we would crash because of using freed up memory much earlier in the command submission.

The best idea I have to track this down further is to add some trace_printk in ttm_bo_bulk_move_helper and amdgpu_bo_destroy and see why and when we are actually using a destroyed BO.

Christian.

Quoted 44 lines

Thanks,
Ray

Quoted 35 lines

BTW: Just pushed this commit to the repository, should show up any second.

Christian.

Quoted 31 lines

Thanks,
Ray

Quoted 28 lines

Regards,
Christian.

Am 10.09.2018 um 10:57 schrieb Huang Rui:

Quoted 14 lines

It avoids to be refered again after freed.

Cc Christian König <[email protected]>
Cc Tom StDenis <[email protected]>

---
   drivers/gpu/drm/ttm/ttm_bo.c | 1 +

   1 file changed, 1 insertion(+)
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
index 138c989..d3ef5f8 100644

drivers/gpu/drm/ttm/ttm_bo.c static struct attribute ttm_bo_count = { · line 59

Quoted 6 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
@@ -54,6 +54,7 @@ static struct attribute ttm_bo_count = {
    static void ttm_bo_default_destroy(struct ttm_buffer_object *bo)
    {
        kfree(bo);
+    bo = NULL;
    }
    static inline int ttm_mem_type_from_place(const struct ttm_place *place,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1e1de90f-dc8c-4395-2375-3351f06d80b4-5C7GfCeVMHo@public.gmane.org

#17 lore raw

Am 10.09.2018 um 15:05 schrieb Tom St Denis:

Quoted 11 lines · Christian König
Quoted 6 lines

Hi Tom,

I'm talking about adding new printks to figure out what the heck is going wrong here.

Thanks,
Christian.

Hi Christian,

Sure, if you want to send me a simple patch that adds more printk I'll gladly give it a try (doubly so since my workstation depends on our staging tree to work properly...).

Just add a printk to ttm_bo_bulk_move_helper to print pos->first and pos->last.

And another one to amdgpu_bo_destroy to printk the value of tbo.

Christian.

Quoted 100 lines

Tom

Quoted 96 lines

Am 10.09.2018 um 14:59 schrieb Tom St Denis:

Quoted 93 lines

Hi Christian,

Are you adding new traces or turning on existing ones?  Would you like me to try them out in my setup?

Tom

Quoted 28 lines · Christian König

Am 10.09.2018 um 14:05 schrieb Huang Rui:

Quoted 19 lines · Christian
Quoted 18 lines

Am 10.09.2018 um 11:23 schrieb Huang Rui:

Quoted 9 lines · Christian König
Quoted 4 lines

Hi Ray,

well those patches doesn't make sense, the pointer is only local to the function.

You're right.
I narrowed it with gdb dump from ttm_bo_bulk_move_lru_tail+0x2b, the use-after-free should be in below codes:

Quoted 6 lines

man = &bulk->tt[i].first->bdev->man[TTM_PL_TT]; ttm_bo_bulk_move_helper(&bulk->tt[i], &man->lru[i], false);

Is there a case, when orignal bo is destroyed in the bulk pos, but it doesn't update pos->first pointer, then we still use it during the bulk moving?

Only when a per VM BO is freed or the VM destroyed.

Quoted 7 lines
Quoted 4 lines

The first case should now be handled by "drm/amdgpu: set bulk_moveable to false when a per VM is released" and when we use a destroyed VM we would see other problems as well.

If a VM instance is teared down, all BOs which belong that VM should be removed from LRU. But how can we submit cmd based on a destroyed VM? You know, we do the bulk move at last step of submission.

Quoted 53 lines

Well exactly that's the point this can't happen :)

Otherwise we would crash because of using freed up memory much earlier in the command submission.

The best idea I have to track this down further is to add some trace_printk in ttm_bo_bulk_move_helper and amdgpu_bo_destroy and see why and when we are actually using a destroyed BO.

Christian.

Quoted 44 lines

Thanks,
Ray

Quoted 35 lines

BTW: Just pushed this commit to the repository, should show up any second.

Christian.

Quoted 31 lines

Thanks,
Ray

Quoted 28 lines

Regards,
Christian.

Am 10.09.2018 um 10:57 schrieb Huang Rui:

Quoted 14 lines

It avoids to be refered again after freed.

Cc Christian König <[email protected]>
Cc Tom StDenis <[email protected]>

---
   drivers/gpu/drm/ttm/ttm_bo.c | 1 +

   1 file changed, 1 insertion(+)
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
index 138c989..d3ef5f8 100644

drivers/gpu/drm/ttm/ttm_bo.c static struct attribute ttm_bo_count = { · line 59

Quoted 6 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
@@ -54,6 +54,7 @@ static struct attribute ttm_bo_count = {
    static void ttm_bo_default_destroy(struct ttm_buffer_object *bo)
    {
        kfree(bo);
+    bo = NULL;
    }
    static inline int ttm_mem_type_from_place(const struct ttm_place *place,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 6d66a7c9-5904-2b23-4a65-f4995753009d-5C7GfCeVMHo@public.gmane.org

author
#19 lore raw
Quoted 12 lines · Christian

Am 10.09.2018 um 15:05 schrieb Tom St Denis:

Quoted 11 lines · Christian König
Quoted 7 lines

Hi Tom,

I'm talking about adding new printks to figure out what the heck is going wrong here.

Thanks,
Christian.

Hi Christian,

Quoted 9 lines
Quoted 3 lines

Sure, if you want to send me a simple patch that adds more printk I'll gladly give it a try (doubly so since my workstation depends on our staging tree to work properly...).

Just add a printk to ttm_bo_bulk_move_helper to print pos->first and pos->last.

And another one to amdgpu_bo_destroy to printk the value of tbo.

Hi Tom,

Could you help to add below traces to check when the bo is freed.

8<---
From 919cabfbf4d202876a510cd51caa9c86cf7c8fd5 Mon Sep 17 00:00:00 2001
From: Huang Rui <[email protected]>
Date: Tue, 11 Sep 2018 15:24:27 +0800
Subject: [PATCH] drm/amdgpu: add traces for lru bulk move

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c |  2 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h  | 47 ++++++++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c     |  1 +
 3 files changed, 50 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +2 -0
index de990bd..ce28326 100644
@@ -89,6 +89,8 @@ static void amdgpu_bo_destroy(struct ttm_buffer_object *tbo)
 	struct amdgpu_device *adev = amdgpu_ttm_adev(tbo->bdev);
 	struct amdgpu_bo *bo = ttm_to_amdgpu_bo(tbo);
 
+	trace_amdgpu_bo_destroy(bo);
+
 	if (bo->pin_count > 0)
 		amdgpu_bo_subtract_pin_size(bo);
 
drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h +47 -0

53 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +1 -0
index ab95a9c..351bc58 100644
@@ -391,6 +391,7 @@ void amdgpu_vm_move_to_lru_tail(struct amdgpu_device *adev,
 			continue;
 
 		ttm_bo_move_to_lru_tail(&bo->tbo, &vm->lru_bulk_move);
+		trace_amdgpu_vm_lru_bulk_move(vm, bo);
 		if (bo->shadow)
 			ttm_bo_move_to_lru_tail(&bo->shadow->tbo,
 						&vm->lru_bulk_move);
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx