Skip to contentSkip to conversation

drm/ttm: Schedule out if possibe in bo delayed delete worker

PATCH In review xinhui pan 4 participants 10 messages Apr 9, 2020 – Apr 10, 2020

Subsystems (3)
lore
Share & tools
lore

xinhui pan

author
#0 lore raw

PATCH 1/1 drm/ttm: Schedule out if possibe in bo delayed delete worker

The delayed delete list is per device which might be very huge. And in a heavy workload test, the list might always not be empty. That will trigger any RCU stall warnings or softlockups in non-preemptible kernels Lets do schedule out if possible in that case.

---
 drivers/gpu/drm/ttm/ttm_bo.c | 1 +
 1 file changed, 1 insertion(+)
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
index 9e07c3f75156..b8d853cab33b 100644
@@ -541,6 +541,7 @@ static bool ttm_bo_delayed_delete(struct ttm_bo_device *bdev, bool remove_all)
 		}
 
 		ttm_bo_put(bo);
+		cond_resched();
 		spin_lock(&glob->lru_lock);
 	}
 	list_splice_tail(&removed, &bdev->ddestroy);
Signature · 6 lines
2.17.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#1 lore raw

Am 09.04.20 um 03:31 schrieb xinhui pan:

Quoted 4 lines

The delayed delete list is per device which might be very huge. And in a heavy workload test, the list might always not be empty. That will trigger any RCU stall warnings or softlockups in non-preemptible kernels Lets do schedule out if possible in that case.

Mhm, I'm not sure if that is actually allowed. This is called from a work item and those are not really supposed to be scheduled away.

Maybe rather change the while into while (!list_empty(&bdev->ddestroy) && !should_reschedule(0)).

Christian.

Quoted 11 lines
---
  drivers/gpu/drm/ttm/ttm_bo.c | 1 +
  1 file changed, 1 insertion(+)
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
index 9e07c3f75156..b8d853cab33b 100644

drivers/gpu/drm/ttm/ttm_bo.c static bool ttm_bo_delayed_delete(struct ttm_bo_device *bdev, bool remove_all) · line 547

Quoted 7 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
@@ -541,6 +541,7 @@ static bool ttm_bo_delayed_delete(struct ttm_bo_device *bdev, bool remove_all)
  		}
 
  		ttm_bo_put(bo);
+		cond_resched();
  		spin_lock(&glob->lru_lock);
  	}
  	list_splice_tail(&removed, &bdev->ddestroy);

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#2 lore raw

Am Donnerstag, den 09.04.2020, 14:35 +0200 schrieb Christian König:

Quoted 8 lines

Am 09.04.20 um 03:31 schrieb xinhui pan:

Quoted 4 lines

The delayed delete list is per device which might be very huge. And in a heavy workload test, the list might always not be empty. That will trigger any RCU stall warnings or softlockups in non-preemptible kernels Lets do schedule out if possible in that case.

Mhm, I'm not sure if that is actually allowed. This is called from a work item and those are not really supposed to be scheduled away.

Huh? Workitems can schedule out just fine, otherwise they would be horribly broken when it comes to sleeping locks. The workqueue code even has measures to keep the workqueues at the expected concurrency level by starting other workitems when one of them goes to sleep.

Regards,
Lucas

Quoted 17 lines

Maybe rather change the while into while (!list_empty(&bdev->ddestroy) && !should_reschedule(0)).

Christian.

Quoted 12 lines
---
  drivers/gpu/drm/ttm/ttm_bo.c | 1 +
  1 file changed, 1 insertion(+)
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
index 9e07c3f75156..b8d853cab33b 100644
@@ -541,6 +541,7 @@ static bool ttm_bo_delayed_delete(struct ttm_bo_device *bdev, bool remove_all)
  		}
 
Quoted 10 lines

drivers/gpu/drm/ttm/ttm_bo.c static bool ttm_bo_delayed_delete(struct ttm_bo_device *bdev, bool remove_all) · line 547

Quoted 5 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
  		ttm_bo_put(bo);
+		cond_resched();
  		spin_lock(&glob->lru_lock);
  	}
  	list_splice_tail(&removed, &bdev->ddestroy);

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#3 lore raw

Am 09.04.20 um 15:25 schrieb Lucas Stach:

Quoted 10 lines

Am Donnerstag, den 09.04.2020, 14:35 +0200 schrieb Christian König:

Quoted 7 lines

Am 09.04.20 um 03:31 schrieb xinhui pan:

Quoted 4 lines

The delayed delete list is per device which might be very huge. And in a heavy workload test, the list might always not be empty. That will trigger any RCU stall warnings or softlockups in non-preemptible kernels Lets do schedule out if possible in that case.

Mhm, I'm not sure if that is actually allowed. This is called from a work item and those are not really supposed to be scheduled away.

Huh? Workitems can schedule out just fine, otherwise they would be horribly broken when it comes to sleeping locks.

Let me refine the sentence: Work items are not really supposed to be scheduled purposely. E.g. you shouldn't call schedule() or cond_resched() like in the case here.

Getting scheduled away because we wait for a lock is of course perfectly fine.

Quoted 3 lines

The workqueue code
even has measures to keep the workqueues at the expected concurrency level by starting other workitems when one of them goes to sleep.

Yeah, and exactly that's what I would say we should avoid here :)

In other words work items can be scheduled away, but they should not if not really necessary (e.g. waiting for a lock).

Otherwise as you said new threads for work item processing are started up and I don't think we want that.

Just returning from the work item and waiting for the next cycle is most likely the better option.

Regards,
Christian.

Quoted 30 lines

Regards,
Lucas

Quoted 26 lines

Maybe rather change the while into while (!list_empty(&bdev->ddestroy) && !should_reschedule(0)).

Christian.

Quoted 10 lines
---
   drivers/gpu/drm/ttm/ttm_bo.c | 1 +
   1 file changed, 1 insertion(+)
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
index 9e07c3f75156..b8d853cab33b 100644

drivers/gpu/drm/ttm/ttm_bo.c static bool ttm_bo_delayed_delete(struct ttm_bo_device *bdev, bool remove_all) · line 547

Quoted 7 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
@@ -541,6 +541,7 @@ static bool ttm_bo_delayed_delete(struct ttm_bo_device *bdev, bool remove_all)
   		}
 
   		ttm_bo_put(bo);
+		cond_resched();
   		spin_lock(&glob->lru_lock);
   	}
   	list_splice_tail(&removed, &bdev->ddestroy);

_______________________________________________
dri-devel mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Fdri-devel&data=02%7C01%7Cchristian.koenig%40amd.com%7C0a47486676a74702f05408d7dc89839c%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220355504145868&sdata=wbRkYBPI6mYuZjKBtQN3AGLDOwqJlWY3XUtwwSiUQHg%3D&reserved=0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#4 lore raw

[AMD Official Use Only - Internal Distribution Only]

Why we break out the loops when there are pending bos to be released?

And I just checked the process_one_work. Right after the work item callback is called, the workqueue itself will call cond_resched. So I think
________________________________
From: Koenig, Christian <[email protected]>
Sent: Thursday, April 9, 2020 9:38:24 PM
To: Lucas Stach <[email protected]>; Pan, Xinhui <[email protected]>; [email protected] <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: [PATCH] drm/ttm: Schedule out if possibe in bo delayed delete worker

Am 09.04.20 um 15:25 schrieb Lucas Stach:

Quoted 10 lines

Am Donnerstag, den 09.04.2020, 14:35 +0200 schrieb Christian König:

Quoted 7 lines

Am 09.04.20 um 03:31 schrieb xinhui pan:

Quoted 4 lines

The delayed delete list is per device which might be very huge. And in a heavy workload test, the list might always not be empty. That will trigger any RCU stall warnings or softlockups in non-preemptible kernels Lets do schedule out if possible in that case.

Mhm, I'm not sure if that is actually allowed. This is called from a work item and those are not really supposed to be scheduled away.

Huh? Workitems can schedule out just fine, otherwise they would be horribly broken when it comes to sleeping locks.

Let me refine the sentence: Work items are not really supposed to be scheduled purposely. E.g. you shouldn't call schedule() or cond_resched() like in the case here.

Getting scheduled away because we wait for a lock is of course perfectly fine.

Quoted 3 lines

The workqueue code
even has measures to keep the workqueues at the expected concurrency level by starting other workitems when one of them goes to sleep.

Yeah, and exactly that's what I would say we should avoid here :)

In other words work items can be scheduled away, but they should not if not really necessary (e.g. waiting for a lock).

Otherwise as you said new threads for work item processing are started up and I don't think we want that.

Just returning from the work item and waiting for the next cycle is most likely the better option.

Regards,
Christian.

Quoted 30 lines

Regards,
Lucas

Quoted 26 lines

Maybe rather change the while into while (!list_empty(&bdev->ddestroy) && !should_reschedule(0)).

Christian.

Quoted 10 lines
---
   drivers/gpu/drm/ttm/ttm_bo.c | 1 +
   1 file changed, 1 insertion(+)
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
index 9e07c3f75156..b8d853cab33b 100644

drivers/gpu/drm/ttm/ttm_bo.c static bool ttm_bo_delayed_delete(struct ttm_bo_device *bdev, bool remove_all) · line 547

Quoted 7 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
@@ -541,6 +541,7 @@ static bool ttm_bo_delayed_delete(struct ttm_bo_device *bdev, bool remove_all)
             }
 
             ttm_bo_put(bo);
+           cond_resched();
             spin_lock(&glob->lru_lock);
     }
     list_splice_tail(&removed, &bdev->ddestroy);

_______________________________________________
dri-devel mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Fdri-devel&amp;data=02%7C01%7Cchristian.koenig%40amd.com%7C0a47486676a74702f05408d7dc89839c%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220355504145868&amp;sdata=wbRkYBPI6mYuZjKBtQN3AGLDOwqJlWY3XUtwwSiUQHg%3D&amp;reserved=0

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#5 lore raw

I think it doesn't matter if workitem schedule out. Even we did not schedule out, the workqueue itself will schedule out later. So it did not break anything with this patch I think.
________________________________
From: Pan, Xinhui <[email protected]>
Sent: Thursday, April 9, 2020 10:07:09 PM
To: Lucas Stach <[email protected]>; [email protected] <[email protected]>; Koenig, Christian <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: [PATCH] drm/ttm: Schedule out if possibe in bo delayed delete worker

Why we break out the loops when there are pending bos to be released?

And I just checked the process_one_work. Right after the work item callback is called, the workqueue itself will call cond_resched. So I think
________________________________
From: Koenig, Christian <[email protected]>
Sent: Thursday, April 9, 2020 9:38:24 PM
To: Lucas Stach <[email protected]>; Pan, Xinhui <[email protected]>; [email protected] <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: [PATCH] drm/ttm: Schedule out if possibe in bo delayed delete worker

Am 09.04.20 um 15:25 schrieb Lucas Stach:

Quoted 10 lines

Am Donnerstag, den 09.04.2020, 14:35 +0200 schrieb Christian König:

Quoted 7 lines

Am 09.04.20 um 03:31 schrieb xinhui pan:

Quoted 4 lines

The delayed delete list is per device which might be very huge. And in a heavy workload test, the list might always not be empty. That will trigger any RCU stall warnings or softlockups in non-preemptible kernels Lets do schedule out if possible in that case.

Mhm, I'm not sure if that is actually allowed. This is called from a work item and those are not really supposed to be scheduled away.

Huh? Workitems can schedule out just fine, otherwise they would be horribly broken when it comes to sleeping locks.

Let me refine the sentence: Work items are not really supposed to be scheduled purposely. E.g. you shouldn't call schedule() or cond_resched() like in the case here.

Getting scheduled away because we wait for a lock is of course perfectly fine.

Quoted 3 lines

The workqueue code
even has measures to keep the workqueues at the expected concurrency level by starting other workitems when one of them goes to sleep.

Yeah, and exactly that's what I would say we should avoid here :)

In other words work items can be scheduled away, but they should not if not really necessary (e.g. waiting for a lock).

Otherwise as you said new threads for work item processing are started up and I don't think we want that.

Just returning from the work item and waiting for the next cycle is most likely the better option.

Regards,
Christian.

Quoted 30 lines

Regards,
Lucas

Quoted 26 lines

Maybe rather change the while into while (!list_empty(&bdev->ddestroy) && !should_reschedule(0)).

Christian.

Quoted 10 lines
---
   drivers/gpu/drm/ttm/ttm_bo.c | 1 +
   1 file changed, 1 insertion(+)
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
index 9e07c3f75156..b8d853cab33b 100644

drivers/gpu/drm/ttm/ttm_bo.c static bool ttm_bo_delayed_delete(struct ttm_bo_device *bdev, bool remove_all) · line 547

Quoted 7 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
@@ -541,6 +541,7 @@ static bool ttm_bo_delayed_delete(struct ttm_bo_device *bdev, bool remove_all)
             }
 
             ttm_bo_put(bo);
+           cond_resched();
             spin_lock(&glob->lru_lock);
     }
     list_splice_tail(&removed, &bdev->ddestroy);

_______________________________________________
dri-devel mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Fdri-devel&amp;data=02%7C01%7Cchristian.koenig%40amd.com%7C0a47486676a74702f05408d7dc89839c%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220355504145868&amp;sdata=wbRkYBPI6mYuZjKBtQN3AGLDOwqJlWY3XUtwwSiUQHg%3D&amp;reserved=0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#6 lore raw

https://elixir.bootlin.com/linux/latest/source/mm/slab.c#L4026

This is another example of the usage of cond_sched.
________________________________
From: Pan, Xinhui <[email protected]>
Sent: Thursday, April 9, 2020 10:11:08 PM
To: Lucas Stach <[email protected]>; [email protected] <[email protected]>; Koenig, Christian <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: [PATCH] drm/ttm: Schedule out if possibe in bo delayed delete worker

I think it doesn't matter if workitem schedule out. Even we did not schedule out, the workqueue itself will schedule out later. So it did not break anything with this patch I think.
________________________________
From: Pan, Xinhui <[email protected]>
Sent: Thursday, April 9, 2020 10:07:09 PM
To: Lucas Stach <[email protected]>; [email protected] <[email protected]>; Koenig, Christian <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: [PATCH] drm/ttm: Schedule out if possibe in bo delayed delete worker

Why we break out the loops when there are pending bos to be released?

And I just checked the process_one_work. Right after the work item callback is called, the workqueue itself will call cond_resched. So I think
________________________________
From: Koenig, Christian <[email protected]>
Sent: Thursday, April 9, 2020 9:38:24 PM
To: Lucas Stach <[email protected]>; Pan, Xinhui <[email protected]>; [email protected] <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: [PATCH] drm/ttm: Schedule out if possibe in bo delayed delete worker

Am 09.04.20 um 15:25 schrieb Lucas Stach:

Quoted 10 lines

Am Donnerstag, den 09.04.2020, 14:35 +0200 schrieb Christian König:

Quoted 7 lines

Am 09.04.20 um 03:31 schrieb xinhui pan:

Quoted 4 lines

The delayed delete list is per device which might be very huge. And in a heavy workload test, the list might always not be empty. That will trigger any RCU stall warnings or softlockups in non-preemptible kernels Lets do schedule out if possible in that case.

Mhm, I'm not sure if that is actually allowed. This is called from a work item and those are not really supposed to be scheduled away.

Huh? Workitems can schedule out just fine, otherwise they would be horribly broken when it comes to sleeping locks.

Let me refine the sentence: Work items are not really supposed to be scheduled purposely. E.g. you shouldn't call schedule() or cond_resched() like in the case here.

Getting scheduled away because we wait for a lock is of course perfectly fine.

Quoted 3 lines

The workqueue code
even has measures to keep the workqueues at the expected concurrency level by starting other workitems when one of them goes to sleep.

Yeah, and exactly that's what I would say we should avoid here :)

In other words work items can be scheduled away, but they should not if not really necessary (e.g. waiting for a lock).

Otherwise as you said new threads for work item processing are started up and I don't think we want that.

Just returning from the work item and waiting for the next cycle is most likely the better option.

Regards,
Christian.

Quoted 30 lines

Regards,
Lucas

Quoted 26 lines

Maybe rather change the while into while (!list_empty(&bdev->ddestroy) && !should_reschedule(0)).

Christian.

Quoted 10 lines
---
   drivers/gpu/drm/ttm/ttm_bo.c | 1 +
   1 file changed, 1 insertion(+)
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
index 9e07c3f75156..b8d853cab33b 100644

drivers/gpu/drm/ttm/ttm_bo.c static bool ttm_bo_delayed_delete(struct ttm_bo_device *bdev, bool remove_all) · line 547

Quoted 7 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
@@ -541,6 +541,7 @@ static bool ttm_bo_delayed_delete(struct ttm_bo_device *bdev, bool remove_all)
             }
 
             ttm_bo_put(bo);
+           cond_resched();
             spin_lock(&glob->lru_lock);
     }
     list_splice_tail(&removed, &bdev->ddestroy);

_______________________________________________
dri-devel mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Fdri-devel&amp;data=02%7C01%7Cchristian.koenig%40amd.com%7C0a47486676a74702f05408d7dc89839c%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220355504145868&amp;sdata=wbRkYBPI6mYuZjKBtQN3AGLDOwqJlWY3XUtwwSiUQHg%3D&amp;reserved=0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#7 lore raw
Quoted 1 line

Why we break out the loops when there are pending bos to be released?

We do this anyway if we can't acquire the necessary locks. Freeing already deleted BOs is just a very lazy background work.

Quoted 1 line

So it did not break anything with this patch I think.

Oh, the patch will certainly work. I'm just not sure if it's the ideal behavior.

Quoted 3 lines

https://elixir.bootlin.com/linux/latest/source/mm/slab.c#L4026

This is another example of the usage of  cond_sched.

Yes, and that is also a good example of what I mean here:

Quoted 4 lines

if (!mutex_trylock
<https://elixir.bootlin.com/linux/latest/ident/mutex_trylock>(&slab_mutex <https://elixir.bootlin.com/linux/latest/ident/slab_mutex>))

		/* Give up. Setup the next iteration. */
		goto  out;

If the function can't acquire the lock immediately it gives up and waits for the next iteration.

I think it would be better if we do this in TTM as well if we spend to much time cleaning up old BOs.

On the other hand you are right that cond_resched() has the advantage that we could spend more time on cleaning up old BOs if there is nothing else for the CPU TODO.

Regards,
Christian.

Am 09.04.20 um 16:24 schrieb Pan, Xinhui:

Quoted 92 lines

https://elixir.bootlin.com/linux/latest/source/mm/slab.c#L4026

This is another example of the usage of  cond_sched.
------------------------------------------------------------------------
*From:* Pan, Xinhui <[email protected]>
*Sent:* Thursday, April 9, 2020 10:11:08 PM
*To:* Lucas Stach <[email protected]>; [email protected] <[email protected]>; Koenig, Christian <[email protected]> *Cc:* [email protected] <[email protected]>
*Subject:* Re: [PATCH] drm/ttm: Schedule out if possibe in bo delayed delete worker
I think it doesn't matter if workitem schedule out. Even we did not schedule out, the workqueue itself will schedule out later. So it did not break anything with this patch I think.
------------------------------------------------------------------------
*From:* Pan, Xinhui <[email protected]>
*Sent:* Thursday, April 9, 2020 10:07:09 PM
*To:* Lucas Stach <[email protected]>; [email protected] <[email protected]>; Koenig, Christian <[email protected]> *Cc:* [email protected] <[email protected]>
*Subject:* Re: [PATCH] drm/ttm: Schedule out if possibe in bo delayed delete worker
Why we break out the loops when there are pending bos to be released?

And I just checked the process_one_work. Right after the work item callback is called,  the workqueue itself will call cond_resched. So I think
------------------------------------------------------------------------
*From:* Koenig, Christian <[email protected]>
*Sent:* Thursday, April 9, 2020 9:38:24 PM
*To:* Lucas Stach <[email protected]>; Pan, Xinhui <[email protected]>; [email protected] <[email protected]> *Cc:* [email protected] <[email protected]>
*Subject:* Re: [PATCH] drm/ttm: Schedule out if possibe in bo delayed delete worker
Am 09.04.20 um 15:25 schrieb Lucas Stach:

Quoted 5 lines

Am Donnerstag, den 09.04.2020, 14:35 +0200 schrieb Christian König:

Quoted 4 lines

Am 09.04.20 um 03:31 schrieb xinhui pan:

Quoted 3 lines

The delayed delete list is per device which might be very huge. And in a heavy workload test, the list might always not be empty. That will trigger any RCU stall warnings or softlockups in non-preemptible

kernels

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Lets do schedule out if possible in that case.

Mhm, I'm not sure if that is actually allowed. This is called from a work item and those are not really supposed to be scheduled away.

Huh? Workitems can schedule out just fine, otherwise they would be horribly broken when it comes to sleeping locks.

Let me refine the sentence: Work items are not really supposed to be scheduled purposely. E.g. you shouldn't call schedule() or cond_resched() like in the case here.

Getting scheduled away because we wait for a lock is of course perfectly fine.

Quoted 3 lines

   The workqueue code
even has measures to keep the workqueues at the expected concurrency level by starting other workitems when one of them goes to sleep.

Yeah, and exactly that's what I would say we should avoid here :)

In other words work items can be scheduled away, but they should not if not really necessary (e.g. waiting for a lock).

Otherwise as you said new threads for work item processing are started up and I don't think we want that.

Just returning from the work item and waiting for the next cycle is most likely the better option.

Regards,
Christian.

Quoted 7 lines

Regards,
Lucas

Quoted 3 lines

Maybe rather change the while into while (!list_empty(&bdev->ddestroy) && !should_reschedule(0)).

Quoted 8 lines
Quoted 8 lines

Christian.

Review

Quoted 6 lines

---
    drivers/gpu/drm/ttm/ttm_bo.c | 1 +

    1 file changed, 1 insertion(+)
(quoted diff) +0 -0

b/drivers/gpu/drm/ttm/ttm_bo.c

Quoted 4 lines
Quoted 4 lines

drivers/gpu/drm/ttm/ttm_bo.c static bool ttm_bo_delayed_delete(struct

Quoted 4 lines

index 9e07c3f75156..b8d853cab33b 100644

drivers/gpu/drm/ttm/ttm_bo.c +0 -0
@@ -541,6 +541,7 @@ static bool ttm_bo_delayed_delete(struct 

ttm_bo_device *bdev, bool remove_all)

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
               }
 
               ttm_bo_put(bo);
+           cond_resched();
 spin_lock(&glob->lru_lock);
       }
       list_splice_tail(&removed, &bdev->ddestroy);
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

_______________________________________________
dri-devel mailing list
[email protected]

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Fdri-devel&amp;data=02%7C01%7Cchristian.koenig%40amd.com%7C0a47486676a74702f05408d7dc89839c%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220355504145868&amp;sdata=wbRkYBPI6mYuZjKBtQN3AGLDOwqJlWY3XUtwwSiUQHg%3D&amp;reserved=0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#8 lore raw
Quoted 7 lines
Quoted 1 line · Christian

Why we break out the loops when there are pending bos to be released?

We do this anyway if we can't acquire the necessary locks. Freeing already deleted BOs is just a very lazy background work.

That is true. eviction will reclaim the BO resource too.

Quoted 14 lines
Quoted 2 lines

So it did not break anything with this patch I think.

Oh, the patch will certainly work. I'm just not sure if it's the ideal behavior.

Quoted 3 lines

https://elixir.bootlin.com/linux/latest/source/mm/slab.c#L4026

This is another example of the usage of cond_sched.

Yes, and that is also a good example of what I mean here:

Quoted 3 lines

if (!mutex_trylock(&slab_mutex))

Quoted 8 lines
Quoted 4 lines

/* Give up. Setup the next iteration. */

goto out;

If the function can't acquire the lock immediately it gives up and waits for the next iteration.

I think it would be better if we do this in TTM as well if we spend to much time cleaning up old BOs.

fair enough.

Quoted 102 lines

On the other hand you are right that cond_resched() has the advantage that we could spend more time on cleaning up old BOs if there is nothing else for the CPU TODO.

Regards,
Christian.

Am 09.04.20 um 16:24 schrieb Pan, Xinhui:

Quoted 94 lines

https://elixir.bootlin.com/linux/latest/source/mm/slab.c#L4026

This is another example of the usage of cond_sched.
From: Pan, Xinhui <[email protected]>
Sent: Thursday, April 9, 2020 10:11:08 PM
To: Lucas Stach <[email protected]>; [email protected] <[email protected]>; Koenig, Christian <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: [PATCH] drm/ttm: Schedule out if possibe in bo delayed delete worker

I think it doesn't matter if workitem schedule out. Even we did not schedule out, the workqueue itself will schedule out later. So it did not break anything with this patch I think.
From: Pan, Xinhui <[email protected]>
Sent: Thursday, April 9, 2020 10:07:09 PM
To: Lucas Stach <[email protected]>; [email protected] <[email protected]>; Koenig, Christian <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: [PATCH] drm/ttm: Schedule out if possibe in bo delayed delete worker

Why we break out the loops when there are pending bos to be released?

And I just checked the process_one_work. Right after the work item callback is called, the workqueue itself will call cond_resched. So I think
From: Koenig, Christian <[email protected]>
Sent: Thursday, April 9, 2020 9:38:24 PM
To: Lucas Stach <[email protected]>; Pan, Xinhui <[email protected]>; [email protected] <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: [PATCH] drm/ttm: Schedule out if possibe in bo delayed delete worker

Am 09.04.20 um 15:25 schrieb Lucas Stach:

Quoted 10 lines

Am Donnerstag, den 09.04.2020, 14:35 +0200 schrieb Christian König:

Quoted 7 lines

Am 09.04.20 um 03:31 schrieb xinhui pan:

Quoted 4 lines

The delayed delete list is per device which might be very huge. And in a heavy workload test, the list might always not be empty. That will trigger any RCU stall warnings or softlockups in non-preemptible kernels Lets do schedule out if possible in that case.

Mhm, I'm not sure if that is actually allowed. This is called from a work item and those are not really supposed to be scheduled away.

Huh? Workitems can schedule out just fine, otherwise they would be horribly broken when it comes to sleeping locks.

Let me refine the sentence: Work items are not really supposed to be scheduled purposely. E.g. you shouldn't call schedule() or cond_resched() like in the case here.

Getting scheduled away because we wait for a lock is of course perfectly fine.

Quoted 3 lines

The workqueue code
even has measures to keep the workqueues at the expected concurrency level by starting other workitems when one of them goes to sleep.

Yeah, and exactly that's what I would say we should avoid here :)

In other words work items can be scheduled away, but they should not if not really necessary (e.g. waiting for a lock).

Otherwise as you said new threads for work item processing are started up and I don't think we want that.

Just returning from the work item and waiting for the next cycle is most likely the better option.

Regards,
Christian.

Quoted 30 lines

Regards,
Lucas

Quoted 26 lines

Maybe rather change the while into while (!list_empty(&bdev->ddestroy) && !should_reschedule(0)).

Christian.

Quoted 10 lines
---
   drivers/gpu/drm/ttm/ttm_bo.c | 1 +
   1 file changed, 1 insertion(+)
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
index 9e07c3f75156..b8d853cab33b 100644

drivers/gpu/drm/ttm/ttm_bo.c static bool ttm_bo_delayed_delete(struct ttm_bo_device *bdev, bool remove_all) · line 547

Quoted 7 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
@@ -541,6 +541,7 @@ static bool ttm_bo_delayed_delete(struct ttm_bo_device *bdev, bool remove_all)
             }
 
             ttm_bo_put(bo);
+           cond_resched();
             spin_lock(&glob->lru_lock);
     }
     list_splice_tail(&removed, &bdev->ddestroy);

_______________________________________________
dri-devel mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Fdri-devel&amp;data=02%7C01%7Cchristian.koenig%40amd.com%7C0a47486676a74702f05408d7dc89839c%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220355504145868&amp;sdata=wbRkYBPI6mYuZjKBtQN3AGLDOwqJlWY3XUtwwSiUQHg%3D&amp;reserved=0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#9 lore raw
In reply to Christian König

We can have both of yours, I think.

Even switch to use spin_trylock, I think we are ok to have cond_resched() Xinhui added in this patch. That can give more chance to urgent task to use cpu.

-David

Quoted 126 lines · Christian König
Quoted 1 line

Why we break out the loops when there are pending bos to be released?

We do this anyway if we can't acquire the necessary locks. Freeing already deleted BOs is just a very lazy background work.

Quoted 1 line

So it did not break anything with this patch I think.

Oh, the patch will certainly work. I'm just not sure if it's the ideal behavior.

Quoted 3 lines

https://elixir.bootlin.com/linux/latest/source/mm/slab.c#L4026

This is another example of the usage of  cond_sched.

Yes, and that is also a good example of what I mean here:

Quoted 4 lines

if (!mutex_trylock
<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Felixir.bootlin.com%2Flinux%2Flatest%2Fident%2Fmutex_trylock&data=02%7C01%7CDavid1.Zhou%40amd.com%7Cfcae774489544a033c0608d7dc969f31%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220411795246517&sdata=XACA%2BgpBkJEgPva9c7Wf6Ca1bAOrNaXARf%2B4ze1Mqyw%3D&reserved=0>(&slab_mutex <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Felixir.bootlin.com%2Flinux%2Flatest%2Fident%2Fslab_mutex&data=02%7C01%7CDavid1.Zhou%40amd.com%7Cfcae774489544a033c0608d7dc969f31%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220411795256512&sdata=z9uAZriS46hBXuYeVaYkAWb%2BPLYwLcCrK%2BWa4DRO0zw%3D&reserved=0>))

		/* Give up. Setup the next iteration. */
		goto  out;

If the function can't acquire the lock immediately it gives up and waits for the next iteration.

I think it would be better if we do this in TTM as well if we spend to much time cleaning up old BOs.

On the other hand you are right that cond_resched() has the advantage that we could spend more time on cleaning up old BOs if there is nothing else for the CPU TODO.

Regards,
Christian.

Am 09.04.20 um 16:24 schrieb Pan, Xinhui:

Quoted 97 lines

https://elixir.bootlin.com/linux/latest/source/mm/slab.c#L4026

This is another example of the usage of  cond_sched.
------------------------------------------------------------------------
*From:* Pan, Xinhui <[email protected]>
*Sent:* Thursday, April 9, 2020 10:11:08 PM
*To:* Lucas Stach <[email protected]>; [email protected] <[email protected]>; Koenig, Christian <[email protected]> *Cc:* [email protected] <[email protected]>
*Subject:* Re: [PATCH] drm/ttm: Schedule out if possibe in bo delayed delete worker
I think it doesn't matter if workitem schedule out. Even we did not schedule out, the workqueue itself will schedule out later. So it did not break anything with this patch I think.
------------------------------------------------------------------------
*From:* Pan, Xinhui <[email protected]>
*Sent:* Thursday, April 9, 2020 10:07:09 PM
*To:* Lucas Stach <[email protected]>; [email protected] <[email protected]>; Koenig, Christian <[email protected]> *Cc:* [email protected] <[email protected]>
*Subject:* Re: [PATCH] drm/ttm: Schedule out if possibe in bo delayed delete worker
Why we break out the loops when there are pending bos to be released?

And I just checked the process_one_work. Right after the work item callback is called,  the workqueue itself will call cond_resched. So I think
------------------------------------------------------------------------
*From:* Koenig, Christian <[email protected]>
*Sent:* Thursday, April 9, 2020 9:38:24 PM
*To:* Lucas Stach <[email protected]>; Pan, Xinhui <[email protected]>; [email protected] <[email protected]> *Cc:* [email protected] <[email protected]>
*Subject:* Re: [PATCH] drm/ttm: Schedule out if possibe in bo delayed delete worker
Am 09.04.20 um 15:25 schrieb Lucas Stach:

Quoted 3 lines

Am Donnerstag, den 09.04.2020, 14:35 +0200 schrieb Christian König:

Quoted 2 lines

Am 09.04.20 um 03:31 schrieb xinhui pan:

Quoted 1 line

The delayed delete list is per device which might be very huge.

And in

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

a heavy workload test, the list might always not be empty. That will trigger any RCU stall warnings or softlockups in non-preemptible

kernels

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Lets do schedule out if possible in that case.

Mhm, I'm not sure if that is actually allowed. This is called from a work item and those are not really supposed to be scheduled away.

Huh? Workitems can schedule out just fine, otherwise they would be horribly broken when it comes to sleeping locks.

Let me refine the sentence: Work items are not really supposed to be scheduled purposely. E.g. you shouldn't call schedule() or cond_resched() like in the case here.

Getting scheduled away because we wait for a lock is of course perfectly fine.

Quoted 3 lines

   The workqueue code
even has measures to keep the workqueues at the expected concurrency level by starting other workitems when one of them goes to sleep.

Yeah, and exactly that's what I would say we should avoid here :)

In other words work items can be scheduled away, but they should not if not really necessary (e.g. waiting for a lock).

Otherwise as you said new threads for work item processing are started up and I don't think we want that.

Just returning from the work item and waiting for the next cycle is most likely the better option.

Regards,
Christian.

Quoted 7 lines

Regards,
Lucas

Quoted 3 lines

Maybe rather change the while into while (!list_empty(&bdev->ddestroy) && !should_reschedule(0)).

Quoted 8 lines
Quoted 8 lines

Christian.

Review

Quoted 6 lines

---
    drivers/gpu/drm/ttm/ttm_bo.c | 1 +

    1 file changed, 1 insertion(+)
(quoted diff) +0 -0

b/drivers/gpu/drm/ttm/ttm_bo.c

Quoted 4 lines
Quoted 4 lines

drivers/gpu/drm/ttm/ttm_bo.c static bool ttm_bo_delayed_delete(struct

Quoted 4 lines

index 9e07c3f75156..b8d853cab33b 100644

drivers/gpu/drm/ttm/ttm_bo.c +0 -0
@@ -541,6 +541,7 @@ static bool ttm_bo_delayed_delete(struct 

ttm_bo_device *bdev, bool remove_all)

Quoted 11 lines
Quoted 11 lines

drivers/gpu/drm/ttm/ttm_bo.c static bool ttm_bo_delayed_delete(struct · line 547

Quoted 7 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
               }
 
               ttm_bo_put(bo);
+           cond_resched();
 spin_lock(&glob->lru_lock);
       }
       list_splice_tail(&removed, &bdev->ddestroy);

_______________________________________________
dri-devel mailing list
[email protected]

Quoted 8 lines
Quoted 2 lines

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Fdri-devel&amp;data=02%7C01%7Cchristian.koenig%40amd.com%7C0a47486676a74702f05408d7dc89839c%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220355504145868&amp;sdata=wbRkYBPI6mYuZjKBtQN3AGLDOwqJlWY3XUtwwSiUQHg%3D&amp;reserved=0 <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Fdri-devel&data=02%7C01%7CDavid1.Zhou%40amd.com%7Cfcae774489544a033c0608d7dc969f31%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220411795266500&sdata=UGbQTN7vjHhNPidodWhXx4sSqUQjtKp4dbCJcztf6ZM%3D&reserved=0>

_______________________________________________
dri-devel mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Fdri-devel&amp;data=02%7C01%7CDavid1.Zhou%40amd.com%7Cfcae774489544a033c0608d7dc969f31%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220411795286493&amp;sdata=SoE%2F8sg9vCb0OwxPF%2FPpKLLUKug%2FAhZDyyvDiQamWp4%3D&amp;reserved=0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx