Skip to contentSkip to conversation

ttm_bo and multiple backing store segments

DISCUSSION Welty, Brian 3 participants 4 messages Jun 29, 2023 – Aug 4, 2023

lore
Share & tools
lore
#0 lore raw

Hi Christian / Thomas,

Wanted to ask if you have explored or thought about adding support in TTM such that a ttm_bo could have more than one underlying backing store segment (that is, to have a tree of ttm_resources)? We are considering to support such BOs for Intel Xe driver.

Some of the benefits:
* devices with page fault support can fault (and migrate) backing store at finer granularity than the entire BO
* BOs can support having multiple backing store segments, which can be in different memory domains/regions
* BO eviction could operate on smaller granularity than entire BO

Or is the thinking that workloads should use SVM/HMM instead of GEM_CREATE if they want above benefits?

Is this something you are open to seeing an RFC series that starts perhaps with just extending ttm_bo_validate() to see how this might shape up?

-Brian

#1 lore raw
Quoted 7 lines · Brian

Hi Christian / Thomas,

Wanted to ask if you have explored or thought about adding support in TTM such that a ttm_bo could have more than one underlying backing store segment (that is, to have a tree of ttm_resources)?
We are considering to support such BOs for Intel Xe driver.

They are indeed the best one to give an opinion here. I just have some dummy questions and comments below.

Quoted 4 lines

Some of the benefits:
* devices with page fault support can fault (and migrate) backing store at finer granularity than the entire BO

what advantage does this bring? to each workload?
is it a performance on huge bo?

Quoted 2 lines

* BOs can support having multiple backing store segments, which can be in different memory domains/regions

what locking challenges would this bring?
is this more targeting gpu + cpu? or only for our multi-tile platforms? and what's the advantage this is bringing to real use cases? (probably the svm/hmm question below answers my questions, but...)

Quoted 1 line

* BO eviction could operate on smaller granularity than entire BO

I believe all the previous doubts apply to this item as well...

Quoted 6 lines

Or is the thinking that workloads should use SVM/HMM instead of GEM_CREATE if they want above benefits?

Is this something you are open to seeing an RFC series that starts perhaps with just extending ttm_bo_validate() to see how this might shape up?

Imho the RFC always help... a piece of code to see the idea usually draws more attention from devs than ask in text mode. But more text explaining the reasons behind are also helpful even with the RFC.

Thanks,
Rodrigo.

Quoted 2 lines

-Brian

#2 lore raw

Hi guys,

massive sorry for the delayed response, this mail felt totally through my radar without being noticed.

Am 17.07.23 um 19:24 schrieb Rodrigo Vivi:

Quoted 6 lines · Brian
Quoted 5 lines

Hi Christian / Thomas,

Wanted to ask if you have explored or thought about adding support in TTM such that a ttm_bo could have more than one underlying backing store segment (that is, to have a tree of ttm_resources)?

We already use something similar on amdgpu where basically the VRAM resources are stitched together from multiple backing pages.

That is not exactly the same, but it comes close.

Quoted 7 lines
Quoted 1 line

We are considering to support such BOs for Intel Xe driver.

They are indeed the best one to give an opinion here. I just have some dummy questions and comments below.

Quoted 3 lines

Some of the benefits:
* devices with page fault support can fault (and migrate) backing store at finer granularity than the entire BO

We've considered that once as well and I even started hacking something together, but the problem was that at least at that point it wasn't doable because of limitations in the Linux memory management.

Basically the extended attributes used to control caching of pages where only definable per VMA! So when one piece of the BO would have been in uncached VRAM while another piece would be in cached system system memory you immediately ran into problems.

I think that issue is fixed by now, but I'm not 100% sure.

In general I think it might be beneficial, but I'm not 100% sure if it's worth the additional complexity.

Regards,
Christian.

Quoted 26 lines

what advantage does this bring? to each workload?
is it a performance on huge bo?

Quoted 2 lines

* BOs can support having multiple backing store segments, which can be in different memory domains/regions

what locking challenges would this bring?
is this more targeting gpu + cpu? or only for our multi-tile platforms? and what's the advantage this is bringing to real use cases? (probably the svm/hmm question below answers my questions, but...)

Quoted 1 line

* BO eviction could operate on smaller granularity than entire BO

I believe all the previous doubts apply to this item as well...

Quoted 5 lines

Or is the thinking that workloads should use SVM/HMM instead of GEM_CREATE if they want above benefits?

Is this something you are open to seeing an RFC series that starts perhaps with just extending ttm_bo_validate() to see how this might shape up?

Imho the RFC always help... a piece of code to see the idea usually draws more attention from devs than ask in text mode. But more text explaining the reasons behind are also helpful even with the RFC.

Thanks,
Rodrigo.

Quoted 1 line

-Brian

#3 lore raw

Finally returning to this, thanks for the replies.

Quoted 8 lines · Christian König

Hi guys,

massive sorry for the delayed response, this mail felt totally through my radar without being noticed.

Am 17.07.23 um 19:24 schrieb Rodrigo Vivi:

Quoted 3 lines · Brian
Quoted 2 lines

Hi Christian / Thomas,

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

Wanted to ask if you have explored or thought about adding support in TTM such that a ttm_bo could have more than one underlying backing store segment (that is, to have a tree of ttm_resources)?

We already use something similar on amdgpu where basically the VRAM resources are stitched together from multiple backing pages.

That is not exactly the same, but it comes close.

I tried searching for awhile for this in amdgpu but wasn't able to find it. Didn't see any signs in amdgpu_vram_mgr.c. Can you point me to where this code lives? I wanted to review and compare the approach...

Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

We are considering to support such BOs for Intel Xe driver.

They are indeed the best one to give an opinion here. I just have some dummy questions and comments below.

Quoted 3 lines

Some of the benefits:
  * devices with page fault support can fault (and migrate) backing store at finer granularity than the entire BO

Quoted 5 lines

We've considered that once as well and I even started hacking something together, but the problem was that at least at that point it wasn't doable because of limitations in the Linux memory management.

Basically the extended attributes used to control caching of pages where only definable per VMA! So when one piece of the BO would have been in uncached VRAM while another piece would be in cached system system memory you immediately ran into problems.

I think that issue is fixed by now, but I'm not 100% sure.

Okay, thanks for mentioning. I didn't come across such issue so far...

Quoted 2 lines

In general I think it might be beneficial, but I'm not 100% sure if it's worth the additional complexity.

Agreed. Well, up next is to put small RFC together then...

Quoted 6 lines

Regards,
Christian.

Quoted 2 lines

what advantage does this bring? to each workload?
is it a performance on huge bo?

Replying to Rodrigo's comments for the rest here...
Yes, providing more rationale is needed. I'll see about beefing up the description with the RFC patches...
Bascially, all aspects of working with BO backing store can operate on smaller granularity.
Including being able to support a BO which is larger than total VRAM.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

  * BOs can support having multiple backing store segments, which can be in different memory domains/regions

what locking challenges would this bring?

Intent would be to still have locking done at the BO level, and not attempt to introduce finer grained locking.

Quoted 2 lines
Quoted 2 lines

is this more targeting gpu + cpu? or only for our multi-tile platforms? and what's the advantage this is bringing to real use cases?

Right, it's able to be leveraged for both types of usage you mentioned.
So with both gpu + cpu accessing a BO, the portion of the BO they are accessing can be placed locally. And with an Xe gt0 + gt1 accessing a BO, we can place segments of it in the tile local to the gt.

Quoted 4 lines
Quoted 4 lines

(probably the svm/hmm question below answers my questions, but...)

Quoted 1 line

  * BO eviction could operate on smaller granularity than entire BO

I believe all the previous doubts apply to this item as well...

Not sure what 'all the previous doubts' refers to...
Agree most of the value is lost if eviction is not updated to operate at finer granularity. Will make sure to explore this.

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

Or is the thinking that workloads should use SVM/HMM instead of GEM_CREATE if they want above benefits?

Is this something you are open to seeing an RFC series that starts perhaps with just extending ttm_bo_validate() to see how this might shape up?

Imho the RFC always help... a piece of code to see the idea usually draws more attention from devs than ask in text mode. But more text explaining the reasons behind are also helpful even with the RFC.

Will work up a small RFC and see where we go with this...

Thanks,
-Brian

Quoted 6 lines
Quoted 5 lines

Thanks,
Rodrigo.

Quoted 1 line

-Brian