ttm_bo and multiple backing store segments
Hi Christian / Thomas,
Wanted to ask if you have explored or thought about adding support in TTM such that a ttm_bo could have more than one underlying backing store segment (that is, to have a tree of ttm_resources)? We are considering to support such BOs for Intel Xe driver.
Some of the benefits:
* devices with page fault support can fault (and migrate) backing store at finer granularity than the entire BO
* BOs can support having multiple backing store segments, which can be in different memory domains/regions
* BO eviction could operate on smaller granularity than entire BO
Or is the thinking that workloads should use SVM/HMM instead of GEM_CREATE if they want above benefits?
Is this something you are open to seeing an RFC series that starts perhaps with just extending ttm_bo_validate() to see how this might shape up?
-Brian
Quoted 7 lines · Brian
Hi Christian / Thomas,
Wanted to ask if you have explored or thought about adding support in TTM such that a
ttm_bocould have more than one underlying backing store segment (that is, to have a tree ofttm_resources)?
We are considering to support such BOs for Intel Xe driver.
They are indeed the best one to give an opinion here. I just have some dummy questions and comments below.
Quoted 4 lines
Some of the benefits:
* devices with page fault support can fault (and migrate) backing store at finer granularity than the entire BO
what advantage does this bring? to each workload?
is it a performance on huge bo?
Quoted 2 lines
* BOs can support having multiple backing store segments, which can be in different memory domains/regions
what locking challenges would this bring?
is this more targeting gpu + cpu? or only for our multi-tile platforms? and what's the advantage this is bringing to real use cases? (probably the svm/hmm question below answers my questions, but...)
Quoted 1 line
* BO eviction could operate on smaller granularity than entire BO
I believe all the previous doubts apply to this item as well...
Quoted 6 lines
Or is the thinking that workloads should use SVM/HMM instead of
GEM_CREATEif they want above benefits?Is this something you are open to seeing an RFC series that starts perhaps with just extending
ttm_bo_validate()to see how this might shape up?
Imho the RFC always help... a piece of code to see the idea usually draws more attention from devs than ask in text mode. But more text explaining the reasons behind are also helpful even with the RFC.
Thanks,
Rodrigo.
Quoted 2 lines
-Brian
Hi guys,
massive sorry for the delayed response, this mail felt totally through my radar without being noticed.
Am 17.07.23 um 19:24 schrieb Rodrigo Vivi:
Quoted 6 lines · Brian
Quoted 5 lines
Hi Christian / Thomas,
Wanted to ask if you have explored or thought about adding support in TTM such that a
ttm_bocould have more than one underlying backing store segment (that is, to have a tree ofttm_resources)?
We already use something similar on amdgpu where basically the VRAM resources are stitched together from multiple backing pages.
That is not exactly the same, but it comes close.
Quoted 7 lines
Quoted 1 line
We are considering to support such BOs for Intel Xe driver.
They are indeed the best one to give an opinion here. I just have some dummy questions and comments below.
Quoted 3 lines
Some of the benefits:
* devices with page fault support can fault (and migrate) backing store at finer granularity than the entire BO
We've considered that once as well and I even started hacking something together, but the problem was that at least at that point it wasn't doable because of limitations in the Linux memory management.
Basically the extended attributes used to control caching of pages where only definable per VMA! So when one piece of the BO would have been in uncached VRAM while another piece would be in cached system system memory you immediately ran into problems.
I think that issue is fixed by now, but I'm not 100% sure.
In general I think it might be beneficial, but I'm not 100% sure if it's worth the additional complexity.
Regards,
Christian.
Quoted 26 lines
what advantage does this bring? to each workload?
is it a performance on huge bo?Quoted 2 lines
* BOs can support having multiple backing store segments, which can be in different memory domains/regions
what locking challenges would this bring?
is this more targeting gpu + cpu? or only for our multi-tile platforms? and what's the advantage this is bringing to real use cases? (probably the svm/hmm question below answers my questions, but...)Quoted 1 line
* BO eviction could operate on smaller granularity than entire BO
I believe all the previous doubts apply to this item as well...
Quoted 5 lines
Or is the thinking that workloads should use SVM/HMM instead of
GEM_CREATEif they want above benefits?Is this something you are open to seeing an RFC series that starts perhaps with just extending
ttm_bo_validate()to see how this might shape up?Imho the RFC always help... a piece of code to see the idea usually draws more attention from devs than ask in text mode. But more text explaining the reasons behind are also helpful even with the RFC.
Thanks,
Rodrigo.Quoted 1 line
-Brian
Finally returning to this, thanks for the replies.
Quoted 8 lines · Christian König
Hi guys,
massive sorry for the delayed response, this mail felt totally through my radar without being noticed.
Am 17.07.23 um 19:24 schrieb Rodrigo Vivi:
Quoted 3 lines · Brian
Quoted 2 lines
Hi Christian / Thomas,
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Wanted to ask if you have explored or thought about adding support in TTM such that a
ttm_bocould have more than one underlying backing store segment (that is, to have a tree ofttm_resources)?We already use something similar on amdgpu where basically the VRAM resources are stitched together from multiple backing pages.
That is not exactly the same, but it comes close.
I tried searching for awhile for this in amdgpu but wasn't able to find it. Didn't see any signs in amdgpu_vram_mgr.c. Can you point me to where this code lives? I wanted to review and compare the approach...
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines
We are considering to support such BOs for Intel Xe driver.
They are indeed the best one to give an opinion here. I just have some dummy questions and comments below.
Quoted 3 lines
Some of the benefits:
* devices with page fault support can fault (and migrate) backing store at finer granularity than the entire BO
Quoted 5 lines
We've considered that once as well and I even started hacking something together, but the problem was that at least at that point it wasn't doable because of limitations in the Linux memory management.
Basically the extended attributes used to control caching of pages where only definable per VMA! So when one piece of the BO would have been in uncached VRAM while another piece would be in cached system system memory you immediately ran into problems.
I think that issue is fixed by now, but I'm not 100% sure.
Okay, thanks for mentioning. I didn't come across such issue so far...
Quoted 2 lines
In general I think it might be beneficial, but I'm not 100% sure if it's worth the additional complexity.
Agreed. Well, up next is to put small RFC together then...
Quoted 6 lines
Regards,
Christian.Quoted 2 lines
what advantage does this bring? to each workload?
is it a performance on huge bo?
Replying to Rodrigo's comments for the rest here...
Yes, providing more rationale is needed. I'll see about beefing up the description with the RFC patches...
Bascially, all aspects of working with BO backing store can operate on smaller granularity.
Including being able to support a BO which is larger than total VRAM.
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
* BOs can support having multiple backing store segments, which can be in different memory domains/regions
what locking challenges would this bring?
Intent would be to still have locking done at the BO level, and not attempt to introduce finer grained locking.
Quoted 2 lines
Quoted 2 lines
is this more targeting gpu + cpu? or only for our multi-tile platforms? and what's the advantage this is bringing to real use cases?
Right, it's able to be leveraged for both types of usage you mentioned.
So with both gpu + cpu accessing a BO, the portion of the BO they are accessing can be placed locally. And with an Xe gt0 + gt1 accessing a BO, we can place segments of it in the tile local to the gt.
Quoted 4 lines
Quoted 4 lines
(probably the svm/hmm question below answers my questions, but...)
Quoted 1 line
* BO eviction could operate on smaller granularity than entire BO
I believe all the previous doubts apply to this item as well...
Not sure what 'all the previous doubts' refers to...
Agree most of the value is lost if eviction is not updated to operate at finer granularity. Will make sure to explore this.
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Or is the thinking that workloads should use SVM/HMM instead of
GEM_CREATEif they want above benefits?Is this something you are open to seeing an RFC series that starts perhaps with just extending
ttm_bo_validate()to see how this might shape up?Imho the RFC always help... a piece of code to see the idea usually draws more attention from devs than ask in text mode. But more text explaining the reasons behind are also helpful even with the RFC.
Will work up a small RFC and see where we go with this...
Thanks,
-Brian
Quoted 6 lines
Quoted 5 lines
Thanks,
Rodrigo.Quoted 1 line
-Brian