drm/mm: Introduce address space shifting
Subsystems (2)
cover drm/mm: Introduce address space shifting
This RFC asks for introduction of an interface which allows to shift a range managed by drm_mm instance without repeating the node list creation.
The long explanation:
Single Root I/O Virtualization is becoming a standard GFX feature in server environments. Virtual Machines provided with direct access to virtualized GFX hardware, in form of VFs, need to support the standard set of features expected by Virtual Machine Managers. These standard features include ability to save the VM state, and later restore the VM, possibly on another machine with different setup. For the restore to succeed, the GFX hardware model must match; but for its configuration, some differences are often allowed. Such alterations may include a different range of non-virtualized resources assigned to the VF, including global address spaces.
If any non-virtualized address space is saved, as part of VM state, on one machine and restored on another, it may happen that the target range differs from source range. To shift the address space, currently creating a new drm_mm object is required, and moving all nodes to the new object while adding the shift.
GFX hardware handled by Xe driver contains Global Graphics Translation Table, which is an example of such non-virtualized resource. Should this interface change be accepted, a series which utilizes this interface in Xe driver will be prepared.
Tomasz Lis (1):
drm_mm: Introduce address space shifting
drivers/gpu/drm/drm_mm.c | 24 ++++++++++++++++++++++++ include/drm/drm_mm.h | 1 + 2 files changed, 25 insertions(+)
Signature · 1 line
2.25.1
RFC 1/1 drm/mm: Introduce address space shifting
Due to resource reprovisioning, sometimes a need arises to move a living address space to a new area, preserving all the nodes and holes stored within.
It is possible to do that by removing all nodes to a temporary list, reiniting the drm_mm instance and re-adding everything while applying a shift to each node. But that is a lot of extra work for a task which could be done internally without any node shuffle operations.
This change introduces an interface which allows to shift the range without pruning the whole drm_mm instance.
Having a drm_mm interface for such shift significantly simplifies the code required to adjust a KMD for a change in base address of a space managed by drm_mm instance.
--- drivers/gpu/drm/drm_mm.c | 24 ++++++++++++++++++++++++ include/drm/drm_mm.h | 1 + 2 files changed, 25 insertions(+)
drivers/gpu/drm/drm_mm.c +24 -0
@@ -917,6 +917,30 @@ struct drm_mm_node *drm_mm_scan_color_evict(struct drm_mm_scan *scan) }EXPORT_SYMBOL (drm_mm_scan_color_evict); +/** +* drm_mm_shift - move the range of addresses managed by this @mm +* @mm: the drm_mm structure instance to shift +* @shift: the shift value to be added to addresses of all nodes +* +* The function shifts all nodes by given offset, moving the address space +* range managed by this @mm. +*/ +void drm_mm_shift (struct drm_mm *mm,s64 shift) +{ +struct drm_mm_node *node; + +/* +* Head node represents a hole, with negative size and start at the end +* of addressable area. This means it is never present within nodes +* list - needs to be shifted separately. +*/ + mm->head_node.start += shift; + +drm_mm_for_each_node (node, mm) + node->start += shift; +} +EXPORT_SYMBOL (drm_mm_shift); +/** * drm_mm_init - initialize a drm-mm allocator * @mm: the drm_mm structure to initialize
include/drm/drm_mm.h +1 -0
@@ -465,6 +465,7 @@ static inline int drm_mm_insert_node(struct drm_mm *mm,void drm_mm_remove_node (struct drm_mm_node *node);void drm_mm_init (struct drm_mm *mm,u64 start,u64 size);void drm_mm_takedown (struct drm_mm *mm); +void drm_mm_shift (struct drm_mm *mm,s64 shift);/** * drm_mm_clean - checks whether an allocator is clean
Signature · 1 line
2.25.1
Am 04.02.25 um 23:41 schrieb Tomasz Lis:
Quoted 3 lines
This RFC asks for introduction of an interface which allows to shift a range managed by
drm_mminstance without repeating the node list creation.
What do you mean with "shift" here? As far as I can see from the code you just modify the start address of each node, e.g. you manipulate the offset.
Quoted 19 lines
The long explanation:
Single Root I/O Virtualization is becoming a standard GFX feature in server environments. Virtual Machines provided with direct access to virtualized GFX hardware, in form of VFs, need to support the standard set of features expected by Virtual Machine Managers. These standard features include ability to save the VM state, and later restore the VM, possibly on another machine with different setup. For the restore to succeed, the GFX hardware model must match; but for its configuration, some differences are often allowed. Such alterations may include a different range of non-virtualized resources assigned to the VF, including global address spaces.
If any non-virtualized address space is saved, as part of VM state, on one machine and restored on another, it may happen that the target range differs from source range. To shift the address space, currently creating a new
drm_mmobject is required, and moving all nodes to the new object while adding the shift.
Quoted 4 lines
GFX hardware handled by Xe driver contains Global Graphics Translation Table, which is an example of such non-virtualized resource. Should this interface change be accepted, a series which utilizes this interface in Xe driver will be prepared.
Well that sounds exactly like what AMD is doing, but we just add the VRAM base to the MM node when calculating the final address in the MC address space.
On the other hand AMD hardware has different address spaces, e.g. VRAM in page table always starts at address 0 while in the MC address space it has a certain offset which differs from device to device.
Regards,
Christian.
Quoted 10 lines
Signed-off-by Tomasz Lis <[email protected]>Tomasz Lis (1):
drm_mm: Introduce address space shiftingdrivers/gpu/drm/drm_mm.c | 24 ++++++++++++++++++++++++ include/drm/drm_mm.h | 1 + 2 files changed, 25 insertions(+)
Quoted 6 lines · Christian König
Am 04.02.25 um 23:41 schrieb Tomasz Lis:
Quoted 3 lines
This RFC asks for introduction of an interface which allows to shift a range managed by
drm_mminstance without repeating the node list creation.What do you mean with "shift" here? As far as I can see from the code you just modify the start address of each node, e.g. you manipulate the offset.
Yes, that's the idea - move the address space to a different base.
Should I use different terminology?
Quoted 20 lines
Quoted 20 lines
The long explanation:
Single Root I/O Virtualization is becoming a standard GFX feature in server environments. Virtual Machines provided with direct access to virtualized GFX hardware, in form of VFs, need to support the standard set of features expected by Virtual Machine Managers. These standard features include ability to save the VM state, and later restore the VM, possibly on another machine with different setup. For the restore to succeed, the GFX hardware model must match; but for its configuration, some differences are often allowed. Such alterations may include a different range of non-virtualized resources assigned to the VF, including global address spaces.
If any non-virtualized address space is saved, as part of VM state, on one machine and restored on another, it may happen that the target range differs from source range. To shift the address space, currently creating a new
drm_mmobject is required, and moving all nodes to the new object while adding the shift.
Quoted 8 lines
Quoted 4 lines
GFX hardware handled by Xe driver contains Global Graphics Translation Table, which is an example of such non-virtualized resource. Should this interface change be accepted, a series which utilizes this interface in Xe driver will be prepared.
Well that sounds exactly like what AMD is doing, but we just add the VRAM base to the MM node when calculating the final address in the MC address space.
On the other hand AMD hardware has different address spaces, e.g. VRAM in page table always starts at address 0 while in the MC address space it has a certain offset which differs from device to device.
We use the drm_mm in a similar manner - some address spaces start at 0, other use a narrower range. But we do not add any base after creation - we have the nodes at final offsets.
There is one more technique we use in few places to restrict range of valid addresses - we create "balloon" nodes which fill up the inaccessible areas. When the accessible range shifts, we then remove all notes, and add them back with balloons resized and the rest of nodes just moved by an offset.
Using the function proposed in this RFC will actually force us to abandon the ballooning approach for VF Global Gfx Translation Table address space and refactor the code to give only the accessible range to drm_mm. That is why I'm sending the RFC without use example - we will have to refactor the Xe code to use it.
-Tomasz
Quoted 15 lines
Regards,
Christian.Quoted 10 lines
Signed-off-by Tomasz Lis <[email protected]>Tomasz Lis (1):
drm_mm: Introduce address space shifting
drivers/gpu/drm/drm_mm.c| 24 ++++++++++++++++++++++++ include/drm/drm_mm.h | 1 +2 files changed, 25 insertions(+)
Am 05.02.25 um 17:02 schrieb Lis, Tomasz:
Quoted 7 lines · Christian König
Quoted 5 lines
Am 04.02.25 um 23:41 schrieb Tomasz Lis:
Quoted 3 lines
This RFC asks for introduction of an interface which allows to shift a range managed by
drm_mminstance without repeating the node list creation.
Quoted 5 lines
Quoted 1 line
What do you mean with "shift" here? As far as I can see from the code you just modify the start address of each node, e.g. you manipulate the offset.
Yes, that's the idea - move the address space to a different base.
Should I use different terminology?
Yes, absolutely. A shift is usually something different than an offset.
Quoted 27 lines
Quoted 27 lines
Quoted 20 lines
The long explanation:
Single Root I/O Virtualization is becoming a standard GFX feature in server environments. Virtual Machines provided with direct access to virtualized GFX hardware, in form of VFs, need to support the standard set of features expected by Virtual Machine Managers. These standard features include ability to save the VM state, and later restore the VM, possibly on another machine with different setup. For the restore to succeed, the GFX hardware model must match; but for its configuration, some differences are often allowed. Such alterations may include a different range of non-virtualized resources assigned to the VF, including global address spaces.
If any non-virtualized address space is saved, as part of VM state, on one machine and restored on another, it may happen that the target range differs from source range. To shift the address space, currently creating a new
drm_mmobject is required, and moving all nodes to the new object while adding the shift.Quoted 4 lines
GFX hardware handled by Xe driver contains Global Graphics Translation Table, which is an example of such non-virtualized resource. Should this interface change be accepted, a series which utilizes this interface in Xe driver will be prepared.
Well that sounds exactly like what AMD is doing, but we just add the VRAM base to the MM node when calculating the final address in the MC address space.
Quoted 7 lines
Quoted 1 line
On the other hand AMD hardware has different address spaces, e.g. VRAM in page table always starts at address 0 while in the MC address space it has a certain offset which differs from device to device.
We use the
drm_mmin a similar manner - some address spaces start at 0, other use a narrower range. But we do not add any base after creation - we have the nodes at final offsets.There is one more technique we use in few places to restrict range of valid addresses - we create "balloon" nodes which fill up the inaccessible areas. When the accessible range shifts, we then remove all notes, and add them back with balloons resized and the rest of nodes just moved by an offset.
Using the function proposed in this RFC will actually force us to abandon the ballooning approach for VF Global Gfx Translation Table address space and refactor the code to give only the accessible range to
drm_mm. That is why I'm sending the RFC without use example - we will have to refactor the Xe code to use it.
Well that approach looks overly complex to me.
Just maintain the VF offsets separately to the drm_mm nodes and apply it in your address calculation whenever necessary. That's basically how all other drivers do it as far as I can see.
This way you don't need to update all nodes whenever your base offsets changes.
Regards,
Christian.
Quoted 18 lines
-Tomasz
Quoted 15 lines
Regards,
Christian.Quoted 10 lines
Signed-off-by Tomasz Lis <[email protected]>Tomasz Lis (1):
drm_mm: Introduce address space shifting
drivers/gpu/drm/drm_mm.c| 24 ++++++++++++++++++++++++ include/drm/drm_mm.h | 1 +2 files changed, 25 insertions(+)
Quoted 6 lines · Christian König
Am 04.03.25 um 16:39 schrieb Lis, Tomasz:
Quoted 3 lines
There was no NACK, and no further questions/comments for a month.
From that, we conclude that the proposed change is considered acceptable.
Well there was also not any comment from Arun nor Matthew who are the de facto maintainers for this particular code.
Quoted 8 lines
Saying that the whole change looks completely unecessary and overly complex to me.
Just maintain the offset for your VF separately to the
drm_mmnodes and you don't need to change all nodes any more when that offset changes.Going to comment on the other reply as well.
Regards,
Christian.
I will make sure to add Arun and Matthew in the further discussion.
Quoted 11 lines · Christian König
Am 05.02.25 um 17:02 schrieb Lis, Tomasz:
Quoted 9 lines · Christian König
Quoted 5 lines
Am 04.02.25 um 23:41 schrieb Tomasz Lis:
Quoted 3 lines
This RFC asks for introduction of an interface which allows to shift a range managed by
drm_mminstance without repeating the node list creation.What do you mean with "shift" here? As far as I can see from the code you just modify the start address of each node, e.g. you manipulate the offset.
Yes, that's the idea - move the address space to a different base.
Should I use different terminology?
Yes, absolutely. A shift is usually something different than an offset.
Ok, instead of "shifting a range" I will refer to that as "adding offset to the range".
Quoted 29 lines
Quoted 29 lines
Quoted 20 lines
Quoted 20 lines
The long explanation:
Single Root I/O Virtualization is becoming a standard GFX feature in server environments. Virtual Machines provided with direct access to virtualized GFX hardware, in form of VFs, need to support the standard set of features expected by Virtual Machine Managers. These standard features include ability to save the VM state, and later restore the VM, possibly on another machine with different setup. For the restore to succeed, the GFX hardware model must match; but for its configuration, some differences are often allowed. Such alterations may include a different range of non-virtualized resources assigned to the VF, including global address spaces.
If any non-virtualized address space is saved, as part of VM state, on one machine and restored on another, it may happen that the target range differs from source range. To shift the address space, currently creating a new
drm_mmobject is required, and moving all nodes to the new object while adding the shift.Quoted 7 lines
Quoted 4 lines
GFX hardware handled by Xe driver contains Global Graphics Translation Table, which is an example of such non-virtualized resource. Should this interface change be accepted, a series which utilizes this interface in Xe driver will be prepared.
Well that sounds exactly like what AMD is doing, but we just add the VRAM base to the MM node when calculating the final address in the MC address space.
On the other hand AMD hardware has different address spaces, e.g. VRAM in page table always starts at address 0 while in the MC address space it has a certain offset which differs from device to device.
We use the
drm_mmin a similar manner - some address spaces start at 0, other use a narrower range. But we do not add any base after creation - we have the nodes at final offsets.
Quoted 8 lines
Quoted 3 lines
There is one more technique we use in few places to restrict range of valid addresses - we create "balloon" nodes which fill up the inaccessible areas. When the accessible range shifts, we then remove all notes, and add them back with balloons resized and the rest of nodes just moved by an offset.
Using the function proposed in this RFC will actually force us to abandon the ballooning approach for VF Global Gfx Translation Table address space and refactor the code to give only the accessible range to
drm_mm. That is why I'm sending the RFC without use example - we will have to refactor the Xe code to use it.Well that approach looks overly complex to me.
Just maintain the VF offsets separately to the
drm_mmnodes and apply it in your address calculation whenever necessary. That's basically how all other drivers do it as far as I can see.This way you don't need to update all nodes whenever your base offsets changes.
My current implementation, without drm_mm modification, involves removing and re-adding all the nodes. The proposed implementation makes the code simpler, not more complex.
For maintaining VF offsets separately - this could be done, if Matthew/Arun agree to this extra addition within Xe driver. However, if that is the preferred implementation, then why the drm_mm_init() declaration contains a start parameter? Do we really want to have support of offsets within drm_mm, but keep the support incomplete?
Are we sure that the other drivers you mention, went for a solution with extra offset in addition to drm_mm start offset because that is the best engineering solution? Maybe I'm missing something, but a solution with multiple offsets stacking together looks to me like a workaround. A workaround likely introduced to avoid spending the time required for discussions and finding an agreement about improving the common drm_mm utility.
Well, let me know what you think.
-Tomasz
Quoted 18 lines
Regards,
Christian.Quoted 14 lines
-Tomasz
Quoted 12 lines
Regards,
Christian.Quoted 9 lines
Signed-off-by Tomasz Lis <[email protected]>Tomasz Lis (1):
drm_mm: Introduce address space shiftingdrivers/gpu/drm/drm_mm.c | 24 ++++++++++++++++++++++++ include/drm/drm_mm.h | 1 + 2 files changed, 25 insertions(+)
There was no NACK, and no further questions/comments for a month.
From that, we conclude that the proposed change is considered acceptable.
In case anyone disagrees, please respond now.
We are starting work on a series which will use the proposed drm_mm function.
-Tomasz
Quoted 37 lines · Tomasz Lis
This RFC asks for introduction of an interface which allows to shift a range managed by
drm_mminstance without repeating the node list creation.The long explanation:
Single Root I/O Virtualization is becoming a standard GFX feature in server environments. Virtual Machines provided with direct access to virtualized GFX hardware, in form of VFs, need to support the standard set of features expected by Virtual Machine Managers. These standard features include ability to save the VM state, and later restore the VM, possibly on another machine with different setup. For the restore to succeed, the GFX hardware model must match; but for its configuration, some differences are often allowed. Such alterations may include a different range of non-virtualized resources assigned to the VF, including global address spaces.
If any non-virtualized address space is saved, as part of VM state, on one machine and restored on another, it may happen that the target range differs from source range. To shift the address space, currently creating a new
drm_mmobject is required, and moving all nodes to the new object while adding the shift.GFX hardware handled by Xe driver contains Global Graphics Translation Table, which is an example of such non-virtualized resource. Should this interface change be accepted, a series which utilizes this interface in Xe driver will be prepared.
Signed-off-by Tomasz Lis <[email protected]>Tomasz Lis (1):
drm_mm: Introduce address space shiftingdrivers/gpu/drm/drm_mm.c | 24 ++++++++++++++++++++++++ include/drm/drm_mm.h | 1 + 2 files changed, 25 insertions(+)
Am 04.03.25 um 16:39 schrieb Lis, Tomasz:
Quoted 3 lines
There was no NACK, and no further questions/comments for a month.
From that, we conclude that the proposed change is considered acceptable.
Well there was also not any comment from Arun nor Matthew who are the de facto maintainers for this particular code.
Saying that the whole change looks completely unecessary and overly complex to me.
Just maintain the offset for your VF separately to the drm_mm nodes and you don't need to change all nodes any more when that offset changes.
Going to comment on the other reply as well.
Regards,
Christian.
Quoted 45 lines
In case anyone disagrees, please respond now.
We are starting work on a series which will use the proposed
drm_mmfunction.-Tomasz
Quoted 37 lines · Tomasz Lis
This RFC asks for introduction of an interface which allows to shift a range managed by
drm_mminstance without repeating the node list creation.The long explanation:
Single Root I/O Virtualization is becoming a standard GFX feature in server environments. Virtual Machines provided with direct access to virtualized GFX hardware, in form of VFs, need to support the standard set of features expected by Virtual Machine Managers. These standard features include ability to save the VM state, and later restore the VM, possibly on another machine with different setup. For the restore to succeed, the GFX hardware model must match; but for its configuration, some differences are often allowed. Such alterations may include a different range of non-virtualized resources assigned to the VF, including global address spaces.
If any non-virtualized address space is saved, as part of VM state, on one machine and restored on another, it may happen that the target range differs from source range. To shift the address space, currently creating a new
drm_mmobject is required, and moving all nodes to the new object while adding the shift.GFX hardware handled by Xe driver contains Global Graphics Translation Table, which is an example of such non-virtualized resource. Should this interface change be accepted, a series which utilizes this interface in Xe driver will be prepared.
Signed-off-by Tomasz Lis <[email protected]>Tomasz Lis (1):
drm_mm: Introduce address space shifting
drivers/gpu/drm/drm_mm.c| 24 ++++++++++++++++++++++++ include/drm/drm_mm.h | 1 +2 files changed, 25 insertions(+)