Skip to contentSkip to conversation

drm/mm: Introduce address space shifting

RFC In review Tomasz Lis 2 participants 8 messages Feb 4, 2025 – Mar 5, 2025

Subsystems (2)
lore
Share & tools
lore

Tomasz Lis

author
#0 lore raw

cover drm/mm: Introduce address space shifting

This RFC asks for introduction of an interface which allows to shift a range managed by drm_mm instance without repeating the node list creation.

The long explanation:

Single Root I/O Virtualization is becoming a standard GFX feature in server environments. Virtual Machines provided with direct access to virtualized GFX hardware, in form of VFs, need to support the standard set of features expected by Virtual Machine Managers. These standard features include ability to save the VM state, and later restore the VM, possibly on another machine with different setup. For the restore to succeed, the GFX hardware model must match; but for its configuration, some differences are often allowed. Such alterations may include a different range of non-virtualized resources assigned to the VF, including global address spaces.

If any non-virtualized address space is saved, as part of VM state, on one machine and restored on another, it may happen that the target range differs from source range. To shift the address space, currently creating a new drm_mm object is required, and moving all nodes to the new object while adding the shift.

GFX hardware handled by Xe driver contains Global Graphics Translation Table, which is an example of such non-virtualized resource. Should this interface change be accepted, a series which utilizes this interface in Xe driver will be prepared.

Tomasz Lis (1):
drm_mm: Introduce address space shifting

 drivers/gpu/drm/drm_mm.c | 24 ++++++++++++++++++++++++
 include/drm/drm_mm.h     |  1 +
 2 files changed, 25 insertions(+)
Signature · 1 line
2.25.1
author
#1 lore raw

RFC 1/1 drm/mm: Introduce address space shifting

Due to resource reprovisioning, sometimes a need arises to move a living address space to a new area, preserving all the nodes and holes stored within.

It is possible to do that by removing all nodes to a temporary list, reiniting the drm_mm instance and re-adding everything while applying a shift to each node. But that is a lot of extra work for a task which could be done internally without any node shuffle operations.

This change introduces an interface which allows to shift the range without pruning the whole drm_mm instance.

Having a drm_mm interface for such shift significantly simplifies the code required to adjust a KMD for a change in base address of a space managed by drm_mm instance.

---
 drivers/gpu/drm/drm_mm.c | 24 ++++++++++++++++++++++++
 include/drm/drm_mm.h     |  1 +
 2 files changed, 25 insertions(+)
drivers/gpu/drm/drm_mm.c +24 -0
index ca254611b382..ce3bd8b5e41f 100644
@@ -917,6 +917,30 @@ struct drm_mm_node *drm_mm_scan_color_evict(struct drm_mm_scan *scan)
 }
 EXPORT_SYMBOL(drm_mm_scan_color_evict);
 
+/**
+ * drm_mm_shift - move the range of addresses managed by this @mm
+ * @mm: the drm_mm structure instance to shift
+ * @shift: the shift value to be added to addresses of all nodes
+ *
+ * The function shifts all nodes by given offset, moving the address space
+ * range managed by this @mm.
+ */
+void drm_mm_shift(struct drm_mm *mm, s64 shift)
+{
+	struct drm_mm_node *node;
+
+	/*
+	 * Head node represents a hole, with negative size and start at the end
+	 * of addressable area. This means it is never present within nodes
+	 * list - needs to be shifted separately.
+	 */
+	mm->head_node.start += shift;
+
+	drm_mm_for_each_node(node, mm)
+		node->start += shift;
+}
+EXPORT_SYMBOL(drm_mm_shift);
+
 /**
  * drm_mm_init - initialize a drm-mm allocator
  * @mm: the drm_mm structure to initialize
include/drm/drm_mm.h +1 -0
index f654874c4ce6..798e5a4f07ad 100644
@@ -465,6 +465,7 @@ static inline int drm_mm_insert_node(struct drm_mm *mm,
 void drm_mm_remove_node(struct drm_mm_node *node);
 void drm_mm_init(struct drm_mm *mm, u64 start, u64 size);
 void drm_mm_takedown(struct drm_mm *mm);
+void drm_mm_shift(struct drm_mm *mm, s64 shift);
 
 /**
  * drm_mm_clean - checks whether an allocator is clean
Signature · 1 line
2.25.1
#2 lore raw
In reply to Tomasz Lis

Am 04.02.25 um 23:41 schrieb Tomasz Lis:

Quoted 3 lines

This RFC asks for introduction of an interface which allows to shift a range managed by drm_mm instance without repeating the node list creation.

What do you mean with "shift" here? As far as I can see from the code you just modify the start address of each node, e.g. you manipulate the offset.

Quoted 19 lines

The long explanation:

Single Root I/O Virtualization is becoming a standard GFX feature in server environments. Virtual Machines provided with direct access to virtualized GFX hardware, in form of VFs, need to support the standard set of features expected by Virtual Machine Managers. These standard features include ability to save the VM state, and later restore the VM, possibly on another machine with different setup. For the restore to succeed, the GFX hardware model must match; but for its configuration, some differences are often allowed. Such alterations may include a different range of non-virtualized resources assigned to the VF, including global address spaces.

If any non-virtualized address space is saved, as part of VM state, on one machine and restored on another, it may happen that the target range differs from source range. To shift the address space, currently creating a new drm_mm object is required, and moving all nodes to the new object while adding the shift.

Quoted 4 lines

GFX hardware handled by Xe driver contains Global Graphics Translation Table, which is an example of such non-virtualized resource. Should this interface change be accepted, a series which utilizes this interface in Xe driver will be prepared.

Well that sounds exactly like what AMD is doing, but we just add the VRAM base to the MM node when calculating the final address in the MC address space.

On the other hand AMD hardware has different address spaces, e.g. VRAM in page table always starts at address 0 while in the MC address space it has a certain offset which differs from device to device.

Regards,
Christian.

Quoted 10 lines

Tomasz Lis (1):
drm_mm: Introduce address space shifting

  drivers/gpu/drm/drm_mm.c | 24 ++++++++++++++++++++++++
  include/drm/drm_mm.h     |  1 +
  2 files changed, 25 insertions(+)
#3 lore raw
Quoted 6 lines · Christian König

Am 04.02.25 um 23:41 schrieb Tomasz Lis:

Quoted 3 lines

This RFC asks for introduction of an interface which allows to shift a range managed by drm_mm instance without repeating the node list creation.

What do you mean with "shift" here? As far as I can see from the code you just modify the start address of each node, e.g. you manipulate the offset.

Yes, that's the idea - move the address space to a different base.

Should I use different terminology?

Quoted 20 lines
Quoted 20 lines

The long explanation:

Single Root I/O Virtualization is becoming a standard GFX feature in server environments. Virtual Machines provided with direct access to virtualized GFX hardware, in form of VFs, need to support the standard set of features expected by Virtual Machine Managers. These standard features include ability to save the VM state, and later restore the VM, possibly on another machine with different setup. For the restore to succeed, the GFX hardware model must match; but for its configuration, some differences are often allowed. Such alterations may include a different range of non-virtualized resources assigned to the VF, including global address spaces.

If any non-virtualized address space is saved, as part of VM state, on one machine and restored on another, it may happen that the target range differs from source range. To shift the address space, currently creating a new drm_mm object is required, and moving all nodes to the new object while adding the shift.

Quoted 8 lines
Quoted 4 lines

GFX hardware handled by Xe driver contains Global Graphics Translation Table, which is an example of such non-virtualized resource. Should this interface change be accepted, a series which utilizes this interface in Xe driver will be prepared.

Well that sounds exactly like what AMD is doing, but we just add the VRAM base to the MM node when calculating the final address in the MC address space.

On the other hand AMD hardware has different address spaces, e.g. VRAM in page table always starts at address 0 while in the MC address space it has a certain offset which differs from device to device.

We use the drm_mm in a similar manner - some address spaces start at 0, other use a narrower range. But we do not add any base after creation - we have the nodes at final offsets.

There is one more technique we use in few places to restrict range of valid addresses - we create "balloon" nodes which fill up the inaccessible areas. When the accessible range shifts, we then remove all notes, and add them back with balloons resized and the rest of nodes just moved by an offset.

Using the function proposed in this RFC will actually force us to abandon the ballooning approach for VF Global Gfx Translation Table address space and refactor the code to give only the accessible range to drm_mm. That is why I'm sending the RFC without use example - we will have to refactor the Xe code to use it.

-Tomasz

Quoted 15 lines

Regards,
Christian.

Quoted 10 lines

Tomasz Lis (1):
   drm_mm: Introduce address space shifting

  drivers/gpu/drm/drm_mm.c | 24 ++++++++++++++++++++++++ include/drm/drm_mm.h     |  1 +

  2 files changed, 25 insertions(+)
#4 lore raw

Am 05.02.25 um 17:02 schrieb Lis, Tomasz:

Quoted 7 lines · Christian König
Quoted 5 lines

Am 04.02.25 um 23:41 schrieb Tomasz Lis:

Quoted 3 lines

This RFC asks for introduction of an interface which allows to shift a range managed by drm_mm instance without repeating the node list creation.

Quoted 5 lines
Quoted 1 line

What do you mean with "shift" here? As far as I can see from the code you just modify the start address of each node, e.g. you manipulate the offset.

Yes, that's the idea - move the address space to a different base.

Should I use different terminology?

Yes, absolutely. A shift is usually something different than an offset.

Quoted 27 lines
Quoted 27 lines
Quoted 20 lines

The long explanation:

Single Root I/O Virtualization is becoming a standard GFX feature in server environments. Virtual Machines provided with direct access to virtualized GFX hardware, in form of VFs, need to support the standard set of features expected by Virtual Machine Managers. These standard features include ability to save the VM state, and later restore the VM, possibly on another machine with different setup. For the restore to succeed, the GFX hardware model must match; but for its configuration, some differences are often allowed. Such alterations may include a different range of non-virtualized resources assigned to the VF, including global address spaces.

If any non-virtualized address space is saved, as part of VM state, on one machine and restored on another, it may happen that the target range differs from source range. To shift the address space, currently creating a new drm_mm object is required, and moving all nodes to the new object while adding the shift.

Quoted 4 lines

GFX hardware handled by Xe driver contains Global Graphics Translation Table, which is an example of such non-virtualized resource. Should this interface change be accepted, a series which utilizes this interface in Xe driver will be prepared.

Well that sounds exactly like what AMD is doing, but we just add the VRAM base to the MM node when calculating the final address in the MC address space.

Quoted 7 lines
Quoted 1 line

On the other hand AMD hardware has different address spaces, e.g. VRAM in page table always starts at address 0 while in the MC address space it has a certain offset which differs from device to device.

We use the drm_mm in a similar manner - some address spaces start at 0, other use a narrower range. But we do not add any base after creation - we have the nodes at final offsets.

There is one more technique we use in few places to restrict range of valid addresses - we create "balloon" nodes which fill up the inaccessible areas. When the accessible range shifts, we then remove all notes, and add them back with balloons resized and the rest of nodes just moved by an offset.

Using the function proposed in this RFC will actually force us to abandon the ballooning approach for VF Global Gfx Translation Table address space and refactor the code to give only the accessible range to drm_mm. That is why I'm sending the RFC without use example - we will have to refactor the Xe code to use it.

Well that approach looks overly complex to me.

Just maintain the VF offsets separately to the drm_mm nodes and apply it in your address calculation whenever necessary. That's basically how all other drivers do it as far as I can see.

This way you don't need to update all nodes whenever your base offsets changes.

Regards,
Christian.

Quoted 18 lines

-Tomasz

Quoted 15 lines

Regards,
Christian.

Quoted 10 lines

Tomasz Lis (1):
   drm_mm: Introduce address space shifting

  drivers/gpu/drm/drm_mm.c | 24 ++++++++++++++++++++++++ include/drm/drm_mm.h     |  1 +

  2 files changed, 25 insertions(+)
#5 lore raw
Quoted 6 lines · Christian König

Am 04.03.25 um 16:39 schrieb Lis, Tomasz:

Quoted 3 lines

There was no NACK, and no further questions/comments for a month.

From that, we conclude that the proposed change is considered acceptable.

Well there was also not any comment from Arun nor Matthew who are the de facto maintainers for this particular code.

Quoted 8 lines

Saying that the whole change looks completely unecessary and overly complex to me.

Just maintain the offset for your VF separately to the drm_mm nodes and you don't need to change all nodes any more when that offset changes.

Going to comment on the other reply as well.

Regards,
Christian.

I will make sure to add Arun and Matthew in the further discussion.

Quoted 11 lines · Christian König

Am 05.02.25 um 17:02 schrieb Lis, Tomasz:

Quoted 9 lines · Christian König
Quoted 5 lines

Am 04.02.25 um 23:41 schrieb Tomasz Lis:

Quoted 3 lines

This RFC asks for introduction of an interface which allows to shift a range managed by drm_mm instance without repeating the node list creation.

What do you mean with "shift" here? As far as I can see from the code you just modify the start address of each node, e.g. you manipulate the offset.

Yes, that's the idea - move the address space to a different base.

Should I use different terminology?

Yes, absolutely. A shift is usually something different than an offset.

Ok, instead of "shifting a range" I will refer to that as "adding offset to the range".

Quoted 29 lines
Quoted 29 lines
Quoted 20 lines
Quoted 20 lines

The long explanation:

Single Root I/O Virtualization is becoming a standard GFX feature in server environments. Virtual Machines provided with direct access to virtualized GFX hardware, in form of VFs, need to support the standard set of features expected by Virtual Machine Managers. These standard features include ability to save the VM state, and later restore the VM, possibly on another machine with different setup. For the restore to succeed, the GFX hardware model must match; but for its configuration, some differences are often allowed. Such alterations may include a different range of non-virtualized resources assigned to the VF, including global address spaces.

If any non-virtualized address space is saved, as part of VM state, on one machine and restored on another, it may happen that the target range differs from source range. To shift the address space, currently creating a new drm_mm object is required, and moving all nodes to the new object while adding the shift.

Quoted 7 lines
Quoted 4 lines

GFX hardware handled by Xe driver contains Global Graphics Translation Table, which is an example of such non-virtualized resource. Should this interface change be accepted, a series which utilizes this interface in Xe driver will be prepared.

Well that sounds exactly like what AMD is doing, but we just add the VRAM base to the MM node when calculating the final address in the MC address space.

On the other hand AMD hardware has different address spaces, e.g. VRAM in page table always starts at address 0 while in the MC address space it has a certain offset which differs from device to device.

We use the drm_mm in a similar manner - some address spaces start at 0, other use a narrower range. But we do not add any base after creation - we have the nodes at final offsets.

Quoted 8 lines
Quoted 3 lines

There is one more technique we use in few places to restrict range of valid addresses - we create "balloon" nodes which fill up the inaccessible areas. When the accessible range shifts, we then remove all notes, and add them back with balloons resized and the rest of nodes just moved by an offset.

Using the function proposed in this RFC will actually force us to abandon the ballooning approach for VF Global Gfx Translation Table address space and refactor the code to give only the accessible range to drm_mm. That is why I'm sending the RFC without use example - we will have to refactor the Xe code to use it.

Well that approach looks overly complex to me.

Just maintain the VF offsets separately to the drm_mm nodes and apply it in your address calculation whenever necessary. That's basically how all other drivers do it as far as I can see.

This way you don't need to update all nodes whenever your base offsets changes.

My current implementation, without drm_mm modification, involves removing and re-adding all the nodes. The proposed implementation makes the code simpler, not more complex.

For maintaining VF offsets separately - this could be done, if Matthew/Arun agree to this extra addition within Xe driver. However, if that is the preferred implementation, then why the drm_mm_init() declaration contains a start parameter? Do we really want to have support of offsets within drm_mm, but keep the support incomplete?

Are we sure that the other drivers you mention, went for a solution with extra offset in addition to drm_mm start offset because that is the best engineering solution? Maybe I'm missing something, but a solution with multiple offsets stacking together looks to me like a workaround. A workaround likely introduced to avoid spending the time required for discussions and finding an agreement about improving the common drm_mm utility.

Well, let me know what you think.

-Tomasz

Quoted 18 lines

Regards,
Christian.

Quoted 14 lines

-Tomasz

Quoted 12 lines

Regards,
Christian.

Quoted 9 lines

Tomasz Lis (1):
   drm_mm: Introduce address space shifting

   drivers/gpu/drm/drm_mm.c | 24 ++++++++++++++++++++++++
   include/drm/drm_mm.h     |  1 +
   2 files changed, 25 insertions(+)
#6 lore raw
In reply to Tomasz Lis

There was no NACK, and no further questions/comments for a month.

From that, we conclude that the proposed change is considered acceptable.

In case anyone disagrees, please respond now.

We are starting work on a series which will use the proposed drm_mm function.

-Tomasz

Quoted 37 lines · Tomasz Lis

This RFC asks for introduction of an interface which allows to shift a range managed by drm_mm instance without repeating the node list creation.

The long explanation:

Single Root I/O Virtualization is becoming a standard GFX feature in server environments. Virtual Machines provided with direct access to virtualized GFX hardware, in form of VFs, need to support the standard set of features expected by Virtual Machine Managers. These standard features include ability to save the VM state, and later restore the VM, possibly on another machine with different setup. For the restore to succeed, the GFX hardware model must match; but for its configuration, some differences are often allowed. Such alterations may include a different range of non-virtualized resources assigned to the VF, including global address spaces.

If any non-virtualized address space is saved, as part of VM state, on one machine and restored on another, it may happen that the target range differs from source range. To shift the address space, currently creating a new drm_mm object is required, and moving all nodes to the new object while adding the shift.

GFX hardware handled by Xe driver contains Global Graphics Translation Table, which is an example of such non-virtualized resource. Should this interface change be accepted, a series which utilizes this interface in Xe driver will be prepared.

Tomasz Lis (1):
drm_mm: Introduce address space shifting

  drivers/gpu/drm/drm_mm.c | 24 ++++++++++++++++++++++++
  include/drm/drm_mm.h     |  1 +
  2 files changed, 25 insertions(+)
#7 lore raw

Am 04.03.25 um 16:39 schrieb Lis, Tomasz:

Quoted 3 lines

There was no NACK, and no further questions/comments for a month.

From that, we conclude that the proposed change is considered acceptable.

Well there was also not any comment from Arun nor Matthew who are the de facto maintainers for this particular code.

Saying that the whole change looks completely unecessary and overly complex to me.

Just maintain the offset for your VF separately to the drm_mm nodes and you don't need to change all nodes any more when that offset changes.

Going to comment on the other reply as well.

Regards,
Christian.

Quoted 45 lines

In case anyone disagrees, please respond now.

We are starting work on a series which will use the proposed drm_mm function.

-Tomasz

Quoted 37 lines · Tomasz Lis

This RFC asks for introduction of an interface which allows to shift a range managed by drm_mm instance without repeating the node list creation.

The long explanation:

Single Root I/O Virtualization is becoming a standard GFX feature in server environments. Virtual Machines provided with direct access to virtualized GFX hardware, in form of VFs, need to support the standard set of features expected by Virtual Machine Managers. These standard features include ability to save the VM state, and later restore the VM, possibly on another machine with different setup. For the restore to succeed, the GFX hardware model must match; but for its configuration, some differences are often allowed. Such alterations may include a different range of non-virtualized resources assigned to the VF, including global address spaces.

If any non-virtualized address space is saved, as part of VM state, on one machine and restored on another, it may happen that the target range differs from source range. To shift the address space, currently creating a new drm_mm object is required, and moving all nodes to the new object while adding the shift.

GFX hardware handled by Xe driver contains Global Graphics Translation Table, which is an example of such non-virtualized resource. Should this interface change be accepted, a series which utilizes this interface in Xe driver will be prepared.

Tomasz Lis (1):
   drm_mm: Introduce address space shifting

  drivers/gpu/drm/drm_mm.c | 24 ++++++++++++++++++++++++ include/drm/drm_mm.h     |  1 +

  2 files changed, 25 insertions(+)