Skip to contentSkip to conversation

virtio-mem: add memory via add_memory_driver_managed()

PATCH In review David Hildenbrand 3 participants 8 messages Jun 11, 2020

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 virtio-mem: add memory via add_memory_driver_managed()

Virtio-mem managed memory is always detected and added by the virtio-mem driver, never using something like the firmware-provided memory map. This is the case after an ordinary system reboot, and has to be guaranteed after kexec. Especially, virtio-mem added memory resources can contain inaccessible parts ("unblocked memory blocks"), blindly forwarding them to a kexec kernel is dangerous, as unplugged memory will get accessed (esp. written).

Let's use the new way of adding special driver-managed memory introduced in commit 75ac4c58bc0d ("mm/memory_hotplug: introduce add_memory_driver_managed()").

This will result in no entries in /sys/firmware/memmap ("raw firmware- provided memory map"), the memory resource will be flagged IORESOURCE_MEM_DRIVER_MANAGED (esp., kexec_file_load() will not place kexec images on this memory), and it is exposed as "System RAM (virtio_mem)" in /proc/iomem, so esp. kexec-tools can properly handle it.

Example /proc/iomem before this change:
[...]
140000000-333ffffff : virtio0

    140000000-147ffffff : System RAM
  334000000-533ffffff : virtio1
    338000000-33fffffff : System RAM
    340000000-347ffffff : System RAM
    348000000-34fffffff : System RAM

[...]

Example /proc/iomem after this change:
[...]
140000000-333ffffff : virtio0

    140000000-147ffffff : System RAM (virtio_mem)
  334000000-533ffffff : virtio1
    338000000-33fffffff : System RAM (virtio_mem)
    340000000-347ffffff : System RAM (virtio_mem)
    348000000-34fffffff : System RAM (virtio_mem)

[...]

Cc "Michael S. Tsirkin" <[email protected]>
Cc Pankaj Gupta <[email protected]>
Cc teawater <[email protected]>

---

Based on latest Linus' tree (and not a tag) because
- virtio-mem has just been merged via the vhost tree
- add_memory_driver_managed() has been merged a week ago via the -mm tree

I'd like to have this patch in 5.8, with the initial merge of virtio-mem if possible (so the user space representation of virtio-mem added memory resources won't change anymore).

---
 drivers/virtio/virtio_mem.c | 25 ++++++++++++++++++++++---
 1 file changed, 22 insertions(+), 3 deletions(-)
drivers/virtio/virtio_mem.c +22 -3
index 50c689f250450..d2eab3558a9e1 100644
@@ -101,6 +101,11 @@ struct virtio_mem {
 
 	/* The parent resource for all memory added via this device. */
 	struct resource *parent_resource;
+	/*
+	 * Copy of "System RAM (virtio_mem)" to be used for
+	 * add_memory_driver_managed().
+	 */
+	const char *resource_name;
 
 	/* Summary of all memory block states. */
 	unsigned long nb_mb_state[VIRTIO_MEM_MB_STATE_COUNT];
@@ -414,8 +419,20 @@ static int virtio_mem_mb_add(struct virtio_mem *vm, unsigned long mb_id)
 	if (nid == NUMA_NO_NODE)
 		nid = memory_add_physaddr_to_nid(addr);
 
+	/*
+	 * When force-unloading the driver and we still have memory added to
+	 * Linux, the resource name has to stay.
+	 */
+	if (!vm->resource_name) {
+		vm->resource_name = kstrdup_const("System RAM (virtio_mem)",
+						  GFP_KERNEL);
+		if (!vm->resource_name)
+			return -ENOMEM;
+	}
+
 	dev_dbg(&vm->vdev->dev, "adding memory block: %lu\n", mb_id);
-	return add_memory(nid, addr, memory_block_size_bytes());
+	return add_memory_driver_managed(nid, addr, memory_block_size_bytes(),
+					 vm->resource_name);
 }
 
 /*
@@ -1890,10 +1907,12 @@ static void virtio_mem_remove(struct virtio_device *vdev)
 	    vm->nb_mb_state[VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL] ||
 	    vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE] ||
 	    vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL] ||
-	    vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE_MOVABLE])
+	    vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE_MOVABLE]) {
 		dev_warn(&vdev->dev, "device still has system memory added\n");
-	else
+	} else {
 		virtio_mem_delete_resource(vm);
+		kfree_const(vm->resource_name);
+	}
 
 	/* remove all tracking data - no locking needed */
 	vfree(vm->mb_state);
Signature · 1 line
2.26.2
#1 lore raw

virtio-mem: add memory via add_memory_driver_managed()

Quoted 44 lines · David Hildenbrand

Virtio-mem managed memory is always detected and added by the virtio-mem driver, never using something like the firmware-provided memory map. This is the case after an ordinary system reboot, and has to be guaranteed after kexec. Especially, virtio-mem added memory resources can contain inaccessible parts ("unblocked memory blocks"), blindly forwarding them to a kexec kernel is dangerous, as unplugged memory will get accessed (esp. written).

Let's use the new way of adding special driver-managed memory introduced in commit 75ac4c58bc0d ("mm/memory_hotplug: introduce add_memory_driver_managed()").

This will result in no entries in /sys/firmware/memmap ("raw firmware- provided memory map"), the memory resource will be flagged IORESOURCE_MEM_DRIVER_MANAGED (esp., kexec_file_load() will not place kexec images on this memory), and it is exposed as "System RAM (virtio_mem)" in /proc/iomem, so esp. kexec-tools can properly handle it.

Example /proc/iomem before this change:
[...]
140000000-333ffffff : virtio0

    140000000-147ffffff : System RAM
  334000000-533ffffff : virtio1
    338000000-33fffffff : System RAM
    340000000-347ffffff : System RAM
    348000000-34fffffff : System RAM

[...]

Example /proc/iomem after this change:
[...]
140000000-333ffffff : virtio0

    140000000-147ffffff : System RAM (virtio_mem)
  334000000-533ffffff : virtio1
    338000000-33fffffff : System RAM (virtio_mem)
    340000000-347ffffff : System RAM (virtio_mem)
    348000000-34fffffff : System RAM (virtio_mem)

[...]

Cc "Michael S. Tsirkin" <[email protected]>
Cc Pankaj Gupta <[email protected]>
Cc teawater <[email protected]>

---

Quoted 7 lines

Based on latest Linus' tree (and not a tag) because
- virtio-mem has just been merged via the vhost tree
- add_memory_driver_managed() has been merged a week ago via the -mm tree

I'd like to have this patch in 5.8, with the initial merge of virtio-mem if possible (so the user space representation of virtio-mem added memory resources won't change anymore).

So my plan is to rebase on top of -rc1 and merge this for rc2 then. I don't like rebase on top of tip as the results are sometimes kind of random.
And let's add a Fixes: tag as well, this way people will remember to pick this.
Makes sense?

Quoted 59 lines
---
 drivers/virtio/virtio_mem.c | 25 ++++++++++++++++++++++---
 1 file changed, 22 insertions(+), 3 deletions(-)
drivers/virtio/virtio_mem.c +22 -3
index 50c689f250450..d2eab3558a9e1 100644
@@ -101,6 +101,11 @@ struct virtio_mem {
 
 	/* The parent resource for all memory added via this device. */
 	struct resource *parent_resource;
+	/*
+	 * Copy of "System RAM (virtio_mem)" to be used for
+	 * add_memory_driver_managed().
+	 */
+	const char *resource_name;
 
 	/* Summary of all memory block states. */
 	unsigned long nb_mb_state[VIRTIO_MEM_MB_STATE_COUNT];
@@ -414,8 +419,20 @@ static int virtio_mem_mb_add(struct virtio_mem *vm, unsigned long mb_id)
 	if (nid == NUMA_NO_NODE)
 		nid = memory_add_physaddr_to_nid(addr);
 
+	/*
+	 * When force-unloading the driver and we still have memory added to
+	 * Linux, the resource name has to stay.
+	 */
+	if (!vm->resource_name) {
+		vm->resource_name = kstrdup_const("System RAM (virtio_mem)",
+						  GFP_KERNEL);
+		if (!vm->resource_name)
+			return -ENOMEM;
+	}
+
 	dev_dbg(&vm->vdev->dev, "adding memory block: %lu\n", mb_id);
-	return add_memory(nid, addr, memory_block_size_bytes());
+	return add_memory_driver_managed(nid, addr, memory_block_size_bytes(),
+					 vm->resource_name);
 }
 
 /*
@@ -1890,10 +1907,12 @@ static void virtio_mem_remove(struct virtio_device *vdev)
 	    vm->nb_mb_state[VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL] ||
 	    vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE] ||
 	    vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL] ||
-	    vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE_MOVABLE])
+	    vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE_MOVABLE]) {
 		dev_warn(&vdev->dev, "device still has system memory added\n");
-	else
+	} else {
 		virtio_mem_delete_resource(vm);
+		kfree_const(vm->resource_name);
+	}
 
 	/* remove all tracking data - no locking needed */
 	vfree(vm->mb_state);
Signature · 1 line
2.26.2
#2 lore raw
Quoted 7 lines
Quoted 3 lines

I'd like to have this patch in 5.8, with the initial merge of virtio-mem if possible (so the user space representation of virtio-mem added memory resources won't change anymore).

So my plan is to rebase on top of -rc1 and merge this for rc2 then. I don't like rebase on top of tip as the results are sometimes kind of random.

Right, I just wanted to get this out early so we can discuss how to proceed.

Quoted 3 lines

And let's add a Fixes: tag as well, this way people will remember to pick this.
Makes sense?

Yes, it's somehow a fix (for kexec). So

Fixes 5f1f79bbc9e26 ("virtio-mem: Paravirtualized memory hotplug")

I can respin after -rc1 with the commit id fixed as noted by Pankaj. Just let me know what you prefer.

Thanks!

Signature · 3 lines
Thanks,

David / dhildenb
#3 lore raw
Quoted 14 lines · David Hildenbrand
Quoted 7 lines
Quoted 3 lines

I'd like to have this patch in 5.8, with the initial merge of virtio-mem if possible (so the user space representation of virtio-mem added memory resources won't change anymore).

So my plan is to rebase on top of -rc1 and merge this for rc2 then. I don't like rebase on top of tip as the results are sometimes kind of random.

Right, I just wanted to get this out early so we can discuss how to proceed.

Quoted 3 lines

And let's add a Fixes: tag as well, this way people will remember to pick this.
Makes sense?

Quoted 8 lines

Yes, it's somehow a fix (for kexec). So

Fixes 5f1f79bbc9e26 ("virtio-mem: Paravirtualized memory hotplug")

I can respin after -rc1 with the commit id fixed as noted by Pankaj. Just let me know what you prefer.

Thanks!

Some once this commit is in Linus' tree, please ping me.

Quoted 4 lines
Signature · 3 lines
Thanks,

David / dhildenb
#4 lore raw
Quoted 21 lines

Am 11.06.2020 um 13:18 schrieb Michael S. Tsirkin <[email protected]>:

Quoted 16 lines · David Hildenbrand
Quoted 7 lines
Quoted 3 lines

I'd like to have this patch in 5.8, with the initial merge of virtio-mem if possible (so the user space representation of virtio-mem added memory resources won't change anymore).

So my plan is to rebase on top of -rc1 and merge this for rc2 then. I don't like rebase on top of tip as the results are sometimes kind of random.

Right, I just wanted to get this out early so we can discuss how to proceed.

Quoted 3 lines

And let's add a Fixes: tag as well, this way people will remember to pick this.
Makes sense?

Yes, it's somehow a fix (for kexec). So

Quoted 8 lines
Quoted 6 lines
Fixes 5f1f79bbc9e26 ("virtio-mem: Paravirtualized memory hotplug")

I can respin after -rc1 with the commit id fixed as noted by Pankaj. Just let me know what you prefer.

Thanks!

Some once this commit is in Linus' tree, please ping me.

It already is as mentioned, only the id was wrong.

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?id=7b7b27214bba1966772f9213cd2d8e5d67f8487f

Quoted 6 lines
Quoted 5 lines
Signature · 4 lines
-- 
Thanks,

David / dhildenb
#5 lore raw
Quoted 57 lines · David Hildenbrand

Am 11.06.2020 um 13:18 schrieb Michael S. Tsirkin <[email protected]>:

On Thu, Jun 11, 2020 at 01:00:24PM +0200, David Hildenbrand wrote:

I'd like to have this patch in 5.8, with the initial merge of virtio-mem

if possible (so the user space representation of virtio-mem added memory

resources won't change anymore).

So my plan is to rebase on top of -rc1 and merge this for rc2 then.

I don't like rebase on top of tip as the results are sometimes kind of

random.

Right, I just wanted to get this out early so we can discuss how to proceed.

And let's add a Fixes: tag as well, this way people will remember to

pick this.

Makes sense?

Yes, it's somehow a fix (for kexec). So

Fixes 5f1f79bbc9e26 ("virtio-mem: Paravirtualized memory hotplug")

I can respin after -rc1 with the commit id fixed as noted by Pankaj.

Just let me know what you prefer.

Thanks!

Quoted 7 lines

Some once this commit is in Linus' tree, please ping me.

It already is as mentioned, only the id was wrong.

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?id=
7b7b27214bba1966772f9213cd2d8e5d67f8487f

OK I pushed this into next based on tip. Let's see what happens.

Quoted 13 lines

--

Thanks,

David / dhildenb

Reviewed-by on v1 1/1
#6 lore raw
In reply to David Hildenbrand
Quoted 11 lines

Virtio-mem managed memory is always detected and added by the virtio-mem driver, never using something like the firmware-provided memory map. This is the case after an ordinary system reboot, and has to be guaranteed after kexec. Especially, virtio-mem added memory resources can contain inaccessible parts ("unblocked memory blocks"), blindly forwarding them to a kexec kernel is dangerous, as unplugged memory will get accessed (esp. written).

Let's use the new way of adding special driver-managed memory introduced in commit 75ac4c58bc0d ("mm/memory_hotplug: introduce add_memory_driver_managed()").

Is this commit id correct?

Quoted 88 lines

This will result in no entries in /sys/firmware/memmap ("raw firmware- provided memory map"), the memory resource will be flagged IORESOURCE_MEM_DRIVER_MANAGED (esp., kexec_file_load() will not place kexec images on this memory), and it is exposed as "System RAM (virtio_mem)" in /proc/iomem, so esp. kexec-tools can properly handle it.

Example /proc/iomem before this change:
[...]
140000000-333ffffff : virtio0

    140000000-147ffffff : System RAM
  334000000-533ffffff : virtio1
    338000000-33fffffff : System RAM
    340000000-347ffffff : System RAM
    348000000-34fffffff : System RAM

[...]

Example /proc/iomem after this change:
[...]
140000000-333ffffff : virtio0

    140000000-147ffffff : System RAM (virtio_mem)
  334000000-533ffffff : virtio1
    338000000-33fffffff : System RAM (virtio_mem)
    340000000-347ffffff : System RAM (virtio_mem)
    348000000-34fffffff : System RAM (virtio_mem)

[...]

Cc "Michael S. Tsirkin" <[email protected]>
Cc Pankaj Gupta <[email protected]>
Cc teawater <[email protected]>

---

Based on latest Linus' tree (and not a tag) because
- virtio-mem has just been merged via the vhost tree
- add_memory_driver_managed() has been merged a week ago via the -mm tree

I'd like to have this patch in 5.8, with the initial merge of virtio-mem if possible (so the user space representation of virtio-mem added memory resources won't change anymore).

---
 drivers/virtio/virtio_mem.c | 25 ++++++++++++++++++++++---
 1 file changed, 22 insertions(+), 3 deletions(-)
drivers/virtio/virtio_mem.c +18 -2
index 50c689f250450..d2eab3558a9e1 100644
@@ -101,6 +101,11 @@ struct virtio_mem {
 
        /* The parent resource for all memory added via this device. */
        struct resource *parent_resource;
+       /*
+        * Copy of "System RAM (virtio_mem)" to be used for
+        * add_memory_driver_managed().
+        */
+       const char *resource_name;
 
        /* Summary of all memory block states. */
        unsigned long nb_mb_state[VIRTIO_MEM_MB_STATE_COUNT];
@@ -414,8 +419,20 @@ static int virtio_mem_mb_add(struct virtio_mem *vm, unsigned long mb_id)
        if (nid == NUMA_NO_NODE)
                nid = memory_add_physaddr_to_nid(addr);
 
+       /*
+        * When force-unloading the driver and we still have memory added to
+        * Linux, the resource name has to stay.
+        */
+       if (!vm->resource_name) {
+               vm->resource_name = kstrdup_const("System RAM (virtio_mem)",
+                                                 GFP_KERNEL);
+               if (!vm->resource_name)
+                       return -ENOMEM;
+       }
+
        dev_dbg(&vm->vdev->dev, "adding memory block: %lu\n", mb_id);
-       return add_memory(nid, addr, memory_block_size_bytes());
+       return add_memory_driver_managed(nid, addr, memory_block_size_bytes(),
+                                        vm->resource_name);
 }
 
 /*
@@ -1890,10 +1907,12 @@ static void virtio_mem_remove(struct virtio_device *vdev)
            vm->nb_mb_state[VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL] ||
            vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE] ||
            vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL] ||
-           vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE_MOVABLE])

drivers/virtio/virtio_mem.c static void virtio_mem_remove(struct virtio_device *vdev) · line 1918

Quoted 10 lines
drivers/virtio/virtio_mem.c +4 -1
+           vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE_MOVABLE]) {
                dev_warn(&vdev->dev, "device still has system memory added\n");
-       else
+       } else {
                virtio_mem_delete_resource(vm);
+               kfree_const(vm->resource_name);
+       }
 
        /* remove all tracking data - no locking needed */
        vfree(vm->mb_state);

Looks good to me.

#7 lore raw
Quoted 8 lines · Pankaj Gupta
Quoted 8 lines

Virtio-mem managed memory is always detected and added by the virtio-mem driver, never using something like the firmware-provided memory map. This is the case after an ordinary system reboot, and has to be guaranteed after kexec. Especially, virtio-mem added memory resources can contain inaccessible parts ("unblocked memory blocks"), blindly forwarding them to a kexec kernel is dangerous, as unplugged memory will get accessed (esp. written).

Quoted 5 lines
Quoted 3 lines

Let's use the new way of adding special driver-managed memory introduced in commit 75ac4c58bc0d ("mm/memory_hotplug: introduce add_memory_driver_managed()").

Is this commit id correct?

Good point, it's the one from next-20200605.

7b7b27214bba

Is the correct one.

[...]

Quoted 4 lines

Looks good to me.

Thanks!

Signature · 3 lines
Thanks,

David / dhildenb