virtio-mem: paravirtualized memory
Subsystems (7)
v2 cover virtio-mem: paravirtualized memory
This is the Linux driver side of virtio-mem. Compared to the QEMU side, it is in a pretty complete and clean state.
virtio-mem is a paravirtualized mechanism of adding/removing memory to/from a VM. We can do this on a 4MB granularity right now. In Linux, all memory is added to the ZONE_NORMAL, so unplugging cannot be guaranteed - but will be more likely to succeed compared to unplugging 128MB+ chunks. We might implement some optimizations in that area in the future that will make memory unplug more reliable.
For now, this is an easy way to give a VM access to more memory and eventually to remove some memory again. I am testing it on x86 and s390x (under QEMU TCG so far only).
This is the follow up on [1], but the concept, user interface and virtio protocol has been heavily changed. I am only including the important parts in this cover letter (because otherwise nobody will read it). Please feel free to ask in case there are any questions.
This series is based on [4] and shows how it is being used. It contains further information. Also have a look at the description of patch nr 4 in this series.
This work is the result of the initital idea of Andrea Arcangeli to host enforce guest access to memory inflated in virtio-balloon using userfaultfd, which turned out to be problematic to implement. That's how I came up with virtio-mem.
--------------------------------------------------------------------------
1. High level concept
--------------------------------------------------------------------------
Each virtio-mem device owns a memory region in the physical address space. The guest is allowed to plug and online up to 'requested_size' of memory. It will not be allowed to plug more than that size. Unplugged memory will be protected by configurable mechanisms (e.g. random discard, userfaultfd protection, etc.). virtio-mem is designed in a way that a guest may never assume to be able to even read unplugged memory. This is a big difference to classical balloon drivers.
The usable memory region might grow over time, so not all parts of the device memory region might be usable from the start. This is an optimization to allow a smarter implementation in the hypervisor (reduce size of dirty bitmaps, size of memory regions ...).
When the device driver starts up, it will query 'requested_size' and start to add memory to the system. This memory is not indicated e.g. via ACPI, so unmodified systems will not silently try to use unplugged memory that they are not supposed to touch.
Updates on the 'requested_size' indicate hypervisor requests to plug or unplug memory.
As each virtio-mem device can belong to a NUMA node, we can easily plug/unplug memory on a NUMA basis. And of course, we can have several independent virtio-mem devices for a VM.
The idea is *not* to add new virtio-mem devices when hotplugging memory, the idea is to resize (grow/shrink) virtio-mem devices.
--------------------------------------------------------------------------
2. Benefits
--------------------------------------------------------------------------
Guest side:
- Increase memory usable by Linux in 4MB steps (vs. section size like 128MB on x86 or 2GB on e.g. some arm if I'm not mistaking)
- Remove struct pages once all 4MB chunks of a section are offline (in contrast to all balloon drivers where this never happens)
- Don't fragment memory, while still being able to unplug smaller chunks than ordinary DIMM sizes.
- Memory hotplug support for architectures that have no proper interface (e.g. s390x misses the external notification part) or e.g. QEMU/Linux support is complicated to implement.
- Automatic management of onlining/offlining in the device driver - no manual interaction from an admin/tool necessary.
QEMU side:
- Resizing (plug/unplug) has a single interface - in contrast to a mixture of ACPI and virtio-balloon. See the example below.
- Migration works out of the box - no need to specify new DIMMs or new sizes on the migration target. It simply works.
- We can resize in arbitrary steps and sizes (in contrast to e.g. ACPI, where we have to know upfront in which granularity we later on want to remove memory or even how much memory we eventually want to add to our guest)
- One interface to rule them (architectures) all :)
--------------------------------------------------------------------------
3. Reboot handling
--------------------------------------------------------------------------
After a reboot, all memory is unplugged. This allows the hypervisor to see if support for virtio-mem is available in the freshly booted system. This way we could charge only for the actually "plugged" memory size. And it avoids to sense for plugged memory in the guest.
E.g. on every size change of a virtio-mem device, we can notify management layers. So we can track how much memory a VM has plugged.
--------------------------------------------------------------------------
4. Example
--------------------------------------------------------------------------
(not including resizable memory regions on the QEMU side yet, so don't
focus on that part - it will consume a lot of memory right now for e.g. dirty bitmaps and memory slot tracking data)
Start QEMU with two virtio-mem devices that provide little memory inititally.
$ qemu-system-x86_64 -m 4G,maxmem=504G \ -smp sockets=2,cores=2 \ [...] -object memory-backend-ram,id=mem0,size=256G \ -device virtio-mem-pci,id=vm0,memdev=mem0,node=0,size=4160M \ -object memory-backend-ram,id=mem1,size=256G \ -device virtio-mem-pci,id=vm1,memdev=mem1,node=1,size=3G
Query the configuration ('size' tells us the guest driver is active):
(qemu) info memory-devices info memory-devices Memory device [virtio-mem]: "vm0" phys-addr: 0x140000000 node: 0 requested-size: 4362076160 size: 4362076160 max-size: 274877906944 block-size: 4194304 memdev: /objects/mem0 Memory device [virtio-mem]: "vm1" phys-addr: 0x4140000000 node: 1 requested-size: 3221225472 size: 3221225472 max-size: 274877906944 block-size: 4194304 memdev: /objects/mem1
Change the size of a virtio-mem device:
(qemu) memory-device-resize vm0 40960 memory-device-resize vm0 40960 ... (qemu) info memory-devices info memory-devices Memory device [virtio-mem]: "vm0" phys-addr: 0x140000000 node: 0 requested-size: 42949672960 size: 42949672960 max-size: 274877906944 block-size: 4194304 memdev: /objects/mem0 ...
Try to unplug memory (KASAN active in the guest - a lot of memory wasted):
(qemu) memory-device-resize vm0 1024 memory-device-resize vm0 1024 ... (qemu) info memory-devices info memory-devices Memory device [virtio-mem]: "vm0" phys-addr: 0x140000000 node: 0 requested-size: 1073741824 size: 6169821184 max-size: 274877906944 block-size: 4194304 memdev: /objects/mem0 ...
I am sharing for now only the linux driver side. The current code can be found at [2]. The QEMU side is still heavily WIP, the current QEMU prototype can be found at [3].
[1] https://lists.gnu.org/archive/html/qemu-devel/2017-06/msg03870.html
[2] https://github.com/davidhildenbrand/linux/tree/virtio-mem
[3] https://github.com/davidhildenbrand/qemu/tree/virtio-mem
[4] https://www.mail-archive.com/[email protected]/msg1698014.html
David Hildenbrand (4):
ACPI: NUMA: export pxm_to_node
s390: mm: support removal of memory
s390: numa: implement memory_add_physaddr_to_nid()
virtio-mem: paravirtualized memory
arch/s390/mm/init.c | 18 +- arch/s390/numa/numa.c | 12 + drivers/acpi/numa.c | 1 + drivers/virtio/Kconfig | 15 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_mem.c | 1040 +++++++++++++++++++++++++++++++ include/uapi/linux/virtio_ids.h | 1 + include/uapi/linux/virtio_mem.h | 134 ++++ 8 files changed, 1216 insertions(+), 6 deletions(-) create mode 100644 drivers/virtio/virtio_mem.c create mode 100644 include/uapi/linux/virtio_mem.h
Signature · 1 line
2.17.0
RFC PATCH v2 1/4 ACPI: NUMA: export pxm_to_node
Will be needed by paravirtualized memory devices.
--- drivers/acpi/numa.c | 1 + 1 file changed, 1 insertion(+)
drivers/acpi/numa.c +1 -0
@@ -50,6 +50,7 @@ int pxm_to_node(int pxm)return NUMA_NO_NODE ;return pxm_to_node_map[pxm]; } +EXPORT_SYMBOL (pxm_to_node);int node_to_pxm (int node) {
Signature · 1 line
2.17.0
Rafael J. Wysocki
maintainerRe: [PATCH RFCv2 1/4] ACPI: NUMA: export pxm_to_node
Quoted 1 line · David Hildenbrand
Will be needed by paravirtualized memory devices.
That's a little information.
It would be good to see the entire series at least.
Quoted 13 lines
Cc "Rafael J. Wysocki" <[email protected]>Cc Len Brown <[email protected]>Signed-off-by David Hildenbrand <[email protected]>--- drivers/acpi/numa.c | 1 + 1 file changed, 1 insertion(+)drivers/acpi/numa.c +0
-0
drivers/acpi/numa.c int pxm_to_node(int pxm) · line 53
Quoted 4 lines
drivers/acpi/numa.c +1
-0@@ -50,6 +50,7 @@ int pxm_to_node(int pxm)return NUMA_NO_NODE ;return pxm_to_node_map[pxm]; } +EXPORT_SYMBOL (pxm_to_node);
EXPORT_SYMBOL_GPL(), please.
Quoted 4 lines
drivers/acpi/numa.c +0
-1int node_to_pxm (int node) { --
Re: [PATCH RFCv2 1/4] ACPI: NUMA: export pxm_to_node
Quoted 6 lines · Rafael J. Wysocki
Quoted 1 line · David Hildenbrand
Will be needed by paravirtualized memory devices.
That's a little information.
It would be good to see the entire series at least.
It's part of this series (guess you only received the cover letter and this patch). Here a link to the patch using it:
https://lkml.org/lkml/2018/5/23/803
Quoted 9 lines
Quoted 9 lines
Cc "Rafael J. Wysocki" <[email protected]>Cc Len Brown <[email protected]>Signed-off-by David Hildenbrand <[email protected]>--- drivers/acpi/numa.c | 1 + 1 file changed, 1 insertion(+)
Quoted 11 lines
drivers/acpi/numa.c int pxm_to_node(int pxm) · line 53
Quoted 9 lines
drivers/acpi/numa.c +1
-0@@ -50,6 +50,7 @@ int pxm_to_node(int pxm)return NUMA_NO_NODE ;return pxm_to_node_map[pxm]; } +EXPORT_SYMBOL (pxm_to_node);
EXPORT_SYMBOL_GPL(), please.
Yes, will do, thanks!
Quoted 5 lines
Quoted 5 lines
drivers/acpi/numa.c +0
-1int node_to_pxm (int node) { --
Signature · 3 lines
Thanks, David / dhildenb
Rafael J. Wysocki
maintainerRe: [PATCH RFCv2 1/4] ACPI: NUMA: export pxm_to_node
Quoted 4 lines · David Hildenbrand
Quoted 3 lines · Rafael J. Wysocki
Quoted 1 line · David Hildenbrand
Will be needed by paravirtualized memory devices.
Quoted 8 lines
Quoted 3 lines
That's a little information.
It would be good to see the entire series at least.
It's part of this series (guess you only received the cover letter and this patch). Here a link to the patch using it:
OK, thanks!
It looks like you have a reason to use it in there, but please note that CONFIG_ACPI_NUMA depends on CONFIG_NUMA, so you don't need to use the latter directly in the #ifdef. Also wouldn't IS_ENABLED() work there?
Moreover, you don't need the local node variable in virtio_mem_translate_node_id().
Re: [PATCH RFCv2 1/4] ACPI: NUMA: export pxm_to_node
Quoted 12 lines · Rafael J. Wysocki
Quoted 11 lines · David Hildenbrand
Quoted 6 lines · Rafael J. Wysocki
Quoted 1 line · David Hildenbrand
Will be needed by paravirtualized memory devices.
That's a little information.
It would be good to see the entire series at least.
It's part of this series (guess you only received the cover letter and this patch). Here a link to the patch using it:
Quoted 8 lines
Quoted 1 line
OK, thanks!
It looks like you have a reason to use it in there, but please note that
CONFIG_ACPI_NUMAdepends onCONFIG_NUMA, so you don't need to use the latter directly in the #ifdef. Also wouldn'tIS_ENABLED()work there?
Thanks for the tip on CONFIG_ACPI_NUMA. Wouldn't IS_ENABLED() require to have a dummy implementation of pxm_to_node() in case drivers/acpi/numa.c is not compiled?
Quoted 4 lines
Moreover, you don't need the local node variable in
virtio_mem_translate_node_id().
Indeed, thanks!
Signature · 3 lines
Thanks, David / dhildenb
Rafael J. Wysocki
maintainerRe: [PATCH RFCv2 1/4] ACPI: NUMA: export pxm_to_node
Quoted 17 lines · David Hildenbrand
Quoted 16 lines · Rafael J. Wysocki
Quoted 4 lines · David Hildenbrand
Quoted 3 lines · Rafael J. Wysocki
Quoted 1 line · David Hildenbrand
Will be needed by paravirtualized memory devices.
Quoted 8 lines
Quoted 3 lines
That's a little information.
It would be good to see the entire series at least.
It's part of this series (guess you only received the cover letter and this patch). Here a link to the patch using it:
OK, thanks!
Quoted 8 lines
Quoted 4 lines
It looks like you have a reason to use it in there, but please note that
CONFIG_ACPI_NUMAdepends onCONFIG_NUMA, so you don't need to use the latter directly in the #ifdef. Also wouldn'tIS_ENABLED()work there?Thanks for the tip on
CONFIG_ACPI_NUMA. Wouldn'tIS_ENABLED()require to have a dummy implementation ofpxm_to_node()in casedrivers/acpi/numa.cis not compiled?
Yes, it would.
But since you want export it, you can very well add one, can't you? I'd even say that it would be prudent to do so.
Re: [PATCH RFCv2 1/4] ACPI: NUMA: export pxm_to_node
Quoted 23 lines · Rafael J. Wysocki
Quoted 22 lines · David Hildenbrand
Quoted 20 lines · Rafael J. Wysocki
Quoted 4 lines · David Hildenbrand
Quoted 3 lines · Rafael J. Wysocki
Quoted 1 line · David Hildenbrand
Will be needed by paravirtualized memory devices.
Quoted 8 lines
Quoted 3 lines
That's a little information.
It would be good to see the entire series at least.
It's part of this series (guess you only received the cover letter and this patch). Here a link to the patch using it:
OK, thanks!
It looks like you have a reason to use it in there, but please note that
CONFIG_ACPI_NUMAdepends onCONFIG_NUMA, so you don't need to use the latter directly in the #ifdef. Also wouldn'tIS_ENABLED()work there?
Quoted 9 lines
Quoted 3 lines
Thanks for the tip on
CONFIG_ACPI_NUMA. Wouldn'tIS_ENABLED()require to have a dummy implementation ofpxm_to_node()in casedrivers/acpi/numa.cis not compiled?Yes, it would.
But since you want export it, you can very well add one, can't you? I'd even say that it would be prudent to do so.
Sure, can do that :)
Signature · 3 lines
Thanks, David / dhildenb
RFC PATCH v2 2/4 s390: mm: support removal of memory
In reply to David HildenbrandWith virtio-mem, we actually want to remove memory again.
--- arch/s390/mm/init.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
arch/s390/mm/init.c +12 -6
@@ -242,12 +242,18 @@ int arch_add_memory(int nid, u64 start, u64 size, struct vmem_altmap *altmap,#ifdef CONFIG_MEMORY_HOTREMOVE int arch_remove_memory (u64 start,u64 size,struct vmem_altmap *altmap) { -/* -* There is no hardware or firmware interface which could trigger a -* hot memory remove on s390. So there is nothing that needs to be -* implemented. -*/ -return -EBUSY ; +const unsigned long start_pfn = start >>PAGE_SHIFT ; +const unsigned long nr_pages = size >>PAGE_SHIFT ; +struct page *page =pfn_to_page (start_pfn); +struct zone *zone; +int ret; + + zone =page_zone (page); + ret =__remove_pages (zone, start_pfn, nr_pages, altmap); +WARN_ON_ONCE (ret); +vmem_remove_mapping (start, size); + +return ret; }#endif #endif /* CONFIG_MEMORY_HOTPLUG */
Signature · 1 line
2.17.0
RFC PATCH v2 3/4 s390: numa: implement memory_add_physaddr_to_nid()
In reply to David HildenbrandThe common interface to be used with add_memory().
--- arch/s390/numa/numa.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
arch/s390/numa/numa.c +12 -0
@@ -44,6 +44,18 @@ int numa_pfn_to_nid(unsigned long pfn)return mode->__pfn_to_nid ? mode->__pfn_to_nid (pfn) :0 ; } +#ifdef CONFIG_MEMORY_HOTPLUG +int memory_add_physaddr_to_nid (u64 addr) +{ +int nid =numa_pfn_to_nid (PFN_DOWN (addr)); + +if (nid <0 ) +return 0 ; +return nid; +} +EXPORT_SYMBOL (memory_add_physaddr_to_nid); +#endif +void numa_update_cpu_topology (void ) {if (mode->update_cpu_topology)
Signature · 1 line
2.17.0
RFC PATCH v2 4/4 virtio-mem: paravirtualized memory
In reply to David HildenbrandEach virtio-mem device owns exactly one memory region. It is responsible for adding/removing memory from that memory region on request in a certain fixed chunk size.
When the device driver starts up, the requested amount of memory is queried and then added to Linux. On request, further memory can be added or removed.
We add memory using add_memory() on a section basis (e.g. 128MB) but keep it offline. We can then online/offline e.g. 4MB chunks from such a section, which basically means adding/removing 4MB from Linux and therefore the guest.
Memory is always onlined as ONLINE, not ONLINE_MOVABLE. We might decide to automatically manage the assignment to specific zones later on (which will require some work in the zone and memory offline/removal code). Having small chunks (e.g. 4MB) compared to bigger chunks (e.g. 128MB) makes it more likely for memore hotunplug to succeed, however a single page (out of 1024) on a 4MB block can hinder it from being able to be offlined. Compared to a typical balloon, we avoid any memory fragmentation.
Another future implementation could be to also support smaller chunks of memory (e.g. 1MB or 256k), which would be a good compromise between being able to unplug more memory (from !MOVABLE zone) and avoiding memory fragmentation. The existing virtio protocol supports this extension, but for now 4MB seem to be possible without a lot of messing around in Linux mm code.
Once all chunks of a section are offline, we can go ahead and remove the memory again using remove_memory(). This also removes the "struct pages", which is nice.
Each virtio-mem device can belong to a NUMA node, which allows us to easily add/remove small chunks of memory to/from a specific NUMA node by using multiple virtio-mem devices. Something that works even when the guest has no idea about the NUMA topology, but the host does - something that can easily happen in cloud environments.
One way to view virtio-mem is as a "resizable DIMM" or a DIMM with many "sub-DIMMS". Right now, x86 and s390x are supported.
Driver/Device removal: Whenever the device is removed/reset, all device
memory is no longer accessible. This is problematic, as we cannot simply remove all memory using remove_memory() from Linux.
Suspend/Hypernate: Only supported if no device memory is plugged. When
the device is reset, all device memory is no longer accessible. See "Driver/Device removal".
--- drivers/virtio/Kconfig | 15 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_mem.c | 1040 +++++++++++++++++++++++++++++++ include/uapi/linux/virtio_ids.h | 1 + include/uapi/linux/virtio_mem.h | 134 ++++ 5 files changed, 1191 insertions(+) create mode 100644 drivers/virtio/virtio_mem.c create mode 100644 include/uapi/linux/virtio_mem.h
drivers/virtio/Kconfig +15 -0
@@ -52,6 +52,21 @@ config VIRTIO_BALLOON If unsure, say M. +config VIRTIO_MEM + tristate "Virtio mem driver" + depends on VIRTIO && MEMORY_HOTPLUG && MEMORY_HOTREMOVE && (X86_64 || S390) + help + Virtio Mem driver + + This driver provides access to virtio-mem paravirtualized memory + devices. + + This driver is experimental and was only tested under s390x and x86_64. + In theory, all architectures that support add_memory/remove_memory + and a reasonably large address space should be supported. + + If unsure, say M. + config VIRTIO_INPUT tristate "Virtio input driver" depends on VIRTIO
drivers/virtio/Makefile +1 -0
@@ -6,3 +6,4 @@ virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o virtio_pci-$(CONFIG_VIRTIO_PCI_LEGACY) += virtio_pci_legacy.o obj-$(CONFIG_VIRTIO_BALLOON) += virtio_balloon.o obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o +obj-$(CONFIG_VIRTIO_MEM) += virtio_mem.o
drivers/virtio/virtio_mem.c new file +1040 -0
1,040 lines not on this page · raw message
include/uapi/linux/virtio_ids.h +1 -0
@@ -43,5 +43,6 @@#define VIRTIO_ID_INPUT 18/* virtio input */ #define VIRTIO_ID_VSOCK 19/* virtio vsock transport */ #define VIRTIO_ID_CRYPTO 20/* virtio crypto */ +#define VIRTIO_ID_MEM 24/* virtio mem */ #endif /* _LINUX_VIRTIO_IDS_H */
include/uapi/linux/virtio_mem.h new file +134 -0
134 lines not on this page · raw message
Signature · 1 line
2.17.0
Quoted 185 lines · David Hildenbrand
This is the Linux driver side of virtio-mem. Compared to the QEMU side, it is in a pretty complete and clean state.
virtio-mem is a paravirtualized mechanism of adding/removing memory to/from a VM. We can do this on a 4MB granularity right now. In Linux, all memory is added to the
ZONE_NORMAL, so unplugging cannot be guaranteed - but will be more likely to succeed compared to unplugging 128MB+ chunks. We might implement some optimizations in that area in the future that will make memory unplug more reliable.For now, this is an easy way to give a VM access to more memory and eventually to remove some memory again. I am testing it on x86 and s390x (under QEMU TCG so far only).
This is the follow up on [1], but the concept, user interface and virtio protocol has been heavily changed. I am only including the important parts in this cover letter (because otherwise nobody will read it). Please feel free to ask in case there are any questions.
This series is based on [4] and shows how it is being used. It contains further information. Also have a look at the description of patch nr 4 in this series.
This work is the result of the initital idea of Andrea Arcangeli to host enforce guest access to memory inflated in virtio-balloon using userfaultfd, which turned out to be problematic to implement. That's how I came up with virtio-mem.
--------------------------------------------------------------------------
1. High level concept
--------------------------------------------------------------------------Each virtio-mem device owns a memory region in the physical address space. The guest is allowed to plug and online up to '
requested_size' of memory. It will not be allowed to plug more than that size. Unplugged memory will be protected by configurable mechanisms (e.g. random discard, userfaultfd protection, etc.). virtio-mem is designed in a way that a guest may never assume to be able to even read unplugged memory. This is a big difference to classical balloon drivers.The usable memory region might grow over time, so not all parts of the device memory region might be usable from the start. This is an optimization to allow a smarter implementation in the hypervisor (reduce size of dirty bitmaps, size of memory regions ...).
When the device driver starts up, it will query '
requested_size' and start to add memory to the system. This memory is not indicated e.g. via ACPI, so unmodified systems will not silently try to use unplugged memory that they are not supposed to touch.Updates on the '
requested_size' indicate hypervisor requests to plug or unplug memory.As each virtio-mem device can belong to a NUMA node, we can easily plug/unplug memory on a NUMA basis. And of course, we can have several independent virtio-mem devices for a VM.
The idea is *not* to add new virtio-mem devices when hotplugging memory, the idea is to resize (grow/shrink) virtio-mem devices.
--------------------------------------------------------------------------
2. Benefits
--------------------------------------------------------------------------Guest side:
- Increase memory usable by Linux in 4MB steps (vs. section size like 128MB on x86 or 2GB on e.g. some arm if I'm not mistaking)
- Remove struct pages once all 4MB chunks of a section are offline (in contrast to all balloon drivers where this never happens)
- Don't fragment memory, while still being able to unplug smaller chunks than ordinary DIMM sizes.
- Memory hotplug support for architectures that have no proper interface (e.g. s390x misses the external notification part) or e.g. QEMU/Linux support is complicated to implement.
- Automatic management of onlining/offlining in the device driver - no manual interaction from an admin/tool necessary.QEMU side:
- Resizing (plug/unplug) has a single interface - in contrast to a mixture of ACPI and virtio-balloon. See the example below.
- Migration works out of the box - no need to specify new DIMMs or new sizes on the migration target. It simply works.
- We can resize in arbitrary steps and sizes (in contrast to e.g. ACPI, where we have to know upfront in which granularity we later on want to remove memory or even how much memory we eventually want to add to our guest)
- One interface to rule them (architectures) all :)--------------------------------------------------------------------------
3. Reboot handling
--------------------------------------------------------------------------After a reboot, all memory is unplugged. This allows the hypervisor to see if support for virtio-mem is available in the freshly booted system. This way we could charge only for the actually "plugged" memory size. And it avoids to sense for plugged memory in the guest.
E.g. on every size change of a virtio-mem device, we can notify management layers. So we can track how much memory a VM has plugged.
--------------------------------------------------------------------------
4. Example
--------------------------------------------------------------------------(not including resizable memory regions on the QEMU side yet, so don't
focus on that part - it will consume a lot of memory right now for e.g. dirty bitmaps and memory slot tracking data)Start QEMU with two virtio-mem devices that provide little memory inititally.
$ qemu-system-x86_64 -m 4G,maxmem=504G \ -smp sockets=2,cores=2 \ [...] -object memory-backend-ram,id=mem0,size=256G \ -device virtio-mem-pci,id=vm0,memdev=mem0,node=0,size=4160M \ -object memory-backend-ram,id=mem1,size=256G \ -device virtio-mem-pci,id=vm1,memdev=mem1,node=1,size=3GQuery the configuration ('size' tells us the guest driver is active):
(qemu) info memory-devices info memory-devices Memory device [virtio-mem]: "vm0" phys-addr: 0x140000000 node: 0 requested-size: 4362076160 size: 4362076160 max-size: 274877906944 block-size: 4194304 memdev: /objects/mem0 Memory device [virtio-mem]: "vm1" phys-addr: 0x4140000000 node: 1 requested-size: 3221225472 size: 3221225472 max-size: 274877906944 block-size: 4194304 memdev: /objects/mem1Change the size of a virtio-mem device:
(qemu) memory-device-resize vm0 40960 memory-device-resize vm0 40960 ... (qemu) info memory-devices info memory-devices Memory device [virtio-mem]: "vm0" phys-addr: 0x140000000 node: 0 requested-size: 42949672960 size: 42949672960 max-size: 274877906944 block-size: 4194304 memdev: /objects/mem0 ...Try to unplug memory (KASAN active in the guest - a lot of memory wasted):
(qemu) memory-device-resize vm0 1024 memory-device-resize vm0 1024 ... (qemu) info memory-devices info memory-devices Memory device [virtio-mem]: "vm0" phys-addr: 0x140000000 node: 0 requested-size: 1073741824 size: 6169821184 max-size: 274877906944 block-size: 4194304 memdev: /objects/mem0 ...I am sharing for now only the linux driver side. The current code can be found at [2]. The QEMU side is still heavily WIP, the current QEMU prototype can be found at [3].
[1] https://lists.gnu.org/archive/html/qemu-devel/2017-06/msg03870.html
[2] https://github.com/davidhildenbrand/linux/tree/virtio-mem
[3] https://github.com/davidhildenbrand/qemu/tree/virtio-mem
[4] https://www.mail-archive.com/[email protected]/msg1698014.htmlDavid Hildenbrand (4):
ACPI: NUMA: exportpxm_to_node
s390: mm: support removal of memory
s390: numa: implementmemory_add_physaddr_to_nid()
virtio-mem: paravirtualized memory
Quoted 12 lines
arch/s390/mm/init.c | 18 +- arch/s390/numa/numa.c | 12 + drivers/acpi/numa.c | 1 + drivers/virtio/Kconfig | 15 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_mem.c | 1040 +++++++++++++++++++++++++++++++ include/uapi/linux/virtio_ids.h | 1 + include/uapi/linux/virtio_mem.h | 134 ++++ 8 files changed, 1216 insertions(+), 6 deletions(-) create mode 100644 drivers/virtio/virtio_mem.c create mode 100644 include/uapi/linux/virtio_mem.h
cc-ing some further mailing lists
Signature · 3 lines
Thanks, David / dhildenb
Quoted 185 lines · David Hildenbrand
This is the Linux driver side of virtio-mem. Compared to the QEMU side, it is in a pretty complete and clean state.
virtio-mem is a paravirtualized mechanism of adding/removing memory to/from a VM. We can do this on a 4MB granularity right now. In Linux, all memory is added to the
ZONE_NORMAL, so unplugging cannot be guaranteed - but will be more likely to succeed compared to unplugging 128MB+ chunks. We might implement some optimizations in that area in the future that will make memory unplug more reliable.For now, this is an easy way to give a VM access to more memory and eventually to remove some memory again. I am testing it on x86 and s390x (under QEMU TCG so far only).
This is the follow up on [1], but the concept, user interface and virtio protocol has been heavily changed. I am only including the important parts in this cover letter (because otherwise nobody will read it). Please feel free to ask in case there are any questions.
This series is based on [4] and shows how it is being used. It contains further information. Also have a look at the description of patch nr 4 in this series.
This work is the result of the initital idea of Andrea Arcangeli to host enforce guest access to memory inflated in virtio-balloon using userfaultfd, which turned out to be problematic to implement. That's how I came up with virtio-mem.
--------------------------------------------------------------------------
1. High level concept
--------------------------------------------------------------------------Each virtio-mem device owns a memory region in the physical address space. The guest is allowed to plug and online up to '
requested_size' of memory. It will not be allowed to plug more than that size. Unplugged memory will be protected by configurable mechanisms (e.g. random discard, userfaultfd protection, etc.). virtio-mem is designed in a way that a guest may never assume to be able to even read unplugged memory. This is a big difference to classical balloon drivers.The usable memory region might grow over time, so not all parts of the device memory region might be usable from the start. This is an optimization to allow a smarter implementation in the hypervisor (reduce size of dirty bitmaps, size of memory regions ...).
When the device driver starts up, it will query '
requested_size' and start to add memory to the system. This memory is not indicated e.g. via ACPI, so unmodified systems will not silently try to use unplugged memory that they are not supposed to touch.Updates on the '
requested_size' indicate hypervisor requests to plug or unplug memory.As each virtio-mem device can belong to a NUMA node, we can easily plug/unplug memory on a NUMA basis. And of course, we can have several independent virtio-mem devices for a VM.
The idea is *not* to add new virtio-mem devices when hotplugging memory, the idea is to resize (grow/shrink) virtio-mem devices.
--------------------------------------------------------------------------
2. Benefits
--------------------------------------------------------------------------Guest side:
- Increase memory usable by Linux in 4MB steps (vs. section size like 128MB on x86 or 2GB on e.g. some arm if I'm not mistaking)
- Remove struct pages once all 4MB chunks of a section are offline (in contrast to all balloon drivers where this never happens)
- Don't fragment memory, while still being able to unplug smaller chunks than ordinary DIMM sizes.
- Memory hotplug support for architectures that have no proper interface (e.g. s390x misses the external notification part) or e.g. QEMU/Linux support is complicated to implement.
- Automatic management of onlining/offlining in the device driver - no manual interaction from an admin/tool necessary.QEMU side:
- Resizing (plug/unplug) has a single interface - in contrast to a mixture of ACPI and virtio-balloon. See the example below.
- Migration works out of the box - no need to specify new DIMMs or new sizes on the migration target. It simply works.
- We can resize in arbitrary steps and sizes (in contrast to e.g. ACPI, where we have to know upfront in which granularity we later on want to remove memory or even how much memory we eventually want to add to our guest)
- One interface to rule them (architectures) all :)--------------------------------------------------------------------------
3. Reboot handling
--------------------------------------------------------------------------After a reboot, all memory is unplugged. This allows the hypervisor to see if support for virtio-mem is available in the freshly booted system. This way we could charge only for the actually "plugged" memory size. And it avoids to sense for plugged memory in the guest.
E.g. on every size change of a virtio-mem device, we can notify management layers. So we can track how much memory a VM has plugged.
--------------------------------------------------------------------------
4. Example
--------------------------------------------------------------------------(not including resizable memory regions on the QEMU side yet, so don't
focus on that part - it will consume a lot of memory right now for e.g. dirty bitmaps and memory slot tracking data)Start QEMU with two virtio-mem devices that provide little memory inititally.
$ qemu-system-x86_64 -m 4G,maxmem=504G \ -smp sockets=2,cores=2 \ [...] -object memory-backend-ram,id=mem0,size=256G \ -device virtio-mem-pci,id=vm0,memdev=mem0,node=0,size=4160M \ -object memory-backend-ram,id=mem1,size=256G \ -device virtio-mem-pci,id=vm1,memdev=mem1,node=1,size=3GQuery the configuration ('size' tells us the guest driver is active):
(qemu) info memory-devices info memory-devices Memory device [virtio-mem]: "vm0" phys-addr: 0x140000000 node: 0 requested-size: 4362076160 size: 4362076160 max-size: 274877906944 block-size: 4194304 memdev: /objects/mem0 Memory device [virtio-mem]: "vm1" phys-addr: 0x4140000000 node: 1 requested-size: 3221225472 size: 3221225472 max-size: 274877906944 block-size: 4194304 memdev: /objects/mem1Change the size of a virtio-mem device:
(qemu) memory-device-resize vm0 40960 memory-device-resize vm0 40960 ... (qemu) info memory-devices info memory-devices Memory device [virtio-mem]: "vm0" phys-addr: 0x140000000 node: 0 requested-size: 42949672960 size: 42949672960 max-size: 274877906944 block-size: 4194304 memdev: /objects/mem0 ...Try to unplug memory (KASAN active in the guest - a lot of memory wasted):
(qemu) memory-device-resize vm0 1024 memory-device-resize vm0 1024 ... (qemu) info memory-devices info memory-devices Memory device [virtio-mem]: "vm0" phys-addr: 0x140000000 node: 0 requested-size: 1073741824 size: 6169821184 max-size: 274877906944 block-size: 4194304 memdev: /objects/mem0 ...I am sharing for now only the linux driver side. The current code can be found at [2]. The QEMU side is still heavily WIP, the current QEMU prototype can be found at [3].
[1] https://lists.gnu.org/archive/html/qemu-devel/2017-06/msg03870.html
[2] https://github.com/davidhildenbrand/linux/tree/virtio-mem
[3] https://github.com/davidhildenbrand/qemu/tree/virtio-mem
[4] https://www.mail-archive.com/[email protected]/msg1698014.htmlDavid Hildenbrand (4):
ACPI: NUMA: exportpxm_to_node
s390: mm: support removal of memory
s390: numa: implementmemory_add_physaddr_to_nid()
virtio-mem: paravirtualized memory
Quoted 12 lines
arch/s390/mm/init.c | 18 +- arch/s390/numa/numa.c | 12 + drivers/acpi/numa.c | 1 + drivers/virtio/Kconfig | 15 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_mem.c | 1040 +++++++++++++++++++++++++++++++ include/uapi/linux/virtio_ids.h | 1 + include/uapi/linux/virtio_mem.h | 134 ++++ 8 files changed, 1216 insertions(+), 6 deletions(-) create mode 100644 drivers/virtio/virtio_mem.c create mode 100644 include/uapi/linux/virtio_mem.h
cc-ing some further mailing lists
Signature · 3 lines
Thanks, David / dhildenb