mm: online/offline 4MB chunks controlled by device driver

online_pages()/offline_pages() theoretically allows us to work on sub-section sizes. This is especially relevant in the context of virtualization. It e.g. allows us to add/remove memory to Linux in a VM in 4MB chunks.

While the whole section is marked as online/offline, we have to know the state of each page. E.g. to not read memory that is not online during kexec() or to properly mark a section as offline as soon as all contained pages are offline.

Signed-off-by David Hildenbrand <[email protected]>

Quoted 4 lines · David Hildenbrand

online_pages()/offline_pages() theoretically allows us to work on sub-section sizes. This is especially relevant in the context of virtualization. It e.g. allows us to add/remove memory to Linux in a VM in 4MB chunks.

Well, theoretically possible but this would require a lot of auditing because the hotplug and per section assumption is quite a spread one.

Quoted 5 lines

While the whole section is marked as online/offline, we have to know the state of each page. E.g. to not read memory that is not online during kexec() or to properly mark a section as offline as soon as all contained pages are offline.

But you cannot use a page flag for that, I am afraid. Page flags are extremely scarce resource. I haven't looked at the rest of the series but _if_ we have a bit spare which I am not really sure about then you should prove there are no other ways around this.

Quoted 2 lines

Signed-off-by David Hildenbrand <[email protected]>

Quoted 4 lines · David Hildenbrand

online_pages()/offline_pages() theoretically allows us to work on sub-section sizes. This is especially relevant in the context of virtualization. It e.g. allows us to add/remove memory to Linux in a VM in 4MB chunks.

Well, theoretically possible but this would require a lot of auditing because the hotplug and per section assumption is quite a spread one.

Quoted 4 lines

While the whole section is marked as online/offline, we have to know the state of each page. E.g. to not read memory that is not online during kexec() or to properly mark a section as offline as soon as all contained pages are offline.

But you cannot use a page flag for that, I am afraid. Page flags are extremely scarce resource. I haven't looked at the rest of the series but _if_ we have a bit spare which I am not really sure about then you should prove there are no other ways around this.

online_pages()/offline_pages() theoretically allows us to work on sub-section sizes. This is especially relevant in the context of virtualization. It e.g. allows us to add/remove memory to Linux in a VM in 4MB chunks.

While the whole section is marked as online/offline, we have to know the state of each page. E.g. to not read memory that is not online during kexec() or to properly mark a section as offline as soon as all contained pages are offline.

+ * PG_offline indicates that a page is offline and the backing storage
+ * might already have been removed (virtualization). Don't touch!

Quoted 9 lines · David Hildenbrand

online_pages()/offline_pages() theoretically allows us to work on sub-section sizes. This is especially relevant in the context of virtualization. It e.g. allows us to add/remove memory to Linux in a VM in 4MB chunks.

While the whole section is marked as online/offline, we have to know the state of each page. E.g. to not read memory that is not online during kexec() or to properly mark a section as offline as soon as all contained pages are offline.

Can you not use PG_reserved for this purpose?

Quoted 2 lines

+ * PG_offline indicates that a page is offline and the backing storage
+ * might already have been removed (virtualization). Don't touch!

* PG_reserved is set for special pages, which can never be swapped out. Some
* of them might not even exist...

They seem pretty congruent to me.

Quoted 5 lines · David Hildenbrand

online_pages()/offline_pages() theoretically allows us to work on sub-section sizes. This is especially relevant in the context of virtualization. It e.g. allows us to add/remove memory to Linux in a VM in 4MB chunks.

Quoted 4 lines

While the whole section is marked as online/offline, we have to know the state of each page. E.g. to not read memory that is not online during kexec() or to properly mark a section as offline as soon as all contained pages are offline.

Can you not use PG_reserved for this purpose?

Quoted 2 lines

+ * PG_offline indicates that a page is offline and the backing storage
+ * might already have been removed (virtualization). Don't touch!

* PG_reserved is set for special pages, which can never be swapped out. Some
* of them might not even exist...

They seem pretty congruent to me.

Quoted 11 lines · Matthew Wilcox

Quoted 9 lines · David Hildenbrand

online_pages()/offline_pages() theoretically allows us to work on sub-section sizes. This is especially relevant in the context of virtualization. It e.g. allows us to add/remove memory to Linux in a VM in 4MB chunks.

While the whole section is marked as online/offline, we have to know the state of each page. E.g. to not read memory that is not online during kexec() or to properly mark a section as offline as soon as all contained pages are offline.

Quoted 1 line

Can you not use PG_reserved for this purpose?

Sounds like your newly introduced "page types" could be useful here? I don't suppose those offline pages would be using mapcount which is aliased there?

Quoted 12 lines · Vlastimil Babka

Quoted 11 lines · Matthew Wilcox

Quoted 9 lines · David Hildenbrand

online_pages()/offline_pages() theoretically allows us to work on sub-section sizes. This is especially relevant in the context of virtualization. It e.g. allows us to add/remove memory to Linux in a VM in 4MB chunks.

While the whole section is marked as online/offline, we have to know the state of each page. E.g. to not read memory that is not online during kexec() or to properly mark a section as offline as soon as all contained pages are offline.

Quoted 5 lines

Quoted 1 line

Can you not use PG_reserved for this purpose?

Sounds like your newly introduced "page types" could be useful here? I don't suppose those offline pages would be using mapcount which is aliased there?

Oh, that's a good point! Yes, this is a perfect use for page_type. We have something like twenty bits available there.

Now you've got me thinking that we can move PG_hwpoison and PG_reserved to be page_type flags too. That'll take us from 23 to 21 bits (on 32-bit, with PG_UNCACHED)

Quoted 11 lines · Matthew Wilcox

Quoted 3 lines · Vlastimil Babka

Sounds like your newly introduced "page types" could be useful here? I don't suppose those offline pages would be using mapcount which is aliased there?

Oh, that's a good point! Yes, this is a perfect use for page_type. We have something like twenty bits available there.

Now you've got me thinking that we can move PG_hwpoison and PG_reserved to be page_type flags too. That'll take us from 23 to 21 bits (on 32-bit, with PG_UNCACHED)

Some things to clarify here. I modified the current RFC to also allow PG_offline on allocated (ballooned) pages (e.g. virtio-balloon).

kdump based dump tools can then easily identify which pages are not to be dumped (either because the content is invalid or not accessible).

I previously stated that ballooned pages would be marked as PG_reserved, which is not true (at least not for virtio-balloon). However this allows me to detect if all pages in a section are offline by looking at (PG_reserved && PG_offline). So I can actually tell if a page is marked as offline and allocated or really offline.

1. The location (not the number!) of PG_hwpoison is basically ABI and cannot be changed. Moving it around will most probably break dump tools. (see kernel/crash_core.c)

2. Exposing PG_offline via kdump will make it ABI as well. And we don't want any complicated validity checks ("is the bit valid or not?"), because that would imply having to make these bits ABI as well. So having PG_offline just like PG_hwpoison part of page_flags is the right thing to do. (see patch nr 4)

3. For determining if all pages of a section are offline (see patch nr 5), I will have to be able to check 1. PG_offline and 2. PG_reserved on any page. Will this be possible by moving e.g. PG_reserved to page types? (especially if some field is suddenly aliased?)

Quoted 26 lines · David Hildenbrand

Quoted 11 lines · Matthew Wilcox

Quoted 3 lines · Vlastimil Babka

Sounds like your newly introduced "page types" could be useful here? I don't suppose those offline pages would be using mapcount which is aliased there?

Oh, that's a good point! Yes, this is a perfect use for page_type. We have something like twenty bits available there.

Now you've got me thinking that we can move PG_hwpoison and PG_reserved to be page_type flags too. That'll take us from 23 to 21 bits (on 32-bit, with PG_UNCACHED)

Some things to clarify here. I modified the current RFC to also allow PG_offline on allocated (ballooned) pages (e.g. virtio-balloon).

kdump based dump tools can then easily identify which pages are not to be dumped (either because the content is invalid or not accessible).

I previously stated that ballooned pages would be marked as PG_reserved, which is not true (at least not for virtio-balloon). However this allows me to detect if all pages in a section are offline by looking at (PG_reserved && PG_offline). So I can actually tell if a page is marked as offline and allocated or really offline.

Quoted 3 lines

1. The location (not the number!) of PG_hwpoison is basically ABI and cannot be changed. Moving it around will most probably break dump tools. (see kernel/crash_core.c)

It's not ABI. It already changed after 4.9 when PG_waiters was introduced by commit 62906027091f.

Quoted 11 lines

2. Exposing PG_offline via kdump will make it ABI as well. And we don't want any complicated validity checks ("is the bit valid or not?"), because that would imply having to make these bits ABI as well. So having PG_offline just like PG_hwpoison part of page_flags is the right thing to do. (see patch nr 4)

3. For determining if all pages of a section are offline (see patch nr 5), I will have to be able to check 1. PG_offline and 2. PG_reserved on any page. Will this be possible by moving e.g. PG_reserved to page types? (especially if some field is suddenly aliased?)

It's possible to tell whether the field is in use as mapcount or page_types; mapcount should always be non-negative, and page_types reserves a few bits to detect under/overflow of mapcount. The slab/slob users of the field will also be positive uses.

Quoted 27 lines · Matthew Wilcox

Quoted 26 lines · David Hildenbrand

Quoted 11 lines · Matthew Wilcox

Quoted 3 lines · Vlastimil Babka

Sounds like your newly introduced "page types" could be useful here? I don't suppose those offline pages would be using mapcount which is aliased there?

Oh, that's a good point! Yes, this is a perfect use for page_type. We have something like twenty bits available there.

Now you've got me thinking that we can move PG_hwpoison and PG_reserved to be page_type flags too. That'll take us from 23 to 21 bits (on 32-bit, with PG_UNCACHED)

Some things to clarify here. I modified the current RFC to also allow PG_offline on allocated (ballooned) pages (e.g. virtio-balloon).

kdump based dump tools can then easily identify which pages are not to be dumped (either because the content is invalid or not accessible).

I previously stated that ballooned pages would be marked as PG_reserved, which is not true (at least not for virtio-balloon). However this allows me to detect if all pages in a section are offline by looking at (PG_reserved && PG_offline). So I can actually tell if a page is marked as offline and allocated or really offline.

Quoted 6 lines

Quoted 3 lines

1. The location (not the number!) of PG_hwpoison is basically ABI and cannot be changed. Moving it around will most probably break dump tools. (see kernel/crash_core.c)

It's not ABI. It already changed after 4.9 when PG_waiters was introduced by commit 62906027091f.

It is, please have a look at the file I pointed you to.

We export the *value* of PG_hwpoison in the ELF file, therefore the *value* can change, but the *location* (page_flags, mapcount, whatever) must not change. Or am I missing something here? I don't think we can move PG_hwpoison that easily.

Also, I can read "For pages that are never mapped to userspace, page->mapcount may be used for storing extra information about page type" - is that true for PG_hwpoison/PG_reserved? I am skeptical.

And we need something similar for PG_offline, because it will become ABI. (I can see that PAGE_BUDDY_MAPCOUNT_VALUE is also exported in an ELF file, so maybe a new page type might work for marking a page offline
- but I have to look at the details first tomorrow)

Quoted 42 lines · David Hildenbrand

Quoted 27 lines · Matthew Wilcox

Quoted 26 lines · David Hildenbrand

Quoted 11 lines · Matthew Wilcox

Quoted 3 lines · Vlastimil Babka

Sounds like your newly introduced "page types" could be useful here? I don't suppose those offline pages would be using mapcount which is aliased there?

Oh, that's a good point! Yes, this is a perfect use for page_type. We have something like twenty bits available there.

Now you've got me thinking that we can move PG_hwpoison and PG_reserved to be page_type flags too. That'll take us from 23 to 21 bits (on 32-bit, with PG_UNCACHED)

Some things to clarify here. I modified the current RFC to also allow PG_offline on allocated (ballooned) pages (e.g. virtio-balloon).

kdump based dump tools can then easily identify which pages are not to be dumped (either because the content is invalid or not accessible).

I previously stated that ballooned pages would be marked as PG_reserved, which is not true (at least not for virtio-balloon). However this allows me to detect if all pages in a section are offline by looking at (PG_reserved && PG_offline). So I can actually tell if a page is marked as offline and allocated or really offline.

Quoted 6 lines

Quoted 3 lines

1. The location (not the number!) of PG_hwpoison is basically ABI and cannot be changed. Moving it around will most probably break dump tools. (see kernel/crash_core.c)

It's not ABI. It already changed after 4.9 when PG_waiters was introduced by commit 62906027091f.

It is, please have a look at the file I pointed you to.

We export the *value* of PG_hwpoison in the ELF file, therefore the *value* can change, but the *location* (page_flags, mapcount, whatever) must not change. Or am I missing something here? I don't think we can move PG_hwpoison that easily.

Quoted 8 lines

Also, I can read "For pages that are never mapped to userspace, page->mapcount may be used for storing extra information about page type" - is that true for PG_hwpoison/PG_reserved? I am skeptical.

And we need something similar for PG_offline, because it will become ABI. (I can see that PAGE_BUDDY_MAPCOUNT_VALUE is also exported in an ELF file, so maybe a new page type might work for marking a page offline
- but I have to look at the details first tomorrow)

Wait wait wait. Who is relying on this? Kdump? Page flags have always been an internal implementation detail and _nobody_ outside of the kernel should ever rely on the specific value. Well, kdump has been cheating but that is because kdump is inherently tight to a specific kernel implementation but that doesn't make it a stable ABI IMHO. Restricting the kernel internals because of a debugging tool would be quite insane.

online_pages()/offline_pages() theoretically allows us to work on sub-section sizes. This is especially relevant in the context of virtualization. It e.g. allows us to add/remove memory to Linux in a VM in 4MB chunks.

While the whole section is marked as online/offline, we have to know the state of each page. E.g. to not read memory that is not online during kexec() or to properly mark a section as offline as soon as all contained pages are offline.

Signed-off-by David Hildenbrand <[email protected]>

---
 include/linux/page-flags.h     | 10 ++++++++++
 include/trace/events/mmflags.h |  9 ++++++++-
 2 files changed, 18 insertions(+), 1 deletion(-)

include/linux/page-flags.h +10 -0

index e34a27727b9a..8ebc4bad7824 100644
@@ -49,6 +49,9 @@
  * PG_hwpoison indicates that a page got corrupted in hardware and contains
  * data with incorrect ECC bits that triggered a machine check. Accessing is
  * not safe since it may cause another machine check. Don't touch!
+ *
+ * PG_offline indicates that a page is offline and the backing storage
+ * might already have been removed (virtualization). Don't touch!
  */
 
 /*

@@ -100,6 +103,9 @@ enum pageflags {
 #if defined(CONFIG_IDLE_PAGE_TRACKING) && defined(CONFIG_64BIT)
 	PG_young,
 	PG_idle,
+#endif
+#ifdef CONFIG_MEMORY_HOTPLUG
+	PG_offline,		/* Page is offline. Don't touch */
 #endif
 	__NR_PAGEFLAGS,
 

@@ -381,6 +387,10 @@ TESTCLEARFLAG(Young, young, PF_ANY)
 PAGEFLAG(Idle, idle, PF_ANY)
 #endif
 
+#ifdef CONFIG_MEMORY_HOTPLUG
+PAGEFLAG(Offline, offline, PF_ANY)
+#endif
+
 /*
  * On an anonymous page mapped into a user virtual memory area,
  * page->mapping points to its anon_vma, not to a struct address_space;

include/trace/events/mmflags.h +6 -0

index a81cffb76d89..14c31209e34a 100644
@@ -79,6 +79,12 @@
 #define IF_HAVE_PG_IDLE(flag,string)
 #endif
 
+#ifdef CONFIG_MEMORY_HOTPLUG
+#define IF_HAVE_PG_OFFLINE(flag,string) ,{1UL << flag, string}
+#else
+#define IF_HAVE_PG_OFFLINE(flag,string)
+#endif
+
 #define __def_pageflag_names						\
 	{1UL << PG_locked,		"locked"	},		\
 	{1UL << PG_waiters,		"waiters"	},		\

Some devices (esp. paravirtualized) might want to control
- when to online/offline a memory block
- how to online memory (MOVABLE/NORMAL)
- in which granularity to online/offline memory

So let's add a new flag "driver_managed" and disallow to change the state by user space. Device onlining/offlining will still work, however the memory will not be actually onlined/offlined. That has to be handled by the device driver that owns the memory.

Quoted 9 lines · David Hildenbrand

Some devices (esp. paravirtualized) might want to control
- when to online/offline a memory block
- how to online memory (MOVABLE/NORMAL)
- in which granularity to online/offline memory

So let's add a new flag "driver_managed" and disallow to change the state by user space. Device onlining/offlining will still work, however the memory will not be actually onlined/offlined. That has to be handled by the device driver that owns the memory.

Is there any reason to create the memblock sysfs interface to this memory at all? ZONE_DEVICE mem hotplug users currently do not do that and manage the memory themselves. It seems you want to achieve the same thing, no?

Quoted 9 lines · David Hildenbrand

Some devices (esp. paravirtualized) might want to control
- when to online/offline a memory block
- how to online memory (MOVABLE/NORMAL)
- in which granularity to online/offline memory

So let's add a new flag "driver_managed" and disallow to change the state by user space. Device onlining/offlining will still work, however the memory will not be actually onlined/offlined. That has to be handled by the device driver that owns the memory.

Is there any reason to create the memblock sysfs interface to this memory at all? ZONE_DEVICE mem hotplug users currently do not do that and manage the memory themselves. It seems you want to achieve the same thing, no?

Some devices (esp. paravirtualized) might want to control
- when to online/offline a memory block
- how to online memory (MOVABLE/NORMAL)
- in which granularity to online/offline memory

So let's add a new flag "driver_managed" and disallow to change the state by user space. Device onlining/offlining will still work, however the memory will not be actually onlined/offlined. That has to be handled by the device driver that owns the memory.

I am right now working on a paravirtualized memory device ("virtio-mem"). These devices control a memory region and the amount of memory available via it. Memory will not be indicated via ACPI and friends, the device driver is responsible for it.

[If you choose to not CC the same set of people on all patches - which is sometimes a legit thing to do - then please cc them to the cover letter at least.]

Quoted 4 lines · David Hildenbrand

I am right now working on a paravirtualized memory device ("virtio-mem"). These devices control a memory region and the amount of memory available via it. Memory will not be indicated via ACPI and friends, the device driver is responsible for it.

How does this compare to other ballooning solutions? And why your driver cannot simply use the existing sections and maintain subsections on top?

Quoted 10 lines · Michal Hocko

[If you choose to not CC the same set of people on all patches - which is sometimes a legit thing to do - then please cc them to the cover letter at least.]

Quoted 4 lines · David Hildenbrand

I am right now working on a paravirtualized memory device ("virtio-mem"). These devices control a memory region and the amount of memory available via it. Memory will not be indicated via ACPI and friends, the device driver is responsible for it.

Quoted 3 lines

How does this compare to other ballooning solutions? And why your driver cannot simply use the existing sections and maintain subsections on top?

(further down in this mail is a small paragraph about that)

"And why your driver cannot simply use the existing sections and maintain subsections on top?"

Can you elaborate how that is going to work? What I do as of now, is to remember for each memory block (basically a section because I want to make it as small as possible) which chunks ("subsections") are online/offline. This works just fine. Is this what you are referring to?

Quoted 11 lines · David Hildenbrand

Quoted 10 lines · Michal Hocko

[If you choose to not CC the same set of people on all patches - which is sometimes a legit thing to do - then please cc them to the cover letter at least.]

Quoted 4 lines · David Hildenbrand

I am right now working on a paravirtualized memory device ("virtio-mem"). These devices control a memory region and the amount of memory available via it. Memory will not be indicated via ACPI and friends, the device driver is responsible for it.

Quoted 5 lines

Quoted 3 lines

How does this compare to other ballooning solutions? And why your driver cannot simply use the existing sections and maintain subsections on top?

(further down in this mail is a small paragraph about that)

Sorry, I just stopped right there and didn't even finsh to the end. Shame on me! I will do my homework and read it carefully (next week).

[...]

Quoted 3 lines

"And why your driver cannot simply use the existing sections and maintain subsections on top?"

Quoted 4 lines

Can you elaborate how that is going to work? What I do as of now, is to remember for each memory block (basically a section because I want to make it as small as possible) which chunks ("subsections") are online/offline. This works just fine. Is this what you are referring to?

Well, basically yes. I meant to suggest you simply mark pages reserved and pull them out. You can reuse some parts of such a struct page for your metadata because we should simply ignore those.

You still have to allocate memmap for the full section but 128MB sections have a nice effect that they fit into a single PMD for sparse-vmemmap. So you do not really need to touch mem sections, all you need is to keep your metadata on top.

[If you choose to not CC the same set of people on all patches - which is sometimes a legit thing to do - then please cc them to the cover letter at least.]

Quoted 3 lines · Michal Hocko

[If you choose to not CC the same set of people on all patches - which is sometimes a legit thing to do - then please cc them to the cover letter at least.]

BTW, sorry for that. The list of people to cc was just too big to handle manually, so I used get_maintainers.sh with git for the same time ... something I will most likely avoid next time :)

Quoted 8 lines · David Hildenbrand

Quoted 3 lines · Michal Hocko

[If you choose to not CC the same set of people on all patches - which is sometimes a legit thing to do - then please cc them to the cover letter at least.]

BTW, sorry for that. The list of people to cc was just too big to handle manually, so I used get_maintainers.sh with git for the same time ... something I will most likely avoid next time :)

I usually have Cc: in all commits and then use the following script as --cc-cmd. You just have to git format-patch the series and then git send-email --cc-cmd=./cc-cmd.sh *.patch
+ some mailing lists

#!/bin/bash

if [[ $1 == *gitsendemail.msg* || $1 == *cover-letter* ]]; then

        grep '<.*@.*>' -h *.patch | sed 's/^.*: //' | sort | uniq

fi

In contrast to existing balloon solutions:
- The device is responsible for its own memory only.

- Works on a coarser granularity (e.g. 4MB because that's what we can online/offline in Linux). We are not using the buddy allocator when unplugging but really search for chunks of memory we can offline.

- A device can belong to exactly one NUMA node. This way we can online/offline memory in a fine granularity NUMA aware.

- Architectures that don't have proper memory hotplug interfaces (e.g. s390x) get memory hotplug support. I have a prototype for s390x.

- Once all 4MB chunks of a memory block are offline, we can remove the memory block and therefore the struct pages (seems to work in my prototype), which is nice.

Todo:
- We might have to add a parameter to offline_pages(), telling it to not try forever but abort in case it takes too long.

On Fri 13-04-18 15:16:24, David Hildenbrand wrote:
[...]

Quoted 2 lines

In contrast to existing balloon solutions:
- The device is responsible for its own memory only.

Please be more specific. Any ballooning driver is responsible for its own memory. So what exactly does that mean?

Quoted 4 lines

- Works on a coarser granularity (e.g. 4MB because that's what we can online/offline in Linux). We are not using the buddy allocator when unplugging but really search for chunks of memory we can offline.

Again, more details please. Virtio driver already tries to scan suitable pages to balloon AFAIK.

Quoted 3 lines

- A device can belong to exactly one NUMA node. This way we can online/offline memory in a fine granularity NUMA aware.

What does prevent existing balloon solutions to be NUMA aware?

Quoted 3 lines

- Architectures that don't have proper memory hotplug interfaces (e.g. s390x) get memory hotplug support. I have a prototype for s390x.

I am pretty sure that s390 does support memory hotplug. Or what do you mean?

Quoted 4 lines

- Once all 4MB chunks of a memory block are offline, we can remove the memory block and therefore the struct pages (seems to work in my prototype), which is nice.

OK, so our existing ballooning solutions indeed do not free up memmaps which is suboptimal.

Quoted 4 lines

Todo:
- We might have to add a parameter to offline_pages(), telling it to not try forever but abort in case it takes too long.

Offlining fails when it see non-migrateable pages but other than that it should always succeed in the finite time. If not then there is a bug to be fixed.

Quoted 7 lines · Michal Hocko

On Fri 13-04-18 15:16:24, David Hildenbrand wrote:
[...]

Quoted 2 lines

In contrast to existing balloon solutions:
- The device is responsible for its own memory only.

Please be more specific. Any ballooning driver is responsible for its own memory. So what exactly does that mean?

Simple example (virtio-balloon):

You boot Linux with 8GB. You hotplug two DIMM2 with 4GB each. You use virtio-balloon to "unplug"(although it's not) 4GB. Memory will be removed on *any* memory the allocator is willing to give away.

Now imagine you want to reboot and keep the 4GB unplugged by e.g. resizing memory/DIMM. What to resize to how much? A DIMM? Two DIMMS? Delete one DIMM and keep only one? Drop all DIMMs and resize the initital memory? This makes implementation in the hypervisor extremely hard (especially thinking about migration).

So a ballooning driver does not inflate on its device memory but simply on *any* memory. That's why you only have one virtio-balloon device per VM. It is basically "per machine", while paravirtualied memory devices manage their assigned memory. Like a resizable DIMM, so to say.

Quoted 5 lines

Quoted 4 lines

- Works on a coarser granularity (e.g. 4MB because that's what we can online/offline in Linux). We are not using the buddy allocator when unplugging but really search for chunks of memory we can offline.

Quoted 2 lines

Again, more details please. Virtio driver already tries to scan suitable pages to balloon AFAIK.

Virtio balloon simply uses alloc_page().

That's it. Also, if we wanted to alloc bigger chunks at a time, we would be limited to MAX_ORDER - 1. But that's a different story.

One goal of paravirtualied memory is to be able to unplug more memory than with simple DIMM devices (e.g. 2GB) without having it to online it as MOVABLE.

We don't want to squeeze the last little piece of memory out if the system like a balloon driver does. So we *don't* want to go down to a page level. If we can't unplug any bigger chunks anymore, we tried our best and can't do anything about it.

That's why I don't like to call it a balloon. It is not meant to be used for purposes an ordinary balloon is used (besides memory unplug) - like reducing the size of the dcache or cooperative memory management (meaning a high frequency of inflate and deflate).

Quoted 4 lines

Quoted 2 lines

- A device can belong to exactly one NUMA node. This way we can online/offline memory in a fine granularity NUMA aware.

What does prevent existing balloon solutions to be NUMA aware?

They usually (e.g. virtio-balloon) only have one "target pages" goal defined. Not per NUMA node. We could implement such a feature (with quite some obstacles - e.g. what happens if NUMA nodes are merged in the guest but the hypervisor wants to verify that memory is ballooned on the right NUMA node?), but with paravirtualized memory devices it comes "naturally" and "for free". You just request to resize a certain memory device, that's it.

Quoted 5 lines

Quoted 2 lines

- Architectures that don't have proper memory hotplug interfaces (e.g. s390x) get memory hotplug support. I have a prototype for s390x.

I am pretty sure that s390 does support memory hotplug. Or what do you mean?

There is no interface for s390x to tell it "hey, I just gave you another DIMM/memory chunk, please online it" like we have on other architectures. Such an interface does not exist. That's also the reason why remove_memory() is not yet supported. All there is is standby memory that Linux will simply add and try to online when booting up. Old mainframe interfaces are different :)

You can read more about that here:

http://lists.gnu.org/archive/html/qemu-devel/2018-02/msg04951.html

Quoted 6 lines

Quoted 3 lines

- Once all 4MB chunks of a memory block are offline, we can remove the memory block and therefore the struct pages (seems to work in my prototype), which is nice.

OK, so our existing ballooning solutions indeed do not free up memmaps which is suboptimal.

And we would have to hack deep into the current offlining code to make it work (at least that's my understanding).

Quoted 8 lines

Quoted 4 lines

Todo:
- We might have to add a parameter to offline_pages(), telling it to not try forever but abort in case it takes too long.

Offlining fails when it see non-migrateable pages but other than that it should always succeed in the finite time. If not then there is a bug to be fixed.

I just found the -EINTR in the offlining code and thought this might be problematic. (e.g. if somebody pins a page that is still to be migrated
- or is that avoided by isolating?) I haven't managed to trigger this scenario yet. Was just a thought, that's why I mentioned it but didn't implement it.

Quoted 15 lines · David Hildenbrand

Quoted 7 lines · Michal Hocko

On Fri 13-04-18 15:16:24, David Hildenbrand wrote:
[...]

Quoted 2 lines

In contrast to existing balloon solutions:
- The device is responsible for its own memory only.

Please be more specific. Any ballooning driver is responsible for its own memory. So what exactly does that mean?

Simple example (virtio-balloon):

You boot Linux with 8GB. You hotplug two DIMM2 with 4GB each. You use virtio-balloon to "unplug"(although it's not) 4GB. Memory will be removed on *any* memory the allocator is willing to give away.

Quoted 5 lines

Now imagine you want to reboot and keep the 4GB unplugged by e.g. resizing memory/DIMM. What to resize to how much? A DIMM? Two DIMMS? Delete one DIMM and keep only one? Drop all DIMMs and resize the initital memory? This makes implementation in the hypervisor extremely hard (especially thinking about migration).

I do not follow. Why does this even matter in the virtualized env.?

Quoted 13 lines

So a ballooning driver does not inflate on its device memory but simply on *any* memory. That's why you only have one virtio-balloon device per VM. It is basically "per machine", while paravirtualied memory devices manage their assigned memory. Like a resizable DIMM, so to say.

Quoted 7 lines

Quoted 4 lines

- Works on a coarser granularity (e.g. 4MB because that's what we can online/offline in Linux). We are not using the buddy allocator when unplugging but really search for chunks of memory we can offline.

Again, more details please. Virtio driver already tries to scan suitable pages to balloon AFAIK.

Quoted 4 lines

Virtio balloon simply uses alloc_page().

That's it. Also, if we wanted to alloc bigger chunks at a time, we would be limited to MAX_ORDER - 1. But that's a different story.

Not really, we do have means to do a pfn walk and then try to isolate
pages one-by-one. Have a look at http://lkml.kernel.org/r/[email protected]

Quoted 3 lines

One goal of paravirtualied memory is to be able to unplug more memory than with simple DIMM devices (e.g. 2GB) without having it to online it as MOVABLE.

I am not sure I understand but any hotplug based solution without handling that memory on ZONE_MOVABLE is a lost bettle. It is simply usable only to hotadd memory. Shrinking it back will most likely fail on many workloads.

Quoted 6 lines

We don't want to squeeze the last little piece of memory out if the system like a balloon driver does. So we *don't* want to go down to a page level. If we can't unplug any bigger chunks anymore, we tried our best and can't do anything about it.

Quoted 4 lines

That's why I don't like to call it a balloon. It is not meant to be used for purposes an ordinary balloon is used (besides memory unplug) - like reducing the size of the dcache or cooperative memory management (meaning a high frequency of inflate and deflate).

OK, so what is the typical usecase? And how does the usual setup looks like?

128MB (e.g. 256MB for s390x but way more for other architectures)

Quoted 5 lines

Quoted 4 lines

Quoted 2 lines

- A device can belong to exactly one NUMA node. This way we can online/offline memory in a fine granularity NUMA aware.

What does prevent existing balloon solutions to be NUMA aware?

Quoted 7 lines

They usually (e.g. virtio-balloon) only have one "target pages" goal defined. Not per NUMA node. We could implement such a feature (with quite some obstacles - e.g. what happens if NUMA nodes are merged in the guest but the hypervisor wants to verify that memory is ballooned on the right NUMA node?), but with paravirtualized memory devices it comes "naturally" and "for free". You just request to resize a certain memory device, that's it.

I do not follow. I expected that gues NUMA topology matches the host one or it is a subset. So both ends should know which memory to {in,de}flate. Maybe I am just to naive because I am not familiar with any ballooning implementation much.

Quoted 9 lines

Quoted 8 lines

Quoted 4 lines

Todo:
- We might have to add a parameter to offline_pages(), telling it to not try forever but abort in case it takes too long.

Offlining fails when it see non-migrateable pages but other than that it should always succeed in the finite time. If not then there is a bug to be fixed.

Quoted 5 lines

I just found the -EINTR in the offlining code and thought this might be problematic. (e.g. if somebody pins a page that is still to be migrated
- or is that avoided by isolating?) I haven't managed to trigger this scenario yet. Was just a thought, that's why I mentioned it but didn't implement it.

Offlining is a 3 stage thing. Check for unmovable pages and fail with EBUSY, isolating free memory and migrating the rest. If the first 2 succeed we expect the migration will finish in a finite time.

Quoted 20 lines

Quoted 7 lines

Quoted 4 lines

- Once all 4MB chunks of a memory block are offline, we can remove the memory block and therefore the struct pages (seems to work in my prototype), which is nice.

OK, so our existing ballooning solutions indeed do not free up memmaps which is suboptimal.

And we would have to hack deep into the current offlining code to make it work (at least that's my understanding).

Quoted 8 lines

Quoted 4 lines

Todo:
- We might have to add a parameter to offline_pages(), telling it to not try forever but abort in case it takes too long.

Offlining fails when it see non-migrateable pages but other than that it should always succeed in the finite time. If not then there is a bug to be fixed.

Quoted 5 lines

I just found the -EINTR in the offlining code and thought this might be problematic. (e.g. if somebody pins a page that is still to be migrated
- or is that avoided by isolating?) I haven't managed to trigger this scenario yet. Was just a thought, that's why I mentioned it but didn't implement it.

Offlining is a 3 stage thing. Check for unmovable pages and fail with EBUSY, isolating free memory and migrating the rest. If the first 2 succeed we expect the migration will finish in a finite time.

BTW I was able to easily produce the case where do_migrate_range() would loop for ever (well at least for multiple minutes, but I assume this would have went on :) )

On Wed 18-04-18 17:46:25, David Hildenbrand wrote:
[...]

Quoted 3 lines

BTW I was able to easily produce the case where do_migrate_range() would loop for ever (well at least for multiple minutes, but I assume this would have went on :) )

I am definitely interested to hear details.

Quoted 6 lines · Michal Hocko

On Wed 18-04-18 17:46:25, David Hildenbrand wrote:
[...]

Quoted 3 lines

BTW I was able to easily produce the case where do_migrate_range() would loop for ever (well at least for multiple minutes, but I assume this would have went on :) )

Quoted 2 lines

I am definitely interested to hear details.

migrate_pages() seems to be returning > 0 all the time. Seems to come from too many -EAGAIN from unmap_and_move().

This in return (did not go further down that road) can be as simple as trylock_page() failing.

Of course, we could have other permanent errors here (-ENOMEM). __offline_pages() ignores all errors coming from do_migrate_range(). So in theory, this can take forever - at least not what I want for my use case. I want it to fail fast. "if this block cannot be offlined, try another one".

I wonder if it is the right thing to do in __offline_pages() to ignore even permanent errors. Anyhow, I think I'll need some way of telling offline_pages "please don't retry forever".

Quoted 7 lines · David Hildenbrand

Quoted 6 lines · Michal Hocko

On Wed 18-04-18 17:46:25, David Hildenbrand wrote:
[...]

Quoted 3 lines

BTW I was able to easily produce the case where do_migrate_range() would loop for ever (well at least for multiple minutes, but I assume this would have went on :) )

Quoted 8 lines

Quoted 2 lines

I am definitely interested to hear details.

migrate_pages() seems to be returning > 0 all the time. Seems to come from too many -EAGAIN from unmap_and_move().

This in return (did not go further down that road) can be as simple as trylock_page() failing.

Yes but we assume that nobody holds the lock for ever so sooner or later we should be able to get the lock.

Quoted 9 lines

Of course, we could have other permanent errors here (-ENOMEM). __offline_pages() ignores all errors coming from do_migrate_range(). So in theory, this can take forever - at least not what I want for my use case. I want it to fail fast. "if this block cannot be offlined, try another one".

I wonder if it is the right thing to do in __offline_pages() to ignore even permanent errors. Anyhow, I think I'll need some way of telling offline_pages "please don't retry forever".

Well, it would be really great to find a way to distinguish permanent errors from temporary ones. But I am not sure this is very easy. Anyway, we should be only looking at migratable pages at this stage of the offline so the migration should eventually succeed. We have a bug if this is not a case and we should address it. Find the page which fails to migrate and see who keeps us from migrating it. This might be a page pin abuser or something else. That is why I've said I am interested in details.