Skip to contentSkip to conversation

xen/balloon: add runtime control for scrubbing ballooned out pages

PATCH In review Marek Marczykowski-Górecki 4 participants 11 messages Sep 6, 2018 – Sep 7, 2018 1 message not in the archive Xen hypervisor interface

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xen/balloon: add runtime control for scrubbing ballooned out pages

Scrubbing pages on initial balloon down can take some time, especially in nested virtualization case (nested EPT is slow). When HVM/PVH guest is started with memory= significantly lower than maxmem=, all the extra pages will be scrubbed before returning to Xen. But since most of them weren't used at all at that point, Xen needs to populate them first (from populate-on-demand pool). In nested virt case (Xen inside KVM) this slows down the guest boot by 15-30s with just 1.5GB needed to be returned to Xen.

Add runtime parameter to enable/disable it, to allow initially disabling scrubbing, then enable it back during boot (for example in initramfs). Such usage relies on assumption that a) most pages ballooned out during initial boot weren't used at all, and b) even if they were, very few secrets are in the guest at that time (before any serious userspace kicks in).

Default behaviour is unchanged.

---
Is module_param() a good thing for this? Other xen-balloon parameters are in /sys/devices/system/xen_memory, so maybe it would make sense to put this one there too? But then, cmdline parameter would need to be added separately and comment about core_param() suggests it shouldn't be used if not absolutely necessary (is it?).

---
 drivers/xen/Kconfig           |  5 ++++-
 drivers/xen/mem-reservation.c |  7 +++++++
 include/xen/mem-reservation.h | 11 ++++++++---
 3 files changed, 19 insertions(+), 4 deletions(-)
drivers/xen/Kconfig +4 -1
index b459edfacff3..7b2c771e1813 100644
@@ -87,7 +87,10 @@ config XEN_SCRUB_PAGES
 	  Scrub pages before returning them to the system for reuse by
 	  other domains.  This makes sure that any confidential data
 	  is not accidentally visible to other domains.  Is it more
-	  secure, but slightly less efficient.
+	  secure, but slightly less efficient. It can be disabled with
+	  mem_reservation.xen_scrub_pages=0 and also controlled at runtime with
+	  /sys/module/mem_reservation/parameters/xen_scrub_pages.
+
 	  If in doubt, say yes.
 
 config XEN_DEV_EVTCHN
drivers/xen/mem-reservation.c +7 -0
index 084799c6180e..5f08e19b6139 100644
@@ -14,6 +14,13 @@
 
 #include <xen/interface/memory.h>
 #include <xen/mem-reservation.h>
+#include <linux/moduleparam.h>
+
+#ifdef CONFIG_XEN_SCRUB_PAGES
+bool __read_mostly xen_scrub_pages = true;
+module_param(xen_scrub_pages, bool, 0644);
+MODULE_PARM_DESC(xen_scrub_pages, "Scrub ballooned pages before giving them back to Xen");
+#endif
 
 /*
  * Use one extent per PAGE_SIZE to avoid to break down the page into
include/xen/mem-reservation.h +8 -3
index 80b52b4945e9..70c08f95bc84 100644
@@ -17,12 +17,17 @@
 
 #include <xen/page.h>
 
+#ifdef CONFIG_XEN_SCRUB_PAGES
+extern bool xen_scrub_pages;
+
 static inline void xenmem_reservation_scrub_page(struct page *page)
 {
-#ifdef CONFIG_XEN_SCRUB_PAGES
-	clear_highpage(page);
-#endif
+	if (xen_scrub_pages)
+		clear_highpage(page);
 }
+#else
+static inline void xenmem_reservation_scrub_page(struct page *page) { }
+#endif
 
 #ifdef CONFIG_XEN_HAVE_PVMMU
 void __xenmem_reservation_va_mapping_update(unsigned long count,
Signature · 1 line
2.17.1
#1 lore raw
Quoted 18 lines · Marek Marczykowski-Górecki

Scrubbing pages on initial balloon down can take some time, especially in nested virtualization case (nested EPT is slow). When HVM/PVH guest is started with memory= significantly lower than maxmem=, all the extra pages will be scrubbed before returning to Xen. But since most of them weren't used at all at that point, Xen needs to populate them first (from populate-on-demand pool). In nested virt case (Xen inside KVM) this slows down the guest boot by 15-30s with just 1.5GB needed to be returned to Xen.

Add runtime parameter to enable/disable it, to allow initially disabling scrubbing, then enable it back during boot (for example in initramfs). Such usage relies on assumption that a) most pages ballooned out during initial boot weren't used at all, and b) even if they were, very few secrets are in the guest at that time (before any serious userspace kicks in).

Default behaviour is unchanged.

Quoted 8 lines

---
Is module_param() a good thing for this? Other xen-balloon parameters are in /sys/devices/system/xen_memory, so maybe it would make sense to put this one there too? But then, cmdline parameter would need to be added separately and comment about core_param() suggests it shouldn't be used if not absolutely necessary (is it?).

You can also use cmdline_find_option() in the balloon driver.

I would prefer that all tunables for the balloon driver live in the same place. (Note that in that case Documentation/ABI/stable/sysfs-devices-system-xen_memory will need to be updated).

-boris

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#2 lore raw
In reply to Marek Marczykowski-Górecki
Quoted 18 lines · Marek Marczykowski-Górecki

Scrubbing pages on initial balloon down can take some time, especially in nested virtualization case (nested EPT is slow). When HVM/PVH guest is started with memory= significantly lower than maxmem=, all the extra pages will be scrubbed before returning to Xen. But since most of them weren't used at all at that point, Xen needs to populate them first (from populate-on-demand pool). In nested virt case (Xen inside KVM) this slows down the guest boot by 15-30s with just 1.5GB needed to be returned to Xen.

Add runtime parameter to enable/disable it, to allow initially disabling scrubbing, then enable it back during boot (for example in initramfs). Such usage relies on assumption that a) most pages ballooned out during initial boot weren't used at all, and b) even if they were, very few secrets are in the guest at that time (before any serious userspace kicks in).

Default behaviour is unchanged.

Quoted 8 lines

---
Is module_param() a good thing for this? Other xen-balloon parameters are in /sys/devices/system/xen_memory, so maybe it would make sense to put this one there too? But then, cmdline parameter would need to be added separately and comment about core_param() suggests it shouldn't be used if not absolutely necessary (is it?).

You can also use cmdline_find_option() in the balloon driver.

I would prefer that all tunables for the balloon driver live in the same place. (Note that in that case Documentation/ABI/stable/sysfs-devices-system-xen_memory will need to be updated).

-boris

#3 lore raw
Quoted 29 lines · Boris Ostrovsky
Quoted 18 lines · Marek Marczykowski-Górecki

Scrubbing pages on initial balloon down can take some time, especially in nested virtualization case (nested EPT is slow). When HVM/PVH guest is started with memory= significantly lower than maxmem=, all the extra pages will be scrubbed before returning to Xen. But since most of them weren't used at all at that point, Xen needs to populate them first (from populate-on-demand pool). In nested virt case (Xen inside KVM) this slows down the guest boot by 15-30s with just 1.5GB needed to be returned to Xen.

Add runtime parameter to enable/disable it, to allow initially disabling scrubbing, then enable it back during boot (for example in initramfs). Such usage relies on assumption that a) most pages ballooned out during initial boot weren't used at all, and b) even if they were, very few secrets are in the guest at that time (before any serious userspace kicks in).

Default behaviour is unchanged.

Quoted 8 lines

---
Is module_param() a good thing for this? Other xen-balloon parameters are in /sys/devices/system/xen_memory, so maybe it would make sense to put this one there too? But then, cmdline parameter would need to be added separately and comment about core_param() suggests it shouldn't be used if not absolutely necessary (is it?).

Quoted 6 lines

You can also use cmdline_find_option() in the balloon driver.

I would prefer that all tunables for the balloon driver live in the same place. (Note that in that case Documentation/ABI/stable/sysfs-devices-system-xen_memory will need to be updated).

Ok, will move that. This will also mean that it will affect only balloon driver, not other users of mem-reservation.c (grant allocator). But given the reason for this tunable, I think that's fine.

Signature · 5 lines
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
#4 lore raw
In reply to Boris Ostrovsky
Quoted 29 lines · Boris Ostrovsky
Quoted 18 lines · Marek Marczykowski-Górecki

Scrubbing pages on initial balloon down can take some time, especially in nested virtualization case (nested EPT is slow). When HVM/PVH guest is started with memory= significantly lower than maxmem=, all the extra pages will be scrubbed before returning to Xen. But since most of them weren't used at all at that point, Xen needs to populate them first (from populate-on-demand pool). In nested virt case (Xen inside KVM) this slows down the guest boot by 15-30s with just 1.5GB needed to be returned to Xen.

Add runtime parameter to enable/disable it, to allow initially disabling scrubbing, then enable it back during boot (for example in initramfs). Such usage relies on assumption that a) most pages ballooned out during initial boot weren't used at all, and b) even if they were, very few secrets are in the guest at that time (before any serious userspace kicks in).

Default behaviour is unchanged.

Quoted 8 lines

---
Is module_param() a good thing for this? Other xen-balloon parameters are in /sys/devices/system/xen_memory, so maybe it would make sense to put this one there too? But then, cmdline parameter would need to be added separately and comment about core_param() suggests it shouldn't be used if not absolutely necessary (is it?).

Quoted 6 lines

You can also use cmdline_find_option() in the balloon driver.

I would prefer that all tunables for the balloon driver live in the same place. (Note that in that case Documentation/ABI/stable/sysfs-devices-system-xen_memory will need to be updated).

Ok, will move that. This will also mean that it will affect only balloon driver, not other users of mem-reservation.c (grant allocator). But given the reason for this tunable, I think that's fine.

Signature · 5 lines
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw
In reply to Boris Ostrovsky
Quoted 1 line · Boris Ostrovsky

You can also use cmdline_find_option() in the balloon driver.

This looks to be x86 only (?!), which doesn't sound right in drivers/xen/. I'll use core_param() for now...

Signature · 5 lines
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw
In reply to Boris Ostrovsky
Quoted 1 line · Boris Ostrovsky

You can also use cmdline_find_option() in the balloon driver.

This looks to be x86 only (?!), which doesn't sound right in drivers/xen/. I'll use core_param() for now...

Signature · 5 lines
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
#7 lore raw
In reply to Marek Marczykowski-Górecki
Quoted 11 lines
Quoted 1 line
Quoted 1 line · On 06.09.18 at 17:33, <[email protected]> wrote:
include/xen/mem-reservation.h +3 -0
@@ -17,12 +17,17 @@
 
 #include <xen/page.h>
 
+#ifdef CONFIG_XEN_SCRUB_PAGES
+extern bool xen_scrub_pages;
+
 static inline void xenmem_reservation_scrub_page(struct page *page)

include/xen/mem-reservation.h line 30

Quoted 10 lines
include/xen/mem-reservation.h +5 -3
 {
-#ifdef CONFIG_XEN_SCRUB_PAGES
-	clear_highpage(page);
-#endif
+	if (xen_scrub_pages)
+		clear_highpage(page);
 }
+#else
+static inline void xenmem_reservation_scrub_page(struct page *page) { }
+#endif

Wouldn't CONFIG_XEN_SCRUB_PAGES then better become CONFIG_XEN_SCRUB_PAGES_DEFAULT, with the value used as initializer of xen_scrub_pages?

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#8 lore raw
In reply to Marek Marczykowski-Górecki
Quoted 11 lines
Quoted 1 line
Quoted 1 line · On 06.09.18 at 17:33, <[email protected]> wrote:
include/xen/mem-reservation.h +3 -0
@@ -17,12 +17,17 @@
 
 #include <xen/page.h>
 
+#ifdef CONFIG_XEN_SCRUB_PAGES
+extern bool xen_scrub_pages;
+
 static inline void xenmem_reservation_scrub_page(struct page *page)

include/xen/mem-reservation.h line 30

Quoted 10 lines
include/xen/mem-reservation.h +5 -3
 {
-#ifdef CONFIG_XEN_SCRUB_PAGES
-	clear_highpage(page);
-#endif
+	if (xen_scrub_pages)
+		clear_highpage(page);
 }
+#else
+static inline void xenmem_reservation_scrub_page(struct page *page) { }
+#endif

Wouldn't CONFIG_XEN_SCRUB_PAGES then better become CONFIG_XEN_SCRUB_PAGES_DEFAULT, with the value used as initializer of xen_scrub_pages?

Jan

A message that is not in the archive (sent off-list or not yet received): [email protected]

maintainer
#10 lore raw
Quoted 17 lines · Jan Beulich
Quoted 17 lines
Quoted 1 line
Quoted 1 line · On 06.09.18 at 17:33, <[email protected]> wrote:
include/xen/mem-reservation.h +5 -3
@@ -17,12 +17,17 @@
 
 #include <xen/page.h>
 
+#ifdef CONFIG_XEN_SCRUB_PAGES
+extern bool xen_scrub_pages;
+
 static inline void xenmem_reservation_scrub_page(struct page *page)
 {
-#ifdef CONFIG_XEN_SCRUB_PAGES
-	clear_highpage(page);
-#endif
+	if (xen_scrub_pages)
+		clear_highpage(page);
Quoted 8 lines

include/xen/mem-reservation.h line 30

Quoted 4 lines
include/xen/mem-reservation.h +3 -0
 }
+#else
+static inline void xenmem_reservation_scrub_page(struct page *page) { }
+#endif

Wouldn't CONFIG_XEN_SCRUB_PAGES then better become CONFIG_XEN_SCRUB_PAGES_DEFAULT, with the value used as initializer of xen_scrub_pages?

Yes, I'd like that better.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

maintainer
#11 lore raw
In reply to a message not in the archive
Quoted 17 lines · Jan Beulich
Quoted 17 lines
Quoted 1 line
Quoted 1 line · On 06.09.18 at 17:33, <[email protected]> wrote:
include/xen/mem-reservation.h +5 -3
@@ -17,12 +17,17 @@
 
 #include <xen/page.h>
 
+#ifdef CONFIG_XEN_SCRUB_PAGES
+extern bool xen_scrub_pages;
+
 static inline void xenmem_reservation_scrub_page(struct page *page)
 {
-#ifdef CONFIG_XEN_SCRUB_PAGES
-	clear_highpage(page);
-#endif
+	if (xen_scrub_pages)
+		clear_highpage(page);
Quoted 8 lines

include/xen/mem-reservation.h line 30

Quoted 4 lines
include/xen/mem-reservation.h +3 -0
 }
+#else
+static inline void xenmem_reservation_scrub_page(struct page *page) { }
+#endif

Wouldn't CONFIG_XEN_SCRUB_PAGES then better become CONFIG_XEN_SCRUB_PAGES_DEFAULT, with the value used as initializer of xen_scrub_pages?

Yes, I'd like that better.

Juergen