Skip to contentSkip to conversation

s390/mm: shared zeropage + KVM fixes

PATCH Superseded David Hildenbrand 2 participants 8 messages Mar 27, 2024 – Apr 11, 2024

Subsystems (6)
Memory management KVM s390 s390 MM +2 more subsystems

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover s390/mm: shared zeropage + KVM fixes

This series fixes one issue with uffd + shared zeropages on s390x and fixes that "ordinary" KVM guests can make use of shared zeropages again.

userfaultfd could currently end up mapping shared zeropages into processes that forbid shared zeropages. This only apples to s390x, relevant for handling PV guests and guests that use storage kets correctly. Fix it by placing a zeroed folio instead of the shared zeropage during UFFDIO_ZEROPAGE instead.

I stumbled over this issue while looking into a customer scenario that is using:

(1) Memory ballooning for dynamic resizing. Start a VM with, say, 100 GiB and inflate the balloon during boot to 60 GiB. The VM has ~40 GiB available and additional memory can be "fake hotplugged" to the VM later on demand by deflating the balloon. Actual memory overcommit is not desired, so physical memory would only be moved between VMs.

(2) Live migration of VMs between sites to evacuate servers in case of emergency.

Without the shared zeropage, during (2), the VM would suddenly consume 100 GiB on the migration source and destination. On the migration source, where we don't excpect memory overcommit, we could easilt end up crashing the VM during migration.

Independent of that, memory handed back to the hypervisor using "free page reporting" would end up consuming actual memory after the migration on the destination, not getting freed up until reused+freed again.

While there might be ways to optimize parts of this in QEMU, we really should just support the shared zeropage again for ordinary VMs.

We only expect legcy guests to make use of storage keys, so let's handle zeropages again when enabling storage keys or when enabling PV. To not break userfaultfd like we did in the past, don't zap the shared zeropages, but instead trigger unsharing faults, just like we do for unsharing KSM pages in break_ksm().

Unsharing faults will simply replace the shared zeropage by a zeroed anonymous folio. We can already trigger the same fault path using GUP, when trying to long-term pin a shared zeropage, but also when unmerging a KSM-placed zeropages, so this is nothing new.

Patch #1 tested on 86-64 by forcing mm_forbids_zeropage() to be 1, and running the uffd selftests.

Patch #2 tested on s390x: the live migration scenario now works as expected, and kvm-unit-tests that trigger usage of skeys work well, whereby I can see detection and unsharing of shared zeropages.

Based on Linus master. Andrew agreed that both patches can go via the s390x tree.

v1 -> v2:
* "mm/userfaultfd: don't place zeropages when zeropages are disallowed" -> Minor "ret" ahndling tweaks
* "s390/mm: re-enable the shared zeropage for !PV and !skeys KVM guests" -> Added Fixes: tag

Cc Christian Borntraeger <[email protected]>
Cc Janosch Frank <[email protected]>
Cc Claudio Imbrenda <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc Vasily Gorbik <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Peter Xu <[email protected]>
Cc Alexander Gordeev <[email protected]>
Cc Sven Schnelle <[email protected]>
Cc Gerald Schaefer <[email protected]>
Cc Andrea Arcangeli <[email protected]>

David Hildenbrand (2):
mm/userfaultfd: don't place zeropages when zeropages are disallowed s390/mm: re-enable the shared zeropage for !PV and !skeys KVM guests

 arch/s390/include/asm/gmap.h        |   2 +-
 arch/s390/include/asm/mmu.h         |   5 +
 arch/s390/include/asm/mmu_context.h |   1 +
 arch/s390/include/asm/pgtable.h     |  15 ++-
 arch/s390/kvm/kvm-s390.c            |   4 +-
 arch/s390/mm/gmap.c                 | 163 +++++++++++++++++++++-------
 mm/userfaultfd.c                    |  34 ++++++
 7 files changed, 177 insertions(+), 47 deletions(-)
Signature · 1 line
2.43.2
#1 lore raw

PATCH v2 1/2 mm/userfaultfd: don't place zeropages when zeropages are disallowed

s390x must disable shared zeropages for processes running VMs, because the VMs could end up making use of "storage keys" or protected virtualization, which are incompatible with shared zeropages.

Yet, with userfaultfd it is possible to insert shared zeropages into such processes. Let's fallback to simply allocating a fresh zeroed anonymous folio and insert that instead.

mm_forbids_zeropage() was introduced in commit 593befa6ab74 ("mm: introduce mm_forbids_zeropage function"), briefly before userfaultfd went upstream.

Note that we don't want to fail the UFFDIO_ZEROPAGE request like we do for hugetlb, it would be rather unexpected. Further, we also cannot really indicated "not supported" to user space ahead of time: it could be that the MM disallows zeropages after userfaultfd was already registered.

Fixes c1a4de99fada ("userfaultfd: mcopy_atomic|mfill_zeropage: UFFDIO_COPY|UFFDIO_ZEROPAGE preparation")
---
 mm/userfaultfd.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
mm/userfaultfd.c +34 -0
index 712160cd41ec..9d385696fb89 100644
@@ -316,6 +316,37 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,
 	goto out;
 }
 
+static int mfill_atomic_pte_zeroed_folio(pmd_t *dst_pmd,
+		 struct vm_area_struct *dst_vma, unsigned long dst_addr)
+{
+	struct folio *folio;
+	int ret = -ENOMEM;
+
+	folio = vma_alloc_zeroed_movable_folio(dst_vma, dst_addr);
+	if (!folio)
+		return ret;
+
+	if (mem_cgroup_charge(folio, dst_vma->vm_mm, GFP_KERNEL))
+		goto out_put;
+
+	/*
+	 * The memory barrier inside __folio_mark_uptodate makes sure that
+	 * zeroing out the folio become visible before mapping the page
+	 * using set_pte_at(). See do_anonymous_page().
+	 */
+	__folio_mark_uptodate(folio);
+
+	ret = mfill_atomic_install_pte(dst_pmd, dst_vma, dst_addr,
+				       &folio->page, true, 0);
+	if (ret)
+		goto out_put;
+
+	return 0;
+out_put:
+	folio_put(folio);
+	return ret;
+}
+
 static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd,
 				     struct vm_area_struct *dst_vma,
 				     unsigned long dst_addr)
@@ -324,6 +355,9 @@ static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd,
 	spinlock_t *ptl;
 	int ret;
 
+	if (mm_forbids_zeropage(dst_vma->mm))
+		return mfill_atomic_pte_zeroed_folio(dst_pmd, dst_vma, dst_addr);
+
 	_dst_pte = pte_mkspecial(pfn_pte(my_zero_pfn(dst_addr),
 					 dst_vma->vm_page_prot));
 	ret = -EAGAIN;
Signature · 1 line
2.43.2
#2 lore raw

Re: [PATCH v2 1/2] mm/userfaultfd: don't place zeropages when zeropages are disallowed

On Wed, Mar 27, 2024 at 06:17:36PM +0100, David Hildenbrand wrote:

Hi David,
...

mm/userfaultfd.c static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd, · line 358

Quoted 8 lines

static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd,

mm/userfaultfd.c +1 -0
 				     struct vm_area_struct *dst_vma,
 				     unsigned long dst_addr)
@@ -324,6 +355,9 @@ static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd,
 	spinlock_t *ptl;
 	int ret;
 
+	if (mm_forbids_zeropage(dst_vma->mm))

I assume, you were going to pass dst_vma->vm_mm here? This patch does not compile otherwise.
...

Thanks!

#3 lore raw

Re: [PATCH v2 1/2] mm/userfaultfd: don't place zeropages when zeropages are disallowed

Quoted 11 lines · Alexander Gordeev

On Wed, Mar 27, 2024 at 06:17:36PM +0100, David Hildenbrand wrote:

Hi David,
...

Quoted 7 lines

static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd,
struct vm_area_struct *dst_vma, unsigned long dst_addr)

mm/userfaultfd.c +0 -0
@@ -324,6 +355,9 @@ static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd,
  	spinlock_t *ptl;
  	int ret;
 
Quoted 4 lines

mm/userfaultfd.c static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd, · line 358

Quoted 1 line
mm/userfaultfd.c +1 -0
+	if (mm_forbids_zeropage(dst_vma->mm))

I assume, you were going to pass dst_vma->vm_mm here? This patch does not compile otherwise.

Ah, I compiled it only on x86, where the parameter is ignored ... and for testing the code path I forced mm_forbids_zeropage to be 1 on x86.

Yes, this must be dst_vma->vm_mm.

Thanks!

Signature · 3 lines
Cheers,

David / dhildenb
#4 lore raw

Re: [PATCH v2 1/2] mm/userfaultfd: don't place zeropages when zeropages are disallowed

Quoted 12 lines · David Hildenbrand
Quoted 11 lines · Alexander Gordeev

On Wed, Mar 27, 2024 at 06:17:36PM +0100, David Hildenbrand wrote:

Hi David,
...

Quoted 7 lines

static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd,
struct vm_area_struct *dst_vma, unsigned long dst_addr)

mm/userfaultfd.c +0 -0
@@ -324,6 +355,9 @@ static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd,
   	spinlock_t *ptl;
   	int ret;
 
Quoted 7 lines
Quoted 4 lines

mm/userfaultfd.c static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd, · line 358

Quoted 1 line
mm/userfaultfd.c +1 -0
+	if (mm_forbids_zeropage(dst_vma->mm))

I assume, you were going to pass dst_vma->vm_mm here? This patch does not compile otherwise.

Ah, I compiled it only on x86, where the parameter is ignored ... and for testing the code path I forced mm_forbids_zeropage to be 1 on x86.

Now I get it, I compiled it all on s390x, but not the individual patches, so patch #2 hid the issue in patch #1. Sneaky. :)

Signature · 3 lines
Cheers,

David / dhildenb
#5 lore raw

PATCH v2 2/2 s390/mm: re-enable the shared zeropage for !PV and !skeys KVM guests

In reply to David Hildenbrand

commit fa41ba0d08de ("s390/mm: avoid empty zero pages for KVM guests to avoid postcopy hangs") introduced an undesired side effect when combined with memory ballooning and VM migration: memory part of the inflated memory balloon will consume memory.

Assuming we have a 100GiB VM and inflated the balloon to 40GiB. Our VM will consume ~60GiB of memory. If we now trigger a VM migration, hypervisors like QEMU will read all VM memory. As s390x does not support the shared zeropage, we'll end up allocating for all previously-inflated memory part of the memory balloon: 50 GiB. So we might easily (unexpectedly) crash the VM on the migration source.

Even worse, hypervisors like QEMU optimize for zeropage migration to not consume memory on the migration destination: when migrating a "page full of zeroes", on the migration destination they check whether the target memory is already zero (by reading the destination memory) and avoid writing to the memory to not allocate memory: however, s390x will also allocate memory here, implying that also on the migration destination, we will end up allocating all previously-inflated memory part of the memory balloon.

This is especially bad if actual memory overcommit was not desired, when memory ballooning is used for dynamic VM memory resizing, setting aside some memory during boot that can be added later on demand. Alternatives like virtio-mem that would avoid this issue are not yet available on s390x.

There could be ways to optimize some cases in user space: before reading memory in an anonymous private mapping on the migration source, check via /proc/self/pagemap if anything is already populated. Similarly check on the migration destination before reading. While that would avoid populating tables full of shared zeropages on all architectures, it's harder to get right and performant, and requires user space changes.

Further, with posctopy live migration we must place a page, so there, "avoid touching memory to avoid allocating memory" is not really possible. (Note that a previously we would have falsely inserted shared zeropages into processes using UFFDIO_ZEROPAGE where mm_forbids_zeropage() would have actually forbidden it)

PV is currently incompatible with memory ballooning, and in the common case, KVM guests don't make use of storage keys. Instead of zapping zeropages when enabling storage keys / PV, that turned out to be problematic in the past, let's do exactly the same we do with KSM pages:
trigger unsharing faults to replace the shared zeropages by proper anonymous folios.

What about added latency when enabling storage kes? Having a lot of zeropages in applicable environments (PV, legacy guests, unittests) is unexpected. Further, KSM could today already unshare the zeropages and unmerging KSM pages when enabling storage kets would unshare the KSM-placed zeropages in the same way, resulting in the same latency.

Fixes fa41ba0d08de ("s390/mm: avoid empty zero pages for KVM guests to avoid postcopy hangs")
---
 arch/s390/include/asm/gmap.h        |   2 +-
 arch/s390/include/asm/mmu.h         |   5 +
 arch/s390/include/asm/mmu_context.h |   1 +
 arch/s390/include/asm/pgtable.h     |  15 ++-
 arch/s390/kvm/kvm-s390.c            |   4 +-
 arch/s390/mm/gmap.c                 | 163 +++++++++++++++++++++-------
 6 files changed, 143 insertions(+), 47 deletions(-)
arch/s390/include/asm/gmap.h +1 -1
index 5cc46e0dde62..9725586f4259 100644
@@ -146,7 +146,7 @@ int gmap_mprotect_notify(struct gmap *, unsigned long start,
 
 void gmap_sync_dirty_log_pmd(struct gmap *gmap, unsigned long dirty_bitmap[4],
 			     unsigned long gaddr, unsigned long vmaddr);
-int gmap_mark_unmergeable(void);
+int s390_disable_cow_sharing(void);
 void s390_unlist_old_asce(struct gmap *gmap);
 int s390_replace_asce(struct gmap *gmap);
 void s390_uv_destroy_pfns(unsigned long count, unsigned long *pfns);
arch/s390/include/asm/mmu.h +5 -0
index bb1b4bef1878..4c2dc7abc285 100644
@@ -32,6 +32,11 @@ typedef struct {
 	unsigned int uses_skeys:1;
 	/* The mmu context uses CMM. */
 	unsigned int uses_cmm:1;
+	/*
+	 * The mmu context allows COW-sharing of memory pages (KSM, zeropage).
+	 * Note that COW-sharing during fork() is currently always allowed.
+	 */
+	unsigned int allow_cow_sharing:1;
 	/* The gmaps associated with this context are allowed to use huge pages. */
 	unsigned int allow_gmap_hpage_1m:1;
 } mm_context_t;
arch/s390/include/asm/mmu_context.h +1 -0
index 929af18b0908..a7789a9f6218 100644
@@ -35,6 +35,7 @@ static inline int init_new_context(struct task_struct *tsk,
 	mm->context.has_pgste = 0;
 	mm->context.uses_skeys = 0;
 	mm->context.uses_cmm = 0;
+	mm->context.allow_cow_sharing = 1;
 	mm->context.allow_gmap_hpage_1m = 0;
 #endif
 	switch (mm->context.asce_limit) {
arch/s390/include/asm/pgtable.h +12 -3
index 60950e7a25f5..1a71cb19c089 100644
@@ -566,10 +566,19 @@ static inline pud_t set_pud_bit(pud_t pud, pgprot_t prot)
 }
 
 /*
- * In the case that a guest uses storage keys
- * faults should no longer be backed by zero pages
+ * As soon as the guest uses storage keys or enables PV, we deduplicate all
+ * mapped shared zeropages and prevent new shared zeropages from getting
+ * mapped.
  */
-#define mm_forbids_zeropage mm_has_pgste
+static inline int mm_forbids_zeropage(struct mm_struct *mm)
+{
+#ifdef CONFIG_PGSTE
+	if (!mm->context.allow_cow_sharing)
+		return 1;
+#endif
+	return 0;
+}
+
 static inline int mm_uses_skeys(struct mm_struct *mm)
 {
 #ifdef CONFIG_PGSTE
arch/s390/kvm/kvm-s390.c +1 -3
index 5147b943a864..db3392f0be21 100644
@@ -2631,9 +2631,7 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
 		if (r)
 			break;
 
-		mmap_write_lock(current->mm);
-		r = gmap_mark_unmergeable();
-		mmap_write_unlock(current->mm);
+		r = s390_disable_cow_sharing();
 		if (r)
 			break;
 
arch/s390/mm/gmap.c +123 -40

193 lines not on this page · raw message

Signature · 1 line
2.43.2
#6 lore raw

Re: [PATCH v2 2/2] s390/mm: re-enable the shared zeropage for !PV and !skeys KVM guests

Quoted 4 lines · David Hildenbrand

index 60950e7a25f5..1a71cb19c089 100644

arch/s390/include/asm/pgtable.h static inline pud_t set_pud_bit(pud_t pud, pgprot_t prot) · line 572

Quoted 10 lines
arch/s390/include/asm/pgtable.h +3 -3
@@ -566,10 +566,19 @@ static inline pud_t set_pud_bit(pud_t pud, pgprot_t prot)
 }
 
 /*
- * In the case that a guest uses storage keys
- * faults should no longer be backed by zero pages
+ * As soon as the guest uses storage keys or enables PV, we deduplicate all
+ * mapped shared zeropages and prevent new shared zeropages from getting
+ * mapped.
  */
-#define mm_forbids_zeropage mm_has_pgste

Should it be the below insted?

#define mm_forbids_zeropage mm_forbids_zeropage

Once I add it, it fails to compile, due to the issue in patch #1.

But then I guess this series was tested with the generic mm_forbids_zeropage() which always returns 0:

#define mm_forbids_zeropage(X) (0)

arch/s390/include/asm/pgtable.h static inline pud_t set_pud_bit(pud_t pud, pgprot_t prot) · line 581

Quoted 9 lines
arch/s390/include/asm/pgtable.h +9 -0
+static inline int mm_forbids_zeropage(struct mm_struct *mm)
+{
+#ifdef CONFIG_PGSTE
+	if (!mm->context.allow_cow_sharing)
+		return 1;
+#endif
+	return 0;
+}
+

Thanks!

#7 lore raw

Re: [PATCH v2 2/2] s390/mm: re-enable the shared zeropage for !PV and !skeys KVM guests

Quoted 18 lines · Alexander Gordeev
Quoted 4 lines · David Hildenbrand

index 60950e7a25f5..1a71cb19c089 100644

arch/s390/include/asm/pgtable.h static inline pud_t set_pud_bit(pud_t pud, pgprot_t prot) · line 572

Quoted 10 lines
arch/s390/include/asm/pgtable.h +3 -3
@@ -566,10 +566,19 @@ static inline pud_t set_pud_bit(pud_t pud, pgprot_t prot)
  }
 
  /*
- * In the case that a guest uses storage keys
- * faults should no longer be backed by zero pages
+ * As soon as the guest uses storage keys or enables PV, we deduplicate all
+ * mapped shared zeropages and prevent new shared zeropages from getting
+ * mapped.
   */
-#define mm_forbids_zeropage mm_has_pgste

Should it be the below insted?

Quoted 3 lines

#define mm_forbids_zeropage mm_forbids_zeropage

Once I add it, it fails to compile, due to the issue in patch #1.

I guess yes. So we tested the unsharing part (why I was seeing unsharing triggering), but would have allowed zeropages afterwards. Sight.

Let me retest and resend. Thanks!

Signature · 3 lines
Cheers,

David / dhildenb