Skip to contentSkip to conversation

KVM: PPC: Book3S HV: Remove user-triggerable WARN_ON

PATCH In review Paul Mackerras 2 participants 9 messages May 28, 2020 – Feb 10, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 KVM: PPC: Book3S HV: Remove user-triggerable WARN_ON

Although in general we do not expect valid PTEs to be found in kvmppc_create_pte when we are inserting a large page mapping, there is one situation where this can occur. That is when dirty page logging is turned off for a memslot while the VM is running. Because the new memslots are installed before the old memslot is flushed in kvmppc_core_commit_memory_region_hv(), there is a window where a hypervisor page fault can try to install a 2MB (or 1GB) page where there are already small page mappings which were installed while dirty page logging was enabled and which have not yet been flushed.

Since we have a situation where valid PTEs can legitimately be found by kvmppc_unmap_free_pte, and which can be triggered by userspace, just remove the WARN_ON_ONCE, since it is undesirable to have userspace able to trigger a kernel warning.

---
 arch/powerpc/kvm/book3s_64_mmu_radix.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
arch/powerpc/kvm/book3s_64_mmu_radix.c +7 -4
index 97b45ea..bc3f795 100644
@@ -429,9 +429,13 @@ void kvmppc_unmap_pte(struct kvm *kvm, pte_t *pte, unsigned long gpa,
  * Callers are responsible for flushing the PWC.
  *
  * When page tables are being unmapped/freed as part of page fault path
- * (full = false), ptes are not expected. There is code to unmap them
- * and emit a warning if encountered, but there may already be data
- * corruption due to the unexpected mappings.
+ * (full = false), valid ptes are generally not expected; however, there
+ * is one situation where they arise, which is when dirty page logging is
+ * turned off for a memslot while the VM is running.  The new memslot
+ * becomes visible to page faults before the memslot commit function
+ * gets to flush the memslot, which can lead to a 2MB page mapping being
+ * installed for a guest physical address where there are already 64kB
+ * (or 4kB) mappings (of sub-pages of the same 2MB page).
  */
 static void kvmppc_unmap_free_pte(struct kvm *kvm, pte_t *pte, bool full,
 				  unsigned int lpid)
@@ -445,7 +449,6 @@ static void kvmppc_unmap_free_pte(struct kvm *kvm, pte_t *pte, bool full,
 		for (it = 0; it < PTRS_PER_PTE; ++it, ++p) {
 			if (pte_val(*p) = 0)
 				continue;
-			WARN_ON_ONCE(1);
 			kvmppc_unmap_pte(kvm, p,
 					 pte_pfn(*p) << PAGE_SHIFT,
 					 PAGE_SHIFT, NULL, lpid);
Signature · 1 line
2.7.4
#1 lore raw

PATCH 2/2 KVM: PPC: Book3S HV: Close race with page faults around memslot flushes

There is a potential race condition between hypervisor page faults and flushing a memslot. It is possible for a page fault to read the memslot before a memslot is updated and then write a PTE to the partition-scoped page tables after kvmppc_radix_flush_memslot has completed. (Note that this race has never been explicitly observed.)

To close this race, it is sufficient to increment the MMU sequence number while the kvm->mmu_lock is held. That will cause mmu_notifier_retry() to return true, and the page fault will then return to the guest without inserting a PTE.

---
 arch/powerpc/kvm/book3s_64_mmu_radix.c | 5 +++++
 1 file changed, 5 insertions(+)
arch/powerpc/kvm/book3s_64_mmu_radix.c +5 -0
index bc3f795..aa41183 100644
@@ -1130,6 +1130,11 @@ void kvmppc_radix_flush_memslot(struct kvm *kvm,
 					 kvm->arch.lpid);
 		gpa += PAGE_SIZE;
 	}
+	/*
+	 * Increase the mmu notifier sequence number to prevent any page
+	 * fault that read the memslot earlier from writing a PTE.
+	 */
+	kvm->mmu_notifier_seq++;
 	spin_unlock(&kvm->mmu_lock);
 }
 
Signature · 1 line
2.7.4
#2 lore raw

PATCH 1/2 KVM: PPC: Book3S HV: Remove shared-TLB optimisation from vCPU TLB coherency logic

In reply to Paul Mackerras

Processors that implement ISA v3.0 or later don't necessarily have threads in a core sharing all translations, and/or TLBIEL does not necessarily invalidate translations on all other threads (the architecture talks only about the effect on translations for "the thread executing the tlbiel instruction".

While this worked for POWER9, it may not for future implementations, so remove it. A POWER9 specific optimisation would have to have a specific CPU feature to check, if it were to be re-added.

---
 arch/powerpc/kvm/book3s_hv.c         | 32 +++++++++++++++++++---------
 arch/powerpc/kvm/book3s_hv_builtin.c |  9 --------
 arch/powerpc/kvm/book3s_hv_rm_mmu.c  |  6 ------
 3 files changed, 22 insertions(+), 25 deletions(-)
arch/powerpc/kvm/book3s_hv.c +22 -9
index 2d8627dbd9f6..752daf43f780 100644
@@ -2588,22 +2588,34 @@ static void radix_flush_cpu(struct kvm *kvm, int cpu, struct kvm_vcpu *vcpu)
 {
 	struct kvm_nested_guest *nested = vcpu->arch.nested;
 	cpumask_t *cpu_in_guest;
+	cpumask_t *need_tlb_flush;
 	int i;
 
-	cpu = cpu_first_thread_sibling(cpu);
 	if (nested) {
-		cpumask_set_cpu(cpu, &nested->need_tlb_flush);
+		need_tlb_flush = &nested->need_tlb_flush;
 		cpu_in_guest = &nested->cpu_in_guest;
 	} else {
-		cpumask_set_cpu(cpu, &kvm->arch.need_tlb_flush);
+		need_tlb_flush = &kvm->arch.need_tlb_flush;
 		cpu_in_guest = &kvm->arch.cpu_in_guest;
 	}
+
+	cpu = cpu_first_thread_sibling(cpu);
+	for (i = 0; i < threads_per_core; ++i)
+		cpumask_set_cpu(cpu + i, need_tlb_flush);
+
 	/*
-	 * Make sure setting of bit in need_tlb_flush precedes
+	 * Make sure setting of bits in need_tlb_flush precedes
 	 * testing of cpu_in_guest bits.  The matching barrier on
 	 * the other side is the first smp_mb() in kvmppc_run_core().
 	 */
 	smp_mb();
+
+	/*
+	 * Pull vcpus out of guests if necessary, such that they'll notice
+	 * the need_tlb_flush bit when they re-enter the guest. If this was
+	 * ever a performance concern, it would be interesting to compare
+	 * with performance of using TLBIE.
+	 */
 	for (i = 0; i < threads_per_core; ++i)
 		if (cpumask_test_cpu(cpu + i, cpu_in_guest))
 			smp_call_function_single(cpu + i, do_nothing, NULL, 1);
@@ -2632,18 +2644,18 @@ static void kvmppc_prepare_radix_vcpu(struct kvm_vcpu *vcpu, int pcpu)
 	 * can move around between pcpus.  To cope with this, when
 	 * a vcpu moves from one pcpu to another, we need to tell
 	 * any vcpus running on the same core as this vcpu previously
-	 * ran to flush the TLB.  The TLB is shared between threads,
-	 * so we use a single bit in .need_tlb_flush for all 4 threads.
+	 * ran to flush the TLB.
 	 */
 	if (prev_cpu != pcpu) {
-		if (prev_cpu >= 0 &&
-		    cpu_first_thread_sibling(prev_cpu) !-		    cpu_first_thread_sibling(pcpu))
-			radix_flush_cpu(kvm, prev_cpu, vcpu);
 		if (nested)
 			nested->prev_cpu[vcpu->arch.nested_vcpu_id] = pcpu;
 		else
 			vcpu->arch.prev_cpu = pcpu;
+
+		if (prev_cpu < 0)
+			return; /* first run */
+
+		radix_flush_cpu(kvm, prev_cpu, vcpu);
 	}
 }
 
arch/powerpc/kvm/book3s_hv_builtin.c +0 -9
index f3d3183249fe..dad118760a4e 100644
@@ -789,15 +789,6 @@ void kvmppc_check_need_tlb_flush(struct kvm *kvm, int pcpu,
 {
 	cpumask_t *need_tlb_flush;
 
-	/*
-	 * On POWER9, individual threads can come in here, but the
-	 * TLB is shared between the 4 threads in a core, hence
-	 * invalidating on one thread invalidates for all.
-	 * Thus we make all 4 threads use the same bit.
-	 */
-	if (cpu_has_feature(CPU_FTR_ARCH_300))
-		pcpu = cpu_first_thread_sibling(pcpu);
-
 	if (nested)
 		need_tlb_flush = &nested->need_tlb_flush;
 	else
arch/powerpc/kvm/book3s_hv_rm_mmu.c +0 -6
index 88da2764c1bb..f87237927096 100644
@@ -62,12 +62,6 @@ static int global_invalidates(struct kvm *kvm)
 		smp_wmb();
 		cpumask_setall(&kvm->arch.need_tlb_flush);
 		cpu = local_paca->kvm_hstate.kvm_vcore->pcpu;
-		/*
-		 * On POWER9, threads are independent but the TLB is shared,
-		 * so use the bit for the first thread to represent the core.
-		 */
-		if (cpu_has_feature(CPU_FTR_ARCH_300))
-			cpu = cpu_first_thread_sibling(cpu);
 		cpumask_clear_cpu(cpu, &kvm->arch.need_tlb_flush);
 	}
 
Signature · 1 line
2.23.0
#3 lore raw

Re: [PATCH 1/2] KVM: PPC: Book3S HV: Remove shared-TLB optimisation from vCPU TLB coherency logic

In reply to Paul Mackerras
Quoted 5 lines · Nicholas Piggin

Processors that implement ISA v3.0 or later don't necessarily have threads in a core sharing all translations, and/or TLBIEL does not necessarily invalidate translations on all other threads (the architecture talks only about the effect on translations for "the thread executing the tlbiel instruction".

It seems to me that to have an implementation where TLB entries created on one thread (say T0) are visible to and usable by another thread (T1), but a tlbiel on thread T0 does not result in the entry being removed from visibility/usability on T1, is a pretty insane implementation. I'm not sure that the architecture envisaged allowing this kind of implementation, though perhaps the language doesn't absolutely prohibit it.

This kind of implementation is what you are allowing for in this patch, isn't it?

The sane implementations would be ones where either (a) TLB entries are private to each thread and tlbiel only works on the local thread, or (b) TLB entries can be shared and tlbiel works across all threads. I think this is the conclusion we collectively came to when working on that bug we worked on towards the end of last year.

Quoted 3 lines

While this worked for POWER9, it may not for future implementations, so remove it. A POWER9 specific optimisation would have to have a specific CPU feature to check, if it were to be re-added.

Did you do any measurements of how much performance impact this has on POWER9? I don't believe this patch will actually be necessary on POWER10, so it seems like this patch is just to allow for some undefined possible future CPU. It may still be worth putting in for the sake of strict architecture compliance if the performance impact is minimal.

Paul.

#4 lore raw

Re: [PATCH 1/2] KVM: PPC: Book3S HV: Remove shared-TLB optimisation from vCPU TLB coherency logic

In reply to Paul Mackerras

Excerpts from Paul Mackerras's message of February 9, 2021 5:19 pm:

Quoted 15 lines · Nicholas Piggin
Quoted 5 lines

Processors that implement ISA v3.0 or later don't necessarily have threads in a core sharing all translations, and/or TLBIEL does not necessarily invalidate translations on all other threads (the architecture talks only about the effect on translations for "the thread executing the tlbiel instruction".

It seems to me that to have an implementation where TLB entries created on one thread (say T0) are visible to and usable by another thread (T1), but a tlbiel on thread T0 does not result in the entry being removed from visibility/usability on T1, is a pretty insane implementation. I'm not sure that the architecture envisaged allowing this kind of implementation, though perhaps the language doesn't absolutely prohibit it.

Quoted 2 lines

This kind of implementation is what you are allowing for in this patch, isn't it?

Not intentionally, and patch 2 removes the possibility.

The main thing it allows is an implementation where TLB entries created by T1 which are visble only to T1 do not get removed by TLBIEL on T0. I also have some concern with ordering of in-flight operations (ptesync, memory ordering, etc) which are mostly avoided with this.

Quoted 5 lines

The sane implementations would be ones where either (a) TLB entries are private to each thread and tlbiel only works on the local thread, or (b) TLB entries can be shared and tlbiel works across all threads. I think this is the conclusion we collectively came to when working on that bug we worked on towards the end of last year.

I think an implementation could have both types. So it's hard to get away from flushing all threads.

If the vCPU ran on T0 then migrated to another core, then before running a vCPU from the same LPAR in this core again, we could flush _just_ T0 and that should be fine, but if the vCPU was scheduled onto T1 first, we would have to flush to catch shared xlates. But if we flush on T1, then we would still have to flush T0 when that ran the LPID again to catch the private xlates (but probably would not have to flush T2 or T3).

We could catch that and optimise it with a shared mask and a private mask, but it seems like a lot of complexity and memory ordering issues for unclear gain.

Quoted 7 lines
Quoted 4 lines

While this worked for POWER9, it may not for future implementations, so remove it. A POWER9 specific optimisation would have to have a specific CPU feature to check, if it were to be re-added.

Did you do any measurements of how much performance impact this has on POWER9?

I don't think I've really been able to generate enough CPU migrations to cause a blip. Not to say it doesn't have an impact just that I don't know how to create a worst case load (short of spamming vCPus with sched_setaffinity calls).

Quoted 5 lines

I don't believe this patch will actually be necessary on POWER10, so it seems like this patch is just to allow for some undefined possible future CPU. It may still be worth putting in for the sake of strict architecture compliance if the performance impact is minimal.

AFAIK we still have an issue upstream with other known fixes in, that was fixed with this patch.

Thanks,
Nick

#5 lore raw

Re: [PATCH 1/2] KVM: PPC: Book3S HV: Remove shared-TLB optimisation from vCPU TLB coherency logic

In reply to Paul Mackerras
Quoted 16 lines · Nicholas Piggin

Excerpts from Paul Mackerras's message of February 9, 2021 5:19 pm:

Quoted 15 lines · Nicholas Piggin
Quoted 5 lines

Processors that implement ISA v3.0 or later don't necessarily have threads in a core sharing all translations, and/or TLBIEL does not necessarily invalidate translations on all other threads (the architecture talks only about the effect on translations for "the thread executing the tlbiel instruction".

It seems to me that to have an implementation where TLB entries created on one thread (say T0) are visible to and usable by another thread (T1), but a tlbiel on thread T0 does not result in the entry being removed from visibility/usability on T1, is a pretty insane implementation. I'm not sure that the architecture envisaged allowing this kind of implementation, though perhaps the language doesn't absolutely prohibit it.

Quoted 7 lines
Quoted 2 lines

This kind of implementation is what you are allowing for in this patch, isn't it?

Not intentionally, and patch 2 removes the possibility.

The main thing it allows is an implementation where TLB entries created by T1 which are visble only to T1 do not get removed by TLBIEL on T0.

I could understand this patch as trying to accommodate both those implementations where TLB entries are private to each thread, and those implementations where TLB entries are shared, without needing to distinguish between them, at the expense of doing unnecessary invalidations on both kinds of implementation.

Quoted 11 lines

I also have some concern with ordering of in-flight operations (ptesync, memory ordering, etc) which are mostly avoided with this.

Quoted 5 lines

The sane implementations would be ones where either (a) TLB entries are private to each thread and tlbiel only works on the local thread, or (b) TLB entries can be shared and tlbiel works across all threads. I think this is the conclusion we collectively came to when working on that bug we worked on towards the end of last year.

I think an implementation could have both types. So it's hard to get away from flushing all threads.

Having both private and shared TLB entries in the same implementation would seem very odd to me. What would determine whether a given entry is shared or private?

Paul.

#6 lore raw

Re: [PATCH 1/2] KVM: PPC: Book3S HV: Remove shared-TLB optimisation from vCPU TLB coherency logic

In reply to Paul Mackerras

Excerpts from Paul Mackerras's message of February 10, 2021 10:39 am:

Quoted 22 lines · Nicholas Piggin
Quoted 21 lines

Excerpts from Paul Mackerras's message of February 9, 2021 5:19 pm:

Quoted 15 lines · Nicholas Piggin
Quoted 5 lines

Processors that implement ISA v3.0 or later don't necessarily have threads in a core sharing all translations, and/or TLBIEL does not necessarily invalidate translations on all other threads (the architecture talks only about the effect on translations for "the thread executing the tlbiel instruction".

It seems to me that to have an implementation where TLB entries created on one thread (say T0) are visible to and usable by another thread (T1), but a tlbiel on thread T0 does not result in the entry being removed from visibility/usability on T1, is a pretty insane implementation. I'm not sure that the architecture envisaged allowing this kind of implementation, though perhaps the language doesn't absolutely prohibit it.

Quoted 2 lines

This kind of implementation is what you are allowing for in this patch, isn't it?

Not intentionally, and patch 2 removes the possibility.

Quoted 8 lines
Quoted 2 lines

The main thing it allows is an implementation where TLB entries created by T1 which are visble only to T1 do not get removed by TLBIEL on T0.

I could understand this patch as trying to accommodate both those implementations where TLB entries are private to each thread, and those implementations where TLB entries are shared, without needing to distinguish between them, at the expense of doing unnecessary invalidations on both kinds of implementation.

That's exactly what it is. Existing code accommodates shared TLBs, this patch additionally allows for private.

Quoted 9 lines
Quoted 9 lines

I also have some concern with ordering of in-flight operations (ptesync, memory ordering, etc) which are mostly avoided with this.

Quoted 5 lines

The sane implementations would be ones where either (a) TLB entries are private to each thread and tlbiel only works on the local thread, or (b) TLB entries can be shared and tlbiel works across all threads. I think this is the conclusion we collectively came to when working on that bug we worked on towards the end of last year.

Quoted 6 lines
Quoted 2 lines

I think an implementation could have both types. So it's hard to get away from flushing all threads.

Having both private and shared TLB entries in the same implementation would seem very odd to me. What would determine whether a given entry is shared or private?

Example: an ERAT or L1 TLB per-thread, and a shared L2 TLB behind that. The L1 may not be PID/LPID tagged so you don't want to cross-invalidate other threads every time, say.

Thanks,
Nick

#7 lore raw

Re: [PATCH 1/2] KVM: PPC: Book3S HV: Remove shared-TLB optimisation from vCPU TLB coherency logic

In reply to Paul Mackerras
Quoted 47 lines · Nicholas Piggin

Excerpts from Paul Mackerras's message of February 10, 2021 10:39 am:

Quoted 22 lines · Nicholas Piggin
Quoted 21 lines

Excerpts from Paul Mackerras's message of February 9, 2021 5:19 pm:

Quoted 15 lines · Nicholas Piggin
Quoted 5 lines

Processors that implement ISA v3.0 or later don't necessarily have threads in a core sharing all translations, and/or TLBIEL does not necessarily invalidate translations on all other threads (the architecture talks only about the effect on translations for "the thread executing the tlbiel instruction".

It seems to me that to have an implementation where TLB entries created on one thread (say T0) are visible to and usable by another thread (T1), but a tlbiel on thread T0 does not result in the entry being removed from visibility/usability on T1, is a pretty insane implementation. I'm not sure that the architecture envisaged allowing this kind of implementation, though perhaps the language doesn't absolutely prohibit it.

Quoted 2 lines

This kind of implementation is what you are allowing for in this patch, isn't it?

Not intentionally, and patch 2 removes the possibility.

Quoted 8 lines
Quoted 2 lines

The main thing it allows is an implementation where TLB entries created by T1 which are visble only to T1 do not get removed by TLBIEL on T0.

I could understand this patch as trying to accommodate both those implementations where TLB entries are private to each thread, and those implementations where TLB entries are shared, without needing to distinguish between them, at the expense of doing unnecessary invalidations on both kinds of implementation.

That's exactly what it is. Existing code accommodates shared TLBs, this patch additionally allows for private.

Quoted 12 lines
Quoted 11 lines

I also have some concern with ordering of in-flight operations (ptesync, memory ordering, etc) which are mostly avoided with this.

Quoted 5 lines

The sane implementations would be ones where either (a) TLB entries are private to each thread and tlbiel only works on the local thread, or (b) TLB entries can be shared and tlbiel works across all threads. I think this is the conclusion we collectively came to when working on that bug we worked on towards the end of last year.

I think an implementation could have both types. So it's hard to get away from flushing all threads.

Quoted 7 lines
Quoted 3 lines

Having both private and shared TLB entries in the same implementation would seem very odd to me. What would determine whether a given entry is shared or private?

Example: an ERAT or L1 TLB per-thread, and a shared L2 TLB behind that. The L1 may not be PID/LPID tagged so you don't want to cross-invalidate other threads every time, say.

That's the insane implementation I referred to above, so we're back to saying we need to allow for that kind of implementation.

Paul.

#8 lore raw

Re: [PATCH 1/2] KVM: PPC: Book3S HV: Remove shared-TLB optimisation from vCPU TLB coherency logic

In reply to Paul Mackerras

Excerpts from Paul Mackerras's message of February 10, 2021 12:46 pm:

Quoted 52 lines · Nicholas Piggin
Quoted 51 lines

Excerpts from Paul Mackerras's message of February 10, 2021 10:39 am:

Quoted 22 lines · Nicholas Piggin
Quoted 21 lines

Excerpts from Paul Mackerras's message of February 9, 2021 5:19 pm:

Quoted 15 lines · Nicholas Piggin
Quoted 5 lines

Processors that implement ISA v3.0 or later don't necessarily have threads in a core sharing all translations, and/or TLBIEL does not necessarily invalidate translations on all other threads (the architecture talks only about the effect on translations for "the thread executing the tlbiel instruction".

It seems to me that to have an implementation where TLB entries created on one thread (say T0) are visible to and usable by another thread (T1), but a tlbiel on thread T0 does not result in the entry being removed from visibility/usability on T1, is a pretty insane implementation. I'm not sure that the architecture envisaged allowing this kind of implementation, though perhaps the language doesn't absolutely prohibit it.

Quoted 2 lines

This kind of implementation is what you are allowing for in this patch, isn't it?

Not intentionally, and patch 2 removes the possibility.

Quoted 8 lines
Quoted 2 lines

The main thing it allows is an implementation where TLB entries created by T1 which are visble only to T1 do not get removed by TLBIEL on T0.

I could understand this patch as trying to accommodate both those implementations where TLB entries are private to each thread, and those implementations where TLB entries are shared, without needing to distinguish between them, at the expense of doing unnecessary invalidations on both kinds of implementation.

That's exactly what it is. Existing code accommodates shared TLBs, this patch additionally allows for private.

Quoted 15 lines
Quoted 11 lines

I also have some concern with ordering of in-flight operations (ptesync, memory ordering, etc) which are mostly avoided with this.

Quoted 5 lines

The sane implementations would be ones where either (a) TLB entries are private to each thread and tlbiel only works on the local thread, or (b) TLB entries can be shared and tlbiel works across all threads. I think this is the conclusion we collectively came to when working on that bug we worked on towards the end of last year.

I think an implementation could have both types. So it's hard to get away from flushing all threads.

Having both private and shared TLB entries in the same implementation would seem very odd to me. What would determine whether a given entry is shared or private?

Quoted 6 lines
Quoted 3 lines

Example: an ERAT or L1 TLB per-thread, and a shared L2 TLB behind that. The L1 may not be PID/LPID tagged so you don't want to cross-invalidate other threads every time, say.

That's the insane implementation I referred to above, so we're back to saying we need to allow for that kind of implementation.

I misunderstood you then.

The really insane implementation we were talking about a couple of months ago is one where a translation can be brought in by T0, then invalidated with TLBIEL on T0, but it is later visible to T1 if it switches to that context.

A combination of private and shared TLBs I don't see as being insane, so long as (there is the appearance that) 1. translations are not created on threads that are not running that context, and 2. tlbiel invalidates translations visible to that thread.

Thanks,
Nick