Skip to contentSkip to conversation

KVM: PPC: Book3S HV: Optimise TLB flushing when a vcpu moves between threads in a core

PATCH In review Nicholas Piggin 3 participants 7 messages Jan 18, 2021 – Feb 9, 2021

Subsystems (2)
lore
Share & tools
lore

Nicholas Piggin

author reviewer
#0 lore raw

PATCH 2/2 KVM: PPC: Book3S HV: Optimise TLB flushing when a vcpu moves between threads in a core

As explained in the comment, there is no need to flush TLBs on all threads in a core when a vcpu moves between threads in the same core.

Thread migrations can be a significant proportion of vcpu migrations, so this can help reduce the TLB flushing and IPI traffic.

---
I believe we can do this and have the TLB coherency correct as per the architecture, but would appreciate someone else verifying my thinking.

Thanks,
Nick

 arch/powerpc/kvm/book3s_hv.c | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)
arch/powerpc/kvm/book3s_hv.c +25 -2
index 752daf43f780..53d0cbfe5933 100644
@@ -2584,7 +2584,7 @@ static void kvmppc_release_hwthread(int cpu)
 	tpaca->kvm_hstate.kvm_split_mode = NULL;
 }
 
-static void radix_flush_cpu(struct kvm *kvm, int cpu, struct kvm_vcpu *vcpu)
+static void radix_flush_cpu(struct kvm *kvm, int cpu, bool core, struct kvm_vcpu *vcpu)
 {
 	struct kvm_nested_guest *nested = vcpu->arch.nested;
 	cpumask_t *cpu_in_guest;
@@ -2599,6 +2599,14 @@ static void radix_flush_cpu(struct kvm *kvm, int cpu, struct kvm_vcpu *vcpu)
 		cpu_in_guest = &kvm->arch.cpu_in_guest;
 	}
 
+	if (!core) {
+		cpumask_set_cpu(cpu, need_tlb_flush);
+		smp_mb();
+		if (cpumask_test_cpu(cpu, cpu_in_guest))
+			smp_call_function_single(cpu, do_nothing, NULL, 1);
+		return;
+	}
+
 	cpu = cpu_first_thread_sibling(cpu);
 	for (i = 0; i < threads_per_core; ++i)
 		cpumask_set_cpu(cpu + i, need_tlb_flush);
@@ -2655,7 +2663,23 @@ static void kvmppc_prepare_radix_vcpu(struct kvm_vcpu *vcpu, int pcpu)
 		if (prev_cpu < 0)
 			return; /* first run */
 
-		radix_flush_cpu(kvm, prev_cpu, vcpu);
+		/*
+		 * If changing cores, all threads on the old core should
+		 * flush, because TLBs can be shared between threads. More
+		 * precisely, the thread we previously ran on should be
+		 * flushed, and the thread to first run a vcpu on the old
+		 * core should flush, but we don't keep enough information
+		 * around to track that, so we flush all.
+		 *
+		 * If changing threads in the same core, only the old thread
+		 * need be flushed.
+		 */
+		if (cpu_first_thread_sibling(prev_cpu) !+		    cpu_first_thread_sibling(pcpu))
+			radix_flush_cpu(kvm, prev_cpu, true, vcpu);
+		else
+			radix_flush_cpu(kvm, prev_cpu, false, vcpu);
+
 	}
 }
 
Signature · 1 line
2.23.0
#1 lore raw

Re: [PATCH 2/2] KVM: PPC: Book3S HV: Optimise TLB flushing when a vcpu moves between threads in a co

Quoted 23 lines · Nicholas Piggin

As explained in the comment, there is no need to flush TLBs on all threads in a core when a vcpu moves between threads in the same core.

Thread migrations can be a significant proportion of vcpu migrations, so this can help reduce the TLB flushing and IPI traffic.

---
I believe we can do this and have the TLB coherency correct as per the architecture, but would appreciate someone else verifying my thinking.

Thanks,
Nick

 arch/powerpc/kvm/book3s_hv.c | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)
arch/powerpc/kvm/book3s_hv.c +0 -0
index 752daf43f780..53d0cbfe5933 100644

arch/powerpc/kvm/book3s_hv.c static void kvmppc_release_hwthread(int cpu) · line 2588

Quoted 6 lines
arch/powerpc/kvm/book3s_hv.c +1 -1
@@ -2584,7 +2584,7 @@ static void kvmppc_release_hwthread(int cpu)
 	tpaca->kvm_hstate.kvm_split_mode = NULL;
 }
 
-static void radix_flush_cpu(struct kvm *kvm, int cpu, struct kvm_vcpu *vcpu)
+static void radix_flush_cpu(struct kvm *kvm, int cpu, bool core, struct kvm_vcpu *vcpu)
 {

Can we rename 'core' to something like 'core_sched' or 'within_core'

arch/powerpc/kvm/book3s_hv.c static void radix_flush_cpu(struct kvm *kvm, int cpu, struct kvm_vcpu *vcpu) · line 2601

Quoted 6 lines
arch/powerpc/kvm/book3s_hv.c +0 -0
 	struct kvm_nested_guest *nested = vcpu->arch.nested;
 	cpumask_t *cpu_in_guest;
@@ -2599,6 +2599,14 @@ static void radix_flush_cpu(struct kvm *kvm, int cpu, struct kvm_vcpu *vcpu)
 		cpu_in_guest = &kvm->arch.cpu_in_guest;
 	}
 

and do

      if (core_sched) {
Quoted 37 lines
arch/powerpc/kvm/book3s_hv.c +24 -1
+	if (!core) {
+		cpumask_set_cpu(cpu, need_tlb_flush);
+		smp_mb();
+		if (cpumask_test_cpu(cpu, cpu_in_guest))
+			smp_call_function_single(cpu, do_nothing, NULL, 1);
+		return;
+	}
+
 	cpu = cpu_first_thread_sibling(cpu);
 	for (i = 0; i < threads_per_core; ++i)
 		cpumask_set_cpu(cpu + i, need_tlb_flush);
@@ -2655,7 +2663,23 @@ static void kvmppc_prepare_radix_vcpu(struct kvm_vcpu *vcpu, int pcpu)
 		if (prev_cpu < 0)
 			return; /* first run */
 
-		radix_flush_cpu(kvm, prev_cpu, vcpu);
+		/*
+		 * If changing cores, all threads on the old core should
+		 * flush, because TLBs can be shared between threads. More
+		 * precisely, the thread we previously ran on should be
+		 * flushed, and the thread to first run a vcpu on the old
+		 * core should flush, but we don't keep enough information
+		 * around to track that, so we flush all.
+		 *
+		 * If changing threads in the same core, only the old thread
+		 * need be flushed.
+		 */
+		if (cpu_first_thread_sibling(prev_cpu) !> +		    cpu_first_thread_sibling(pcpu))
+			radix_flush_cpu(kvm, prev_cpu, true, vcpu);
+		else
+			radix_flush_cpu(kvm, prev_cpu, false, vcpu);
+
 	}
 }
 
Signature · 1 line
2.23.0
author reviewer
#2 lore raw

Re: [PATCH 2/2] KVM: PPC: Book3S HV: Optimise TLB flushing when a vcpu moves between threads in a co

In reply to Nicholas Piggin

Excerpts from Aneesh Kumar K.V's message of January 19, 2021 7:45 pm:

Quoted 42 lines
Quoted 23 lines · Nicholas Piggin

As explained in the comment, there is no need to flush TLBs on all threads in a core when a vcpu moves between threads in the same core.

Thread migrations can be a significant proportion of vcpu migrations, so this can help reduce the TLB flushing and IPI traffic.

---
I believe we can do this and have the TLB coherency correct as per the architecture, but would appreciate someone else verifying my thinking.

Thanks,
Nick

 arch/powerpc/kvm/book3s_hv.c | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)
arch/powerpc/kvm/book3s_hv.c +0 -0
index 752daf43f780..53d0cbfe5933 100644

arch/powerpc/kvm/book3s_hv.c static void kvmppc_release_hwthread(int cpu) · line 2588

Quoted 6 lines
arch/powerpc/kvm/book3s_hv.c +1 -1
@@ -2584,7 +2584,7 @@ static void kvmppc_release_hwthread(int cpu)
 	tpaca->kvm_hstate.kvm_split_mode = NULL;
 }
 
-static void radix_flush_cpu(struct kvm *kvm, int cpu, struct kvm_vcpu *vcpu)
+static void radix_flush_cpu(struct kvm *kvm, int cpu, bool core, struct kvm_vcpu *vcpu)
 {

Can we rename 'core' to something like 'core_sched' or 'within_core'

arch/powerpc/kvm/book3s_hv.c static void radix_flush_cpu(struct kvm *kvm, int cpu, struct kvm_vcpu *vcpu) · line 2601

Quoted 6 lines
arch/powerpc/kvm/book3s_hv.c +0 -0
 	struct kvm_nested_guest *nested = vcpu->arch.nested;
 	cpumask_t *cpu_in_guest;
@@ -2599,6 +2599,14 @@ static void radix_flush_cpu(struct kvm *kvm, int cpu, struct kvm_vcpu *vcpu)
 		cpu_in_guest = &kvm->arch.cpu_in_guest;
 	}
 

and do

Quoted 1 line
      if (core_sched) {

This function is called to flush guest TLBs on this cpu / all threads on this cpu core. I don't think it helps to bring any "why" logic into it because the caller has already dealt with that.

Thanks,
Nick

Quoted 40 lines
Quoted 39 lines
arch/powerpc/kvm/book3s_hv.c +25 -1
+	if (!core) {
+		cpumask_set_cpu(cpu, need_tlb_flush);
+		smp_mb();
+		if (cpumask_test_cpu(cpu, cpu_in_guest))
+			smp_call_function_single(cpu, do_nothing, NULL, 1);
+		return;
+	}
+
 	cpu = cpu_first_thread_sibling(cpu);
 	for (i = 0; i < threads_per_core; ++i)
 		cpumask_set_cpu(cpu + i, need_tlb_flush);
@@ -2655,7 +2663,23 @@ static void kvmppc_prepare_radix_vcpu(struct kvm_vcpu *vcpu, int pcpu)
 		if (prev_cpu < 0)
 			return; /* first run */
 
-		radix_flush_cpu(kvm, prev_cpu, vcpu);
+		/*
+		 * If changing cores, all threads on the old core should
+		 * flush, because TLBs can be shared between threads. More
+		 * precisely, the thread we previously ran on should be
+		 * flushed, and the thread to first run a vcpu on the old
+		 * core should flush, but we don't keep enough information
+		 * around to track that, so we flush all.
+		 *
+		 * If changing threads in the same core, only the old thread
+		 * need be flushed.
+		 */
+		if (cpu_first_thread_sibling(prev_cpu) !=
+		    cpu_first_thread_sibling(pcpu))
+			radix_flush_cpu(kvm, prev_cpu, true, vcpu);
+		else
+			radix_flush_cpu(kvm, prev_cpu, false, vcpu);
+
 	}
 }
 
Signature · 1 line
2.23.0
#3 lore raw

Re: [PATCH 2/2] KVM: PPC: Book3S HV: Optimise TLB flushing when a vcpu moves between threads in a co

In reply to Nicholas Piggin
Quoted 11 lines · Nicholas Piggin

As explained in the comment, there is no need to flush TLBs on all threads in a core when a vcpu moves between threads in the same core.

Thread migrations can be a significant proportion of vcpu migrations, so this can help reduce the TLB flushing and IPI traffic.

---
I believe we can do this and have the TLB coherency correct as per the architecture, but would appreciate someone else verifying my thinking.

So far I have not been able to convince myself that migrating within a core is really different from migrating across cores as far as the architecture is concerned. If you're trying to allow for an implementation where TLB entries are shared but tlbiel only works (effectively and completely) on the local thread, then I don't think you can do this. If a TLB entry is created on T0, then the vcpu moves to T1 and does a tlbiel, then the guest task on that vcpu migrates to the vcpu that is on T2, it might still see a stale TLB entry.

Paul.

#4 lore raw

Re: [PATCH 2/2] KVM: PPC: Book3S HV: Optimise TLB flushing when a vcpu moves between threads in a co

In reply to Nicholas Piggin
Quoted 45 lines · Nicholas Piggin

Excerpts from Aneesh Kumar K.V's message of January 19, 2021 7:45 pm:

Quoted 42 lines
Quoted 23 lines · Nicholas Piggin

As explained in the comment, there is no need to flush TLBs on all threads in a core when a vcpu moves between threads in the same core.

Thread migrations can be a significant proportion of vcpu migrations, so this can help reduce the TLB flushing and IPI traffic.

---
I believe we can do this and have the TLB coherency correct as per the architecture, but would appreciate someone else verifying my thinking.

Thanks,
Nick

 arch/powerpc/kvm/book3s_hv.c | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)
arch/powerpc/kvm/book3s_hv.c +0 -0
index 752daf43f780..53d0cbfe5933 100644

arch/powerpc/kvm/book3s_hv.c static void kvmppc_release_hwthread(int cpu) · line 2588

Quoted 6 lines
arch/powerpc/kvm/book3s_hv.c +1 -1
@@ -2584,7 +2584,7 @@ static void kvmppc_release_hwthread(int cpu)
 	tpaca->kvm_hstate.kvm_split_mode = NULL;
 }
 
-static void radix_flush_cpu(struct kvm *kvm, int cpu, struct kvm_vcpu *vcpu)
+static void radix_flush_cpu(struct kvm *kvm, int cpu, bool core, struct kvm_vcpu *vcpu)
 {

Can we rename 'core' to something like 'core_sched' or 'within_core'

arch/powerpc/kvm/book3s_hv.c static void radix_flush_cpu(struct kvm *kvm, int cpu, struct kvm_vcpu *vcpu) · line 2601

Quoted 6 lines
arch/powerpc/kvm/book3s_hv.c +0 -0
 	struct kvm_nested_guest *nested = vcpu->arch.nested;
 	cpumask_t *cpu_in_guest;
@@ -2599,6 +2599,14 @@ static void radix_flush_cpu(struct kvm *kvm, int cpu, struct kvm_vcpu *vcpu)
 		cpu_in_guest = &kvm->arch.cpu_in_guest;
 	}
 

and do

Quoted 1 line
      if (core_sched) {
Quoted 3 lines

This function is called to flush guest TLBs on this cpu / all threads on this cpu core. I don't think it helps to bring any "why" logic into it because the caller has already dealt with that.

I agree with Aneesh that the name "core" doesn't really help the reader to know what the parameter means. Either it needs a comment or a more descriptive name.

Paul.

author reviewer
#5 lore raw

Re: [PATCH 2/2] KVM: PPC: Book3S HV: Optimise TLB flushing when a vcpu moves between threads in a co

In reply to Nicholas Piggin

Excerpts from Paul Mackerras's message of February 9, 2021 5:26 pm:

Quoted 46 lines · Nicholas Piggin
Quoted 45 lines

Excerpts from Aneesh Kumar K.V's message of January 19, 2021 7:45 pm:

Quoted 43 lines
Quoted 23 lines · Nicholas Piggin

As explained in the comment, there is no need to flush TLBs on all threads in a core when a vcpu moves between threads in the same core.

Thread migrations can be a significant proportion of vcpu migrations, so this can help reduce the TLB flushing and IPI traffic.

---
I believe we can do this and have the TLB coherency correct as per the architecture, but would appreciate someone else verifying my thinking.

Thanks,
Nick

 arch/powerpc/kvm/book3s_hv.c | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)
arch/powerpc/kvm/book3s_hv.c +0 -0
index 752daf43f780..53d0cbfe5933 100644

arch/powerpc/kvm/book3s_hv.c static void kvmppc_release_hwthread(int cpu) · line 2588

Quoted 6 lines
arch/powerpc/kvm/book3s_hv.c +1 -1
@@ -2584,7 +2584,7 @@ static void kvmppc_release_hwthread(int cpu)
 	tpaca->kvm_hstate.kvm_split_mode = NULL;
 }
 
-static void radix_flush_cpu(struct kvm *kvm, int cpu, struct kvm_vcpu *vcpu)
+static void radix_flush_cpu(struct kvm *kvm, int cpu, bool core, struct kvm_vcpu *vcpu)
 {

Can we rename 'core' to something like 'core_sched' or 'within_core'

arch/powerpc/kvm/book3s_hv.c static void radix_flush_cpu(struct kvm *kvm, int cpu, struct kvm_vcpu *vcpu) · line 2601

Quoted 6 lines
arch/powerpc/kvm/book3s_hv.c +0 -0
 	struct kvm_nested_guest *nested = vcpu->arch.nested;
 	cpumask_t *cpu_in_guest;
@@ -2599,6 +2599,14 @@ static void radix_flush_cpu(struct kvm *kvm, int cpu, struct kvm_vcpu *vcpu)
 		cpu_in_guest = &kvm->arch.cpu_in_guest;
 	}
 

and do

      if (core_sched) {
Quoted 7 lines
Quoted 3 lines

This function is called to flush guest TLBs on this cpu / all threads on this cpu core. I don't think it helps to bring any "why" logic into it because the caller has already dealt with that.

I agree with Aneesh that the name "core" doesn't really help the reader to know what the parameter means. Either it needs a comment or a more descriptive name.

Okay. 'all_threads_in_core' is less typing than a comment :)

Thanks,
Nick

author reviewer
#6 lore raw

Re: [PATCH 2/2] KVM: PPC: Book3S HV: Optimise TLB flushing when a vcpu moves between threads in a co

In reply to Nicholas Piggin

Excerpts from Paul Mackerras's message of February 9, 2021 5:23 pm:

Quoted 13 lines · Nicholas Piggin
Quoted 11 lines

As explained in the comment, there is no need to flush TLBs on all threads in a core when a vcpu moves between threads in the same core.

Thread migrations can be a significant proportion of vcpu migrations, so this can help reduce the TLB flushing and IPI traffic.

---
I believe we can do this and have the TLB coherency correct as per the architecture, but would appreciate someone else verifying my thinking.

Quoted 8 lines

So far I have not been able to convince myself that migrating within a core is really different from migrating across cores as far as the architecture is concerned. If you're trying to allow for an implementation where TLB entries are shared but tlbiel only works (effectively and completely) on the local thread, then I don't think you can do this. If a TLB entry is created on T0, then the vcpu moves to T1 and does a tlbiel, then the guest task on that vcpu migrates to the vcpu that is on T2, it might still see a stale TLB entry.

The difference is that the guest TLBIEL will still execute on the same core, so it should take care of the shared / core-wide translations that were set up. Therefore you just have to worry about the private ones, and in that case you only need to invalidate the threads that it ran on.

Thanks,
Nick