Skip to contentSkip to conversation

KVM: PPC: Book3S HV: Always save guest pmu for guest capable of nesting

PATCH In review Suraj Jitindar Singh 3 participants 7 messages Jul 3, 2019 – Jul 18, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/3 KVM: PPC: Book3S HV: Always save guest pmu for guest capable of nesting

The performance monitoring unit (PMU) registers are saved on guest exit when the guest has set the pmcregs_in_use flag in its lppaca, if it exists, or unconditionally if it doesn't. If a nested guest is being run then the hypervisor doesn't, and in most cases can't, know if the pmu registers are in use since it doesn't know the location of the lppaca for the nested guest, although it may have one for its immediate guest. This results in the values of these registers being lost across nested guest entry and exit in the case where the nested guest was making use of the performance monitoring facility while it's nested guest hypervisor wasn't.

Further more the hypervisor could interrupt a guest hypervisor between when it has loaded up the pmu registers and it calling H_ENTER_NESTED or between returning from the nested guest to the guest hypervisor and the guest hypervisor reading the pmu registers, in kvmhv_p9_guest_entry(). This means that it isn't sufficient to just save the pmu registers when entering or exiting a nested guest, but that it is necessary to always save the pmu registers whenever a guest is capable of running nested guests to ensure the register values aren't lost in the context switch.

Ensure the pmu register values are preserved by always saving their value into the vcpu struct when a guest is capable of running nested guests.

This should have minimal performance impact however any impact can be avoided by booting a guest with "-machine pseries,cap-nested-hvúlse" on the qemu commandline.

Fixes 95a6432ce903 "KVM: PPC: Book3S HV: Streamlined guest entry/exit path on P9 for radix guests"
---
 arch/powerpc/kvm/book3s_hv.c | 2 ++
 1 file changed, 2 insertions(+)
arch/powerpc/kvm/book3s_hv.c +2 -0
index ec1804f822af..b682a429f3ef 100644
@@ -3654,6 +3654,8 @@ int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,
 		vcpu->arch.vpa.dirty = 1;
 		save_pmu = lp->pmcregs_in_use;
 	}
+	/* Must save pmu if this guest is capable of running nested guests */
+	save_pmu |= nesting_enabled(vcpu->kvm);
 
 	kvmhv_save_guest_pmu(vcpu, save_pmu);
 
Signature · 1 line
2.13.6
#1 lore raw

PATCH 2/3 PPC: PMC: Set pmcregs_in_use in paca when running as LPAR

The ability to run nested guests under KVM means that a guest can also act as a hypervisor for it's own nested guest. Currently ppc_set_pmu_inuse() assumes that either FW_FEATURE_LPAR is set, indicating a guest environment, and so sets the pmcregs_in_use flag in the lppaca, or that it isn't set, indicating a hypervisor environment, and so sets the pmcregs_in_use flag in the paca.

The pmcregs_in_use flag in the lppaca is used to communicate this information to a hypervisor and so must be set in a guest environment. The pmcregs_in_use flag in the paca is used by KVM code to determine whether the host state of the performance monitoring unit (PMU) must be saved and restored when running a guest.

Thus when a guest also acts as a hypervisor it must set this bit in both places since it needs to ensure both that the real hypervisor saves it's pmu registers when it runs (requires pmcregs_in_use flag in lppaca), and that it saves it's own pmu registers when running a nested guest (requires pmcregs_in_use flag in paca).

Modify ppc_set_pmu_inuse() so that the pmcregs_in_use bit is set in both the lppaca and the paca when a guest (LPAR) is running with the capability of running it's own guests (CONFIG_KVM_BOOK3S_HV_POSSIBLE).

Fixes 95a6432ce903 "KVM: PPC: Book3S HV: Streamlined guest entry/exit path on P9 for radix guests"
---
 arch/powerpc/include/asm/pmc.h | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
arch/powerpc/include/asm/pmc.h +2 -3
index dc9a1ca70edf..c6bbe9778d3c 100644
@@ -27,11 +27,10 @@ static inline void ppc_set_pmu_inuse(int inuse)
 #ifdef CONFIG_PPC_PSERIES
 		get_lppaca()->pmcregs_in_use = inuse;
 #endif
-	} else {
+	}
 #ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
-		get_paca()->pmcregs_in_use = inuse;
+	get_paca()->pmcregs_in_use = inuse;
 #endif
-	}
 #endif
 }
 
Signature · 1 line
2.13.6
#2 lore raw

PATCH 3/3 KVM: PPC: Book3S HV: Save and restore guest visible PSSCR bits on pseries

In reply to Suraj Jitindar Singh

The performance stop status and control register (PSSCR) is used to control the power saving facilities of the processor. This register has various fields, some of which can be modified only in hypervisor state, and others which can be modified in both hypervisor and priviledged non-hypervisor state. The bits which can be modified in priviledged non-hypervisor state are referred to as guest visible.

Currently the L0 hypervisor saves and restores both it's own host value as well as the guest value of the psscr when context switching between the hypervisor and guest. However a nested hypervisor running it's own nested guests (as indicated by kvmhv_on_pseries()) doesn't context switch the psscr register. This means that if a nested (L2) guest modified the psscr that the L1 guest hypervisor will run with this value, and if the L1 guest hypervisor modified this value and then goes to run the nested (L2) guest again that the L2 psscr value will be lost.

Fix this by having the (L1) nested hypervisor save and restore both its host and the guest psscr value when entering and exiting a nested (L2) guest. Note that only the guest visible parts of the psscr are context switched since this is all the L1 nested hypervisor can access, this is fine however as these are the only fields the L0 hypervisor provides guest control of anyway and so all other fields are ignored.

This could also have been implemented by adding the psscr register to the hv_regs passed to the L0 hypervisor as input to the H_ENTER_NESTED hcall, however this would have meant updating the structure layout and thus required modifications to both the L0 and L1 kernels. Whereas the approach used doesn't require L0 kernel modifications while achieving the same result.

Fixes 95a6432ce903 "KVM: PPC: Book3S HV: Streamlined guest entry/exit path on P9 for radix guests"
---
 arch/powerpc/kvm/book3s_hv.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
arch/powerpc/kvm/book3s_hv.c +11 -0
index b682a429f3ef..cde3f5a4b3e4 100644
@@ -3569,9 +3569,18 @@ int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,
 	mtspr(SPRN_DEC, vcpu->arch.dec_expires - mftb());
 
 	if (kvmhv_on_pseries()) {
+		/*
+		 * We need to save and restore the guest visible part of the
+		 * psscr (i.e. using SPRN_PSSCR_PR) since the hypervisor
+		 * doesn't do this for us. Note only required if pseries since
+		 * this is done in kvmhv_load_hv_regs_and_go() below otherwise.
+		 */
+		unsigned long host_psscr;
 		/* call our hypervisor to load up HV regs and go */
 		struct hv_guest_state hvregs;
 
+		host_psscr = mfspr(SPRN_PSSCR_PR);
+		mtspr(SPRN_PSSCR_PR, vcpu->arch.psscr);
 		kvmhv_save_hv_regs(vcpu, &hvregs);
 		hvregs.lpcr = lpcr;
 		vcpu->arch.regs.msr = vcpu->arch.shregs.msr;
@@ -3590,6 +3599,8 @@ int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,
 		vcpu->arch.shregs.msr = vcpu->arch.regs.msr;
 		vcpu->arch.shregs.dar = mfspr(SPRN_DAR);
 		vcpu->arch.shregs.dsisr = mfspr(SPRN_DSISR);
+		vcpu->arch.psscr = mfspr(SPRN_PSSCR_PR);
+		mtspr(SPRN_PSSCR_PR, host_psscr);
 
 		/* H_CEDE has to be handled now, not later */
 		if (trap = BOOK3S_INTERRUPT_SYSCALL && !vcpu->arch.nested &&
Signature · 1 line
2.13.6
#3 lore raw
In reply to Suraj Jitindar Singh
Quoted 24 lines · Suraj Jitindar Singh

The performance monitoring unit (PMU) registers are saved on guest exit when the guest has set the pmcregs_in_use flag in its lppaca, if it exists, or unconditionally if it doesn't. If a nested guest is being run then the hypervisor doesn't, and in most cases can't, know if the pmu registers are in use since it doesn't know the location of the lppaca for the nested guest, although it may have one for its immediate guest. This results in the values of these registers being lost across nested guest entry and exit in the case where the nested guest was making use of the performance monitoring facility while it's nested guest hypervisor wasn't.

Further more the hypervisor could interrupt a guest hypervisor between when it has loaded up the pmu registers and it calling H_ENTER_NESTED or between returning from the nested guest to the guest hypervisor and the guest hypervisor reading the pmu registers, in kvmhv_p9_guest_entry(). This means that it isn't sufficient to just save the pmu registers when entering or exiting a nested guest, but that it is necessary to always save the pmu registers whenever a guest is capable of running nested guests to ensure the register values aren't lost in the context switch.

Ensure the pmu register values are preserved by always saving their value into the vcpu struct when a guest is capable of running nested guests.

Quoted 5 lines

This should have minimal performance impact however any impact can be avoided by booting a guest with "-machine pseries,cap-nested-hvúlse" on the qemu commandline.

Fixes 95a6432ce903 "KVM: PPC: Book3S HV: Streamlined guest entry/exit path on P9 for radix guests"

I'm not clear why this and the next commit are marked as fixing the above commit. Wasn't it broken prior to that commit as well?

cheers

Quoted 20 lines
---
 arch/powerpc/kvm/book3s_hv.c | 2 ++
 1 file changed, 2 insertions(+)
arch/powerpc/kvm/book3s_hv.c +2 -0
index ec1804f822af..b682a429f3ef 100644
@@ -3654,6 +3654,8 @@ int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,
 		vcpu->arch.vpa.dirty = 1;
 		save_pmu = lp->pmcregs_in_use;
 	}
+	/* Must save pmu if this guest is capable of running nested guests */
+	save_pmu |= nesting_enabled(vcpu->kvm);
 
 	kvmhv_save_guest_pmu(vcpu, save_pmu);
 
Signature · 1 line
2.13.6
#4 lore raw
Quoted 47 lines · Michael Ellerman
Quoted 46 lines · Suraj Jitindar Singh

The performance monitoring unit (PMU) registers are saved on guest exit
when the guest has set the pmcregs_in_use flag in its lppaca, if it exists, or unconditionally if it doesn't. If a nested guest is being
run then the hypervisor doesn't, and in most cases can't, know if the
pmu registers are in use since it doesn't know the location of the lppaca
for the nested guest, although it may have one for its immediate guest.
This results in the values of these registers being lost across nested
guest entry and exit in the case where the nested guest was making use
of the performance monitoring facility while it's nested guest hypervisor
wasn't.

Further more the hypervisor could interrupt a guest hypervisor between
when it has loaded up the pmu registers and it calling H_ENTER_NESTED or
between returning from the nested guest to the guest hypervisor and the
guest hypervisor reading the pmu registers, in kvmhv_p9_guest_entry().
This means that it isn't sufficient to just save the pmu registers when
entering or exiting a nested guest, but that it is necessary to always
save the pmu registers whenever a guest is capable of running nested guests
to ensure the register values aren't lost in the context switch.

Ensure the pmu register values are preserved by always saving their value into the vcpu struct when a guest is capable of running nested
guests.

This should have minimal performance impact however any impact can be
avoided by booting a guest with "-machine pseries,cap-nested- hvúlse"
on the qemu commandline.

Quoted 5 lines
Quoted 2 lines
Fixes 95a6432ce903 "KVM: PPC: Book3S HV: Streamlined guest

entry/exit path on P9 for radix guests"

I'm not clear why this and the next commit are marked as fixing the above commit. Wasn't it broken prior to that commit as well?

That was the commit which introduced the entry path which we use for a nested guest, the path on which we need to be saving and restoring the pmu registers and so where the new code was introduced.

It wasn't technically broken prior to that commit since you couldn't run nested prior to that commit, and in fact it's a few commits after that one where we actually enabled the ability to run nested guests.

However since that's the code which introduced the nested entry path it seemed like the best fit for the fixes tag for people who will be looking for fixes in that area. Also all the other nested entry path fixes used that fixes tag so it ties them together nicely.

Thanks,
Suraj

Quoted 26 lines

cheers

Quoted 23 lines
---
 arch/powerpc/kvm/book3s_hv.c | 2 ++
 1 file changed, 2 insertions(+)
arch/powerpc/kvm/book3s_hv.c +1 -0
index ec1804f822af..b682a429f3ef 100644
@@ -3654,6 +3654,8 @@ int kvmhv_p9_guest_entry(struct kvm_vcpu
 *vcpu, u64 time_limit,
 		vcpu->arch.vpa.dirty = 1;
 		save_pmu = lp->pmcregs_in_use;
 	}
+	/* Must save pmu if this guest is capable of running

nested guests */

arch/powerpc/kvm/book3s_hv.c +1 -0
+	save_pmu |= nesting_enabled(vcpu->kvm);
 
 	kvmhv_save_guest_pmu(vcpu, save_pmu);
Signature · 2 lines
-- 
2.13.6
#5 lore raw
Quoted 2 lines · Suraj Jitindar Singh
Quoted 1 line · Michael Ellerman

Suraj Jitindar Singh <[email protected]> writes:

...

Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
Fixes 95a6432ce903 "KVM: PPC: Book3S HV: Streamlined guest

entry/exit path on P9 for radix guests"

I'm not clear why this and the next commit are marked as fixing the above commit. Wasn't it broken prior to that commit as well?

That was the commit which introduced the entry path which we use for a nested guest, the path on which we need to be saving and restoring the pmu registers and so where the new code was introduced.

OK, I thought that commit was an unrelated optimisation. Agree that is a good target if it is the commit that introduced the nested path.

cheers

#6 lore raw
In reply to Suraj Jitindar Singh
Quoted 24 lines · Suraj Jitindar Singh

The performance monitoring unit (PMU) registers are saved on guest exit when the guest has set the pmcregs_in_use flag in its lppaca, if it exists, or unconditionally if it doesn't. If a nested guest is being run then the hypervisor doesn't, and in most cases can't, know if the pmu registers are in use since it doesn't know the location of the lppaca for the nested guest, although it may have one for its immediate guest. This results in the values of these registers being lost across nested guest entry and exit in the case where the nested guest was making use of the performance monitoring facility while it's nested guest hypervisor wasn't.

Further more the hypervisor could interrupt a guest hypervisor between when it has loaded up the pmu registers and it calling H_ENTER_NESTED or between returning from the nested guest to the guest hypervisor and the guest hypervisor reading the pmu registers, in kvmhv_p9_guest_entry(). This means that it isn't sufficient to just save the pmu registers when entering or exiting a nested guest, but that it is necessary to always save the pmu registers whenever a guest is capable of running nested guests to ensure the register values aren't lost in the context switch.

Ensure the pmu register values are preserved by always saving their value into the vcpu struct when a guest is capable of running nested guests.

Quoted 7 lines

This should have minimal performance impact however any impact can be avoided by booting a guest with "-machine pseries,cap-nested-hvúlse" on the qemu commandline.

Fixes 95a6432ce903 "KVM: PPC: Book3S HV: Streamlined guest entry/exit path on P9 for radix guests"

Series applied to powerpc fixes, thanks.

https://git.kernel.org/powerpc/c/63279eeb7f93abb1692573c26f1e038e1a87358b

cheers