Skip to contentSkip to conversation

KVM: x86: add hint to skip hidden rdpkru under kvm_load_host_xsave_state

PATCH In review Jon Kohler 7 participants 19 messages May 7, 2021 – May 19, 2021 1 message not in the archive

Subsystems (3)
lore
Share & tools
lore

Jon Kohler

author
#0 lore raw

PATCH 1/1 KVM: x86: add hint to skip hidden rdpkru under kvm_load_host_xsave_state

kvm_load_host_xsave_state handles xsave on vm exit, part of which is managing memory protection key state. The latest arch.pkru is updated with a rdpkru, and if that doesn't match the base host_pkru (which about 70% of the time), we issue a __write_pkru.

__write_pkru issues another rdpkru internally to try to avoid the wrpkru, so we're reading the same value back to back when we 100% of the time know that we need to go directly to wrpkru. This is a 100% branch miss and extra work that can be skipped.

To improve performance, add a hint to __write_pkru so that callers can specify if they've already done the check themselves. The compiler seems to filter this branch this out completely when the hint is true, so incrementally tighter code is generated as well.

While we're in this section of code, optimize if condition ordering prior to __write_pkru in both kvm_load_{guest|host}_xsave_state.

For both functions, flip the ordering of the || condition so that arch.xcr0 & XFEATURE_MASK_PKRU is checked first, which when instrumented in our evironment appeared to be always true and less overall work than kvm_read_cr4_bits.

For kvm_load_guest_xsave_state, hoist arch.pkru != host_pkru ahead one position. When instrumented, I saw this be true roughly ~70% of the time vs the other conditions which were almost always true. With this change, we will avoid 3rd condition check ~30% of the time.

---
 arch/x86/include/asm/fpu/internal.h  |  2 +-
 arch/x86/include/asm/pgtable.h       |  2 +-
 arch/x86/include/asm/special_insns.h | 10 ++++++----
 arch/x86/kvm/x86.c                   | 14 +++++++-------
 4 files changed, 15 insertions(+), 13 deletions(-)
arch/x86/include/asm/fpu/internal.h +1 -1
index 8d33ad80704f..0860bbadb422 100644
@@ -583,7 +583,7 @@ static inline void switch_fpu_finish(struct fpu *new_fpu)
 		if (pk)
 			pkru_val = pk->pkru;
 	}
-	__write_pkru(pkru_val);
+	__write_pkru(pkru_val, false);
 
 	/*
 	 * Expensive PASID MSR write will be avoided in update_pasid() because
arch/x86/include/asm/pgtable.h +1 -1
index b1099f2d9800..8c8d4796b864 100644
@@ -151,7 +151,7 @@ static inline void write_pkru(u32 pkru)
 	fpregs_lock();
 	if (pk)
 		pk->pkru = pkru;
-	__write_pkru(pkru);
+	__write_pkru(pkru, false);
 	fpregs_unlock();
 }
 
arch/x86/include/asm/special_insns.h +6 -4
index 2acd6cb62328..f4ac8ec3f096 100644
@@ -104,13 +104,15 @@ static inline void wrpkru(u32 pkru)
 		     : : "a" (pkru), "c"(ecx), "d"(edx));
 }
 
-static inline void __write_pkru(u32 pkru)
+static inline void __write_pkru(u32 pkru, bool skip_comparison)
 {
 	/*
 	 * WRPKRU is relatively expensive compared to RDPKRU.
-	 * Avoid WRPKRU when it would not change the value.
+	 * Avoid WRPKRU when it would not change the value,
+	 * except when the caller has already done the
+	 * comparison, in which case skip redundant RDPKRU.
 	 */
-	if (pkru == rdpkru())
+	if (!skip_comparison && pkru == rdpkru())
 		return;
 
 	wrpkru(pkru);
@@ -122,7 +124,7 @@ static inline u32 rdpkru(void)
 	return 0;
 }
 
-static inline void __write_pkru(u32 pkru)
+static inline void __write_pkru(u32 pkru, bool skip_comparison)
 {
 }
 #endif
arch/x86/kvm/x86.c +7 -7
index cebdaa1e3cf5..cd95adbd140c 100644
@@ -912,10 +912,10 @@ void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
 	}
 
 	if (static_cpu_has(X86_FEATURE_PKU) &&
-	    (kvm_read_cr4_bits(vcpu, X86_CR4_PKE) ||
-	     (vcpu->arch.xcr0 & XFEATURE_MASK_PKRU)) &&
-	    vcpu->arch.pkru != vcpu->arch.host_pkru)
-		__write_pkru(vcpu->arch.pkru);
+	    vcpu->arch.pkru != vcpu->arch.host_pkru &&
+	    ((vcpu->arch.xcr0 & XFEATURE_MASK_PKRU) ||
+	     kvm_read_cr4_bits(vcpu, X86_CR4_PKE)))
+		__write_pkru(vcpu->arch.pkru, false);
 }
 EXPORT_SYMBOL_GPL(kvm_load_guest_xsave_state);
 
@@ -925,11 +925,11 @@ void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
 		return;
 
 	if (static_cpu_has(X86_FEATURE_PKU) &&
-	    (kvm_read_cr4_bits(vcpu, X86_CR4_PKE) ||
-	     (vcpu->arch.xcr0 & XFEATURE_MASK_PKRU))) {
+	    ((vcpu->arch.xcr0 & XFEATURE_MASK_PKRU) ||
+	     kvm_read_cr4_bits(vcpu, X86_CR4_PKE))) {
 		vcpu->arch.pkru = rdpkru();
 		if (vcpu->arch.pkru != vcpu->arch.host_pkru)
-			__write_pkru(vcpu->arch.host_pkru);
+			__write_pkru(vcpu->arch.host_pkru, true);
 	}
 
 	if (kvm_read_cr4_bits(vcpu, X86_CR4_OSXSAVE)) {

--
2.30.1 (Apple Git-130)

maintainer
#1 lore raw

Nice one. The patch can be made simpler though (I think).

On 07/05/21 18:44, Jon Kohler wrote:
@@ -122,7 +124,7 @@ static inline u32 rdpkru(void)

Quoted 14 lines
  	return 0;
  }
arch/x86/include/asm/special_insns.h +1 -1
-static inline void __write_pkru(u32 pkru)
+static inline void __write_pkru(u32 pkru, bool skip_comparison)
  {
  }
  #endif
arch/x86/kvm/x86.c +0 -0
index cebdaa1e3cf5..cd95adbd140c 100644
@@ -912,10 +912,10 @@ void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
  	}

arch/x86/kvm/x86.c void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu) · line 918

Quoted 10 lines
arch/x86/kvm/x86.c +4 -4
 
  	if (static_cpu_has(X86_FEATURE_PKU) &&
-	    (kvm_read_cr4_bits(vcpu, X86_CR4_PKE) ||
-	     (vcpu->arch.xcr0 & XFEATURE_MASK_PKRU)) &&
-	    vcpu->arch.pkru != vcpu->arch.host_pkru)
-		__write_pkru(vcpu->arch.pkru);
+	    vcpu->arch.pkru != vcpu->arch.host_pkru &&
+	    ((vcpu->arch.xcr0 & XFEATURE_MASK_PKRU) ||
+	     kvm_read_cr4_bits(vcpu, X86_CR4_PKE)))
+		__write_pkru(vcpu->arch.pkru, false);

This can be optimized as well, can't it? This means that the only case that needs the rdpkru is in switch_fpu_finish, and __write_pkru can be removed completely:

- do the rdpkru+wrpkru in switch_fpu_finish

- just use wrpkru in KVM

Paolo

Quoted 21 lines
arch/x86/kvm/x86.c +3 -4
  }
  EXPORT_SYMBOL_GPL(kvm_load_guest_xsave_state);
 
@@ -925,11 +925,11 @@ void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
  		return;
 
  	if (static_cpu_has(X86_FEATURE_PKU) &&
-	    (kvm_read_cr4_bits(vcpu, X86_CR4_PKE) ||
-	     (vcpu->arch.xcr0 & XFEATURE_MASK_PKRU))) {
+	    ((vcpu->arch.xcr0 & XFEATURE_MASK_PKRU) ||
+	     kvm_read_cr4_bits(vcpu, X86_CR4_PKE))) {
  		vcpu->arch.pkru = rdpkru();
  		if (vcpu->arch.pkru != vcpu->arch.host_pkru)
-			__write_pkru(vcpu->arch.host_pkru);
+			__write_pkru(vcpu->arch.host_pkru, true);
  	}
 
  	if (kvm_read_cr4_bits(vcpu, X86_CR4_OSXSAVE)) {
--

2.30.1 (Apple Git-130)

#2 lore raw
Quoted 7 lines · Paolo Bonzini

This can be optimized as well, can't it?  This means that the only case that needs the rdpkru is in switch_fpu_finish, and __write_pkru can be removed completely:

- do the rdpkru+wrpkru in switch_fpu_finish

- just use wrpkru in KVM

I was going to suggest exactly the same thing. It doesn't require the compiler to be smart, and wrpkru() is available in the same header as __write_pkru().

I also detest the mysterious true/false arguments to functions where you have no clue what they're doing at the call site without comments.

author
#3 lore raw
Quoted 9 lines · Dave Hansen
Quoted 4 lines · Paolo Bonzini

This can be optimized as well, can't it? This means that the only case that needs the rdpkru is in switch_fpu_finish, and __write_pkru can be removed completely:

Quoted 7 lines
Quoted 3 lines

- do the rdpkru+wrpkru in switch_fpu_finish

- just use wrpkru in KVM

I was going to suggest exactly the same thing. It doesn't require the compiler to be smart, and wrpkru() is available in the same header as __write_pkru().

Thanks Paolo, Dave. Good suggestion, I’ll work up a v2 and send it out.

Quoted 3 lines

I also detest the mysterious true/false arguments to functions where you have no clue what they're doing at the call site without comments.

Noted, my apologies. Thats a good point, I won’t make that mistake again.

#4 lore raw
In reply to Jon Kohler
Quoted 5 lines · Jon Kohler

kvm_load_host_xsave_state handles xsave on vm exit, part of which is managing memory protection key state. The latest arch.pkru is updated with a rdpkru, and if that doesn't match the base host_pkru (which about 70% of the time), we issue a __write_pkru.

This thread caused me to read the code, and I don't get how it's even remotely correct.

First, kvm_load_guest_fpu() has this delight:

    /*
     * Guests with protected state can't have it set by the hypervisor,
     * so skip trying to set it.
     */
    if (vcpu->arch.guest_fpu)
        /* PKRU is separately restored in kvm_x86_ops.run. */
        __copy_kernel_to_fpregs(&vcpu->arch.guest_fpu->state,
                    ~XFEATURE_MASK_PKRU);

That's nice, but it fails to restore XINUSE[PKRU]. As far as I know, that bit is live, and the only way to restore it to 0 is with XRSTOR(S).

Quoted 6 lines
arch/x86/kvm/x86.c +0 -0
index cebdaa1e3cf5..cd95adbd140c 100644
@@ -912,10 +912,10 @@ void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
        }

arch/x86/kvm/x86.c void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu) · line 918

Quoted 10 lines
arch/x86/kvm/x86.c +4 -4
 
        if (static_cpu_has(X86_FEATURE_PKU) &&
-           (kvm_read_cr4_bits(vcpu, X86_CR4_PKE) ||
-            (vcpu->arch.xcr0 & XFEATURE_MASK_PKRU)) &&
-           vcpu->arch.pkru != vcpu->arch.host_pkru)
-               __write_pkru(vcpu->arch.pkru);
+           vcpu->arch.pkru != vcpu->arch.host_pkru &&
+           ((vcpu->arch.xcr0 & XFEATURE_MASK_PKRU) ||
+            kvm_read_cr4_bits(vcpu, X86_CR4_PKE)))
+               __write_pkru(vcpu->arch.pkru, false);

Please tell me I'm missing something (e.g. KVM very cleverly managing the PKRU register using intercepts) that makes this reliably load the guest value. An innocent or malicious guest could easily make that condition evaluate to false, thus allowing the host PKRU value to be live in guest mode. (Or is something fancy going on here?)

I don't even want to think about what happens if a perf NMI hits and accesses host user memory while the guest PKRU is live (on VMX -- I think this can't happen on SVM).

Quoted 6 lines
arch/x86/kvm/x86.c +0 -0
 }
 EXPORT_SYMBOL_GPL(kvm_load_guest_xsave_state);
 
@@ -925,11 +925,11 @@ void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
                return;
 

arch/x86/kvm/x86.c void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu) · line 933

Quoted 10 lines
arch/x86/kvm/x86.c +3 -3
        if (static_cpu_has(X86_FEATURE_PKU) &&
-           (kvm_read_cr4_bits(vcpu, X86_CR4_PKE) ||
-            (vcpu->arch.xcr0 & XFEATURE_MASK_PKRU))) {
+           ((vcpu->arch.xcr0 & XFEATURE_MASK_PKRU) ||
+            kvm_read_cr4_bits(vcpu, X86_CR4_PKE))) {
                vcpu->arch.pkru = rdpkru();
                if (vcpu->arch.pkru != vcpu->arch.host_pkru)
-                       __write_pkru(vcpu->arch.host_pkru);
+                       __write_pkru(vcpu->arch.host_pkru, true);
        }

Suppose the guest writes to PKRU and then, without exiting, sets PKE = 0 and XCR0[PKRU] = 0. (Or are the intercepts such that this can't happen except on SEV where maybe SEV magic makes the problem go away?)

I admit I'm fairly mystified as to why KVM doesn't handle PKRU like the rest of guest XSTATE.

author
#5 lore raw
Quoted 25 lines · Andy Lutomirski
Quoted 5 lines · Jon Kohler

kvm_load_host_xsave_state handles xsave on vm exit, part of which is managing memory protection key state. The latest arch.pkru is updated with a rdpkru, and if that doesn't match the base host_pkru (which about 70% of the time), we issue a __write_pkru.

This thread caused me to read the code, and I don't get how it's even remotely correct.

First, kvm_load_guest_fpu() has this delight:

   /*
    * Guests with protected state can't have it set by the hypervisor,
    * so skip trying to set it.
    */
   if (vcpu->arch.guest_fpu)
       /* PKRU is separately restored in kvm_x86_ops.run. */
       __copy_kernel_to_fpregs(&vcpu->arch.guest_fpu->state,
                   ~XFEATURE_MASK_PKRU);
Quoted 3 lines

That's nice, but it fails to restore XINUSE[PKRU]. As far as I know, that bit is live, and the only way to restore it to 0 is with XRSTOR(S).

Thanks, Andy - Adding Tom Lendacky to this thread as that Particular snippet was last touched in ~5.11 in ed02b213098a.

Quoted 24 lines
Quoted 7 lines
arch/x86/kvm/x86.c +0 -0
index cebdaa1e3cf5..cd95adbd140c 100644
@@ -912,10 +912,10 @@ void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
       }

arch/x86/kvm/x86.c void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu) · line 918

Quoted 10 lines
arch/x86/kvm/x86.c +4 -4
 
       if (static_cpu_has(X86_FEATURE_PKU) &&
-           (kvm_read_cr4_bits(vcpu, X86_CR4_PKE) ||
-            (vcpu->arch.xcr0 & XFEATURE_MASK_PKRU)) &&
-           vcpu->arch.pkru != vcpu->arch.host_pkru)
-               __write_pkru(vcpu->arch.pkru);
+           vcpu->arch.pkru != vcpu->arch.host_pkru &&
+           ((vcpu->arch.xcr0 & XFEATURE_MASK_PKRU) ||
+            kvm_read_cr4_bits(vcpu, X86_CR4_PKE)))
+               __write_pkru(vcpu->arch.pkru, false);

Please tell me I'm missing something (e.g. KVM very cleverly managing the PKRU register using intercepts) that makes this reliably load the guest value. An innocent or malicious guest could easily make that condition evaluate to false, thus allowing the host PKRU value to be live in guest mode. (Or is something fancy going on here?)

Quoted 3 lines

I don't even want to think about what happens if a perf NMI hits and accesses host user memory while the guest PKRU is live (on VMX -- I think this can't happen on SVM).

Perhaps Babu / Dave can comment on the exiting logic here? Babu did some additional work to fix save/restore on 37486135d3a.

From this changes perspective, I’m just trying to get to the resultant evaluation a bit quicker, which this change (and the v3) seems to do ok with; however, if I’ve ran my ship into a larger ice berg, happy to collaborate to make it better overall.

Quoted 22 lines
Quoted 7 lines
arch/x86/kvm/x86.c +0 -0
 }
 EXPORT_SYMBOL_GPL(kvm_load_guest_xsave_state);
 
@@ -925,11 +925,11 @@ void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
               return;
 

arch/x86/kvm/x86.c void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu) · line 933

Quoted 10 lines
arch/x86/kvm/x86.c +3 -3
       if (static_cpu_has(X86_FEATURE_PKU) &&
-           (kvm_read_cr4_bits(vcpu, X86_CR4_PKE) ||
-            (vcpu->arch.xcr0 & XFEATURE_MASK_PKRU))) {
+           ((vcpu->arch.xcr0 & XFEATURE_MASK_PKRU) ||
+            kvm_read_cr4_bits(vcpu, X86_CR4_PKE))) {
               vcpu->arch.pkru = rdpkru();
               if (vcpu->arch.pkru != vcpu->arch.host_pkru)
-                       __write_pkru(vcpu->arch.host_pkru);
+                       __write_pkru(vcpu->arch.host_pkru, true);
       }

Suppose the guest writes to PKRU and then, without exiting, sets PKE = 0 and XCR0[PKRU] = 0. (Or are the intercepts such that this can't happen except on SEV where maybe SEV magic makes the problem go away?)

Quoted 2 lines

I admit I'm fairly mystified as to why KVM doesn't handle PKRU like the rest of guest XSTATE.

I’ll defer to Dave/Babu here. This code has been this way for a bit now, I’m not sure at first glance if that situation could occur intentionally or accidentally, but that would evaluate the same both before and after this change, no?

#6 lore raw
Quoted 25 lines · Jon Kohler
Quoted 25 lines · Andy Lutomirski
Quoted 5 lines · Jon Kohler

kvm_load_host_xsave_state handles xsave on vm exit, part of which is managing memory protection key state. The latest arch.pkru is updated with a rdpkru, and if that doesn't match the base host_pkru (which about 70% of the time), we issue a __write_pkru.

This thread caused me to read the code, and I don't get how it's even remotely correct.

First, kvm_load_guest_fpu() has this delight:

   /*
    * Guests with protected state can't have it set by the hypervisor,
    * so skip trying to set it.
    */
   if (vcpu->arch.guest_fpu)
       /* PKRU is separately restored in kvm_x86_ops.run. */
       __copy_kernel_to_fpregs(&vcpu->arch.guest_fpu->state,
                   ~XFEATURE_MASK_PKRU);
Quoted 6 lines
Quoted 3 lines

That's nice, but it fails to restore XINUSE[PKRU]. As far as I know, that bit is live, and the only way to restore it to 0 is with XRSTOR(S).

Thanks, Andy - Adding Tom Lendacky to this thread as that Particular snippet was last touched in ~5.11 in ed02b213098a.

It sounds like Andy's comment is separate from the changes associated with commit ed02b213098a, right?

Commit ed02b213098a just added the check for vcpu->arch.guest_fpu to support SEV-ES guests. Since the hypervisor can't save/restore those registers directly when running under SEV-ES, we skip this. The state will be restored when VMRUN is performed.

Thanks,
Tom

Quoted 67 lines
Quoted 25 lines
Quoted 8 lines
arch/x86/kvm/x86.c +0 -0
index cebdaa1e3cf5..cd95adbd140c 100644
@@ -912,10 +912,10 @@ void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
       }

arch/x86/kvm/x86.c void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu) · line 918

Quoted 10 lines
arch/x86/kvm/x86.c +4 -4
 
       if (static_cpu_has(X86_FEATURE_PKU) &&
-           (kvm_read_cr4_bits(vcpu, X86_CR4_PKE) ||
-            (vcpu->arch.xcr0 & XFEATURE_MASK_PKRU)) &&
-           vcpu->arch.pkru != vcpu->arch.host_pkru)
-               __write_pkru(vcpu->arch.pkru);
+           vcpu->arch.pkru != vcpu->arch.host_pkru &&
+           ((vcpu->arch.xcr0 & XFEATURE_MASK_PKRU) ||
+            kvm_read_cr4_bits(vcpu, X86_CR4_PKE)))
+               __write_pkru(vcpu->arch.pkru, false);

Please tell me I'm missing something (e.g. KVM very cleverly managing the PKRU register using intercepts) that makes this reliably load the guest value. An innocent or malicious guest could easily make that condition evaluate to false, thus allowing the host PKRU value to be live in guest mode. (Or is something fancy going on here?)

Quoted 3 lines

I don't even want to think about what happens if a perf NMI hits and accesses host user memory while the guest PKRU is live (on VMX -- I think this can't happen on SVM).

Perhaps Babu / Dave can comment on the exiting logic here? Babu did some additional work to fix save/restore on 37486135d3a.

From this changes perspective, I’m just trying to get to the resultant evaluation a bit quicker, which this change (and the v3) seems to do ok with; however, if I’ve ran my ship into a larger ice berg, happy to collaborate to make it better overall.

Quoted 22 lines
Quoted 7 lines
arch/x86/kvm/x86.c +0 -0
 }
 EXPORT_SYMBOL_GPL(kvm_load_guest_xsave_state);
 
@@ -925,11 +925,11 @@ void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
               return;
 

arch/x86/kvm/x86.c void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu) · line 933

Quoted 10 lines
arch/x86/kvm/x86.c +3 -3
       if (static_cpu_has(X86_FEATURE_PKU) &&
-           (kvm_read_cr4_bits(vcpu, X86_CR4_PKE) ||
-            (vcpu->arch.xcr0 & XFEATURE_MASK_PKRU))) {
+           ((vcpu->arch.xcr0 & XFEATURE_MASK_PKRU) ||
+            kvm_read_cr4_bits(vcpu, X86_CR4_PKE))) {
               vcpu->arch.pkru = rdpkru();
               if (vcpu->arch.pkru != vcpu->arch.host_pkru)
-                       __write_pkru(vcpu->arch.host_pkru);
+                       __write_pkru(vcpu->arch.host_pkru, true);
       }

Suppose the guest writes to PKRU and then, without exiting, sets PKE = 0 and XCR0[PKRU] = 0. (Or are the intercepts such that this can't happen except on SEV where maybe SEV magic makes the problem go away?)

Quoted 2 lines

I admit I'm fairly mystified as to why KVM doesn't handle PKRU like the rest of guest XSTATE.

I’ll defer to Dave/Babu here. This code has been this way for a bit now, I’m not sure at first glance if that situation could occur intentionally or accidentally, but that would evaluate the same both before and after this change, no?

maintainer
#7 lore raw
In reply to Andy Lutomirski
Quoted 3 lines · Andy Lutomirski

That's nice, but it fails to restore XINUSE[PKRU]. As far as I know, that bit is live, and the only way to restore it to 0 is with XRSTOR(S).

The manual says "It is possible for XINUSE[i] to be 1 even when state component i is in its initial configuration" so this is architecturally valid. Does the XINUSE optimization matter for PKRU which is a single word?

Quoted 7 lines
Quoted 7 lines
arch/x86/kvm/x86.c +0 -0
index cebdaa1e3cf5..cd95adbd140c 100644
@@ -912,10 +912,10 @@ void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
         }
 
Quoted 15 lines

arch/x86/kvm/x86.c void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu) · line 918

Quoted 9 lines
arch/x86/kvm/x86.c +4 -4
         if (static_cpu_has(X86_FEATURE_PKU) &&
-           (kvm_read_cr4_bits(vcpu, X86_CR4_PKE) ||
-            (vcpu->arch.xcr0 & XFEATURE_MASK_PKRU)) &&
-           vcpu->arch.pkru != vcpu->arch.host_pkru)
-               __write_pkru(vcpu->arch.pkru);
+           vcpu->arch.pkru != vcpu->arch.host_pkru &&
+           ((vcpu->arch.xcr0 & XFEATURE_MASK_PKRU) ||
+            kvm_read_cr4_bits(vcpu, X86_CR4_PKE)))
+               __write_pkru(vcpu->arch.pkru, false);

Please tell me I'm missing something (e.g. KVM very cleverly managing the PKRU register using intercepts) that makes this reliably load the guest value. An innocent or malicious guest could easily make that condition evaluate to false, thus allowing the host PKRU value to be live in guest mode. (Or is something fancy going on here?)

RDPKRU/WRPKRU cannot be used unless CR4.PKE=1, but PKRU can still be accessed using XSAVE/XRSTOR. However both CR4 and XCR0 have their writes trapped, so the guest will not use the host PKRU value before the next vmexit if CR4.PKE=0 and XCR0.PKRU=0.

Quoted 3 lines

I don't even want to think about what happens if a perf NMI hits and accesses host user memory while the guest PKRU is live (on VMX -- I think this can't happen on SVM).

This is indeed a problem, which indeed cannot happen on SVM but is there on VMX. Note that the function above is not handling all of the xstate, it's handling the *XSAVE state*, that is XCR0, XSS and PKRU. Thus the window is small, but it's there.

Is it solvable at all, without having PKRU fields in the VMCS (and without masking NMIs in the LAPIC which would be too expensive)? Dave, Sean, what do you think?

Quoted 6 lines
Quoted 6 lines
arch/x86/kvm/x86.c +0 -0
  }
  EXPORT_SYMBOL_GPL(kvm_load_guest_xsave_state);
 
@@ -925,11 +925,11 @@ void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
                 return;
 
Quoted 14 lines

arch/x86/kvm/x86.c void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu) · line 933

Quoted 10 lines
arch/x86/kvm/x86.c +3 -3
         if (static_cpu_has(X86_FEATURE_PKU) &&
-           (kvm_read_cr4_bits(vcpu, X86_CR4_PKE) ||
-            (vcpu->arch.xcr0 & XFEATURE_MASK_PKRU))) {
+           ((vcpu->arch.xcr0 & XFEATURE_MASK_PKRU) ||
+            kvm_read_cr4_bits(vcpu, X86_CR4_PKE))) {
                 vcpu->arch.pkru = rdpkru();
                 if (vcpu->arch.pkru != vcpu->arch.host_pkru)
-                       __write_pkru(vcpu->arch.host_pkru);
+                       __write_pkru(vcpu->arch.host_pkru, true);
         }

Suppose the guest writes to PKRU and then, without exiting, sets PKE = 0 and XCR0[PKRU] = 0. (Or are the intercepts such that this can't happen except on SEV where maybe SEV magic makes the problem go away?)

Yes, see above. KVM needs to trap CR4 and XCR0 anyway (CR4 because you don't want the guest to clear e.g. MCE, XCR0 to forbid setting bits that the host kernel does not have in its own xstate).

Quoted 2 lines

I admit I'm fairly mystified as to why KVM doesn't handle PKRU like the rest of guest XSTATE.

Because the rest of the guest XSTATE is live too early. The problem you mention above with respect to perf, where you access host memory with the guest PKRU, would be very much amplified.

It is basically the opposite problem of what you have in switch_fpu_finish, which loads PKRU eagerly while delaying the rest to the switch to userland.

Paolo

#8 lore raw
Quoted 5 lines · Paolo Bonzini
Quoted 3 lines · Andy Lutomirski

I don't even want to think about what happens if a perf NMI hits and accesses host user memory while the guest PKRU is live (on VMX -- I think this can't happen on SVM).

Quoted 8 lines

This is indeed a problem, which indeed cannot happen on SVM but is there on VMX. Note that the function above is not handling all of the xstate, it's handling the *XSAVE state*, that is XCR0, XSS and PKRU. Thus the window is small, but it's there.

Is it solvable at all, without having PKRU fields in the VMCS (and without masking NMIs in the LAPIC which would be too expensive)? Dave, Sean, what do you think?

The least awful solution would be to have the NMI handler restore the host's PKRU. The NMI handler would need to save/restore the register, a la CR2, but the whole thing could be optimized to run if and only if the NMI lands in the window where the guest's PKRU is loaded.

#9 lore raw
Quoted 10 lines · Sean Christopherson
Quoted 9 lines · Paolo Bonzini
Quoted 3 lines · Andy Lutomirski

I don't even want to think about what happens if a perf NMI hits and accesses host user memory while the guest PKRU is live (on VMX -- I think this can't happen on SVM).

This is indeed a problem, which indeed cannot happen on SVM but is there on VMX. Note that the function above is not handling all of the xstate, it's handling the *XSAVE state*, that is XCR0, XSS and PKRU. Thus the window is small, but it's there.

Quoted 7 lines
Quoted 3 lines

Is it solvable at all, without having PKRU fields in the VMCS (and without masking NMIs in the LAPIC which would be too expensive)? Dave, Sean, what do you think?

The least awful solution would be to have the NMI handler restore the host's PKRU. The NMI handler would need to save/restore the register, a la CR2, but the whole thing could be optimized to run if and only if the NMI lands in the window where the guest's PKRU is loaded.

What were you thinking about? Something like:

	*this_cpu_ptr(&need_nmi_wpkru) = 1
	// Enter Guest
	__write_pkru(vcpu->arch.pkru);
	*this_cpu_ptr(&need_nmi_wpkru) = 0

And then in the NMI handler:

	u32 pkru;

	if (*this_cpu_ptr(&need_nmi_wpkru)) {
		pkru = rdpku();
		__write_pkru(vcpu->arch.pkru);
	}
	...
	copy_*_user_nmi(... whatever ...);
	...
	if (*this_cpu_ptr(&need_nmi_wpkru))
		__write_pkru(pkru);

?

I was thinking we could just blow away PKRU without saving/restoring it in the NMI handler, but that might clobber PKRU in the window between need_nmi_wpkru=1 and entering the guest.

But, the save/restore seems doable especially since we can do it in C and don't have to mess with the NMI stack or anything.

#10 lore raw
Quoted 26 lines · Dave Hansen
Quoted 10 lines · Sean Christopherson
Quoted 9 lines · Paolo Bonzini
Quoted 3 lines · Andy Lutomirski

I don't even want to think about what happens if a perf NMI hits and accesses host user memory while the guest PKRU is live (on VMX -- I think this can't happen on SVM).

This is indeed a problem, which indeed cannot happen on SVM but is there on VMX. Note that the function above is not handling all of the xstate, it's handling the *XSAVE state*, that is XCR0, XSS and PKRU. Thus the window is small, but it's there.

Quoted 7 lines
Quoted 3 lines

Is it solvable at all, without having PKRU fields in the VMCS (and without masking NMIs in the LAPIC which would be too expensive)? Dave, Sean, what do you think?

The least awful solution would be to have the NMI handler restore the host's PKRU. The NMI handler would need to save/restore the register, a la CR2, but the whole thing could be optimized to run if and only if the NMI lands in the window where the guest's PKRU is loaded.

What were you thinking about? Something like:

	*this_cpu_ptr(&need_nmi_wpkru) = 1
	// Enter Guest
	__write_pkru(vcpu->arch.pkru);
	*this_cpu_ptr(&need_nmi_wpkru) = 0
Quoted 7 lines

And then in the NMI handler:

	u32 pkru;

	if (*this_cpu_ptr(&need_nmi_wpkru)) {
		pkru = rdpku();
		__write_pkru(vcpu->arch.pkru);

This isn't correct, vcpu->arch.pkru holds the guest value, the NMI handler needs to load the host value. I was thinking KVM would stash away the current host value, and the NMI handler would check the saved value against rdpkru().

Quoted 7 lines
	}
	...
	copy_*_user_nmi(... whatever ...);
	...
	if (*this_cpu_ptr(&need_nmi_wpkru))
		__write_pkru(pkru);
Quoted 5 lines

?

I was thinking we could just blow away PKRU without saving/restoring it in the NMI handler, but that might clobber PKRU in the window between need_nmi_wpkru=1 and entering the guest.

Yep. It would also blow away the guest's value if the guest did WRPKU while it was running since KVM would never get a chance to read/save the guest's new value.

Quoted 2 lines

But, the save/restore seems doable especially since we can do it in C and don't have to mess with the NMI stack or anything.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#12 lore raw
Quoted 7 lines · Andy Lutomirski
Quoted 5 lines

The least awful solution would be to have the NMI handler restore the host's PKRU. The NMI handler would need to save/restore the register, a la CR2, but the whole thing could be optimized to run if and only if the NMI lands in the window where the guest's PKRU is loaded.

Or set a flag causing nmi_uaccess_ok() to return false.

Oh, that doesn't sound too bad. The VMENTER/EXIT paths are also essentially a context switch.

Will widening the window where nmi_uaccess_okay()==false anger any of the perf folks? It looks like perf knows how to handle it nicely.

#13 lore raw
Quoted 11 lines · Dave Hansen
Quoted 7 lines · Andy Lutomirski
Quoted 5 lines

The least awful solution would be to have the NMI handler restore the host's PKRU. The NMI handler would need to save/restore the register, a la CR2, but the whole thing could be optimized to run if and only if the NMI lands in the window where the guest's PKRU is loaded.

Or set a flag causing nmi_uaccess_ok() to return false.

Oh, that doesn't sound too bad. The VMENTER/EXIT paths are also essentially a context switch.

I like that idea, too.

The flag might also be useful to fix the issue where the NMI handler activates PEBS after KVM disables it. Jim?

Quoted 2 lines

Will widening the window where nmi_uaccess_okay()==false anger any of the perf folks? It looks like perf knows how to handle it nicely.

#14 lore raw
Quoted 11 lines · Sean Christopherson
Quoted 9 lines · Dave Hansen
Quoted 7 lines · Andy Lutomirski
Quoted 5 lines

The least awful solution would be to have the NMI handler restore the host's PKRU. The NMI handler would need to save/restore the register, a la CR2, but the whole thing could be optimized to run if and only if the NMI lands in the window where the guest's PKRU is loaded.

Or set a flag causing nmi_uaccess_ok() to return false.

Quoted 7 lines
Quoted 2 lines

Oh, that doesn't sound too bad. The VMENTER/EXIT paths are also essentially a context switch.

I like that idea, too.

The flag might also be useful to fix the issue where the NMI handler activates PEBS after KVM disables it. Jim?

The issue is actually that the NMI handler *clears* IA32_PEBS_ENABLE bits after giving out the host value of the MSR to KVM. If we were to block the NMI handler from modifying IA32_PEBS_ENABLE until after the next VM-exit, that could solve this issue. I don't know if it makes sense to piggyback on nmi_uaccess(), though.

Quoted 2 lines
Quoted 2 lines

Will widening the window where nmi_uaccess_okay()==false anger any of the perf folks? It looks like perf knows how to handle it nicely.

#15 lore raw
Quoted 17 lines · Jim Mattson
Quoted 16 lines · Sean Christopherson
Quoted 11 lines · Dave Hansen
Quoted 7 lines · Andy Lutomirski
Quoted 5 lines

The least awful solution would be to have the NMI handler restore the host's PKRU. The NMI handler would need to save/restore the register, a la CR2, but the whole thing could be optimized to run if and only if the NMI lands in the window where the guest's PKRU is loaded.

Or set a flag causing nmi_uaccess_ok() to return false.

Oh, that doesn't sound too bad. The VMENTER/EXIT paths are also essentially a context switch.

I like that idea, too.

Quoted 8 lines
Quoted 2 lines

The flag might also be useful to fix the issue where the NMI handler activates PEBS after KVM disables it. Jim?

The issue is actually that the NMI handler *clears* IA32_PEBS_ENABLE bits after giving out the host value of the MSR to KVM. If we were to block the NMI handler from modifying IA32_PEBS_ENABLE until after the next VM-exit, that could solve this issue. I don't know if it makes sense to piggyback on nmi_uaccess(), though.

I wasn't thinking about using nmi_uaccess_okay(), but rather whatever flag is added so that can KVM can inform the NMI handler that KVM is in the middle of its version of a context switch.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Will widening the window where nmi_uaccess_okay()==false anger any of the perf folks? It looks like perf knows how to handle it nicely.

#16 lore raw
In reply to Paolo Bonzini
Quoted 9 lines · Paolo Bonzini
Quoted 3 lines · Andy Lutomirski

That's nice, but it fails to restore XINUSE[PKRU].  As far as I know, that bit is live, and the only way to restore it to 0 is with XRSTOR(S).

The manual says "It is possible for XINUSE[i] to be 1 even when state component i is in its initial configuration" so this is architecturally valid.  Does the XINUSE optimization matter for PKRU which is a single word?

In Linux with normal userspace, virtually never.

The hardware defaults PKRU to 0x0 which means "no restrictions on any keys". Linux defaults PKRU via 'init_pkru_value' to the most restrictive value. This ensures that new non-zero-pkey-assigned memory is protected by default.

But, that also means PKRU is virtually never in its init state in Linux.
An app would probably need to manipulate PKRU with XRSTOR to get
XINUSE[PKRU]=0.

It would only even *possibly* be useful if running a KVM guest that had PKRU=0x0 (sorry I don't consider things using KVM "normal userspace" :P ).

#17 lore raw
Quoted 18 lines · Dave Hansen
Quoted 9 lines · Paolo Bonzini
Quoted 3 lines · Andy Lutomirski

That's nice, but it fails to restore XINUSE[PKRU].  As far as I know, that bit is live, and the only way to restore it to 0 is with XRSTOR(S).

The manual says "It is possible for XINUSE[i] to be 1 even when state component i is in its initial configuration" so this is architecturally valid.  Does the XINUSE optimization matter for PKRU which is a single word?

In Linux with normal userspace, virtually never.

The hardware defaults PKRU to 0x0 which means "no restrictions on any keys". Linux defaults PKRU via 'init_pkru_value' to the most restrictive value. This ensures that new non-zero-pkey-assigned memory is protected by default.

Quoted 7 lines

But, that also means PKRU is virtually never in its init state in Linux.
An app would probably need to manipulate PKRU with XRSTOR to get
XINUSE[PKRU]=0.

It would only even *possibly* be useful if running a KVM guest that had PKRU=0x0 (sorry I don't consider things using KVM "normal userspace" :P ).

There was at least one report from the rr camp of glibc behaving differently depending on the result of XGETBV(1). It's at least impolite to change the XINUSE register for a guest behind its back.

Admittedly that particular report wasn't about PKRU, and *Linux* guests won't run with XINUSE[PKRU]=0 under normal circumstances, but non-Linux guests could certainly do so.

#18 lore raw
In reply to Andy Lutomirski
Quoted 3 lines · Andy Lutomirski

I don't even want to think about what happens if a perf NMI hits and accesses host user memory while the guest PKRU is live (on VMX -- I think this can't happen on SVM).

What's the relevant difference between SVM and VMX here? I'm missing something.

maintainer
#19 lore raw
Quoted 7 lines · Dave Hansen
Quoted 3 lines · Andy Lutomirski

I don't even want to think about what happens if a perf NMI hits and accesses host user memory while the guest PKRU is live (on VMX -- I think this can't happen on SVM).

What's the relevant difference between SVM and VMX here? I'm missing something.

SVM has the global interrupt flag that blocks NMIs and SMIs, and PKRU is loaded while GIF=0.

Paolo