Skip to contentSkip to conversation

x86: clear RDRAND CPUID bit on AMD family 15h/16h

PATCH Superseded Jan Beulich 3 participants 6 messages Aug 29, 2019

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86: clear RDRAND CPUID bit on AMD family 15h/16h

Inspired by Linux commit c49a0a80137c7ca7d6ced4c812c9e07a949f6f24:

There have been reports of RDRAND issues after resuming from suspend on some AMD family 15h and family 16h systems. This issue stems from a BIOS not performing the proper steps during resume to ensure RDRAND continues to function properly.

Update the CPU initialization to clear the RDRAND CPUID bit for any family 15h and 16h processor that supports RDRAND. If it is known that the family 15h or family 16h system does not have an RDRAND resume issue or that the system will not be placed in suspend, the "cpuid=rdrand" kernel parameter can be used to stop the clearing of the RDRAND CPUID bit.

Note, that clearing the RDRAND CPUID bit does not prevent a processor that normally supports the RDRAND instruction from executing it. So any code that determined the support based on family and model won't #UD.

---
Slightly RFC, in particular because of the change to parse_xen_cpuid():
Alternative approach suggestions are welcome.

docs/misc/xen-command-line.pandoc +4 -0
@@ -488,6 +488,10 @@ The Speculation Control hardware feature
 be ignored, e.g. `no-ibrsb`, at which point Xen won't use them itself, and
 won't offer them to guests.
 
+`rdrand` can be used to override the default disabling of the feature on certain
+AMD systems.  Its negative form can of course also be used to suppress use and
+exposure of the feature.
+
 ### cpuid_mask_cpu
 > `= fam_0f_rev_[cdefg] | fam_10_rev_[bc] | fam_11_rev_b`
 
xen/arch/x86/cpu/amd.c +12 -0
@@ -641,6 +641,18 @@ static void init_amd(struct cpuinfo_x86
 		if (acpi_smi_cmd && (acpi_enable_value | acpi_disable_value))
 			amd_acpi_c1e_quirk = true;
 		break;
+
+	case 0x15: case 0x16:
+		/*
+		 * There are too many Fam15/Fam16 systems where upon resume
+		 * from S3 firmware fails to re-setup properly functioning
+		 * RDRAND.  Clear the feature unless force-enabled on the
+		 * command line.
+		 */
+		if (c == &boot_cpu_data &&
+		    !is_forced_cpu_cap(X86_FEATURE_RDRAND))
+			setup_clear_cpu_cap(X86_FEATURE_RDRAND);
+		break;
 	}
 
 	display_cacheinfo(c);
xen/arch/x86/cpu/common.c +5 -0
@@ -94,6 +94,11 @@ void __init setup_force_cpu_cap(unsigned
 	__set_bit(cap, boot_cpu_data.x86_capability);
 }
 
+bool __init is_forced_cpu_cap(unsigned int cap)
+{
+	return test_bit(cap, forced_caps);
+}
+
 static void default_init(struct cpuinfo_x86 * c)
 {
 	/* Not much we can do here... */
xen/arch/x86/cpuid.c +7 -0
@@ -74,6 +74,13 @@ static int __init parse_xen_cpuid(const
                 setup_clear_cpu_cap(X86_FEATURE_AMD_SSBD);
             }
         }
+        else if ( (val = parse_boolean("rdrand", s, ss)) >= 0 )
+        {
+            if ( !val )
+                setup_clear_cpu_cap(X86_FEATURE_RDRAND);
+            else if ( cpuid_ecx(1) & cpufeat_mask(X86_FEATURE_RDRAND) )
+                setup_force_cpu_cap(X86_FEATURE_RDRAND);
+        }
         else
             rc = -EINVAL;
 
xen/include/asm-x86/processor.h +1 -0
@@ -166,6 +166,7 @@ extern const struct x86_cpu_id *x86_matc
 extern void identify_cpu(struct cpuinfo_x86 *);
 extern void setup_clear_cpu_cap(unsigned int);
 extern void setup_force_cpu_cap(unsigned int);
+extern bool is_forced_cpu_cap(unsigned int);
 extern void print_cpu_info(unsigned int cpu);
 extern unsigned int init_intel_cacheinfo(struct cpuinfo_x86 *c);
 

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#1 lore raw
Quoted 13 lines · Jan Beulich

Inspired by Linux commit c49a0a80137c7ca7d6ced4c812c9e07a949f6f24:

There have been reports of RDRAND issues after resuming from suspend on some AMD family 15h and family 16h systems. This issue stems from a BIOS not performing the proper steps during resume to ensure RDRAND continues to function properly.

Update the CPU initialization to clear the RDRAND CPUID bit for any family 15h and 16h processor that supports RDRAND. If it is known that the family 15h or family 16h system does not have an RDRAND resume issue or that the system will not be placed in suspend, the "cpuid=rdrand" kernel parameter can be used to stop the clearing of the RDRAND CPUID bit.

Quoted 8 lines

Note, that clearing the RDRAND CPUID bit does not prevent a processor that normally supports the RDRAND instruction from executing it. So any code that determined the support based on family and model won't #UD.

---
Slightly RFC, in particular because of the change to parse_xen_cpuid():
Alternative approach suggestions are welcome.

This issue was on my radar, but I hadn't got around to starting a patch yet.

I was wondering if we could perhaps default it to whether S3 was available, but looking at the code which prints "ACPI sleep modes: S3", it doesn't appear to be related to any real ACPI information.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#2 lore raw
Quoted 14 lines · Andrew Cooper
Quoted 13 lines · Jan Beulich

Inspired by Linux commit c49a0a80137c7ca7d6ced4c812c9e07a949f6f24:

There have been reports of RDRAND issues after resuming from suspend on some AMD family 15h and family 16h systems. This issue stems from a BIOS not performing the proper steps during resume to ensure RDRAND continues to function properly.

Update the CPU initialization to clear the RDRAND CPUID bit for any family 15h and 16h processor that supports RDRAND. If it is known that the family 15h or family 16h system does not have an RDRAND resume issue or that the system will not be placed in suspend, the "cpuid=rdrand" kernel parameter can be used to stop the clearing of the RDRAND CPUID bit.

Quoted 3 lines
Quoted 3 lines

Note, that clearing the RDRAND CPUID bit does not prevent a processor that normally supports the RDRAND instruction from executing it. So any code that determined the support based on family and model won't #UD.

Yeah, but it will no longer see random numbers after resume.

Quoted 11 lines
Quoted 5 lines

---
Slightly RFC, in particular because of the change to parse_xen_cpuid():
Alternative approach suggestions are welcome.

This issue was on my radar, but I hadn't got around to starting a patch yet.

I was wondering if we could perhaps default it to whether S3 was available, but looking at the code which prints "ACPI sleep modes: S3", it doesn't appear to be related to any real ACPI information.

Wouldn't it make more sense to inhibit suspend/resume instead?

I'm quite sure a machine running Xen is very rarely put to S3.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw
Quoted 15 lines · Juergen Gross
Quoted 14 lines · Andrew Cooper
Quoted 13 lines · Jan Beulich

Inspired by Linux commit c49a0a80137c7ca7d6ced4c812c9e07a949f6f24:

There have been reports of RDRAND issues after resuming from suspend on some AMD family 15h and family 16h systems. This issue stems from a BIOS not performing the proper steps during resume to ensure RDRAND continues to function properly.

Update the CPU initialization to clear the RDRAND CPUID bit for any family 15h and 16h processor that supports RDRAND. If it is known that the family 15h or family 16h system does not have an RDRAND resume issue or that the system will not be placed in suspend, the "cpuid=rdrand" kernel parameter can be used to stop the clearing of the RDRAND CPUID bit.

Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

Note, that clearing the RDRAND CPUID bit does not prevent a processor that normally supports the RDRAND instruction from executing it. So any code that determined the support based on family and model won't #UD.

Yeah, but it will no longer see random numbers after resume.

That's the implication; note that I've taken the text from the Linux commit.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

---
Slightly RFC, in particular because of the change to parse_xen_cpuid():
Alternative approach suggestions are welcome.

Quoted 7 lines
Quoted 5 lines

This issue was on my radar, but I hadn't got around to starting a patch yet.

I was wondering if we could perhaps default it to whether S3 was available, but looking at the code which prints "ACPI sleep modes: S3", it doesn't appear to be related to any real ACPI information.

Wouldn't it make more sense to inhibit suspend/resume instead?

That's an alternative option. But if anything I'd see us providing both, not the one you suggest instead of what the patch here does.

Quoted 1 line

I'm quite sure a machine running Xen is very rarely put to S3.

I'm not at all sure about this.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw
Quoted 36 lines · Jan Beulich
Quoted 15 lines · Juergen Gross
Quoted 14 lines · Andrew Cooper
Quoted 13 lines · Jan Beulich

Inspired by Linux commit c49a0a80137c7ca7d6ced4c812c9e07a949f6f24:

There have been reports of RDRAND issues after resuming from suspend on some AMD family 15h and family 16h systems. This issue stems from a BIOS not performing the proper steps during resume to ensure RDRAND continues to function properly.

Update the CPU initialization to clear the RDRAND CPUID bit for any family 15h and 16h processor that supports RDRAND. If it is known that the family 15h or family 16h system does not have an RDRAND resume issue or that the system will not be placed in suspend, the "cpuid=rdrand" kernel parameter can be used to stop the clearing of the RDRAND CPUID bit.

Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

Note, that clearing the RDRAND CPUID bit does not prevent a processor that normally supports the RDRAND instruction from executing it. So any code that determined the support based on family and model won't #UD.

Yeah, but it will no longer see random numbers after resume.

That's the implication; note that I've taken the text from the Linux commit.

Quoted 11 lines
Quoted 10 lines
Quoted 4 lines

---
Slightly RFC, in particular because of the change to parse_xen_cpuid():
Alternative approach suggestions are welcome.

This issue was on my radar, but I hadn't got around to starting a patch yet.

I was wondering if we could perhaps default it to whether S3 was available, but looking at the code which prints "ACPI sleep modes: S3", it doesn't appear to be related to any real ACPI information.

Quoted 8 lines
Quoted 1 line

Wouldn't it make more sense to inhibit suspend/resume instead?

That's an alternative option. But if anything I'd see us providing both, not the one you suggest instead of what the patch here does.

Quoted 1 line

I'm quite sure a machine running Xen is very rarely put to S3.

I'm not at all sure about this.

Suspend/resume in Xen was broken for more than 3 months in the late 4.12 development phase and nobody noticed it. Only when I started my core scheduling series I came across the issue.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw
Quoted 55 lines · Juergen Gross
Quoted 54 lines · Jan Beulich
Quoted 22 lines · Juergen Gross
Quoted 21 lines · Andrew Cooper
Quoted 20 lines · Jan Beulich

Inspired by Linux commit c49a0a80137c7ca7d6ced4c812c9e07a949f6f24:

      There have been reports of RDRAND issues after resuming from suspend on
      some AMD family 15h and family 16h systems. This issue stems from a BIOS
      not performing the proper steps during resume to ensure RDRAND continues
      to function properly.

      Update the CPU initialization to clear the RDRAND CPUID bit for any family
      15h and 16h processor that supports RDRAND. If it is known that the family
      15h or family 16h system does not have an RDRAND resume issue or that the
      system will not be placed in suspend, the "cpuid=rdrand" kernel parameter
      can be used to stop the clearing of the RDRAND CPUID bit.

Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

      Note, that clearing the RDRAND CPUID bit does not prevent a processor
      that normally supports the RDRAND instruction from executing it. So any
      code that determined the support based on family and model won't #UD.

Yeah, but it will no longer see random numbers after resume.

That's the implication; note that I've taken the text from the Linux commit.

Quoted 9 lines
Quoted 9 lines
Quoted 5 lines

---
Slightly RFC, in particular because of the change to parse_xen_cpuid():
Alternative approach suggestions are welcome.

This issue was on my radar, but I hadn't got around to starting a patch yet.

Quoted 6 lines
Quoted 4 lines

I was wondering if we could perhaps default it to whether S3 was available, but looking at the code which prints "ACPI sleep modes:
S3",
it doesn't appear to be related to any real ACPI information.

Wouldn't it make more sense to inhibit suspend/resume instead?

That's an alternative option. But if anything I'd see us providing both, not the one you suggest instead of what the patch here does.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

I'm quite sure a machine running Xen is very rarely put to S3.

I'm not at all sure about this.

Suspend/resume in Xen was broken for more than 3 months in the late 4.12 development phase and nobody noticed it. Only when I started my core scheduling series I came across the issue.

OpenXT and Qubes will come after you with sticks...

For traditional server scenarios, S3 is rare to non-existent, but Xen also runs in a number of well known cases where S3 is very important. The reason S3 breaks frequently (see also Jan's IRQ adjustments broke it for a while in staging), is because none of the developers who work in these areas use it.

I now have a test scenario using a CoffeeLake system sat on my desk, but it only gets testing when I'm poking the low level code, or when I suspect a change might have an impact.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel