Skip to contentSkip to conversation

x86/fpu/xstate: Add more diagnostic information on inconsistent xstate sizes

RFC PATCH In review Fenghua Yu 3 participants 7 messages Apr 5, 2023 – Apr 12, 2023 x86

lore
Share & tools
lore

Fenghua Yu

author
#0 lore raw

RFC PATCH 1/1 x86/fpu/xstate: Add more diagnostic information on inconsistent xstate sizes

A warning is emitted when xstate sizes are found inconsistent. But the warning message doesn't show enough information to diagnose the issue.

Provide more detailed xstate size information to help debug the issue. As a hypothetical example, on a platform that may report incorrect xstate size in CPUID.0xd.1:EBX, the diagnostic information after the warning will show:

[    0.000000] x86/fpu: max_features=0x407
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: xstate_offset[10]: 832, xstate_sizes[10]:    8
[    0.000000] x86/fpu: total size: 840 bytes
[    0.000000] x86/fpu: XCR0=0x7, IA32_XSS=0x400
[    0.000000] x86/fpu: kernel_size from CPUID.0xd.0x1:EBX: 576 bytes

XCR0 | IA32_XSS is 0x407 which is consistent with max_features. CPUID.0xd.0x1:EBX should report the size of the xsave area containing xstate components corresponding to bits set in XCR0 | IA32_XSS. But it only reports 576 bytes of xsave area which doesn't count xstate sizes for AVX (offset 2 and 256 bytes) and PASID (offset 10 and 8 bytes). This confirms that the platform reports xstate size incorrectly through the CPUID bits.

---
 arch/x86/kernel/fpu/xstate.c | 33 +++++++++++++++++++++++++++++++--
 1 file changed, 31 insertions(+), 2 deletions(-)
arch/x86/kernel/fpu/xstate.c +31 -2
index 0bab497c9436..5f27fcdc6c90 100644
@@ -602,8 +602,37 @@ static bool __init paranoid_xstate_size_valid(unsigned int kernel_size)
 		}
 	}
 	size = xstate_calculate_size(fpu_kernel_cfg.max_features, compacted);
-	XSTATE_WARN_ON(size != kernel_size,
-		       "size %u != kernel_size %u\n", size, kernel_size);
+	if (size != kernel_size) {
+		u64 xcr0, ia32_xss;
+
+		XSTATE_WARN_ON(1, "size %u != kernel_size %u\n",
+			       size, kernel_size);
+
+		/* Show more information to help diagnose the size issue. */
+		pr_info("x86/fpu: max_features=0x%llx\n",
+			fpu_kernel_cfg.max_features);
+		print_xstate_offset_size();
+		pr_info("x86/fpu: total size: %u bytes\n", size);
+		xcr0 = xgetbv(XCR_XFEATURE_ENABLED_MASK);
+		if (compacted) {
+			rdmsrl(MSR_IA32_XSS, ia32_xss);
+			pr_info("x86/fpu: XCR0=0x%llx, IA32_XSS=0x%llx\n",
+				xcr0, ia32_xss);
+		} else {
+			pr_info("x86/fpu: XCR0=0x%llx\n", xcr0);
+		}
+		/*
+		 * In compact case, CPUID.0xd.0x1:EBX reports the size of
+		 * the XSAVE size containing all the state components
+		 * corresponding to bits set in XCR0 | IA32_XSS.
+		 *
+		 * Otherwise, CPUID.0xd.0x0:EBX reports the size of an XSAVE
+		 * area containing all the *user* state components
+		 * corresponding to bits set in XCR0.
+		 */
+		pr_info("x86/fpu: kernel_size from CPUID.0xd.0x%x:EBX: %u bytes\n",
+			compacted ? 1 : 0, kernel_size);
+	}
 	return size == kernel_size;
 }
 
Signature · 1 line
2.37.1
#1 lore raw
Quoted 44 lines · Fenghua Yu

A warning is emitted when xstate sizes are found inconsistent. But the warning message doesn't show enough information to diagnose the issue.

Provide more detailed xstate size information to help debug the issue. As a hypothetical example, on a platform that may report incorrect xstate size in CPUID.0xd.1:EBX, the diagnostic information after the warning will show:

[    0.000000] x86/fpu: max_features=0x407
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: xstate_offset[10]: 832, xstate_sizes[10]:    8
[    0.000000] x86/fpu: total size: 840 bytes
[    0.000000] x86/fpu: XCR0=0x7, IA32_XSS=0x400
[    0.000000] x86/fpu: kernel_size from CPUID.0xd.0x1:EBX: 576 bytes

XCR0 | IA32_XSS is 0x407 which is consistent with max_features. CPUID.0xd.0x1:EBX should report the size of the xsave area containing xstate components corresponding to bits set in XCR0 | IA32_XSS. But it only reports 576 bytes of xsave area which doesn't count xstate sizes for AVX (offset 2 and 256 bytes) and PASID (offset 10 and 8 bytes). This confirms that the platform reports xstate size incorrectly through the CPUID bits.

---
  arch/x86/kernel/fpu/xstate.c | 33 +++++++++++++++++++++++++++++++--
  1 file changed, 31 insertions(+), 2 deletions(-)
arch/x86/kernel/fpu/xstate.c +4 -2
index 0bab497c9436..5f27fcdc6c90 100644
@@ -602,8 +602,37 @@ static bool __init paranoid_xstate_size_valid(unsigned int kernel_size)
  		}
  	}
  	size = xstate_calculate_size(fpu_kernel_cfg.max_features, compacted);
-	XSTATE_WARN_ON(size != kernel_size,
-		       "size %u != kernel_size %u\n", size, kernel_size);
+	if (size != kernel_size) {
+		u64 xcr0, ia32_xss;
+
+		XSTATE_WARN_ON(1, "size %u != kernel_size %u\n",

arch/x86/kernel/fpu/xstate.c static bool __init paranoid_xstate_size_valid(unsigned int kernel_size) · line 618

Quoted 10 lines
arch/x86/kernel/fpu/xstate.c +10 -0
+			       size, kernel_size);
+
+		/* Show more information to help diagnose the size issue. */
+		pr_info("x86/fpu: max_features=0x%llx\n",
+			fpu_kernel_cfg.max_features);
+		print_xstate_offset_size();
+		pr_info("x86/fpu: total size: %u bytes\n", size);
+		xcr0 = xgetbv(XCR_XFEATURE_ENABLED_MASK);
+		if (compacted) {
+			rdmsrl(MSR_IA32_XSS, ia32_xss);

This shouldn't be directly read here because of the LBR state component.

See the function comment:

* Independent XSAVE features allocate their own buffers and are not
* covered by these checks. Only the size of the buffer for task->fpu
* is checked here.

But, isn't that max_features bitmask pretty much about it?

Quoted 6 lines
arch/x86/kernel/fpu/xstate.c +6 -0
+			pr_info("x86/fpu: XCR0=0x%llx, IA32_XSS=0x%llx\n",
+				xcr0, ia32_xss);
+		} else {
+			pr_info("x86/fpu: XCR0=0x%llx\n", xcr0);
+		}
+		/*

arch/x86/kernel/fpu/xstate.c static bool __init paranoid_xstate_size_valid(unsigned int kernel_size) · line 634

Quoted 10 lines
arch/x86/kernel/fpu/xstate.c +10 -0
+		 * In compact case, CPUID.0xd.0x1:EBX reports the size of
+		 * the XSAVE size containing all the state components
+		 * corresponding to bits set in XCR0 | IA32_XSS.
+		 *
+		 * Otherwise, CPUID.0xd.0x0:EBX reports the size of an XSAVE
+		 * area containing all the *user* state components
+		 * corresponding to bits set in XCR0.
+		 */
+		pr_info("x86/fpu: kernel_size from CPUID.0xd.0x%x:EBX: %u bytes\n",
+			compacted ? 1 : 0, kernel_size);

Include Thiago who asked this.

Thanks,
Chang

author
#2 lore raw

Hi, Chang,

Quoted 59 lines · Chang S. Bae
Quoted 45 lines · Fenghua Yu

A warning is emitted when xstate sizes are found inconsistent. But the warning message doesn't show enough information to diagnose the issue.

Provide more detailed xstate size information to help debug the issue. As a hypothetical example, on a platform that may report incorrect xstate size in CPUID.0xd.1:EBX, the diagnostic information after the warning will show:
[    0.000000] x86/fpu: max_features=0x407
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256 [    0.000000] x86/fpu: xstate_offset[10]: 832, xstate_sizes[10]:    8 [    0.000000] x86/fpu: total size: 840 bytes
[    0.000000] x86/fpu: XCR0=0x7, IA32_XSS=0x400
[    0.000000] x86/fpu: kernel_size from CPUID.0xd.0x1:EBX: 576 bytes

XCR0 | IA32_XSS is 0x407 which is consistent with max_features. CPUID.0xd.0x1:EBX should report the size of the xsave area containing xstate components corresponding to bits set in XCR0 | IA32_XSS. But it only reports 576 bytes of xsave area which doesn't count xstate sizes for AVX (offset 2 and 256 bytes) and PASID (offset 10 and 8 bytes). This confirms that the platform reports xstate size incorrectly through the CPUID bits.

---
  arch/x86/kernel/fpu/xstate.c | 33 +++++++++++++++++++++++++++++++--

  1 file changed, 31 insertions(+), 2 deletions(-)
arch/x86/kernel/fpu/xstate.c +4 -2
index 0bab497c9436..5f27fcdc6c90 100644
@@ -602,8 +602,37 @@ static bool __init 
 paranoid_xstate_size_valid(unsigned int kernel_size)
           }
       }
       size = xstate_calculate_size(fpu_kernel_cfg.max_features, compacted);
-    XSTATE_WARN_ON(size != kernel_size,
-               "size %u != kernel_size %u\n", size, kernel_size);
+    if (size != kernel_size) {
+        u64 xcr0, ia32_xss;
+
+        XSTATE_WARN_ON(1, "size %u != kernel_size %u\n",

arch/x86/kernel/fpu/xstate.c static bool __init · line 619

Quoted 10 lines
arch/x86/kernel/fpu/xstate.c +10 -0
+                   size, kernel_size);
+
+        /* Show more information to help diagnose the size issue. */
+        pr_info("x86/fpu: max_features=0x%llx\n",
+            fpu_kernel_cfg.max_features);
+        print_xstate_offset_size();
+        pr_info("x86/fpu: total size: %u bytes\n", size);
+        xcr0 = xgetbv(XCR_XFEATURE_ENABLED_MASK);
+        if (compacted) {
+            rdmsrl(MSR_IA32_XSS, ia32_xss);

This shouldn't be directly read here because of the LBR state component.

Quoted 7 lines

See the function comment:

 * Independent XSAVE features allocate their own buffers and are not
 * covered by these checks. Only the size of the buffer for task->fpu
 * is checked here.

But, isn't that max_features bitmask pretty much about it?

How about getting IA32_XSS from xfeatures_mask_supervisor()? That's how to get kernel_size by setting IA32_XSS without independent features in get_xsave_compacted_size().

Thanks.

-Fenghua

#3 lore raw
Quoted 33 lines · Fenghua Yu
Quoted 32 lines · Chang S. Bae
Quoted 16 lines · Fenghua Yu
arch/x86/kernel/fpu/xstate.c +4 -2
index 0bab497c9436..5f27fcdc6c90 100644
@@ -602,8 +602,37 @@ static bool __init 
 paranoid_xstate_size_valid(unsigned int kernel_size)
           }
       }
       size = xstate_calculate_size(fpu_kernel_cfg.max_features, compacted);
-    XSTATE_WARN_ON(size != kernel_size,
-               "size %u != kernel_size %u\n", size, kernel_size);
+    if (size != kernel_size) {
+        u64 xcr0, ia32_xss;
+
+        XSTATE_WARN_ON(1, "size %u != kernel_size %u\n",

arch/x86/kernel/fpu/xstate.c static bool __init · line 619

Quoted 10 lines
arch/x86/kernel/fpu/xstate.c +10 -0
+                   size, kernel_size);
+
+        /* Show more information to help diagnose the size issue. */
+        pr_info("x86/fpu: max_features=0x%llx\n",
+            fpu_kernel_cfg.max_features);
+        print_xstate_offset_size();
+        pr_info("x86/fpu: total size: %u bytes\n", size);
+        xcr0 = xgetbv(XCR_XFEATURE_ENABLED_MASK);
+        if (compacted) {
+            rdmsrl(MSR_IA32_XSS, ia32_xss);

This shouldn't be directly read here because of the LBR state component.

See the function comment:

Quoted 7 lines
Quoted 5 lines

  * Independent XSAVE features allocate their own buffers and are not
  * covered by these checks. Only the size of the buffer for task->fpu
  * is checked here.

But, isn't that max_features bitmask pretty much about it?

How about getting IA32_XSS from xfeatures_mask_supervisor()? That's how to get kernel_size by setting IA32_XSS without independent features in get_xsave_compacted_size()

I think what it tests here is comparing the sizes between the kernel code and microcode calculations on the same input, which is the max_features bitmask.

We know that the kernel code calculates the size based on it and also takes it to write down there -- XCR0 and IA32_XSS. Then, showing that bitmask looks to be enough I thought, no?

I still expect some acknowledgment of what is coded here for the kernel calculation details.

Thanks,
Chang

author
#4 lore raw

Hi, Chang,

Quoted 38 lines · Chang S. Bae
Quoted 37 lines · Fenghua Yu
Quoted 36 lines · Chang S. Bae
Quoted 16 lines · Fenghua Yu
arch/x86/kernel/fpu/xstate.c +4 -2
index 0bab497c9436..5f27fcdc6c90 100644
@@ -602,8 +602,37 @@ static bool __init 
 paranoid_xstate_size_valid(unsigned int kernel_size)
           }
       }
       size = xstate_calculate_size(fpu_kernel_cfg.max_features, compacted);
-    XSTATE_WARN_ON(size != kernel_size,
-               "size %u != kernel_size %u\n", size, kernel_size);
+    if (size != kernel_size) {
+        u64 xcr0, ia32_xss;
+
+        XSTATE_WARN_ON(1, "size %u != kernel_size %u\n",

arch/x86/kernel/fpu/xstate.c static bool __init · line 619

Quoted 10 lines
arch/x86/kernel/fpu/xstate.c +10 -0
+                   size, kernel_size);
+
+        /* Show more information to help diagnose the size issue. */
+        pr_info("x86/fpu: max_features=0x%llx\n",
+            fpu_kernel_cfg.max_features);
+        print_xstate_offset_size();
+        pr_info("x86/fpu: total size: %u bytes\n", size);
+        xcr0 = xgetbv(XCR_XFEATURE_ENABLED_MASK);
+        if (compacted) {
+            rdmsrl(MSR_IA32_XSS, ia32_xss);

This shouldn't be directly read here because of the LBR state component.

See the function comment:

  * Independent XSAVE features allocate their own buffers and are not
  * covered by these checks. Only the size of the buffer for task->fpu
  * is checked here.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

But, isn't that max_features bitmask pretty much about it?

How about getting IA32_XSS from xfeatures_mask_supervisor()? That's how to get kernel_size by setting IA32_XSS without independent features in get_xsave_compacted_size()

I think what it tests here is comparing the sizes between the kernel code and microcode calculations on the same input, which is the max_features bitmask.

We know that the kernel code calculates the size based on it and also takes it to write down there -- XCR0 and IA32_XSS. Then, showing that bitmask looks to be enough I thought, no?

First of all, max_features is shown already.

Kernel_size from CPUID.0xd.0x1:EBX takes XCR0 | IA32_XSS as input. Platform may take wrong XCR0 or IA32_XSS and get wrong kernel_size. The purpose of this patch is to provide more debug info to help debug platform/kernel issue. So instead of a whole max_features, xgetbv() to get XCR0 and xfeatures_mask_supervisor() to get IA32_XSS provides more debug info in case platform may have issue in XCR0 or IA32_XSS.

In other words, splitting max_features into XCR0 and IA32_XSS and showing them individually provide more useful debug info than one single max_features value.

Does it make sense?

Quoted 2 lines

I still expect some acknowledgment of what is coded here for the kernel calculation details.

The kernel calculation is shown in
+ print_xstate_offset_size();
+ pr_info("x86/fpu: total size: %u bytes\n", size);

Isn't that detailed enough to show offset and size of each xstate and sum of sizes?

After that,
+ pr_info("x86/fpu: kernel_size from CPUID.0xd.0x%x:EBX: %u bytes\n",
+ compacted ? 1 : 0, kernel_size);
shows how kernel_size is calculated from CPUID?

Using the above debug info, a real platform CPUID issue is shown clearly.

What other details are needed?

Thanks.

-Fenghua

#5 lore raw
Quoted 2 lines · Fenghua Yu

First of all, max_features is shown already.

Yes.

Quoted 5 lines

Kernel_size from CPUID.0xd.0x1:EBX takes XCR0 | IA32_XSS as input. Platform may take wrong XCR0 or IA32_XSS and get wrong kernel_size. The > purpose of this patch is to provide more debug info to help debug platform/kernel issue. So instead of a whole max_features, xgetbv() to get XCR0 and xfeatures_mask_supervisor() to get IA32_XSS provides more debug info in case platform may have issue in XCR0 or IA32_XSS.

In other words, splitting max_features into XCR0 and IA32_XSS and showing them individually provide more useful debug info than one single max_features value. > Does it make sense?

Hmm, I don't get it. I don't think whether the microcode takes those register values wrong or miscalculates the size does matter here.

print_xstate_offset_size() or something can decode the mask and readily shows off how it was calculated here. Then, probably that's it.

Quoted 8 lines
Quoted 1 line

I still expect some acknowledgment of what is coded here for the kernel calculation details.

The kernel calculation is shown in
+        print_xstate_offset_size();
+        pr_info("x86/fpu: total size: %u bytes\n", size);

Isn't that detailed enough to show offset and size of each xstate and sum of sizes?

Quoted 8 lines

After that,
+    pr_info("x86/fpu: kernel_size from CPUID.0xd.0x%x:EBX: %u bytes\n", +               compacted ? 1 : 0, kernel_size);
shows how kernel_size is calculated from CPUID?

Using the above debug info, a real platform CPUID issue is shown clearly.

What other details are needed?

I recall it was also asked to show which features are off or mismatched as compared to the CPU calculation. I'm not so sure about it.

Thanks,
Chang

#6 lore raw
In reply to Fenghua Yu
Quoted 5 lines · Fenghua Yu

In other words, splitting max_features into XCR0 and IA32_XSS and showing them individually provide more useful debug info than one single max_features value.

Does it make sense?

Not to me.

Quoted 10 lines
Quoted 2 lines

I still expect some acknowledgment of what is coded here for the kernel calculation details.

The kernel calculation is shown in
+        print_xstate_offset_size();
+        pr_info("x86/fpu: total size: %u bytes\n", size);

Isn't that detailed enough to show offset and size of each xstate and sum of sizes?

Quoted 8 lines

After that,
+    pr_info("x86/fpu: kernel_size from CPUID.0xd.0x%x:EBX: %u bytes\n", +               compacted ? 1 : 0, kernel_size);
shows how kernel_size is calculated from CPUID?

Using the above debug info, a real platform CPUID issue is shown clearly.

What other details are needed?

I was kinda hoping this would be a simple, non-controversial patch that would get us better debugging info the next time that the microcode or a bad VMM screws up. This patch isn't turning out to be as simple as I hoped.

I was wrong. Let's just drop this.