Skip to contentSkip to conversation

x86: add CPU flags supported inside libvirt

PATCH In review Denis V. Lunev 4 participants 8 messages Jul 18, 2019 – Aug 14, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86: add CPU flags supported inside libvirt

There are the following flags available in libvirt inside cpu_map.xm

    <feature name='cvt16'>
      <cpuid function='0x80000001' ecx='0x00040000'/>
    </feature>
    <feature name='cmt'> <!-- cqm -->
      <cpuid eax_in='0x07' ecx_in='0x00' ebx='0x00001000'/>
    </feature>

We have faced the problem that QEMU does not start once these flags are present in the domain.xml.

This patch just adds proper names into the map.

CC Paolo Bonzini <[email protected]>
CC Richard Henderson <[email protected]>
CC Eduardo Habkost <[email protected]>
CC Nikolay Shirokovskiy <[email protected]>
CC Peter Krempa <[email protected]>
CC Daniel P. Berrangé <[email protected]>
---
 target/i386/cpu.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
target/i386/cpu.c +2 -2
index 805ce95247..88ba4dad47 100644
@@ -870,7 +870,7 @@ static FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
             "lahf-lm", "cmp-legacy", "svm", "extapic",
             "cr8legacy", "abm", "sse4a", "misalignsse",
             "3dnowprefetch", "osvw", "ibs", "xop",
-            "skinit", "wdt", NULL, "lwp",
+            "skinit", "wdt", "cvt16", "lwp",
             "fma4", "tce", NULL, "nodeid-msr",
             NULL, "tbm", "topoext", "perfctr-core",
             "perfctr-nb", NULL, NULL, NULL,
@@ -1044,7 +1044,7 @@ static FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
             "fsgsbase", "tsc-adjust", NULL, "bmi1",
             "hle", "avx2", NULL, "smep",
             "bmi2", "erms", "invpcid", "rtm",
-            NULL, NULL, "mpx", NULL,
+            "cmt", NULL, "mpx", NULL,
             "avx512f", "avx512dq", "rdseed", "adx",
             "smap", "avx512ifma", "pcommit", "clflushopt",
             "clwb", "intel-pt", "avx512pf", "avx512er",
Signature · 1 line
2.17.1
#1 lore raw
Quoted 38 lines · Denis V. Lunev

There are the following flags available in libvirt inside cpu_map.xm

    <feature name='cvt16'>
      <cpuid function='0x80000001' ecx='0x00040000'/>
    </feature>
    <feature name='cmt'> <!-- cqm -->
      <cpuid eax_in='0x07' ecx_in='0x00' ebx='0x00001000'/>
    </feature>

We have faced the problem that QEMU does not start once these flags are present in the domain.xml.

This patch just adds proper names into the map.

CC Paolo Bonzini <[email protected]>
CC Richard Henderson <[email protected]>
CC Eduardo Habkost <[email protected]>
CC Nikolay Shirokovskiy <[email protected]>
CC Peter Krempa <[email protected]>
CC Daniel P. Berrangé <[email protected]>
---
 target/i386/cpu.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
target/i386/cpu.c +1 -1
index 805ce95247..88ba4dad47 100644
@@ -870,7 +870,7 @@ static FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
             "lahf-lm", "cmp-legacy", "svm", "extapic",
             "cr8legacy", "abm", "sse4a", "misalignsse",
             "3dnowprefetch", "osvw", "ibs", "xop",
-            "skinit", "wdt", NULL, "lwp",
+            "skinit", "wdt", "cvt16", "lwp",
             "fma4", "tce", NULL, "nodeid-msr",
             NULL, "tbm", "topoext", "perfctr-core",
             "perfctr-nb", NULL, NULL, NULL,

target/i386/cpu.c static FeatureWordInfo feature_word_info[FEATURE_WORDS] = { · line 1050

Quoted 8 lines
target/i386/cpu.c +1 -1
@@ -1044,7 +1044,7 @@ static FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
             "fsgsbase", "tsc-adjust", NULL, "bmi1",
             "hle", "avx2", NULL, "smep",
             "bmi2", "erms", "invpcid", "rtm",
-            NULL, NULL, "mpx", NULL,
+            "cmt", NULL, "mpx", NULL,
             "avx512f", "avx512dq", "rdseed", "adx",
             "smap", "avx512ifma", "pcommit", "clflushopt",
             "clwb", "intel-pt", "avx512pf", "avx512er",

Oops, nice catch! I've queued the patch for 4.1.

Paolo

#2 lore raw
Quoted 49 lines · Paolo Bonzini
Quoted 38 lines · Denis V. Lunev

There are the following flags available in libvirt inside cpu_map.xm

    <feature name='cvt16'>
      <cpuid function='0x80000001' ecx='0x00040000'/>
    </feature>
    <feature name='cmt'> <!-- cqm -->
      <cpuid eax_in='0x07' ecx_in='0x00' ebx='0x00001000'/>
    </feature>

We have faced the problem that QEMU does not start once these flags are present in the domain.xml.

This patch just adds proper names into the map.

CC Paolo Bonzini <[email protected]>
CC Richard Henderson <[email protected]>
CC Eduardo Habkost <[email protected]>
CC Nikolay Shirokovskiy <[email protected]>
CC Peter Krempa <[email protected]>
CC Daniel P. Berrangé <[email protected]>
---
 target/i386/cpu.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
target/i386/cpu.c +1 -1
index 805ce95247..88ba4dad47 100644
@@ -870,7 +870,7 @@ static FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
             "lahf-lm", "cmp-legacy", "svm", "extapic",
             "cr8legacy", "abm", "sse4a", "misalignsse",
             "3dnowprefetch", "osvw", "ibs", "xop",
-            "skinit", "wdt", NULL, "lwp",
+            "skinit", "wdt", "cvt16", "lwp",
             "fma4", "tce", NULL, "nodeid-msr",
             NULL, "tbm", "topoext", "perfctr-core",
             "perfctr-nb", NULL, NULL, NULL,

target/i386/cpu.c static FeatureWordInfo feature_word_info[FEATURE_WORDS] = { · line 1050

Quoted 8 lines
target/i386/cpu.c +1 -1
@@ -1044,7 +1044,7 @@ static FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
             "fsgsbase", "tsc-adjust", NULL, "bmi1",
             "hle", "avx2", NULL, "smep",
             "bmi2", "erms", "invpcid", "rtm",
-            NULL, NULL, "mpx", NULL,
+            "cmt", NULL, "mpx", NULL,
             "avx512f", "avx512dq", "rdseed", "adx",
             "smap", "avx512ifma", "pcommit", "clflushopt",
             "clwb", "intel-pt", "avx512pf", "avx512er",

Oops, nice catch! I've queued the patch for 4.1.

Quoted 1 line

Paolo

I have written small script to find differences between CPU features in QEMU and libvirt.

#!/bin/bash
LIST=awk "/name/{split(\\\$2, arr, \"'\"); print arr[2]}" x86_features.xml for feat in $LIST; do
    var=grep \"$feat\" target/i386/cpu.c
    if [ -z "$var" ]; then
        echo $feat
    fi
done

There are the following list of features present in libvirt and missed in QEMU:

osxsave - removed in qemu in f1a23522b03
ospke - removed in qemu in 9ccb9784b57
pconfig - removed in qemu in 712f807e196.
mbm_total
mbm_local

Two last features are described as follows:
  <!-- cpuid level 0x0000000f:1 (edx) --> <feature name='mbm_total'>
    <cpuid eax_in='0x0f' ecx_in='0x01' edx='0x00000002'/> </feature>
  <feature name='mbm_local'>
    <cpuid eax_in='0x0f' ecx_in='0x01' edx='0x00000004'/> </feature>
This leaf is not supported in QEMU at all.

According to Intel 64 and IA-32 Architecture Developer manual vol. 2a these bits are used for
EDX Bit 00: Supports L3 occupancy monitoring if 1. Bit 01: Supports L3 Total Bandwidth monitoring if 1. Bit 02: Supports L3 Local Bandwidth monitoring if 1. Bits 31 - 03: Reserved.

Thus technically these 5 bits are able to produce problems for QEMU if they will be found in domain.xml

Den

#3 lore raw
In reply to Denis V. Lunev
Quoted 3 lines · Denis V. Lunev

There are the following flags available in libvirt inside cpu_map.xm

    <feature name='cvt16'>
      <cpuid function='0x80000001' ecx='0x00040000'/>

This is bit 18...

Quoted 25 lines
    </feature>
    <feature name='cmt'> <!-- cqm -->
      <cpuid eax_in='0x07' ecx_in='0x00' ebx='0x00001000'/>
    </feature>

We have faced the problem that QEMU does not start once these flags are present in the domain.xml.

This patch just adds proper names into the map.

CC Paolo Bonzini <[email protected]>
CC Richard Henderson <[email protected]>
CC Eduardo Habkost <[email protected]>
CC Nikolay Shirokovskiy <[email protected]>
CC Peter Krempa <[email protected]>
CC Daniel P. Berrangé <[email protected]>
---
 target/i386/cpu.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
target/i386/cpu.c +0 -0
index 805ce95247..88ba4dad47 100644

target/i386/cpu.c static FeatureWordInfo feature_word_info[FEATURE_WORDS] = { · line 873

Quoted 5 lines
target/i386/cpu.c +1 -1
@@ -870,7 +870,7 @@ static FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
             "lahf-lm", "cmp-legacy", "svm", "extapic",
             "cr8legacy", "abm", "sse4a", "misalignsse",
             "3dnowprefetch", "osvw", "ibs", "xop",
-            "skinit", "wdt", NULL, "lwp",
+            "skinit", "wdt", "cvt16", "lwp",

...this is bit 14.

Anyway, the cvt16 bit was removed on purpose, and was never supported. See:
http://mid.mail-archive.com/[email protected]

target/i386/cpu.c static FeatureWordInfo feature_word_info[FEATURE_WORDS] = { · line 1047

Quoted 9 lines
target/i386/cpu.c +1 -1
             "fma4", "tce", NULL, "nodeid-msr",
             NULL, "tbm", "topoext", "perfctr-core",
             "perfctr-nb", NULL, NULL, NULL,
@@ -1044,7 +1044,7 @@ static FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
             "fsgsbase", "tsc-adjust", NULL, "bmi1",
             "hle", "avx2", NULL, "smep",
             "bmi2", "erms", "invpcid", "rtm",
-            NULL, NULL, "mpx", NULL,
+            "cmt", NULL, "mpx", NULL,

This is one is named "cqm" on Linux (X86_FEATURE_CQM). I prefer to keep consistency with the name already in use by Linux than the one in libvirt that was never used.

You can still add a "cmt" alias property if you think it would be useful.

Also, I see no code implementing migration of MSR_IA32_QM_EVTSEL, MSR_IA32_QM_CTR, or other RDT-M state. If the feature is not safe for migration yet, you need to explicitly add the feature to .unmigratable_flags.

Quoted 6 lines
target/i386/cpu.c +0 -0
             "avx512f", "avx512dq", "rdseed", "adx",
             "smap", "avx512ifma", "pcommit", "clflushopt",
             "clwb", "intel-pt", "avx512pf", "avx512er",
Signature · 1 line
2.17.1
Signature · 1 line
Eduardo
#4 lore raw
Quoted 6 lines · Eduardo Habkost

This is one is named "cqm" on Linux (X86_FEATURE_CQM). I prefer to keep consistency with the name already in use by Linux than the one in libvirt that was never used.

You can still add a "cmt" alias property if you think it would be useful.

Actually KVM does not mark it as supported:

        const u32 kvm_cpuid_7_0_ebx_x86_features =
                F(FSGSBASE) | F(BMI1) | F(HLE) | F(AVX2) | F(SMEP) |
                F(BMI2) | F(ERMS) | f_invpcid | F(RTM) | f_mpx | F(RDSEED) |
                F(ADX) | F(SMAP) | F(AVX512IFMA) | F(AVX512F) | F(AVX512PF) |
                F(AVX512ER) | F(AVX512CD) | F(CLFLUSHOPT) | F(CLWB) | F(AVX512DQ) |
                F(SHA_NI) | F(AVX512BW) | F(AVX512VL) | f_intel_pt;

Paolo

#5 lore raw
Quoted 8 lines · Paolo Bonzini
Quoted 6 lines · Eduardo Habkost

This is one is named "cqm" on Linux (X86_FEATURE_CQM). I prefer to keep consistency with the name already in use by Linux than the one in libvirt that was never used.

You can still add a "cmt" alias property if you think it would be useful.

Quoted 8 lines

Actually KVM does not mark it as supported:

        const u32 kvm_cpuid_7_0_ebx_x86_features =
                F(FSGSBASE) | F(BMI1) | F(HLE) | F(AVX2) | F(SMEP) |
                F(BMI2) | F(ERMS) | f_invpcid | F(RTM) | f_mpx | F(RDSEED) |
                F(ADX) | F(SMAP) | F(AVX512IFMA) | F(AVX512F) | F(AVX512PF) |
                F(AVX512ER) | F(AVX512CD) | F(CLFLUSHOPT) | F(CLWB) | F(AVX512DQ) |
                F(SHA_NI) | F(AVX512BW) | F(AVX512VL) | f_intel_pt;

We can still add the feature name if we also set it on .unmigratable_features, but I don't see why it would be useful. Is anybody working to support exposing RDT-M to guests?

Signature · 1 line
Eduardo
#6 lore raw
Quoted 11 lines · Eduardo Habkost
Quoted 8 lines

Actually KVM does not mark it as supported:

        const u32 kvm_cpuid_7_0_ebx_x86_features =
                F(FSGSBASE) | F(BMI1) | F(HLE) | F(AVX2) | F(SMEP) |
                F(BMI2) | F(ERMS) | f_invpcid | F(RTM) | f_mpx | F(RDSEED) |
                F(ADX) | F(SMAP) | F(AVX512IFMA) | F(AVX512F) | F(AVX512PF) |
                F(AVX512ER) | F(AVX512CD) | F(CLFLUSHOPT) | F(CLWB) | F(AVX512DQ) |
                F(SHA_NI) | F(AVX512BW) | F(AVX512VL) | f_intel_pt;

We can still add the feature name if we also set it on .unmigratable_features, but I don't see why it would be useful. Is anybody working to support exposing RDT-M to guests?

I don't think so.

Paolo

#7 lore raw
In reply to Denis V. Lunev
Quoted 9 lines · Denis V. Lunev

There are the following flags available in libvirt inside cpu_map.xm

    <feature name='cvt16'>
      <cpuid function='0x80000001' ecx='0x00040000'/>
    </feature>
    <feature name='cmt'> <!-- cqm -->
      <cpuid eax_in='0x07' ecx_in='0x00' ebx='0x00001000'/>
    </feature>

We have faced the problem that QEMU does not start once these flags are present in the domain.xml.

Libvirt should not add this to the XML by itself (when using host-model CPU, for example) so the user must have asked for the feature explicitly. Thus I don't see any problem with QEMU refusing to start with such configuration. And the workaround is easy, just don't do it.

I'm not sure about cvt16, but IIRC cmt and mbm_* features were added as a way to detect whether the host CPU supports perf monitoring counters. I think tt was not the brightest idea, but there's no reason why QEMU should support enabling these features. Unless it actually makes sense for QEMU.

If there are any issues with libvirt passing these features to QEMU without explicit request from the user, we should address them in libvirt.

Jirka