target/i386/cpu: introduce new CPU models for x86-64 ABI levels
v2 cover target/i386/cpu: introduce new CPU models for x86-64 ABI levels
This series is motivated by this blog that describes how RHEL-9 will recommend use of the x86-64-v2 microarchitectural ABI level:
The implication of compiling code with -march=x86-64-v2 is that this code will no longer be guaranteed to be runnable on a number of the CPU models exposed by the x86_64 target emulator, most notably qemu64 which is the default.
This series is not proposing to change the QEMU default CPU model for x86_64 target. I show how this is can trivially be done in patch 3, but not suggesting that we actually do that, as upstream is quite conservative in dropping support for old host hardware.
New CPU models
==============
It is instead exploring the possibilities of defining new CPU models in QEMU to closely match the x86-64 uarch ABI levels in a relatively vendor agnostic manner. This could be used by downstream vendors who wish to target specific uarch ABI levels in custom machine types.
One of the nice things about "qemu64" is that its naming presents it as effectively being a vendor-neutral model (if we ignore that vendor=AMD is in fact reported in CPUID).
If we look at the feature set fo x86-64-v2 ABI, we see that the QEMU "Nehalem" model is the closest match. This is also happens to be runnable on AMD Opteron G4/G5 and EPYC hosts. None the less, the use of an Intel specific CPU model name on an AMD host feels uncomfortable.
Vendor neutral naming
=====================
The idea behind this series is thus to introduce new CPU model names with vendor neutral naming, to more directly correlate with defined x86-64 uarch ABI levels. We don't want to just invent CPUs with a completely arbitrary set of CPU features as history has shown that brings its own problems. eg a guest uses features A and B, but only does CPUID checks for existence of feature B, assuming that B implies A.
The specification for x86-64 ABI levels uses x86-64-vNN naming but this clashes with QEMU's use of "vNN" for versioning. I felt it would be confusing to end up with CPU model names like "x86-64-v1-v1". Thus I've used an "-abiNNN" suffix instead. Also note I have an "-abi1" suffix here for the baseline. Arguably we could just drop the suffix entirely for the baseline.
A further note is that we could have used "qemu64" as the naming prefix, eg qemu64-abi2, qemu64-abi3, etc. Alot of people hold negative opinions of the qemu64 model in general though, so I felt a clean break with the past might be desirable, even though the x86-64-abi1 CPU model is effectively identical to qemu64.
Runnability of new models
=========================
The goal of the x86-64-abiNNN CPU models is that they should be runnable on any physical host which supports the CPUIDs features for that uarch ABI level. It is hard to figure out what exact set of CPUID features we should report. The uarch ABI document only specifies the minimum expectation, but we can't define a CPU in QEMU using only the minimum set as that would omit countless important features.
Thus to determine the feature set to use in x86-64-abiNNN CPU models, this series used a script (see patch 4) which did the following:
* Find list of current CPU models that satisfy ABI NN
* From this list, calculate the lowest common denominator (LCD) of CPUID features
* From the LCD, find the existing CPU model that has the fewest extra features over the LCD.
* Use that closest match, minus the extra features
to define x86-64-abiNN
This approach works well for the baseline ABI, and level 2 ABI
For ABI level 3 there is a bit of a larger divergance in feature sets, because the LCD between Intel and EPYC models is relatively large. We omit aes pcid erms invpcid tsc-deadline x2apic pclmulqdq
For ABI level 4 the divergance is small. I believe this is a deceptive situation that arises because currently only Intel CPUs in QEMU are able to satisfy ABI level 4 currently. None of the EPYC CPUs defined in QEMU are able to satisfy it.
I'm concerned that if future EPYC CPU generations are added to QEMU which *do* indeed satisfy x86-64 v4, then QEMU's x86-64-abi4 CPUID may prove to be too greedy and thus be unable to run on the future CPUs. In theory we can address this through use of CPU versioning, but ideally we would not have to deal with that.
Nested virt caveats
===================
A further caveat is that none of these generic models define any VMX feature set. At the very least I believe this means that nested virt can't live migrate, but maybe this breaks ability to use nested virt entirely. I'm unclear what the best approach is to deal with this, that would let the user do both "-cpu x86-64-abi2,svm=on" and "-cpu x86-64-abi2,vmx=on"
Side channel mitigation caveats
===============================
Finally none of the x86-64-abiNN models include any of the features related to CPU side channel mitigations. We can't add them without sacrificing the ability to run on some physical hardware that would otherwise satisfy the uarch ABI level.
This makes me a little uneasy. One of the main downsides of using "qemu64" is that it leaves guests vulnerable to the side channel attacks. Introducing new general purpose CPUs that still have this flaw feels undesirable. I don't know whethe to prioritize safety, while sacrificing runability.
Crypto accelerator caveats
==========================
Similarly I'm not a huge fan of leaving out the "aes" instruction for accelerated crypto, as missing "aes" is also one of the key factors in making qemu64 a bad choice.
If we include 'aes' in x86-64-abi2, then we loose support for Nehalem hosts.
If we include 'aes' in x86-64-abi3 then we further loose support for Dhyana hosts (an EPYC derived CPU).
Wrap up
=======
Overall this series shows some CPU models for QEMU that can map to each x86-64 ABI level, but I'm unsure whether it is wise to actually go forward with this idea given the various caveats.
I think they'll mostly only be useful as a built-in default for machine types. If anyone is going to explicitly specify a CPU model they'll almost always be better off picking a vendor specific model or host passthrough. There can be cases where people have a mix of AMD and Intel bare metal machines they need portability across but this feel rare.
Whatever else comes of this series, at least patch 1 should be useful, giving people an idea of what ABI levels each QEMU CPU can satisfy in the documentation.
Changed in v2:
- Don't document any of the CPU aliases, since they vary per machine type, only the concrete types.
- Formally add the script for re-generating the docs
Daniel P. Berrangé (4):
docs: add a table showing x86-64 ABI compatibility levels target/i386: define CPU models to model x86-64 ABI levels
scripts: helper to generate x86_64 CPU ABI compat info NOT FOR MERGE target/i386: use x86-64-abi1 CPU model as default on
x86_64
MAINTAINERS | 2 +- docs/system/cpu-models-x86-abi.csv | 75 +++++++++++ docs/system/cpu-models-x86.rst.inc | 22 ++++ hw/i386/pc_piix.c | 3 + hw/i386/pc_q35.c | 3 + scripts/cpu-x86-uarch-abi.py | 194 +++++++++++++++++++++++++++++ target/i386/cpu.c | 156 +++++++++++++++++++++++ target/i386/cpu.h | 2 +- 8 files changed, 455 insertions(+), 2 deletions(-) create mode 100644 docs/system/cpu-models-x86-abi.csv create mode 100644 scripts/cpu-x86-uarch-abi.py
Signature · 1 line
2.31.1
PATCH v2 1/4 docs: add a table showing x86-64 ABI compatibility levels
It is useful to know which CPUs satisfy each x86-64 ABI compatibility level, when dealing with guest OS that require something newer than the baseline ABI.
These ABI levels are defined in:
https://gitlab.com/x86-psABIs/x86-64-ABI/
and supported by GCC, Clang, glibc and more.
--- MAINTAINERS | 2 +- docs/system/cpu-models-x86-abi.csv | 67 ++++++++++++++++++++++++++++++ docs/system/cpu-models-x86.rst.inc | 22 ++++++++++ 3 files changed, 90 insertions(+), 1 deletion(-) create mode 100644 docs/system/cpu-models-x86-abi.csv
MAINTAINERS +1 -1
@@ -329,7 +329,7 @@ F: tests/tcg/i386/ F: tests/tcg/x86_64/ F: hw/i386/ F: disas/i386.c -F: docs/system/cpu-models-x86.rst.inc +F: docs/system/cpu-models-x86* T: git https://gitlab.com/ehabkost/qemu.git x86-next Xtensa TCG CPUs
docs/system/cpu-models-x86-abi.csv new file +67 -0
67 lines not on this page · raw message
docs/system/cpu-models-x86.rst.inc +22 -0
@@ -39,6 +39,28 @@ CPU, as they would with "Host passthrough", but gives much of the benefit of passthrough, while making live migration safe. +ABI compatibility levels for CPU models +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The x86_64 architecture has a number of `ABI compatibility levels`_ +defined. Traditionally most operating systems and toolchains would +only target the original baseline ABI. It is expected that in +future OS and toolchains are likely to target newer ABIs. The +table that follows illustrates which ABI compatibility levels +can be satisfied by the QEMU CPU models. Note that the table only +lists the long term stable CPU model versions (eg Haswell-v4). +In addition to whats listed, there are also many CPU model +aliases which resolve to a different CPU model version, +depending on the machine type is in use. + +.. _ABI compatibility levels: https://gitlab.com/x86-psABIs/x86-64-ABI/ + +.. csv-table:: x86-64 ABI compatibility levels + :file: cpu-models-x86-abi.csv + :widths: 40,15,15,15,15 + :header-rows: 2 + + Preferred CPU models for Intel x86 hosts ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Signature · 1 line
2.31.1
PATCH v2 2/4 target/i386: define CPU models to model x86-64 ABI levels
In reply to Daniel P. BerrangéTo paraphrase:
In 2020, AMD, Intel, Red Hat, and SUSE worked together to define three microarchitecture levels on top of the historical x86-64 baseline:
* x86-64: original x86_64 baseline instruction set
* x86-64-v2: vector instructions up to Streaming SIMD Extensions 4.2 (SSE4.2) and Supplemental Streaming SIMD Extensions 3 (SSSE3), the POPCNT instruction, and CMPXCHG16B
* x86-64-v3: vector instructions up to AVX2, MOVBE, and additional bit-manipulation instructions.
* x86-64-v4: vector instructions from some of the
AVX-512 variants.
This list of features is defined in the doc at:
https://gitlab.com/x86-psABIs/x86-64-ABI/
QEMU has historically defaulted to the "qemu64" CPU model on x86_64 targets, which is approximately the x86-64 baseline ABI, with 'SVM' added.
It is thought it might be desirable if QEMU could provide CPU models that closely correspond to the ABI levels, while offering portability across the maximum number of physical CPUs.
Historically we've found that defining CPU models with an arbitrary combination of CPU features can be problematic, as some guest OS will not check all features they use, and instead assume that if they see feature "XX", then "YY" will always exist. This is reasonable in bare metal, but subject to breakage in virtualization.
Thus in defining the CPU models for the ABI levels, this patch attempted to base them off an existing CPU model.
While each x86-64-abiNNN has a designated vendor, they are designed to be vendor agnostic models. ie they are capable of running on any physical x86_64 CPU model that satisfies the ABI level. eg although the x86-64-abi2 model is based on Nehalem, it should be able to run guests on an Opteron_G4/G5/EPYC host, since those CPUs support the features required by the x86-64 v2 ABI.
More precisely the models were defined as:
* x86-64-abi1: close match for Opteron_G1, minus
vme
* x86-64-abi2: perfect match for Nehalem
* x86-64-abi3: close match of Haswell-noTSX, minus aes pcid erms invpcid tsc-deadline x2apic pclmulqdq
* x86-64-abi4: close match of Skylake-Server-noTSX-IBRS, minus spec-ctrl
None of the CPU models declare any VMX/SVM capability features. IOW, even if a "vmx"/"svm" flag is added, it will still be unsafe to attempt to live migrate the L1 guest if there are any L2 guests running with hardware virtualization.
Given their vendor agnostic design, these CPU models are primarily thought tobe useful as the default CPU model for machine types. QEMU upstream is quite conservative in mandating new hardware features, but a downstream vendor may choose to mandate a newer x86-64 ABI level for downstream only machine types.
Note that TCG is not capable of supporting the 2 newest ABI levels currently due to missing features:
* x86-64-abi3:
CPUID.01H:ECX.fma [bit 12]
CPUID.01H:ECX.avx [bit 28]
CPUID.01H:ECX.f16c [bit 29]
CPUID.07H:EBX.avx2 [bit 5]
* x86-64-abi4:
CPUID.01H:ECX.pcid [bit 17]
CPUID.01H:ECX.x2apic [bit 21]
CPUID.01H:ECX.tsc-deadline [bit 24]
CPUID.07H:EBX.invpcid [bit 10]
CPUID.07H:EBX.avx512f [bit 16]
CPUID.07H:EBX.avx512dq [bit 17]
CPUID.07H:EBX.rdseed [bit 18]
CPUID.07H:EBX.avx512cd [bit 28]
CPUID.07H:EBX.avx512bw [bit 30]
CPUID.07H:EBX.avx512vl [bit 31]
CPUID.80000001H:ECX.3dnowprefetch [bit 8]
CPUID.0DH:EAX.xsavec [bit 1]
--- docs/system/cpu-models-x86-abi.csv | 8 ++ target/i386/cpu.c | 156 +++++++++++++++++++++++++++++ 2 files changed, 164 insertions(+)
docs/system/cpu-models-x86-abi.csv +8 -0
@@ -65,3 +65,11 @@ pentium3-v1,,,, phenom-v1,✅,,, qemu32-v1,,,, qemu64-v1,✅,,, +x86-64-abi1,✅,,, +x86-64-abi1-v1,✅,,, +x86-64-abi2,✅,✅,, +x86-64-abi2-v1,✅,✅,, +x86-64-abi3,✅,✅,✅, +x86-64-abi3-v1,✅,✅,✅, +x86-64-abi4,✅,✅,✅,✅ +x86-64-abi4-v1,✅,✅,✅,✅
target/i386/cpu.c +156 -0
162 lines not on this page · raw message
Signature · 1 line
2.31.1
PATCH v2 3/4 scripts: helper to generate x86_64 CPU ABI compat info
In reply to Daniel P. BerrangéThis script is what is used to generate the docs data table in:
docs/system/cpu-models-x86-abi.csv
It can be useful to run if adding new CPU models / versions and the csv needs updating.
--- scripts/cpu-x86-uarch-abi.py | 194 +++++++++++++++++++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100644 scripts/cpu-x86-uarch-abi.py
scripts/cpu-x86-uarch-abi.py new file +194 -0
@@ -0,0 +1,194 @@ +#!/usr/bin/python3 +# +# SPDX-License-Identifier: GPL-2.0-or-later +# +# A script to generate a CSV file showing the x86_64 ABI +# compatibility levels for each CPU model. +# + +from qemu import qmp +import sys + +if len(sys.argv) != 1: + print("syntax: %s QMP-SOCK\n\n" % __file__ + + "Where QMP-SOCK points to a QEMU process such as\n\n" + + " # qemu-system-x86_64 -qmp unix:/tmp/qmp,server,nowait " + + "-display none -accel kvm", file=sys.stderr) + sys.exit(1) + +# Mandatory CPUID features for each microarch ABI level +levels = [ + [ # x86-64 baseline + "cmov", + "cx8", + "fpu", + "fxsr", + "mmx", + "syscall", + "sse", + "sse2", + ], + [ # x86-64-v2 + "cx16", + "lahf-lm", + "popcnt", + "pni", + "sse4.1", + "sse4.2", + "ssse3", + ], + [ # x86-64-v3 + "avx", + "avx2", + "bmi1", + "bmi2", + "f16c", + "fma", + "abm", + "movbe", + ], + [ # x86-64-v4 + "avx512f", + "avx512bw", + "avx512cd", + "avx512dq", + "avx512vl", + ], +] + +# Assumes externally launched process such as +# +# qemu-system-x86_64 -qmp unix:/tmp/qmp,server,nowait -display none -accel kvm +# +# Note different results will be obtained with TCG, as +# TCG masks out certain features otherwise present in +# the CPU model definitions, as does KVM. + + +sock = sys.argv[1] +cmd = sys.argv[2] +shell = qmp.QEMUMonitorProtocol(sock) +shell.connect() + +models = shell.cmd("query-cpu-definitions") + +# These QMP props don't correspond to CPUID fatures +# so ignore them +skip = [ + "family", + "min-level", + "min-xlevel", + "vendor", + "model", + "model-id", + "stepping", +] + +names = [] + +for model in models["return"]: + if "alias-of" in model: + continue + names.append(model["name"]) + +models = {} + +for name in sorted(names): + cpu = shell.cmd("query-cpu-model-expansion", + { "type": "static", + "model": { "name": name }}) + + got = {} + for (feature, present) in cpu["return"]["model"]["props"].items(): + if present and feature not in skip: + got[feature] = True + + if name in ["host", "max", "base"]: + continue + + models[name] = { + # Dict of all present features in this CPU model + "features": got, + + # Whether each x86-64 ABI level is satisfied + "levels": [False, False, False, False], + + # Number of extra CPUID features compared to the x86-64 ABI level + "distance":[-1, -1, -1, -1], + + # CPUID features present in model, but not in ABI level + "delta":[[], [], [], []], + + # CPUID features in ABI level but not present in model + "missing": [[], [], [], []], + } + + +# Calculate whether the CPU models satisfy each ABI level +for name in models.keys(): + for level in range(len(levels)): + got = set(models[name]["features"]) + want = set(levels[level]) + missing = want - got + match = True + if len(missing) > 0: + match = False + models[name]["levels"][level] = match + models[name]["missing"][level] = missing + +# Cache list of CPU models satisfying each ABI level +abi_models = [ + [], + [], + [], + [], +] + +for name in models.keys(): + for level in range(len(levels)): + if models[name]["levels"][level]: + abi_models[level].append(name) + + +for level in range(len(abi_models)): + # Find the union of features in all CPU models satisfying this ABI + allfeatures = {} + for name in abi_models[level]: + for feat in models[name]["features"]: + allfeatures[feat] = True + + # Find the intersection of features in all CPU models satisfying this ABI + commonfeatures = [] + for feat in allfeatures: + present = True + for name in models.keys(): + if not models[name]["levels"][level]: + continue + if feat not in models[name]["features"]: + present = False + if present: + commonfeatures.append(feat) + + # Determine how many extra features are present compared to the lowest + # common denominator + for name in models.keys(): + if not models[name]["levels"][level]: + continue + + delta = set(models[name]["features"].keys()) - set(commonfeatures) + models[name]["distance"][level] = len(delta) + models[name]["delta"][level] = delta + +def print_uarch_abi_csv(): + print("# Automatically generated from '%s'" % __file__) + print("Model,baseline,v2,v3,v4") + for name in models.keys(): + print(name, end="") + for level in range(len(levels)): + if models[name]["levels"][level]: + print(",✅", end="") + else: + print(",", end="") + print() + +print_uarch_abi_csv()
Signature · 1 line
2.31.1
PATCH v2 4/4 NOT FOR MERGE target/i386: use x86-64-abi1 CPU model as default on x86_64
In reply to Daniel P. BerrangéThe only differences between x86-64-abi1 and qemu64 is that the former does not have the 'vme' or 'svm' flags.
In practice I don't think we should make this change, because it doesn't especially add any value as-is. The only possible case is around 'svm' because KVM already masks that feature, but TCG allows it. Thus using x86-64-abi1 would mean that KVM and TCG expose a more consistent feature set.
Also note that while libvirt can cope with default CPUs changing now, it can't with the default CPU being a model name that it does not already know about.
--- hw/i386/pc_piix.c | 3 +++ hw/i386/pc_q35.c | 3 +++ target/i386/cpu.h | 2 +- 3 files changed, 7 insertions(+), 1 deletion(-)
hw/i386/pc_piix.c +3 -0
@@ -444,6 +444,9 @@ static void pc_i440fx_5_2_machine_options(MachineClass *m) m->is_default = false;compat_props_add (m->compat_props, hw_compat_5_2, hw_compat_5_2_len);compat_props_add (m->compat_props, pc_compat_5_2, pc_compat_5_2_len); +#ifdef TARGET_X86_64 + m->default_cpu_type =X86_CPU_TYPE_NAME ("qemu64" ); +#endif } DEFINE_I440FX_MACHINE(v5_2, "pc-i440fx-5.2", NULL,
hw/i386/pc_q35.c +3 -0
@@ -372,6 +372,9 @@ static void pc_q35_5_2_machine_options(MachineClass *m) m->alias =NULL ;compat_props_add (m->compat_props, hw_compat_5_2, hw_compat_5_2_len);compat_props_add (m->compat_props, pc_compat_5_2, pc_compat_5_2_len); +#ifdef TARGET_X86_64 + m->default_cpu_type =X86_CPU_TYPE_NAME ("qemu64" ); +#endif } DEFINE_Q35_MACHINE(v5_2, "pc-q35-5.2", NULL,
target/i386/cpu.h +1 -1
@@ -2001,7 +2001,7 @@ uint64_t cpu_get_tsc(CPUX86State *env);#define CPU_RESOLVING_TYPE TYPE_X86_CPU#ifdef TARGET_X86_64 -#define TARGET_DEFAULT_CPU_TYPE X86_CPU_TYPE_NAME("qemu64") +#define TARGET_DEFAULT_CPU_TYPE X86_CPU_TYPE_NAME("x86-64-abi1")#else #define TARGET_DEFAULT_CPU_TYPE X86_CPU_TYPE_NAME("qemu32")#endif
Signature · 1 line
2.31.1