target/arm/kvm: enable SVE in guests
cover target/arm/kvm: enable SVE in guests
With the recent KVM guest SVE support pull request [1] KVM will be ready for guests with SVE. This series provides the QEMU bits for that enablement. The series starts with the bits needed for the KVM SVE ioctls. Then it enables the arm 'max'cpu type, which with TCG already supports SVE, to also support SVE when using KVM. Next a new QMP query is added that allows users to ask which vector lengths are supported by the host, allowing them to select a valid set of vectors for the guest. In order to select those vectors a new property 'sve-vls-map' is added to the 'max' cpu type, and then also to the 'host' cpu type. The table below shows the resulting user interfaces.
CPU type | accel | sve-max-vq | sve-vls-map
-------------------------------------------
1) max | tcg | $MAX_VQ | $VLS_MAP 2) max | kvm | $MAX_VQ | $VLS_MAP 3) host | kvm | N/A | $VLS_MAP
Where for (1) $MAX_VQ sets the maximum vq and smaller vqs are all supported when $VLS_MAP is zero, or when the vqs are selected in $VLS_MAP.
(2) is the same as (1) except KVM has the final say on what vqs are valid.
(3) doesn't accept sve-max-vq because a guest that uses this property without sve-vls-map cannot be safely migrated.
There is never any need to provide both properties, but if both are provided then they are checked for consistency.
The QMP query returns a list of valid vq lists. For example, if a guest can use vqs 1, 2, 3, and 4, then the following list will be returned
[ [ 1 ], [ 1, 2 ], [ 1, 2, 3 ], [ 1, 2, 3, 4 ] ]
Another example might be 1, 2, 4, as the architecture states 3 is optional. In that case the list would be
[ [ 1 ], [ 1, 2 ], [ 1, 2, 4 ] ]
This may look redundant, but it's necessary to provide a future- proof query, because while KVM currently requires vector sets to be strict truncations of the full valid vector set, that may change at some point. Additionally, TCG doesn't have this restriction so more vector sets can be returned that aren't so easily derived from the full set already. (Note, this series doesn't do that yet. See the TODO below.)
And now for what might be a bit more controversial; how we input the valid vector set with sve-vls-map. Well, sve-vls-map is a 64-bit bitmap, which is admittedly not user friendly and also not the same size as KVM's vls bitmap (which is 8 64-bit words). Here's the justification:
1) We want to use the QEMU command line in order for the information to be migrated without needing to add more VM state.
2) It's not easy/pretty to input arrays on the QEMU command line.
3) We already need to use the QMP query to get a valid set, which isn't user friendly either, meaning we're already in libvirt territory.
4) A 64-bit map (supporting up to 8192-bit vectors) is likely big enough for quite some time (currently KVM and TCG only support 2048-bit vectors).
5) If user friendliness is needed more than migratability then the 'max' cpu type can be used with the sve-max-vq property.
6) It's possible to probe the full valid vector set from the
command line by using something like sve-vls-map=0xffff and
then, when it fails, the error message will state the correct
map, e.g. 0xb.TODO:
1) More testing. Initial testing looks good, and I'm doing more now, but wanted to get the series out for review in parallel.
2) Extension to target/arm/arch_dump.c for SVE state. I'll try to get to this early next week.
3) Modify the QMP query to output all valid vector sets for TCG, rather than just the ones derived by truncation as is required by KVM.
[1] https://www.spinics.net/lists/kvm-arm/msg35844.html
Thanks!
Andrew Jones (13):
target/arm/kvm64: fix error returns
update-linux-headers: Add sve_context.h to asm-arm64
HACK: linux header update
target/arm/kvm: Move the get/put of fpsimd registers out target/arm/kvm: Add kvm_arch_get/put_sve target/arm/kvm: max cpu: Enable SVE when available target/arm/kvm: max cpu: Allow sve max vector length setting target/arm/monitor: Add query-sve-vector-lengths target/arm/kvm: Export kvm_arm_get_sve_vls target/arm/monitor: kvm: only return valid sve vector sets target/arm/cpu64: max cpu: Introduce sve-vls-map target/arm/kvm: max cpu: Add support for sve-vls-map target/arm/kvm: host cpu: Add support for sve-vls-map
linux-headers/asm-arm64/kvm.h | 41 +++ linux-headers/asm-arm64/sve_context.h | 53 ++++ linux-headers/linux/kvm.h | 5 + qapi/target.json | 34 +++ scripts/update-linux-headers.sh | 3 + target/arm/cpu.c | 5 + target/arm/cpu.h | 9 + target/arm/cpu64.c | 81 ++++- target/arm/helper.c | 10 +- target/arm/kvm.c | 5 + target/arm/kvm64.c | 418 ++++++++++++++++++++++---- target/arm/kvm_arm.h | 32 ++ target/arm/monitor.c | 106 +++++++ tests/qmp-cmd-test.c | 1 + 14 files changed, 732 insertions(+), 71 deletions(-) create mode 100644 linux-headers/asm-arm64/sve_context.h
Signature · 1 line
2.20.1
PATCH 1/13 target/arm/kvm64: fix error returns
A couple return -EINVAL's forget their '-'s.
--- target/arm/kvm64.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
target/arm/kvm64.c +2 -2
@@ -841,7 +841,7 @@ int kvm_arch_put_registers(CPUState *cs, int level)write_cpustate_to_list (cpu, true);if (!write_list_to_kvmstate (cpu, level)) { -return EINVAL ; +return -EINVAL ; }kvm_arm_sync_mpstate_to_kvm (cpu);
@@ -982,7 +982,7 @@ int kvm_arch_get_registers(CPUState *cs) }if (!write_kvmstate_to_list (cpu)) { -return EINVAL ; +return -EINVAL ; }/* Note that it's OK to have registers which aren't in CPUState, * so we can ignore a failure return here.
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH 01/13] target/arm/kvm64: fix error returns
Hi,
Quoted 22 lines · Andrew Jones
A couple return -EINVAL's forget their '-'s.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/kvm64.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)target/arm/kvm64.c +1
-1@@ -841,7 +841,7 @@ int kvm_arch_put_registers(CPUState *cs, int level)write_cpustate_to_list (cpu, true);if (!write_list_to_kvmstate (cpu, level)) { -return EINVAL ; +return -EINVAL ; }kvm_arm_sync_mpstate_to_kvm (cpu);
target/arm/kvm64.c int kvm_arch_get_registers(CPUState *cs) · line 988
Quoted 8 lines
target/arm/kvm64.c +1
-1@@ -982,7 +982,7 @@ int kvm_arch_get_registers(CPUState *cs) }if (!write_kvmstate_to_list (cpu)) { -return EINVAL ; +return -EINVAL ; }/* Note that it's OK to have registers which aren't in CPUState, * so we can ignore a failure return here.
note, if I am not wrong, in existing call sites the returned value is never tested actually.
Thanks
Eric
PATCH 2/13 update-linux-headers: Add sve_context.h to asm-arm64
In reply to Andrew Jones--- scripts/update-linux-headers.sh | 3 +++ 1 file changed, 3 insertions(+)
scripts/update-linux-headers.sh +3 -0
@@ -99,6 +99,9 @@ for arch in $ARCHLIST; do cp "$tmpdir/include/asm/$header" "$output/linux-headers/asm-$arch" done + if [ $arch = arm64 ]; then + cp "$tmpdir/include/asm/sve_context.h" "$output/linux-headers/asm-arm64/" + fi if [ $arch = mips ]; then cp "$tmpdir/include/asm/sgidefs.h" "$output/linux-headers/asm-mips/" cp "$tmpdir/include/asm/unistd_o32.h" "$output/linux-headers/asm-mips/"
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH 02/13] update-linux-headers: Add sve_context.h to asm-arm64
Hi,
Quoted 11 lines · Andrew Jones
Signed-off-by Andrew Jones <[email protected]>--- scripts/update-linux-headers.sh | 3 +++ 1 file changed, 3 insertions(+)scripts/update-linux-headers.sh +0
-0
scripts/update-linux-headers.sh for arch in $ARCHLIST; do · line 107
Quoted 9 lines
scripts/update-linux-headers.sh +3
-0@@ -99,6 +99,9 @@ for arch in $ARCHLIST; do cp "$tmpdir/include/asm/$header" "$output/linux-headers/asm-$arch" done + if [ $arch = arm64 ]; then + cp "$tmpdir/include/asm/sve_context.h" "$output/linux-headers/asm-arm64/" + fi if [ $arch = mips ]; then cp "$tmpdir/include/asm/sgidefs.h" "$output/linux-headers/asm-mips/" cp "$tmpdir/include/asm/unistd_o32.h" "$output/linux-headers/asm-mips/"
Eric
Re: [Qemu-devel] [PATCH 02/13] update-linux-headers: Add sve_context.h to asm-arm64
Quoted 16 lines · Auger Eric
Hi,
Quoted 13 lines · Andrew Jones
Signed-off-by Andrew Jones <[email protected]>--- scripts/update-linux-headers.sh | 3 +++ 1 file changed, 3 insertions(+)scripts/update-linux-headers.sh +0
-0@@ -99,6 +99,9 @@ for arch in $ARCHLIST; do cp "$tmpdir/include/asm/$header" "$output/linux-headers/asm-$arch" done
Quoted 9 lines
scripts/update-linux-headers.sh for arch in $ARCHLIST; do · line 107
Quoted 7 lines
scripts/update-linux-headers.sh +3
-0+ if [ $arch = arm64 ]; then + cp "$tmpdir/include/asm/sve_context.h" "$output/linux-headers/asm-arm64/" + fi if [ $arch = mips ]; then cp "$tmpdir/include/asm/sgidefs.h" "$output/linux-headers/asm-mips/" cp "$tmpdir/include/asm/unistd_o32.h" "$output/linux-headers/asm-mips/"Reviewed-by Eric Auger <[email protected]>
Thanks, but I noticed we don't need this anymore. Not since b1b9e0dc78a8e applied an identical patch. I've dropped this from the v2 I'm working on now.
Thanks,
drew
PATCH 3/13 HACK: linux header update
In reply to Andrew Jones--- linux-headers/asm-arm64/kvm.h | 41 +++++++++++++++++++++ linux-headers/asm-arm64/sve_context.h | 53 +++++++++++++++++++++++++++ linux-headers/linux/kvm.h | 5 +++ 3 files changed, 99 insertions(+) create mode 100644 linux-headers/asm-arm64/sve_context.h
linux-headers/asm-arm64/kvm.h +41 -0
@@ -35,6 +35,7 @@#include <linux/psci.h> #include <linux/types.h> #include <asm/ptrace.h> +#include <asm/sve_context.h> #define __KVM_HAVE_GUEST_DEBUG#define __KVM_HAVE_IRQ_LINE
@@ -102,6 +103,7 @@ struct kvm_regs {#define KVM_ARM_VCPU_EL1_32BIT 1/* CPU running a 32bit VM */ #define KVM_ARM_VCPU_PSCI_0_2 2/* CPU uses PSCI v0.2 */ #define KVM_ARM_VCPU_PMU_V3 3/* Support guest PMUv3 */ +#define KVM_ARM_VCPU_SVE 4/* enable SVE for this CPU */ struct kvm_vcpu_init {__u32 target;
@@ -226,6 +228,45 @@ struct kvm_vcpu_events { KVM_REG_ARM_FW | ((r) & 0xffff))#define KVM_REG_ARM_PSCI_VERSION KVM_REG_ARM_FW_REG(0) +/* SVE registers */ +#define KVM_REG_ARM64_SVE (0x15 << KVM_REG_ARM_COPROC_SHIFT) + +/* Z- and P-regs occupy blocks at the following offsets within this range: */ +#define KVM_REG_ARM64_SVE_ZREG_BASE 0 +#define KVM_REG_ARM64_SVE_PREG_BASE 0x400 +#define KVM_REG_ARM64_SVE_FFR_BASE 0x600 + +#define KVM_ARM64_SVE_NUM_ZREGS __SVE_NUM_ZREGS +#define KVM_ARM64_SVE_NUM_PREGS __SVE_NUM_PREGS + +#define KVM_ARM64_SVE_MAX_SLICES 32 + +#define KVM_REG_ARM64_SVE_ZREG (n, i) \ + (KVM_REG_ARM64 | KVM_REG_ARM64_SVE | KVM_REG_ARM64_SVE_ZREG_BASE | \ + KVM_REG_SIZE_U2048 | \ + (((n) & (KVM_ARM64_SVE_NUM_ZREGS - 1)) << 5) | \ + ((i) & (KVM_ARM64_SVE_MAX_SLICES - 1))) + +#define KVM_REG_ARM64_SVE_PREG (n, i) \ + (KVM_REG_ARM64 | KVM_REG_ARM64_SVE | KVM_REG_ARM64_SVE_PREG_BASE | \ + KVM_REG_SIZE_U256 | \ + (((n) & (KVM_ARM64_SVE_NUM_PREGS - 1)) << 5) | \ + ((i) & (KVM_ARM64_SVE_MAX_SLICES - 1))) + +#define KVM_REG_ARM64_SVE_FFR (i) \ + (KVM_REG_ARM64 | KVM_REG_ARM64_SVE | KVM_REG_ARM64_SVE_FFR_BASE | \ + KVM_REG_SIZE_U256 | \ + ((i) & (KVM_ARM64_SVE_MAX_SLICES - 1))) + +#define KVM_ARM64_SVE_VQ_MIN __SVE_VQ_MIN +#define KVM_ARM64_SVE_VQ_MAX __SVE_VQ_MAX + +/* Vector lengths pseudo-register: */ +#define KVM_REG_ARM64_SVE_VLS (KVM_REG_ARM64 | KVM_REG_ARM64_SVE | \ + KVM_REG_SIZE_U512 | 0xffff) +#define KVM_ARM64_SVE_VLS_WORDS \ + ((KVM_ARM64_SVE_VQ_MAX - KVM_ARM64_SVE_VQ_MIN) / 64 + 1) +/* Device Control API: ARM VGIC */ #define KVM_DEV_ARM_VGIC_GRP_ADDR 0#define KVM_DEV_ARM_VGIC_GRP_DIST_REGS 1
linux-headers/asm-arm64/sve_context.h new file +53 -0
53 lines not on this page · raw message
linux-headers/linux/kvm.h +5 -0
@@ -988,6 +988,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_ARM_VM_IPA_SIZE 165#define KVM_CAP_MANUAL_DIRTY_LOG_PROTECT 166#define KVM_CAP_HYPERV_CPUID 167 +#define KVM_CAP_ARM_SVE 168#ifdef KVM_CAP_IRQ_ROUTING
@@ -1145,6 +1146,7 @@ struct kvm_dirty_tlb {#define KVM_REG_SIZE_U256 0x0050000000000000ULL#define KVM_REG_SIZE_U512 0x0060000000000000ULL#define KVM_REG_SIZE_U1024 0x0070000000000000ULL +#define KVM_REG_SIZE_U2048 0x0080000000000000ULLstruct kvm_reg_list {__u64 n;/* number of regs */
@@ -1440,6 +1442,9 @@ struct kvm_enc_region {/* Available with KVM_CAP_HYPERV_CPUID */ #define KVM_GET_SUPPORTED_HV_CPUID _IOWR(KVMIO, 0xc1, struct kvm_cpuid2) +/* Available with KVM_CAP_ARM_SVE */ +#define KVM_ARM_VCPU_FINALIZE _IOW(KVMIO, 0xc2, int) +/* Secure Encrypted Virtualization command */ enum sev_cmd_id {/* Guest initialization commands */
Signature · 1 line
2.20.1
PATCH 4/13 target/arm/kvm: Move the get/put of fpsimd registers out
In reply to Andrew JonesMove the getting/putting of the fpsimd registers out of kvm_arch_get/put_registers() into their own helper functions to prepare for alternatively getting/putting SVE registers.
No functional change.
--- target/arm/kvm64.c | 148 +++++++++++++++++++++++++++------------------ 1 file changed, 88 insertions(+), 60 deletions(-)
target/arm/kvm64.c +88 -60
@@ -706,13 +706,53 @@ int kvm_arm_cpreg_level(uint64_t regidx)#define AARCH64_SIMD_CTRL_REG (x) (KVM_REG_ARM64 | KVM_REG_SIZE_U32 | \ KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x)) +static int kvm_arch_put_fpsimd (CPUState *cs) +{ +ARMCPU *cpu =ARM_CPU (cs); +CPUARMState *env = &cpu->env; +struct kvm_one_reg reg; +uint32_t fpr; +int i, ret; + +for (i =0 ; i <32 ; i++) { +uint64_t *q =aa64_vfp_qreg (env, i); +#ifdef HOST_WORDS_BIGENDIAN +uint64_t fp_val[2 ] = { q[1 ], q[0 ] }; + reg.addr = (uintptr_t )fp_val; +#else + reg.addr = (uintptr_t )q; +#endif + reg.id =AARCH64_SIMD_CORE_REG (fp_regs.vregs[i]); + ret =kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); +if (ret) { +return ret; + } + } + + reg.addr = (uintptr_t )(&fpr); + fpr =vfp_get_fpsr (env); + reg.id =AARCH64_SIMD_CTRL_REG (fp_regs.fpsr); + ret =kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); +if (ret) { +return ret; + } + + reg.addr = (uintptr_t )(&fpr); + fpr =vfp_get_fpcr (env); + reg.id =AARCH64_SIMD_CTRL_REG (fp_regs.fpcr); + ret =kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); +if (ret) { +return ret; + } + +return 0 ; +} +int kvm_arch_put_registers (CPUState *cs,int level) {struct kvm_one_reg reg; -uint32_t fpr;uint64_t val; -int i; -int ret; +int i, ret;unsigned int el;ARMCPU *cpu =ARM_CPU (cs);
@@ -802,33 +842,7 @@ int kvm_arch_put_registers(CPUState *cs, int level) } } -/* Advanced SIMD and FP registers. */ -for (i =0 ; i <32 ; i++) { -uint64_t *q =aa64_vfp_qreg (env, i); -#ifdef HOST_WORDS_BIGENDIAN -uint64_t fp_val[2 ] = { q[1 ], q[0 ] }; - reg.addr = (uintptr_t )fp_val; -#else - reg.addr = (uintptr_t )q; -#endif - reg.id =AARCH64_SIMD_CORE_REG (fp_regs.vregs[i]); - ret =kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); -if (ret) { -return ret; - } - } - - reg.addr = (uintptr_t )(&fpr); - fpr =vfp_get_fpsr (env); - reg.id =AARCH64_SIMD_CTRL_REG (fp_regs.fpsr); - ret =kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); -if (ret) { -return ret; - } - - fpr =vfp_get_fpcr (env); - reg.id =AARCH64_SIMD_CTRL_REG (fp_regs.fpcr); - ret =kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); + ret =kvm_arch_put_fpsimd (cs);if (ret) {return ret; }
@@ -849,14 +863,54 @@ int kvm_arch_put_registers(CPUState *cs, int level)return ret; } +static int kvm_arch_get_fpsimd (CPUState *cs) +{ +ARMCPU *cpu =ARM_CPU (cs); +CPUARMState *env = &cpu->env; +struct kvm_one_reg reg; +uint32_t fpr; +int i, ret; + +for (i =0 ; i <32 ; i++) { +uint64_t *q =aa64_vfp_qreg (env, i); + reg.id =AARCH64_SIMD_CORE_REG (fp_regs.vregs[i]); + reg.addr = (uintptr_t )q; + ret =kvm_vcpu_ioctl (cs,KVM_GET_ONE_REG , ®); +if (ret) { +return ret; + }else { +#ifdef HOST_WORDS_BIGENDIAN +uint64_t t; + t = q[0 ], q[0 ] = q[1 ], q[1 ] = t; +#endif + } + } + + reg.addr = (uintptr_t )(&fpr); + reg.id =AARCH64_SIMD_CTRL_REG (fp_regs.fpsr); + ret =kvm_vcpu_ioctl (cs,KVM_GET_ONE_REG , ®); +if (ret) { +return ret; + } +vfp_set_fpsr (env, fpr); + + reg.addr = (uintptr_t )(&fpr); + reg.id =AARCH64_SIMD_CTRL_REG (fp_regs.fpcr); + ret =kvm_vcpu_ioctl (cs,KVM_GET_ONE_REG , ®); +if (ret) { +return ret; + } +vfp_set_fpcr (env, fpr); + +return 0 ; +} +int kvm_arch_get_registers (CPUState *cs) {struct kvm_one_reg reg;uint64_t val; -uint32_t fpr;unsigned int el; -int i; -int ret; +int i, ret;ARMCPU *cpu =ARM_CPU (cs);CPUARMState *env = &cpu->env;
@@ -945,36 +999,10 @@ int kvm_arch_get_registers(CPUState *cs) env->spsr = env->banked_spsr[i]; } -/* Advanced SIMD and FP registers */ -for (i =0 ; i <32 ; i++) { -uint64_t *q =aa64_vfp_qreg (env, i); - reg.id =AARCH64_SIMD_CORE_REG (fp_regs.vregs[i]); - reg.addr = (uintptr_t )q; - ret =kvm_vcpu_ioctl (cs,KVM_GET_ONE_REG , ®); -if (ret) { -return ret; - }else { -#ifdef HOST_WORDS_BIGENDIAN -uint64_t t; - t = q[0 ], q[0 ] = q[1 ], q[1 ] = t; -#endif - } - } - - reg.addr = (uintptr_t )(&fpr); - reg.id =AARCH64_SIMD_CTRL_REG (fp_regs.fpsr); - ret =kvm_vcpu_ioctl (cs,KVM_GET_ONE_REG , ®); -if (ret) { -return ret; - } -vfp_set_fpsr (env, fpr); - - reg.id =AARCH64_SIMD_CTRL_REG (fp_regs.fpcr); - ret =kvm_vcpu_ioctl (cs,KVM_GET_ONE_REG , ®); + ret =kvm_arch_get_fpsimd (cs);if (ret) {return ret; } -vfp_set_fpcr (env, fpr); ret =kvm_get_vcpu_events (cpu);if (ret) {
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH 04/13] target/arm/kvm: Move the get/put of fpsimd registers out
Hi Drew,
Quoted 41 lines · Andrew Jones
Move the getting/putting of the fpsimd registers out of kvm_arch_get/put_registers() into their own helper functions to prepare for alternatively getting/putting SVE registers.
No functional change.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/kvm64.c | 148 +++++++++++++++++++++++++++------------------ 1 file changed, 88 insertions(+), 60 deletions(-)target/arm/kvm64.c +22
-0@@ -706,13 +706,53 @@ int kvm_arm_cpreg_level(uint64_t regidx)#define AARCH64_SIMD_CTRL_REG (x) (KVM_REG_ARM64 | KVM_REG_SIZE_U32 | \ KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x)) +static int kvm_arch_put_fpsimd (CPUState *cs) +{ +ARMCPU *cpu =ARM_CPU (cs); +CPUARMState *env = &cpu->env; +struct kvm_one_reg reg; +uint32_t fpr; +int i, ret; + +for (i =0 ; i <32 ; i++) { +uint64_t *q =aa64_vfp_qreg (env, i); +#ifdef HOST_WORDS_BIGENDIAN +uint64_t fp_val[2 ] = { q[1 ], q[0 ] }; + reg.addr = (uintptr_t )fp_val; +#else + reg.addr = (uintptr_t )q; +#endif + reg.id =AARCH64_SIMD_CORE_REG (fp_regs.vregs[i]); + ret =kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); +if (ret) { +return ret; + } + }
target/arm/kvm64.c int kvm_arm_cpreg_level(uint64_t regidx) · line 740
Quoted 10 lines
target/arm/kvm64.c +10
-0+ + reg.addr = (uintptr_t )(&fpr); + fpr =vfp_get_fpsr (env); + reg.id =AARCH64_SIMD_CTRL_REG (fp_regs.fpsr); + ret =kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); +if (ret) { +return ret; + } + + reg.addr = (uintptr_t )(&fpr);
I don't think you need this assignment
Quoted 82 lines
target/arm/kvm64.c +34
-30+ fpr =vfp_get_fpcr (env); + reg.id =AARCH64_SIMD_CTRL_REG (fp_regs.fpcr); + ret =kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); +if (ret) { +return ret; + } + +return 0 ; +} +int kvm_arch_put_registers (CPUState *cs,int level) {struct kvm_one_reg reg; -uint32_t fpr;uint64_t val; -int i; -int ret; +int i, ret;unsigned int el;ARMCPU *cpu =ARM_CPU (cs);@@ -802,33 +842,7 @@ int kvm_arch_put_registers(CPUState *cs, int level) } } -/* Advanced SIMD and FP registers. */ -for (i =0 ; i <32 ; i++) { -uint64_t *q =aa64_vfp_qreg (env, i); -#ifdef HOST_WORDS_BIGENDIAN -uint64_t fp_val[2 ] = { q[1 ], q[0 ] }; - reg.addr = (uintptr_t )fp_val; -#else - reg.addr = (uintptr_t )q; -#endif - reg.id =AARCH64_SIMD_CORE_REG (fp_regs.vregs[i]); - ret =kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); -if (ret) { -return ret; - } - } - - reg.addr = (uintptr_t )(&fpr); - fpr =vfp_get_fpsr (env); - reg.id =AARCH64_SIMD_CTRL_REG (fp_regs.fpsr); - ret =kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); -if (ret) { -return ret; - } - - fpr =vfp_get_fpcr (env); - reg.id =AARCH64_SIMD_CTRL_REG (fp_regs.fpcr); - ret =kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); + ret =kvm_arch_put_fpsimd (cs);if (ret) {return ret; }@@ -849,14 +863,54 @@ int kvm_arch_put_registers(CPUState *cs, int level)return ret; } +static int kvm_arch_get_fpsimd (CPUState *cs) +{ +ARMCPU *cpu =ARM_CPU (cs); +CPUARMState *env = &cpu->env; +struct kvm_one_reg reg; +uint32_t fpr; +int i, ret; + +for (i =0 ; i <32 ; i++) { +uint64_t *q =aa64_vfp_qreg (env, i); + reg.id =AARCH64_SIMD_CORE_REG (fp_regs.vregs[i]); + reg.addr = (uintptr_t )q; + ret =kvm_vcpu_ioctl (cs,KVM_GET_ONE_REG , ®); +if (ret) { +return ret; + }else { +#ifdef HOST_WORDS_BIGENDIAN +uint64_t t; + t = q[0 ], q[0 ] = q[1 ], q[1 ] = t; +#endif + } + }
target/arm/kvm64.c int kvm_arch_put_registers(CPUState *cs, int level) · line 897
Quoted 10 lines
target/arm/kvm64.c +10
-0+ + reg.addr = (uintptr_t )(&fpr); + reg.id =AARCH64_SIMD_CTRL_REG (fp_regs.fpsr); + ret =kvm_vcpu_ioctl (cs,KVM_GET_ONE_REG , ®); +if (ret) { +return ret; + } +vfp_set_fpsr (env, fpr); + + reg.addr = (uintptr_t )(&fpr);
same here
Quoted 51 lines
target/arm/kvm64.c +11
-27+ reg.id =AARCH64_SIMD_CTRL_REG (fp_regs.fpcr); + ret =kvm_vcpu_ioctl (cs,KVM_GET_ONE_REG , ®); +if (ret) { +return ret; + } +vfp_set_fpcr (env, fpr); + +return 0 ; +} +int kvm_arch_get_registers (CPUState *cs) {struct kvm_one_reg reg;uint64_t val; -uint32_t fpr;unsigned int el; -int i; -int ret; +int i, ret;ARMCPU *cpu =ARM_CPU (cs);CPUARMState *env = &cpu->env;@@ -945,36 +999,10 @@ int kvm_arch_get_registers(CPUState *cs) env->spsr = env->banked_spsr[i]; } -/* Advanced SIMD and FP registers */ -for (i =0 ; i <32 ; i++) { -uint64_t *q =aa64_vfp_qreg (env, i); - reg.id =AARCH64_SIMD_CORE_REG (fp_regs.vregs[i]); - reg.addr = (uintptr_t )q; - ret =kvm_vcpu_ioctl (cs,KVM_GET_ONE_REG , ®); -if (ret) { -return ret; - }else { -#ifdef HOST_WORDS_BIGENDIAN -uint64_t t; - t = q[0 ], q[0 ] = q[1 ], q[1 ] = t; -#endif - } - } - - reg.addr = (uintptr_t )(&fpr); - reg.id =AARCH64_SIMD_CTRL_REG (fp_regs.fpsr); - ret =kvm_vcpu_ioctl (cs,KVM_GET_ONE_REG , ®); -if (ret) { -return ret; - } -vfp_set_fpsr (env, fpr); -
target/arm/kvm64.c int kvm_arch_get_registers(CPUState *cs) · line 1008
Quoted 10 lines
target/arm/kvm64.c +1
-3- reg.id =AARCH64_SIMD_CTRL_REG (fp_regs.fpcr); - ret =kvm_vcpu_ioctl (cs,KVM_GET_ONE_REG , ®); + ret =kvm_arch_get_fpsimd (cs);if (ret) {return ret; } -vfp_set_fpcr (env, fpr); ret =kvm_get_vcpu_events (cpu);if (ret) {
Besides
Thanks
Eric
Re: [Qemu-devel] [PATCH 04/13] target/arm/kvm: Move the get/put of fpsimd registers out
Quoted 47 lines · Auger Eric
Hi Drew,
Quoted 44 lines · Andrew Jones
Move the getting/putting of the fpsimd registers out of kvm_arch_get/put_registers() into their own helper functions to prepare for alternatively getting/putting SVE registers.
No functional change.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/kvm64.c | 148 +++++++++++++++++++++++++++------------------ 1 file changed, 88 insertions(+), 60 deletions(-)target/arm/kvm64.c +25
-0@@ -706,13 +706,53 @@ int kvm_arm_cpreg_level(uint64_t regidx)#define AARCH64_SIMD_CTRL_REG (x) (KVM_REG_ARM64 | KVM_REG_SIZE_U32 | \ KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x)) +static int kvm_arch_put_fpsimd (CPUState *cs) +{ +ARMCPU *cpu =ARM_CPU (cs); +CPUARMState *env = &cpu->env; +struct kvm_one_reg reg; +uint32_t fpr; +int i, ret; + +for (i =0 ; i <32 ; i++) { +uint64_t *q =aa64_vfp_qreg (env, i); +#ifdef HOST_WORDS_BIGENDIAN +uint64_t fp_val[2 ] = { q[1 ], q[0 ] }; + reg.addr = (uintptr_t )fp_val; +#else + reg.addr = (uintptr_t )q; +#endif + reg.id =AARCH64_SIMD_CORE_REG (fp_regs.vregs[i]); + ret =kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); +if (ret) { +return ret; + } + } + + reg.addr = (uintptr_t )(&fpr); + fpr =vfp_get_fpsr (env);
Quoted 8 lines
target/arm/kvm64.c int kvm_arm_cpreg_level(uint64_t regidx) · line 740
Quoted 7 lines
target/arm/kvm64.c +7
-0+ reg.id =AARCH64_SIMD_CTRL_REG (fp_regs.fpsr); + ret =kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); +if (ret) { +return ret; + } + + reg.addr = (uintptr_t )(&fpr);I don't think you need this assignment
You're right, and they weren't in the original code, but I added them in order to be sure that the register get/set blocks were complete units. It's one thing to factor stuff like that out of a loop, but here we had two almost independent blocks so it looked a bit odd.
[...]
Quoted 4 lines
Besides
Reviewed-by Eric Auger <[email protected]>
Thanks,
drew
PATCH 5/13 target/arm/kvm: Add kvm_arch_get/put_sve
In reply to Andrew JonesThese are the SVE equivalents to kvm_arch_get/put_fpsimd.
--- target/arm/kvm64.c | 127 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 123 insertions(+), 4 deletions(-)
target/arm/kvm64.c +123 -4
@@ -658,11 +658,12 @@ int kvm_arch_init_vcpu(CPUState *cs)bool kvm_arm_reg_syncs_via_cpreg_list (uint64_t regidx) {/* Return true if the regidx is a register we should synchronize -* via the cpreg_tuples array (ie is not a core reg we sync by -* hand in kvm_arch_get/put_registers()) +* via the cpreg_tuples array (ie is not a core or sve reg that +* we sync by hand in kvm_arch_get/put_registers()) */ switch (regidx &KVM_REG_ARM_COPROC_MASK ) {case KVM_REG_ARM_CORE : +case KVM_REG_ARM64_SVE :return false;default :return true;
@@ -748,6 +749,61 @@ static int kvm_arch_put_fpsimd(CPUState *cs)return 0 ; } +static int kvm_arch_put_sve (CPUState *cs) +{ +ARMCPU *cpu =ARM_CPU (cs); +CPUARMState *env = &cpu->env; +struct kvm_one_reg reg; +int n, ret; + +for (n =0 ; n <KVM_ARM64_SVE_NUM_ZREGS ; n++) { +uint64_t *q =aa64_vfp_qreg (env, n); +#ifdef HOST_WORDS_BIGENDIAN +uint64_t d[ARM_MAX_VQ *2 ]; +int i; +for (i =0 ; i < cpu->sve_max_vq *2 ; i++) { + d[i] = q[cpu->sve_max_vq *2 -1 - i]; + } + reg.addr = (uintptr_t )d; +#else + reg.addr = (uintptr_t )q; +#endif + reg.id =KVM_REG_ARM64_SVE_ZREG (n,0 ); + ret =kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); +if (ret) { +return ret; + } + } + +for (n =0 ; n <KVM_ARM64_SVE_NUM_PREGS ; n++) { +uint64_t *p = &env->vfp.pregs[n].p[0 ]; +#ifdef HOST_WORDS_BIGENDIAN +uint64_t d[ARM_MAX_VQ *2 ]; +int i; +for (i =0 ; i < cpu->sve_max_vq *2 /8 ; i++) { + d[i] = p[cpu->sve_max_vq *2 /8 -1 - i]; + } + reg.addr = (uintptr_t )d; +#else + reg.addr = (uintptr_t )p; +#endif + reg.id =KVM_REG_ARM64_SVE_PREG (n,0 ); + ret =kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); +if (ret) { +return ret; + } + } + + reg.addr = (uintptr_t )&env->vfp.pregs[FFR_PRED_NUM ].p[0 ]; + reg.id =KVM_REG_ARM64_SVE_FFR (0 ); + ret =kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); +if (ret) { +return ret; + } + +return 0 ; +} +int kvm_arch_put_registers (CPUState *cs,int level) {struct kvm_one_reg reg;
@@ -842,7 +898,11 @@ int kvm_arch_put_registers(CPUState *cs, int level) } } - ret =kvm_arch_put_fpsimd (cs); +if (!cpu_isar_feature (aa64_sve, cpu)) { + ret =kvm_arch_put_fpsimd (cs); + }else { + ret =kvm_arch_put_sve (cs); + }if (ret) {return ret; }
@@ -905,6 +965,61 @@ static int kvm_arch_get_fpsimd(CPUState *cs)return 0 ; } +static int kvm_arch_get_sve (CPUState *cs) +{ +ARMCPU *cpu =ARM_CPU (cs); +CPUARMState *env = &cpu->env; +struct kvm_one_reg reg; +int n, ret; + +for (n =0 ; n <KVM_ARM64_SVE_NUM_ZREGS ; n++) { +uint64_t *q =aa64_vfp_qreg (env, n); + reg.id =KVM_REG_ARM64_SVE_ZREG (n,0 ); + reg.addr = (uintptr_t )q; + ret =kvm_vcpu_ioctl (cs,KVM_GET_ONE_REG , ®); +if (ret) { +return ret; + }else { +#ifdef HOST_WORDS_BIGENDIAN +int i =0 , j = cpu->sve_max_vq *2 -1 ; +while (i < j) { +uint64_t t; + t = q[i], q[i] = q[j], q[j] = t; + ++i, --j; + } +#endif + } + } + +for (n =0 ; n <KVM_ARM64_SVE_NUM_PREGS ; n++) { +uint64_t *p = &env->vfp.pregs[n].p[0 ]; + reg.id =KVM_REG_ARM64_SVE_PREG (n,0 ); + reg.addr = (uintptr_t )p; + ret =kvm_vcpu_ioctl (cs,KVM_GET_ONE_REG , ®); +if (ret) { +return ret; + }else { +#ifdef HOST_WORDS_BIGENDIAN +int i =0 , j = cpu->sve_max_vq *2 /8 -1 ; +while (i < j) { +uint64_t t; + t = q[i], q[i] = q[j], q[j] = t; + ++i, --j; + } +#endif + } + } + + reg.addr = (uintptr_t )&env->vfp.pregs[FFR_PRED_NUM ].p[0 ]; + reg.id =KVM_REG_ARM64_SVE_FFR (0 ); + ret =kvm_vcpu_ioctl (cs,KVM_GET_ONE_REG , ®); +if (ret) { +return ret; + } + +return 0 ; +} +int kvm_arch_get_registers (CPUState *cs) {struct kvm_one_reg reg;
@@ -999,7 +1114,11 @@ int kvm_arch_get_registers(CPUState *cs) env->spsr = env->banked_spsr[i]; } - ret =kvm_arch_get_fpsimd (cs); +if (!cpu_isar_feature (aa64_sve, cpu)) { + ret =kvm_arch_get_fpsimd (cs); + }else { + ret =kvm_arch_get_sve (cs); + }if (ret) {return ret; }
Signature · 1 line
2.20.1
Re: [PATCH 05/13] target/arm/kvm: Add kvm_arch_get/put_sve
Quoted 6 lines · Andrew Jones
These are the SVE equivalents to kvm_arch_get/put_fpsimd.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/kvm64.c | 127 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 123 insertions(+), 4 deletions(-)
[...]
Quoted 5 lines
target/arm/kvm64.c +5
-0+static int kvm_arch_put_sve (CPUState *cs) +{ +ARMCPU *cpu =ARM_CPU (cs); +CPUARMState *env = &cpu->env; +struct kvm_one_reg reg;
target/arm/kvm64.c static int kvm_arch_put_fpsimd(CPUState *cs) · line 766
Quoted 10 lines
target/arm/kvm64.c +10
-0+int n, ret; + +for (n =0 ; n <KVM_ARM64_SVE_NUM_ZREGS ; n++) { +uint64_t *q =aa64_vfp_qreg (env, n); +#ifdef HOST_WORDS_BIGENDIAN +uint64_t d[ARM_MAX_VQ *2 ]; +int i; +for (i =0 ; i < cpu->sve_max_vq *2 ; i++) { + d[i] = q[cpu->sve_max_vq *2 -1 - i]; + }
Out of interest, why do all this swabbing? It seems expensive.
Cheers
---Dave
Re: [Qemu-devel] [PATCH 05/13] target/arm/kvm: Add kvm_arch_get/put_sve
Quoted 19 lines · Dave Martin
Quoted 6 lines · Andrew Jones
These are the SVE equivalents to kvm_arch_get/put_fpsimd.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/kvm64.c | 127 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 123 insertions(+), 4 deletions(-)[...]
Quoted 9 lines
target/arm/kvm64.c +9
-0+static int kvm_arch_put_sve (CPUState *cs) +{ +ARMCPU *cpu =ARM_CPU (cs); +CPUARMState *env = &cpu->env; +struct kvm_one_reg reg; +int n, ret; + +for (n =0 ; n <KVM_ARM64_SVE_NUM_ZREGS ; n++) { +uint64_t *q =aa64_vfp_qreg (env, n);
Quoted 9 lines
target/arm/kvm64.c static int kvm_arch_put_fpsimd(CPUState *cs) · line 766
Quoted 6 lines
target/arm/kvm64.c +6
-0+#ifdef HOST_WORDS_BIGENDIAN +uint64_t d[ARM_MAX_VQ *2 ]; +int i; +for (i =0 ; i < cpu->sve_max_vq *2 ; i++) { + d[i] = q[cpu->sve_max_vq *2 -1 - i]; + }Out of interest, why do all this swabbing? It seems expensive.
QEMU keeps its 128-bit and larger words in the same order (least significant word first) for both host endian types. We need to do word swapping every time we set/get them to/from KVM.
Thanks,
drew
Re: [Qemu-devel] [PATCH 05/13] target/arm/kvm: Add kvm_arch_get/put_sve
Quoted 30 lines · Andrew Jones
Quoted 19 lines · Dave Martin
Quoted 6 lines · Andrew Jones
These are the SVE equivalents to kvm_arch_get/put_fpsimd.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/kvm64.c | 127 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 123 insertions(+), 4 deletions(-)[...]
Quoted 9 lines
target/arm/kvm64.c +9
-0+static int kvm_arch_put_sve (CPUState *cs) +{ +ARMCPU *cpu =ARM_CPU (cs); +CPUARMState *env = &cpu->env; +struct kvm_one_reg reg; +int n, ret; + +for (n =0 ; n <KVM_ARM64_SVE_NUM_ZREGS ; n++) { +uint64_t *q =aa64_vfp_qreg (env, n);Quoted 9 lines
target/arm/kvm64.c static int kvm_arch_put_fpsimd(CPUState *cs) · line 766
Quoted 6 lines
target/arm/kvm64.c +6
-0+#ifdef HOST_WORDS_BIGENDIAN +uint64_t d[ARM_MAX_VQ *2 ]; +int i; +for (i =0 ; i < cpu->sve_max_vq *2 ; i++) { + d[i] = q[cpu->sve_max_vq *2 -1 - i]; + }Out of interest, why do all this swabbing? It seems expensive.
Quoted 3 lines
QEMU keeps its 128-bit and larger words in the same order (least significant word first) for both host endian types. We need to do word swapping every time we set/get them to/from KVM.
I'm not sure whether this is appropriate here, though it depends on what QEMU does with the data.
Something non-obvious to be aware of:
As exposed through the signal frame and the KVM ABI, the memory representation of an SVE reg is invariant with respect to the endianness.
IIUC, the byte order seen for a V-reg in KVM_REG_ARM_CORE and for the equivalent Z-reg in KVM_REG_ARM64_SVE would be the opposite of each other on BE, but the same on LE.
This is a feature of the archtiecture: a V-reg can be stored as a single value, but Z-regs are in general too big to be treated as a single
value: they are always treated as a sequence of elements, and the largest element size supported is 64 bits, not 128. IIUC, there is no direct native way to store with 128-bit swabbing: some explicit data processing operation would also be needed to swap adjacent 64-bit elements in the vector around the store/load.
This is not specified in the ABI documentation -- I should address that.
If this is infeasible for KVM to work with, we could perhaps change it, but I'm not too keen on that at this stage. KVM_REG_ARM64_SVE_VLS has a similar behaviour: it's a vector of 64-bit possibly-swabbed words, not a single possibly-swabbed 512-bit word.
Looking at the kernel, I may have screwed up in places where the two representations interact, like fpsimd_to_sve(). I should take a look at that... This doesn't affect the KVM ABI though.
Cheers
---Dave
Re: [Qemu-devel] [PATCH 05/13] target/arm/kvm: Add kvm_arch_get/put_sve
Quoted 8 lines · Dave Martin
Quoted 3 lines · Andrew Jones
QEMU keeps its 128-bit and larger words in the same order (least significant word first) for both host endian types. We need to do word swapping every time we set/get them to/from KVM.
I'm not sure whether this is appropriate here, though it depends on what QEMU does with the data.
The layout is optimised for TCG emulation to be able to work with it, I think (rth would have the definite reason, though).
Quoted 5 lines
Something non-obvious to be aware of:
As exposed through the signal frame and the KVM ABI, the memory representation of an SVE reg is invariant with respect to the endianness.
Yes; we handle this conversion as we write out the signal frame:
https://github.com/qemu/qemu/blob/master/linux-user/aarch64/signal.c#L184
thanks
-- PMM
Re: [Qemu-devel] [PATCH 05/13] target/arm/kvm: Add kvm_arch_get/put_sve
Quoted 7 lines · Peter Maydell
Quoted 6 lines · Dave Martin
Quoted 3 lines · Andrew Jones
QEMU keeps its 128-bit and larger words in the same order (least significant word first) for both host endian types. We need to do word swapping every time we set/get them to/from KVM.
Quoted 6 lines
Quoted 2 lines
I'm not sure whether this is appropriate here, though it depends on what QEMU does with the data.
The layout is optimised for TCG emulation to be able to work with it, I think (rth would have the definite reason, though).
So long as we are agreed about the ABI, this is none of my concern :)
Quoted 8 lines
Quoted 5 lines
Something non-obvious to be aware of:
As exposed through the signal frame and the KVM ABI, the memory representation of an SVE reg is invariant with respect to the endianness.
Yes; we handle this conversion as we write out the signal frame:
https://github.com/qemu/qemu/blob/master/linux-user/aarch64/signal.c#L184
Right. I hadn't focused consciously on this, since the architecture does the work for us in the kernel (mostly).
I will check the documentation to make sure the behaviour is clearly described, anyhow.
Cheers
---Dave
Re: [PATCH 05/13] target/arm/kvm: Add kvm_arch_get/put_sve
In reply to Dave MartinQuoted 19 lines · Dave Martin
Quoted 6 lines · Andrew Jones
These are the SVE equivalents to kvm_arch_get/put_fpsimd.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/kvm64.c | 127 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 123 insertions(+), 4 deletions(-)[...]
Quoted 9 lines
target/arm/kvm64.c +9
-0+static int kvm_arch_put_sve (CPUState *cs) +{ +ARMCPU *cpu =ARM_CPU (cs); +CPUARMState *env = &cpu->env; +struct kvm_one_reg reg; +int n, ret; + +for (n =0 ; n <KVM_ARM64_SVE_NUM_ZREGS ; n++) { +uint64_t *q =aa64_vfp_qreg (env, n);
Quoted 8 lines
target/arm/kvm64.c static int kvm_arch_put_fpsimd(CPUState *cs) · line 766
Quoted 6 lines
target/arm/kvm64.c +6
-0+#ifdef HOST_WORDS_BIGENDIAN +uint64_t d[ARM_MAX_VQ *2 ]; +int i; +for (i =0 ; i < cpu->sve_max_vq *2 ; i++) { + d[i] = q[cpu->sve_max_vq *2 -1 - i]; + }Out of interest, why do all this swabbing? It seems expensive.
Indeed, to me this seems to be the wrong kind of swabbing here. Exactly what format is KVM expecting? Surely it should be the one used by the unpredicated LDR/STR instructions. Anything else would seem to be working against the architecture.
If so, the format is, architecturally, a stream of bytes in index order, which corresponds to a little-endian stream of words. So the loop I'd expect to see here is
for (i = 0, n = cpu->sve_max_vq; i < n; ++i) {
d[i] = bswap64(q[i]);
}r~
Re: [PATCH 05/13] target/arm/kvm: Add kvm_arch_get/put_sve
Quoted 34 lines · Richard Henderson
Quoted 19 lines · Dave Martin
Quoted 6 lines · Andrew Jones
These are the SVE equivalents to kvm_arch_get/put_fpsimd.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/kvm64.c | 127 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 123 insertions(+), 4 deletions(-)[...]
Quoted 9 lines
target/arm/kvm64.c +9
-0+static int kvm_arch_put_sve (CPUState *cs) +{ +ARMCPU *cpu =ARM_CPU (cs); +CPUARMState *env = &cpu->env; +struct kvm_one_reg reg; +int n, ret; + +for (n =0 ; n <KVM_ARM64_SVE_NUM_ZREGS ; n++) { +uint64_t *q =aa64_vfp_qreg (env, n);Quoted 8 lines
target/arm/kvm64.c static int kvm_arch_put_fpsimd(CPUState *cs) · line 766
Quoted 6 lines
target/arm/kvm64.c +6
-0+#ifdef HOST_WORDS_BIGENDIAN +uint64_t d[ARM_MAX_VQ *2 ]; +int i; +for (i =0 ; i < cpu->sve_max_vq *2 ; i++) { + d[i] = q[cpu->sve_max_vq *2 -1 - i]; + }Out of interest, why do all this swabbing? It seems expensive.
Indeed, to me this seems to be the wrong kind of swabbing here. Exactly what format is KVM expecting? Surely it should be the one used by the unpredicated LDR/STR instructions. Anything else would seem to be working against the architecture.
Quoted 7 lines
If so, the format is, architecturally, a stream of bytes in index order, which corresponds to a little-endian stream of words. So the loop I'd expect to see here is
for (i = 0, n = cpu->sve_max_vq; i < n; ++i) { d[i] = bswap64(q[i]); }
That's the opposite of what we do for fpsimd registers though. I'm fine with doing whatever KVM/TCG needs, but so far I was just following the same pattern we already have.
Thanks,
drew
Re: [PATCH 05/13] target/arm/kvm: Add kvm_arch_get/put_sve
Quoted 9 lines · Andrew Jones
Quoted 9 lines
Indeed, to me this seems to be the wrong kind of swabbing here. Exactly what format is KVM expecting? Surely it should be the one used by the unpredicated LDR/STR instructions. Anything else would seem to be working against the architecture.
If so, the format is, architecturally, a stream of bytes in index order, which corresponds to a little-endian stream of words. So the loop I'd expect to see here is
Quoted 7 lines
Quoted 3 lines
for (i = 0, n = cpu->sve_max_vq; i < n; ++i) { d[i] = bswap64(q[i]); }That's the opposite of what we do for fpsimd registers though. I'm fine with doing whatever KVM/TCG needs, but so far I was just following the same pattern we already have.
The behaviour of the hardware is different for LDR of fpsimd registers.
FP&SIMD LDR operates on datasize (8, 16, 32, 64, 128 bits). SVE LDR always operates on bytes.
r~
Re: [PATCH 05/13] target/arm/kvm: Add kvm_arch_get/put_sve
In reply to Andrew JonesQuoted 6 lines · Andrew Jones
These are the SVE equivalents to kvm_arch_get/put_fpsimd.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/kvm64.c | 127 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 123 insertions(+), 4 deletions(-)
[...]
Quoted 10 lines
target/arm/kvm64.c +10
-0+static int kvm_arch_put_sve (CPUState *cs) +{ +ARMCPU *cpu =ARM_CPU (cs); +CPUARMState *env = &cpu->env; +struct kvm_one_reg reg; +int n, ret; + +for (n =0 ; n <KVM_ARM64_SVE_NUM_ZREGS ; n++) { +uint64_t *q =aa64_vfp_qreg (env, n); +#ifdef HOST_WORDS_BIGENDIAN
target/arm/kvm64.c static int kvm_arch_put_fpsimd(CPUState *cs) · line 771
Quoted 10 lines
target/arm/kvm64.c +10
-0+uint64_t d[ARM_MAX_VQ *2 ]; +int i; +for (i =0 ; i < cpu->sve_max_vq *2 ; i++) { + d[i] = q[cpu->sve_max_vq *2 -1 - i]; + } + reg.addr = (uintptr_t )d; +#else + reg.addr = (uintptr_t )q; +#endif + reg.id =KVM_REG_ARM64_SVE_ZREG (n,0 );
Will this silently go wrong if more than one slice is required (i.e., the register size grows beyond 8192 bits?)
[...]
Cheers
---Dave
Re: [Qemu-devel] [PATCH 05/13] target/arm/kvm: Add kvm_arch_get/put_sve
Quoted 25 lines · Dave Martin
Quoted 6 lines · Andrew Jones
These are the SVE equivalents to kvm_arch_get/put_fpsimd.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/kvm64.c | 127 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 123 insertions(+), 4 deletions(-)[...]
Quoted 15 lines
target/arm/kvm64.c +15
-0+static int kvm_arch_put_sve (CPUState *cs) +{ +ARMCPU *cpu =ARM_CPU (cs); +CPUARMState *env = &cpu->env; +struct kvm_one_reg reg; +int n, ret; + +for (n =0 ; n <KVM_ARM64_SVE_NUM_ZREGS ; n++) { +uint64_t *q =aa64_vfp_qreg (env, n); +#ifdef HOST_WORDS_BIGENDIAN +uint64_t d[ARM_MAX_VQ *2 ]; +int i; +for (i =0 ; i < cpu->sve_max_vq *2 ; i++) { + d[i] = q[cpu->sve_max_vq *2 -1 - i]; + }
Quoted 8 lines
target/arm/kvm64.c static int kvm_arch_put_fpsimd(CPUState *cs) · line 771
Quoted 5 lines
target/arm/kvm64.c +5
-0+ reg.addr = (uintptr_t )d; +#else + reg.addr = (uintptr_t )q; +#endif + reg.id =KVM_REG_ARM64_SVE_ZREG (n,0 );Will this silently go wrong if more than one slice is required (i.e., the register size grows beyond 8192 bits?)
Yeah, I could probably implement the slice loop now and add a function that returns 1 (for now) like your vcpu_sve_slices() function in KVM. I'll do that for v2.
Thanks,
drew
Quoted 6 lines
[...]
Cheers
---Dave
Re: [Qemu-devel] [PATCH 05/13] target/arm/kvm: Add kvm_arch_get/put_sve
Quoted 35 lines · Andrew Jones
Quoted 25 lines · Dave Martin
Quoted 6 lines · Andrew Jones
These are the SVE equivalents to kvm_arch_get/put_fpsimd.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/kvm64.c | 127 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 123 insertions(+), 4 deletions(-)[...]
Quoted 15 lines
target/arm/kvm64.c +15
-0+static int kvm_arch_put_sve (CPUState *cs) +{ +ARMCPU *cpu =ARM_CPU (cs); +CPUARMState *env = &cpu->env; +struct kvm_one_reg reg; +int n, ret; + +for (n =0 ; n <KVM_ARM64_SVE_NUM_ZREGS ; n++) { +uint64_t *q =aa64_vfp_qreg (env, n); +#ifdef HOST_WORDS_BIGENDIAN +uint64_t d[ARM_MAX_VQ *2 ]; +int i; +for (i =0 ; i < cpu->sve_max_vq *2 ; i++) { + d[i] = q[cpu->sve_max_vq *2 -1 - i]; + }Quoted 8 lines
target/arm/kvm64.c static int kvm_arch_put_fpsimd(CPUState *cs) · line 771
Quoted 5 lines
target/arm/kvm64.c +5
-0+ reg.addr = (uintptr_t )d; +#else + reg.addr = (uintptr_t )q; +#endif + reg.id =KVM_REG_ARM64_SVE_ZREG (n,0 );Will this silently go wrong if more than one slice is required (i.e., the register size grows beyond 8192 bits?)
Quoted 3 lines
Yeah, I could probably implement the slice loop now and add a function that returns 1 (for now) like your
vcpu_sve_slices()function in KVM. I'll do that for v2.
Or just add a sanity check that the vector length is <= 2048 bits.
Support for larger vectors is untestable for now, since the kernel doesn't support that and would never expose it.
On that point, could TCG easily be made to expose a larger vector length to the kernel? I'd be interested to see what happened.
The kernel should warn and hide the larger vector lengths from KVM, but I've not been able to test that.
It's only worth trying this out if the hacks to QEMU to enable testing this were pretty trivial, though.
Cheers
---Dave
Re: [Qemu-devel] [PATCH 05/13] target/arm/kvm: Add kvm_arch_get/put_sve
Quoted 2 lines · Dave Martin
On that point, could TCG easily be made to expose a larger vector length to the kernel? I'd be interested to see what happened.
It would be easy enough to extend the maximum vector length within TCG.
Increase ARM_MAX_VQ. Alter the couple of places where we manipulate ZCR.LEN to extend the current 4-bit mask.
How large do you need the max to be, for testing?
r~
Re: [Qemu-devel] [PATCH 05/13] target/arm/kvm: Add kvm_arch_get/put_sve
Quoted 10 lines · Richard Henderson
Quoted 2 lines · Dave Martin
On that point, could TCG easily be made to expose a larger vector length to the kernel? I'd be interested to see what happened.
It would be easy enough to extend the maximum vector length within TCG.
Increase
ARM_MAX_VQ. Alter the couple of places where we manipulate ZCR.LEN to extend the current 4-bit mask.How large do you need the max to be, for testing?
Anything upwards of 256 bytes is interesting.
The architecture reserves space for it to grow up to 9 bits, though it's unlikely it would ever get that large in reality.
So if you wanted to go crazy, you might be able to go up to 8192 bytes.
This is just for fun, since it goes outside the architecture and Linux officially doesn't support it today in any case. So definitely not a priority!
Cheers
---Dave
PATCH 6/13 target/arm/kvm: max cpu: Enable SVE when available
In reply to Andrew JonesEnable SVE in the KVM guest when the 'max' cpu type is configured and KVM supports it. KVM SVE requires use of the new finalize vcpu ioctl, so we add that now too.
--- target/arm/cpu64.c | 1 + target/arm/kvm.c | 5 +++++ target/arm/kvm64.c | 16 +++++++++++++++- target/arm/kvm_arm.h | 12 ++++++++++++ 4 files changed, 33 insertions(+), 1 deletion(-)
target/arm/cpu64.c +1 -0
@@ -292,6 +292,7 @@ static void aarch64_max_initfn(Object *obj)if (kvm_enabled ()) {kvm_arm_set_cpu_features_from_host (cpu); + cpu->sve_max_vq =ARM_MAX_VQ ; }else {uint64_t t;uint32_t u;
target/arm/kvm.c +5 -0
@@ -50,6 +50,11 @@ int kvm_arm_vcpu_init(CPUState *cs)return kvm_vcpu_ioctl (cs,KVM_ARM_VCPU_INIT , &init); } +int kvm_arm_vcpu_finalize (CPUState *cs,int feature) +{ +return kvm_vcpu_ioctl (cs,KVM_ARM_VCPU_FINALIZE , &feature); +} +void kvm_arm_init_serror_injection (CPUState *cs) { cap_has_inject_serror_esr = kvm_check_extension(cs->kvm_state,
target/arm/kvm64.c +15 -1
@@ -622,13 +622,20 @@ int kvm_arch_init_vcpu(CPUState *cs) cpu->kvm_init_features[0 ] |=1 <<KVM_ARM_VCPU_EL1_32BIT ; }if (!kvm_check_extension (cs->kvm_state,KVM_CAP_ARM_PMU_V3 )) { - cpu->has_pmu = false; + cpu->has_pmu = false; }if (cpu->has_pmu) { cpu->kvm_init_features[0 ] |=1 <<KVM_ARM_VCPU_PMU_V3 ; }else {unset_feature (&env->features,ARM_FEATURE_PMU ); } +if (cpu->sve_max_vq) { +if (!kvm_check_extension (cs->kvm_state,KVM_CAP_ARM_SVE )) { + cpu->sve_max_vq =0 ; + }else { + cpu->kvm_init_features[0 ] |=1 <<KVM_ARM_VCPU_SVE ; + } + }/* Do KVM_ARM_VCPU_INIT ioctl */ ret =kvm_arm_vcpu_init (cs);
@@ -636,6 +643,13 @@ int kvm_arch_init_vcpu(CPUState *cs)return ret; } +if (cpu->sve_max_vq) { + ret =kvm_arm_vcpu_finalize (cs,KVM_ARM_VCPU_SVE ); +if (ret) { +return ret; + } + } +/* * When KVM is in use, PSCI is emulated in-kernel and not by qemu. * Currently KVM has its own idea about MPIDR assignment, so we
target/arm/kvm_arm.h +12 -0
@@ -27,6 +27,18 @@*/ int kvm_arm_vcpu_init (CPUState *cs); +/** +* kvm_arm_vcpu_finalize +* @cs: CPUState +* @feature: int +* +* Finalizes the configuration of the specified VCPU feature +* by invoking the KVM_ARM_VCPU_FINALIZE ioctl. +* +* Returns: 0 if success else < 0 error code +*/ +int kvm_arm_vcpu_finalize (CPUState *cs,int feature); +/** * kvm_arm_register_device: * @mr: memory region for this device
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH 06/13] target/arm/kvm: max cpu: Enable SVE when available
Hi Drew,
Quoted 17 lines · Andrew Jones
Enable SVE in the KVM guest when the 'max' cpu type is configured and KVM supports it. KVM SVE requires use of the new finalize vcpu ioctl, so we add that now too.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/cpu64.c | 1 + target/arm/kvm.c | 5 +++++ target/arm/kvm64.c | 16 +++++++++++++++- target/arm/kvm_arm.h | 12 ++++++++++++ 4 files changed, 33 insertions(+), 1 deletion(-)target/arm/cpu64.c +0
-0
target/arm/cpu64.c static void aarch64_max_initfn(Object *obj) · line 295
Quoted 4 lines
target/arm/cpu64.c +1
-0@@ -292,6 +292,7 @@ static void aarch64_max_initfn(Object *obj)if (kvm_enabled ()) {kvm_arm_set_cpu_features_from_host (cpu); + cpu->sve_max_vq =ARM_MAX_VQ ;
same line in the !kvm_enabled path. Maybe you can set the sve_max_vq field in a subsequent patch and just introduce the finalize and capability checking in that patch?
Quoted 24 lines
target/arm/cpu64.c +0
-0}else {uint64_t t;uint32_t u;target/arm/kvm.c +5
-0@@ -50,6 +50,11 @@ int kvm_arm_vcpu_init(CPUState *cs)return kvm_vcpu_ioctl (cs,KVM_ARM_VCPU_INIT , &init); } +int kvm_arm_vcpu_finalize (CPUState *cs,int feature) +{ +return kvm_vcpu_ioctl (cs,KVM_ARM_VCPU_FINALIZE , &feature); +} +void kvm_arm_init_serror_injection (CPUState *cs) { cap_has_inject_serror_esr = kvm_check_extension(cs->kvm_state,target/arm/kvm64.c +0
-0
target/arm/kvm64.c int kvm_arch_init_vcpu(CPUState *cs) · line 625
Quoted 5 lines
target/arm/kvm64.c +1
-1@@ -622,13 +622,20 @@ int kvm_arch_init_vcpu(CPUState *cs) cpu->kvm_init_features[0 ] |=1 <<KVM_ARM_VCPU_EL1_32BIT ; }if (!kvm_check_extension (cs->kvm_state,KVM_CAP_ARM_PMU_V3 )) { - cpu->has_pmu = false; + cpu->has_pmu = false;
nit: maybe document this unrelated indent fix in the commit msg?
Quoted 35 lines
target/arm/kvm64.c +14
-0}if (cpu->has_pmu) { cpu->kvm_init_features[0 ] |=1 <<KVM_ARM_VCPU_PMU_V3 ; }else {unset_feature (&env->features,ARM_FEATURE_PMU ); } +if (cpu->sve_max_vq) { +if (!kvm_check_extension (cs->kvm_state,KVM_CAP_ARM_SVE )) { + cpu->sve_max_vq =0 ; + }else { + cpu->kvm_init_features[0 ] |=1 <<KVM_ARM_VCPU_SVE ; + } + }/* Do KVM_ARM_VCPU_INIT ioctl */ ret =kvm_arm_vcpu_init (cs);@@ -636,6 +643,13 @@ int kvm_arch_init_vcpu(CPUState *cs)return ret; } +if (cpu->sve_max_vq) { + ret =kvm_arm_vcpu_finalize (cs,KVM_ARM_VCPU_SVE ); +if (ret) { +return ret; + } + } +/* * When KVM is in use, PSCI is emulated in-kernel and not by qemu. * Currently KVM has its own idea about MPIDR assignment, so we target/arm/kvm_arm.h +0
-0
target/arm/kvm_arm.h line 33
Quoted 7 lines
target/arm/kvm_arm.h +4
-0@@ -27,6 +27,18 @@*/ int kvm_arm_vcpu_init (CPUState *cs); +/** +* kvm_arm_vcpu_finalize +* @cs: CPUState +* @feature: int
feature bitmap or bit?
Quoted 12 lines
target/arm/kvm_arm.h +8
-0+* +* Finalizes the configuration of the specified VCPU feature +* by invoking the KVM_ARM_VCPU_FINALIZE ioctl. +* +* Returns: 0 if success else < 0 error code +*/ +int kvm_arm_vcpu_finalize (CPUState *cs,int feature); +/** * kvm_arm_register_device: * @mr: memory region for this device
Re: [Qemu-devel] [PATCH 06/13] target/arm/kvm: max cpu: Enable SVE when available
Quoted 21 lines · Auger Eric
Hi Drew,
Quoted 18 lines · Andrew Jones
Enable SVE in the KVM guest when the 'max' cpu type is configured and KVM supports it. KVM SVE requires use of the new finalize vcpu ioctl, so we add that now too.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/cpu64.c | 1 + target/arm/kvm.c | 5 +++++ target/arm/kvm64.c | 16 +++++++++++++++- target/arm/kvm_arm.h | 12 ++++++++++++ 4 files changed, 33 insertions(+), 1 deletion(-)target/arm/cpu64.c +0
-0@@ -292,6 +292,7 @@ static void aarch64_max_initfn(Object *obj)
Quoted 6 lines
target/arm/cpu64.c static void aarch64_max_initfn(Object *obj) · line 295
Quoted 3 lines
target/arm/cpu64.c +1
-0if (kvm_enabled ()) {kvm_arm_set_cpu_features_from_host (cpu); + cpu->sve_max_vq =ARM_MAX_VQ ;same line in the !kvm_enabled path. Maybe you can set the
sve_max_vqfield in a subsequent patch and just introduce the finalize and capability checking in that patch?
This gets changed in a subsequent patch, so factoring now would be wasted code motion. I'm not sure the finalize function is worth its own patch, so I'm inclined to leave this as is.
Quoted 22 lines
Quoted 22 lines
target/arm/cpu64.c +0
-0}else {uint64_t t;uint32_t u;target/arm/kvm.c +5
-0@@ -50,6 +50,11 @@ int kvm_arm_vcpu_init(CPUState *cs)return kvm_vcpu_ioctl (cs,KVM_ARM_VCPU_INIT , &init); } +int kvm_arm_vcpu_finalize (CPUState *cs,int feature) +{ +return kvm_vcpu_ioctl (cs,KVM_ARM_VCPU_FINALIZE , &feature); +} +void kvm_arm_init_serror_injection (CPUState *cs) { cap_has_inject_serror_esr = kvm_check_extension(cs->kvm_state,target/arm/kvm64.c +0
-1--- a/target/arm/kvm64.c
Quoted 8 lines
target/arm/kvm64.c int kvm_arch_init_vcpu(CPUState *cs) · line 625
Quoted 7 lines
+++
b/target/arm/kvm64.ctarget/arm/kvm64.c +1
-1@@ -622,13 +622,20 @@ int kvm_arch_init_vcpu(CPUState *cs) cpu->kvm_init_features[0 ] |=1 <<KVM_ARM_VCPU_EL1_32BIT ; }if (!kvm_check_extension (cs->kvm_state,KVM_CAP_ARM_PMU_V3 )) { - cpu->has_pmu = false; + cpu->has_pmu = false;nit: maybe document this unrelated indent fix in the commit msg?
It's pretty obvious without extra commentary, IMHO.
Quoted 38 lines
Quoted 38 lines
target/arm/kvm64.c +14
-0}if (cpu->has_pmu) { cpu->kvm_init_features[0 ] |=1 <<KVM_ARM_VCPU_PMU_V3 ; }else {unset_feature (&env->features,ARM_FEATURE_PMU ); } +if (cpu->sve_max_vq) { +if (!kvm_check_extension (cs->kvm_state,KVM_CAP_ARM_SVE )) { + cpu->sve_max_vq =0 ; + }else { + cpu->kvm_init_features[0 ] |=1 <<KVM_ARM_VCPU_SVE ; + } + }/* Do KVM_ARM_VCPU_INIT ioctl */ ret =kvm_arm_vcpu_init (cs);@@ -636,6 +643,13 @@ int kvm_arch_init_vcpu(CPUState *cs)return ret; } +if (cpu->sve_max_vq) { + ret =kvm_arm_vcpu_finalize (cs,KVM_ARM_VCPU_SVE ); +if (ret) { +return ret; + } + } +/* * When KVM is in use, PSCI is emulated in-kernel and not by qemu. * Currently KVM has its own idea about MPIDR assignment, so we target/arm/kvm_arm.h +0
-0@@ -27,6 +27,18 @@*/ int kvm_arm_vcpu_init (CPUState *cs);
Quoted 5 lines
target/arm/kvm_arm.h line 33
Quoted 4 lines
target/arm/kvm_arm.h +4
-0+/** +* kvm_arm_vcpu_finalize +* @cs: CPUState +* @feature: int feature bitmap or bit?
Neither. I can improve this by stating these integers must be one of the set defined in the "KVM_ARM_VCPU_FINALIZE" section of kernel doc Documentation/virtual/kvm/api.txt though.
Quoted 4 lines
Quoted 4 lines
target/arm/kvm_arm.h +4
-0+* +* Finalizes the configuration of the specified VCPU feature +* by invoking the KVM_ARM_VCPU_FINALIZE ioctl. +*
Quoted 9 lines
target/arm/kvm_arm.h line 44
Quoted 8 lines
target/arm/kvm_arm.h +4
-0+* Returns: 0 if success else < 0 error code +*/ +int kvm_arm_vcpu_finalize (CPUState *cs,int feature); +/** * kvm_arm_register_device: * @mr: memory region for this device
Thanks,
drew
PATCH 7/13 target/arm/kvm: max cpu: Allow sve max vector length setting
In reply to Andrew JonesAllow the cpu type 'max' sve-max-vq property to work with kvm too. If the property is not specified then the maximum kvm supports is used. If it is specified we check that kvm supports that exact length or error out if it doesn't.
--- target/arm/cpu.h | 4 +++ target/arm/cpu64.c | 7 ++-- target/arm/kvm64.c | 80 ++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 86 insertions(+), 5 deletions(-)
target/arm/cpu.h +4 -0
@@ -3122,6 +3122,10 @@ static inline uint64_t arm_sctlr(CPUARMState *env, int el) } } +static inline int arm_cpu_fls64 (uint64_t v) +{ +return !v ?0 :64 -clz64 (v); +}/* Return true if the processor is in big-endian mode. */ static inline bool arm_cpu_data_is_big_endian (CPUARMState *env)
target/arm/cpu64.c +4 -3
@@ -292,7 +292,7 @@ static void aarch64_max_initfn(Object *obj)if (kvm_enabled ()) {kvm_arm_set_cpu_features_from_host (cpu); - cpu->sve_max_vq =ARM_MAX_VQ ; + cpu->sve_max_vq =-1 ;/* set in kvm_arch_init_vcpu() */ }else {uint64_t t;uint32_t u;
@@ -374,9 +374,10 @@ static void aarch64_max_initfn(Object *obj)#endif cpu->sve_max_vq =ARM_MAX_VQ ; -object_property_add (obj,"sve-max-vq" ,"uint32" , cpu_max_get_sve_vq, - cpu_max_set_sve_vq,NULL ,NULL , &error_fatal); } + +object_property_add (obj,"sve-max-vq" ,"uint32" , cpu_max_get_sve_vq, + cpu_max_set_sve_vq,NULL ,NULL , &error_fatal); }struct ARMCPUInfo {
target/arm/kvm64.c +78 -2
104 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH 07/13] target/arm/kvm: max cpu: Allow sve max vector length setting
target/arm/cpu64.c static void aarch64_max_initfn(Object *obj) · line 295
Quoted 6 lines · Andrew Jones
target/arm/cpu64.c +1
-1@@ -292,7 +292,7 @@ static void aarch64_max_initfn(Object *obj)if (kvm_enabled ()) {kvm_arm_set_cpu_features_from_host (cpu); - cpu->sve_max_vq =ARM_MAX_VQ ; + cpu->sve_max_vq =-1 ;/* set in kvm_arch_init_vcpu() */
Perhaps a better comment is "unspecified, to be finalized in ..."
target/arm/cpu64.c static void aarch64_max_initfn(Object *obj) · line 380
Quoted 2 lines
target/arm/cpu64.c +2
-0+object_property_add (obj,"sve-max-vq" ,"uint32" , cpu_max_get_sve_vq, + cpu_max_set_sve_vq,NULL ,NULL , &error_fatal);
as one can set it here, via the command-line.
target/arm/kvm64.c int kvm_arch_init_vcpu(CPUState *cs) · line 711
Quoted 11 lines
target/arm/kvm64.c +7
-0@@ -644,6 +702,24 @@ int kvm_arch_init_vcpu(CPUState *cs) }if (cpu->sve_max_vq) { +uint64_t sve_vls[KVM_ARM64_SVE_VLS_WORDS ]; + ret =kvm_arm_get_sve_vls (cs, sve_vls); +if (ret <0 ) { +return ret; + } +if (cpu->sve_max_vq ==-1 ) { + cpu->sve_max_vq = ret;
You still cannot set a value larger than ARM_MAX_VQ, as that is the fixed amount of space allocated within the cpu structure.
r~
Re: [Qemu-devel] [PATCH 07/13] target/arm/kvm: max cpu: Allow sve max vector length setting
Quoted 9 lines · Richard Henderson
target/arm/cpu64.c static void aarch64_max_initfn(Object *obj) · line 295
Quoted 6 lines · Andrew Jones
target/arm/cpu64.c +1
-1@@ -292,7 +292,7 @@ static void aarch64_max_initfn(Object *obj)if (kvm_enabled ()) {kvm_arm_set_cpu_features_from_host (cpu); - cpu->sve_max_vq =ARM_MAX_VQ ; + cpu->sve_max_vq =-1 ;/* set in kvm_arch_init_vcpu() */ Perhaps a better comment is "unspecified, to be finalized in ..."
OK
Quoted 9 lines
target/arm/cpu64.c static void aarch64_max_initfn(Object *obj) · line 380
Quoted 3 lines
target/arm/cpu64.c +2
-0+object_property_add (obj,"sve-max-vq" ,"uint32" , cpu_max_get_sve_vq, + cpu_max_set_sve_vq,NULL ,NULL , &error_fatal);as one can set it here, via the command-line.
Quoted 3 lines
(quoted diff) +0
-0@@ -644,6 +702,24 @@ int kvm_arch_init_vcpu(CPUState *cs) }
Quoted 12 lines
target/arm/kvm64.c int kvm_arch_init_vcpu(CPUState *cs) · line 711
Quoted 8 lines
target/arm/kvm64.c +7
-0if (cpu->sve_max_vq) { +uint64_t sve_vls[KVM_ARM64_SVE_VLS_WORDS ]; + ret =kvm_arm_get_sve_vls (cs, sve_vls); +if (ret <0 ) { +return ret; + } +if (cpu->sve_max_vq ==-1 ) { + cpu->sve_max_vq = ret;You still cannot set a value larger than
ARM_MAX_VQ, as that is the fixed amount of space allocated within the cpu structure.
Good point. I should check against ARM_MAX_VQ here.
Thanks,
drew
Re: [Qemu-devel] [PATCH 07/13] target/arm/kvm: max cpu: Allow sve max vector length setting
In reply to Andrew JonesHi Drew,
Quoted 64 lines · Andrew Jones
Allow the cpu type 'max' sve-max-vq property to work with kvm too. If the property is not specified then the maximum kvm supports is used. If it is specified we check that kvm supports that exact length or error out if it doesn't.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/cpu.h | 4 +++ target/arm/cpu64.c | 7 ++-- target/arm/kvm64.c | 80 ++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 86 insertions(+), 5 deletions(-)target/arm/cpu.h +4
-0@@ -3122,6 +3122,10 @@ static inline uint64_t arm_sctlr(CPUARMState *env, int el) } } +static inline int arm_cpu_fls64 (uint64_t v) +{ +return !v ?0 :64 -clz64 (v); +}/* Return true if the processor is in big-endian mode. */ static inline bool arm_cpu_data_is_big_endian (CPUARMState *env)target/arm/cpu64.c +4
-3@@ -292,7 +292,7 @@ static void aarch64_max_initfn(Object *obj)if (kvm_enabled ()) {kvm_arm_set_cpu_features_from_host (cpu); - cpu->sve_max_vq =ARM_MAX_VQ ; + cpu->sve_max_vq =-1 ;/* set in kvm_arch_init_vcpu() */ }else {uint64_t t;uint32_t u;@@ -374,9 +374,10 @@ static void aarch64_max_initfn(Object *obj)#endif cpu->sve_max_vq =ARM_MAX_VQ ; -object_property_add (obj,"sve-max-vq" ,"uint32" , cpu_max_get_sve_vq, - cpu_max_set_sve_vq,NULL ,NULL , &error_fatal); } + +object_property_add (obj,"sve-max-vq" ,"uint32" , cpu_max_get_sve_vq, + cpu_max_set_sve_vq, NULL, NULL, &error_fatal);> }struct ARMCPUInfo {target/arm/kvm64.c +4
-0@@ -446,6 +446,59 @@ void kvm_arm_pmu_set_irq(CPUState *cs, int irq) } } +static int kvm_arm_get_sve_vls (CPUState *cs,uint64_t sve_vls[]) +{ +struct kvm_one_reg reg = { + .id =KVM_REG_ARM64_SVE_VLS ,
target/arm/kvm64.c void kvm_arm_pmu_set_irq(CPUState *cs, int irq) · line 462
Quoted 10 lines
target/arm/kvm64.c +10
-0+ .addr = (uint64_t)&sve_vls[0], + }; +int i, ret; + + ret =kvm_vcpu_ioctl (cs,KVM_GET_ONE_REG , ®); +if (ret) { +return ret; + } + + ret =0 ;
is it mandated?
Quoted 11 lines
target/arm/kvm64.c +11
-0+for (i =KVM_ARM64_SVE_VLS_WORDS -1 ; i >=0 ; --i) { +if (sve_vls[i]) { + ret =arm_cpu_fls64 (sve_vls[i]) + i *64 ; +break ; + } + } + +return ret; +} + +static int kvm_arm_set_sve_vls (CPUState *cs,uint64_t sve_vls[],int max_vq)
target/arm/kvm64.c void kvm_arm_pmu_set_irq(CPUState *cs, int irq) · line 483
Quoted 10 lines
target/arm/kvm64.c +10
-0+{ +struct kvm_one_reg reg = { + .id =KVM_REG_ARM64_SVE_VLS , + .addr = (uint64_t )&sve_vls[0 ], + }; +int i; + +for (i =KVM_ARM64_SVE_VLS_WORDS -1 ; i >=0 ; --i) { +if (sve_vls[i]) { +int vq =arm_cpu_fls64 (sve_vls[i]) + i *64 ;
nit: add a line
target/arm/kvm64.c void kvm_arm_pmu_set_irq(CPUState *cs, int irq) · line 485
Quoted 2 lines
target/arm/kvm64.c +2
-0+while (vq > max_vq) { + sve_vls[i] &= ~BIT_MASK (vq -1 );
Isn't BIT_MASK for 32b. MAKE_64BIT_MASK?
target/arm/kvm64.c void kvm_arm_pmu_set_irq(CPUState *cs, int irq) · line 488
Quoted 3 lines
target/arm/kvm64.c +3
-0+ vq =arm_cpu_fls64 (sve_vls[i]) + i *64 ; + } +if (vq < max_vq) {
I don't really get this check: having vq less than max_vq does not seems weird. Do you absolutely want vq=max_vq?
target/arm/kvm64.c void kvm_arm_pmu_set_irq(CPUState *cs, int irq) · line 490
Quoted 2 lines
target/arm/kvm64.c +2
-0+error_report ("sve-max-vq=%d is not a valid length" , max_vq); +error_printf ("next lowest is %d\n" , vq);
why mixing error_report/printf?
Quoted 37 lines
target/arm/kvm64.c +17
-2+return -EINVAL ; + } +if (vq == max_vq) { +break ; + } + } + } + +return kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); +} +static inline void set_feature (uint64_t *features,int feature) { *features |=1ULL << feature;@@ -605,7 +658,7 @@ int kvm_arch_init_vcpu(CPUState *cs)if (cpu->kvm_target ==QEMU_KVM_ARM_TARGET_NONE || !object_dynamic_cast (OBJECT (cpu),TYPE_AARCH64_CPU )) { -fprintf (stderr,"KVM is not supported for this guest CPU type\n" ); +error_report ("KVM is not supported for this guest CPU type" );return -EINVAL ; }@@ -631,7 +684,12 @@ int kvm_arch_init_vcpu(CPUState *cs) }if (cpu->sve_max_vq) {if (!kvm_check_extension (cs->kvm_state,KVM_CAP_ARM_SVE )) { - cpu->sve_max_vq =0 ; + if (cpu->sve_max_vq == -1) {> + cpu->sve_max_vq = 0; + }else { +error_report ("This KVM host does not support SVE" ); +return -EINVAL ; + } }else { cpu->kvm_init_features[0 ] |=1 <<KVM_ARM_VCPU_SVE ; }
target/arm/kvm64.c int kvm_arch_init_vcpu(CPUState *cs) · line 705
Quoted 4 lines
target/arm/kvm64.c +1
-0@@ -644,6 +702,24 @@ int kvm_arch_init_vcpu(CPUState *cs) }if (cpu->sve_max_vq) { +uint64_t sve_vls[KVM_ARM64_SVE_VLS_WORDS ];
line
target/arm/kvm64.c int kvm_arch_init_vcpu(CPUState *cs) · line 713
Quoted 7 lines
target/arm/kvm64.c +7
-0+ ret =kvm_arm_get_sve_vls (cs, sve_vls); +if (ret <0 ) { +return ret; + } + if (cpu->sve_max_vq == -1) {> + cpu->sve_max_vq = ret; + }else if (cpu->sve_max_vq > ret) { + error_report("This KVM host does not support SVE vectors "
I would rephrase the error mesg with something like:
This KVM host supports SVE vectors of max VQ=%d whereas requested VQ is %d
target/arm/kvm64.c int kvm_arch_init_vcpu(CPUState *cs) · line 725
Quoted 13 lines
target/arm/kvm64.c +9
-0+"of length %d quadwords (%d bytes)" , + cpu->sve_max_vq, cpu->sve_max_vq * 16); +return -EINVAL ; + }else { + ret =kvm_arm_set_sve_vls (cs, sve_vls, cpu->sve_max_vq); +if (ret <0 ) { +return ret; + } + } ret =kvm_arm_vcpu_finalize (cs,KVM_ARM_VCPU_SVE );if (ret) {return ret;
Thanks
Eric
Re: [Qemu-devel] [PATCH 07/13] target/arm/kvm: max cpu: Allow sve max vector length setting
Quoted 62 lines · Auger Eric
Hi Drew,
Quoted 60 lines · Andrew Jones
Allow the cpu type 'max' sve-max-vq property to work with kvm too. If the property is not specified then the maximum kvm supports is used. If it is specified we check that kvm supports that exact length or error out if it doesn't.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/cpu.h | 4 +++ target/arm/cpu64.c | 7 ++-- target/arm/kvm64.c | 80 ++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 86 insertions(+), 5 deletions(-)target/arm/cpu.h +4
-0@@ -3122,6 +3122,10 @@ static inline uint64_t arm_sctlr(CPUARMState *env, int el) } } +static inline int arm_cpu_fls64 (uint64_t v) +{ +return !v ?0 :64 -clz64 (v); +}/* Return true if the processor is in big-endian mode. */ static inline bool arm_cpu_data_is_big_endian (CPUARMState *env)target/arm/cpu64.c +4
-3@@ -292,7 +292,7 @@ static void aarch64_max_initfn(Object *obj)if (kvm_enabled ()) {kvm_arm_set_cpu_features_from_host (cpu); - cpu->sve_max_vq =ARM_MAX_VQ ; + cpu->sve_max_vq =-1 ;/* set in kvm_arch_init_vcpu() */ }else {uint64_t t;uint32_t u;@@ -374,9 +374,10 @@ static void aarch64_max_initfn(Object *obj)#endif cpu->sve_max_vq =ARM_MAX_VQ ; -object_property_add (obj,"sve-max-vq" ,"uint32" , cpu_max_get_sve_vq, - cpu_max_set_sve_vq,NULL ,NULL , &error_fatal); } + +object_property_add (obj,"sve-max-vq" ,"uint32" , cpu_max_get_sve_vq, + cpu_max_set_sve_vq, NULL, NULL, &error_fatal);> }struct ARMCPUInfo {target/arm/kvm64.c +0
-0@@ -446,6 +446,59 @@ void kvm_arm_pmu_set_irq(CPUState *cs, int irq) } }
Quoted 15 lines
Quoted 4 lines
target/arm/kvm64.c +4
-0+static int kvm_arm_get_sve_vls (CPUState *cs,uint64_t sve_vls[]) +{ +struct kvm_one_reg reg = { + .id =KVM_REG_ARM64_SVE_VLS ,target/arm/kvm64.c void kvm_arm_pmu_set_irq(CPUState *cs, int irq) · line 462
Quoted 10 lines
target/arm/kvm64.c +10
-0+ .addr = (uint64_t)&sve_vls[0], + }; +int i, ret; + + ret =kvm_vcpu_ioctl (cs,KVM_GET_ONE_REG , ®); +if (ret) { +return ret; + } + + ret =0 ;is it mandated?
Yes, this can be removed, as we know ret=0 here.
Quoted 22 lines
Quoted 11 lines
target/arm/kvm64.c +11
-0+for (i =KVM_ARM64_SVE_VLS_WORDS -1 ; i >=0 ; --i) { +if (sve_vls[i]) { + ret =arm_cpu_fls64 (sve_vls[i]) + i *64 ; +break ; + } + } + +return ret; +} + +static int kvm_arm_set_sve_vls (CPUState *cs,uint64_t sve_vls[],int max_vq)target/arm/kvm64.c void kvm_arm_pmu_set_irq(CPUState *cs, int irq) · line 483
Quoted 10 lines
target/arm/kvm64.c +10
-0+{ +struct kvm_one_reg reg = { + .id =KVM_REG_ARM64_SVE_VLS , + .addr = (uint64_t )&sve_vls[0 ], + }; +int i; + +for (i =KVM_ARM64_SVE_VLS_WORDS -1 ; i >=0 ; --i) { +if (sve_vls[i]) { +int vq =arm_cpu_fls64 (sve_vls[i]) + i *64 ;nit: add a line
Quoted 3 lines
target/arm/kvm64.c void kvm_arm_pmu_set_irq(CPUState *cs, int irq) · line 485
Quoted 2 lines
target/arm/kvm64.c +2
-0+while (vq > max_vq) { + sve_vls[i] &= ~BIT_MASK (vq -1 );Isn't
BIT_MASKfor 32b.MAKE_64BIT_MASK?
We should just need a 'UL', not a 'ULL'. I think I'll change all my BIT_MASK() usage to just BIT() for v2 though, even though there doesn't appear to be a difference for the cases I've used it.
Quoted 5 lines
target/arm/kvm64.c void kvm_arm_pmu_set_irq(CPUState *cs, int irq) · line 488
Quoted 3 lines
target/arm/kvm64.c +3
-0+ vq =arm_cpu_fls64 (sve_vls[i]) + i *64 ; + } +if (vq < max_vq) {I don't really get this check: having vq less than
max_vqdoes not seems weird. Do you absolutely want vq=max_vq?
In this context 'vq' is the max vq KVM supports and max_vq is the maximum the user requested. So if the user wants a maximum greater than what is possible, then that's an error.
Quoted 3 lines
target/arm/kvm64.c void kvm_arm_pmu_set_irq(CPUState *cs, int irq) · line 490
Quoted 2 lines
target/arm/kvm64.c +2
-0+error_report ("sve-max-vq=%d is not a valid length" , max_vq); +error_printf ("next lowest is %d\n" , vq);why mixing error_report/printf?
This is how we're supposed to do two line error messages, afaik.
Quoted 49 lines
Quoted 37 lines
target/arm/kvm64.c +17
-2+return -EINVAL ; + } +if (vq == max_vq) { +break ; + } + } + } + +return kvm_vcpu_ioctl (cs,KVM_SET_ONE_REG , ®); +} +static inline void set_feature (uint64_t *features,int feature) { *features |=1ULL << feature;@@ -605,7 +658,7 @@ int kvm_arch_init_vcpu(CPUState *cs)if (cpu->kvm_target ==QEMU_KVM_ARM_TARGET_NONE || !object_dynamic_cast (OBJECT (cpu),TYPE_AARCH64_CPU )) { -fprintf (stderr,"KVM is not supported for this guest CPU type\n" ); +error_report ("KVM is not supported for this guest CPU type" );return -EINVAL ; }@@ -631,7 +684,12 @@ int kvm_arch_init_vcpu(CPUState *cs) }if (cpu->sve_max_vq) {if (!kvm_check_extension (cs->kvm_state,KVM_CAP_ARM_SVE )) { - cpu->sve_max_vq =0 ; + if (cpu->sve_max_vq == -1) {> + cpu->sve_max_vq = 0; + }else { +error_report ("This KVM host does not support SVE" ); +return -EINVAL ; + } }else { cpu->kvm_init_features[0 ] |=1 <<KVM_ARM_VCPU_SVE ; }target/arm/kvm64.c int kvm_arch_init_vcpu(CPUState *cs) · line 705
Quoted 4 lines
target/arm/kvm64.c +1
-0@@ -644,6 +702,24 @@ int kvm_arch_init_vcpu(CPUState *cs) }if (cpu->sve_max_vq) { +uint64_t sve_vls[KVM_ARM64_SVE_VLS_WORDS ];line
target/arm/kvm64.c int kvm_arch_init_vcpu(CPUState *cs) · line 713
Quoted 7 lines
target/arm/kvm64.c +7
-0+ ret =kvm_arm_get_sve_vls (cs, sve_vls); +if (ret <0 ) { +return ret; + } + if (cpu->sve_max_vq == -1) {> + cpu->sve_max_vq = ret; + }else if (cpu->sve_max_vq > ret) { + error_report("This KVM host does not support SVE vectors "
Quoted 2 lines
I would rephrase the error mesg with something like:
This KVM host supports SVE vectors of max VQ=%d whereas requested VQ is %d
Yup, giving a better hint as to what the host does support is a good idea.
Quoted 13 lines
target/arm/kvm64.c int kvm_arch_init_vcpu(CPUState *cs) · line 725
Quoted 13 lines
target/arm/kvm64.c +9
-0+"of length %d quadwords (%d bytes)" , + cpu->sve_max_vq, cpu->sve_max_vq * 16); +return -EINVAL ; + }else { + ret =kvm_arm_set_sve_vls (cs, sve_vls, cpu->sve_max_vq); +if (ret <0 ) { +return ret; + } + } ret =kvm_arm_vcpu_finalize (cs,KVM_ARM_VCPU_SVE );if (ret) {return ret;
Quoted 3 lines
Thanks
Eric
Thanks,
drew
PATCH 8/13 target/arm/monitor: Add query-sve-vector-lengths
In reply to Andrew JonesProvide a QMP interface to query the supported SVE vector lengths. A migratable guest will need to explicitly specify a valid set of lengths on the command line and that set can be obtained from the list returned with this QMP command.
This patch only introduces the QMP command with the TCG implementation. The result may not yet be correct for KVM. Following patches ensure the KVM result is correct.
--- qapi/target.json | 34 ++++++++++++++++++++++++ target/arm/monitor.c | 62 ++++++++++++++++++++++++++++++++++++++++++++ tests/qmp-cmd-test.c | 1 + 3 files changed, 97 insertions(+)
qapi/target.json +34 -0
@@ -397,6 +397,40 @@ { 'command': 'query-gic-capabilities', 'returns': ['GICCapability'], 'if': 'defined(TARGET_ARM)' } +## +# @SVEVectorLengths: +# +# The struct contains a list of integers where each integer is a valid +# SVE vector length for a KVM guest on this host. The vector lengths +# are in quadword (128-bit) units, e.g. '4' means 512 bits (64 bytes). +# +# @vls: list of vector lengths in quadwords. +# +# Since: 4.1 +## +{ 'struct': 'SVEVectorLengths', + 'data': { 'vls': ['int'] }, + 'if': 'defined(TARGET_ARM)' } + +## +# @query-sve-vector-lengths: +# +# This command is ARM-only. It will return a list of SVEVectorLengths +# objects. The list describes all valid SVE vector length sets. +# +# Returns: a list of SVEVectorLengths objects +# +# Since: 4.1 +# +# -> { "execute": "query-sve-vector-lengths" } +# <- { "return": [ { "vls": [ 1 ] }, +# { "vls": [ 1, 2 ] }, +# { "vls": [ 1, 2, 4 ] } ] } +# +## +{ 'command': 'query-sve-vector-lengths', 'returns': ['SVEVectorLengths'], + 'if': 'defined(TARGET_ARM)' } + ## # @CpuModelExpansionInfo: #
target/arm/monitor.c +62 -0
71 lines not on this page · raw message
tests/qmp-cmd-test.c +1 -0
@@ -90,6 +90,7 @@ static bool query_is_blacklisted(const char *cmd)/* Success depends on target arch: */ "query-cpu-definitions" ,/* arm, i386, ppc, s390x */ "query-gic-capabilities" ,/* arm */ +"query-sve-vector-lengths" ,/* arm */ /* Success depends on target-specific build configuration: */ "query-pci", /* CONFIG_PCI *//* Success depends on launching SEV guest */
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH 08/13] target/arm/monitor: Add query-sve-vector-lengths
Quoted 21 lines · Andrew Jones
Provide a QMP interface to query the supported SVE vector lengths. A migratable guest will need to explicitly specify a valid set of lengths on the command line and that set can be obtained from the list returned with this QMP command.
This patch only introduces the QMP command with the TCG implementation. The result may not yet be correct for KVM. Following patches ensure the KVM result is correct.
Signed-off-by Andrew Jones <[email protected]>--- qapi/target.json | 34 ++++++++++++++++++++++++ target/arm/monitor.c | 62 ++++++++++++++++++++++++++++++++++++++++++++ tests/qmp-cmd-test.c | 1 + 3 files changed, 97 insertions(+)qapi/target.json +0
-0
qapi/target.json line 403
Quoted 7 lines
qapi/target.json +4
-0@@ -397,6 +397,40 @@ { 'command': 'query-gic-capabilities', 'returns': ['GICCapability'], 'if': 'defined(TARGET_ARM)' } +## +# @SVEVectorLengths: +# +# The struct contains a list of integers where each integer is a valid
Suggest to s/The struct contains/Contains/.
qapi/target.json line 405
Quoted 2 lines
qapi/target.json +2
-0+# SVE vector length for a KVM guest on this host. The vector lengths +# are in quadword (128-bit) units, e.g. '4' means 512 bits (64 bytes).
Any particular reason for counting quad-words instead of bytes, or perhaps bits?
qapi/target.json line 418
Quoted 13 lines
qapi/target.json +13
-0+# +# @vls: list of vector lengths in quadwords. +# +# Since: 4.1 +## +{ 'struct': 'SVEVectorLengths', + 'data': { 'vls': ['int'] }, + 'if': 'defined(TARGET_ARM)' } + +## +# @query-sve-vector-lengths: +# +# This command is ARM-only. It will return a list of SVEVectorLengths
No other target-specific command documents its target-specificness like this. Suggest
# Query valid SVE vector length sets.
Quoted 23 lines
qapi/target.json +15
-0+# objects. The list describes all valid SVE vector length sets. +# +# Returns: a list of SVEVectorLengths objects +# +# Since: 4.1 +# +# -> { "execute": "query-sve-vector-lengths" } +# <- { "return": [ { "vls": [ 1 ] }, +# { "vls": [ 1, 2 ] }, +# { "vls": [ 1, 2, 4 ] } ] } +# +## +{ 'command': 'query-sve-vector-lengths', 'returns': ['SVEVectorLengths'], + 'if': 'defined(TARGET_ARM)' } + ## # @CpuModelExpansionInfo: #target/arm/monitor.c +0
-0
target/arm/monitor.c line 27
Quoted 4 lines
target/arm/monitor.c +1
-0@@ -24,6 +24,7 @@#include "hw/boards.h" #include "kvm_arm.h" #include "qapi/qapi-commands-target.h" +#include "monitor/hmp-target.h"
Uh, hmp-target.h when the patch is supposedly about QMP only...
target/arm/monitor.c GICCapabilityList *qmp_query_gic_capabilities(Error **errp) · line 89
Quoted 11 lines
target/arm/monitor.c +4
-0static GICCapability *gic_cap_new (int version) {@@ -82,3 +83,64 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)return head; } + +static SVEVectorLengths *qmp_sve_vls_get (void ) +{ +CPUArchState *env =mon_get_cpu_env ();
Aha, you need it for mon_get_cpu_env().
mon_get_cpu_env() returns the current monitor's current CPU. This is an HMP thing, QMP commands should never access it.
Looks like you use it to find one of the CPUs, so you can access its ->sve_max_vq.
"One of the CPUs" smells odd: what if they aren't all the same? Perhaps that can't happen. I don't know, you tell me :)
If any CPU will do, what about simply using first_cpu?
Quoted 5 lines
target/arm/monitor.c +5
-0+ARMCPU *cpu =arm_env_get_cpu (env); +SVEVectorLengths *vls =g_new (SVEVectorLengths,1 ); +intList **v = &vls->vls; +int i; +
target/arm/monitor.c GICCapabilityList *qmp_query_gic_capabilities(Error **errp) · line 104
Quoted 10 lines
target/arm/monitor.c +10
-0+if (cpu->sve_max_vq ==0 ) { + *v =g_new0 (intList,1 );/* one vl of 0 means none supported */ +return vls; + } + +for (i =1 ; i <= cpu->sve_max_vq; ++i) { + *v =g_new0 (intList,1 ); + (*v)->value = i; + v = &(*v)->next; + }
What this loop does is not immediately obvious. I think you could use a function comment.
Quoted 14 lines
target/arm/monitor.c +14
-0+ +return vls; +} + +static SVEVectorLengths *qmp_sve_vls_dup_and_truncate (SVEVectorLengths *vls) +{ +SVEVectorLengths *trunc_vls; +intList **v, *p = vls->vls; + +if (!p->next) { +return NULL ; + } + + trunc_vls =g_new (SVEVectorLengths,1 );
target/arm/monitor.c GICCapabilityList *qmp_query_gic_capabilities(Error **errp) · line 128
Quoted 10 lines
target/arm/monitor.c +10
-0+ v = &trunc_vls->vls; + +for (; p->next; p = p->next) { + *v =g_new0 (intList,1 ); + (*v)->value = p->value; + v = &(*v)->next; + } + +return trunc_vls; +}
More so.
Quoted 30 lines
target/arm/monitor.c +18
-0+ +SVEVectorLengthsList *qmp_query_sve_vector_lengths (Error **errp) +{ +SVEVectorLengthsList *vls_list =g_new0 (SVEVectorLengthsList,1 ); +SVEVectorLengths *vls =qmp_sve_vls_get (); + +while (vls) { + vls_list->value = vls; + vls =qmp_sve_vls_dup_and_truncate (vls); +if (vls) { +SVEVectorLengthsList *next = vls_list; + vls_list =g_new0 (SVEVectorLengthsList,1 ); + vls_list->next = next; + } + } + +return vls_list; +}tests/qmp-cmd-test.c +1
-0@@ -90,6 +90,7 @@ static bool query_is_blacklisted(const char *cmd)/* Success depends on target arch: */ "query-cpu-definitions" ,/* arm, i386, ppc, s390x */ "query-gic-capabilities" ,/* arm */ +"query-sve-vector-lengths" ,/* arm */ /* Success depends on target-specific build configuration: */ "query-pci", /* CONFIG_PCI *//* Success depends on launching SEV guest */
Re: [Qemu-devel] [PATCH 08/13] target/arm/monitor: Add query-sve-vector-lengths
Quoted 26 lines · Markus Armbruster
Quoted 24 lines · Andrew Jones
Provide a QMP interface to query the supported SVE vector lengths. A migratable guest will need to explicitly specify a valid set of lengths on the command line and that set can be obtained from the list returned with this QMP command.
This patch only introduces the QMP command with the TCG implementation. The result may not yet be correct for KVM. Following patches ensure the KVM result is correct.
Signed-off-by Andrew Jones <[email protected]>--- qapi/target.json | 34 ++++++++++++++++++++++++ target/arm/monitor.c | 62 ++++++++++++++++++++++++++++++++++++++++++++ tests/qmp-cmd-test.c | 1 + 3 files changed, 97 insertions(+)qapi/target.json +0
-0@@ -397,6 +397,40 @@ { 'command': 'query-gic-capabilities', 'returns': ['GICCapability'], 'if': 'defined(TARGET_ARM)' }
Quoted 6 lines
qapi/target.json line 403
Quoted 4 lines
qapi/target.json +4
-0+## +# @SVEVectorLengths: +# +# The struct contains a list of integers where each integer is a validSuggest to s/The struct contains/Contains/.
OK
Quoted 6 lines
qapi/target.json line 405
Quoted 3 lines
qapi/target.json +2
-0+# SVE vector length for a KVM guest on this host. The vector lengths +# are in quadword (128-bit) units, e.g. '4' means 512 bits (64 bytes).Any particular reason for counting quad-words instead of bytes, or perhaps bits?
It can be considered just bits here, but when set in sve-vls-map those bits are chosen to mean quadwords as that allows us to get up to 8192-bit vectors with a single 64-bit word. Maybe I should write more of that here to clarify.
Quoted 17 lines
Quoted 4 lines
qapi/target.json +3
-0+# +# @vls: list of vector lengths in quadwords. +#qapi/target.json line 418
Quoted 10 lines
qapi/target.json +10
-0+# Since: 4.1 +## +{ 'struct': 'SVEVectorLengths', + 'data': { 'vls': ['int'] }, + 'if': 'defined(TARGET_ARM)' } + +## +# @query-sve-vector-lengths: +# +# This command is ARM-only. It will return a list of SVEVectorLengthsNo other target-specific command documents its target-specificness like this. Suggest
Well, it's pretty similar to query-gic-capabilities, which is what I used as a template, but I'm happy to change it to whatever you suggest :)
Quoted 18 lines
# Query valid SVE vector length sets.
Quoted 5 lines
qapi/target.json +5
-0+# objects. The list describes all valid SVE vector length sets. +# +# Returns: a list of SVEVectorLengths objects +# +# Since: 4.1qapi/target.json line 433
Quoted 10 lines
qapi/target.json +10
-0+# +# -> { "execute": "query-sve-vector-lengths" } +# <- { "return": [ { "vls": [ 1 ] }, +# { "vls": [ 1, 2 ] }, +# { "vls": [ 1, 2, 4 ] } ] } +# +## +{ 'command': 'query-sve-vector-lengths', 'returns': ['SVEVectorLengths'], + 'if': 'defined(TARGET_ARM)' } +
Yup, will do
Quoted 32 lines
target/arm/monitor.c line 27
Quoted 12 lines
qapi/target.json +0
-0## # @CpuModelExpansionInfo: #target/arm/monitor.c +1
-0@@ -24,6 +24,7 @@#include "hw/boards.h" #include "kvm_arm.h" #include "qapi/qapi-commands-target.h" +#include "monitor/hmp-target.h" Uh,
hmp-target.hwhen the patch is supposedly about QMP only...target/arm/monitor.c GICCapabilityList *qmp_query_gic_capabilities(Error **errp) · line 89
Quoted 11 lines
target/arm/monitor.c +4
-0static GICCapability *gic_cap_new (int version) {@@ -82,3 +83,64 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)return head; } + +static SVEVectorLengths *qmp_sve_vls_get (void ) +{ +CPUArchState *env =mon_get_cpu_env ();Aha, you need it for
mon_get_cpu_env().
mon_get_cpu_env()returns the current monitor's current CPU. This is an HMP thing, QMP commands should never access it.
Quoted 7 lines
Looks like you use it to find one of the CPUs, so you can access its ->sve_max_vq.
"One of the CPUs" smells odd: what if they aren't all the same? Perhaps that can't happen. I don't know, you tell me :)
If any CPU will do, what about simply using
first_cpu?
first_cpu will work. We currently only allow the same vector length set for all cpus. I'll change it and drop the HMP things.
Quoted 11 lines
Quoted 11 lines
target/arm/monitor.c +10
-0+ARMCPU *cpu =arm_env_get_cpu (env); +SVEVectorLengths *vls =g_new (SVEVectorLengths,1 ); +intList **v = &vls->vls; +int i; + +if (cpu->sve_max_vq ==0 ) { + *v =g_new0 (intList,1 );/* one vl of 0 means none supported */ +return vls; + } +
Quoted 8 lines
target/arm/monitor.c GICCapabilityList *qmp_query_gic_capabilities(Error **errp) · line 104
Quoted 5 lines
target/arm/monitor.c +5
-0+for (i =1 ; i <= cpu->sve_max_vq; ++i) { + *v =g_new0 (intList,1 ); + (*v)->value = i; + v = &(*v)->next; + }What this loop does is not immediately obvious. I think you could use a function comment.
OK
Quoted 19 lines
Quoted 19 lines
target/arm/monitor.c +18
-0+ +return vls; +} + +static SVEVectorLengths *qmp_sve_vls_dup_and_truncate (SVEVectorLengths *vls) +{ +SVEVectorLengths *trunc_vls; +intList **v, *p = vls->vls; + +if (!p->next) { +return NULL ; + } + + trunc_vls =g_new (SVEVectorLengths,1 ); + v = &trunc_vls->vls; + +for (; p->next; p = p->next) { + *v =g_new0 (intList,1 );
Quoted 8 lines
target/arm/monitor.c GICCapabilityList *qmp_query_gic_capabilities(Error **errp) · line 128
Quoted 6 lines
target/arm/monitor.c +6
-0+ (*v)->value = p->value; + v = &(*v)->next; + } + +return trunc_vls; +}More so.
More OK :)
Quoted 23 lines
Quoted 23 lines
target/arm/monitor.c +18
-0+ +SVEVectorLengthsList *qmp_query_sve_vector_lengths (Error **errp) +{ +SVEVectorLengthsList *vls_list =g_new0 (SVEVectorLengthsList,1 ); +SVEVectorLengths *vls =qmp_sve_vls_get (); + +while (vls) { + vls_list->value = vls; + vls =qmp_sve_vls_dup_and_truncate (vls); +if (vls) { +SVEVectorLengthsList *next = vls_list; + vls_list =g_new0 (SVEVectorLengthsList,1 ); + vls_list->next = next; + } + } + +return vls_list; +}tests/qmp-cmd-test.c +0
-0
Quoted 8 lines
tests/qmp-cmd-test.c static bool query_is_blacklisted(const char *cmd) · line 96
Quoted 8 lines
tests/qmp-cmd-test.c +1
-0@@ -90,6 +90,7 @@ static bool query_is_blacklisted(const char *cmd)/* Success depends on target arch: */ "query-cpu-definitions" ,/* arm, i386, ppc, s390x */ "query-gic-capabilities" ,/* arm */ +"query-sve-vector-lengths" ,/* arm */ /* Success depends on target-specific build configuration: */ "query-pci", /* CONFIG_PCI *//* Success depends on launching SEV guest */
Thanks,
drew
Re: [Qemu-devel] [PATCH 08/13] target/arm/monitor: Add query-sve-vector-lengths
Quoted 43 lines · Andrew Jones
Quoted 26 lines · Markus Armbruster
Quoted 24 lines · Andrew Jones
Provide a QMP interface to query the supported SVE vector lengths. A migratable guest will need to explicitly specify a valid set of lengths on the command line and that set can be obtained from the list returned with this QMP command.
This patch only introduces the QMP command with the TCG implementation. The result may not yet be correct for KVM. Following patches ensure the KVM result is correct.
Signed-off-by Andrew Jones <[email protected]>--- qapi/target.json | 34 ++++++++++++++++++++++++ target/arm/monitor.c | 62 ++++++++++++++++++++++++++++++++++++++++++++ tests/qmp-cmd-test.c | 1 + 3 files changed, 97 insertions(+)qapi/target.json +0
-0@@ -397,6 +397,40 @@ { 'command': 'query-gic-capabilities', 'returns': ['GICCapability'], 'if': 'defined(TARGET_ARM)' }Quoted 6 lines
qapi/target.json line 403
Quoted 4 lines
qapi/target.json +4
-0+## +# @SVEVectorLengths: +# +# The struct contains a list of integers where each integer is a validSuggest to s/The struct contains/Contains/.
OK
Quoted 6 lines
qapi/target.json line 405
Quoted 3 lines
qapi/target.json +2
-0+# SVE vector length for a KVM guest on this host. The vector lengths +# are in quadword (128-bit) units, e.g. '4' means 512 bits (64 bytes).Any particular reason for counting quad-words instead of bytes, or perhaps bits?
Quoted 4 lines
It can be considered just bits here, but when set in sve-vls-map those bits are chosen to mean quadwords as that allows us to get up to 8192-bit vectors with a single 64-bit word. Maybe I should write more of that here to clarify.
Please do.
In general, QMP prefers the plain units over arbitrary multiples, e.g. Byte, not Mebibyte. Sadly, many violations of this rule have crept in.
More complete documentation should help us determine the unit we want here.
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
qapi/target.json +8
-0+# +# @vls: list of vector lengths in quadwords. +# +# Since: 4.1 +## +{ 'struct': 'SVEVectorLengths', + 'data': { 'vls': ['int'] }, + 'if': 'defined(TARGET_ARM)' }Quoted 8 lines
qapi/target.json line 418
Quoted 5 lines
qapi/target.json +5
-0+ +## +# @query-sve-vector-lengths: +# +# This command is ARM-only. It will return a list of SVEVectorLengthsNo other target-specific command documents its target-specificness like this. Suggest
Quoted 2 lines
Well, it's pretty similar to query-gic-capabilities, which is what I used as a template, but I'm happy to change it to whatever you suggest :)
Here's the documentation we generate for query-gic-capabilities:
-- Command: query-gic-capabilities
This command is ARM-only. It will return a list of GICCapability
objects that describe its capability bits.
Returns: a list of GICCapability objects.
Since: 2.6
Example:
-> { "execute": "query-gic-capabilities" }
<- { "return": [{ "version": 2, "emulated": true, "kernel": false },
{ "version": 3, "emulated": false, "kernel": true } ] }
If: 'defined(TARGET_ARM)'The "If:" line is generated from the schema's condition. It's not as readable as I'd like it to be, but it's there, and it always matches the code.
"This command is ARM-only" is redundant. Escaped review. A patch to drop it would be welcome.
Quoted 11 lines
Quoted 11 lines
# Query valid SVE vector length sets.
Quoted 9 lines
qapi/target.json +9
-0+# objects. The list describes all valid SVE vector length sets. +# +# Returns: a list of SVEVectorLengths objects +# +# Since: 4.1 +# +# -> { "execute": "query-sve-vector-lengths" } +# <- { "return": [ { "vls": [ 1 ] }, +# { "vls": [ 1, 2 ] },
Quoted 8 lines
Quoted 6 lines
qapi/target.json line 433
Quoted 6 lines
qapi/target.json +6
-0+# { "vls": [ 1, 2, 4 ] } ] } +# +## +{ 'command': 'query-sve-vector-lengths', 'returns': ['SVEVectorLengths'], + 'if': 'defined(TARGET_ARM)' } +Yup, will do
Took me a few seconds to connect this to my "# Query valid SVE vector length sets." line %-}
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
qapi/target.json +0
-0## # @CpuModelExpansionInfo: #
[...]
PATCH 9/13 target/arm/kvm: Export kvm_arm_get_sve_vls
In reply to Andrew Jones--- target/arm/kvm64.c | 7 +++++-- target/arm/kvm_arm.h | 20 ++++++++++++++++++++ 2 files changed, 25 insertions(+), 2 deletions(-)
target/arm/kvm64.c +5 -2
@@ -446,7 +446,8 @@ void kvm_arm_pmu_set_irq(CPUState *cs, int irq) } } -static int kvm_arm_get_sve_vls (CPUState *cs,uint64_t sve_vls[]) +int kvm_arm_get_sve_vls (CPUState *cs, +uint64_t sve_vls[KVM_ARM64_SVE_VLS_WORDS ]) {struct kvm_one_reg reg = { .id =KVM_REG_ARM64_SVE_VLS ,
@@ -470,7 +471,9 @@ static int kvm_arm_get_sve_vls(CPUState *cs, uint64_t sve_vls[])return ret; } -static int kvm_arm_set_sve_vls (CPUState *cs,uint64_t sve_vls[],int max_vq) +static int kvm_arm_set_sve_vls (CPUState *cs, +uint64_t sve_vls[KVM_ARM64_SVE_VLS_WORDS ], +int max_vq) {struct kvm_one_reg reg = { .id =KVM_REG_ARM64_SVE_VLS ,
target/arm/kvm_arm.h +20 -0
@@ -248,6 +248,26 @@ int kvm_arm_vgic_probe(void);void kvm_arm_pmu_set_irq (CPUState *cs,int irq);void kvm_arm_pmu_init (CPUState *cs); +/** +* kvm_arm_get_sve_vls +* @cs: CPUState +* @sve_vls: valid vector length bitmap +* +* Get the valid vector length bitmap. If a bit 'bit' is set +* then the host supports a vector length of (bit * 16) bytes. +* +* For example, if +* +* sve_vls[0] = 0xb and +* sve_vls[1 ... KVM_ARM64_SVE_VLS_WORDS-1] = 0, +* +* then the host supports 16, 32, and 64 byte vector lengths. +* +* Returns: the highest set bit if successful else < 0 error code +*/ +int kvm_arm_get_sve_vls (CPUState *cs, +uint64_t sve_vls[KVM_ARM64_SVE_VLS_WORDS ]); +#else static inline void kvm_arm_set_cpu_features_from_host (ARMCPU *cpu)
Signature · 1 line
2.20.1
PATCH 10/13 target/arm/monitor: kvm: only return valid sve vector sets
In reply to Andrew JonesWhile the TCG SVE implementation can implement all vector lengths which are a quadword multiple, up to some maximum length, KVM can only provide what the host supports, and not all multiples are required to be supported by the architecture. With this patch we extend the QMP query to ask KVM for the valid vectors when KVM is enabled.
--- target/arm/monitor.c | 43 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 42 insertions(+), 1 deletion(-)
target/arm/monitor.c +42 -1
@@ -84,6 +84,41 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)return head; } +#ifdef CONFIG_KVM +static SVEVectorLengths *qmp_kvm_sve_vls_get (void ) +{ +CPUArchState *env =mon_get_cpu_env (); +ARMCPU *cpu =arm_env_get_cpu (env); +uint64_t sve_vls[KVM_ARM64_SVE_VLS_WORDS ]; +SVEVectorLengths *vls =g_new (SVEVectorLengths,1 ); +intList **v = &vls->vls; +int ret, i; + + ret =kvm_arm_get_sve_vls (CPU (cpu), sve_vls); +if (ret <=0 ) { + *v =g_new0 (intList,1 );/* one vl of 0 means none supported */ +return vls; + } + +for (i =KVM_ARM64_SVE_VQ_MIN ; i <= ret; ++i) { +int bitval = (sve_vls[(i -KVM_ARM64_SVE_VQ_MIN ) /64 ] >> + ((i -KVM_ARM64_SVE_VQ_MIN ) %64 )) &1 ; +if (bitval) { + *v =g_new0 (intList,1 ); + (*v)->value = i; + v = &(*v)->next; + } + } + +return vls; +} +#else +static SVEVectorLengths *qmp_kvm_sve_vls_get (void ) +{ +return NULL ; +} +#endif +static SVEVectorLengths *qmp_sve_vls_get (void ) {CPUArchState *env =mon_get_cpu_env ();
@@ -130,7 +165,13 @@ static SVEVectorLengths *qmp_sve_vls_dup_and_truncate(SVEVectorLengths *vls)SVEVectorLengthsList *qmp_query_sve_vector_lengths (Error **errp) {SVEVectorLengthsList *vls_list =g_new0 (SVEVectorLengthsList,1 ); -SVEVectorLengths *vls =qmp_sve_vls_get (); +SVEVectorLengths *vls; + +if (kvm_enabled ()) { + vls =qmp_kvm_sve_vls_get (); + }else { + vls =qmp_sve_vls_get (); + }while (vls) { vls_list->value = vls;
Signature · 1 line
2.20.1
PATCH 11/13 target/arm/cpu64: max cpu: Introduce sve-vls-map
In reply to Andrew JonesIntroduce another cpu property to control SVE vector lengths, sve-vls-map, which allows the user to explicitly select the set of vector lengths the guest can use. The map must conform to QEMU's limits and architectural constraints, checked when the property is set. Inconsistencies with sve-max-vq are also checked. The bit number of a set bit in the map represents the allowed vector length in number of quadwords.
Note, as the map is implemented with a single 64-bit word we currently only support up to 8192-bit vectors. As QEMU and KVM only support up to 2048-bit vectors then this sufficient now, and probably for some time. Extending the bitmap beyond a single word will likely require changing the property to a string and adding yet another parser to QEMU.
--- target/arm/cpu.c | 4 +++ target/arm/cpu.h | 3 ++ target/arm/cpu64.c | 70 +++++++++++++++++++++++++++++++++++++++++--- target/arm/helper.c | 10 ++++++- target/arm/monitor.c | 9 ++++-- 5 files changed, 88 insertions(+), 8 deletions(-)
target/arm/cpu.c +4 -0
@@ -1188,6 +1188,10 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)set_feature (env,ARM_FEATURE_VBAR ); } +if (!kvm_enabled () && !cpu->sve_vls_map) { + cpu->sve_vls_map =BIT_MASK (cpu->sve_max_vq) -1 ; + } +register_cp_regs_for_features (cpu);arm_cpu_register_gdb_regs_for_features (cpu);
target/arm/cpu.h +3 -0
@@ -920,6 +920,9 @@ struct ARMCPU {/* Used to set the maximum vector length the cpu will support. */ uint32_t sve_max_vq; + +/* Each bit represents a supported vector length of (bitnum * 16) bytes */ +uint64_t sve_vls_map; };static inline ARMCPU *arm_env_get_cpu (CPUARMState *env)
target/arm/cpu64.c +66 -4
86 lines not on this page · raw message
target/arm/helper.c +9 -1
@@ -5254,12 +5254,20 @@ uint32_t sve_zcr_len_for_el(CPUARMState *env, int el)static void zcr_write (CPUARMState *env,const ARMCPRegInfo *ri,uint64_t value) { +ARMCPU *cpu =arm_env_get_cpu (env);int cur_el =arm_current_el (env);int old_len =sve_zcr_len_for_el (env, cur_el);int new_len;/* Bits other than [3:0] are RAZ/WI. */ -raw_write (env, ri, value &0xf ); + value &=0xf ; + +if (value && !(BIT_MASK (value) & cpu->sve_vls_map)) { +uint64_t map = cpu->sve_vls_map & (BIT_MASK (value) -1 ); + value =arm_cpu_fls64 (map) -1 ; + } + +raw_write (env, ri, value);/* * Because we arrived here, we know both FP and SVE are enabled;
target/arm/monitor.c +6 -3
@@ -133,9 +133,12 @@ static SVEVectorLengths *qmp_sve_vls_get(void) }for (i =1 ; i <= cpu->sve_max_vq; ++i) { - *v =g_new0 (intList,1 ); - (*v)->value = i; - v = &(*v)->next; +int bitval = (cpu->sve_vls_map >> (i -1 )) &1 ; +if (bitval) { + *v =g_new0 (intList,1 ); + (*v)->value = i; + v = &(*v)->next; + } }return vls;
Signature · 1 line
2.20.1
Re: [PATCH 11/13] target/arm/cpu64: max cpu: Introduce sve-vls-map
Quoted 24 lines · Andrew Jones
Introduce another cpu property to control SVE vector lengths, sve-vls-map, which allows the user to explicitly select the set of vector lengths the guest can use. The map must conform to QEMU's limits and architectural constraints, checked when the property is set. Inconsistencies with sve-max-vq are also checked. The bit number of a set bit in the map represents the allowed vector length in number of quadwords.
Note, as the map is implemented with a single 64-bit word we currently only support up to 8192-bit vectors. As QEMU and KVM only support up to 2048-bit vectors then this sufficient now, and probably for some time. Extending the bitmap beyond a single word will likely require changing the property to a string and adding yet another parser to QEMU.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/cpu.c | 4 +++ target/arm/cpu.h | 3 ++ target/arm/cpu64.c | 70 +++++++++++++++++++++++++++++++++++++++++--- target/arm/helper.c | 10 ++++++- target/arm/monitor.c | 9 ++++-- 5 files changed, 88 insertions(+), 8 deletions(-)
[...]
target/arm/cpu.h struct ARMCPU { · line 925
Quoted 11 lines
target/arm/cpu.h +3
-0@@ -920,6 +920,9 @@ struct ARMCPU {/* Used to set the maximum vector length the cpu will support. */ uint32_t sve_max_vq; + +/* Each bit represents a supported vector length of (bitnum * 16) bytes */ +uint64_t sve_vls_map;
Just to be clear, the representation here is different from the representation in KVM_REG_ARM64_SVE_VLS?
In the latter, bit n represents vector length ((n + 1) * 16) bytes.
(QEMU is free to choose its own internal representation, naturally.)
[...]
Quoted 1 line
target/arm/cpu64.c +0
-0
[...]
target/arm/cpu64.c static void cpu_max_set_sve_vq(Object *obj, Visitor *v, const char *name, · line 308
Quoted 13 lines
target/arm/cpu64.c +13
-0+static void cpu_set_sve_vls_map (Object *obj,Visitor *v,const char *name, +void *opaque,Error **errp) +{ +ARMCPU *cpu =ARM_CPU (obj); +Error *err =NULL ; +uint64_t mask = ~(BIT_MASK (ARM_MAX_VQ -1 ) -1 ); +int i; + +visit_type_uint64 (v, name, &cpu->sve_vls_map, errp); + +if (!err) { +if (cpu->sve_vls_map ==0 ) { +error_setg (&err,"SVE vector length map cannot be zero" );
Maybe say "empty" here, since the map represents a set?
(But it this is just for debug rather than reporting errors to the user, it probably doesn't matter much.)
[...]
Quoted 6 lines
target/arm/helper.c +0
-0@@ -5254,12 +5254,20 @@ uint32_t sve_zcr_len_for_el(CPUARMState *env, int el) static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
target/arm/helper.c uint32_t sve_zcr_len_for_el(CPUARMState *env, int el) · line 5263
Quoted 10 lines
target/arm/helper.c +2
-1uint64_t value) { +ARMCPU *cpu =arm_env_get_cpu (env);int cur_el =arm_current_el (env);int old_len =sve_zcr_len_for_el (env, cur_el);int new_len;/* Bits other than [3:0] are RAZ/WI. */ -raw_write (env, ri, value &0xf ); + value &=0xf ;
You might want to sanity-check that the max vq you configured for the vcpu is <= 16 here.
target/arm/helper.c uint32_t sve_zcr_len_for_el(CPUARMState *env, int el) · line 5270
Quoted 7 lines
target/arm/helper.c +7
-0+ +if (value && !(BIT_MASK (value) & cpu->sve_vls_map)) { +uint64_t map = cpu->sve_vls_map & (BIT_MASK (value) -1 ); + value =arm_cpu_fls64 (map) -1 ; + } + +raw_write (env, ri, value);
[...]
Cheers
---Dave
Re: [Qemu-devel] [PATCH 11/13] target/arm/cpu64: max cpu: Introduce sve-vls-map
Quoted 43 lines · Dave Martin
Quoted 24 lines · Andrew Jones
Introduce another cpu property to control SVE vector lengths, sve-vls-map, which allows the user to explicitly select the set of vector lengths the guest can use. The map must conform to QEMU's limits and architectural constraints, checked when the property is set. Inconsistencies with sve-max-vq are also checked. The bit number of a set bit in the map represents the allowed vector length in number of quadwords.
Note, as the map is implemented with a single 64-bit word we currently only support up to 8192-bit vectors. As QEMU and KVM only support up to 2048-bit vectors then this sufficient now, and probably for some time. Extending the bitmap beyond a single word will likely require changing the property to a string and adding yet another parser to QEMU.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/cpu.c | 4 +++ target/arm/cpu.h | 3 ++ target/arm/cpu64.c | 70 +++++++++++++++++++++++++++++++++++++++++--- target/arm/helper.c | 10 ++++++- target/arm/monitor.c | 9 ++++-- 5 files changed, 88 insertions(+), 8 deletions(-)[...]
target/arm/cpu.h struct ARMCPU { · line 925
Quoted 11 lines
target/arm/cpu.h +3
-0@@ -920,6 +920,9 @@ struct ARMCPU {/* Used to set the maximum vector length the cpu will support. */ uint32_t sve_max_vq; + +/* Each bit represents a supported vector length of (bitnum * 16) bytes */ +uint64_t sve_vls_map;Just to be clear, the representation here is different from the representation in
KVM_REG_ARM64_SVE_VLS?
Quoted 1 line
In the latter, bit n represents vector length ((n + 1) * 16) bytes.
KVM also uses bitnum * 16. bitnum is the the bit number, not the shift. So you can't have bitnum=0. 'n' must be a shift, as it would need to allow zero in order to represent KVM_ARM64_SVE_VQ_MIN.
Quoted 9 lines
(QEMU is free to choose its own internal representation, naturally.)
[...]
Quoted 1 line
target/arm/cpu64.c +0
-0[...]
Quoted 15 lines
target/arm/cpu64.c static void cpu_max_set_sve_vq(Object *obj, Visitor *v, const char *name, · line 308
Quoted 13 lines
target/arm/cpu64.c +13
-0+static void cpu_set_sve_vls_map (Object *obj,Visitor *v,const char *name, +void *opaque,Error **errp) +{ +ARMCPU *cpu =ARM_CPU (obj); +Error *err =NULL ; +uint64_t mask = ~(BIT_MASK (ARM_MAX_VQ -1 ) -1 ); +int i; + +visit_type_uint64 (v, name, &cpu->sve_vls_map, errp); + +if (!err) { +if (cpu->sve_vls_map ==0 ) { +error_setg (&err,"SVE vector length map cannot be zero" );Maybe say "empty" here, since the map represents a set?
Empty does sound better when considering the map represents a set, but the user will be inputting a number for the map, so zero will be what they attempted to input.
Quoted 3 lines
(But it this is just for debug rather than reporting errors to the user, it probably doesn't matter much.)
I'm also not too worried about which term we use, so I'm happy to change it if you'd like.
Quoted 16 lines
[...]
Quoted 13 lines
target/arm/helper.c +1
-0@@ -5254,12 +5254,20 @@ uint32_t sve_zcr_len_for_el(CPUARMState *env, int el)static void zcr_write (CPUARMState *env,const ARMCPRegInfo *ri,uint64_t value) { +ARMCPU *cpu =arm_env_get_cpu (env);int cur_el =arm_current_el (env);int old_len =sve_zcr_len_for_el (env, cur_el);int new_len;
Quoted 6 lines
target/arm/helper.c uint32_t sve_zcr_len_for_el(CPUARMState *env, int el) · line 5263
Quoted 3 lines
target/arm/helper.c +1
-1/* Bits other than [3:0] are RAZ/WI. */ -raw_write (env, ri, value &0xf ); + value &=0xf ;You might want to sanity-check that the max vq you configured for the vcpu is <= 16 here.
It's already checked by the cpu property set function at input time.
Quoted 11 lines
Quoted 8 lines
target/arm/helper.c +7
-0+ +if (value && !(BIT_MASK (value) & cpu->sve_vls_map)) { +uint64_t map = cpu->sve_vls_map & (BIT_MASK (value) -1 ); + value =arm_cpu_fls64 (map) -1 ; + } + +raw_write (env, ri, value);[...]
Quoted 3 lines
Cheers
---Dave
Thanks,
drew
Re: [Qemu-devel] [PATCH 11/13] target/arm/cpu64: max cpu: Introduce sve-vls-map
Quoted 44 lines · Andrew Jones
Quoted 43 lines · Dave Martin
Quoted 24 lines · Andrew Jones
Introduce another cpu property to control SVE vector lengths, sve-vls-map, which allows the user to explicitly select the set of vector lengths the guest can use. The map must conform to QEMU's limits and architectural constraints, checked when the property is set. Inconsistencies with sve-max-vq are also checked. The bit number of a set bit in the map represents the allowed vector length in number of quadwords.
Note, as the map is implemented with a single 64-bit word we currently only support up to 8192-bit vectors. As QEMU and KVM only support up to 2048-bit vectors then this sufficient now, and probably for some time. Extending the bitmap beyond a single word will likely require changing the property to a string and adding yet another parser to QEMU.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/cpu.c | 4 +++ target/arm/cpu.h | 3 ++ target/arm/cpu64.c | 70 +++++++++++++++++++++++++++++++++++++++++--- target/arm/helper.c | 10 ++++++- target/arm/monitor.c | 9 ++++-- 5 files changed, 88 insertions(+), 8 deletions(-)[...]
target/arm/cpu.h struct ARMCPU { · line 925
Quoted 11 lines
target/arm/cpu.h +3
-0@@ -920,6 +920,9 @@ struct ARMCPU {/* Used to set the maximum vector length the cpu will support. */ uint32_t sve_max_vq; + +/* Each bit represents a supported vector length of (bitnum * 16) bytes */ +uint64_t sve_vls_map;Just to be clear, the representation here is different from the representation in
KVM_REG_ARM64_SVE_VLS?
Quoted 3 lines
Quoted 1 line
In the latter, bit n represents vector length ((n + 1) * 16) bytes.
KVM also uses bitnum * 16. bitnum is the the bit number, not the shift.
Can you point to the relevant kernel code? This isn't what I thought I wrote...
Quoted 2 lines
So you can't have bitnum=0. 'n' must be a shift, as it would need to allow zero in order to represent
KVM_ARM64_SVE_VQ_MIN.
[...]
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
target/arm/cpu.h +7
-0+static void cpu_set_sve_vls_map (Object *obj,Visitor *v,const char *name, +void *opaque,Error **errp) +{ +ARMCPU *cpu =ARM_CPU (obj); +Error *err =NULL ; +uint64_t mask = ~(BIT_MASK (ARM_MAX_VQ -1 ) -1 ); +int i;Quoted 8 lines
target/arm/cpu.h
Quoted 6 lines
target/arm/cpu.h +6
-0+ +visit_type_uint64 (v, name, &cpu->sve_vls_map, errp); + +if (!err) { +if (cpu->sve_vls_map ==0 ) { +error_setg (&err,"SVE vector length map cannot be zero" );Maybe say "empty" here, since the map represents a set?
Quoted 3 lines
Empty does sound better when considering the map represents a set, but the user will be inputting a number for the map, so zero will be what they attempted to input.
If the user is inputting the encoded number directly, I guess it makes sense to describe the empty set as "zero".
Quoted 5 lines
Quoted 2 lines
(But it this is just for debug rather than reporting errors to the user, it probably doesn't matter much.)
I'm also not too worried about which term we use, so I'm happy to change it if you'd like.
I guess I don't have a strong opinion.
[...]
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
target/arm/helper.c +1
-0@@ -5254,12 +5254,20 @@ uint32_t sve_zcr_len_for_el(CPUARMState *env, int el)static void zcr_write (CPUARMState *env,const ARMCPRegInfo *ri,uint64_t value) { +ARMCPU *cpu =arm_env_get_cpu (env);int cur_el =arm_current_el (env);int old_len =sve_zcr_len_for_el (env, cur_el);int new_len;
Quoted 8 lines
Quoted 6 lines
target/arm/helper.c uint32_t sve_zcr_len_for_el(CPUARMState *env, int el) · line 5263
Quoted 3 lines
target/arm/helper.c +1
-1/* Bits other than [3:0] are RAZ/WI. */ -raw_write (env, ri, value &0xf ); + value &=0xf ;You might want to sanity-check that the max vq you configured for the vcpu is <= 16 here.
It's already checked by the cpu property set function at input time.
The check is in one place, but the assumptions based on it are potentially all over the place -- it may be tricky to track them all down if you ever add support for >8192-bit vectors.
(Not a big deal from my point of view, but I'd migration to larger vectors to be as smooth as reasonably possible if we ever find we have to do it.)
Cheers
---Dave
Re: [Qemu-devel] [PATCH 11/13] target/arm/cpu64: max cpu: Introduce sve-vls-map
Quoted 45 lines · Dave Martin
Quoted 44 lines · Andrew Jones
Quoted 43 lines · Dave Martin
Quoted 24 lines · Andrew Jones
Introduce another cpu property to control SVE vector lengths, sve-vls-map, which allows the user to explicitly select the set of vector lengths the guest can use. The map must conform to QEMU's limits and architectural constraints, checked when the property is set. Inconsistencies with sve-max-vq are also checked. The bit number of a set bit in the map represents the allowed vector length in number of quadwords.
Note, as the map is implemented with a single 64-bit word we currently only support up to 8192-bit vectors. As QEMU and KVM only support up to 2048-bit vectors then this sufficient now, and probably for some time. Extending the bitmap beyond a single word will likely require changing the property to a string and adding yet another parser to QEMU.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/cpu.c | 4 +++ target/arm/cpu.h | 3 ++ target/arm/cpu64.c | 70 +++++++++++++++++++++++++++++++++++++++++--- target/arm/helper.c | 10 ++++++- target/arm/monitor.c | 9 ++++-- 5 files changed, 88 insertions(+), 8 deletions(-)[...]
target/arm/cpu.h struct ARMCPU { · line 925
Quoted 11 lines
target/arm/cpu.h +3
-0@@ -920,6 +920,9 @@ struct ARMCPU {/* Used to set the maximum vector length the cpu will support. */ uint32_t sve_max_vq; + +/* Each bit represents a supported vector length of (bitnum * 16) bytes */ +uint64_t sve_vls_map;Just to be clear, the representation here is different from the representation in
KVM_REG_ARM64_SVE_VLS?
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
In the latter, bit n represents vector length ((n + 1) * 16) bytes.
KVM also uses bitnum * 16. bitnum is the the bit number, not the shift.
Can you point to the relevant kernel code? This isn't what I thought I wrote...
The Documentation/virtual/kvm/api.txt documentation has
Quoted 6 lines
if (vq >=
SVE_VQ_MIN&& vq <=SVE_VQ_MAX&&
((vector_lengths[(vq -KVM_ARM64_SVE_VQ_MIN) / 64] >>((vq - KVM_ARM64_SVE_VQ_MIN) % 64)) & 1)) /* Vector length vq * 16 bytes supported */ else /* Vector length vq * 16 bytes not supported */
Taking vq=1, we check (vector_lengths[0] >> 0) to see if we have the length (1 * 16) bytes. Since bitnum 1 is (1 << 0) that means the shift is zero and (bitnum * 16) is the same as (vq * 16). With ((n + 1) * 16), n would have to be zero, which is not a valid bitnum, but is a valid bit shift.
Quoted 61 lines
Quoted 3 lines
So you can't have bitnum=0. 'n' must be a shift, as it would need to allow zero in order to represent
KVM_ARM64_SVE_VQ_MIN.[...]
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
target/arm/cpu.h +7
-0+static void cpu_set_sve_vls_map (Object *obj,Visitor *v,const char *name, +void *opaque,Error **errp) +{ +ARMCPU *cpu =ARM_CPU (obj); +Error *err =NULL ; +uint64_t mask = ~(BIT_MASK (ARM_MAX_VQ -1 ) -1 ); +int i;Quoted 8 lines
target/arm/cpu.h
Quoted 6 lines
target/arm/cpu.h +6
-0+ +visit_type_uint64 (v, name, &cpu->sve_vls_map, errp); + +if (!err) { +if (cpu->sve_vls_map ==0 ) { +error_setg (&err,"SVE vector length map cannot be zero" );Maybe say "empty" here, since the map represents a set?
Quoted 3 lines
Empty does sound better when considering the map represents a set, but the user will be inputting a number for the map, so zero will be what they attempted to input.
If the user is inputting the encoded number directly, I guess it makes sense to describe the empty set as "zero".
Quoted 5 lines
Quoted 2 lines
(But it this is just for debug rather than reporting errors to the user, it probably doesn't matter much.)
I'm also not too worried about which term we use, so I'm happy to change it if you'd like.
I guess I don't have a strong opinion.
[...]
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
target/arm/helper.c +1
-0@@ -5254,12 +5254,20 @@ uint32_t sve_zcr_len_for_el(CPUARMState *env, int el)static void zcr_write (CPUARMState *env,const ARMCPRegInfo *ri,uint64_t value) { +ARMCPU *cpu =arm_env_get_cpu (env);int cur_el =arm_current_el (env);int old_len =sve_zcr_len_for_el (env, cur_el);int new_len;Quoted 8 lines
Quoted 6 lines
target/arm/helper.c uint32_t sve_zcr_len_for_el(CPUARMState *env, int el) · line 5263
Quoted 3 lines
target/arm/helper.c +1
-1/* Bits other than [3:0] are RAZ/WI. */ -raw_write (env, ri, value &0xf ); + value &=0xf ;You might want to sanity-check that the max vq you configured for the vcpu is <= 16 here.
It's already checked by the cpu property set function at input time.
Quoted 8 lines
The check is in one place, but the assumptions based on it are potentially all over the place -- it may be tricky to track them all down if you ever add support for >8192-bit vectors.
(Not a big deal from my point of view, but I'd migration to larger vectors to be as smooth as reasonably possible if we ever find we have to do it.)
Adding asserts might be a good idea to catch bugs when expanding the supported vector lengths. I can certainly add one here. Maybe Richard has some ideas as well for this.
Thanks,
drew
Re: [Qemu-devel] [PATCH 11/13] target/arm/cpu64: max cpu: Introduce sve-vls-map
Quoted 62 lines · Andrew Jones
Quoted 45 lines · Dave Martin
Quoted 44 lines · Andrew Jones
Quoted 43 lines · Dave Martin
Quoted 24 lines · Andrew Jones
Introduce another cpu property to control SVE vector lengths, sve-vls-map, which allows the user to explicitly select the set of vector lengths the guest can use. The map must conform to QEMU's limits and architectural constraints, checked when the property is set. Inconsistencies with sve-max-vq are also checked. The bit number of a set bit in the map represents the allowed vector length in number of quadwords.
Note, as the map is implemented with a single 64-bit word we currently only support up to 8192-bit vectors. As QEMU and KVM only support up to 2048-bit vectors then this sufficient now, and probably for some time. Extending the bitmap beyond a single word will likely require changing the property to a string and adding yet another parser to QEMU.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/cpu.c | 4 +++ target/arm/cpu.h | 3 ++ target/arm/cpu64.c | 70 +++++++++++++++++++++++++++++++++++++++++--- target/arm/helper.c | 10 ++++++- target/arm/monitor.c | 9 ++++-- 5 files changed, 88 insertions(+), 8 deletions(-)[...]
target/arm/cpu.h struct ARMCPU { · line 925
Quoted 11 lines
target/arm/cpu.h +3
-0@@ -920,6 +920,9 @@ struct ARMCPU {/* Used to set the maximum vector length the cpu will support. */ uint32_t sve_max_vq; + +/* Each bit represents a supported vector length of (bitnum * 16) bytes */ +uint64_t sve_vls_map;Just to be clear, the representation here is different from the representation in
KVM_REG_ARM64_SVE_VLS?Quoted 6 lines
Quoted 3 lines
Quoted 1 line
In the latter, bit n represents vector length ((n + 1) * 16) bytes.
KVM also uses bitnum * 16. bitnum is the the bit number, not the shift.
Can you point to the relevant kernel code? This isn't what I thought I wrote...
The
Documentation/virtual/kvm/api.txtdocumentation hasQuoted 6 lines
if (vq >=
SVE_VQ_MIN&& vq <=SVE_VQ_MAX&&
((vector_lengths[(vq -KVM_ARM64_SVE_VQ_MIN) / 64] >>((vq - KVM_ARM64_SVE_VQ_MIN) % 64)) & 1)) /* Vector length vq * 16 bytes supported */ else /* Vector length vq * 16 bytes not supported */
Quoted 5 lines
Taking vq=1, we check (vector_lengths[0] >> 0) to see if we have the length (1 * 16) bytes. Since bitnum 1 is (1 << 0) that means the shift is zero and (bitnum * 16) is the same as (vq * 16). With ((n + 1) * 16), n would have to be zero, which is not a valid bitnum, but is a valid bit shift.
OK, so it sounds like the interpretation of the KVM ABI is what I intended.
What is "bitnum" though? Most of the time, people seem to number bits starting from 0 -- though of course that's just a convention.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
So you can't have bitnum=0. 'n' must be a shift, as it would need to allow zero in order to represent
KVM_ARM64_SVE_VQ_MIN.
[...]
Quoted 26 lines
Quoted 26 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
target/arm/helper.c +1
-0@@ -5254,12 +5254,20 @@ uint32_t sve_zcr_len_for_el(CPUARMState *env, int el)static void zcr_write (CPUARMState *env,const ARMCPRegInfo *ri,uint64_t value) { +ARMCPU *cpu =arm_env_get_cpu (env);int cur_el =arm_current_el (env);int old_len =sve_zcr_len_for_el (env, cur_el);int new_len;Quoted 8 lines
Quoted 6 lines
target/arm/helper.c uint32_t sve_zcr_len_for_el(CPUARMState *env, int el) · line 5263
Quoted 3 lines
target/arm/helper.c +1
-1/* Bits other than [3:0] are RAZ/WI. */ -raw_write (env, ri, value &0xf ); + value &=0xf ;You might want to sanity-check that the max vq you configured for the vcpu is <= 16 here.
It's already checked by the cpu property set function at input time.
The check is in one place, but the assumptions based on it are potentially all over the place -- it may be tricky to track them all down if you ever add support for >8192-bit vectors.
Quoted 8 lines
Quoted 4 lines
(Not a big deal from my point of view, but I'd migration to larger vectors to be as smooth as reasonably possible if we ever find we have to do it.)
Adding asserts might be a good idea to catch bugs when expanding the supported vector lengths. I can certainly add one here. Maybe Richard has some ideas as well for this.
Sure, just wanted to make the point, since this might be an issue in the future.
The compromise in the kernel implementation was similar: in many places the code is agnostic to larger vectors, but in a few (such as the ioctl reg accessors) this seemed too cumbersome to justify for now, so I went with a simpler implementation with some added sanity-checks.
Cheers
---Dave
Re: [Qemu-devel] [PATCH 11/13] target/arm/cpu64: max cpu: Introduce sve-vls-map
Quoted 69 lines · Dave Martin
Quoted 62 lines · Andrew Jones
Quoted 45 lines · Dave Martin
Quoted 44 lines · Andrew Jones
Quoted 43 lines · Dave Martin
Quoted 24 lines · Andrew Jones
Introduce another cpu property to control SVE vector lengths, sve-vls-map, which allows the user to explicitly select the set of vector lengths the guest can use. The map must conform to QEMU's limits and architectural constraints, checked when the property is set. Inconsistencies with sve-max-vq are also checked. The bit number of a set bit in the map represents the allowed vector length in number of quadwords.
Note, as the map is implemented with a single 64-bit word we currently only support up to 8192-bit vectors. As QEMU and KVM only support up to 2048-bit vectors then this sufficient now, and probably for some time. Extending the bitmap beyond a single word will likely require changing the property to a string and adding yet another parser to QEMU.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/cpu.c | 4 +++ target/arm/cpu.h | 3 ++ target/arm/cpu64.c | 70 +++++++++++++++++++++++++++++++++++++++++--- target/arm/helper.c | 10 ++++++- target/arm/monitor.c | 9 ++++-- 5 files changed, 88 insertions(+), 8 deletions(-)[...]
target/arm/cpu.h struct ARMCPU { · line 925
Quoted 11 lines
target/arm/cpu.h +3
-0@@ -920,6 +920,9 @@ struct ARMCPU {/* Used to set the maximum vector length the cpu will support. */ uint32_t sve_max_vq; + +/* Each bit represents a supported vector length of (bitnum * 16) bytes */ +uint64_t sve_vls_map;Just to be clear, the representation here is different from the representation in
KVM_REG_ARM64_SVE_VLS?Quoted 6 lines
Quoted 3 lines
Quoted 1 line
In the latter, bit n represents vector length ((n + 1) * 16) bytes.
KVM also uses bitnum * 16. bitnum is the the bit number, not the shift.
Can you point to the relevant kernel code? This isn't what I thought I wrote...
The
Documentation/virtual/kvm/api.txtdocumentation hasQuoted 6 lines
if (vq >=
SVE_VQ_MIN&& vq <=SVE_VQ_MAX&&
((vector_lengths[(vq -KVM_ARM64_SVE_VQ_MIN) / 64] >>((vq - KVM_ARM64_SVE_VQ_MIN) % 64)) & 1)) /* Vector length vq * 16 bytes supported */ else /* Vector length vq * 16 bytes not supported */Quoted 5 lines
Taking vq=1, we check (vector_lengths[0] >> 0) to see if we have the length (1 * 16) bytes. Since bitnum 1 is (1 << 0) that means the shift is zero and (bitnum * 16) is the same as (vq * 16). With ((n + 1) * 16), n would have to be zero, which is not a valid bitnum, but is a valid bit shift.
Quoted 4 lines
OK, so it sounds like the interpretation of the KVM ABI is what I intended.
What is "bitnum" though? Most of the time, people seem to number bits starting from 0 -- though of course that's just a convention.
Eh, I see the problem. I used the word 'number', when I meant 'position'. Indeed the bit number of bit position 1 is zero. Sorry about the confusion. For v2 I can try to ensure I don't use the wrong terminology.
Quoted 41 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
So you can't have bitnum=0. 'n' must be a shift, as it would need to allow zero in order to represent
KVM_ARM64_SVE_VQ_MIN.[...]
Quoted 26 lines
Quoted 26 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
target/arm/helper.c +1
-0@@ -5254,12 +5254,20 @@ uint32_t sve_zcr_len_for_el(CPUARMState *env, int el)static void zcr_write (CPUARMState *env,const ARMCPRegInfo *ri,uint64_t value) { +ARMCPU *cpu =arm_env_get_cpu (env);int cur_el =arm_current_el (env);int old_len =sve_zcr_len_for_el (env, cur_el);int new_len;Quoted 8 lines
Quoted 6 lines
target/arm/helper.c uint32_t sve_zcr_len_for_el(CPUARMState *env, int el) · line 5263
Quoted 3 lines
target/arm/helper.c +1
-1/* Bits other than [3:0] are RAZ/WI. */ -raw_write (env, ri, value &0xf ); + value &=0xf ;You might want to sanity-check that the max vq you configured for the vcpu is <= 16 here.
It's already checked by the cpu property set function at input time.
The check is in one place, but the assumptions based on it are potentially all over the place -- it may be tricky to track them all down if you ever add support for >8192-bit vectors.
Quoted 8 lines
Quoted 4 lines
(Not a big deal from my point of view, but I'd migration to larger vectors to be as smooth as reasonably possible if we ever find we have to do it.)
Adding asserts might be a good idea to catch bugs when expanding the supported vector lengths. I can certainly add one here. Maybe Richard has some ideas as well for this.
Quoted 8 lines
Sure, just wanted to make the point, since this might be an issue in the future.
The compromise in the kernel implementation was similar: in many places the code is agnostic to larger vectors, but in a few (such as the ioctl reg accessors) this seemed too cumbersome to justify for now, so I went with a simpler implementation with some added sanity-checks.
Thanks,
drew
Re: [Qemu-devel] [PATCH 11/13] target/arm/cpu64: max cpu: Introduce sve-vls-map
Quoted 70 lines · Andrew Jones
Quoted 69 lines · Dave Martin
Quoted 67 lines · Andrew Jones
Quoted 45 lines · Dave Martin
Quoted 44 lines · Andrew Jones
Quoted 43 lines · Dave Martin
Quoted 24 lines · Andrew Jones
Introduce another cpu property to control SVE vector lengths, sve-vls-map, which allows the user to explicitly select the set of vector lengths the guest can use. The map must conform to QEMU's limits and architectural constraints, checked when the property is set. Inconsistencies with sve-max-vq are also checked. The bit number of a set bit in the map represents the allowed vector length in number of quadwords.
Note, as the map is implemented with a single 64-bit word we currently only support up to 8192-bit vectors. As QEMU and KVM only support up to 2048-bit vectors then this sufficient now, and probably for some time. Extending the bitmap beyond a single word will likely require changing the property to a string and adding yet another parser to QEMU.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/cpu.c | 4 +++ target/arm/cpu.h | 3 ++ target/arm/cpu64.c | 70 +++++++++++++++++++++++++++++++++++++++++--- target/arm/helper.c | 10 ++++++- target/arm/monitor.c | 9 ++++-- 5 files changed, 88 insertions(+), 8 deletions(-)[...]
target/arm/cpu.h struct ARMCPU { · line 925
Quoted 11 lines
target/arm/cpu.h +3
-0@@ -920,6 +920,9 @@ struct ARMCPU {/* Used to set the maximum vector length the cpu will support. */ uint32_t sve_max_vq; + +/* Each bit represents a supported vector length of (bitnum * 16) bytes */ +uint64_t sve_vls_map;Just to be clear, the representation here is different from the representation in
KVM_REG_ARM64_SVE_VLS?Quoted 6 lines
Quoted 3 lines
Quoted 1 line
In the latter, bit n represents vector length ((n + 1) * 16) bytes.
KVM also uses bitnum * 16. bitnum is the the bit number, not the shift.
Can you point to the relevant kernel code? This isn't what I thought I wrote...
The
Documentation/virtual/kvm/api.txtdocumentation hasQuoted 6 lines
if (vq >=
SVE_VQ_MIN&& vq <=SVE_VQ_MAX&&
((vector_lengths[(vq -KVM_ARM64_SVE_VQ_MIN) / 64] >>((vq - KVM_ARM64_SVE_VQ_MIN) % 64)) & 1)) /* Vector length vq * 16 bytes supported */ else /* Vector length vq * 16 bytes not supported */Taking vq=1, we check (vector_lengths[0] >> 0) to see if we have the length (1 * 16) bytes. Since bitnum 1 is (1 << 0) that means the shift is zero and (bitnum * 16) is the same as (vq * 16). With ((n + 1) * 16), n would have to be zero, which is not a valid bitnum, but is a valid bit shift.
Quoted 8 lines
Quoted 4 lines
OK, so it sounds like the interpretation of the KVM ABI is what I intended.
What is "bitnum" though? Most of the time, people seem to number bits starting from 0 -- though of course that's just a convention.
Eh, I see the problem. I used the word 'number', when I meant 'position'. Indeed the bit number of bit position 1 is zero. Sorry about the confusion. For v2 I can try to ensure I don't use the wrong terminology.
I'm not sure there's an actual problem here. I found the terminology unfamiliar, that's all.
My main concern was that people might be confused about the kernel ABI here, but it sounds like your interpretation is as I intended.
But by all means clarify the comments if you think there's a risk of QEMU folks being confused.
[...]
Cheers
---Dave
Re: [Qemu-devel] [PATCH 11/13] target/arm/cpu64: max cpu: Introduce sve-vls-map
In reply to Andrew JonesQuoted 8 lines · Andrew Jones
Introduce another cpu property to control SVE vector lengths, sve-vls-map, which allows the user to explicitly select the set of vector lengths the guest can use. The map must conform to QEMU's limits and architectural constraints, checked when the property is set. Inconsistencies with sve-max-vq are also checked. The bit number of a set bit in the map represents the allowed vector length in number of quadwords.
Quoted 8 lines
Note, as the map is implemented with a single 64-bit word we currently only support up to 8192-bit vectors. As QEMU and KVM only support up to 2048-bit vectors then this sufficient now, and probably for some time. Extending the bitmap beyond a single word will likely require changing the property to a string and adding yet another parser to QEMU.
Signed-off-by Andrew Jones <[email protected]>
Please add an example of how the new property should be used on the command line to your commit message. I'd ask for a documentation update as well if we had any to update.
Re: [Qemu-devel] [PATCH 11/13] target/arm/cpu64: max cpu: Introduce sve-vls-map
Quoted 17 lines · Markus Armbruster
Quoted 15 lines · Andrew Jones
Introduce another cpu property to control SVE vector lengths, sve-vls-map, which allows the user to explicitly select the set of vector lengths the guest can use. The map must conform to QEMU's limits and architectural constraints, checked when the property is set. Inconsistencies with sve-max-vq are also checked. The bit number of a set bit in the map represents the allowed vector length in number of quadwords.
Note, as the map is implemented with a single 64-bit word we currently only support up to 8192-bit vectors. As QEMU and KVM only support up to 2048-bit vectors then this sufficient now, and probably for some time. Extending the bitmap beyond a single word will likely require changing the property to a string and adding yet another parser to QEMU.
Quoted 5 lines
Quoted 1 line
Signed-off-by Andrew Jones <[email protected]>Please add an example of how the new property should be used on the command line to your commit message. I'd ask for a documentation update as well if we had any to update.
Will do.
Thanks,
drew
PATCH 12/13 target/arm/kvm: max cpu: Add support for sve-vls-map
In reply to Andrew JonesThe max cpu type can have its SVE vector lengths explicitly set with the sve-vls-map property. This patch allows that property to work when KVM is in use. The map must conform to additional constraints for KVM which are checked at vcpu init.
--- target/arm/cpu64.c | 7 +++--- target/arm/kvm64.c | 56 +++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 56 insertions(+), 7 deletions(-)
target/arm/cpu64.c +3 -4
@@ -310,7 +310,7 @@ static void cpu_set_sve_vls_map(Object *obj, Visitor *v, const char *name,error_setg (&err,"SVE vector length map has unsupported lengths" );error_append_hint (&err,"Valid vector lengths in range [1-%d]\n" ,ARM_MAX_VQ ); - }else if (cpu->sve_max_vq !=ARM_MAX_VQ && + }else if (cpu->sve_max_vq !=ARM_MAX_VQ && cpu->sve_max_vq !=-1 && cpu->sve_max_vq !=arm_cpu_fls64 (cpu->sve_vls_map)) {/* * If the user provides both sve-max-vq and sve-vls-map, with
@@ -433,13 +433,12 @@ static void aarch64_max_initfn(Object *obj)#endif cpu->sve_max_vq =ARM_MAX_VQ ; - -object_property_add (obj,"sve-vls-map" ,"uint64" , cpu_get_sve_vls_map, - cpu_set_sve_vls_map,NULL ,NULL , &error_fatal); }object_property_add (obj,"sve-max-vq" ,"uint32" , cpu_max_get_sve_vq, cpu_max_set_sve_vq,NULL ,NULL , &error_fatal); +object_property_add (obj,"sve-vls-map" ,"uint64" , cpu_get_sve_vls_map, + cpu_set_sve_vls_map,NULL ,NULL , &error_fatal); }struct ARMCPUInfo {
target/arm/kvm64.c +53 -3
74 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH 13/13 target/arm/kvm: host cpu: Add support for sve-vls-map
In reply to Andrew JonesAllow the host cpu type to enable SVE in guests with the sve-vls-map cpu property.
--- target/arm/cpu.c | 1 + target/arm/cpu.h | 2 ++ target/arm/cpu64.c | 12 +++++++++--- 3 files changed, 12 insertions(+), 3 deletions(-)
target/arm/cpu.c +1 -0
@@ -2222,6 +2222,7 @@ static void arm_host_initfn(Object *obj)ARMCPU *cpu =ARM_CPU (obj);kvm_arm_set_cpu_features_from_host (cpu); +aarch64_add_sve_vls_map_property (obj);arm_cpu_post_init (obj); }
target/arm/cpu.h +2 -0
@@ -976,11 +976,13 @@ int aarch64_cpu_gdb_write_register(CPUState *cpu, uint8_t *buf, int reg);void aarch64_sve_narrow_vq (CPUARMState *env,unsigned vq);void aarch64_sve_change_el (CPUARMState *env,int old_el,int new_el,bool el0_a64); +void aarch64_add_sve_vls_map_property (Object *obj);#else static inline void aarch64_sve_narrow_vq (CPUARMState *env,unsigned vq) { }static inline void aarch64_sve_change_el (CPUARMState *env,int o,int n,bool a) { } +void aarch64_add_sve_vls_map_property (Object *obj) { }#endif target_ulong do_arm_semihosting (CPUARMState *env);
target/arm/cpu64.c +9 -3
@@ -310,7 +310,8 @@ static void cpu_set_sve_vls_map(Object *obj, Visitor *v, const char *name,error_setg (&err,"SVE vector length map has unsupported lengths" );error_append_hint (&err,"Valid vector lengths in range [1-%d]\n" ,ARM_MAX_VQ ); - }else if (cpu->sve_max_vq !=ARM_MAX_VQ && cpu->sve_max_vq !=-1 && + }else if (cpu->sve_max_vq && cpu->sve_max_vq !=ARM_MAX_VQ && + cpu->sve_max_vq !=-1 && cpu->sve_max_vq !=arm_cpu_fls64 (cpu->sve_vls_map)) {/* * If the user provides both sve-max-vq and sve-vls-map, with
@@ -340,6 +341,12 @@ static void cpu_set_sve_vls_map(Object *obj, Visitor *v, const char *name,error_propagate (errp, err); } +void aarch64_add_sve_vls_map_property (Object *obj) +{ +object_property_add (obj,"sve-vls-map" ,"uint64" , cpu_get_sve_vls_map, + cpu_set_sve_vls_map,NULL ,NULL , &error_fatal); +} +/* -cpu max: if KVM is enabled, like -cpu host (best possible with this host); * otherwise, a CPU with as many features enabled as our emulation supports. * The version of '-cpu max' for qemu-system-arm is defined in cpu.c;
@@ -437,8 +444,7 @@ static void aarch64_max_initfn(Object *obj)object_property_add (obj,"sve-max-vq" ,"uint32" , cpu_max_get_sve_vq, cpu_max_set_sve_vq,NULL ,NULL , &error_fatal); -object_property_add (obj,"sve-vls-map" ,"uint64" , cpu_get_sve_vls_map, - cpu_set_sve_vls_map,NULL ,NULL , &error_fatal); +aarch64_add_sve_vls_map_property (obj); }struct ARMCPUInfo {
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH 13/13] target/arm/kvm: host cpu: Add support for sve-vls-map
Quoted 27 lines · Andrew Jones
Allow the host cpu type to enable SVE in guests with the sve-vls-map cpu property.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/cpu.c | 1 + target/arm/cpu.h | 2 ++ target/arm/cpu64.c | 12 +++++++++--- 3 files changed, 12 insertions(+), 3 deletions(-)target/arm/cpu.c +1
-0@@ -2222,6 +2222,7 @@ static void arm_host_initfn(Object *obj)ARMCPU *cpu =ARM_CPU (obj);kvm_arm_set_cpu_features_from_host (cpu); +aarch64_add_sve_vls_map_property (obj);arm_cpu_post_init (obj); }target/arm/cpu.h +0
-0
target/arm/cpu.h int aarch64_cpu_gdb_write_register(CPUState *cpu, uint8_t *buf, int reg); · line 985
Quoted 10 lines
target/arm/cpu.h +2
-0@@ -976,11 +976,13 @@ int aarch64_cpu_gdb_write_register(CPUState *cpu, uint8_t *buf, int reg);void aarch64_sve_narrow_vq (CPUARMState *env,unsigned vq);void aarch64_sve_change_el (CPUARMState *env,int old_el,int new_el,bool el0_a64); +void aarch64_add_sve_vls_map_property (Object *obj);#else static inline void aarch64_sve_narrow_vq (CPUARMState *env,unsigned vq) { }static inline void aarch64_sve_change_el (CPUARMState *env,int o,int n,bool a) { } +void aarch64_add_sve_vls_map_property (Object *obj) { }
qemu/target/arm/cpu.h:985:6: error: no previous prototype for ‘aarch64_add_sve_vls_map_property’ [-Werror=missing-prototypes]
void aarch64_add_sve_vls_map_property(Object *obj) { }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Did you forget static inline?
Quoted 3 lines
target/arm/cpu.h +0
-0#endif target_ulong do_arm_semihosting (CPUARMState *env);
[...]
Re: [Qemu-devel] [PATCH 13/13] target/arm/kvm: host cpu: Add support for sve-vls-map
Quoted 42 lines · Markus Armbruster
Quoted 27 lines · Andrew Jones
Allow the host cpu type to enable SVE in guests with the sve-vls-map cpu property.
Signed-off-by Andrew Jones <[email protected]>--- target/arm/cpu.c | 1 + target/arm/cpu.h | 2 ++ target/arm/cpu64.c | 12 +++++++++--- 3 files changed, 12 insertions(+), 3 deletions(-)target/arm/cpu.c +1
-0@@ -2222,6 +2222,7 @@ static void arm_host_initfn(Object *obj)ARMCPU *cpu =ARM_CPU (obj);kvm_arm_set_cpu_features_from_host (cpu); +aarch64_add_sve_vls_map_property (obj);arm_cpu_post_init (obj); }target/arm/cpu.h +0
-0target/arm/cpu.h int aarch64_cpu_gdb_write_register(CPUState *cpu, uint8_t *buf, int reg); · line 985
Quoted 10 lines
target/arm/cpu.h +2
-0@@ -976,11 +976,13 @@ int aarch64_cpu_gdb_write_register(CPUState *cpu, uint8_t *buf, int reg);void aarch64_sve_narrow_vq (CPUARMState *env,unsigned vq);void aarch64_sve_change_el (CPUARMState *env,int old_el,int new_el,bool el0_a64); +void aarch64_add_sve_vls_map_property (Object *obj);#else static inline void aarch64_sve_narrow_vq (CPUARMState *env,unsigned vq) { }static inline void aarch64_sve_change_el (CPUARMState *env,int o,int n,bool a) { } +void aarch64_add_sve_vls_map_property (Object *obj) { }
qemu/target/arm/cpu.h:985:6: error: no previous prototype for ‘aarch64_add_sve_vls_map_property’ [-Werror=missing-prototypes]
voidaarch64_add_sve_vls_map_property(Object *obj){ }
Quoted 3 lines
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Did you forget static inline?
I sure did. Thanks for finding this!
drew
Quoted 5 lines
Quoted 4 lines
target/arm/cpu.h +0
-0#endif target_ulong do_arm_semihosting (CPUARMState *env);[...]
On Sun, 2019-05-12 at 10:36 +0200, Andrew Jones wrote:
[...]
Quoted 9 lines
CPU type | accel | sve-max-vq | sve-vls-map
-------------------------------------------1) max | tcg | $MAX_VQ | $VLS_MAP 2) max | kvm | $MAX_VQ | $VLS_MAP 3) host | kvm | N/A | $VLS_MAPWhere for (1) $MAX_VQ sets the maximum vq and smaller vqs are all supported when $VLS_MAP is zero, or when the vqs are selected in $VLS_MAP.
I'm a bit confused by the nomenclature here. VL clearly stands for Vector Length, but what does VQ stand for? You seem to be using the two terms pretty much interchangeably throughout the cover letter.
[...]
Quoted 2 lines
There is never any need to provide both properties, but if both are provided then they are checked for consistency.
I would personally just error out when both are provided.
Quoted 10 lines
The QMP query returns a list of valid vq lists. For example, if a guest can use vqs 1, 2, 3, and 4, then the following list will be returned
[ [ 1 ], [ 1, 2 ], [ 1, 2, 3 ], [ 1, 2, 3, 4 ] ]
Another example might be 1, 2, 4, as the architecture states 3 is optional. In that case the list would be
[ [ 1 ], [ 1, 2 ], [ 1, 2, 4 ] ]
I think the proposed QMP command is problematic, because it reports the valid vector lengths for either KVM or TCG based on which accelerator is currently enabled: it should report all information at all times instead, similarly to how query-gic-capabilities does it.
[...]
Quoted 18 lines
And now for what might be a bit more controversial; how we input the valid vector set with sve-vls-map. Well, sve-vls-map is a 64-bit bitmap, which is admittedly not user friendly and also not the same size as KVM's vls bitmap (which is 8 64-bit words). Here's the justification:
1) We want to use the QEMU command line in order for the information to be migrated without needing to add more VM state.
2) It's not easy/pretty to input arrays on the QEMU command line.
3) We already need to use the QMP query to get a valid set, which isn't user friendly either, meaning we're already in libvirt territory.
4) A 64-bit map (supporting up to 8192-bit vectors) is likely big enough for quite some time (currently KVM and TCG only support 2048-bit vectors).
5) If user friendliness is needed more than migratability then the 'max' cpu type can be used with the sve-max-vq property.
6) It's possible to probe the full valid vector set from the
Quoted 3 lines
command line by using something like sve-vls-map=0xffff and
then, when it fails, the error message will state the correct
map, e.g. 0xb.I don't have a problem with having to use a bitmap internally, though libvirt will clearly want to expose a more approachable interface to users.
However, QMP reporting the information in the current format means we'd have to build an additional parser on top of the bitmap handling and conversion routines we'll clearly need to make this work; plus it just feels weird that the information reported by QMP can't be used on the command line without going through some tranformation first.
Wouldn't it make more sense to both accept and report bitmaps?
Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
Quoted 8 lines · Andrea Bolognani
On Sun, 2019-05-12 at 10:36 +0200, Andrew Jones wrote:
[...]Quoted 6 lines
CPU type | accel | sve-max-vq | sve-vls-map
-------------------------------------------1) max | tcg | $MAX_VQ | $VLS_MAP 2) max | kvm | $MAX_VQ | $VLS_MAP 3) host | kvm | N/A | $VLS_MAP
Quoted 7 lines
Quoted 3 lines
Where for (1) $MAX_VQ sets the maximum vq and smaller vqs are all supported when $VLS_MAP is zero, or when the vqs are selected in $VLS_MAP.
I'm a bit confused by the nomenclature here. VL clearly stands for Vector Length, but what does VQ stand for? You seem to be using the two terms pretty much interchangeably throughout the cover letter.
From the Linux end, "vector length" or VL refers to the size of a vector register, either in no particular unit or in bytes.
"VQ" refers specifically to the vector length in 128-bit quadwords.
In some situations, neither terminology is obviously better than the other, such as in the way KVM_REG_ARM64_SVE_VLS is encoded.
Quoted 15 lines
[...]
Quoted 2 lines
There is never any need to provide both properties, but if both are provided then they are checked for consistency.
I would personally just error out when both are provided.
Quoted 9 lines
The QMP query returns a list of valid vq lists. For example, if a guest can use vqs 1, 2, 3, and 4, then the following list will be returned
[ [ 1 ], [ 1, 2 ], [ 1, 2, 3 ], [ 1, 2, 3, 4 ] ]
Another example might be 1, 2, 4, as the architecture states 3 is optional. In that case the list would be
Quoted 7 lines
Quoted 1 line
[ [ 1 ], [ 1, 2 ], [ 1, 2, 4 ] ]
I think the proposed QMP command is problematic, because it reports the valid vector lengths for either KVM or TCG based on which accelerator is currently enabled: it should report all information at all times instead, similarly to how query-gic-capabilities does it.
I wonder if this is premature flexibility?
The size of these lists is going to get cumbersome if the architecture is ever extended. Even today, we might need over 100 items in this (nested) list. If this is to be presented to the user this will be far from friendly, it could get much worse if the architecutre changes in future to allow larger vectors or more flexible virtualisation.
Could we just have a list of supported vector lengths and a possibly empty list of additional capabilities that describe what kinds of flexibility are allowed?
So, for example, we might support vector lengths of 1, 2, 4 and 8 quadwords, with the the ability to clamp the max vector length the guest sees: the kernel ABI guarantees that you can do this, even if you can't disable/enable each individual vector length independently.
So, [ 1, 2, 4, 8 ] seems sufficient to describe this in a forwards compatible way.
Some day, we might report { "independent", [ 1, 2, 4, 8, 16, 32, ... ] }
I'm guessing about the data representation here.
[...]
Cheers
---Dave
Quoted 65 lines · Dave Martin
Quoted 8 lines · Andrea Bolognani
On Sun, 2019-05-12 at 10:36 +0200, Andrew Jones wrote:
[...]Quoted 6 lines
CPU type | accel | sve-max-vq | sve-vls-map
-------------------------------------------1) max | tcg | $MAX_VQ | $VLS_MAP 2) max | kvm | $MAX_VQ | $VLS_MAP 3) host | kvm | N/A | $VLS_MAPQuoted 7 lines
Quoted 3 lines
Where for (1) $MAX_VQ sets the maximum vq and smaller vqs are all supported when $VLS_MAP is zero, or when the vqs are selected in $VLS_MAP.
I'm a bit confused by the nomenclature here. VL clearly stands for Vector Length, but what does VQ stand for? You seem to be using the two terms pretty much interchangeably throughout the cover letter.
From the Linux end, "vector length" or VL refers to the size of a vector register, either in no particular unit or in bytes.
"VQ" refers specifically to the vector length in 128-bit quadwords.
In some situations, neither terminology is obviously better than the other, such as in the way
KVM_REG_ARM64_SVE_VLSis encoded.Quoted 15 lines
[...]
Quoted 2 lines
There is never any need to provide both properties, but if both are provided then they are checked for consistency.
I would personally just error out when both are provided.
Quoted 9 lines
The QMP query returns a list of valid vq lists. For example, if a guest can use vqs 1, 2, 3, and 4, then the following list will be returned
[ [ 1 ], [ 1, 2 ], [ 1, 2, 3 ], [ 1, 2, 3, 4 ] ]
Another example might be 1, 2, 4, as the architecture states 3 is optional. In that case the list would be
Quoted 7 lines
Quoted 1 line
[ [ 1 ], [ 1, 2 ], [ 1, 2, 4 ] ]
I think the proposed QMP command is problematic, because it reports the valid vector lengths for either KVM or TCG based on which accelerator is currently enabled: it should report all information at all times instead, similarly to how query-gic-capabilities does it.
I wonder if this is premature flexibility?
The size of these lists is going to get cumbersome if the architecture is ever extended. Even today, we might need over 100 items in this (nested) list. If this is to be presented to the user this will be far from friendly, it could get much worse if the architecutre changes in future to allow larger vectors or more flexible virtualisation.
Could we just have a list of supported vector lengths and a possibly empty list of additional capabilities that describe what kinds of flexibility are allowed?
So, for example, we might support vector lengths of 1, 2, 4 and 8 quadwords, with the the ability to clamp the max vector length the guest sees: the kernel ABI guarantees that you can do this, even if you can't disable/enable each individual vector length independently.
Quoted 7 lines
So, [ 1, 2, 4, 8 ] seems sufficient to describe this in a forwards compatible way.
Some day, we might report { "independent", [ 1, 2, 4, 8, 16, 32, ... ] }
I'm guessing about the data representation here.
I think that could work, and something along those lines even crossed my mind. Let's see what libvirt folk say. I'm not overly concerned about user friendless here though, as users aren't running QMP commands and parsing json by hand too much.
Thanks,
drew
Quoted 3 lines · Andrew Jones
Quoted 2 lines · Dave Martin
Quoted 1 line · Andrea Bolognani
On Sun, 2019-05-12 at 10:36 +0200, Andrew Jones wrote:
[...]
Quoted 39 lines
Quoted 39 lines
Quoted 9 lines
Quoted 9 lines
The QMP query returns a list of valid vq lists. For example, if a guest can use vqs 1, 2, 3, and 4, then the following list will be returned
[ [ 1 ], [ 1, 2 ], [ 1, 2, 3 ], [ 1, 2, 3, 4 ] ]
Another example might be 1, 2, 4, as the architecture states 3 is optional. In that case the list would be
Quoted 7 lines
Quoted 1 line
[ [ 1 ], [ 1, 2 ], [ 1, 2, 4 ] ]
I think the proposed QMP command is problematic, because it reports the valid vector lengths for either KVM or TCG based on which accelerator is currently enabled: it should report all information at all times instead, similarly to how query-gic-capabilities does it.
I wonder if this is premature flexibility?
The size of these lists is going to get cumbersome if the architecture is ever extended. Even today, we might need over 100 items in this (nested) list. If this is to be presented to the user this will be far from friendly, it could get much worse if the architecutre changes in future to allow larger vectors or more flexible virtualisation.
Could we just have a list of supported vector lengths and a possibly empty list of additional capabilities that describe what kinds of flexibility are allowed?
So, for example, we might support vector lengths of 1, 2, 4 and 8 quadwords, with the the ability to clamp the max vector length the guest sees: the kernel ABI guarantees that you can do this, even if you can't disable/enable each individual vector length independently.
So, [ 1, 2, 4, 8 ] seems sufficient to describe this in a forwards compatible way.
Some day, we might report { "independent", [ 1, 2, 4, 8, 16, 32, ... ] }
Quoted 7 lines
Quoted 2 lines
I'm guessing about the data representation here.
I think that could work, and something along those lines even crossed my mind. Let's see what libvirt folk say. I'm not overly concerned about user friendless here though, as users aren't running QMP commands and parsing json by hand too much.
One concern I have is that if the architecture ever supports masking out vector lengths individually, the size of this description becomes something like O(2^n) in the maximum vector length, instead of O(n^2).
That could be very painful to deal with...
Cheers
---Dave
Quoted 8 lines · Andrea Bolognani
On Sun, 2019-05-12 at 10:36 +0200, Andrew Jones wrote:
[...]Quoted 6 lines
CPU type | accel | sve-max-vq | sve-vls-map
-------------------------------------------1) max | tcg | $MAX_VQ | $VLS_MAP 2) max | kvm | $MAX_VQ | $VLS_MAP 3) host | kvm | N/A | $VLS_MAP
Quoted 7 lines
Quoted 3 lines
Where for (1) $MAX_VQ sets the maximum vq and smaller vqs are all supported when $VLS_MAP is zero, or when the vqs are selected in $VLS_MAP.
I'm a bit confused by the nomenclature here. VL clearly stands for Vector Length, but what does VQ stand for? You seem to be using the two terms pretty much interchangeably throughout the cover letter.
As Dave pointed out, they're both lengths, but VQ specifically points out that the unit is 'Q'uadwords. We could use VQS instead of VLS, "Vector Lengths" sounds better.
Quoted 6 lines
[...]
Quoted 2 lines
There is never any need to provide both properties, but if both are provided then they are checked for consistency.
I would personally just error out when both are provided.
I'm fine with that if nobody else objects.
Quoted 10 lines
Quoted 10 lines
The QMP query returns a list of valid vq lists. For example, if a guest can use vqs 1, 2, 3, and 4, then the following list will be returned
[ [ 1 ], [ 1, 2 ], [ 1, 2, 3 ], [ 1, 2, 3, 4 ] ]
Another example might be 1, 2, 4, as the architecture states 3 is optional. In that case the list would be
Quoted 7 lines
Quoted 1 line
[ [ 1 ], [ 1, 2 ], [ 1, 2, 4 ] ]
I think the proposed QMP command is problematic, because it reports the valid vector lengths for either KVM or TCG based on which accelerator is currently enabled: it should report all information at all times instead, similarly to how query-gic-capabilities does it.
OK, and then with a flag stating which is when then. Dave points out we may want to reduce the list to a single set and then add flags to indicate what can be done with it in order to derive other sets. What do you think about that?
Quoted 28 lines
[...]
Quoted 18 lines
And now for what might be a bit more controversial; how we input the valid vector set with sve-vls-map. Well, sve-vls-map is a 64-bit bitmap, which is admittedly not user friendly and also not the same size as KVM's vls bitmap (which is 8 64-bit words). Here's the justification:
1) We want to use the QEMU command line in order for the information to be migrated without needing to add more VM state.
2) It's not easy/pretty to input arrays on the QEMU command line.
3) We already need to use the QMP query to get a valid set, which isn't user friendly either, meaning we're already in libvirt territory.
4) A 64-bit map (supporting up to 8192-bit vectors) is likely big enough for quite some time (currently KVM and TCG only support 2048-bit vectors).
5) If user friendliness is needed more than migratability then the 'max' cpu type can be used with the sve-max-vq property.
6) It's possible to probe the full valid vector set from theQuoted 3 lines
command line by using something like sve-vls-map=0xffff and then, when it fails, the error message will state the correct map, e.g. 0xb.I don't have a problem with having to use a bitmap internally, though libvirt will clearly want to expose a more approachable interface to users.
Quoted 7 lines
However, QMP reporting the information in the current format means we'd have to build an additional parser on top of the bitmap handling and conversion routines we'll clearly need to make this work; plus it just feels weird that the information reported by QMP can't be used on the command line without going through some tranformation first.
Wouldn't it make more sense to both accept and report bitmaps?
If we eventually need more than one word for the bitmap then it'll require parsing and bitmap composition code in libvirt anyway. I was thinking by pointing out each bit separately that we could boundlessly grow the list without having to change anything in libvirt later.
Thanks,
drew
Quoted 13 lines · Andrew Jones
Quoted 12 lines · Andrea Bolognani
On Sun, 2019-05-12 at 10:36 +0200, Andrew Jones wrote:
[...]Quoted 9 lines
CPU type | accel | sve-max-vq | sve-vls-map
-------------------------------------------1) max | tcg | $MAX_VQ | $VLS_MAP 2) max | kvm | $MAX_VQ | $VLS_MAP 3) host | kvm | N/A | $VLS_MAPWhere for (1) $MAX_VQ sets the maximum vq and smaller vqs are all supported when $VLS_MAP is zero, or when the vqs are selected in $VLS_MAP.
Quoted 7 lines
Quoted 3 lines
I'm a bit confused by the nomenclature here. VL clearly stands for Vector Length, but what does VQ stand for? You seem to be using the two terms pretty much interchangeably throughout the cover letter.
As Dave pointed out, they're both lengths, but VQ specifically points out that the unit is 'Q'uadwords. We could use VQS instead of VLS, "Vector Lengths" sounds better.
Alright, it makes sense - once you've managed to figure out what exactly a "quadword" is, at least :)
Since we expect management applications to use QMP to discover what vector lengths are supported and then provide an explicit map, I think it's fair to say that the ability to specify a single maximum vector length is going to be exclusively used as a convenience for command line users.
In that light, I think it would be reasonable for the usage to look along the lines of
-cpu host,sve-vl-map=0xd # machine-friendly variant -cpu max,sve-vl-max=512 # user-friendly variant
with documentation clearly pointing out that the two options expect completely different formats - but that was the case even before, so we would have had to document that anyway.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
The QMP query returns a list of valid vq lists. For example, if a guest can use vqs 1, 2, 3, and 4, then the following list will be returned
[ [ 1 ], [ 1, 2 ], [ 1, 2, 3 ], [ 1, 2, 3, 4 ] ]
Another example might be 1, 2, 4, as the architecture states 3 is optional. In that case the list would be
[ [ 1 ], [ 1, 2 ], [ 1, 2, 4 ] ]
Quoted 7 lines
Quoted 5 lines
I think the proposed QMP command is problematic, because it reports the valid vector lengths for either KVM or TCG based on which accelerator is currently enabled: it should report all information at all times instead, similarly to how query-gic-capabilities does it.
OK, and then with a flag stating which is when then.
Correct.
Quoted 4 lines
Dave points out
we may want to reduce the list to a single set and then add flags to indicate what can be done with it in order to derive other sets. What do you think about that?
So right now all that can be done is truncating the list by removing an arbitrary number of elements from the end, right? Eg. if you have [ 1, 2, 4 ] you can use either that or [ 1, 2 ] or [ 1 ]. But in the future you might also be able to mask single elements in the middle of the list, thus enabling things like [ 1, 4 ].
That doesn't sound very complicated to support in libvirt, though I have to say that I'm not a big fan of this proposal because as far as I can see it basically means implementing the very same logic twice, once in QEMU and then once more in libvirt.
Quoted 33 lines
Quoted 33 lines
[...]
Quoted 18 lines
And now for what might be a bit more controversial; how we input the valid vector set with sve-vls-map. Well, sve-vls-map is a 64-bit bitmap, which is admittedly not user friendly and also not the same size as KVM's vls bitmap (which is 8 64-bit words). Here's the justification:
1) We want to use the QEMU command line in order for the information to be migrated without needing to add more VM state.
2) It's not easy/pretty to input arrays on the QEMU command line.
3) We already need to use the QMP query to get a valid set, which isn't user friendly either, meaning we're already in libvirt territory.
4) A 64-bit map (supporting up to 8192-bit vectors) is likely big enough for quite some time (currently KVM and TCG only support 2048-bit vectors).
5) If user friendliness is needed more than migratability then the 'max' cpu type can be used with the sve-max-vq property.
6) It's possible to probe the full valid vector set from theQuoted 3 lines
command line by using something like sve-vls-map=0xffff and then, when it fails, the error message will state the correct map, e.g. 0xb.I don't have a problem with having to use a bitmap internally, though libvirt will clearly want to expose a more approachable interface to users.
However, QMP reporting the information in the current format means we'd have to build an additional parser on top of the bitmap handling and conversion routines we'll clearly need to make this work; plus it just feels weird that the information reported by QMP can't be used on the command line without going through some tranformation first.
Quoted 7 lines
Quoted 1 line
Wouldn't it make more sense to both accept and report bitmaps?
If we eventually need more than one word for the bitmap then it'll require parsing and bitmap composition code in libvirt anyway. I was thinking by pointing out each bit separately that we could boundlessly grow the list without having to change anything in libvirt later.
If the size of the bitmap on the KVM side is 512 bits, why don't we just make it that size on the QEMU side too from the start?
Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
Quoted 31 lines · Andrea Bolognani
Quoted 13 lines · Andrew Jones
Quoted 12 lines · Andrea Bolognani
On Sun, 2019-05-12 at 10:36 +0200, Andrew Jones wrote:
[...]Quoted 9 lines
CPU type | accel | sve-max-vq | sve-vls-map
-------------------------------------------1) max | tcg | $MAX_VQ | $VLS_MAP 2) max | kvm | $MAX_VQ | $VLS_MAP 3) host | kvm | N/A | $VLS_MAPWhere for (1) $MAX_VQ sets the maximum vq and smaller vqs are all supported when $VLS_MAP is zero, or when the vqs are selected in $VLS_MAP.
Quoted 7 lines
Quoted 3 lines
I'm a bit confused by the nomenclature here. VL clearly stands for Vector Length, but what does VQ stand for? You seem to be using the two terms pretty much interchangeably throughout the cover letter.
As Dave pointed out, they're both lengths, but VQ specifically points out that the unit is 'Q'uadwords. We could use VQS instead of VLS, "Vector Lengths" sounds better.
Alright, it makes sense - once you've managed to figure out what exactly a "quadword" is, at least :)
Since we expect management applications to use QMP to discover what vector lengths are supported and then provide an explicit map, I think it's fair to say that the ability to specify a single maximum vector length is going to be exclusively used as a convenience for command line users.
Quoted 5 lines
In that light, I think it would be reasonable for the usage to look along the lines of
-cpu host,sve-vl-map=0xd # machine-friendly variant -cpu max,sve-vl-max=512 # user-friendly variant
We already have sve-max-vq, so I'm not sure we want to rename it.
Quoted 37 lines
with documentation clearly pointing out that the two options expect completely different formats - but that was the case even before, so we would have had to document that anyway.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
The QMP query returns a list of valid vq lists. For example, if a guest can use vqs 1, 2, 3, and 4, then the following list will be returned
[ [ 1 ], [ 1, 2 ], [ 1, 2, 3 ], [ 1, 2, 3, 4 ] ]
Another example might be 1, 2, 4, as the architecture states 3 is optional. In that case the list would be
[ [ 1 ], [ 1, 2 ], [ 1, 2, 4 ] ]
Quoted 7 lines
Quoted 5 lines
I think the proposed QMP command is problematic, because it reports the valid vector lengths for either KVM or TCG based on which accelerator is currently enabled: it should report all information at all times instead, similarly to how query-gic-capabilities does it.
OK, and then with a flag stating which is when then.
Correct.
Quoted 4 lines
Dave points out
we may want to reduce the list to a single set and then add flags to indicate what can be done with it in order to derive other sets. What do you think about that?So right now all that can be done is truncating the list by removing an arbitrary number of elements from the end, right? Eg. if you have [ 1, 2, 4 ] you can use either that or [ 1, 2 ] or [ 1 ]. But in the future you might also be able to mask single elements in the middle of the list, thus enabling things like [ 1, 4 ].
Quoted 4 lines
That doesn't sound very complicated to support in libvirt, though I have to say that I'm not a big fan of this proposal because as far as I can see it basically means implementing the very same logic twice, once in QEMU and then once more in libvirt.
So if big tables of bits aren't a problem for QMP queries, then I'll just leave the design as it is.
Quoted 36 lines
Quoted 36 lines
Quoted 35 lines
[...]
Quoted 18 lines
And now for what might be a bit more controversial; how we input the valid vector set with sve-vls-map. Well, sve-vls-map is a 64-bit bitmap, which is admittedly not user friendly and also not the same size as KVM's vls bitmap (which is 8 64-bit words). Here's the justification:
1) We want to use the QEMU command line in order for the information to be migrated without needing to add more VM state.
2) It's not easy/pretty to input arrays on the QEMU command line.
3) We already need to use the QMP query to get a valid set, which isn't user friendly either, meaning we're already in libvirt territory.
4) A 64-bit map (supporting up to 8192-bit vectors) is likely big enough for quite some time (currently KVM and TCG only support 2048-bit vectors).
5) If user friendliness is needed more than migratability then the 'max' cpu type can be used with the sve-max-vq property.
6) It's possible to probe the full valid vector set from theQuoted 3 lines
command line by using something like sve-vls-map=0xffff and then, when it fails, the error message will state the correct map, e.g. 0xb.I don't have a problem with having to use a bitmap internally, though libvirt will clearly want to expose a more approachable interface to users.
However, QMP reporting the information in the current format means we'd have to build an additional parser on top of the bitmap handling and conversion routines we'll clearly need to make this work; plus it just feels weird that the information reported by QMP can't be used on the command line without going through some tranformation first.
Wouldn't it make more sense to both accept and report bitmaps?
Quoted 8 lines
Quoted 5 lines
If we eventually need more than one word for the bitmap then it'll require parsing and bitmap composition code in libvirt anyway. I was thinking by pointing out each bit separately that we could boundlessly grow the list without having to change anything in libvirt later.
If the size of the bitmap on the KVM side is 512 bits, why don't we just make it that size on the QEMU side too from the start?
I'd still only want to input 64-bits on the command line, otherwise we get into inputting arrays, which isn't easy. KVM's interface is meant for expansion, but it won't be using most of those bits for quite some time either.
Thanks,
drew
Quoted 7 lines · Andrew Jones
Quoted 6 lines · Andrea Bolognani
Since we expect management applications to use QMP to discover what vector lengths are supported and then provide an explicit map, I think it's fair to say that the ability to specify a single maximum vector length is going to be exclusively used as a convenience for command line users.
Quoted 7 lines
Quoted 5 lines
In that light, I think it would be reasonable for the usage to look along the lines of
-cpu host,sve-vl-map=0xd # machine-friendly variant -cpu max,sve-vl-max=512 # user-friendly variant
We already have sve-max-vq, so I'm not sure we want to rename it.
Oh, I didn't realize that was the case. And of course it already takes a number of quadwords as argument, I suppose? That's pretty unfortunate :(
Perhaps we could consider deprecating it in favor of a user-friendly variant that's actually suitable for regular humans, like the one I suggest above?
[...]
Quoted 11 lines
Quoted 11 lines
Quoted 4 lines
Dave points out
we may want to reduce the list to a single set and then add flags to indicate what can be done with it in order to derive other sets. What do you think about that?So right now all that can be done is truncating the list by removing an arbitrary number of elements from the end, right? Eg. if you have [ 1, 2, 4 ] you can use either that or [ 1, 2 ] or [ 1 ]. But in the future you might also be able to mask single elements in the middle of the list, thus enabling things like [ 1, 4 ].
Quoted 7 lines
Quoted 4 lines
That doesn't sound very complicated to support in libvirt, though I have to say that I'm not a big fan of this proposal because as far as I can see it basically means implementing the very same logic twice, once in QEMU and then once more in libvirt.
So if big tables of bits aren't a problem for QMP queries, then I'll just leave the design as it is.
I thought about it a bit more and perhaps the simplified design is better after all.
Whatever the interface looks like on the QEMU side, we're going to want to offer libvirt users two options for configuring vector
lengths: listing all desired vector lengths explicitly (basically sev-vl-map but more verbose and readable) and providing just the biggest desired vector length (like in sev-max-vq).
In the latter case, we'll want to expand the user-provided value into an explicit list anyway in order to guarantee guest ABI stability, and doing so when a single bitmap has been obtained via QMP seems like it would be more manageable.
Sorry for the flip-flop, but after all isn't this exactly what upstream design discussion is supposed to be all about? :)
[...]
Quoted 7 lines
Quoted 2 lines
If the size of the bitmap on the KVM side is 512 bits, why don't we just make it that size on the QEMU side too from the start?
I'd still only want to input 64-bits on the command line, otherwise we get into inputting arrays, which isn't easy. KVM's interface is meant for expansion, but it won't be using most of those bits for quite some time either.
I'm probably missing something entirely obvious, but couldn't you just have a single, possibly fairly massive (up to 128 hex digits if I'm not mistaken) value on the command line and just work with that one, no arrays necessary?
Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
Quoted 10 lines · Andrea Bolognani
Quoted 1 line · Andrew Jones
We already have sve-max-vq, so I'm not sure we want to rename it.
Oh, I didn't realize that was the case. And of course it already takes a number of quadwords as argument, I suppose? That's pretty unfortunate :(
Perhaps we could consider deprecating it in favor of a user-friendly variant that's actually suitable for regular humans, like the one I suggest above?
Why is =4 less user-friendly than =512?
I don't actually see "total bits in vector" as more user-friendly than "number of quadwords" when it comes to non-powers-of-2 like =7 vs =896 or =13 vs =1664.
r~
Quoted 8 lines · Richard Henderson
Quoted 7 lines · Andrea Bolognani
Quoted 1 line · Andrew Jones
We already have sve-max-vq, so I'm not sure we want to rename it.
Oh, I didn't realize that was the case. And of course it already takes a number of quadwords as argument, I suppose? That's pretty unfortunate :(
Quoted 8 lines
Quoted 3 lines
Perhaps we could consider deprecating it in favor of a user-friendly variant that's actually suitable for regular humans, like the one I suggest above?
Why is =4 less user-friendly than =512?
I don't actually see "total bits in vector" as more user-friendly than "number of quadwords" when it comes to non-powers-of-2 like =7 vs =896 or =13 vs =1664.
I would wager most people are intimately familiar with bits, bytes and multiples due to having to work with them daily. Quadwords, not so much.
Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
Quoted 13 lines · Andrea Bolognani
Quoted 12 lines · Richard Henderson
Quoted 10 lines · Andrea Bolognani
Quoted 1 line · Andrew Jones
We already have sve-max-vq, so I'm not sure we want to rename it.
Oh, I didn't realize that was the case. And of course it already takes a number of quadwords as argument, I suppose? That's pretty unfortunate :(
Perhaps we could consider deprecating it in favor of a user-friendly variant that's actually suitable for regular humans, like the one I suggest above?
Quoted 8 lines
Quoted 4 lines
Why is =4 less user-friendly than =512?
I don't actually see "total bits in vector" as more user-friendly than "number of quadwords" when it comes to non-powers-of-2 like =7 vs =896 or =13 vs =1664.
I would wager most people are intimately familiar with bits, bytes and multiples due to having to work with them daily. Quadwords, not so much.
Generally I tend to agree. For kvmtool I leaned torward quadwords purely because
16,32,48,64,80,96,112,128,144,160,176,192,208
is a big pain to type compared with
1,2,3,4,5,6,7,8,9,10,11,12,13
Even though I prefer to specify vector lengths in bytes everywhere else in the Linux user API (precisely to avoid the confusion you object to).
This isn't finalised yet for kvmtool -- I need to rework the patches and may not include it at all initially: kvmtool doesn't support migration, which is the main usecase for being able to specify an exact set of vector lengths AFAICT.
Since this is otherwise only useful for migration, experimentation or machine-driven configuration, a bitmask
0x1fff
as some have suggested may well be a pragmatic alternative for kvmtool.
Cheers
---Dave
Quoted 28 lines · Dave Martin
Quoted 9 lines · Andrea Bolognani
Quoted 4 lines · Richard Henderson
Why is =4 less user-friendly than =512?
I don't actually see "total bits in vector" as more user-friendly than "number of quadwords" when it comes to non-powers-of-2 like =7 vs =896 or =13 vs =1664.
I would wager most people are intimately familiar with bits, bytes and multiples due to having to work with them daily. Quadwords, not so much.
Generally I tend to agree. For kvmtool I leaned torward quadwords purely because
16,32,48,64,80,96,112,128,144,160,176,192,208is a big pain to type compared with
1,2,3,4,5,6,7,8,9,10,11,12,13Even though I prefer to specify vector lengths in bytes everywhere else in the Linux user API (precisely to avoid the confusion you object to).
This isn't finalised yet for kvmtool -- I need to rework the patches and may not include it at all initially: kvmtool doesn't support migration, which is the main usecase for being able to specify an exact set of vector lengths AFAICT.
Quoted 6 lines
Since this is otherwise only useful for migration, experimentation or machine-driven configuration, a bitmask
0x1fffas some have suggested may well be a pragmatic alternative for kvmtool.
Just to be clear, I have suggested using bits (or bytes or megabytes depending on the exact value) only for the command-line-user-oriented sve-vl-max option, which would take a single value.
For interoperation with the management layer, on the other hand, using a bitmap is perfectly fine, and whether the values encoded within are expressed in quadwords or whatever other format is largely irrelevant, so long as it it's properly documented of course.
Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
Quoted 36 lines · Andrea Bolognani
Quoted 28 lines · Dave Martin
Quoted 9 lines · Andrea Bolognani
Quoted 4 lines · Richard Henderson
Why is =4 less user-friendly than =512?
I don't actually see "total bits in vector" as more user-friendly than "number of quadwords" when it comes to non-powers-of-2 like =7 vs =896 or =13 vs =1664.
I would wager most people are intimately familiar with bits, bytes and multiples due to having to work with them daily. Quadwords, not so much.
Generally I tend to agree. For kvmtool I leaned torward quadwords purely because
16,32,48,64,80,96,112,128,144,160,176,192,208is a big pain to type compared with
1,2,3,4,5,6,7,8,9,10,11,12,13Even though I prefer to specify vector lengths in bytes everywhere else in the Linux user API (precisely to avoid the confusion you object to).
This isn't finalised yet for kvmtool -- I need to rework the patches and may not include it at all initially: kvmtool doesn't support migration, which is the main usecase for being able to specify an exact set of vector lengths AFAICT.
Quoted 6 lines
Since this is otherwise only useful for migration, experimentation or machine-driven configuration, a bitmask
0x1fffas some have suggested may well be a pragmatic alternative for kvmtool.
Quoted 8 lines
Just to be clear, I have suggested using bits (or bytes or megabytes depending on the exact value) only for the command-line-user-oriented sve-vl-max option, which would take a single value.
For interoperation with the management layer, on the other hand, using a bitmap is perfectly fine, and whether the values encoded within are expressed in quadwords or whatever other format is largely irrelevant, so long as it it's properly documented of course.
Seems fair.
Cheers
---Dave
Quoted 9 lines · Andrea Bolognani
I thought about it a bit more and perhaps the simplified design is better after all.
Whatever the interface looks like on the QEMU side, we're going to want to offer libvirt users two options for configuring vector
lengths: listing all desired vector lengths explicitly (basically sev-vl-map but more verbose and readable) and providing just the biggest desired vector length (like in sev-max-vq).
Quoted 7 lines
In the latter case, we'll want to expand the user-provided value into an explicit list anyway in order to guarantee guest ABI stability, and doing so when a single bitmap has been obtained via QMP seems like it would be more manageable.
Sorry for the flip-flop, but after all isn't this exactly what upstream design discussion is supposed to be all about? :)
Yup, no problem. I'm actually liking the idea of the single map plus flags. We won't need two implementations (QEMU and libvirt), we'll only need one (libvirt). The QEMU QMP side will only need to state what should be implemented using the flags. Also, as we already agreed, we need TCG and KVM flags anyway, so we're already in flag land.
Quoted 10 lines
[...]
Quoted 7 lines
Quoted 2 lines
If the size of the bitmap on the KVM side is 512 bits, why don't we just make it that size on the QEMU side too from the start?
I'd still only want to input 64-bits on the command line, otherwise we get into inputting arrays, which isn't easy. KVM's interface is meant for expansion, but it won't be using most of those bits for quite some time either.
Quoted 5 lines
I'm probably missing something entirely obvious, but couldn't you just have a single, possibly fairly massive (up to 128 hex digits if I'm not mistaken) value on the command line and just work with that one, no arrays necessary?
We could, and I like the idea. It just hadn't crossed my mind due to implementation tunnel vision.
Thanks,
drew
Quoted 23 lines · Andrew Jones
With the recent KVM guest SVE support pull request [1] KVM will be ready for guests with SVE. This series provides the QEMU bits for that enablement. The series starts with the bits needed for the KVM SVE ioctls. Then it enables the arm 'max'cpu type, which with TCG already supports SVE, to also support SVE when using KVM. Next a new QMP query is added that allows users to ask which vector lengths are supported by the host, allowing them to select a valid set of vectors for the guest. In order to select those vectors a new property 'sve-vls-map' is added to the 'max' cpu type, and then also to the 'host' cpu type. The table below shows the resulting user interfaces.
CPU type | accel | sve-max-vq | sve-vls-map
-------------------------------------------1) max | tcg | $MAX_VQ | $VLS_MAP 2) max | kvm | $MAX_VQ | $VLS_MAP 3) host | kvm | N/A | $VLS_MAPWhere for (1) $MAX_VQ sets the maximum vq and smaller vqs are all supported when $VLS_MAP is zero, or when the vqs are selected in $VLS_MAP.
Quoted 5 lines
(2) is the same as (1) except KVM has the final say on what vqs are valid.
(3) doesn't accept sve-max-vq because a guest that uses this property without sve-vls-map cannot be safely migrated.
Is this "migrated between two hosts with the same host CPU type but with different kernel versions which expose different subsets of the host's permitted vector lengths" ?
thanks
-- PMM
Quoted 27 lines · Peter Maydell
Quoted 26 lines · Andrew Jones
With the recent KVM guest SVE support pull request [1] KVM will be ready for guests with SVE. This series provides the QEMU bits for that enablement. The series starts with the bits needed for the KVM SVE ioctls. Then it enables the arm 'max'cpu type, which with TCG already supports SVE, to also support SVE when using KVM. Next a new QMP query is added that allows users to ask which vector lengths are supported by the host, allowing them to select a valid set of vectors for the guest. In order to select those vectors a new property 'sve-vls-map' is added to the 'max' cpu type, and then also to the 'host' cpu type. The table below shows the resulting user interfaces.
CPU type | accel | sve-max-vq | sve-vls-map
-------------------------------------------1) max | tcg | $MAX_VQ | $VLS_MAP 2) max | kvm | $MAX_VQ | $VLS_MAP 3) host | kvm | N/A | $VLS_MAPWhere for (1) $MAX_VQ sets the maximum vq and smaller vqs are all supported when $VLS_MAP is zero, or when the vqs are selected in $VLS_MAP.
(2) is the same as (1) except KVM has the final say on what vqs are valid.
Quoted 7 lines
Quoted 2 lines
(3) doesn't accept sve-max-vq because a guest that uses this property without sve-vls-map cannot be safely migrated.
Is this "migrated between two hosts with the same host CPU type but with different kernel versions which expose different subsets of the host's permitted vector lengths" ?
That's one example. Also, I see attempting to make/use only migrate-safe cpu properties for our primary KVM cpu type ('host') as a step in the right direction to eventually making a more migratable KVM cpu type. But I do see that (3) is a bit of an oxymoron considering the 'host' cpu type isn't really migrate-safe unless the hosts are identical (which should technically require the same host kernel too) anyway.
Thanks,
drew
Quoted 5 lines · Andrew Jones
CPU type | accel | sve-max-vq | sve-vls-map
-------------------------------------------1) max | tcg | $MAX_VQ | $VLS_MAP 2) max | kvm | $MAX_VQ | $VLS_MAP 3) host | kvm | N/A | $VLS_MAP
This doesn't seem right. Why is -cpu host not whatever the host supports? It certainly has been so far. I really don't see how -cpu max makes any sense for kvm.
Quoted 9 lines
The QMP query returns a list of valid vq lists. For example, if a guest can use vqs 1, 2, 3, and 4, then the following list will be returned
[ [ 1 ], [ 1, 2 ], [ 1, 2, 3 ], [ 1, 2, 3, 4 ] ]
Another example might be 1, 2, 4, as the architecture states 3 is optional. In that case the list would be
Quoted 6 lines
[ [ 1 ], [ 1, 2 ], [ 1, 2, 4 ] ]
This may look redundant, but it's necessary to provide a future- proof query, because while KVM currently requires vector sets to be strict truncations of the full valid vector set, that may change at some point.
How and why would that make sense?
Real hardware is going to support one set of vector lengths. Whether VQ=3 is valid or not is not going to depend on the maximum VQ, surely.
I'll also note that if we want to support the theoretical beyond-current-architecture maximum VQ=512, such that migration works seemlessly with current hardware, then we're going to have to change the migration format.
So far I'm supporting only the current architecture maximum VQ=16. Which seemed plenty, given that the first round of hardware only supports VQ=4.
r~
Quoted 9 lines · Richard Henderson
Quoted 5 lines · Andrew Jones
CPU type | accel | sve-max-vq | sve-vls-map
-------------------------------------------1) max | tcg | $MAX_VQ | $VLS_MAP 2) max | kvm | $MAX_VQ | $VLS_MAP 3) host | kvm | N/A | $VLS_MAPThis doesn't seem right. Why is -cpu host not whatever the host supports? It certainly has been so far.
-cpu host can support whatever the host (hardware + KVM ) supports, but if a user doesn't want to expose all of it to the guest, then the user doesn't have to. For example, the host cpu may have a PMU, but the guest doesn't necessarily get one (-cpu host,pmu=off).
Quoted 3 lines
I really don't see how -cpu max makes any sense for kvm.
It's already supported for kvm. This series just extends that support to match tcg's sve support. The reason it's supported is that you can then use '-cpu max' along with '-machine accel=kvm:tcg' in a command line and it'll just work.
Quoted 12 lines
Quoted 12 lines
The QMP query returns a list of valid vq lists. For example, if a guest can use vqs 1, 2, 3, and 4, then the following list will be returned
[ [ 1 ], [ 1, 2 ], [ 1, 2, 3 ], [ 1, 2, 3, 4 ] ]
Another example might be 1, 2, 4, as the architecture states 3 is optional. In that case the list would be
[ [ 1 ], [ 1, 2 ], [ 1, 2, 4 ] ]
Quoted 8 lines
Quoted 4 lines
This may look redundant, but it's necessary to provide a future- proof query, because while KVM currently requires vector sets to be strict truncations of the full valid vector set, that may change at some point.
How and why would that make sense?
Real hardware is going to support one set of vector lengths.
The guest's view of the hardware will be a single set. That set can be different for different guests though, within the constraints of the architecture and KVM or TCG implementations.
Quoted 2 lines
Whether VQ=3 is
valid or not is not going to depend on the maximum VQ, surely.
Exactly. That's why we need a way to explicitly state what is supported. We can't assume anything from the max VQ alone. Additionally, while strict truncation is required now for KVM, things may be more flexible later. TCG is already more flexible. For TCG, all sets that at least include all the power-of-2 lengths up to the maximum VQ are valid, as the architecture states. Plus, all the sets that can be derived by adding one ore more optional lengths to those sets are also valid. Listing each of them allows management software to know what's going to work and what's not without having to know all the rules itself.
Quoted 9 lines
I'll also note that if we want to support the theoretical beyond-current-architecture maximum VQ=512, such that migration works seemlessly with current hardware, then we're going to have to change the migration format.
So far I'm supporting only the current architecture maximum VQ=16. Which seemed plenty, given that the first round of hardware only supports VQ=4.
I agree. The changes won't be small to QEMU, but hopefully we can design a QMP query that won't need to change, saving libvirt some pain.
Thanks,
drew
Quoted 11 lines · Richard Henderson
Quoted 5 lines · Andrew Jones
CPU type | accel | sve-max-vq | sve-vls-map
-------------------------------------------1) max | tcg | $MAX_VQ | $VLS_MAP 2) max | kvm | $MAX_VQ | $VLS_MAP 3) host | kvm | N/A | $VLS_MAPThis doesn't seem right. Why is -cpu host not whatever the host supports? It certainly has been so far. I really don't see how -cpu max makes any sense for kvm.
The point of '-cpu max' is that it works and gives you the best thing QEMU can support regardless of what accelerator is in use. This means that you don't need to do tedious workarounds like "if KVM then -cpu host else -cpu somethingelse".
thanks
-- PMM