spapr: set vsmt to MAX(8, smp_threads)
PATCH v3 1/1 spapr: set vsmt to MAX(8, smp_threads)
We ignore silently the value of smp_threads when we set the default VSMT value, and if smp_threads is greater than VSMT kernel is going into trouble later.
("spapr: Adjust default VSMT value for better migration compatibility")
---
Notes:
v3: use MAX(8, smp_threads) and let KVM to return an error
if nb_threads is too big
update subject to reflect the change
v2: display a specific error message when the default VSMT is used
fix subject
hw/ppc/spapr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
hw/ppc/spapr.c +1 -1
@@ -2310,7 +2310,7 @@ static void spapr_set_vsmt_mode(sPAPRMachineState *spapr, Error **errp)* the value that we'd get with KVM on POWER8, the * overwhelmingly common case in production systems. */ - spapr->vsmt =8 ; + spapr->vsmt =MAX (8 , smp_threads); }/* KVM: If necessary, set the SMT mode: */
Signature · 1 line
2.14.3
Quoted 4 lines · On Fri, 9 Feb 2018 09:18:58 +0100 Laurent Vivier <[email protected]> wrote:
We ignore silently the value of
smp_threadswhen we set the default VSMT value, and ifsmp_threadsis greater than VSMT kernel is going into trouble later.
Hi Laurent,
I've looked a bit more and I'm not sure what kernel troubles you're referring to, but several places in QEMU where we use kvm_ppc_smt() later on do assume that smp_threads > kvm_ppc_smt(). Basically, everywhere we compute a vCPU id:
In spapr_init_cpus() when creating DRC connectors:
int core_id = i * smp_threads;
if (mc->has_hotpluggable_cpus) {
spapr_dr_connector_new(OBJECT(spapr), TYPE_SPAPR_DRC_CPU,
(core_id / smp_threads) * smt);
}or in spapr_cpu_core_realize() when creating vCPUs:
cpu->vcpu_id = (cc->core_id * spapr->vsmt / smp_threads) + i;
It is visible by adding some printfs in the current code base. This is what happens when passing -smp cores=2,threads=16 without your patch:
DRC connector to vcpu_id 0
CPU vcpu_id 0
CPU vcpu_id 1
CPU vcpu_id 2
CPU vcpu_id 3
CPU vcpu_id 4
CPU vcpu_id 5
CPU vcpu_id 6
CPU vcpu_id 7
CPU vcpu_id 8
CPU vcpu_id 9
CPU vcpu_id 10
CPU vcpu_id 11
CPU vcpu_id 12
CPU vcpu_id 13
CPU vcpu_id 14
CPU vcpu_id 15
DRC connector to vcpu_id 8
^^^
should be 16
CPU vcpu_id 8
^^^
should start numbering at 16
CPU vcpu_id 9
CPU vcpu_id 10
CPU vcpu_id 11
CPU vcpu_id 12
CPU vcpu_id 13
CPU vcpu_id 14
CPU vcpu_id 15
CPU vcpu_id 16
CPU vcpu_id 17
CPU vcpu_id 18
CPU vcpu_id 19
CPU vcpu_id 20
CPU vcpu_id 21
CPU vcpu_id 22
CPU vcpu_id 23
qemu-system-ppc64: kvm_init_vcpu failed: File exists
^^^^
CPU 8 already created by the first core
I'm not feeling comfortable with the rest of the code silently depending on the fact that spapr_set_vsmt_mode() terminates QEMU if it cannot enforce smp_threads <= kvm_ppc_smt().
Anyway, with your patch, the same command line as above gives:
qemu-system-ppc64: Failed to set KVM's VSMT mode to 16 (errno -22) On PPC, a VM with 16 threads/core on a host with 8 threads/core requires
the use of VSMT mode 16.
This KVM seems to be too old to support VSMT.
This hammer is big enough to fix the vCPU ids miscalculations, so:
Quoted 30 lines
Fixes 8904e5a750("spapr: Adjust default VSMT value for better migration compatibility")
Signed-off-by Laurent Vivier <[email protected]>---
Notes:
v3: useMAX(8, smp_threads)and let KVM to return an error
ifnb_threadsis too big
update subject to reflect the changev2: display a specific error message when the default VSMT is used
fix subjecthw/ppc/spapr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)hw/ppc/spapr.c +1
-1@@ -2310,7 +2310,7 @@ static void spapr_set_vsmt_mode(sPAPRMachineState *spapr, Error **errp)* the value that we'd get with KVM on POWER8, the * overwhelmingly common case in production systems. */ - spapr->vsmt =8 ; + spapr->vsmt =MAX (8 , smp_threads); }/* KVM: If necessary, set the SMT mode: */
Quoted 72 lines · Greg Kurz
Quoted 4 lines · On Fri, 9 Feb 2018 09:18:58 +0100 Laurent Vivier <[email protected]> wrote:
We ignore silently the value of
smp_threadswhen we set the default VSMT value, and ifsmp_threadsis greater than VSMT kernel is going into trouble later.Hi Laurent,
I've looked a bit more and I'm not sure what kernel troubles you're referring to, but several places in QEMU where we use
kvm_ppc_smt()later on do assume thatsmp_threads>kvm_ppc_smt(). Basically, everywhere we compute a vCPU id:In
spapr_init_cpus()when creating DRC connectors:int core_id = i * smp_threads; if (mc->has_hotpluggable_cpus) { spapr_dr_connector_new(OBJECT(spapr), TYPE_SPAPR_DRC_CPU, (core_id / smp_threads) * smt); }or in
spapr_cpu_core_realize()when creating vCPUs:cpu->vcpu_id = (cc->core_id * spapr->vsmt / smp_threads) + i;It is visible by adding some printfs in the current code base. This is what happens when passing -smp cores=2,threads=16 without your patch:
DRC connector to
vcpu_id0
CPUvcpu_id0
CPUvcpu_id1
CPUvcpu_id2
CPUvcpu_id3
CPUvcpu_id4
CPUvcpu_id5
CPUvcpu_id6
CPUvcpu_id7
CPUvcpu_id8
CPUvcpu_id9
CPUvcpu_id10
CPUvcpu_id11
CPUvcpu_id12
CPUvcpu_id13
CPUvcpu_id14
CPUvcpu_id15
DRC connector tovcpu_id8
^^^
should be 16
CPUvcpu_id8^^^should start numbering at 16
CPUvcpu_id9
CPUvcpu_id10
CPUvcpu_id11
CPUvcpu_id12
CPUvcpu_id13
CPUvcpu_id14
CPUvcpu_id15
CPUvcpu_id16
CPUvcpu_id17
CPUvcpu_id18
CPUvcpu_id19
CPUvcpu_id20
CPUvcpu_id21
CPUvcpu_id22
CPUvcpu_id23
qemu-system-ppc64:kvm_init_vcpufailed: File exists
^^^^
CPU 8 already created by the first core
Quoted 3 lines
I'm not feeling comfortable with the rest of the code silently depending on the fact that
spapr_set_vsmt_mode()terminates QEMU if it cannot enforcesmp_threads<=kvm_ppc_smt().
I'm not quite sure what you're suggesting as an alternative, though.
Quoted 43 lines
Anyway, with your patch, the same command line as above gives:
qemu-system-ppc64: Failed to set KVM's VSMT mode to 16 (errno -22) On PPC, a VM with 16 threads/core on a host with 8 threads/core requires
the use of VSMT mode 16.
This KVM seems to be too old to support VSMT.This hammer is big enough to fix the vCPU ids miscalculations, so:
Reviewed-by Greg Kurz <[email protected]>Quoted 30 lines
Fixes 8904e5a750("spapr: Adjust default VSMT value for better migration compatibility")
Signed-off-by Laurent Vivier <[email protected]>---
Notes:
v3: useMAX(8, smp_threads)and let KVM to return an error
ifnb_threadsis too big
update subject to reflect the changev2: display a specific error message when the default VSMT is used
fix subjecthw/ppc/spapr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)hw/ppc/spapr.c +1
-1@@ -2310,7 +2310,7 @@ static void spapr_set_vsmt_mode(sPAPRMachineState *spapr, Error **errp)* the value that we'd get with KVM on POWER8, the * overwhelmingly common case in production systems. */ - spapr->vsmt =8 ; + spapr->vsmt =MAX (8 , smp_threads); }/* KVM: If necessary, set the SMT mode: */
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Quoted 73 lines · On Sat, 10 Feb 2018 20:23:07 +1100 David Gibson <[email protected]>…
Quoted 72 lines · Greg Kurz
Quoted 4 lines · On Fri, 9 Feb 2018 09:18:58 +0100 Laurent Vivier <[email protected]> wrote:
We ignore silently the value of
smp_threadswhen we set the default VSMT value, and ifsmp_threadsis greater than VSMT kernel is going into trouble later.Hi Laurent,
I've looked a bit more and I'm not sure what kernel troubles you're referring to, but several places in QEMU where we use
kvm_ppc_smt()later on do assume thatsmp_threads>kvm_ppc_smt(). Basically, everywhere we compute a vCPU id:In
spapr_init_cpus()when creating DRC connectors:int core_id = i * smp_threads; if (mc->has_hotpluggable_cpus) { spapr_dr_connector_new(OBJECT(spapr), TYPE_SPAPR_DRC_CPU, (core_id / smp_threads) * smt); }or in
spapr_cpu_core_realize()when creating vCPUs:cpu->vcpu_id = (cc->core_id * spapr->vsmt / smp_threads) + i;It is visible by adding some printfs in the current code base. This is what happens when passing -smp cores=2,threads=16 without your patch:
DRC connector to
vcpu_id0
CPUvcpu_id0
CPUvcpu_id1
CPUvcpu_id2
CPUvcpu_id3
CPUvcpu_id4
CPUvcpu_id5
CPUvcpu_id6
CPUvcpu_id7
CPUvcpu_id8
CPUvcpu_id9
CPUvcpu_id10
CPUvcpu_id11
CPUvcpu_id12
CPUvcpu_id13
CPUvcpu_id14
CPUvcpu_id15
DRC connector tovcpu_id8
^^^
should be 16
CPUvcpu_id8^^^should start numbering at 16
CPUvcpu_id9
CPUvcpu_id10
CPUvcpu_id11
CPUvcpu_id12
CPUvcpu_id13
CPUvcpu_id14
CPUvcpu_id15
CPUvcpu_id16
CPUvcpu_id17
CPUvcpu_id18
CPUvcpu_id19
CPUvcpu_id20
CPUvcpu_id21
CPUvcpu_id22
CPUvcpu_id23
qemu-system-ppc64:kvm_init_vcpufailed: File exists
^^^^
CPU 8 already created by the first core
Quoted 6 lines
Quoted 3 lines
I'm not feeling comfortable with the rest of the code silently depending on the fact that
spapr_set_vsmt_mode()terminates QEMU if it cannot enforcesmp_threads<=kvm_ppc_smt().I'm not quite sure what you're suggesting as an alternative, though.
I haven't suggested anything yet :)
But I was thinking of:
- having a single function to compute the vcpu_id, instead of open-coding the formula in several places like the current code does,
- this function should ensure all pre-requisites to compute the vcpu_id are met (including trying to set VSMT) or return an error
Quoted 44 lines
Quoted 43 lines
Anyway, with your patch, the same command line as above gives:
qemu-system-ppc64: Failed to set KVM's VSMT mode to 16 (errno -22) On PPC, a VM with 16 threads/core on a host with 8 threads/core requires
the use of VSMT mode 16.
This KVM seems to be too old to support VSMT.This hammer is big enough to fix the vCPU ids miscalculations, so:
Reviewed-by Greg Kurz <[email protected]>Quoted 30 lines
Fixes 8904e5a750("spapr: Adjust default VSMT value for better migration compatibility")
Signed-off-by Laurent Vivier <[email protected]>---
Notes:
v3: useMAX(8, smp_threads)and let KVM to return an error
ifnb_threadsis too big
update subject to reflect the changev2: display a specific error message when the default VSMT is used
fix subjecthw/ppc/spapr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)hw/ppc/spapr.c +1
-1@@ -2310,7 +2310,7 @@ static void spapr_set_vsmt_mode(sPAPRMachineState *spapr, Error **errp)* the value that we'd get with KVM on POWER8, the * overwhelmingly common case in production systems. */ - spapr->vsmt =8 ; + spapr->vsmt =MAX (8 , smp_threads); }/* KVM: If necessary, set the SMT mode: */
Quoted 80 lines · Greg Kurz
Quoted 77 lines · On Sat, 10 Feb 2018 20:23:07 +1100 David Gibson <[email protected]>…
Quoted 75 lines · Greg Kurz
Quoted 4 lines · On Fri, 9 Feb 2018 09:18:58 +0100 Laurent Vivier <[email protected]> wrote:
We ignore silently the value of
smp_threadswhen we set the default VSMT value, and ifsmp_threadsis greater than VSMT kernel is going into trouble later.Hi Laurent,
I've looked a bit more and I'm not sure what kernel troubles you're referring to, but several places in QEMU where we use
kvm_ppc_smt()later on do assume thatsmp_threads>kvm_ppc_smt(). Basically, everywhere we compute a vCPU id:In
spapr_init_cpus()when creating DRC connectors:int core_id = i * smp_threads; if (mc->has_hotpluggable_cpus) { spapr_dr_connector_new(OBJECT(spapr), TYPE_SPAPR_DRC_CPU, (core_id / smp_threads) * smt); }or in
spapr_cpu_core_realize()when creating vCPUs:cpu->vcpu_id = (cc->core_id * spapr->vsmt / smp_threads) + i;It is visible by adding some printfs in the current code base. This is what happens when passing -smp cores=2,threads=16 without your patch:
DRC connector to
vcpu_id0
CPUvcpu_id0
CPUvcpu_id1
CPUvcpu_id2
CPUvcpu_id3
CPUvcpu_id4
CPUvcpu_id5
CPUvcpu_id6
CPUvcpu_id7
CPUvcpu_id8
CPUvcpu_id9
CPUvcpu_id10
CPUvcpu_id11
CPUvcpu_id12
CPUvcpu_id13
CPUvcpu_id14
CPUvcpu_id15
DRC connector tovcpu_id8
^^^
should be 16
CPUvcpu_id8^^^should start numbering at 16
CPUvcpu_id9
CPUvcpu_id10
CPUvcpu_id11
CPUvcpu_id12
CPUvcpu_id13
CPUvcpu_id14
CPUvcpu_id15
CPUvcpu_id16
CPUvcpu_id17
CPUvcpu_id18
CPUvcpu_id19
CPUvcpu_id20
CPUvcpu_id21
CPUvcpu_id22
CPUvcpu_id23
qemu-system-ppc64:kvm_init_vcpufailed: File exists
^^^^
CPU 8 already created by the first coreI'm not feeling comfortable with the rest of the code silently depending on the fact that
spapr_set_vsmt_mode()terminates QEMU if it cannot enforcesmp_threads<=kvm_ppc_smt().
Quoted 8 lines
Quoted 2 lines
I'm not quite sure what you're suggesting as an alternative, though.
I haven't suggested anything yet :)
But I was thinking of:
- having a single function to compute thevcpu_id, instead of open-coding the formula in several places like the current code does,
That seems like a good idea.
Quoted 2 lines
- this function should ensure all pre-requisites to compute the
vcpu_idare met (including trying to set VSMT) or return an error
I'm much more dubious about this. Checks that can be performed passively I'm ok with, but I think something that looks like a simple calculation helper shouldn't be having side-effects like poking at KVM state.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
David Gibson
AppliedQuoted 8 lines · Laurent Vivier
We ignore silently the value of
smp_threadswhen we set the default VSMT value, and ifsmp_threadsis greater than VSMT kernel is going into trouble later.Fixes 8904e5a750("spapr: Adjust default VSMT value for better migration compatibility")
Signed-off-by Laurent Vivier <[email protected]>
Applied, thanks.
Quoted 26 lines
---
Notes:
v3: useMAX(8, smp_threads)and let KVM to return an error
ifnb_threadsis too big
update subject to reflect the changev2: display a specific error message when the default VSMT is used
fix subjecthw/ppc/spapr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)hw/ppc/spapr.c +1
-1@@ -2310,7 +2310,7 @@ static void spapr_set_vsmt_mode(sPAPRMachineState *spapr, Error **errp)* the value that we'd get with KVM on POWER8, the * overwhelmingly common case in production systems. */ - spapr->vsmt =8 ; + spapr->vsmt =MAX (8 , smp_threads); }/* KVM: If necessary, set the SMT mode: */
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson