ARM virt: Introduce CPU clusters topology support
v5 cover ARM virt: Introduce CPU clusters topology support
Hi,
This series introduces the new CPU clusters topology parameter and enable the support for it on ARM virt machines.
Background and descriptions:
The new Cluster-Aware Scheduling support has landed in Linux 5.16, which has been proved to benefit the scheduling performance (e.g. load balance and wake_affine strategy) for both x86_64 and AArch64. We can see the PR [1] or the actual patch series [2] for reference.
So since Linux 5.16 we have four-level arch-neutral CPU topology definition like below and a new scheduler level for clusters.
struct cpu_topology {
int thread_id;
int core_id;
int cluster_id;
int package_id;
int llc_id;
cpumask_t thread_sibling;
cpumask_t core_sibling;
cpumask_t cluster_sibling;
cpumask_t llc_sibling;
}A cluster generally means a group of CPU cores which share L2 cache or other mid-level resources, and it is the shared resources that is used to improve scheduler's behavior. From the point of view of the size range, it's between CPU die and CPU core. For example, on some ARM64 Kunpeng servers, we have 6 clusters in each NUMA node, and 4 CPU cores in each cluster. The 4 CPU cores share a separate L2 cache and a L3 cache tag, which brings cache affinity advantage.
[1] https://lore.kernel.org/lkml/163572864855.3357115.17938524897008353101.tglx@xen13/
[2] https://lkml.org/lkml/2021/9/24/178
In virtualization, on the Hosts which have pClusters, if we can design a vCPU topology with cluster level for guest kernel and have a dedicated vCPU pinning. A Cluster-Aware Guest kernel can also make use of the cache affinity of CPU clusters to gain similar scheduling performance. So this series introduce clusters support in the vCPU topology on ARM virt machines.
The patches are arranged mainly in two parts:
The first part (patch 1-7):
- Implement infrastructure for CPU cluster level topology support, including the SMP documentation, configuration and parsing, adding testcases for clusters.
The second part (part 8-14):
- Enable CPU cluster support on ARM virt machines, so that users can specify a 4-level CPU hierarchy sockets/clusters/cores/threads. And the 4-level topology will be described to guest kernel through ACPI PPTT and DT cpu-map.
Changelog:v3->v4:
- Significant change from v3 to v4, since the whole series is reworked based on latest QEMU SMP frame.
- v3: https://patchew.org/QEMU/[email protected]/
v4->v5:
- newly added patches 4-7
- rebased on Philippe series: "tests/unit: Rework test-smp-parse tests"
https://patchew.org/QEMU/[email protected]/
- v4: https://patchew.org/QEMU/[email protected]/
Yanan Wang (14):
qemu-options: Improve readability of SMP related Docs hw/core/machine: Introduce CPU cluster topology support hw/core/machine: Wrap target specific parameters together tests/unit/test-smp-parse: Add testcases for CPU clusters tests/unit/test-smp-parse: No need to explicitly zero MachineClass
members
tests/unit/test-smp-parse: Keep default MIN/MAX CPUs in
machine_base_class_init
MAINTAINERS: Self-recommended as reviewer of "Machine core" hw/arm/virt: Support clusters on ARM virt machines hw/arm/virt: Support cluster level in DT cpu-map hw/acpi/aml-build: Improve scalability of PPTT generation hw/arm/virt-acpi-build: Make an ARM specific PPTT generator tests/acpi/bios-tables-test: Allow changes to virt/PPTT file hw/arm/virt-acpi-build: Support cluster level in PPTT generation tests/acpi/bios-table-test: Update expected virt/PPTT file
MAINTAINERS | 1 + hw/acpi/aml-build.c | 66 +---------------- hw/arm/virt-acpi-build.c | 92 +++++++++++++++++++++++- hw/arm/virt.c | 16 +++-- hw/core/machine-smp.c | 29 ++++++-- hw/core/machine.c | 3 + include/hw/acpi/aml-build.h | 5 +- include/hw/boards.h | 6 +- qapi/machine.json | 5 +- qemu-options.hx | 91 ++++++++++++++++++----- softmmu/vl.c | 3 + tests/data/acpi/virt/PPTT | Bin 76 -> 96 bytes tests/unit/test-smp-parse.c | 140 ++++++++++++++++++++++++++++++------ 13 files changed, 332 insertions(+), 125 deletions(-)
--
2.27.0
PATCH v5 1/14 qemu-options: Improve readability of SMP related Docs
We have a description in qemu-options.hx for each CPU topology parameter to explain what it exactly means, and also an extra declaration for the target-specific one, e.g. "for PC only" when describing "dies", and "for PC, it's on one die" when describing "cores".
Now we are going to introduce one more non-generic parameter "clusters", it will make the Doc less readable and if we still continue to use the legacy way to describe it.
So let's at first make two tweaks of the Docs to improve the readability and also scalability:
1) In the -help text: Delete the extra specific declaration and describe each topology parameter level by level. Then add a note to declare that different machines may support different subsets and the actual meaning of the supported parameters will vary accordingly.
2) In the rST text: List all the sub-hierarchies currently supported in QEMU, and correspondingly give an example of -smp configuration for each of them.
--- qemu-options.hx | 76 ++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 59 insertions(+), 17 deletions(-)
qemu-options.hx +59 -17
@@ -207,14 +207,26 @@ ERST DEF("smp", HAS_ARG, QEMU_OPTION_smp, "-smp [[cpus=]n][,maxcpus=maxcpus][,sockets=sockets][,dies=dies][,cores=cores][,threads=threads]\n" - " set the number of CPUs to 'n' [default=1]\n" + " set the number of initial CPUs to 'n' [default=1]\n" " maxcpus= maximum number of total CPUs, including\n" " offline CPUs for hotplug, etc\n" - " sockets= number of discrete sockets in the system\n" - " dies= number of CPU dies on one socket (for PC only)\n" - " cores= number of CPU cores on one socket (for PC, it's on one die)\n" - " threads= number of threads on one CPU core\n", - QEMU_ARCH_ALL) + " sockets= number of sockets on the machine board\n" + " dies= number of dies in one socket\n" + " cores= number of cores in one die\n" + " threads= number of threads in one core\n" + "Note: Different machines may have different subsets of the CPU topology\n" + " parameters supported, so the actual meaning of the supported parameters\n" + " will vary accordingly. For example, for a machine type that supports a\n" + " three-level CPU hierarchy of sockets/cores/threads, the parameters will\n" + " sequentially mean as below:\n" + " sockets means the number of sockets on the machine board\n" + " cores means the number of cores in one socket\n" + " threads means the number of threads in one core\n" + " For a particular machine type board, an expected CPU topology hierarchy\n" + " can be defined through the supported sub-option. Unsupported parameters\n" + " can also be provided in addition to the sub-option, but their values\n" + " must be set as 1 in the purpose of correct parsing.\n", + QEMU_ARCH_ALL) SRST ``-smp [[cpus=]n][,maxcpus=maxcpus][,sockets=sockets][,dies=dies][,cores=cores][,threads=threads]`` Simulate a SMP system with '\ ``n``\ ' CPUs initially present on
@@ -225,27 +237,57 @@ SRST initial CPU count will match the maximum number. When only one of them is given then the omitted one will be set to its counterpart's value. Both parameters may be specified, but the maximum number of CPUs must - be equal to or greater than the initial CPU count. Both parameters are - subject to an upper limit that is determined by the specific machine - type chosen. - - To control reporting of CPU topology information, the number of sockets, - dies per socket, cores per die, and threads per core can be specified. - The sum `` sockets * cores * dies * threads `` must be equal to the - maximum CPU count. CPU targets may only support a subset of the topology - parameters. Where a CPU target does not support use of a particular - topology parameter, its value should be assumed to be 1 for the purpose - of computing the CPU maximum count. + be equal to or greater than the initial CPU count. Product of the + CPU topology hierarchy must be equal to the maximum number of CPUs. + Both parameters are subject to an upper limit that is determined by + the specific machine type chosen. + + To control reporting of CPU topology information, values of the topology + parameters can be specified. Machines may only support a subset of the + parameters and different machines may have different subsets supported + which vary depending on capacity of the corresponding CPU targets. So + for a particular machine type board, an expected topology hierarchy can + be defined through the supported sub-option. Unsupported parameters can + also be provided in addition to the sub-option, but their values must be + set as 1 in the purpose of correct parsing. Either the initial CPU count, or at least one of the topology parameters must be specified. The specified parameters must be greater than zero, explicit configuration like "cpus=0" is not allowed. Values for any omitted parameters will be computed from those which are given. + + For example, the following sub-option defines a CPU topology hierarchy + (2 sockets totally on the machine, 2 cores per socket, 2 threads per + core) for a machine that only supports sockets/cores/threads. + Some members of the option can be omitted but their values will be + automatically computed: + + :: + + -smp 8,sockets=2,cores=2,threads=2,maxcpus=8 + + The following sub-option defines a CPU topology hierarchy (2 sockets + totally on the machine, 2 dies per socket, 2 cores per die, 2 threads + per core) for PC machines which support sockets/dies/cores/threads. + Some members of the option can be omitted but their values will be + automatically computed: + + :: + + -smp 16,sockets=2,dies=2,cores=2,threads=2,maxcpus=16 + Historically preference was given to the coarsest topology parameters when computing missing values (ie sockets preferred over cores, which were preferred over threads), however, this behaviour is considered liable to change. Prior to 6.2 the preference was sockets over cores over threads. Since 6.2 the preference is cores over sockets over threads. + + For example, the following option defines a machine board with 2 sockets + of 1 core before 6.2 and 1 socket of 2 cores after 6.2: + + :: + + -smp 2 ERST DEF("numa", HAS_ARG, QEMU_OPTION_numa,
Signature · 1 line
2.27.0
Re: [PATCH v5 01/14] qemu-options: Improve readability of SMP related Docs
Quoted 21 lines · Yanan Wang
We have a description in qemu-options.hx for each CPU topology parameter to explain what it exactly means, and also an extra declaration for the target-specific one, e.g. "for PC only" when describing "dies", and "for PC, it's on one die" when describing "cores".
Now we are going to introduce one more non-generic parameter "clusters", it will make the Doc less readable and if we still continue to use the legacy way to describe it.
So let's at first make two tweaks of the Docs to improve the readability and also scalability:
1) In the -help text: Delete the extra specific declaration and describe each topology parameter level by level. Then add a note to declare that different machines may support different subsets and the actual meaning of the supported parameters will vary accordingly.
2) In the rST text: List all the sub-hierarchies currently supported in QEMU, and correspondingly give an example of -smp configuration for each of them.
Quoted 4 lines
Signed-off-by Yanan Wang <[email protected]>--- qemu-options.hx | 76 ++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 59 insertions(+), 17 deletions(-)
PATCH v5 3/14 hw/core/machine: Wrap target specific parameters together
In reply to Yanan Wang viaWrap the CPU target specific parameters together into a single variable except generic sockets/cores/threads, to make related code lines shorter and more concise.
No functional change intended.
--- hw/core/machine-smp.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-)
hw/core/machine-smp.c +10 -7
@@ -79,6 +79,7 @@ void machine_parse_smp_config(MachineState *ms,unsigned cores = config->has_cores ? config->cores :0 ;unsigned threads = config->has_threads ? config->threads :0 ;unsigned maxcpus = config->has_maxcpus ? config->maxcpus :0 ; +unsigned others;/* * Specified CPU topology parameters must be greater than zero,
@@ -111,6 +112,8 @@ void machine_parse_smp_config(MachineState *ms, dies = dies >0 ? dies :1 ; clusters = clusters >0 ? clusters :1 ; + others = dies * clusters; +/* compute missing values based on the provided ones */ if (cpus ==0 && maxcpus ==0 ) { sockets = sockets >0 ? sockets :1 ;
@@ -124,30 +127,30 @@ void machine_parse_smp_config(MachineState *ms,if (sockets ==0 ) { cores = cores >0 ? cores :1 ; threads = threads >0 ? threads :1 ; - sockets = maxcpus / (dies * clusters * cores * threads); + sockets = maxcpus / (cores * threads * others); }else if (cores ==0 ) { threads = threads >0 ? threads :1 ; - cores = maxcpus / (sockets * dies * clusters * threads); + cores = maxcpus / (sockets * threads * others); } }else {/* prefer cores over sockets since 6.2 */ if (cores ==0 ) { sockets = sockets >0 ? sockets :1 ; threads = threads >0 ? threads :1 ; - cores = maxcpus / (sockets * dies * clusters * threads); + cores = maxcpus / (sockets * threads * others); }else if (sockets ==0 ) { threads = threads >0 ? threads :1 ; - sockets = maxcpus / (dies * clusters * cores * threads); + sockets = maxcpus / (cores * threads * others); } }/* try to calculate omitted threads at last */ if (threads ==0 ) { - threads = maxcpus / (sockets * dies * clusters * cores); + threads = maxcpus / (sockets * cores * others); } } - maxcpus = maxcpus >0 ? maxcpus : sockets * dies * clusters * cores * threads; + maxcpus = maxcpus >0 ? maxcpus : sockets * cores * threads * others; cpus = cpus >0 ? cpus : maxcpus; ms->smp.cpus = cpus;
@@ -159,7 +162,7 @@ void machine_parse_smp_config(MachineState *ms, ms->smp.max_cpus = maxcpus;/* sanity-check of the computed topology */ -if (sockets * dies * clusters * cores * threads != maxcpus) { +if (sockets * cores * threads * others != maxcpus) { g_autofree char *topo_msg = cpu_hierarchy_to_string(ms);error_setg (errp,"Invalid CPU topology: " "product of the hierarchy must match maxcpus: "
Signature · 1 line
2.27.0
Re: [PATCH v5 03/14] hw/core/machine: Wrap target specific parameters together
Quoted 24 lines · Yanan Wang
Wrap the CPU target specific parameters together into a single variable except generic sockets/cores/threads, to make related code lines shorter and more concise.
No functional change intended.
Signed-off-by Yanan Wang <[email protected]>--- hw/core/machine-smp.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-)hw/core/machine-smp.c +1
-0@@ -79,6 +79,7 @@ void machine_parse_smp_config(MachineState *ms,unsigned cores = config->has_cores ? config->cores :0 ;unsigned threads = config->has_threads ? config->threads :0 ;unsigned maxcpus = config->has_maxcpus ? config->maxcpus :0 ; +unsigned others;/* * Specified CPU topology parameters must be greater than zero,
hw/core/machine-smp.c void machine_parse_smp_config(MachineState *ms, · line 115
Quoted 4 lines
hw/core/machine-smp.c +1
-0@@ -111,6 +112,8 @@ void machine_parse_smp_config(MachineState *ms, dies = dies >0 ? dies :1 ; clusters = clusters >0 ? clusters :1 ; + others = dies * clusters;
This doesn't seem clearer to me...
Quoted 49 lines
hw/core/machine-smp.c +7
-7/* compute missing values based on the provided ones */ if (cpus ==0 && maxcpus ==0 ) { sockets = sockets >0 ? sockets :1 ;@@ -124,30 +127,30 @@ void machine_parse_smp_config(MachineState *ms,if (sockets ==0 ) { cores = cores >0 ? cores :1 ; threads = threads >0 ? threads :1 ; - sockets = maxcpus / (dies * clusters * cores * threads); + sockets = maxcpus / (cores * threads * others); }else if (cores ==0 ) { threads = threads >0 ? threads :1 ; - cores = maxcpus / (sockets * dies * clusters * threads); + cores = maxcpus / (sockets * threads * others); } }else {/* prefer cores over sockets since 6.2 */ if (cores ==0 ) { sockets = sockets >0 ? sockets :1 ; threads = threads >0 ? threads :1 ; - cores = maxcpus / (sockets * dies * clusters * threads); + cores = maxcpus / (sockets * threads * others); }else if (sockets ==0 ) { threads = threads >0 ? threads :1 ; - sockets = maxcpus / (dies * clusters * cores * threads); + sockets = maxcpus / (cores * threads * others); } }/* try to calculate omitted threads at last */ if (threads ==0 ) { - threads = maxcpus / (sockets * dies * clusters * cores); + threads = maxcpus / (sockets * cores * others); } } - maxcpus = maxcpus >0 ? maxcpus : sockets * dies * clusters * cores * threads; + maxcpus = maxcpus >0 ? maxcpus : sockets * cores * threads * others; cpus = cpus >0 ? cpus : maxcpus; ms->smp.cpus = cpus;@@ -159,7 +162,7 @@ void machine_parse_smp_config(MachineState *ms, ms->smp.max_cpus = maxcpus;/* sanity-check of the computed topology */ -if (sockets * dies * clusters * cores * threads != maxcpus) { +if (sockets * cores * threads * others != maxcpus) { g_autofree char *topo_msg = cpu_hierarchy_to_string(ms);error_setg (errp,"Invalid CPU topology: " "product of the hierarchy must match maxcpus: "
Re: [PATCH v5 03/14] hw/core/machine: Wrap target specific parameters together
Quoted 22 lines · Philippe Mathieu-Daudé
Quoted 21 lines · Yanan Wang
Wrap the CPU target specific parameters together into a single variable except generic sockets/cores/threads, to make related code lines shorter and more concise.
No functional change intended.
Signed-off-by Yanan Wang <[email protected]>--- hw/core/machine-smp.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-)hw/core/machine-smp.c +1
-0@@ -79,6 +79,7 @@ void machine_parse_smp_config(MachineState *ms,unsigned cores = config->has_cores ? config->cores :0 ;unsigned threads = config->has_threads ? config->threads :0 ;unsigned maxcpus = config->has_maxcpus ? config->maxcpus :0 ; +unsigned others;
Quoted 8 lines
hw/core/machine-smp.c void machine_parse_smp_config(MachineState *ms, · line 115
Quoted 7 lines
hw/core/machine-smp.c +1
-0/* * Specified CPU topology parameters must be greater than zero, @@ -111,6 +112,8 @@ void machine_parse_smp_config(MachineState *ms, dies = dies >0 ? dies :1 ; clusters = clusters >0 ? clusters :1 ; + others = dies * clusters;This doesn't seem clearer to me...
Ok. We can just drop this patch.
Thanks,
Yanan
Quoted 50 lines
Quoted 41 lines
hw/core/machine-smp.c +6
-6/* compute missing values based on the provided ones */ if (cpus ==0 && maxcpus ==0 ) { sockets = sockets >0 ? sockets :1 ;@@ -124,30 +127,30 @@ void machine_parse_smp_config(MachineState *ms,if (sockets ==0 ) { cores = cores >0 ? cores :1 ; threads = threads >0 ? threads :1 ; - sockets = maxcpus / (dies * clusters * cores * threads); + sockets = maxcpus / (cores * threads * others); }else if (cores ==0 ) { threads = threads >0 ? threads :1 ; - cores = maxcpus / (sockets * dies * clusters * threads); + cores = maxcpus / (sockets * threads * others); } }else {/* prefer cores over sockets since 6.2 */ if (cores ==0 ) { sockets = sockets >0 ? sockets :1 ; threads = threads >0 ? threads :1 ; - cores = maxcpus / (sockets * dies * clusters * threads); + cores = maxcpus / (sockets * threads * others); }else if (sockets ==0 ) { threads = threads >0 ? threads :1 ; - sockets = maxcpus / (dies * clusters * cores * threads); + sockets = maxcpus / (cores * threads * others); } }/* try to calculate omitted threads at last */ if (threads ==0 ) { - threads = maxcpus / (sockets * dies * clusters * cores); + threads = maxcpus / (sockets * cores * others); } } - maxcpus = maxcpus >0 ? maxcpus : sockets * dies * clusters * cores * threads; + maxcpus = maxcpus >0 ? maxcpus : sockets * cores * threads * others; cpus = cpus >0 ? cpus : maxcpus; ms->smp.cpus = cpus;hw/core/machine-smp.c void machine_parse_smp_config(MachineState *ms, · line 168
Quoted 8 lines
hw/core/machine-smp.c +1
-1@@ -159,7 +162,7 @@ void machine_parse_smp_config(MachineState *ms, ms->smp.max_cpus = maxcpus;/* sanity-check of the computed topology */ -if (sockets * dies * clusters * cores * threads != maxcpus) { +if (sockets * cores * threads * others != maxcpus) { g_autofree char *topo_msg = cpu_hierarchy_to_string(ms);error_setg (errp,"Invalid CPU topology: " "product of the hierarchy must match maxcpus: ".
PATCH v5 2/14 hw/core/machine: Introduce CPU cluster topology support
In reply to Yanan Wang viaThe new Cluster-Aware Scheduling support has landed in Linux 5.16, which has been proved to benefit the scheduling performance (e.g. load balance and wake_affine strategy) on both x86_64 and AArch64.
So now in Linux 5.16 we have four-level arch-neutral CPU topology definition like below and a new scheduler level for clusters.
struct cpu_topology {
int thread_id;
int core_id;
int cluster_id;
int package_id;
int llc_id;
cpumask_t thread_sibling;
cpumask_t core_sibling;
cpumask_t cluster_sibling;
cpumask_t llc_sibling;
}A cluster generally means a group of CPU cores which share L2 cache or other mid-level resources, and it is the shared resources that is used to improve scheduler's behavior. From the point of view of the size range, it's between CPU die and CPU core. For example, on some ARM64 Kunpeng servers, we have 6 clusters in each NUMA node, and 4 CPU cores in each cluster. The 4 CPU cores share a separate L2 cache and a L3 cache tag, which brings cache affinity advantage.
In virtualization, on the Hosts which have pClusters, if we can design a vCPU topology with cluster level for guest kernel and have a dedicated vCPU pinning. A Cluster-Aware Guest kernel can also make use of the cache affinity of CPU clusters to gain similar scheduling performance.
This patch adds infrastructure for CPU cluster level topology configuration and parsing, so that the user can specify cluster parameter if their machines support it.
--- hw/core/machine-smp.c | 26 +++++++++++++++++++------- hw/core/machine.c | 3 +++ include/hw/boards.h | 6 +++++- qapi/machine.json | 5 ++++- qemu-options.hx | 7 ++++--- softmmu/vl.c | 3 +++ 6 files changed, 38 insertions(+), 12 deletions(-)
hw/core/machine-smp.c +19 -7
@@ -37,6 +37,10 @@ static char *cpu_hierarchy_to_string(MachineState *ms)g_string_append_printf (s," * dies (%u)" , ms->smp.dies); } +if (mc->smp_props.clusters_supported) { +g_string_append_printf (s," * clusters (%u)" , ms->smp.clusters); + } +g_string_append_printf (s," * cores (%u)" , ms->smp.cores);g_string_append_printf (s," * threads (%u)" , ms->smp.threads);
@@ -71,6 +75,7 @@ void machine_parse_smp_config(MachineState *ms,unsigned cpus = config->has_cpus ? config->cpus :0 ;unsigned sockets = config->has_sockets ? config->sockets :0 ;unsigned dies = config->has_dies ? config->dies :0 ; +unsigned clusters = config->has_clusters ? config->clusters :0 ;unsigned cores = config->has_cores ? config->cores :0 ;unsigned threads = config->has_threads ? config->threads :0 ;unsigned maxcpus = config->has_maxcpus ? config->maxcpus :0 ;
@@ -82,6 +87,7 @@ void machine_parse_smp_config(MachineState *ms,if ((config->has_cpus && config->cpus ==0 ) || (config->has_sockets && config->sockets ==0 ) || (config->has_dies && config->dies ==0 ) || + (config->has_clusters && config->clusters ==0 ) || (config->has_cores && config->cores ==0 ) || (config->has_threads && config->threads ==0 ) || (config->has_maxcpus && config->maxcpus ==0 )) {
@@ -97,8 +103,13 @@ void machine_parse_smp_config(MachineState *ms,error_setg (errp,"dies not supported by this machine's CPU topology" );return ; } +if (!mc->smp_props.clusters_supported && clusters >1 ) { +error_setg (errp,"clusters not supported by this machine's CPU topology" ); +return ; + } dies = dies >0 ? dies :1 ; + clusters = clusters >0 ? clusters :1 ;/* compute missing values based on the provided ones */ if (cpus ==0 && maxcpus ==0 ) {
@@ -113,41 +124,42 @@ void machine_parse_smp_config(MachineState *ms,if (sockets ==0 ) { cores = cores >0 ? cores :1 ; threads = threads >0 ? threads :1 ; - sockets = maxcpus / (dies * cores * threads); + sockets = maxcpus / (dies * clusters * cores * threads); }else if (cores ==0 ) { threads = threads >0 ? threads :1 ; - cores = maxcpus / (sockets * dies * threads); + cores = maxcpus / (sockets * dies * clusters * threads); } }else {/* prefer cores over sockets since 6.2 */ if (cores ==0 ) { sockets = sockets >0 ? sockets :1 ; threads = threads >0 ? threads :1 ; - cores = maxcpus / (sockets * dies * threads); + cores = maxcpus / (sockets * dies * clusters * threads); }else if (sockets ==0 ) { threads = threads >0 ? threads :1 ; - sockets = maxcpus / (dies * cores * threads); + sockets = maxcpus / (dies * clusters * cores * threads); } }/* try to calculate omitted threads at last */ if (threads ==0 ) { - threads = maxcpus / (sockets * dies * cores); + threads = maxcpus / (sockets * dies * clusters * cores); } } - maxcpus = maxcpus >0 ? maxcpus : sockets * dies * cores * threads; + maxcpus = maxcpus >0 ? maxcpus : sockets * dies * clusters * cores * threads; cpus = cpus >0 ? cpus : maxcpus; ms->smp.cpus = cpus; ms->smp.sockets = sockets; ms->smp.dies = dies; + ms->smp.clusters = clusters; ms->smp.cores = cores; ms->smp.threads = threads; ms->smp.max_cpus = maxcpus;/* sanity-check of the computed topology */ -if (sockets * dies * cores * threads != maxcpus) { +if (sockets * dies * clusters * cores * threads != maxcpus) { g_autofree char *topo_msg = cpu_hierarchy_to_string(ms);error_setg (errp,"Invalid CPU topology: " "product of the hierarchy must match maxcpus: "
hw/core/machine.c +3 -0
@@ -742,10 +742,12 @@ static void machine_get_smp(Object *obj, Visitor *v, const char *name, .has_cpus = true, .cpus = ms->smp.cpus, .has_sockets = true, .sockets = ms->smp.sockets, .has_dies = true, .dies = ms->smp.dies, + .has_clusters = true, .clusters = ms->smp.clusters, .has_cores = true, .cores = ms->smp.cores, .has_threads = true, .threads = ms->smp.threads, .has_maxcpus = true, .maxcpus = ms->smp.max_cpus, }; +if (!visit_type_SMPConfiguration (v, name, &config, &error_abort)) {return ; }
@@ -932,6 +934,7 @@ static void machine_initfn(Object *obj) ms->smp.max_cpus = mc->default_cpus; ms->smp.sockets =1 ; ms->smp.dies =1 ; + ms->smp.clusters =1 ; ms->smp.cores =1 ; ms->smp.threads =1 ; }
include/hw/boards.h +5 -1
@@ -129,10 +129,12 @@ typedef struct {* SMPCompatProps: * @prefer_sockets - whether sockets are preferred over cores in smp parsing * @dies_supported - whether dies are supported by the machine +* @clusters_supported - whether clusters are supported by the machine */ typedef struct {bool prefer_sockets;bool dies_supported; +bool clusters_supported; }SMPCompatProps ;/**
@@ -299,7 +301,8 @@ typedef struct DeviceMemoryState {* @cpus: the number of present logical processors on the machine * @sockets: the number of sockets on the machine * @dies: the number of dies in one socket -* @cores: the number of cores in one die +* @clusters: the number of clusters in one die +* @cores: the number of cores in one cluster * @threads: the number of threads in one core * @max_cpus: the maximum number of logical processors on the machine */
@@ -307,6 +310,7 @@ typedef struct CpuTopology {unsigned int cpus;unsigned int sockets;unsigned int dies; +unsigned int clusters;unsigned int cores;unsigned int threads;unsigned int max_cpus;
qapi/machine.json +4 -1
@@ -1404,7 +1404,9 @@ # # @dies: number of dies per socket in the CPU topology # -# @cores: number of cores per die in the CPU topology +# @clusters: number of clusters per die in the CPU topology +# +# @cores: number of cores per cluster in the CPU topology # # @threads: number of threads per core in the CPU topology #
@@ -1416,6 +1418,7 @@ '*cpus': 'int', '*sockets': 'int', '*dies': 'int', + '*clusters': 'int', '*cores': 'int', '*threads': 'int', '*maxcpus': 'int' } }
qemu-options.hx +4 -3
@@ -206,13 +206,14 @@ SRST ERST DEF("smp", HAS_ARG, QEMU_OPTION_smp, - "-smp [[cpus=]n][,maxcpus=maxcpus][,sockets=sockets][,dies=dies][,cores=cores][,threads=threads]\n" + "-smp [[cpus=]n][,maxcpus=maxcpus][,sockets=sockets][,dies=dies][,clusters=clusters][,cores=cores][,threads=threads]\n" " set the number of initial CPUs to 'n' [default=1]\n" " maxcpus= maximum number of total CPUs, including\n" " offline CPUs for hotplug, etc\n" " sockets= number of sockets on the machine board\n" " dies= number of dies in one socket\n" - " cores= number of cores in one die\n" + " clusters= number of clusters in one die\n" + " cores= number of cores in one cluster\n" " threads= number of threads in one core\n" "Note: Different machines may have different subsets of the CPU topology\n" " parameters supported, so the actual meaning of the supported parameters\n"
@@ -228,7 +229,7 @@ DEF("smp", HAS_ARG, QEMU_OPTION_smp, " must be set as 1 in the purpose of correct parsing.\n", QEMU_ARCH_ALL) SRST -``-smp [[cpus=]n][,maxcpus=maxcpus][,sockets=sockets][,dies=dies][,cores=cores][,threads=threads]`` +``-smp [[cpus=]n][,maxcpus=maxcpus][,sockets=sockets][,dies=dies][,clusters=clusters][,cores=cores][,threads=threads]`` Simulate a SMP system with '\ ``n``\ ' CPUs initially present on the machine type board. On boards supporting CPU hotplug, the optional '\ ``maxcpus``\ ' parameter can be set to enable further CPUs to be
softmmu/vl.c +3 -0
@@ -726,6 +726,9 @@ static QemuOptsList qemu_smp_opts = { }, { .name = "dies", .type = QEMU_OPT_NUMBER, + }, { + .name = "clusters", + .type = QEMU_OPT_NUMBER, }, { .name = "cores", .type = QEMU_OPT_NUMBER,
Signature · 1 line
2.27.0
Re: [PATCH v5 02/14] hw/core/machine: Introduce CPU cluster topology support
Hi,
Quoted 26 lines · Yanan Wang
The new Cluster-Aware Scheduling support has landed in Linux 5.16, which has been proved to benefit the scheduling performance (e.g. load balance and
wake_affinestrategy) on bothx86_64and AArch64.So now in Linux 5.16 we have four-level arch-neutral CPU topology definition like below and a new scheduler level for clusters.
struct cpu_topology { int thread_id; int core_id; int cluster_id; int package_id; int llc_id; cpumask_t thread_sibling; cpumask_t core_sibling; cpumask_t cluster_sibling; cpumask_t llc_sibling; }A cluster generally means a group of CPU cores which share L2 cache or other mid-level resources, and it is the shared resources that is used to improve scheduler's behavior. From the point of view of the size range, it's between CPU die and CPU core. For example, on some ARM64 Kunpeng servers, we have 6 clusters in each NUMA node, and 4 CPU cores in each cluster. The 4 CPU cores share a separate L2 cache and a L3 cache tag, which brings cache affinity advantage.
Quoted 1 line
In virtualization, on the Hosts which have pClusters, if we can
Maybe [*] -> reference to pClusters?
Quoted 24 lines
design a vCPU topology with cluster level for guest kernel and have a dedicated vCPU pinning. A Cluster-Aware Guest kernel can also make use of the cache affinity of CPU clusters to gain similar scheduling performance.
This patch adds infrastructure for CPU cluster level topology configuration and parsing, so that the user can specify cluster parameter if their machines support it.
Signed-off-by Yanan Wang <[email protected]>--- hw/core/machine-smp.c | 26 +++++++++++++++++++------- hw/core/machine.c | 3 +++ include/hw/boards.h | 6 +++++- qapi/machine.json | 5 ++++- qemu-options.hx | 7 ++++--- softmmu/vl.c | 3 +++ 6 files changed, 38 insertions(+), 12 deletions(-)qapi/machine.json +0
-0
qapi/machine.json line 1407
Quoted 5 lines
qapi/machine.json +1
-1@@ -1404,7 +1404,9 @@ # # @dies: number of dies per socket in the CPU topology # -# @cores: number of cores per die in the CPU topology +# @clusters: number of clusters per die in the CPU topology
Missing:
# (since 7.0)
qapi/machine.json line 1424
Quoted 13 lines
qapi/machine.json +3
-0+# +# @cores: number of cores per cluster in the CPU topology # # @threads: number of threads per core in the CPU topology #@@ -1416,6 +1418,7 @@ '*cpus': 'int', '*sockets': 'int', '*dies': 'int', + '*clusters': 'int', '*cores': 'int', '*threads': 'int', '*maxcpus': 'int' } }
If you want I can update the doc when applying.
Thanks,
Phil.
Re: [PATCH v5 02/14] hw/core/machine: Introduce CPU cluster topology support
Hi Philippe,
Thanks for your review.
Quoted 29 lines · Philippe Mathieu-Daudé
Hi,
Quoted 26 lines · Yanan Wang
The new Cluster-Aware Scheduling support has landed in Linux 5.16, which has been proved to benefit the scheduling performance (e.g. load balance and
wake_affinestrategy) on bothx86_64and AArch64.So now in Linux 5.16 we have four-level arch-neutral CPU topology definition like below and a new scheduler level for clusters.
struct cpu_topology { int thread_id; int core_id; int cluster_id; int package_id; int llc_id; cpumask_t thread_sibling; cpumask_t core_sibling; cpumask_t cluster_sibling; cpumask_t llc_sibling; }A cluster generally means a group of CPU cores which share L2 cache or other mid-level resources, and it is the shared resources that is used to improve scheduler's behavior. From the point of view of the size range, it's between CPU die and CPU core. For example, on some ARM64 Kunpeng servers, we have 6 clusters in each NUMA node, and 4 CPU cores in each cluster. The 4 CPU cores share a separate L2 cache and a L3 cache tag, which brings cache affinity advantage.
Quoted 2 lines
Quoted 1 line
In virtualization, on the Hosts which have pClusters, if we can
Maybe [*] -> reference to pClusters?
Hm, I'm not sure what kind of reference is appropriate here. The third paragraph in the commit message has explained what a cluster generally means. We can also read the description of clusters in Linux kernel Kconfig files: [1] and [2].
[1]arm64: https://github.com/torvalds/linux/blob/master/arch/arm64/Kconfig
config SCHED_CLUSTER
bool "Cluster scheduler support"
help
Cluster scheduler support improves the CPU scheduler's decision making when dealing with machines that have clusters of CPUs. Cluster usually means a couple of CPUs which are placed closely by sharing mid-level caches, last-level cache tags or internal busses.
[2]x86: https://github.com/torvalds/linux/blob/master/arch/x86/Kconfig
config SCHED_CLUSTER
bool "Cluster scheduler support"
depends on SMP
default y
help
Cluster scheduler support improves the CPU scheduler's decision making when dealing with machines that have clusters of CPUs. Cluster usually means a couple of CPUs which are placed closely by sharing mid-level caches, last-level cache tags or internal busses.
Quoted 30 lines
Quoted 23 lines
design a vCPU topology with cluster level for guest kernel and have a dedicated vCPU pinning. A Cluster-Aware Guest kernel can also make use of the cache affinity of CPU clusters to gain similar scheduling performance.
This patch adds infrastructure for CPU cluster level topology configuration and parsing, so that the user can specify cluster parameter if their machines support it.
Signed-off-by Yanan Wang <[email protected]>--- hw/core/machine-smp.c | 26 +++++++++++++++++++------- hw/core/machine.c | 3 +++ include/hw/boards.h | 6 +++++- qapi/machine.json | 5 ++++- qemu-options.hx | 7 ++++--- softmmu/vl.c | 3 +++ 6 files changed, 38 insertions(+), 12 deletions(-)qapi/machine.json +0
-0qapi/machine.json line 1407
Quoted 5 lines
qapi/machine.json +1
-1@@ -1404,7 +1404,9 @@ # # @dies: number of dies per socket in the CPU topology # -# @cores: number of cores per die in the CPU topology +# @clusters: number of clusters per die in the CPU topologyMissing:
Quoted 1 line
# (since 7.0)
Ah, yes.
Quoted 6 lines
Quoted 6 lines
qapi/machine.json +2
-0+# +# @cores: number of cores per cluster in the CPU topology # # @threads: number of threads per core in the CPU topology #@@ -1416,6 +1418,7 @@
Quoted 8 lines
qapi/machine.json line 1424
Quoted 7 lines
qapi/machine.json +1
-0'*cpus': 'int', '*sockets': 'int', '*dies': 'int', + '*clusters': 'int', '*cores': 'int', '*threads': 'int', '*maxcpus': 'int' } }If you want I can update the doc when applying.
Do you mean the missing "since 7.0"?
If you have a plan to apply the first 1-7 patches separately and I don't need to respin, please help to update it, thank you! :)
Thanks,
Yanan
Quoted 5 lines
Thanks,
Phil.
.
Re: [PATCH v5 02/14] hw/core/machine: Introduce CPU cluster topology support
Quoted 33 lines · wangyanan (Y)
Hi Philippe,
Thanks for your review.Quoted 29 lines · Philippe Mathieu-Daudé
Hi,
Quoted 26 lines · Yanan Wang
The new Cluster-Aware Scheduling support has landed in Linux 5.16, which has been proved to benefit the scheduling performance (e.g. load balance and
wake_affinestrategy) on bothx86_64and AArch64.So now in Linux 5.16 we have four-level arch-neutral CPU topology definition like below and a new scheduler level for clusters. struct
cpu_topology{
intthread_id;
intcore_id;
intcluster_id;
intpackage_id;
intllc_id;
cpumask_tthread_sibling;
cpumask_tcore_sibling;
cpumask_tcluster_sibling;
cpumask_tllc_sibling;
}A cluster generally means a group of CPU cores which share L2 cache or other mid-level resources, and it is the shared resources that is used to improve scheduler's behavior. From the point of view of the size range, it's between CPU die and CPU core. For example, on some ARM64 Kunpeng servers, we have 6 clusters in each NUMA node, and 4 CPU cores in each cluster. The 4 CPU cores share a separate L2 cache and a L3 cache tag, which brings cache affinity advantage.
Quoted 3 lines
Quoted 2 lines
Quoted 1 line
In virtualization, on the Hosts which have pClusters, if we can
Maybe [*] -> reference to pClusters?
Hm, I'm not sure what kind of reference is appropriate here.
So I guess the confusion comes from a simple typo =)
Is it OK if I replace "pClusters" by "Clusters"?
Quoted 73 lines
The third paragraph in the commit message has explained what a cluster generally means. We can also read the description of clusters in Linux kernel
Kconfigfiles: [1] and [2].[1]arm64: https://github.com/torvalds/linux/blob/master/arch/arm64/Kconfig
config
SCHED_CLUSTER
bool "Cluster scheduler support"
help
Cluster scheduler support improves the CPU scheduler's decision making when dealing with machines that have clusters of CPUs. Cluster usually means a couple of CPUs which are placed closely by sharing mid-level caches, last-level cache tags or internal busses.[2]x86: https://github.com/torvalds/linux/blob/master/arch/x86/Kconfig
config
SCHED_CLUSTER
bool "Cluster scheduler support"
depends on SMP
default y
help
Cluster scheduler support improves the CPU scheduler's decision making when dealing with machines that have clusters of CPUs. Cluster usually means a couple of CPUs which are placed closely by sharing mid-level caches, last-level cache tags or internal busses.Quoted 30 lines
Quoted 23 lines
design a vCPU topology with cluster level for guest kernel and have a dedicated vCPU pinning. A Cluster-Aware Guest kernel can also make use of the cache affinity of CPU clusters to gain similar scheduling performance.
This patch adds infrastructure for CPU cluster level topology configuration and parsing, so that the user can specify cluster parameter if their machines support it.
Signed-off-by Yanan Wang <[email protected]>---
hw/core/machine-smp.c| 26 +++++++++++++++++++------- hw/core/machine.c | 3 +++ include/hw/boards.h | 6 +++++- qapi/machine.json | 5 ++++- qemu-options.hx | 7 ++++--- softmmu/vl.c | 3 +++6 files changed, 38 insertions(+), 12 deletions(-)qapi/machine.json +0
-0qapi/machine.json line 1407
Quoted 5 lines
qapi/machine.json +1
-1@@ -1404,7 +1404,9 @@ # # @dies: number of dies per socket in the CPU topology # -# @cores: number of cores per die in the CPU topology +# @clusters: number of clusters per die in the CPU topologyMissing:
Quoted 1 line
# (since 7.0)
Ah, yes.
Quoted 6 lines
Quoted 6 lines
qapi/machine.json +2
-0+# +# @cores: number of cores per cluster in the CPU topology # # @threads: number of threads per core in the CPU topology #@@ -1416,6 +1418,7 @@Quoted 8 lines
qapi/machine.json line 1424
Quoted 7 lines
qapi/machine.json +1
-0'*cpus': 'int', '*sockets': 'int', '*dies': 'int', + '*clusters': 'int', '*cores': 'int', '*threads': 'int', '*maxcpus': 'int' } }If you want I can update the doc when applying.
Quoted 3 lines
Do you mean the missing "since 7.0"?
If you have a plan to apply the first 1-7 patches separately and I don't need to respin, please help to update it, thank you! :)
Yes, that is the plan.
Quoted 9 lines
Thanks,
YananQuoted 5 lines
Thanks,
Phil.
.
Re: [PATCH v5 02/14] hw/core/machine: Introduce CPU cluster topology support
Quoted 34 lines · Philippe Mathieu-Daudé
Quoted 33 lines · wangyanan (Y)
Hi Philippe,
Thanks for your review.Quoted 29 lines · Philippe Mathieu-Daudé
Hi,
Quoted 26 lines · Yanan Wang
The new Cluster-Aware Scheduling support has landed in Linux 5.16, which has been proved to benefit the scheduling performance (e.g. load balance and
wake_affinestrategy) on bothx86_64and AArch64.So now in Linux 5.16 we have four-level arch-neutral CPU topology definition like below and a new scheduler level for clusters. struct
cpu_topology{
intthread_id;
intcore_id;
intcluster_id;
intpackage_id;
intllc_id;
cpumask_tthread_sibling;
cpumask_tcore_sibling;
cpumask_tcluster_sibling;
cpumask_tllc_sibling;
}A cluster generally means a group of CPU cores which share L2 cache or other mid-level resources, and it is the shared resources that is used to improve scheduler's behavior. From the point of view of the size range, it's between CPU die and CPU core. For example, on some ARM64 Kunpeng servers, we have 6 clusters in each NUMA node, and 4 CPU cores in each cluster. The 4 CPU cores share a separate L2 cache and a L3 cache tag, which brings cache affinity advantage.
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
Quoted 1 line
In virtualization, on the Hosts which have pClusters, if we can
Maybe [*] -> reference to pClusters?
Hm, I'm not sure what kind of reference is appropriate here.
So I guess the confusion comes from a simple typo =)
I tried to mean "physical clusters" on host by pClusters, on the contrary to "virtual clusters" on guest. But obviously it brings confusion.
Quoted 1 line
Is it OK if I replace "pClusters" by "Clusters"?
Sure, it's clearer to just use "clusters", please do that.
Quoted 73 lines
Quoted 73 lines
The third paragraph in the commit message has explained what a cluster generally means. We can also read the description of clusters in Linux kernel
Kconfigfiles: [1] and [2].[1]arm64: https://github.com/torvalds/linux/blob/master/arch/arm64/Kconfig
config
SCHED_CLUSTER
bool "Cluster scheduler support"
help
Cluster scheduler support improves the CPU scheduler's decision making when dealing with machines that have clusters of CPUs. Cluster usually means a couple of CPUs which are placed closely by sharing mid-level caches, last-level cache tags or internal busses.[2]x86: https://github.com/torvalds/linux/blob/master/arch/x86/Kconfig
config
SCHED_CLUSTER
bool "Cluster scheduler support"
depends on SMP
default y
help
Cluster scheduler support improves the CPU scheduler's decision making when dealing with machines that have clusters of CPUs. Cluster usually means a couple of CPUs which are placed closely by sharing mid-level caches, last-level cache tags or internal busses.Quoted 30 lines
Quoted 23 lines
design a vCPU topology with cluster level for guest kernel and have a dedicated vCPU pinning. A Cluster-Aware Guest kernel can also make use of the cache affinity of CPU clusters to gain similar scheduling performance.
This patch adds infrastructure for CPU cluster level topology configuration and parsing, so that the user can specify cluster parameter if their machines support it.
Signed-off-by Yanan Wang <[email protected]>--- hw/core/machine-smp.c | 26 +++++++++++++++++++------- hw/core/machine.c | 3 +++ include/hw/boards.h | 6 +++++- qapi/machine.json | 5 ++++- qemu-options.hx | 7 ++++--- softmmu/vl.c | 3 +++ 6 files changed, 38 insertions(+), 12 deletions(-)qapi/machine.json +0
-0qapi/machine.json line 1407
Quoted 5 lines
qapi/machine.json +1
-1@@ -1404,7 +1404,9 @@ # # @dies: number of dies per socket in the CPU topology # -# @cores: number of cores per die in the CPU topology +# @clusters: number of clusters per die in the CPU topologyMissing:
Quoted 1 line
# (since 7.0)
Ah, yes.
Quoted 14 lines
qapi/machine.json line 1424
Quoted 13 lines
qapi/machine.json +3
-0+# +# @cores: number of cores per cluster in the CPU topology # # @threads: number of threads per core in the CPU topology #@@ -1416,6 +1418,7 @@ '*cpus': 'int', '*sockets': 'int', '*dies': 'int', + '*clusters': 'int', '*cores': 'int', '*threads': 'int', '*maxcpus': 'int' } }If you want I can update the doc when applying.
Quoted 4 lines
Quoted 3 lines
Do you mean the missing "since 7.0"?
If you have a plan to apply the first 1-7 patches separately and I don't need to respin, please help to update it, thank you! :)Yes, that is the plan.
Thank you! I will pack the rest for ARM into next version separately after you queue the generic part.
Thanks,
Yanan
Quoted 8 lines
Quoted 7 lines
Thanks,
YananQuoted 5 lines
Thanks,
Phil.
.
.
Re: [PATCH v5 02/14] hw/core/machine: Introduce CPU cluster topology support
Quoted 36 lines · wangyanan (Y)
Quoted 34 lines · Philippe Mathieu-Daudé
Quoted 33 lines · wangyanan (Y)
Hi Philippe,
Thanks for your review.Quoted 29 lines · Philippe Mathieu-Daudé
Hi,
Quoted 26 lines · Yanan Wang
The new Cluster-Aware Scheduling support has landed in Linux 5.16, which has been proved to benefit the scheduling performance (e.g. load balance and
wake_affinestrategy) on bothx86_64and AArch64.So now in Linux 5.16 we have four-level arch-neutral CPU topology definition like below and a new scheduler level for clusters. struct
cpu_topology{
intthread_id;
intcore_id;
intcluster_id;
intpackage_id;
intllc_id;
cpumask_tthread_sibling;
cpumask_tcore_sibling;
cpumask_tcluster_sibling;
cpumask_tllc_sibling;
}A cluster generally means a group of CPU cores which share L2 cache or other mid-level resources, and it is the shared resources that is used to improve scheduler's behavior. From the point of view of the size range, it's between CPU die and CPU core. For example, on some ARM64 Kunpeng servers, we have 6 clusters in each NUMA node, and 4 CPU cores in each cluster. The 4 CPU cores share a separate L2 cache and a L3 cache tag, which brings cache affinity advantage.
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
Quoted 1 line
In virtualization, on the Hosts which have pClusters, if we can
Maybe [*] -> reference to pClusters?
Hm, I'm not sure what kind of reference is appropriate here.
So I guess the confusion comes from a simple typo =)
I tried to mean "physical clusters" on host by pClusters, on the contrary to "virtual clusters" on guest. But obviously it brings confusion.
OK, I got confused because you don't use "vClusters".
Quoted 2 lines
Quoted 1 line
Is it OK if I replace "pClusters" by "Clusters"?
Sure, it's clearer to just use "clusters", please do that.
OK.
Markus Armbruster
Acked-byRe: [PATCH v5 02/14] hw/core/machine: Introduce CPU cluster topology support
In reply to Philippe Mathieu-DaudéQuoted 3 lines · Philippe Mathieu-Daudé
Hi,
On 12/28/21 10:22, Yanan Wang wrote:
[...]
Quoted 15 lines
qapi/machine.json line 1407
Quoted 10 lines
qapi/machine.json +1
-1@@ -1404,7 +1404,9 @@ # # @dies: number of dies per socket in the CPU topology # -# @cores: number of cores per die in the CPU topology +# @clusters: number of clusters per die in the CPU topologyMissing:
# (since 7.0)
Quoted 15 lines
qapi/machine.json line 1424
Quoted 13 lines
qapi/machine.json +3
-0+# +# @cores: number of cores per cluster in the CPU topology # # @threads: number of threads per core in the CPU topology #@@ -1416,6 +1418,7 @@ '*cpus': 'int', '*sockets': 'int', '*dies': 'int', + '*clusters': 'int', '*cores': 'int', '*threads': 'int', '*maxcpus': 'int' } }If you want I can update the doc when applying.
With the update, QAPU schema
PATCH v5 4/14 tests/unit/test-smp-parse: Add testcases for CPU clusters
In reply to Yanan Wang viaAdd testcases for parsing of the four-level CPU topology hierarchy, ie sockets/clusters/cores/threads, which will be supported on ARM virt machines.
--- tests/unit/test-smp-parse.c | 130 ++++++++++++++++++++++++++++++++++-- 1 file changed, 123 insertions(+), 7 deletions(-)
tests/unit/test-smp-parse.c +123 -7
@@ -61,6 +61,20 @@ .has_maxcpus = hf, .maxcpus = f, \ } +/* +* Currently a 4-level topology hierarchy is supported on ARM virt machines +* -sockets/clusters/cores/threads +*/ +#define SMP_CONFIG_WITH_CLUSTERS (ha, a, hb, b, hc, c, hd, d, he, e, hf, f) \ + { \ + .has_cpus = ha, .cpus = a, \ + .has_sockets = hb, .sockets = b, \ + .has_clusters = hc, .clusters = c, \ + .has_cores = hd, .cores = d, \ + .has_threads = he, .threads = e, \ + .has_maxcpus = hf, .maxcpus = f, \ + } +/** * @config - the given SMP configuration * @expect_prefer_sockets - the expected parsing result for the
@@ -290,6 +304,10 @@ static const struct SMPTestData data_generic_invalid[] = {/* config: -smp 2,dies=2 */ .config = SMP_CONFIG_WITH_DIES(T, 2, F, 0, T, 2, F, 0, F, 0, F, 0), .expect_error = "dies not supported by this machine's CPU topology", + }, { +/* config: -smp 2,clusters=2 */ + .config = SMP_CONFIG_WITH_CLUSTERS(T, 2, F, 0, T, 2, F, 0, F, 0, F, 0), + .expect_error = "clusters not supported by this machine's CPU topology", }, {/* config: -smp 8,sockets=2,cores=4,threads=2,maxcpus=8 */ .config = SMP_CONFIG_GENERIC(T, 8, T, 2, T, 4, T, 2, T, 8),
@@ -337,20 +355,40 @@ static const struct SMPTestData data_with_dies_invalid[] = { }, }; +static const struct SMPTestData data_with_clusters_invalid[] = { + { +/* config: -smp 16,sockets=2,clusters=2,cores=4,threads=2,maxcpus=16 */ + .config =SMP_CONFIG_WITH_CLUSTERS (T ,16 ,T ,2 ,T ,2 ,T ,4 ,T ,2 ,T ,16 ), + .expect_error ="Invalid CPU topology: " +"product of the hierarchy must match maxcpus: " +"sockets (2) * clusters (2) * cores (4) * threads (2) " +"!= maxcpus (16)" , + }, { +/* config: -smp 34,sockets=2,clusters=2,cores=4,threads=2,maxcpus=32 */ + .config =SMP_CONFIG_WITH_CLUSTERS (T ,34 ,T ,2 ,T ,2 ,T ,4 ,T ,2 ,T ,32 ), + .expect_error ="Invalid CPU topology: " +"maxcpus must be equal to or greater than smp: " +"sockets (2) * clusters (2) * cores (4) * threads (2) " +"== maxcpus (32) < smp_cpus (34)" , + }, +}; +static char *smp_config_to_string (const SMPConfiguration *config) {return g_strdup_printf ("(SMPConfiguration) {\n" -" .has_cpus = %5s, cpus = %" PRId64",\n" -" .has_sockets = %5s, sockets = %" PRId64",\n" -" .has_dies = %5s, dies = %" PRId64",\n" -" .has_cores = %5s, cores = %" PRId64",\n" -" .has_threads = %5s, threads = %" PRId64",\n" -" .has_maxcpus = %5s, maxcpus = %" PRId64",\n" +" .has_cpus = %5s, cpus = %" PRId64",\n" +" .has_sockets = %5s, sockets = %" PRId64",\n" +" .has_dies = %5s, dies = %" PRId64",\n" +" .has_clusters = %5s, clusters = %" PRId64",\n" +" .has_cores = %5s, cores = %" PRId64",\n" +" .has_threads = %5s, threads = %" PRId64",\n" +" .has_maxcpus = %5s, maxcpus = %" PRId64",\n" "}" , config->has_cpus ?"true" :"false" , config->cpus, config->has_sockets ?"true" :"false" , config->sockets, config->has_dies ?"true" :"false" , config->dies, + config->has_clusters ?"true" :"false" , config->clusters, config->has_cores ?"true" :"false" , config->cores, config->has_threads ?"true" :"false" , config->threads, config->has_maxcpus ?"true" :"false" , config->maxcpus);
@@ -363,11 +401,12 @@ static char *cpu_topology_to_string(const CpuTopology *topo)" .cpus = %u,\n" " .sockets = %u,\n" " .dies = %u,\n" +" .clusters = %u,\n" " .cores = %u,\n" " .threads = %u,\n" " .max_cpus = %u,\n" "}" , - topo->cpus, topo->sockets, topo->dies, + topo->cpus, topo->sockets, topo->dies, topo->clusters, topo->cores, topo->threads, topo->max_cpus); }
@@ -391,6 +430,7 @@ static void check_parse(MachineState *ms, const SMPConfiguration *config, (ms->smp.cpus == expect_topo->cpus) && (ms->smp.sockets == expect_topo->sockets) && (ms->smp.dies == expect_topo->dies) && + (ms->smp.clusters == expect_topo->clusters) && (ms->smp.cores == expect_topo->cores) && (ms->smp.threads == expect_topo->threads) && (ms->smp.max_cpus == expect_topo->max_cpus)) {
@@ -472,6 +512,11 @@ static void unsupported_params_init(const MachineClass *mc, SMPTestData *data) data->expect_prefer_sockets.dies =1 ; data->expect_prefer_cores.dies =1 ; } + +if (!mc->smp_props.clusters_supported) { + data->expect_prefer_sockets.clusters =1 ; + data->expect_prefer_cores.clusters =1 ; + } }static void machine_base_class_init (ObjectClass *oc,void *data)
@@ -491,6 +536,7 @@ static void machine_generic_valid_class_init(ObjectClass *oc, void *data) mc->max_cpus =MAX_CPUS ; mc->smp_props.dies_supported = false; + mc->smp_props.clusters_supported = false; }static void machine_generic_invalid_class_init (ObjectClass *oc,void *data)
@@ -502,6 +548,7 @@ static void machine_generic_invalid_class_init(ObjectClass *oc, void *data) mc->max_cpus =511 ; mc->smp_props.dies_supported = false; + mc->smp_props.clusters_supported = false; }static void machine_with_dies_class_init (ObjectClass *oc,void *data)
@@ -512,6 +559,18 @@ static void machine_with_dies_class_init(ObjectClass *oc, void *data) mc->max_cpus =MAX_CPUS ; mc->smp_props.dies_supported = true; + mc->smp_props.clusters_supported = false; +} + +static void machine_with_clusters_class_init (ObjectClass *oc,void *data) +{ +MachineClass *mc =MACHINE_CLASS (oc); + + mc->min_cpus =MIN_CPUS ; + mc->max_cpus =MAX_CPUS ; + + mc->smp_props.clusters_supported = true; + mc->smp_props.dies_supported = false; }static void test_generic_valid (const void *opaque)
@@ -607,6 +666,56 @@ static void test_with_dies(const void *opaque)object_unref (obj); } +static void test_with_clusters (const void *opaque) +{ +const char *machine_type = opaque; +Object *obj =object_new (machine_type); +MachineState *ms =MACHINE (obj); +MachineClass *mc =MACHINE_GET_CLASS (obj); +SMPTestData data = {}; +unsigned int num_clusters =2 ; +int i; + +for (i =0 ; i <ARRAY_SIZE (data_generic_valid); i++) { + data = data_generic_valid[i]; +unsupported_params_init (mc, &data); + +/* when clusters parameter is omitted, it will be set as 1 */ + data.expect_prefer_sockets.clusters =1 ; + data.expect_prefer_cores.clusters =1 ; + +smp_parse_test (ms, &data, true); + +/* when clusters parameter is specified */ + data.config.has_clusters = true; + data.config.clusters = num_clusters; +if (data.config.has_cpus) { + data.config.cpus *= num_clusters; + } +if (data.config.has_maxcpus) { + data.config.maxcpus *= num_clusters; + } + + data.expect_prefer_sockets.clusters = num_clusters; + data.expect_prefer_sockets.cpus *= num_clusters; + data.expect_prefer_sockets.max_cpus *= num_clusters; + data.expect_prefer_cores.clusters = num_clusters; + data.expect_prefer_cores.cpus *= num_clusters; + data.expect_prefer_cores.max_cpus *= num_clusters; + +smp_parse_test (ms, &data, true); + } + +for (i =0 ; i <ARRAY_SIZE (data_with_clusters_invalid); i++) { + data = data_with_clusters_invalid[i]; +unsupported_params_init (mc, &data); + +smp_parse_test (ms, &data, false); + } + +object_unref (obj); +} +/* Type info of the tested machine */ static const TypeInfo smp_machine_types[] = { {
@@ -628,6 +737,10 @@ static const TypeInfo smp_machine_types[] = { .name = MACHINE_TYPE_NAME("smp-with-dies"), .parent = TYPE_MACHINE, .class_init = machine_with_dies_class_init, + }, { + .name = MACHINE_TYPE_NAME("smp-with-clusters"), + .parent = TYPE_MACHINE, + .class_init = machine_with_clusters_class_init, } };
@@ -648,6 +761,9 @@ int main(int argc, char *argv[])g_test_add_data_func ("/test-smp-parse/with_dies" ,MACHINE_TYPE_NAME ("smp-with-dies" ), test_with_dies); +g_test_add_data_func ("/test-smp-parse/with_clusters" , +MACHINE_TYPE_NAME ("smp-with-clusters" ), + test_with_clusters);g_test_run ();
Signature · 1 line
2.27.0
Re: [PATCH v5 04/14] tests/unit/test-smp-parse: Add testcases for CPU clusters
Quoted 8 lines · Yanan Wang
Add testcases for parsing of the four-level CPU topology hierarchy, ie sockets/clusters/cores/threads, which will be supported on ARM virt machines.
Signed-off-by Yanan Wang <[email protected]>--- tests/unit/test-smp-parse.c | 130 ++++++++++++++++++++++++++++++++++-- 1 file changed, 123 insertions(+), 7 deletions(-)
PATCH v5 5/14 tests/unit/test-smp-parse: No need to explicitly zero MachineClass members
In reply to Yanan Wang viaThe default value of the MachineClass members is 0, which means we don't have to explicitly zero them. Also the value of "mc->smp_props.prefer_sockets" will be taken care of by smp_parse_test(), we don't necessarily need the statement in machine_base_class_init() either.
--- tests/unit/test-smp-parse.c | 10 ---------- 1 file changed, 10 deletions(-)
tests/unit/test-smp-parse.c +0 -10
@@ -523,8 +523,6 @@ static void machine_base_class_init(ObjectClass *oc, void *data) {MachineClass *mc =MACHINE_CLASS (oc); - mc->smp_props.prefer_sockets = true; - mc->name =g_strdup (SMP_MACHINE_NAME ); }
@@ -534,9 +532,6 @@ static void machine_generic_valid_class_init(ObjectClass *oc, void *data) mc->min_cpus =MIN_CPUS ; mc->max_cpus =MAX_CPUS ; - - mc->smp_props.dies_supported = false; - mc->smp_props.clusters_supported = false; }static void machine_generic_invalid_class_init (ObjectClass *oc,void *data)
@@ -546,9 +541,6 @@ static void machine_generic_invalid_class_init(ObjectClass *oc, void *data)/* Force invalid min CPUs and max CPUs */ mc->min_cpus =2 ; mc->max_cpus =511 ; - - mc->smp_props.dies_supported = false; - mc->smp_props.clusters_supported = false; }static void machine_with_dies_class_init (ObjectClass *oc,void *data)
@@ -559,7 +551,6 @@ static void machine_with_dies_class_init(ObjectClass *oc, void *data) mc->max_cpus =MAX_CPUS ; mc->smp_props.dies_supported = true; - mc->smp_props.clusters_supported = false; }static void machine_with_clusters_class_init (ObjectClass *oc,void *data)
@@ -570,7 +561,6 @@ static void machine_with_clusters_class_init(ObjectClass *oc, void *data) mc->max_cpus =MAX_CPUS ; mc->smp_props.clusters_supported = true; - mc->smp_props.dies_supported = false; }static void test_generic_valid (const void *opaque)
Signature · 1 line
2.27.0
Re: [PATCH v5 05/14] tests/unit/test-smp-parse: No need to explicitly zero MachineClass members
Quoted 10 lines · Yanan Wang
The default value of the MachineClass members is 0, which means we don't have to explicitly zero them. Also the value of "
mc->smp_props.prefer_sockets" will be taken care of bysmp_parse_test(), we don't necessarily need the statement inmachine_base_class_init()either.Signed-off-by Yanan Wang <[email protected]>--- tests/unit/test-smp-parse.c | 10 ---------- 1 file changed, 10 deletions(-)
PATCH v5 7/14 MAINTAINERS: Self-recommended as reviewer of "Machine core"
In reply to Yanan Wang viaI've built interests in the generic machine subsystem and have also been working on projects related to this part, self-recommand myself as a reviewer so that I can help to review some patches familiar to me, and have a chance to learn more continuously.
--- MAINTAINERS | 1 + 1 file changed, 1 insertion(+)
MAINTAINERS +1 -0
@@ -1631,6 +1631,7 @@ Machine core M: Eduardo Habkost <[email protected]> M: Marcel Apfelbaum <[email protected]> R: Philippe Mathieu-Daudé <[email protected]> +R: Yanan Wang <[email protected]> S: Supported F: cpu.c F: hw/core/cpu.c
Signature · 1 line
2.27.0
Re: [PATCH v5 07/14] MAINTAINERS: Self-recommended as reviewer of "Machine core"
Quoted 10 lines · Yanan Wang
I've built interests in the generic machine subsystem and have also been working on projects related to this part, self-recommand myself as a reviewer so that I can help to review some patches familiar to me, and have a chance to learn more continuously.
Signed-off-by Yanan Wang <[email protected]>--- MAINTAINERS | 1 + 1 file changed, 1 insertion(+)
PATCH v5 6/14 tests/unit/test-smp-parse: Keep default MIN/MAX CPUs in machine_base_class_init
In reply to Yanan Wang viaMost machine types in test-smp-parse will be OK to have the default MIN/MAX CPUs except "smp-generic-invalid", let's keep the default values in machine_base_class_init which will be inherited. And if we hope a different value for a specific machine, modify it in its own initialization function.
--- tests/unit/test-smp-parse.c | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-)
tests/unit/test-smp-parse.c +2 -14
@@ -523,15 +523,10 @@ static void machine_base_class_init(ObjectClass *oc, void *data) {MachineClass *mc =MACHINE_CLASS (oc); - mc->name =g_strdup (SMP_MACHINE_NAME ); -} - -static void machine_generic_valid_class_init (ObjectClass *oc,void *data) -{ -MachineClass *mc =MACHINE_CLASS (oc); - mc->min_cpus =MIN_CPUS ; mc->max_cpus =MAX_CPUS ; + + mc->name =g_strdup (SMP_MACHINE_NAME ); }static void machine_generic_invalid_class_init (ObjectClass *oc,void *data)
@@ -547,9 +542,6 @@ static void machine_with_dies_class_init(ObjectClass *oc, void *data) {MachineClass *mc =MACHINE_CLASS (oc); - mc->min_cpus =MIN_CPUS ; - mc->max_cpus =MAX_CPUS ; - mc->smp_props.dies_supported = true; }
@@ -557,9 +549,6 @@ static void machine_with_clusters_class_init(ObjectClass *oc, void *data) {MachineClass *mc =MACHINE_CLASS (oc); - mc->min_cpus =MIN_CPUS ; - mc->max_cpus =MAX_CPUS ; - mc->smp_props.clusters_supported = true; }
@@ -718,7 +707,6 @@ static const TypeInfo smp_machine_types[] = { }, { .name = MACHINE_TYPE_NAME("smp-generic-valid"), .parent = TYPE_MACHINE, - .class_init = machine_generic_valid_class_init, }, { .name = MACHINE_TYPE_NAME("smp-generic-invalid"), .parent = TYPE_MACHINE,
Signature · 1 line
2.27.0
Re: [PATCH v5 06/14] tests/unit/test-smp-parse: Keep default MIN/MAX CPUs in machine_base_class_init
Quoted 10 lines · Yanan Wang
Most machine types in test-smp-parse will be OK to have the default MIN/MAX CPUs except "smp-generic-invalid", let's keep the default values in
machine_base_class_initwhich will be inherited. And if we hope a different value for a specific machine, modify it in its own initialization function.Signed-off-by Yanan Wang <[email protected]>--- tests/unit/test-smp-parse.c | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-)
PATCH v5 9/14 hw/arm/virt: Support cluster level in DT cpu-map
In reply to Yanan Wang viaSupport one cluster level between core and physical package in the cpu-map of Arm/virt devicetree. This is also consistent with Linux Doc "Documentation/devicetree/bindings/cpu/cpu-topology.txt".
--- hw/arm/virt.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
hw/arm/virt.c +8 -7
@@ -430,9 +430,8 @@ static void fdt_add_cpu_nodes(const VirtMachineState *vms)* can contain several layers of clustering within a single physical * package and cluster nodes can be contained in parent cluster nodes. * -* Given that cluster is not yet supported in the vCPU topology, -* we currently generate one cluster node within each socket node -* by default. +* Note: currently we only support one layer of clustering within +* each physical package. */ qemu_fdt_add_subnode (ms->fdt,"/cpus/cpu-map" );
@@ -442,14 +441,16 @@ static void fdt_add_cpu_nodes(const VirtMachineState *vms)if (ms->smp.threads >1 ) { map_path =g_strdup_printf ( -"/cpus/cpu-map/socket%d/cluster0/core%d/thread%d" , - cpu / (ms->smp.cores * ms->smp.threads), +"/cpus/cpu-map/socket%d/cluster%d/core%d/thread%d" , + cpu / (ms->smp.clusters * ms->smp.cores * ms->smp.threads), + (cpu / (ms->smp.cores * ms->smp.threads)) % ms->smp.clusters, (cpu / ms->smp.threads) % ms->smp.cores, cpu % ms->smp.threads); }else { map_path =g_strdup_printf ( -"/cpus/cpu-map/socket%d/cluster0/core%d" , - cpu / ms->smp.cores, +"/cpus/cpu-map/socket%d/cluster%d/core%d" , + cpu / (ms->smp.clusters * ms->smp.cores), + (cpu / ms->smp.cores) % ms->smp.clusters, cpu % ms->smp.cores); }qemu_fdt_add_path (ms->fdt, map_path);
Signature · 1 line
2.27.0
PATCH v5 8/14 hw/arm/virt: Support clusters on ARM virt machines
In reply to Yanan Wang viaIn implementations of ARM64 architecture, at most there could be a CPU topology hierarchy like "sockets/dies/clusters/cores/threads" defined. For example, some ARM64 server chip Kunpeng 920 totally has 2 sockets, 2 NUMA nodes (also represent CPU dies range) in each socket, 6 clusters in each NUMA node, 4 CPU cores in each cluster.
Clusters within the same NUMA share the L3 cache data and cores within the same cluster share a L2 cache and a L3 cache tag. Given that designing a vCPU topology with cluster level for the guest can gain scheduling performance improvement, let's support this new parameter on ARM virt machines.
After this, we can define a 4-level CPU topology hierarchy like:
cpus=*,maxcpus=*,sockets=*,clusters=*,cores=*,threads=*.
--- hw/arm/virt.c | 1 + qemu-options.hx | 10 ++++++++++ 2 files changed, 11 insertions(+)
hw/arm/virt.c +1 -0
@@ -2700,6 +2700,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) hc->unplug_request = virt_machine_device_unplug_request_cb; hc->unplug = virt_machine_device_unplug_cb; mc->nvdimm_supported = true; + mc->smp_props.clusters_supported = true; mc->auto_enable_numa_with_memhp = true; mc->auto_enable_numa_with_memdev = true; mc->default_ram_id ="mach-virt.ram" ;
qemu-options.hx +10 -0
@@ -277,6 +277,16 @@ SRST -smp 16,sockets=2,dies=2,cores=2,threads=2,maxcpus=16 + The following sub-option defines a CPU topology hierarchy (2 sockets + totally on the machine, 2 clusters per socket, 2 cores per cluster, + 2 threads per core) for ARM virt machines which support sockets/clusters + /cores/threads. Some members of the option can be omitted but their values + will be automatically computed: + + :: + + -smp 16,sockets=2,clusters=2,cores=2,threads=2,maxcpus=16 + Historically preference was given to the coarsest topology parameters when computing missing values (ie sockets preferred over cores, which were preferred over threads), however, this behaviour is considered
Signature · 1 line
2.27.0
PATCH v5 10/14 hw/acpi/aml-build: Improve scalability of PPTT generation
In reply to Yanan Wang viaCurrently we generate a PPTT table of n-level processor hierarchy with n-level loops in build_pptt(). It works fine as now there are only three CPU topology parameters. But the code may become less scalable with the processor hierarchy levels increasing.
This patch only improves the scalability of build_pptt by reducing the loops, and intends to make no functional change.
--- hw/acpi/aml-build.c | 50 +++++++++++++++++++++++++++++---------------- 1 file changed, 32 insertions(+), 18 deletions(-)
hw/acpi/aml-build.c +32 -18
@@ -2001,7 +2001,10 @@ static void build_processor_hierarchy_node(GArray *tbl, uint32_t flags,void build_pptt (GArray *table_data,BIOSLinker *linker,MachineState *ms,const char *oem_id,const char *oem_table_id) { -int pptt_start = table_data->len; +GQueue *list =g_queue_new (); +guint pptt_start = table_data->len; +guint father_offset; +guint length, i;int uid =0 ;int socket;AcpiTable table = { .sig ="PPTT" , .rev =2 ,
@@ -2010,9 +2013,8 @@ void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms,acpi_table_begin (&table, table_data);for (socket =0 ; socket < ms->smp.sockets; socket++) { -uint32_t socket_offset = table_data->len - pptt_start; -int core; - +g_queue_push_tail (list, +GUINT_TO_POINTER (table_data->len - pptt_start)); build_processor_hierarchy_node( table_data,/*
@@ -2021,35 +2023,47 @@ void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms,*/ (1 <<0 ), 0, socket, NULL, 0); + } -for (core =0 ; core < ms->smp.cores; core++) { -uint32_t core_offset = table_data->len - pptt_start; -int thread; + length =g_queue_get_length (list); +for (i =0 ; i < length; i++) { +int core; + father_offset =GPOINTER_TO_UINT (g_queue_pop_head (list)); +for (core =0 ; core < ms->smp.cores; core++) {if (ms->smp.threads >1 ) { +g_queue_push_tail (list, +GUINT_TO_POINTER (table_data->len - pptt_start));build_processor_hierarchy_node ( table_data, (0 <<0 ),/* not a physical package */ - socket_offset, core,NULL ,0 ); - -for (thread =0 ; thread < ms->smp.threads; thread++) { -build_processor_hierarchy_node ( - table_data, - (1 <<1 ) |/* ACPI Processor ID valid */ - (1 <<2 ) |/* Processor is a Thread */ - (1 <<3 ),/* Node is a Leaf */ - core_offset, uid++,NULL ,0 ); - } + father_offset, core,NULL ,0 ); }else {build_processor_hierarchy_node ( table_data, (1 <<1 ) |/* ACPI Processor ID valid */ (1 <<3 ),/* Node is a Leaf */ - socket_offset, uid++,NULL ,0 ); + father_offset, uid++,NULL ,0 ); } } } + length =g_queue_get_length (list); +for (i =0 ; i < length; i++) { +int thread; + + father_offset =GPOINTER_TO_UINT (g_queue_pop_head (list)); +for (thread =0 ; thread < ms->smp.threads; thread++) { +build_processor_hierarchy_node ( + table_data, + (1 <<1 ) |/* ACPI Processor ID valid */ + (1 <<2 ) |/* Processor is a Thread */ + (1 <<3 ),/* Node is a Leaf */ + father_offset, uid++,NULL ,0 ); + } + } + +g_queue_free (list);acpi_table_end (linker, &table); }
Signature · 1 line
2.27.0
PATCH v5 11/14 hw/arm/virt-acpi-build: Make an ARM specific PPTT generator
In reply to Yanan Wang viaWe have a generic build_pptt() in hw/acpi/aml-build.c but it's currently only used in ARM acpi initialization. Now we are going to support the new CPU cluster parameter which is currently only supported by ARM, it won't be a very good idea to add it to the generic build_pptt() as it will make the code complex and hard to maintain especially when we also support CPU cache topology hierarchy in build_pptt() too. Note that the cache topology design also varies between different CPU targets.
So an ARM specific PPTT generator becomes necessary now. Given that the generic one is currently only used by ARM, let's just move build_pptt() from aml-build.c to virt-acpi-build.c with minor update.
--- hw/acpi/aml-build.c | 80 ++----------------------------------- hw/arm/virt-acpi-build.c | 77 ++++++++++++++++++++++++++++++++++- include/hw/acpi/aml-build.h | 5 ++- 3 files changed, 81 insertions(+), 81 deletions(-)
hw/acpi/aml-build.c +3 -77
@@ -1968,10 +1968,9 @@ void build_slit(GArray *table_data, BIOSLinker *linker, MachineState *ms,* ACPI spec, Revision 6.3 * 5.2.29.1 Processor hierarchy node structure (Type 0) */ -static void build_processor_hierarchy_node (GArray *tbl,uint32_t flags, -uint32_t parent,uint32_t id, -uint32_t *priv_rsrc, -uint32_t priv_num) +void build_processor_hierarchy_node (GArray *tbl,uint32_t flags, +uint32_t parent,uint32_t id, +uint32_t *priv_rsrc,uint32_t priv_num) {int i;
@@ -1994,79 +1993,6 @@ static void build_processor_hierarchy_node(GArray *tbl, uint32_t flags, } } -/* -* ACPI spec, Revision 6.3 -* 5.2.29 Processor Properties Topology Table (PPTT) -*/ -void build_pptt (GArray *table_data,BIOSLinker *linker,MachineState *ms, -const char *oem_id,const char *oem_table_id) -{ -GQueue *list =g_queue_new (); -guint pptt_start = table_data->len; -guint father_offset; -guint length, i; -int uid =0 ; -int socket; -AcpiTable table = { .sig ="PPTT" , .rev =2 , - .oem_id = oem_id, .oem_table_id = oem_table_id }; - -acpi_table_begin (&table, table_data); - -for (socket =0 ; socket < ms->smp.sockets; socket++) { -g_queue_push_tail (list, -GUINT_TO_POINTER (table_data->len - pptt_start)); -build_processor_hierarchy_node ( - table_data, -/* -* Physical package - represents the boundary -* of a physical package -*/ - (1 <<0 ), -0 , socket,NULL ,0 ); - } - - length =g_queue_get_length (list); -for (i =0 ; i < length; i++) { -int core; - - father_offset =GPOINTER_TO_UINT (g_queue_pop_head (list)); -for (core =0 ; core < ms->smp.cores; core++) { -if (ms->smp.threads >1 ) { -g_queue_push_tail (list, -GUINT_TO_POINTER (table_data->len - pptt_start)); -build_processor_hierarchy_node ( - table_data, - (0 <<0 ),/* not a physical package */ - father_offset, core,NULL ,0 ); - }else { -build_processor_hierarchy_node ( - table_data, - (1 <<1 ) |/* ACPI Processor ID valid */ - (1 <<3 ),/* Node is a Leaf */ - father_offset, uid++,NULL ,0 ); - } - } - } - - length =g_queue_get_length (list); -for (i =0 ; i < length; i++) { -int thread; - - father_offset =GPOINTER_TO_UINT (g_queue_pop_head (list)); -for (thread =0 ; thread < ms->smp.threads; thread++) { -build_processor_hierarchy_node ( - table_data, - (1 <<1 ) |/* ACPI Processor ID valid */ - (1 <<2 ) |/* Processor is a Thread */ - (1 <<3 ),/* Node is a Leaf */ - father_offset, uid++,NULL ,0 ); - } - } - -g_queue_free (list); -acpi_table_end (linker, &table); -} -/* build rev1/rev3/rev5.1 FADT */ void build_fadt (GArray *tbl,BIOSLinker *linker,const AcpiFadtData *f,const char *oem_id,const char *oem_table_id)
hw/arm/virt-acpi-build.c +75 -2
89 lines not on this page · raw message
include/hw/acpi/aml-build.h +3 -2
@@ -489,8 +489,9 @@ void build_srat_memory(GArray *table_data, uint64_t base,void build_slit (GArray *table_data,BIOSLinker *linker,MachineState *ms,const char *oem_id,const char *oem_table_id); -void build_pptt (GArray *table_data,BIOSLinker *linker,MachineState *ms, -const char *oem_id,const char *oem_table_id); +void build_processor_hierarchy_node (GArray *tbl,uint32_t flags, +uint32_t parent,uint32_t id, +uint32_t *priv_rsrc,uint32_t priv_num);void build_fadt (GArray *tbl,BIOSLinker *linker,const AcpiFadtData *f,const char *oem_id,const char *oem_table_id);
Signature · 1 line
2.27.0
PATCH v5 12/14 tests/acpi/bios-tables-test: Allow changes to virt/PPTT file
In reply to Yanan Wang viaList test/data/acpi/virt/PPTT as the expected files allowed to be changed in tests/qtest/bios-tables-test-allowed-diff.h
--- tests/qtest/bios-tables-test-allowed-diff.h | 1 + 1 file changed, 1 insertion(+)
tests/qtest/bios-tables-test-allowed-diff.h +1 -0
@@ -1 +1,2 @@/* List of comma-separated changed AML files to ignore */ +"tests/data/acpi/virt/PPTT",
Signature · 1 line
2.27.0
PATCH v5 13/14 hw/arm/virt-acpi-build: Support cluster level in PPTT generation
In reply to Yanan Wang viaSupport cluster level in generation of ACPI Processor Properties Topology Table (PPTT) for ARM virt machines.
--- hw/arm/virt-acpi-build.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
hw/arm/virt-acpi-build.c +15 -0
@@ -840,6 +840,21 @@ build_pptt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) 0, socket, NULL, 0); } + length =g_queue_get_length (list); +for (i =0 ; i < length; i++) { +int cluster; + + father_offset =GPOINTER_TO_UINT (g_queue_pop_head (list)); +for (cluster =0 ; cluster < ms->smp.clusters; cluster++) { +g_queue_push_tail (list, +GUINT_TO_POINTER (table_data->len - pptt_start)); +build_processor_hierarchy_node ( + table_data, + (0 <<0 ),/* not a physical package */ + father_offset, cluster,NULL ,0 ); + } + } + length =g_queue_get_length (list);for (i =0 ; i < length; i++) {int core;
Signature · 1 line
2.27.0
PATCH v5 14/14 tests/acpi/bios-table-test: Update expected virt/PPTT file
In reply to Yanan Wang viaRun ./tests/data/acpi/rebuild-expected-aml.sh from build directory to update PPTT binary. Also empty bios-tables-test-allowed-diff.h.
The disassembled differences between actual and expected PPTT:
/*
* Intel ACPI Component Architecture
* AML/ASL+ Disassembler version 20180810 (64-bit version)
* Copyright (c) 2000 - 2018 Intel Corporation
*
- * Disassembly of tests/data/acpi/virt/PPTT, Mon Oct 25 20:24:53 2021
+ * Disassembly of /tmp/aml-BPI5B1, Mon Oct 25 20:24:53 2021 *
* ACPI Data Table [PPTT]
*
* Format: [HexOffset DecimalOffset ByteLength] FieldName : FieldValue */
[000h 0000 4] Signature : "PPTT" [Processor Properties Topology Table] -[004h 0004 4] Table Length : 0000004C +[004h 0004 4] Table Length : 00000060 [008h 0008 1] Revision : 02 -[009h 0009 1] Checksum : A8 +[009h 0009 1] Checksum : 48 [00Ah 0010 6] Oem ID : "BOCHS " [010h 0016 8] Oem Table ID : "BXPC " [018h 0024 4] Oem Revision : 00000001 [01Ch 0028 4] Asl Compiler ID : "BXPC" [020h 0032 4] Asl Compiler Revision : 00000001
[024h 0036 1] Subtable Type : 00 [Processor Hierarchy Node]
[025h 0037 1] Length : 14
[026h 0038 2] Reserved : 0000
[028h 0040 4] Flags (decoded below) : 00000001
Physical package : 1
ACPI Processor ID valid : 0
[02Ch 0044 4] Parent : 00000000
[030h 0048 4] ACPI Processor ID : 00000000
[034h 0052 4] Private Resource Number : 00000000 [038h 0056 1] Subtable Type : 00 [Processor Hierarchy Node]
[039h 0057 1] Length : 14
[03Ah 0058 2] Reserved : 0000
-[03Ch 0060 4] Flags (decoded below) : 0000000A
+[03Ch 0060 4] Flags (decoded below) : 00000000
Physical package : 0
- ACPI Processor ID valid : 1
+ ACPI Processor ID valid : 0
[040h 0064 4] Parent : 00000024
[044h 0068 4] ACPI Processor ID : 00000000
[048h 0072 4] Private Resource Number : 00000000-Raw Table Data: Length 76 (0x4C)
+[04Ch 0076 1] Subtable Type : 00 [Processor Hierarchy Node] +[04Dh 0077 1] Length : 14 +[04Eh 0078 2] Reserved : 0000 +[050h 0080 4] Flags (decoded below) : 0000000A + Physical package : 0 + ACPI Processor ID valid : 1 +[054h 0084 4] Parent : 00000038 +[058h 0088 4] ACPI Processor ID : 00000000 +[05Ch 0092 4] Private Resource Number : 00000000
+
+Raw Table Data: Length 96 (0x60)
- 0000: 50 50 54 54 4C 00 00 00 02 A8 42 4F 43 48 53 20 // PPTTL.....BOCHS
+ 0000: 50 50 54 54 60 00 00 00 02 48 42 4F 43 48 53 20 // PPTT`....HBOCHS
0010: 42 58 50 43 20 20 20 20 01 00 00 00 42 58 50 43 // BXPC ....BXPC
0020: 01 00 00 00 00 14 00 00 01 00 00 00 00 00 00 00 // ................
- 0030: 00 00 00 00 00 00 00 00 00 14 00 00 0A 00 00 00 // ................
- 0040: 24 00 00 00 00 00 00 00 00 00 00 00 // $...........
+ 0030: 00 00 00 00 00 00 00 00 00 14 00 00 00 00 00 00 // ................
+ 0040: 24 00 00 00 00 00 00 00 00 00 00 00 00 14 00 00 // $...............
+ 0050: 0A 00 00 00 38 00 00 00 00 00 00 00 00 00 00 00 // ....8...........--- tests/data/acpi/virt/PPTT | Bin 76 -> 96 bytes tests/qtest/bios-tables-test-allowed-diff.h | 1 - 2 files changed, 1 deletion(-)
tests/data/acpi/virt/PPTT binary +0 -0
tests/qtest/bios-tables-test-allowed-diff.h +0 -1
@@ -1,2 +1 @@/* List of comma-separated changed AML files to ignore */ -"tests/data/acpi/virt/PPTT",
Signature · 1 line
2.27.0
Quoted 5 lines · Yanan Wang
Hi,
This series introduces the new CPU clusters topology parameter and enable the support for it on ARM virt machines.
Quoted 10 lines
Yanan Wang (14):
qemu-options: Improve readability of SMP related Docs hw/core/machine: Introduce CPU cluster topology support hw/core/machine: Wrap target specific parameters together tests/unit/test-smp-parse: Add testcases for CPU clusters tests/unit/test-smp-parse: No need to explicitly zero MachineClass
members
tests/unit/test-smp-parse: Keep default MIN/MAX CPUs in
machine_base_class_init
MAINTAINERS: Self-recommended as reviewer of "Machine core"
Patches 1, 2, 4-7 queued, thanks.
The rest ARM & ACPI part (patches 8-14) have been packed into v6:
v6: https://patchew.org/QEMU/[email protected]/
Thanks,
Yanan
Quoted 104 lines · Yanan Wang
Hi,
This series introduces the new CPU clusters topology parameter and enable the support for it on ARM virt machines.
Background and descriptions:
The new Cluster-Aware Scheduling support has landed in Linux 5.16, which has been proved to benefit the scheduling performance (e.g. load balance andwake_affinestrategy) for bothx86_64and AArch64. We can see the PR [1] or the actual patch series [2] for reference.So since Linux 5.16 we have four-level arch-neutral CPU topology definition like below and a new scheduler level for clusters.
struct cpu_topology { int thread_id; int core_id; int cluster_id; int package_id; int llc_id; cpumask_t thread_sibling; cpumask_t core_sibling; cpumask_t cluster_sibling; cpumask_t llc_sibling; }A cluster generally means a group of CPU cores which share L2 cache or other mid-level resources, and it is the shared resources that is used to improve scheduler's behavior. From the point of view of the size range, it's between CPU die and CPU core. For example, on some ARM64 Kunpeng servers, we have 6 clusters in each NUMA node, and 4 CPU cores in each cluster. The 4 CPU cores share a separate L2 cache and a L3 cache tag, which brings cache affinity advantage.
[1] https://lore.kernel.org/lkml/163572864855.3357115.17938524897008353101.tglx@xen13/
[2] https://lkml.org/lkml/2021/9/24/178In virtualization, on the Hosts which have pClusters, if we can design a vCPU topology with cluster level for guest kernel and have a dedicated vCPU pinning. A Cluster-Aware Guest kernel can also make use of the cache affinity of CPU clusters to gain similar scheduling performance. So this series introduce clusters support in the vCPU topology on ARM virt machines.
The patches are arranged mainly in two parts:
The first part (patch 1-7):
- Implement infrastructure for CPU cluster level topology support, including the SMP documentation, configuration and parsing, adding testcases for clusters.The second part (part 8-14):
- Enable CPU cluster support on ARM virt machines, so that users can specify a 4-level CPU hierarchy sockets/clusters/cores/threads. And the 4-level topology will be described to guest kernel through ACPI PPTT and DT cpu-map.Changelog:
v3->v4:
- Significant change from v3 to v4, since the whole series is reworked based on latest QEMU SMP frame.
- v3: https://patchew.org/QEMU/[email protected]/
v4->v5:
- newly added patches 4-7
- rebased on Philippe series: "tests/unit: Rework test-smp-parse tests"
https://patchew.org/QEMU/[email protected]/
- v4: https://patchew.org/QEMU/[email protected]/Yanan Wang (14):
qemu-options: Improve readability of SMP related Docs hw/core/machine: Introduce CPU cluster topology support hw/core/machine: Wrap target specific parameters together tests/unit/test-smp-parse: Add testcases for CPU clusters tests/unit/test-smp-parse: No need to explicitly zero MachineClass
members
tests/unit/test-smp-parse: Keep default MIN/MAX CPUs in
machine_base_class_init
MAINTAINERS: Self-recommended as reviewer of "Machine core" hw/arm/virt: Support clusters on ARM virt machines hw/arm/virt: Support cluster level in DT cpu-map hw/acpi/aml-build: Improve scalability of PPTT generation hw/arm/virt-acpi-build: Make an ARM specific PPTT generator tests/acpi/bios-tables-test: Allow changes tovirt/PPTTfile hw/arm/virt-acpi-build: Support cluster level in PPTT generation tests/acpi/bios-table-test: Update expectedvirt/PPTTfileMAINTAINERS | 1 + hw/acpi/aml-build.c | 66 +---------------- hw/arm/virt-acpi-build.c | 92 +++++++++++++++++++++++- hw/arm/virt.c | 16 +++-- hw/core/machine-smp.c | 29 ++++++-- hw/core/machine.c | 3 + include/hw/acpi/aml-build.h | 5 +- include/hw/boards.h | 6 +- qapi/machine.json | 5 +- qemu-options.hx | 91 ++++++++++++++++++----- softmmu/vl.c | 3 + tests/data/acpi/virt/PPTT | Bin 76 -> 96 bytes tests/unit/test-smp-parse.c | 140 ++++++++++++++++++++++++++++++------ 13 files changed, 332 insertions(+), 125 deletions(-)--
2.27.0.