Skip to contentSkip to conversation

hw/arm: MPAM Emulation + PPTT cache description.

RFC PATCH In review Jonathan Cameron via 4 participants 11 messages Aug 8, 2023 – Aug 23, 2023 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

cover hw/arm: MPAM Emulation + PPTT cache description.

Aim of this bit of emulation is to use it for testing James Morse's kernel tree - in particularly letting us poke the corner cases. Right now I'm not that focused on upstreaming this (too many other things in my backlog), but any feedback on the approach etc welcome and perhaps the PPTT part is useful independent of MPAM support.

Current kernel branch (one outstanding bug reported but that's hard to hit and requires setting the narrowing target number of IDs to 1 which is bonkers):
https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git/log/?h=mpam/snapshot/v6.5-rc1

Supported:
* PPTT cache description - this is necessary for the cross references MPAM table entries use to establish which Cache any given control set influences. I included option for generating shared tables which were a common choice prior to MPAM needing those cross references.
* CPU emulation for MPAM. Given we aren't doing anything with the content this is just a case of adding the MPAM_IDR register and read/write registers to control current PARTID / PMG group.
* MPAM MSC emulation for caches and memory controllers.
Multiple RIS support allows up to 16 such elements to be controlled via a single interface (used only for memory currently.
Most controls wired up, though introspection interface and sanity checks only cover some of them so far. No monitoring yet.
* ACPI tables and device instantiation in ARM Virt. ACPI only because the kernel patches clearly state the DT binding is a WIP.
* A hack to add lots of caches to the MAX cpu via the relevant CPU registers
- these are read back to generate the PPTT Table and MPAM devices.

TODO:
- Dealing with case of no NUMA nodes. Currently we don't start if NUMA nodes aren't specified and mpam=on. Defaulting to a single NUMA node if MPAM is enabled may make more sense.
- Error injection / reporting on invalid parameters.
- Monitor support.
- Wire up the interrupts properly.
- Tighten checks on unexpected values to further help with catching bugs in kernel code (a few already found and fixed by James).
- ACPI table test (yeah I'm lazy).
- Remove remaining 'fixed' constraints on number of partitions etc so they can be different across controllers / different levels of the hierarchy.
- Expand the qmp introspection interface to cover the missing parts.

Example command line (who doesn't love SMT arm machines?):
aarch64-softmmu/qemu-system-aarch64 -D test.log -d unimp \
-M virt,nvdimm=on,gic-version=3,mpam=on,mpam_min_msc=on \
-m 4g -cpu max,core-count=2 \
-smp 16,sockets=1,clusters=4,threads=2,cache-cluster-start-level=2,cache-node-start-level=3 \
-kernel Image \
-drive if=none,file=full.qcow2,format=qcow2,id=hd \
-device pcie-root-port,id=root_port1 -device virtio-blk-pci,drive=hd \
-qmp-pretty tcp:localhost:4445,server=on,wait=off \
...
-nographic -no-reboot -append 'earlycon root=/dev/vda2' \
-bios QEMU_EFI.fd \
-object memory-backend-ram,size=1G,id=mem0 \
-object memory-backend-ram,size=1G,id=mem1 \
-object memory-backend-ram,size=1G,id=mem2 \
-object memory-backend-ram,size=1G,id=mem3 \
-numa node,nodeid=0,cpus=0-3,memdev=mem0 \
-numa node,nodeid=1,cpus=4-7,memdev=mem1 \
-numa node,nodeid=2,cpus=8-11,memdev=mem2 \
-numa node,nodeid=3,cpus=12-15,memdev=mem3

QMP comamnds:

{ "execute": "qmp_capabilities" }
{ "execute": "query-mpam-cache",

  "arguments": {
    "level": 3

}
}

Will return something like (reformatted as the pretty version is 'long') An 'ideal' version of this interface will take some more thought as it needs to balance readability and clarity with complex implementation of the code to 'interpret' the register values.

{
    "return": [
        {
            "cpu": 0,
            "level": 3,
            "regs": [
                {
                    "mbwumon-idr": 0,
                    "idr": 758514712831,
                    "cfg-cpbm": [
                        { "words": [ 4294967295 ] },
                        { "words": [ 0 ] },
                        { "words": [ 0 ] },
                        { "words": [ 0 ] },	
                        { "words": [ 0 ] },
			....
			{ "words": [ 0 ] }
                    ],
                    "partid-nrw-idr": 31,
                    "mbw-idr": 0,
                    "csumon-idr": 0,
                    "esr": 0,
                    "ecr": 1,
                    "cfg-part-sel": 0,
                    "iidr": 44042038,
                    "cpor-idr": 32,
                    "msmon-idr": 0,
                    "ccap-idr": 2952791044,
                    "aidr": 17,
                    "pri-idr": 35
                }
            ],
            "type": 3
        }
    ]
}

Jonathan Cameron (5):
hw/acpi: Add PPTT cache descriptions
HACK: target/arm/tcg: Add some more caches to cpu=max target/arm: Add support for MPAM CPU registers hw/arm: Add MPAM emulation.
hw/arm/virt: Add MPAM MSCs for memory controllers and caches.

 qapi/machine.json           |   8 +-
 qapi/mpam.json              |  78 ++++
 qapi/qapi-schema.json       |   1 +
 include/hw/acpi/aml-build.h |  19 +-
 include/hw/arm/mpam.h       |  13 +
 include/hw/arm/virt.h       |   2 +
 include/hw/boards.h         |   4 +
 target/arm/cpu.h            |  15 +
 hw/acpi/aml-build.c         | 189 +++++++-
 hw/arm/mpam-qapi-stubs.c    |   9 +
 hw/arm/mpam-qapi.c          |  58 +++
 hw/arm/mpam.c               | 886 ++++++++++++++++++++++++++++++++++++
 hw/arm/virt-acpi-build.c    | 327 ++++++++++++-
 hw/arm/virt.c               | 134 ++++++
 hw/core/machine-smp.c       |   8 +
 hw/loongarch/acpi-build.c   |   2 +-
 target/arm/cpu.c            |  10 +-
 target/arm/helper.c         |  30 ++
 target/arm/tcg/cpu64.c      |  12 +
 hw/arm/Kconfig              |   4 +
 hw/arm/meson.build          |   4 +
 qapi/meson.build            |   1 +
 22 files changed, 1803 insertions(+), 11 deletions(-)
 create mode 100644 qapi/mpam.json
 create mode 100644 include/hw/arm/mpam.h
 create mode 100644 hw/arm/mpam-qapi-stubs.c
 create mode 100644 hw/arm/mpam-qapi.c
 create mode 100644 hw/arm/mpam.c
Signature · 1 line
2.39.2
#1 lore raw

RFC PATCH 1/5 hw/acpi: Add PPTT cache descriptions

Current PPTT tables generated by QEMU only provide information on CPU topology and neglect the description of Caches.

This patch adds flexible definition of those caches and updates the table version to 3 to allow for the per CPU cache instance IDs needed for cross references from the MPAM table.

If MPAM is not being used, then a unified description can be used, greatly reducing the resulting table size.

New machine parameters are used to control the cache toplogy.
cache-cluster-start-level: Which caches are associated with the cluster
level of the topology. e.g cache-cluster-start-level=2 results in shared l2 cache across a cluster.
cache-numa-start-level: Which caches are associate with the NUMA (in qemu
this is currently the physical package level). For example cache-cluster-start-level=2,cache-numa-start-level=3 gives private l1, cluster shared l2 and package shared L3.

FIXME: Test updates.

---
 qapi/machine.json           |   8 +-
 include/hw/acpi/aml-build.h |  19 +++-
 include/hw/boards.h         |   4 +
 hw/acpi/aml-build.c         | 189 ++++++++++++++++++++++++++++++++++--
 hw/arm/virt-acpi-build.c    | 130 ++++++++++++++++++++++++-
 hw/core/machine-smp.c       |   8 ++
 hw/loongarch/acpi-build.c   |   2 +-
 7 files changed, 350 insertions(+), 10 deletions(-)
qapi/machine.json +7 -1
index a08b6576ca..cc86784641 100644
@@ -1494,6 +1494,10 @@
 # @maxcpus: maximum number of hotpluggable virtual CPUs in the virtual
 #     machine
 #
+# @cache-cluster-start-level: Level of first cache attached to cluster
+#
+# @cache-node-start-level: Level of first cache attached to cluster
+#
 # Since: 6.1
 ##
 { 'struct': 'SMPConfiguration', 'data': {
@@ -1503,7 +1507,9 @@
      '*clusters': 'int',
      '*cores': 'int',
      '*threads': 'int',
-     '*maxcpus': 'int' } }
+     '*maxcpus': 'int',
+     '*cache-cluster-start-level': 'int',
+     '*cache-node-start-level': 'int'} }
 
 ##
 # @x-query-irq:
include/hw/acpi/aml-build.h +18 -1
index d1fb08514b..055b74820d 100644
@@ -489,8 +489,25 @@ void build_srat_memory(GArray *table_data, uint64_t base,
 void build_slit(GArray *table_data, BIOSLinker *linker, MachineState *ms,
                 const char *oem_id, const char *oem_table_id);
 
+typedef enum ACPIPPTTCacheType {
+    DATA,
+    INSTRUCTION,
+    UNIFIED,
+} ACPIPPTTCacheType;
+
+typedef struct ACPIPPTTCache {
+    ACPIPPTTCacheType type;
+    int sets;
+    int size;
+    int associativity;
+    int linesize;
+    unsigned int pptt_id;
+    int level;
+} ACPIPPTTCache;
+
 void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms,
-                const char *oem_id, const char *oem_table_id);
+                const char *oem_id, const char *oem_table_id,
+                int num_caches, ACPIPPTTCache *caches);
 
 void build_fadt(GArray *tbl, BIOSLinker *linker, const AcpiFadtData *f,
                 const char *oem_id, const char *oem_table_id);
include/hw/boards.h +4 -0
index ed83360198..6e8ab92684 100644
@@ -316,6 +316,8 @@ typedef struct DeviceMemoryState {
  * @cores: the number of cores in one cluster
  * @threads: the number of threads in one core
  * @max_cpus: the maximum number of logical processors on the machine
+ * @cache_cluster_start_level: First cache level attached to cluster
+ * @cache_node_start_level: First cache level attached to node
  */
 typedef struct CpuTopology {
     unsigned int cpus;
@@ -325,6 +327,8 @@ typedef struct CpuTopology {
     unsigned int cores;
     unsigned int threads;
     unsigned int max_cpus;
+    unsigned int cache_cluster_start_level;
+    unsigned int cache_node_start_level;
 } CpuTopology;
 
 /**
hw/acpi/aml-build.c +183 -6

251 lines not on this page · raw message

hw/arm/virt-acpi-build.c +129 -1

151 lines not on this page · raw message

hw/core/machine-smp.c +8 -0
index 0f4d9b6f7a..cbb0bf1bc7 100644
@@ -81,6 +81,10 @@ void machine_parse_smp_config(MachineState *ms,
     unsigned cores   = config->has_cores ? config->cores : 0;
     unsigned threads = config->has_threads ? config->threads : 0;
     unsigned maxcpus = config->has_maxcpus ? config->maxcpus : 0;
+    unsigned cache_cl_start = config->has_cache_cluster_start_level ?
+        config->cache_cluster_start_level : 0;
+    unsigned cache_nd_start = config->has_cache_node_start_level ?
+        config->cache_node_start_level : 0;
 
     /*
      * Specified CPU topology parameters must be greater than zero,
@@ -161,6 +165,10 @@ void machine_parse_smp_config(MachineState *ms,
     ms->smp.max_cpus = maxcpus;
 
     mc->smp_props.has_clusters = config->has_clusters;
+    if (mc->smp_props.has_clusters) {
+        ms->smp.cache_cluster_start_level = cache_cl_start;
+        ms->smp.cache_node_start_level = cache_nd_start;
+    }
 
     /* sanity-check of the computed topology */
     if (sockets * dies * clusters * cores * threads != maxcpus) {
hw/loongarch/acpi-build.c +1 -1
index 0b62c3a2f7..51d4ed9a19 100644
@@ -439,7 +439,7 @@ static void acpi_build(AcpiBuildTables *tables, MachineState *machine)
 
     acpi_add_table(table_offsets, tables_blob);
     build_pptt(tables_blob, tables->linker, machine,
-               lams->oem_id, lams->oem_table_id);
+               lams->oem_id, lams->oem_table_id, 0, NULL);
 
     acpi_add_table(table_offsets, tables_blob);
     build_srat(tables_blob, tables->linker, machine);
Signature · 1 line
2.39.2
#2 lore raw

Re: [RFC PATCH 1/5] hw/acpi: Add PPTT cache descriptions

Hi Jonathan,

Quoted 12 lines · Jonathan Cameron via

Date: Tue, 8 Aug 2023 12:57:09 +0100
From: Jonathan Cameron via <[email protected]>
Subject: [RFC PATCH 1/5] hw/acpi: Add PPTT cache descriptions
X-Mailer: git-send-email 2.39.2

Current PPTT tables generated by QEMU only provide information on CPU topology and neglect the description of Caches.

This patch adds flexible definition of those caches and updates the table version to 3 to allow for the per CPU cache instance IDs needed for cross references from the MPAM table.

Quoted 7 lines

If MPAM is not being used, then a unified description can be used, greatly reducing the resulting table size.

New machine parameters are used to control the cache toplogy.
cache-cluster-start-level: Which caches are associated with the cluster
level of the topology. e.g cache-cluster-start-level=2 results in shared l2 cache across a cluster.

So the i/d cache are at core level by default and we don't need to configure its topology, right?

Quoted 2 lines

cache-numa-start-level: Which caches are associate with the NUMA (in qemu
this is currently the physical package level).

I'm a bit confused about the connection of this numa and l3. Does there "NUMA" refer to socket level?

Quoted 3 lines

For example
cache-cluster-start-level=2,cache-numa-start-level=3 gives private l1, cluster shared l2 and package shared L3.

Okay, you list the topology as: l1 per core, l2 per cluster and l3 per socket.

For this case, I think my QOM topology proposal [1] (this is the underlying general topology implementation, compatible with symmetric and heterogeneous, and I'm working on this QOM topology as a superset of smp) is compatible with your command.

And I understand the difference between my "x-l2-cache-topo=[core|cluster]" for x86 and yours is that I named the l2 cache, while you took level count as the parameter.

What if I extend my symmetric cache topology commands for i386 as "l2-cache=cluster,l3-cache=socket (*)"?

Compared to cache-cluster-start-level=2,cache-numa-start-level=3, are there some specific cases that cache-xxx-start-level can solves but (*) command cannot?

[1]: https://mail.gnu.org/archive/html/qemu-devel/2023-02/msg05167.html

Quoted 19 lines

FIXME: Test updates.

---
 qapi/machine.json           |   8 +-
 include/hw/acpi/aml-build.h |  19 +++-
 include/hw/boards.h         |   4 +
 hw/acpi/aml-build.c         | 189 ++++++++++++++++++++++++++++++++++--
 hw/arm/virt-acpi-build.c    | 130 ++++++++++++++++++++++++-
 hw/core/machine-smp.c       |   8 ++
 hw/loongarch/acpi-build.c   |   2 +-
 7 files changed, 350 insertions(+), 10 deletions(-)
qapi/machine.json +0 -0
index a08b6576ca..cc86784641 100644

qapi/machine.json line 1499

Quoted 6 lines
qapi/machine.json +3 -0
@@ -1494,6 +1494,10 @@
 # @maxcpus: maximum number of hotpluggable virtual CPUs in the virtual
 #     machine
 #
+# @cache-cluster-start-level: Level of first cache attached to cluster
+#
+# @cache-node-start-level: Level of first cache attached to cluster

node or numa?

Thanks,
Zhao

Quoted 526 lines
qapi/machine.json +4 -1
+#
 # Since: 6.1
 ##
 { 'struct': 'SMPConfiguration', 'data': {
@@ -1503,7 +1507,9 @@
      '*clusters': 'int',
      '*cores': 'int',
      '*threads': 'int',
-     '*maxcpus': 'int' } }
+     '*maxcpus': 'int',
+     '*cache-cluster-start-level': 'int',
+     '*cache-node-start-level': 'int'} }
 
 ##
 # @x-query-irq:
include/hw/acpi/aml-build.h +18 -1
index d1fb08514b..055b74820d 100644
@@ -489,8 +489,25 @@ void build_srat_memory(GArray *table_data, uint64_t base,
 void build_slit(GArray *table_data, BIOSLinker *linker, MachineState *ms,
                 const char *oem_id, const char *oem_table_id);
 
+typedef enum ACPIPPTTCacheType {
+    DATA,
+    INSTRUCTION,
+    UNIFIED,
+} ACPIPPTTCacheType;
+
+typedef struct ACPIPPTTCache {
+    ACPIPPTTCacheType type;
+    int sets;
+    int size;
+    int associativity;
+    int linesize;
+    unsigned int pptt_id;
+    int level;
+} ACPIPPTTCache;
+
 void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms,
-                const char *oem_id, const char *oem_table_id);
+                const char *oem_id, const char *oem_table_id,
+                int num_caches, ACPIPPTTCache *caches);
 
 void build_fadt(GArray *tbl, BIOSLinker *linker, const AcpiFadtData *f,
                 const char *oem_id, const char *oem_table_id);
include/hw/boards.h +4 -0
index ed83360198..6e8ab92684 100644
@@ -316,6 +316,8 @@ typedef struct DeviceMemoryState {
  * @cores: the number of cores in one cluster
  * @threads: the number of threads in one core
  * @max_cpus: the maximum number of logical processors on the machine
+ * @cache_cluster_start_level: First cache level attached to cluster
+ * @cache_node_start_level: First cache level attached to node
  */
 typedef struct CpuTopology {
     unsigned int cpus;
@@ -325,6 +327,8 @@ typedef struct CpuTopology {
     unsigned int cores;
     unsigned int threads;
     unsigned int max_cpus;
+    unsigned int cache_cluster_start_level;
+    unsigned int cache_node_start_level;
 } CpuTopology;
 
 /**
hw/acpi/aml-build.c +183 -6
index ea331a20d1..e103cd638f 100644
@@ -1994,32 +1994,175 @@ static void build_processor_hierarchy_node(GArray *tbl, uint32_t flags,
     }
 }
 
+static void build_cache_nodes(GArray *tbl, ACPIPPTTCache *cache,
+                              uint32_t next_offset,
+                              bool has_id, unsigned int id)
+{
+    int val;
+
+    /* Type 1 - cache */
+    build_append_byte(tbl, 1);
+    /* Length */
+    build_append_byte(tbl, 28);
+    /* Reserved */
+    build_append_int_noprefix(tbl, 0, 2);
+    /* Flags - everything except possibly the ID */
+    build_append_int_noprefix(tbl, has_id ? 0xff : 0x7f, 4);
+    /* Offset of next cache up */
+    build_append_int_noprefix(tbl, next_offset, 4);
+    build_append_int_noprefix(tbl, cache->size, 4);
+    build_append_int_noprefix(tbl, cache->sets, 4);
+    build_append_byte(tbl, cache->associativity);
+    /* Read and Write allocate amd WB */
+    val = 0x3 | (1 << 4);
+    switch (cache->type) {
+    case INSTRUCTION:
+        val |= (1 << 2);
+        break;
+    case DATA:
+        val |= (0 << 2); /* Data */
+        break;
+    case UNIFIED:
+        val |= (3 << 2); /* Unified */
+        break;
+    }
+    build_append_byte(tbl, val);
+    build_append_int_noprefix(tbl, cache->linesize, 2);
+    build_append_int_noprefix(tbl,
+                              has_id ?
+                              (cache->type << 24) | (cache->level << 16) | id :
+                              0, 4);
+}
+
+static void build_caches_subset(GArray *table_data, uint32_t pptt_start,
+                                int num_caches, ACPIPPTTCache *caches,
+                                bool assign_ids, int base_id,
+                                uint8_t level_high, uint8_t level_low,
+                                uint32_t *data_offset, uint32_t *instr_offset)
+{
+    uint32_t next_level_offset_data = 0, next_level_offset_instruction = 0;
+    uint32_t this_offset, next_offset = 0;
+    int c, l;
+
+    /* Walk caches from top to bottom */
+
+    for (l = level_high; l >= level_low; l--) { /* Walk down levels */
+        for (c = 0; c < num_caches; c++) {
+            if (caches[c].level != l) {
+                continue;
+            }
+
+            /* Assume only unified above l1 for now */
+            this_offset = table_data->len - pptt_start;
+            switch (caches[c].type) {
+            case INSTRUCTION:
+                next_offset = next_level_offset_instruction;
+                break;
+            case DATA:
+                next_offset = next_level_offset_data;
+                break;
+            case UNIFIED:
+                /* Either is fine here - hopefully */
+                next_offset = next_level_offset_instruction;
+                break;
+            }
+            build_cache_nodes(table_data, &caches[c], next_offset,
+                              assign_ids, base_id);
+            switch (caches[c].type) {
+            case INSTRUCTION:
+                next_level_offset_instruction = this_offset;
+                break;
+            case DATA:
+                next_level_offset_data = this_offset;
+                break;
+            case UNIFIED:
+                next_level_offset_instruction = this_offset;
+                next_level_offset_data = this_offset;
+                break;
+            }
+            *data_offset = next_level_offset_data;
+            *instr_offset = next_level_offset_instruction;
+        }
+    }
+}
+
 /*
  * ACPI spec, Revision 6.3
  * 5.2.29 Processor Properties Topology Table (PPTT)
  */
 void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms,
-                const char *oem_id, const char *oem_table_id)
+                const char *oem_id, const char *oem_table_id,
+                int num_caches, ACPIPPTTCache *caches)
 {
+    bool share_structs = false;
     MachineClass *mc = MACHINE_GET_CLASS(ms);
     CPUArchIdList *cpus = ms->possible_cpus;
     int64_t socket_id = -1, cluster_id = -1, core_id = -1;
     uint32_t socket_offset = 0, cluster_offset = 0, core_offset = 0;
     uint32_t pptt_start = table_data->len;
     int n;
-    AcpiTable table = { .sig = "PPTT", .rev = 2,
+    AcpiTable table = { .sig = "PPTT", .rev = 3,
                         .oem_id = oem_id, .oem_table_id = oem_table_id };
+    uint32_t l1_data_offset = 0;
+    uint32_t l1_instr_offset = 0;
+    uint32_t cluster_data_offset = 0;
+    uint32_t cluster_instr_offset = 0;
+    uint32_t node_data_offset = 0;
+    uint32_t node_instr_offset = 0;
+    int top_node = 7;
+    int top_cluster = 7;
+    int top_core = 7;
 
     acpi_table_begin(&table, table_data);
 
+    /* Let us have a unified cache description for now */
+
+    if (share_structs && num_caches >= 1) {
+        if (ms->smp.cache_node_start_level) {
+            build_caches_subset(table_data, pptt_start, num_caches, caches,
+                                false, 0,
+                                top_node, ms->smp.cache_node_start_level,
+                                &node_data_offset, &node_instr_offset);
+            top_cluster = ms->smp.cache_node_start_level - 1;
+        }
+        /* Assumption that some caches below this */
+        if (ms->smp.cache_cluster_start_level) {
+            build_caches_subset(table_data, pptt_start, num_caches, caches,
+                                false, 0,
+                                top_cluster,  ms->smp.cache_cluster_start_level,
+                                &cluster_data_offset, &cluster_instr_offset);
+            top_core = ms->smp.cache_cluster_start_level - 1;
+        }
+        build_caches_subset(table_data, pptt_start, num_caches, caches,
+                            false, 0,
+                            top_core , 0,
+                            &l1_data_offset, &l1_instr_offset);
+    }
+
     /*
      * This works with the assumption that cpus[n].props.*_id has been
      * sorted from top to down levels in mc->possible_cpu_arch_ids().
      * Otherwise, the unexpected and duplicated containers will be
      * created.
      */
+
     for (n = 0; n < cpus->len; n++) {
         if (cpus->cpus[n].props.socket_id != socket_id) {
+            uint32_t priv_rsrc[2];
+            int num_priv = 0;
+
+            if (!share_structs && ms->smp.cache_node_start_level) {
+                build_caches_subset(table_data, pptt_start, num_caches, caches,
+                                    true, n,
+                                    top_node, ms->smp.cache_node_start_level,
+                                    &node_data_offset, &node_instr_offset);
+                top_cluster = ms->smp.cache_node_start_level - 1;
+            }
+            priv_rsrc[0] = node_instr_offset;
+            priv_rsrc[1] = node_data_offset;
+            if (node_instr_offset || node_data_offset) {
+                num_priv = node_instr_offset == node_data_offset ? 1 : 2;
+            }
             assert(cpus->cpus[n].props.socket_id > socket_id);
             socket_id = cpus->cpus[n].props.socket_id;
             cluster_id = -1;
@@ -2027,36 +2170,70 @@ void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms,
             socket_offset = table_data->len - pptt_start;
             build_processor_hierarchy_node(table_data,
                 (1 << 0), /* Physical package */
-                0, socket_id, NULL, 0);
+                0, socket_id, priv_rsrc, num_priv);
         }
 
+
         if (mc->smp_props.clusters_supported && mc->smp_props.has_clusters) {
             if (cpus->cpus[n].props.cluster_id != cluster_id) {
+                uint32_t priv_rsrc[2];
+                int num_priv = 0;
+
+                if (!share_structs && ms->smp.cache_cluster_start_level) {
+                    build_caches_subset(table_data, pptt_start, num_caches,
+                                        caches, true, n,
+                                        top_cluster,
+                                        ms->smp.cache_cluster_start_level,
+                                        &cluster_data_offset,
+                                        &cluster_instr_offset);
+                    top_core = ms->smp.cache_cluster_start_level - 1;
+                }
+                priv_rsrc[0] = cluster_instr_offset;
+                priv_rsrc[1] = cluster_data_offset;
+
                 assert(cpus->cpus[n].props.cluster_id > cluster_id);
                 cluster_id = cpus->cpus[n].props.cluster_id;
                 core_id = -1;
                 cluster_offset = table_data->len - pptt_start;
+
+                if (cluster_instr_offset || cluster_data_offset) {
+                    num_priv = cluster_instr_offset == cluster_data_offset ?
+                        1 : 2;
+                }
                 build_processor_hierarchy_node(table_data,
                     (0 << 0), /* Not a physical package */
-                    socket_offset, cluster_id, NULL, 0);
+                    socket_offset, cluster_id, priv_rsrc, num_priv);
             }
         } else {
             cluster_offset = socket_offset;
         }
 
+        if (!share_structs &&
+            cpus->cpus[n].props.core_id != core_id) {
+            build_caches_subset(table_data, pptt_start, num_caches, caches,
+                                true, n,
+                                top_core , 0,
+                                &l1_data_offset, &l1_instr_offset);
+        }
         if (ms->smp.threads == 1) {
+            uint32_t priv_rsrc[2] = { l1_instr_offset, l1_data_offset };
+
             build_processor_hierarchy_node(table_data,
                 (1 << 1) | /* ACPI Processor ID valid */
                 (1 << 3),  /* Node is a Leaf */
-                cluster_offset, n, NULL, 0);
+                cluster_offset, n, priv_rsrc,
+                l1_instr_offset == l1_data_offset ? 1 : 2);
         } else {
             if (cpus->cpus[n].props.core_id != core_id) {
+                uint32_t priv_rsrc[2] = { l1_instr_offset, l1_data_offset };
+
                 assert(cpus->cpus[n].props.core_id > core_id);
                 core_id = cpus->cpus[n].props.core_id;
                 core_offset = table_data->len - pptt_start;
                 build_processor_hierarchy_node(table_data,
                     (0 << 0), /* Not a physical package */
-                    cluster_offset, core_id, NULL, 0);
+                    cluster_offset, core_id, priv_rsrc,
+                    l1_instr_offset == l1_data_offset ? 1 : 2);
             }
 
             build_processor_hierarchy_node(table_data,
hw/arm/virt-acpi-build.c +129 -1
index 6b674231c2..ec8fdcefff 100644
@@ -922,6 +922,129 @@ static void acpi_align_size(GArray *blob, unsigned align)
     g_array_set_size(blob, ROUND_UP(acpi_data_len(blob), align));
 }
 
+static unsigned int virt_get_caches(VirtMachineState *vms,
+                                    ACPIPPTTCache *caches)
+{
+    ARMCPU *armcpu = ARM_CPU(qemu_get_cpu(0));
+    bool ccidx = cpu_isar_feature(any_ccidx, armcpu);
+    unsigned int num_cache, i;
+    int level_instr = 1, level_data = 1;
+
+    for (i = 0, num_cache = 0; i < 7; i++, num_cache++) {
+        int type = (armcpu->clidr >> (3 * i)) & 7;
+        int bank_index;
+        int level = 0;
+        ACPIPPTTCacheType cache_type = INSTRUCTION;
+
+        if (type == 0) {
+            break;
+        }
+
+        switch (type) {
+        case 1:
+            cache_type = INSTRUCTION;
+            level = level_instr;
+            break;
+        case 2:
+            cache_type = DATA;
+            level = level_data;
+            break;
+        case 4:
+            cache_type = UNIFIED;
+            level = level_instr > level_data ? level_instr : level_data;
+            break;
+        case 3: /* Split - Do data first */
+            cache_type = DATA;
+            level = level_data;
+            break;
+        }
+        /*
+         * ccsidr is indexed using both the level and whether it is
+         * an instruction cache. Unified caches use the same storage
+         * as data caches.
+         */
+        bank_index = (i * 2) | ((type == 1) ? 1 : 0);
+        if (ccidx) {
+            caches[num_cache] = (ACPIPPTTCache) {
+                .type =  cache_type,
+                .level = level,
+                .linesize = 1 << (FIELD_EX64(armcpu->ccsidr[bank_index],
+                                             CCSIDR_EL1,
+                                             CCIDX_LINESIZE) + 4),
+                .associativity = FIELD_EX64(armcpu->ccsidr[bank_index],
+                                            CCSIDR_EL1,
+                                            CCIDX_ASSOCIATIVITY) + 1,
+                .sets = FIELD_EX64(armcpu->ccsidr[bank_index], CCSIDR_EL1,
+                                   CCIDX_NUMSETS) + 1,
+            };
+        } else {
+            caches[num_cache] = (ACPIPPTTCache) {
+                .type =  cache_type,
+                .level = level,
+                .linesize = 1 << (FIELD_EX64(armcpu->ccsidr[bank_index],
+                                             CCSIDR_EL1, LINESIZE) + 4),
+                .associativity = FIELD_EX64(armcpu->ccsidr[bank_index],
+                                            CCSIDR_EL1,
+                                            ASSOCIATIVITY) + 1,
+                .sets = FIELD_EX64(armcpu->ccsidr[bank_index], CCSIDR_EL1,
+                                   NUMSETS) + 1,
+            };
+        }
+        caches[num_cache].size = caches[num_cache].associativity *
+            caches[num_cache].sets * caches[num_cache].linesize;
+
+        /* Break one 'split' entry up into two records */
+        if (type == 3) {
+            num_cache++;
+            bank_index = (i * 2) | 1;
+            if (ccidx) {
+                /* Instruction cache: bottom bit set when reading banked reg */
+                caches[num_cache] = (ACPIPPTTCache) {
+                    .type = INSTRUCTION,
+                    .level = level_instr,
+                    .linesize = 1 << (FIELD_EX64(armcpu->ccsidr[bank_index],
+                                                 CCSIDR_EL1,
+                                                 CCIDX_LINESIZE) + 4),
+                    .associativity = FIELD_EX64(armcpu->ccsidr[bank_index],
+                                                CCSIDR_EL1,
+                                                CCIDX_ASSOCIATIVITY) + 1,
+                    .sets = FIELD_EX64(armcpu->ccsidr[bank_index], CCSIDR_EL1,
+                                       CCIDX_NUMSETS) + 1,
+                };
+            } else {
+                caches[num_cache] = (ACPIPPTTCache) {
+                    .type = INSTRUCTION,
+                    .level = level_instr,
+                    .linesize = 1 << (FIELD_EX64(armcpu->ccsidr[bank_index],
+                                                 CCSIDR_EL1, LINESIZE) + 4),
+                    .associativity = FIELD_EX64(armcpu->ccsidr[bank_index],
+                                                CCSIDR_EL1,
+                                                ASSOCIATIVITY) + 1,
+                    .sets = FIELD_EX64(armcpu->ccsidr[bank_index], CCSIDR_EL1,
+                                       NUMSETS) + 1,
+                };
+            }
+            caches[num_cache].size = caches[num_cache].associativity *
+                caches[num_cache].sets * caches[num_cache].linesize;
+        }
+        switch (type) {
+        case 1:
+            level_instr++;
+            break;
+        case 2:
+            level_data++;
+            break;
+        case 3:
+        case 4:
+            level_instr++;
+            level_data++;
+            break;
+        }
+    }
+
+    return num_cache;
+}
+
 static
 void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
 {
@@ -930,6 +1053,8 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
     unsigned dsdt, xsdt;
     GArray *tables_blob = tables->table_data;
     MachineState *ms = MACHINE(vms);
+    ACPIPPTTCache caches[16]; /* Can select up to 16 */
+    unsigned int num_cache;
 
     table_offsets = g_array_new(false, true /* clear */,
                                         sizeof(uint32_t));
@@ -949,10 +1074,13 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
     acpi_add_table(table_offsets, tables_blob);
     build_madt(tables_blob, tables->linker, vms);
 
+    num_cache = virt_get_caches(vms, caches);
+
     if (!vmc->no_cpu_topology) {
         acpi_add_table(table_offsets, tables_blob);
         build_pptt(tables_blob, tables->linker, ms,
-                   vms->oem_id, vms->oem_table_id);
+                   vms->oem_id, vms->oem_table_id,
+                   num_cache, caches);
     }
 
     acpi_add_table(table_offsets, tables_blob);
hw/core/machine-smp.c +8 -0
index 0f4d9b6f7a..cbb0bf1bc7 100644
@@ -81,6 +81,10 @@ void machine_parse_smp_config(MachineState *ms,
     unsigned cores   = config->has_cores ? config->cores : 0;
     unsigned threads = config->has_threads ? config->threads : 0;
     unsigned maxcpus = config->has_maxcpus ? config->maxcpus : 0;
+    unsigned cache_cl_start = config->has_cache_cluster_start_level ?
+        config->cache_cluster_start_level : 0;
+    unsigned cache_nd_start = config->has_cache_node_start_level ?
+        config->cache_node_start_level : 0;
 
     /*
      * Specified CPU topology parameters must be greater than zero,
@@ -161,6 +165,10 @@ void machine_parse_smp_config(MachineState *ms,
     ms->smp.max_cpus = maxcpus;
 
     mc->smp_props.has_clusters = config->has_clusters;
+    if (mc->smp_props.has_clusters) {
+        ms->smp.cache_cluster_start_level = cache_cl_start;
+        ms->smp.cache_node_start_level = cache_nd_start;
+    }
 
     /* sanity-check of the computed topology */
     if (sockets * dies * clusters * cores * threads != maxcpus) {
hw/loongarch/acpi-build.c +1 -1
index 0b62c3a2f7..51d4ed9a19 100644
@@ -439,7 +439,7 @@ static void acpi_build(AcpiBuildTables *tables, MachineState *machine)
 
     acpi_add_table(table_offsets, tables_blob);
     build_pptt(tables_blob, tables->linker, machine,
-               lams->oem_id, lams->oem_table_id);
+               lams->oem_id, lams->oem_table_id, 0, NULL);
 
     acpi_add_table(table_offsets, tables_blob);
     build_srat(tables_blob, tables->linker, machine);
Signature · 1 line
2.39.2
#3 lore raw

Re: [RFC PATCH 1/5] hw/acpi: Add PPTT cache descriptions

Quoted 18 lines · On Mon, 14 Aug 2023 17:50:58 +0800 Zhao Liu <[email protected]> wrote:

Hi Jonathan,

Quoted 15 lines · Jonathan Cameron via

Date: Tue, 8 Aug 2023 12:57:09 +0100
From: Jonathan Cameron via <[email protected]>
Subject: [RFC PATCH 1/5] hw/acpi: Add PPTT cache descriptions
X-Mailer: git-send-email 2.39.2

Current PPTT tables generated by QEMU only provide information on CPU topology and neglect the description of Caches.

This patch adds flexible definition of those caches and updates the table version to 3 to allow for the per CPU cache instance IDs needed for cross references from the MPAM table.

If MPAM is not being used, then a unified description can be used, greatly reducing the resulting table size.

Quoted 7 lines
Quoted 4 lines

New machine parameters are used to control the cache toplogy.
cache-cluster-start-level: Which caches are associated with the cluster
level of the topology. e.g cache-cluster-start-level=2 results in shared l2 cache across a cluster.

So the i/d cache are at core level by default and we don't need to configure its topology, right?

Exactly. Default is everything private. Everything below cache-cluster-start-level remains so.

Quoted 6 lines
Quoted 3 lines

cache-numa-start-level: Which caches are associate with the NUMA (in qemu
this is currently the physical package level).

I'm a bit confused about the connection of this numa and l3. Does there "NUMA" refer to socket level?

Fair point. We don't really have enough flexibility in QEMU so far to represent all the complexities seen in fairly standard systems. Can take one I have to hand

l1i, l1d, l2 private caches.
l3 tags shared by cluster (that just looks like a processor container in PPTT with no caches associated with it)
l3 shared by die
Numa domain per die,
2 Numa domains (dies) per Socket.
N sockets.

So far we are a level short in QEMU representation.

Quoted 5 lines
Quoted 4 lines

For example
cache-cluster-start-level=2,cache-numa-start-level=3 gives private l1, cluster shared l2 and package shared L3.

Quoted 7 lines

Okay, you list the topology as: l1 per core, l2 per cluster and l3 per socket.

For this case, I think my QOM topology proposal [1] (this is the underlying general topology implementation, compatible with symmetric and heterogeneous, and I'm working on this QOM topology as a superset of smp) is compatible with your command.

I've been on holiday and it will take a few more days to catch up. After that I plan to take a close look at your proposal.

Quoted 7 lines

And I understand the difference between my "x-l2-cache-topo=[core|cluster]" for x86 and yours is that I named the l2 cache, while you took level count as the parameter.

What if I extend my symmetric cache topology commands for i386 as "l2-cache=cluster,l3-cache=socket (*)"?

That would work fine for me.

Quoted 6 lines

Compared to cache-cluster-start-level=2,cache-numa-start-level=3, are there some specific cases that cache-xxx-start-level can solves but (*) command cannot?

[1]: https://mail.gnu.org/archive/html/qemu-devel/2023-02/msg05167.html

None, though we'd probably want some sanity checking as l2-cache=socket,l3-cache=cluster isn't representable in PPTT. Arguably I should have that for my start-level based approach as well.

Quoted 15 lines
Quoted 15 lines

FIXME: Test updates.

---
 qapi/machine.json           |   8 +-
 include/hw/acpi/aml-build.h |  19 +++-
 include/hw/boards.h         |   4 +
 hw/acpi/aml-build.c         | 189 ++++++++++++++++++++++++++++++++++--
 hw/arm/virt-acpi-build.c    | 130 ++++++++++++++++++++++++-
 hw/core/machine-smp.c       |   8 ++
 hw/loongarch/acpi-build.c   |   2 +-
 7 files changed, 350 insertions(+), 10 deletions(-)
Quoted 13 lines

qapi/machine.json line 1499

Quoted 11 lines
qapi/machine.json +3 -0
index a08b6576ca..cc86784641 100644
@@ -1494,6 +1494,10 @@
 # @maxcpus: maximum number of hotpluggable virtual CPUs in the virtual
 #     machine
 #
+# @cache-cluster-start-level: Level of first cache attached to cluster
+#
+# @cache-node-start-level: Level of first cache attached to cluster  

node or numa?

oops - should be consistent on that. Thanks!

Jonathan

Quoted 531 lines

Thanks,
Zhao

Quoted 526 lines
qapi/machine.json +4 -1
+#
 # Since: 6.1
 ##
 { 'struct': 'SMPConfiguration', 'data': {
@@ -1503,7 +1507,9 @@
      '*clusters': 'int',
      '*cores': 'int',
      '*threads': 'int',
-     '*maxcpus': 'int' } }
+     '*maxcpus': 'int',
+     '*cache-cluster-start-level': 'int',
+     '*cache-node-start-level': 'int'} }
 
 ##
 # @x-query-irq:
include/hw/acpi/aml-build.h +18 -1
index d1fb08514b..055b74820d 100644
@@ -489,8 +489,25 @@ void build_srat_memory(GArray *table_data, uint64_t base,
 void build_slit(GArray *table_data, BIOSLinker *linker, MachineState *ms,
                 const char *oem_id, const char *oem_table_id);
 
+typedef enum ACPIPPTTCacheType {
+    DATA,
+    INSTRUCTION,
+    UNIFIED,
+} ACPIPPTTCacheType;
+
+typedef struct ACPIPPTTCache {
+    ACPIPPTTCacheType type;
+    int sets;
+    int size;
+    int associativity;
+    int linesize;
+    unsigned int pptt_id;
+    int level;
+} ACPIPPTTCache;
+
 void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms,
-                const char *oem_id, const char *oem_table_id);
+                const char *oem_id, const char *oem_table_id,
+                int num_caches, ACPIPPTTCache *caches);
 
 void build_fadt(GArray *tbl, BIOSLinker *linker, const AcpiFadtData *f,
                 const char *oem_id, const char *oem_table_id);
include/hw/boards.h +4 -0
index ed83360198..6e8ab92684 100644
@@ -316,6 +316,8 @@ typedef struct DeviceMemoryState {
  * @cores: the number of cores in one cluster
  * @threads: the number of threads in one core
  * @max_cpus: the maximum number of logical processors on the machine
+ * @cache_cluster_start_level: First cache level attached to cluster
+ * @cache_node_start_level: First cache level attached to node
  */
 typedef struct CpuTopology {
     unsigned int cpus;
@@ -325,6 +327,8 @@ typedef struct CpuTopology {
     unsigned int cores;
     unsigned int threads;
     unsigned int max_cpus;
+    unsigned int cache_cluster_start_level;
+    unsigned int cache_node_start_level;
 } CpuTopology;
 
 /**
hw/acpi/aml-build.c +183 -6
index ea331a20d1..e103cd638f 100644
@@ -1994,32 +1994,175 @@ static void build_processor_hierarchy_node(GArray *tbl, uint32_t flags,
     }
 }
 
+static void build_cache_nodes(GArray *tbl, ACPIPPTTCache *cache,
+                              uint32_t next_offset,
+                              bool has_id, unsigned int id)
+{
+    int val;
+
+    /* Type 1 - cache */
+    build_append_byte(tbl, 1);
+    /* Length */
+    build_append_byte(tbl, 28);
+    /* Reserved */
+    build_append_int_noprefix(tbl, 0, 2);
+    /* Flags - everything except possibly the ID */
+    build_append_int_noprefix(tbl, has_id ? 0xff : 0x7f, 4);
+    /* Offset of next cache up */
+    build_append_int_noprefix(tbl, next_offset, 4);
+    build_append_int_noprefix(tbl, cache->size, 4);
+    build_append_int_noprefix(tbl, cache->sets, 4);
+    build_append_byte(tbl, cache->associativity);
+    /* Read and Write allocate amd WB */
+    val = 0x3 | (1 << 4);
+    switch (cache->type) {
+    case INSTRUCTION:
+        val |= (1 << 2);
+        break;
+    case DATA:
+        val |= (0 << 2); /* Data */
+        break;
+    case UNIFIED:
+        val |= (3 << 2); /* Unified */
+        break;
+    }
+    build_append_byte(tbl, val);
+    build_append_int_noprefix(tbl, cache->linesize, 2);
+    build_append_int_noprefix(tbl,
+                              has_id ?
+                              (cache->type << 24) | (cache->level << 16) | id :
+                              0, 4);
+}
+
+static void build_caches_subset(GArray *table_data, uint32_t pptt_start,
+                                int num_caches, ACPIPPTTCache *caches,
+                                bool assign_ids, int base_id,
+                                uint8_t level_high, uint8_t level_low,
+                                uint32_t *data_offset, uint32_t *instr_offset)
+{
+    uint32_t next_level_offset_data = 0, next_level_offset_instruction = 0;
+    uint32_t this_offset, next_offset = 0;
+    int c, l;
+
+    /* Walk caches from top to bottom */
+
+    for (l = level_high; l >= level_low; l--) { /* Walk down levels */
+        for (c = 0; c < num_caches; c++) {
+            if (caches[c].level != l) {
+                continue;
+            }
+
+            /* Assume only unified above l1 for now */
+            this_offset = table_data->len - pptt_start;
+            switch (caches[c].type) {
+            case INSTRUCTION:
+                next_offset = next_level_offset_instruction;
+                break;
+            case DATA:
+                next_offset = next_level_offset_data;
+                break;
+            case UNIFIED:
+                /* Either is fine here - hopefully */
+                next_offset = next_level_offset_instruction;
+                break;
+            }
+            build_cache_nodes(table_data, &caches[c], next_offset,
+                              assign_ids, base_id);
+            switch (caches[c].type) {
+            case INSTRUCTION:
+                next_level_offset_instruction = this_offset;
+                break;
+            case DATA:
+                next_level_offset_data = this_offset;
+                break;
+            case UNIFIED:
+                next_level_offset_instruction = this_offset;
+                next_level_offset_data = this_offset;
+                break;
+            }
+            *data_offset = next_level_offset_data;
+            *instr_offset = next_level_offset_instruction;
+        }
+    }
+}
+
 /*
  * ACPI spec, Revision 6.3
  * 5.2.29 Processor Properties Topology Table (PPTT)
  */
 void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms,
-                const char *oem_id, const char *oem_table_id)
+                const char *oem_id, const char *oem_table_id,
+                int num_caches, ACPIPPTTCache *caches)
 {
+    bool share_structs = false;
     MachineClass *mc = MACHINE_GET_CLASS(ms);
     CPUArchIdList *cpus = ms->possible_cpus;
     int64_t socket_id = -1, cluster_id = -1, core_id = -1;
     uint32_t socket_offset = 0, cluster_offset = 0, core_offset = 0;
     uint32_t pptt_start = table_data->len;
     int n;
-    AcpiTable table = { .sig = "PPTT", .rev = 2,
+    AcpiTable table = { .sig = "PPTT", .rev = 3,
                         .oem_id = oem_id, .oem_table_id = oem_table_id };
+    uint32_t l1_data_offset = 0;
+    uint32_t l1_instr_offset = 0;
+    uint32_t cluster_data_offset = 0;
+    uint32_t cluster_instr_offset = 0;
+    uint32_t node_data_offset = 0;
+    uint32_t node_instr_offset = 0;
+    int top_node = 7;
+    int top_cluster = 7;
+    int top_core = 7;
 
     acpi_table_begin(&table, table_data);
 
+    /* Let us have a unified cache description for now */
+
+    if (share_structs && num_caches >= 1) {
+        if (ms->smp.cache_node_start_level) {
+            build_caches_subset(table_data, pptt_start, num_caches, caches,
+                                false, 0,
+                                top_node, ms->smp.cache_node_start_level,
+                                &node_data_offset, &node_instr_offset);
+            top_cluster = ms->smp.cache_node_start_level - 1;
+        }
+        /* Assumption that some caches below this */
+        if (ms->smp.cache_cluster_start_level) {
+            build_caches_subset(table_data, pptt_start, num_caches, caches,
+                                false, 0,
+                                top_cluster,  ms->smp.cache_cluster_start_level,
+                                &cluster_data_offset, &cluster_instr_offset);
+            top_core = ms->smp.cache_cluster_start_level - 1;
+        }
+        build_caches_subset(table_data, pptt_start, num_caches, caches,
+                            false, 0,
+                            top_core , 0,
+                            &l1_data_offset, &l1_instr_offset);
+    }
+
     /*
      * This works with the assumption that cpus[n].props.*_id has been
      * sorted from top to down levels in mc->possible_cpu_arch_ids().
      * Otherwise, the unexpected and duplicated containers will be
      * created.
      */
+
     for (n = 0; n < cpus->len; n++) {
         if (cpus->cpus[n].props.socket_id != socket_id) {
+            uint32_t priv_rsrc[2];
+            int num_priv = 0;
+
+            if (!share_structs && ms->smp.cache_node_start_level) {
+                build_caches_subset(table_data, pptt_start, num_caches, caches,
+                                    true, n,
+                                    top_node, ms->smp.cache_node_start_level,
+                                    &node_data_offset, &node_instr_offset);
+                top_cluster = ms->smp.cache_node_start_level - 1;
+            }
+            priv_rsrc[0] = node_instr_offset;
+            priv_rsrc[1] = node_data_offset;
+            if (node_instr_offset || node_data_offset) {
+                num_priv = node_instr_offset == node_data_offset ? 1 : 2;
+            }
             assert(cpus->cpus[n].props.socket_id > socket_id);
             socket_id = cpus->cpus[n].props.socket_id;
             cluster_id = -1;
@@ -2027,36 +2170,70 @@ void build_pptt(GArray *table_data, BIOSLinker *linker, MachineState *ms,
             socket_offset = table_data->len - pptt_start;
             build_processor_hierarchy_node(table_data,
                 (1 << 0), /* Physical package */
-                0, socket_id, NULL, 0);
+                0, socket_id, priv_rsrc, num_priv);
         }
 
+
         if (mc->smp_props.clusters_supported && mc->smp_props.has_clusters) {
             if (cpus->cpus[n].props.cluster_id != cluster_id) {
+                uint32_t priv_rsrc[2];
+                int num_priv = 0;
+
+                if (!share_structs && ms->smp.cache_cluster_start_level) {
+                    build_caches_subset(table_data, pptt_start, num_caches,
+                                        caches, true, n,
+                                        top_cluster,
+                                        ms->smp.cache_cluster_start_level,
+                                        &cluster_data_offset,
+                                        &cluster_instr_offset);
+                    top_core = ms->smp.cache_cluster_start_level - 1;
+                }
+                priv_rsrc[0] = cluster_instr_offset;
+                priv_rsrc[1] = cluster_data_offset;
+
                 assert(cpus->cpus[n].props.cluster_id > cluster_id);
                 cluster_id = cpus->cpus[n].props.cluster_id;
                 core_id = -1;
                 cluster_offset = table_data->len - pptt_start;
+
+                if (cluster_instr_offset || cluster_data_offset) {
+                    num_priv = cluster_instr_offset == cluster_data_offset ?
+                        1 : 2;
+                }
                 build_processor_hierarchy_node(table_data,
                     (0 << 0), /* Not a physical package */
-                    socket_offset, cluster_id, NULL, 0);
+                    socket_offset, cluster_id, priv_rsrc, num_priv);
             }
         } else {
             cluster_offset = socket_offset;
         }
 
+        if (!share_structs &&
+            cpus->cpus[n].props.core_id != core_id) {
+            build_caches_subset(table_data, pptt_start, num_caches, caches,
+                                true, n,
+                                top_core , 0,
+                                &l1_data_offset, &l1_instr_offset);
+        }
         if (ms->smp.threads == 1) {
+            uint32_t priv_rsrc[2] = { l1_instr_offset, l1_data_offset };
+
             build_processor_hierarchy_node(table_data,
                 (1 << 1) | /* ACPI Processor ID valid */
                 (1 << 3),  /* Node is a Leaf */
-                cluster_offset, n, NULL, 0);
+                cluster_offset, n, priv_rsrc,
+                l1_instr_offset == l1_data_offset ? 1 : 2);
         } else {
             if (cpus->cpus[n].props.core_id != core_id) {
+                uint32_t priv_rsrc[2] = { l1_instr_offset, l1_data_offset };
+
                 assert(cpus->cpus[n].props.core_id > core_id);
                 core_id = cpus->cpus[n].props.core_id;
                 core_offset = table_data->len - pptt_start;
                 build_processor_hierarchy_node(table_data,
                     (0 << 0), /* Not a physical package */
-                    cluster_offset, core_id, NULL, 0);
+                    cluster_offset, core_id, priv_rsrc,
+                    l1_instr_offset == l1_data_offset ? 1 : 2);
             }
 
             build_processor_hierarchy_node(table_data,
hw/arm/virt-acpi-build.c +129 -1
index 6b674231c2..ec8fdcefff 100644
@@ -922,6 +922,129 @@ static void acpi_align_size(GArray *blob, unsigned align)
     g_array_set_size(blob, ROUND_UP(acpi_data_len(blob), align));
 }
 
+static unsigned int virt_get_caches(VirtMachineState *vms,
+                                    ACPIPPTTCache *caches)
+{
+    ARMCPU *armcpu = ARM_CPU(qemu_get_cpu(0));
+    bool ccidx = cpu_isar_feature(any_ccidx, armcpu);
+    unsigned int num_cache, i;
+    int level_instr = 1, level_data = 1;
+
+    for (i = 0, num_cache = 0; i < 7; i++, num_cache++) {
+        int type = (armcpu->clidr >> (3 * i)) & 7;
+        int bank_index;
+        int level = 0;
+        ACPIPPTTCacheType cache_type = INSTRUCTION;
+
+        if (type == 0) {
+            break;
+        }
+
+        switch (type) {
+        case 1:
+            cache_type = INSTRUCTION;
+            level = level_instr;
+            break;
+        case 2:
+            cache_type = DATA;
+            level = level_data;
+            break;
+        case 4:
+            cache_type = UNIFIED;
+            level = level_instr > level_data ? level_instr : level_data;
+            break;
+        case 3: /* Split - Do data first */
+            cache_type = DATA;
+            level = level_data;
+            break;
+        }
+        /*
+         * ccsidr is indexed using both the level and whether it is
+         * an instruction cache. Unified caches use the same storage
+         * as data caches.
+         */
+        bank_index = (i * 2) | ((type == 1) ? 1 : 0);
+        if (ccidx) {
+            caches[num_cache] = (ACPIPPTTCache) {
+                .type =  cache_type,
+                .level = level,
+                .linesize = 1 << (FIELD_EX64(armcpu->ccsidr[bank_index],
+                                             CCSIDR_EL1,
+                                             CCIDX_LINESIZE) + 4),
+                .associativity = FIELD_EX64(armcpu->ccsidr[bank_index],
+                                            CCSIDR_EL1,
+                                            CCIDX_ASSOCIATIVITY) + 1,
+                .sets = FIELD_EX64(armcpu->ccsidr[bank_index], CCSIDR_EL1,
+                                   CCIDX_NUMSETS) + 1,
+            };
+        } else {
+            caches[num_cache] = (ACPIPPTTCache) {
+                .type =  cache_type,
+                .level = level,
+                .linesize = 1 << (FIELD_EX64(armcpu->ccsidr[bank_index],
+                                             CCSIDR_EL1, LINESIZE) + 4),
+                .associativity = FIELD_EX64(armcpu->ccsidr[bank_index],
+                                            CCSIDR_EL1,
+                                            ASSOCIATIVITY) + 1,
+                .sets = FIELD_EX64(armcpu->ccsidr[bank_index], CCSIDR_EL1,
+                                   NUMSETS) + 1,
+            };
+        }
+        caches[num_cache].size = caches[num_cache].associativity *
+            caches[num_cache].sets * caches[num_cache].linesize;
+
+        /* Break one 'split' entry up into two records */
+        if (type == 3) {
+            num_cache++;
+            bank_index = (i * 2) | 1;
+            if (ccidx) {
+                /* Instruction cache: bottom bit set when reading banked reg */
+                caches[num_cache] = (ACPIPPTTCache) {
+                    .type = INSTRUCTION,
+                    .level = level_instr,
+                    .linesize = 1 << (FIELD_EX64(armcpu->ccsidr[bank_index],
+                                                 CCSIDR_EL1,
+                                                 CCIDX_LINESIZE) + 4),
+                    .associativity = FIELD_EX64(armcpu->ccsidr[bank_index],
+                                                CCSIDR_EL1,
+                                                CCIDX_ASSOCIATIVITY) + 1,
+                    .sets = FIELD_EX64(armcpu->ccsidr[bank_index], CCSIDR_EL1,
+                                       CCIDX_NUMSETS) + 1,
+                };
+            } else {
+                caches[num_cache] = (ACPIPPTTCache) {
+                    .type = INSTRUCTION,
+                    .level = level_instr,
+                    .linesize = 1 << (FIELD_EX64(armcpu->ccsidr[bank_index],
+                                                 CCSIDR_EL1, LINESIZE) + 4),
+                    .associativity = FIELD_EX64(armcpu->ccsidr[bank_index],
+                                                CCSIDR_EL1,
+                                                ASSOCIATIVITY) + 1,
+                    .sets = FIELD_EX64(armcpu->ccsidr[bank_index], CCSIDR_EL1,
+                                       NUMSETS) + 1,
+                };
+            }
+            caches[num_cache].size = caches[num_cache].associativity *
+                caches[num_cache].sets * caches[num_cache].linesize;
+        }
+        switch (type) {
+        case 1:
+            level_instr++;
+            break;
+        case 2:
+            level_data++;
+            break;
+        case 3:
+        case 4:
+            level_instr++;
+            level_data++;
+            break;
+        }
+    }
+
+    return num_cache;
+}
+
 static
 void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
 {
@@ -930,6 +1053,8 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
     unsigned dsdt, xsdt;
     GArray *tables_blob = tables->table_data;
     MachineState *ms = MACHINE(vms);
+    ACPIPPTTCache caches[16]; /* Can select up to 16 */
+    unsigned int num_cache;
 
     table_offsets = g_array_new(false, true /* clear */,
                                         sizeof(uint32_t));
@@ -949,10 +1074,13 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
     acpi_add_table(table_offsets, tables_blob);
     build_madt(tables_blob, tables->linker, vms);
 
+    num_cache = virt_get_caches(vms, caches);
+
     if (!vmc->no_cpu_topology) {
         acpi_add_table(table_offsets, tables_blob);
         build_pptt(tables_blob, tables->linker, ms,
-                   vms->oem_id, vms->oem_table_id);
+                   vms->oem_id, vms->oem_table_id,
+                   num_cache, caches);
     }
 
     acpi_add_table(table_offsets, tables_blob);
hw/core/machine-smp.c +8 -0
index 0f4d9b6f7a..cbb0bf1bc7 100644
@@ -81,6 +81,10 @@ void machine_parse_smp_config(MachineState *ms,
     unsigned cores   = config->has_cores ? config->cores : 0;
     unsigned threads = config->has_threads ? config->threads : 0;
     unsigned maxcpus = config->has_maxcpus ? config->maxcpus : 0;
+    unsigned cache_cl_start = config->has_cache_cluster_start_level ?
+        config->cache_cluster_start_level : 0;
+    unsigned cache_nd_start = config->has_cache_node_start_level ?
+        config->cache_node_start_level : 0;
 
     /*
      * Specified CPU topology parameters must be greater than zero,
@@ -161,6 +165,10 @@ void machine_parse_smp_config(MachineState *ms,
     ms->smp.max_cpus = maxcpus;
 
     mc->smp_props.has_clusters = config->has_clusters;
+    if (mc->smp_props.has_clusters) {
+        ms->smp.cache_cluster_start_level = cache_cl_start;
+        ms->smp.cache_node_start_level = cache_nd_start;
+    }
 
     /* sanity-check of the computed topology */
     if (sockets * dies * clusters * cores * threads != maxcpus) {
hw/loongarch/acpi-build.c +1 -1
index 0b62c3a2f7..51d4ed9a19 100644
@@ -439,7 +439,7 @@ static void acpi_build(AcpiBuildTables *tables, MachineState *machine)
 
     acpi_add_table(table_offsets, tables_blob);
     build_pptt(tables_blob, tables->linker, machine,
-               lams->oem_id, lams->oem_table_id);
+               lams->oem_id, lams->oem_table_id, 0, NULL);
 
     acpi_add_table(table_offsets, tables_blob);
     build_srat(tables_blob, tables->linker, machine);
Signature · 1 line
2.39.2
#4 lore raw

RFC PATCH 2/5 HACK: target/arm/tcg: Add some more caches to cpu=max

In reply to Jonathan Cameron via

Used to drive the MPAM cache intialization and to exercise more of the PPTT cache entry generation code. Perhaps a default L3 cache is acceptable for max?

---
 target/arm/tcg/cpu64.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
target/arm/tcg/cpu64.c +12 -0
index 8019f00bc3..2af67739f6 100644
@@ -711,6 +711,17 @@ void aarch64_max_tcg_initfn(Object *obj)
     uint64_t t;
     uint32_t u;
 
+    /*
+     * Expanded cache set
+     */
+    cpu->clidr = 0x8204923; /* 4 4 4 4 3 in 3 bit fields */
+    cpu->ccsidr[0] = 0x000000ff0000001aull; /* 64KB L1 dcache */
+    cpu->ccsidr[1] = 0x000000ff0000001aull; /* 64KB L1 icache */
+    cpu->ccsidr[2] = 0x000007ff0000003aull; /* 1MB L2 unified cache */
+    cpu->ccsidr[4] = 0x000007ff0000007cull; /* 2MB L3 cache 128B line */
+    cpu->ccsidr[6] = 0x00007fff0000007cull; /* 16MB L4 cache 128B line */
+    cpu->ccsidr[8] = 0x0007ffff0000007cull; /* 2048MB L5 cache 128B line */
+
     /*
      * Reset MIDR so the guest doesn't mistake our 'max' CPU type for a real
      * one and try to apply errata workarounds or use impdef features we
@@ -828,6 +839,7 @@ void aarch64_max_tcg_initfn(Object *obj)
     t = FIELD_DP64(t, ID_AA64MMFR2, BBM, 2);      /* FEAT_BBM at level 2 */
     t = FIELD_DP64(t, ID_AA64MMFR2, EVT, 2);      /* FEAT_EVT */
     t = FIELD_DP64(t, ID_AA64MMFR2, E0PD, 1);     /* FEAT_E0PD */
+    t = FIELD_DP64(t, ID_AA64MMFR2, CCIDX, 1);      /* FEAT_TTCNP */
     cpu->isar.id_aa64mmfr2 = t;
 
     t = cpu->isar.id_aa64zfr0;
Signature · 1 line
2.39.2
#5 lore raw

Re: [RFC PATCH 2/5] HACK: target/arm/tcg: Add some more caches to cpu=max

Quoted 18 lines · Jonathan Cameron

Used to drive the MPAM cache intialization and to exercise more of the PPTT cache entry generation code. Perhaps a default L3 cache is acceptable for max?

---
 target/arm/tcg/cpu64.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
target/arm/tcg/cpu64.c +1 -0
index 8019f00bc3..2af67739f6 100644
@@ -711,6 +711,17 @@ void aarch64_max_tcg_initfn(Object *obj)
     uint64_t t;
     uint32_t u;
 
+    /*

target/arm/tcg/cpu64.c void aarch64_max_tcg_initfn(Object *obj) · line 724

Quoted 10 lines
target/arm/tcg/cpu64.c +10 -0
+     * Expanded cache set
+     */
+    cpu->clidr = 0x8204923; /* 4 4 4 4 3 in 3 bit fields */
+    cpu->ccsidr[0] = 0x000000ff0000001aull; /* 64KB L1 dcache */
+    cpu->ccsidr[1] = 0x000000ff0000001aull; /* 64KB L1 icache */
+    cpu->ccsidr[2] = 0x000007ff0000003aull; /* 1MB L2 unified cache */
+    cpu->ccsidr[4] = 0x000007ff0000007cull; /* 2MB L3 cache 128B line */
+    cpu->ccsidr[6] = 0x00007fff0000007cull; /* 16MB L4 cache 128B line */
+    cpu->ccsidr[8] = 0x0007ffff0000007cull; /* 2048MB L5 cache 128B line */
+

I think Peter in another thread wondered if we should have a generic function for expanding the cache idr registers based on a abstract lane definition.

Quoted 11 lines
target/arm/tcg/cpu64.c +1 -0
     /*
      * Reset MIDR so the guest doesn't mistake our 'max' CPU type for a real
      * one and try to apply errata workarounds or use impdef features we
@@ -828,6 +839,7 @@ void aarch64_max_tcg_initfn(Object *obj)
     t = FIELD_DP64(t, ID_AA64MMFR2, BBM, 2);      /* FEAT_BBM at level 2 */
     t = FIELD_DP64(t, ID_AA64MMFR2, EVT, 2);      /* FEAT_EVT */
     t = FIELD_DP64(t, ID_AA64MMFR2, E0PD, 1);     /* FEAT_E0PD */
+    t = FIELD_DP64(t, ID_AA64MMFR2, CCIDX, 1);      /* FEAT_TTCNP */
     cpu->isar.id_aa64mmfr2 = t;
 
     t = cpu->isar.id_aa64zfr0;
Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
#6 lore raw

Re: [RFC PATCH 2/5] HACK: target/arm/tcg: Add some more caches to cpu=max

Quoted 19 lines · On Mon, 14 Aug 2023 11:13:58 +0100 Alex Bennée <[email protected]> wrote:
Quoted 17 lines · Jonathan Cameron

Used to drive the MPAM cache intialization and to exercise more of the PPTT cache entry generation code. Perhaps a default L3 cache is acceptable for max?

---
 target/arm/tcg/cpu64.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
target/arm/tcg/cpu64.c +0 -0
index 8019f00bc3..2af67739f6 100644
@@ -711,6 +711,17 @@ void aarch64_max_tcg_initfn(Object *obj)
     uint64_t t;
     uint32_t u;
 
Quoted 16 lines

target/arm/tcg/cpu64.c void aarch64_max_tcg_initfn(Object *obj) · line 724

Quoted 11 lines
target/arm/tcg/cpu64.c +11 -0
+    /*
+     * Expanded cache set
+     */
+    cpu->clidr = 0x8204923; /* 4 4 4 4 3 in 3 bit fields */
+    cpu->ccsidr[0] = 0x000000ff0000001aull; /* 64KB L1 dcache */
+    cpu->ccsidr[1] = 0x000000ff0000001aull; /* 64KB L1 icache */
+    cpu->ccsidr[2] = 0x000007ff0000003aull; /* 1MB L2 unified cache */
+    cpu->ccsidr[4] = 0x000007ff0000007cull; /* 2MB L3 cache 128B line */
+    cpu->ccsidr[6] = 0x00007fff0000007cull; /* 16MB L4 cache 128B line */
+    cpu->ccsidr[8] = 0x0007ffff0000007cull; /* 2048MB L5 cache 128B line */
+  

I think Peter in another thread wondered if we should have a generic function for expanding the cache idr registers based on a abstract lane definition.

Great!

This response?
https://lore.kernel.org/qemu-devel/CAFEAcA_Lzj1LEutMro72fCfqiCWtOpd+5b-YPcfKv8Bg1f+rCg@mail.gmail.com/

That might get us somewhere but ultimately I think we need a general way to push this stuff in as parameters of the CPU or a CPU definition with a wide enough set of caches to allow us to poke the boundaries and hang a typical MPAM setup off it. Would people mind adding at least an L3 to max? The L4 and above is useful for checking the PPTT building code works, but that's probably more a development time activity than an every day one.

Jonathan

Quoted 13 lines
Quoted 11 lines
target/arm/tcg/cpu64.c +1 -0
     /*
      * Reset MIDR so the guest doesn't mistake our 'max' CPU type for a real
      * one and try to apply errata workarounds or use impdef features we
@@ -828,6 +839,7 @@ void aarch64_max_tcg_initfn(Object *obj)
     t = FIELD_DP64(t, ID_AA64MMFR2, BBM, 2);      /* FEAT_BBM at level 2 */
     t = FIELD_DP64(t, ID_AA64MMFR2, EVT, 2);      /* FEAT_EVT */
     t = FIELD_DP64(t, ID_AA64MMFR2, E0PD, 1);     /* FEAT_E0PD */
+    t = FIELD_DP64(t, ID_AA64MMFR2, CCIDX, 1);      /* FEAT_TTCNP */
     cpu->isar.id_aa64mmfr2 = t;
 
     t = cpu->isar.id_aa64zfr0;  
#7 lore raw

Re: [RFC PATCH 2/5] HACK: target/arm/tcg: Add some more caches to cpu=max

Quoted 39 lines · Jonathan Cameron via
Quoted 19 lines · On Mon, 14 Aug 2023 11:13:58 +0100 Alex Bennée <[email protected]> wrote:
Quoted 17 lines · Jonathan Cameron

Used to drive the MPAM cache intialization and to exercise more of the PPTT cache entry generation code. Perhaps a default L3 cache is acceptable for max?

---
  target/arm/tcg/cpu64.c | 12 ++++++++++++
  1 file changed, 12 insertions(+)
target/arm/tcg/cpu64.c +0 -0
index 8019f00bc3..2af67739f6 100644
@@ -711,6 +711,17 @@ void aarch64_max_tcg_initfn(Object *obj)
      uint64_t t;
      uint32_t u;
 
Quoted 16 lines

target/arm/tcg/cpu64.c void aarch64_max_tcg_initfn(Object *obj) · line 724

Quoted 11 lines
target/arm/tcg/cpu64.c +11 -0
+    /*
+     * Expanded cache set
+     */
+    cpu->clidr = 0x8204923; /* 4 4 4 4 3 in 3 bit fields */
+    cpu->ccsidr[0] = 0x000000ff0000001aull; /* 64KB L1 dcache */
+    cpu->ccsidr[1] = 0x000000ff0000001aull; /* 64KB L1 icache */
+    cpu->ccsidr[2] = 0x000007ff0000003aull; /* 1MB L2 unified cache */
+    cpu->ccsidr[4] = 0x000007ff0000007cull; /* 2MB L3 cache 128B line */
+    cpu->ccsidr[6] = 0x00007fff0000007cull; /* 16MB L4 cache 128B line */
+    cpu->ccsidr[8] = 0x0007ffff0000007cull; /* 2048MB L5 cache 128B line */
+

I think Peter in another thread wondered if we should have a generic function for expanding the cache idr registers based on a abstract lane definition.

Quoted 4 lines

Great!

This response?
https://lore.kernel.org/qemu-devel/CAFEAcA_Lzj1LEutMro72fCfqiCWtOpd+5b-YPcfKv8Bg1f+rCg@mail.gmail.com/

Followed up with

https://lore.kernel.org/qemu-devel/[email protected]/

r~

#8 lore raw

RFC PATCH 3/5 target/arm: Add support for MPAM CPU registers

In reply to Jonathan Cameron via

It is common to support MPAM on CPU cores, but not in the rest of the system, so there is little disadvantage in always enabling these.

---
 target/arm/cpu.h    | 15 +++++++++++++++
 target/arm/cpu.c    | 10 +++++++++-
 target/arm/helper.c | 30 ++++++++++++++++++++++++++++++
 3 files changed, 54 insertions(+), 1 deletion(-)
target/arm/cpu.h +15 -0
index 88e5accda6..8d28e22291 100644
@@ -735,6 +735,10 @@ typedef struct CPUArchState {
      * to keep the offsets into the rest of the structure smaller.
      */
     ARMVectorReg zarray[ARM_MAX_VQ * 16];
+
+    uint64_t mpam0_el1;
+    uint64_t mpam1_el1;
+
 #endif
 
     struct CPUBreakpoint *cpu_breakpoint[16];
@@ -1043,6 +1047,7 @@ struct ArchCPU {
         uint64_t id_aa64zfr0;
         uint64_t id_aa64smfr0;
         uint64_t reset_pmcr_el0;
+        uint64_t mpamidr_el1;
     } isar;
     uint64_t midr;
     uint32_t revidr;
@@ -2327,6 +2332,16 @@ FIELD(DBGDEVID, DOUBLELOCK, 20, 4)
 FIELD(DBGDEVID, AUXREGS, 24, 4)
 FIELD(DBGDEVID, CIDMASK, 28, 4)
 
+FIELD(MPAMIDR, PARTID_MAX, 0, 16)
+FIELD(MPAMIDR, HAS_HCR, 17, 1)
+FIELD(MPAMIDR, VMR_MAX, 18, 3)
+FIELD(MPAMIDR, PMG_MAX, 32, 8)
+FIELD(MPAMIDR, HAS_ALTSP, 57, 1)
+FIELD(MPAMIDR, HAS_TIDR, 58, 1)
+FIELD(MPAMIDR, SP4, 59, 1)
+FIELD(MPAMIDR, HAS_FORCE_NS, 60, 1)
+FIELD(MPAMIDR, HAS_SDEFLT, 61, 1)
+
 FIELD(MVFR0, SIMDREG, 0, 4)
 FIELD(MVFR0, FPSP, 4, 4)
 FIELD(MVFR0, FPDP, 8, 4)
target/arm/cpu.c +9 -1
index 93c28d50e5..d85a3ec8a2 100644
@@ -305,6 +305,9 @@ static void arm_cpu_reset_hold(Object *obj)
         env->cp15.rvbar = cpu->rvbar_prop;
         env->pc = env->cp15.rvbar;
 #endif
+
+        env->mpam1_el1 = 1ULL << 63;
+
     } else {
 #if defined(CONFIG_USER_ONLY)
         /* Userspace expects access to cp10 and cp11 for FP/Neon */
@@ -2097,7 +2100,12 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)
             FIELD_DP32(cpu->isar.id_pfr0, ID_PFR0, AMU, 0);
         /* FEAT_MPAM (Memory Partitioning and Monitoring Extension) */
         cpu->isar.id_aa64pfr0 =
-            FIELD_DP64(cpu->isar.id_aa64pfr0, ID_AA64PFR0, MPAM, 0);
+            FIELD_DP64(cpu->isar.id_aa64pfr0, ID_AA64PFR0, MPAM, 1);
+        cpu->isar.mpamidr_el1 =
+            FIELD_DP64(cpu->isar.mpamidr_el1, MPAMIDR, PARTID_MAX, 63);
+        cpu->isar.mpamidr_el1 =
+            FIELD_DP64(cpu->isar.mpamidr_el1, MPAMIDR, PMG_MAX, 3);
+
         /* FEAT_NV (Nested Virtualization) */
         cpu->isar.id_aa64mmfr2 =
             FIELD_DP64(cpu->isar.id_aa64mmfr2, ID_AA64MMFR2, NV, 0);
target/arm/helper.c +30 -0
index 50f61e42ca..dbeb8d9fa6 100644
@@ -8072,7 +8072,17 @@ static const ARMCPRegInfo actlr2_hactlr2_reginfo[] = {
       .access = PL2_RW, .type = ARM_CP_CONST,
       .resetvalue = 0 },
 };
+/*
+static uint64_t mpam_el1_read(CPUARMState *env, const ARMCPRegInfo *ri)
+{
+    return 0;
+}
 
+static void mpam_el1_write(CPUARMState *env, const ARMCPRegInfo *ri, uint64_t val)
+{
+    return;
+}
+*/
 void register_cp_regs_for_features(ARMCPU *cpu)
 {
     /* Register all the coprocessor registers based on feature bits */
@@ -8404,6 +8414,26 @@ void register_cp_regs_for_features(ARMCPU *cpu)
               .access = PL1_R, .type = ARM_CP_CONST,
               .accessfn = access_aa64_tid3,
               .resetvalue = 0 },
+
+            /* Should be separate feature */
+            { .name = "MPAMIDR_EL1", .state = ARM_CP_STATE_AA64,
+              .opc0 = 3, .opc1 = 0, .crn = 0xa, .crm = 4, .opc2 = 4,
+              .access = PL1_R, .type = ARM_CP_CONST,
+              .accessfn = access_aa64_tid3,
+              .resetvalue = cpu->isar.mpamidr_el1 },
+            /* TODO: check the accessfn and whether we need a reset value for these */
+            { .name = "MPAM0_EL1", .state = ARM_CP_STATE_AA64,
+              .opc0 = 3, .opc1 = 0, .crn = 0xa, .crm = 5, .opc2 = 1,
+              .access = PL1_RW, .type = ARM_CP_ALIAS,
+              .accessfn = access_aa64_tid3,
+              .fieldoffset = offsetof(CPUARMState, mpam0_el1),
+            },
+            { .name = "MPAM1_EL1", .state = ARM_CP_STATE_AA64,
+              .opc0 = 3, .opc1 = 0, .crn = 0xa, .crm = 5, .opc2 = 0,
+              .access = PL1_RW, .type = ARM_CP_ALIAS,
+              .accessfn = access_aa64_tid3,
+              .fieldoffset = offsetof(CPUARMState, mpam1_el1),
+            },
             { .name = "MVFR0_EL1", .state = ARM_CP_STATE_AA64,
               .opc0 = 3, .opc1 = 0, .crn = 0, .crm = 3, .opc2 = 0,
               .access = PL1_R, .type = ARM_CP_CONST,
Signature · 1 line
2.39.2
#9 lore raw

RFC PATCH 4/5 hw/arm: Add MPAM emulation.

In reply to Jonathan Cameron via

Note this doesn't 'do' anything other than provide an introspection interface. The intent here is to support work on the Linux kernel support and for that a functional emulation of the interface is useful.

---
 qapi/mpam.json           |  78 ++++
 qapi/qapi-schema.json    |   1 +
 include/hw/arm/mpam.h    |  12 +
 include/hw/arm/virt.h    |   2 +
 hw/arm/mpam-qapi-stubs.c |   9 +
 hw/arm/mpam-qapi.c       |  58 +++
 hw/arm/mpam.c            | 886 +++++++++++++++++++++++++++++++++++++++
 hw/arm/Kconfig           |   4 +
 hw/arm/meson.build       |   4 +
 qapi/meson.build         |   1 +
 10 files changed, 1055 insertions(+)
qapi/mpam.json new file +78 -0
new file mode 100644
index 0000000000..f4990ef96b
@@ -0,0 +1,78 @@
+# -*- Mode: Python -*-
+# vim: filetype=python
+
+##
+# = ARM MPAM State Introspection
+##
+
+##
+# @MpamBm:
+##
+{ 'struct': 'MpamBm',
+  'data' : { 'words' : [ 'int' ]
+           }
+}
+
+##
+# @MpamRegs:
+#
+# Per RIS Register State
+#
+##
+{ 'struct' : 'MpamRegs',
+  'data' : { 'idr' : 'int',
+             'iidr' : 'int',
+             'aidr' : 'int',
+             'cpor-idr': 'int',
+             'ccap-idr': 'int',
+             'mbw-idr': 'int',
+             'pri-idr': 'int',
+             'partid-nrw-idr': 'int',
+             'msmon-idr': 'int',
+             'csumon-idr': 'int',
+             'mbwumon-idr': 'int',
+             'ecr': 'int',
+             'esr': 'int',
+             'cfg-part-sel': 'int',
+             'cfg-cpbm': ['MpamBm']#lazy
+          }
+}
+
+##
+# @MpamCacheInfo:
+#
+# Information about MPAM Cache MSCs
+#
+# @cpu: First CPU of the set associated with the cache
+#
+# @level: Level of cache
+#
+# @type: type of cache - make an enum
+#
+# Since: 9.0
+##
+
+{ 'struct': 'MpamCacheInfo',
+  'data' : { 'cpu' : 'int',
+             'level' : 'int',
+             'type' : 'int',
+             'regs' : ['MpamRegs']
+           }
+}
+
+##
+# @query-mpam-cache:
+#
+# Get a list of MpamInfo for all Cache Related MSC
+#
+# @level: Provide a cache level to filter against
+#
+# Returns: a list of @MpamCacheInfo describing all
+#   the MPAM cache MSC instances
+#
+# Since: 9.0
+##
+{ 'command': 'query-mpam-cache',
+  'data': { '*level': 'int' },
+  'returns' : ['MpamCacheInfo'],
+  'allow-preconfig': true }
qapi/qapi-schema.json +1 -0
index 6594afba31..ea3ee75841 100644
@@ -79,3 +79,4 @@
 { 'include': 'virtio.json' }
 { 'include': 'cryptodev.json' }
 { 'include': 'cxl.json' }
+{ 'include': 'mpam.json' }
include/hw/arm/mpam.h new file +12 -0
new file mode 100644
index 0000000000..7bd88d57bc
@@ -0,0 +1,12 @@
+#ifndef _MPAM_H_
+#define _MPAM_H_
+
+#include "qom/object.h"
+#include "qapi/qapi-commands-mpam.h"
+
+#define TYPE_MPAM_MSC_MEM "mpam-msc-mem"
+#define TYPE_MPAM_MSC_CACHE "mpam-msc-cache"
+
+void mpam_cache_fill_info(Object *obj, MpamCacheInfo *info);
+
+#endif /* _MPAM_H_ */
include/hw/arm/virt.h +2 -0
index e1ddbea96b..ac015a391a 100644
@@ -71,6 +71,7 @@ enum {
     VIRT_SMMU,
     VIRT_UART,
     VIRT_MMIO,
+    VIRT_MPAM_MSC,
     VIRT_RTC,
     VIRT_FW_CFG,
     VIRT_PCIE,
@@ -160,6 +161,7 @@ struct VirtMachineState {
     bool ras;
     bool mte;
     bool dtb_randomness;
+    bool mpam, mpam_min_msc;
     OnOffAuto acpi;
     VirtGICType gic_version;
     VirtIOMMUType iommu;
hw/arm/mpam-qapi-stubs.c new file +9 -0
new file mode 100644
index 0000000000..40ccb7de9a
@@ -0,0 +1,9 @@
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qapi/qapi-commands-mpam.h"
+
+MpamCacheInfoList *qmp_query_mpam_cache(bool has_level, int64_t level, Error **errp)
+{
+    return NULL;
+}
hw/arm/mpam-qapi.c new file +58 -0

58 lines not on this page · raw message

hw/arm/mpam.c new file +886 -0

886 lines not on this page · raw message

hw/arm/Kconfig +4 -0
index 7e68348440..e44259910c 100644
@@ -32,6 +32,7 @@ config ARM_VIRT
     select VIRTIO_MEM_SUPPORTED
     select ACPI_CXL
     select ACPI_HMAT
+    select MPAM
 
 config CHEETAH
     bool
@@ -593,6 +594,9 @@ config FSL_IMX7
 config ARM_SMMUV3
     bool
 
+config MPAM
+    bool
+
 config FSL_IMX6UL
     bool
     default y
hw/arm/meson.build +4 -0
index 11eb9112f8..08f72befa1 100644
@@ -67,6 +67,10 @@ arm_ss.add(when: 'CONFIG_XEN', if_true: files('xen_arm.c'))
 arm_ss.add_all(xen_ss)
 
 system_ss.add(when: 'CONFIG_ARM_SMMUV3', if_true: files('smmu-common.c'))
+system_ss.add(when: 'CONFIG_MPAM', if_true: files('mpam.c', 'mpam-qapi.c'))
+system_ss.add(when: 'CONFIG_MPAM', if_false: files('mpam-qapi-stubs.c'))
+system_ss.add(when: 'CONFIG_ALL', if_true: files('mpam-qapi-stubs.c'))
+
 system_ss.add(when: 'CONFIG_EXYNOS4', if_true: files('exynos4_boards.c'))
 system_ss.add(when: 'CONFIG_RASPI', if_true: files('bcm2835_peripherals.c'))
 system_ss.add(when: 'CONFIG_TOSA', if_true: files('tosa.c'))
qapi/meson.build +1 -0
index 60a668b343..8e9a45330b 100644
@@ -41,6 +41,7 @@ qapi_all_modules = [
   'migration',
   'misc',
   'misc-target',
+  'mpam',
   'net',
   'pragma',
   'qom',
Signature · 1 line
2.39.2
#10 lore raw

RFC PATCH 5/5 hw/arm/virt: Add MPAM MSCs for memory controllers and caches.

In reply to Jonathan Cameron via

Allow sharing of MSC instances (using RIS) for memory controllers. Currently cached controllers are not using RIS (tend to be more in a system so more clever logic needed to auto allocate them).

No DT support yet. The kernel bindings are considered unstable so premature to add too much on that front.

Note that for now MPAM MSC creation is dependent on having SRAT and hence you need some numa nodes defined.

---
 include/hw/arm/mpam.h    |   1 +
 hw/arm/virt-acpi-build.c | 197 +++++++++++++++++++++++++++++++++++++++
 hw/arm/virt.c            | 134 ++++++++++++++++++++++++++
 3 files changed, 332 insertions(+)
include/hw/arm/mpam.h +1 -0
index 7bd88d57bc..8f47c8806f 100644
@@ -7,6 +7,7 @@
 #define TYPE_MPAM_MSC_MEM "mpam-msc-mem"
 #define TYPE_MPAM_MSC_CACHE "mpam-msc-cache"
 
+#define MPAM_SIZE 0x4000 /* Big enough for anyone ;) */
 void mpam_cache_fill_info(Object *obj, MpamCacheInfo *info);
 
 #endif /* _MPAM_H_ */
hw/arm/virt-acpi-build.c +197 -0

215 lines not on this page · raw message

hw/arm/virt.c +134 -0

170 lines not on this page · raw message

Signature · 1 line
2.39.2