Skip to contentSkip to conversation

acpi: report numa nodes for device memory using GI

PATCH Superseded [email protected] 3 participants 7 messages Nov 19, 2023 – Nov 30, 2023 Arm/Amlogic Meson SoC support

v4 of 19 Superseded Latest is v19 →

lore
Share & tools
lore
#0 lore raw

v4 cover acpi: report numa nodes for device memory using GI

From: Ankit Agrawal <[email protected]>

There are upcoming devices which allow CPU to cache coherently access their memory. It is sensible to expose such memory as NUMA nodes separate from the sysmem node to the OS. The ACPI spec provides a scheme in SRAT called Generic Initiator Affinity Structure [1] to allow an association between a Proximity Domain (PXM) and a Generic Initiator (GI) (e.g. heterogeneous processors and accelerators, GPUs, and I/O devices with integrated compute or DMA engines).

While a single node per device may cover several use cases, it is however insufficient for a full utilization of the NVIDIA GPUs MIG (Mult-Instance GPUs) [2] feature. The feature allows partitioning of the GPU device resources (including device memory) into several (upto 8) isolated instances. Each of the partitioned memory requires a dedicated NUMA node to operate. The partitions are not fixed and they can be created/deleted at runtime.

Linux OS does not provide a means to dynamically create/destroy NUMA nodes and such feature implementation is expected to be non-trivial. The nodes that OS discovers at the boot time while parsing SRAT remains fixed. So we utilize the GI Affinity structures that allows association between nodes and devices. Multiple GI structures per device/BDF is possible, allowing creation of multiple nodes in the VM by exposing unique PXM in each of these structures.

Implement the mechanism to build the GI affinity structures as Qemu currently does not. Introduce a new acpi-generic-initiator object that allows an association of a set of nodes with a device. During SRAT creation, all such objected are identified and used to add the GI Affinity Structures. Currently, only PCI device is supported and an error is returned for acpi device.

The admin will create a range of 8 nodes and associate that with the device using the acpi-generic-initiator object. While a configuration of less than 8 nodes per device is allowed, such configuration will prevent utilization of the feature to the fullest. This setting is applicable to all the Grace+Hopper systems. The following is an example of the Qemu command line arguments to create 8 nodes and link them to the device 'dev0':

-numa node,nodeid=2 \
-numa node,nodeid=3 \
-numa node,nodeid=4 \
-numa node,nodeid=5 \
-numa node,nodeid=6 \
-numa node,nodeid=7 \
-numa node,nodeid=8 \
-numa node,nodeid=9 \
-device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.0,addr=04.0,rombar=0,id=dev0 \
-object acpi-generic-initiator,id=gi0,pci-dev=dev0,host-nodes=2-9 \

The performance benefits can be realized by providing the NUMA node distances appropriately (through libvirt tags or Qemu params). The admin can get the distance among nodes in hardware using numactl -H.

This series goes along with the vfio-pci variant driver [3] under review.

Applied over v8.2.0.

[1] ACPI Spec 6.3, Section 5.2.16.6
[2] https://www.nvidia.com/en-in/technologies/multi-instance-gpu
[3] https://lore.kernel.org/all/[email protected]/

Link for v3:
https://lore.kernel.org/all/[email protected]/

v3 -> v4
- changed the ':' delimited way to a uint16 array to communicate the nodes associated with the device.
- added asserts to handle invalid inputs.
- addressed other miscellaneous v3 comments.

v2 -> v3
- changed param to accept a ':' delimited list of numa nodes, instead of a range.
- Removed nvidia-acpi-generic-initiator object.
- Addressed miscellaneous comments in v2.

v1 -> v2
- Removed dependency on sysfs to communicate the feature with variant module.
- Use GI Affinity SRAT structure instead of Memory Affinity.
- No DSDT entries needed to communicate the PXM for the device. SRAT GI structure is used instead.
- New objects introduced to establish link between device and nodes.

Ankit Agrawal (2):
qom: new object to associate device to numa node hw/acpi: Implement the SRAT GI affinity structure

 hw/acpi/acpi-generic-initiator.c         | 185 +++++++++++++++++++++++
 hw/acpi/meson.build                      |   1 +
 hw/arm/virt-acpi-build.c                 |   3 +
 include/hw/acpi/acpi-generic-initiator.h |  56 +++++++
 qapi/qom.json                            |  18 +++
 5 files changed, 263 insertions(+)
 create mode 100644 hw/acpi/acpi-generic-initiator.c
 create mode 100644 include/hw/acpi/acpi-generic-initiator.h
Signature · 1 line
2.34.1
#1 lore raw

PATCH v4 1/2 qom: new object to associate device to numa node

From: Ankit Agrawal <[email protected]>

NVIDIA GPU's support MIG (Mult-Instance GPUs) feature [1], which allows partitioning of the GPU device resources (including device memory) into several (upto 8) isolated instances. Each of the partitioned memory needs a dedicated NUMA node to operate. The partitions are not fixed and they can be created/deleted at runtime.

Unfortunately Linux OS does not provide a means to dynamically create/destroy NUMA nodes and such feature implementation is not expected to be trivial. The nodes that OS discovers at the boot time while parsing SRAT remains fixed. So we utilize the Generic Initiator Affinity structures that allows association between nodes and devices. Multiple GI structures per BDF is possible, allowing creation of multiple nodes by exposing unique PXM in each of these structures.

Introduce a new acpi-generic-initiator object to allow host admin provide the device and the corresponding NUMA nodes. Qemu maintain this association and use this object to build the requisite GI Affinity Structure.

An admin can provide the range of nodes through a uint16 array host-nodes and link it to a device by providing its id. Currently, only PCI device is supported and an error is returned for acpi device. The following sample creates 8 nodes and link them to the PCI device dev0:

-numa node,nodeid=2 \
-numa node,nodeid=3 \
-numa node,nodeid=4 \
-numa node,nodeid=5 \
-numa node,nodeid=6 \
-numa node,nodeid=7 \
-numa node,nodeid=8 \
-numa node,nodeid=9 \
-device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.0,addr=04.0,rombar=0,id=dev0 \
-object acpi-generic-initiator,id=gi0,pci-dev=dev0,host-nodes=2-9 \

[1] https://www.nvidia.com/en-in/technologies/multi-instance-gpu

---
 hw/acpi/acpi-generic-initiator.c         | 84 ++++++++++++++++++++++++
 hw/acpi/meson.build                      |  1 +
 include/hw/acpi/acpi-generic-initiator.h | 30 +++++++++
 qapi/qom.json                            | 18 +++++
 4 files changed, 133 insertions(+)
 create mode 100644 hw/acpi/acpi-generic-initiator.c
 create mode 100644 include/hw/acpi/acpi-generic-initiator.h
hw/acpi/acpi-generic-initiator.c new file +84 -0
new file mode 100644
index 0000000000..5ea51cb81e
@@ -0,0 +1,84 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved
+ */
+
+#include "qemu/osdep.h"
+#include "hw/qdev-properties.h"
+#include "qapi/error.h"
+#include "qapi/qapi-builtin-visit.h"
+#include "qapi/visitor.h"
+#include "qom/object_interfaces.h"
+#include "qom/object.h"
+#include "hw/qdev-core.h"
+#include "hw/vfio/vfio-common.h"
+#include "hw/vfio/pci.h"
+#include "hw/pci/pci_device.h"
+#include "sysemu/numa.h"
+#include "hw/acpi/acpi-generic-initiator.h"
+
+OBJECT_DEFINE_TYPE_WITH_INTERFACES(AcpiGenericInitiator, acpi_generic_initiator,
+                   ACPI_GENERIC_INITIATOR, OBJECT,
+                   { TYPE_USER_CREATABLE },
+                   { NULL })
+
+OBJECT_DECLARE_SIMPLE_TYPE(AcpiGenericInitiator, ACPI_GENERIC_INITIATOR)
+
+static void acpi_generic_initiator_init(Object *obj)
+{
+    AcpiGenericInitiator *gi = ACPI_GENERIC_INITIATOR(obj);
+    gi->device = NULL;
+    gi->nodelist = NULL;
+}
+
+static void acpi_generic_initiator_finalize(Object *obj)
+{
+    AcpiGenericInitiator *gi = ACPI_GENERIC_INITIATOR(obj);
+
+    g_free(gi->device);
+    qapi_free_uint16List(gi->nodelist);
+}
+
+static void acpi_generic_initiator_set_pci_device(Object *obj, const char *val,
+                                                  Error **errp)
+{
+    AcpiGenericInitiator *gi = ACPI_GENERIC_INITIATOR(obj);
+
+    gi->device = g_strdup(val);
+}
+
+static void acpi_generic_initiator_set_acpi_device(Object *obj, const char *val,
+                                                   Error **errp)
+{
+    error_setg(errp, "Generic Initiator ACPI device not supported");
+}
+
+static void
+acpi_generic_initiator_set_host_nodes(Object *obj, Visitor *v, const char *name,
+                                      void *opaque, Error **errp)
+{
+    AcpiGenericInitiator *gi = ACPI_GENERIC_INITIATOR(obj);
+    uint16List *l;
+
+    visit_type_uint16List(v, name, &(gi->nodelist), errp);
+
+    for (l = gi->nodelist; l; l = l->next) {
+        if (l->value >= MAX_NODES) {
+            error_setg(errp, "Invalid host-nodes value: %d", l->value);
+            qapi_free_uint16List(gi->nodelist);
+            return;
+        }
+    }
+}
+
+static void acpi_generic_initiator_class_init(ObjectClass *oc, void *data)
+{
+    object_class_property_add_str(oc, ACPI_GENERIC_INITIATOR_PCI_DEVICE_PROP,
+                                  NULL, acpi_generic_initiator_set_pci_device);
+    object_class_property_add_str(oc, ACPI_GENERIC_INITIATOR_ACPI_DEVICE_PROP,
+                                  NULL, acpi_generic_initiator_set_acpi_device);
+    object_class_property_add(oc, ACPI_GENERIC_INITIATOR_HOSTNODE_PROP, "int",
+        NULL,
+        acpi_generic_initiator_set_host_nodes,
+        NULL, NULL);
+}
hw/acpi/meson.build +1 -0
index fc1b952379..2268589519 100644
@@ -1,5 +1,6 @@
 acpi_ss = ss.source_set()
 acpi_ss.add(files(
+  'acpi-generic-initiator.c',
   'acpi_interface.c',
   'aml-build.c',
   'bios-linker-loader.c',
include/hw/acpi/acpi-generic-initiator.h new file +30 -0
new file mode 100644
index 0000000000..db3ed02c80
@@ -0,0 +1,30 @@
+#ifndef ACPI_GENERIC_INITIATOR_H
+#define ACPI_GENERIC_INITIATOR_H
+
+#include "hw/mem/pc-dimm.h"
+#include "hw/acpi/bios-linker-loader.h"
+#include "qemu/uuid.h"
+#include "hw/acpi/aml-build.h"
+#include "qom/object.h"
+#include "qom/object_interfaces.h"
+
+#define TYPE_ACPI_GENERIC_INITIATOR "acpi-generic-initiator"
+
+#define ACPI_GENERIC_INITIATOR_PCI_DEVICE_PROP "pci-dev"
+#define ACPI_GENERIC_INITIATOR_ACPI_DEVICE_PROP "acpi-dev"
+#define ACPI_GENERIC_INITIATOR_HOSTNODE_PROP "host-nodes"
+
+typedef struct AcpiGenericInitiator {
+    /* private */
+    Object parent;
+
+    /* public */
+    char *device;
+    uint16List *nodelist;
+} AcpiGenericInitiator;
+
+typedef struct AcpiGenericInitiatorClass {
+        ObjectClass parent_class;
+} AcpiGenericInitiatorClass;
+
+#endif
qapi/qom.json +18 -0
index c53ef978ff..f726f5ea41 100644
@@ -794,6 +794,22 @@
 { 'struct': 'VfioUserServerProperties',
   'data': { 'socket': 'SocketAddress', 'device': 'str' } }
 
+##
+# @AcpiGenericInitiatorProperties:
+#
+# Properties for acpi-generic-initiator objects.
+#
+# @pci-dev: PCI device ID to be associated with the node
+#
+# @acpi-dev: ACPI device ID to be associated with the node
+#
+# @host-nodes: numa node list
+#
+# Since: 8.2
+##
+{ 'struct': 'AcpiGenericInitiatorProperties',
+  'data': { '*pci-dev': 'str', '*acpi-dev': 'str', 'host-nodes': ['uint16'] } }
+
 ##
 # @RngProperties:
 #
@@ -911,6 +927,7 @@
 ##
 { 'enum': 'ObjectType',
   'data': [
+    'acpi-generic-initiator',
     'authz-list',
     'authz-listfile',
     'authz-pam',
@@ -981,6 +998,7 @@
             'id': 'str' },
   'discriminator': 'qom-type',
   'data': {
+      'acpi-generic-initiator':     'AcpiGenericInitiatorProperties',
       'authz-list':                 'AuthZListProperties',
       'authz-listfile':             'AuthZListFileProperties',
       'authz-pam':                  'AuthZPAMProperties',
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH v4 1/2] qom: new object to associate device to numa node

Quoted 59 lines · On Sun, 19 Nov 2023 18:31:10 +0530 <[email protected]> wrote:

From: Ankit Agrawal <[email protected]>

NVIDIA GPU's support MIG (Mult-Instance GPUs) feature [1], which allows partitioning of the GPU device resources (including device memory) into several (upto 8) isolated instances. Each of the partitioned memory needs a dedicated NUMA node to operate. The partitions are not fixed and they can be created/deleted at runtime.

Unfortunately Linux OS does not provide a means to dynamically create/destroy NUMA nodes and such feature implementation is not expected to be trivial. The nodes that OS discovers at the boot time while parsing SRAT remains fixed. So we utilize the Generic Initiator Affinity structures that allows association between nodes and devices. Multiple GI structures per BDF is possible, allowing creation of multiple nodes by exposing unique PXM in each of these structures.

Introduce a new acpi-generic-initiator object to allow host admin provide the device and the corresponding NUMA nodes. Qemu maintain this association and use this object to build the requisite GI Affinity Structure.

An admin can provide the range of nodes through a uint16 array host-nodes and link it to a device by providing its id. Currently, only PCI device is supported and an error is returned for acpi device. The following sample creates 8 nodes and link them to the PCI device dev0:

(quoted diff) +0 -10
-numa node,nodeid=2 \
-numa node,nodeid=3 \
-numa node,nodeid=4 \
-numa node,nodeid=5 \
-numa node,nodeid=6 \
-numa node,nodeid=7 \
-numa node,nodeid=8 \
-numa node,nodeid=9 \
-device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.0,addr=04.0,rombar=0,id=dev0 \
-object acpi-generic-initiator,id=gi0,pci-dev=dev0,host-nodes=2-9 \

[1] https://www.nvidia.com/en-in/technologies/multi-instance-gpu

---
 hw/acpi/acpi-generic-initiator.c         | 84 ++++++++++++++++++++++++
 hw/acpi/meson.build                      |  1 +
 include/hw/acpi/acpi-generic-initiator.h | 30 +++++++++
 qapi/qom.json                            | 18 +++++
 4 files changed, 133 insertions(+)
 create mode 100644 hw/acpi/acpi-generic-initiator.c
 create mode 100644 include/hw/acpi/acpi-generic-initiator.h
hw/acpi/acpi-generic-initiator.c new file +5 -0
new file mode 100644
index 0000000000..5ea51cb81e
@@ -0,0 +1,84 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved
+ */
+

hw/acpi/acpi-generic-initiator.c line 15

Quoted 10 lines
hw/acpi/acpi-generic-initiator.c new file +10 -0
+#include "qemu/osdep.h"
+#include "hw/qdev-properties.h"
+#include "qapi/error.h"
+#include "qapi/qapi-builtin-visit.h"
+#include "qapi/visitor.h"
+#include "qom/object_interfaces.h"
+#include "qom/object.h"
+#include "hw/qdev-core.h"
+#include "hw/vfio/vfio-common.h"
+#include "hw/vfio/pci.h"

There's nothing related to vfio here except for the example use case, surely you don't need the above two headers.

Quoted 54 lines
hw/acpi/acpi-generic-initiator.c new file +54 -0
+#include "hw/pci/pci_device.h"
+#include "sysemu/numa.h"
+#include "hw/acpi/acpi-generic-initiator.h"
+
+OBJECT_DEFINE_TYPE_WITH_INTERFACES(AcpiGenericInitiator, acpi_generic_initiator,
+                   ACPI_GENERIC_INITIATOR, OBJECT,
+                   { TYPE_USER_CREATABLE },
+                   { NULL })
+
+OBJECT_DECLARE_SIMPLE_TYPE(AcpiGenericInitiator, ACPI_GENERIC_INITIATOR)
+
+static void acpi_generic_initiator_init(Object *obj)
+{
+    AcpiGenericInitiator *gi = ACPI_GENERIC_INITIATOR(obj);
+    gi->device = NULL;
+    gi->nodelist = NULL;
+}
+
+static void acpi_generic_initiator_finalize(Object *obj)
+{
+    AcpiGenericInitiator *gi = ACPI_GENERIC_INITIATOR(obj);
+
+    g_free(gi->device);
+    qapi_free_uint16List(gi->nodelist);
+}
+
+static void acpi_generic_initiator_set_pci_device(Object *obj, const char *val,
+                                                  Error **errp)
+{
+    AcpiGenericInitiator *gi = ACPI_GENERIC_INITIATOR(obj);
+
+    gi->device = g_strdup(val);
+}
+
+static void acpi_generic_initiator_set_acpi_device(Object *obj, const char *val,
+                                                   Error **errp)
+{
+    error_setg(errp, "Generic Initiator ACPI device not supported");
+}
+
+static void
+acpi_generic_initiator_set_host_nodes(Object *obj, Visitor *v, const char *name,
+                                      void *opaque, Error **errp)
+{
+    AcpiGenericInitiator *gi = ACPI_GENERIC_INITIATOR(obj);
+    uint16List *l;
+
+    visit_type_uint16List(v, name, &(gi->nodelist), errp);
+
+    for (l = gi->nodelist; l; l = l->next) {
+        if (l->value >= MAX_NODES) {
+            error_setg(errp, "Invalid host-nodes value: %d", l->value);
+            qapi_free_uint16List(gi->nodelist);
+            return;

hw/acpi/acpi-generic-initiator.c line 79

Quoted 10 lines
hw/acpi/acpi-generic-initiator.c new file +10 -0
+        }
+    }
+}
+
+static void acpi_generic_initiator_class_init(ObjectClass *oc, void *data)
+{
+    object_class_property_add_str(oc, ACPI_GENERIC_INITIATOR_PCI_DEVICE_PROP,
+                                  NULL, acpi_generic_initiator_set_pci_device);
+    object_class_property_add_str(oc, ACPI_GENERIC_INITIATOR_ACPI_DEVICE_PROP,
+                                  NULL, acpi_generic_initiator_set_acpi_device);

This doesn't allow introspection of acpi-dev support, the property shouldn't be added until the support is available. At least we've thought about it now and we might use a comment to describe the intention that ACPI devices could be supported by this option in the future.

Quoted 62 lines
hw/acpi/acpi-generic-initiator.c new file +5 -0
+    object_class_property_add(oc, ACPI_GENERIC_INITIATOR_HOSTNODE_PROP, "int",
+        NULL,
+        acpi_generic_initiator_set_host_nodes,
+        NULL, NULL);
+}
hw/acpi/meson.build +1 -0
index fc1b952379..2268589519 100644
@@ -1,5 +1,6 @@
 acpi_ss = ss.source_set()
 acpi_ss.add(files(
+  'acpi-generic-initiator.c',
   'acpi_interface.c',
   'aml-build.c',
   'bios-linker-loader.c',
include/hw/acpi/acpi-generic-initiator.h new file +30 -0
new file mode 100644
index 0000000000..db3ed02c80
@@ -0,0 +1,30 @@
+#ifndef ACPI_GENERIC_INITIATOR_H
+#define ACPI_GENERIC_INITIATOR_H
+
+#include "hw/mem/pc-dimm.h"
+#include "hw/acpi/bios-linker-loader.h"
+#include "qemu/uuid.h"
+#include "hw/acpi/aml-build.h"
+#include "qom/object.h"
+#include "qom/object_interfaces.h"
+
+#define TYPE_ACPI_GENERIC_INITIATOR "acpi-generic-initiator"
+
+#define ACPI_GENERIC_INITIATOR_PCI_DEVICE_PROP "pci-dev"
+#define ACPI_GENERIC_INITIATOR_ACPI_DEVICE_PROP "acpi-dev"
+#define ACPI_GENERIC_INITIATOR_HOSTNODE_PROP "host-nodes"
+
+typedef struct AcpiGenericInitiator {
+    /* private */
+    Object parent;
+
+    /* public */
+    char *device;
+    uint16List *nodelist;
+} AcpiGenericInitiator;
+
+typedef struct AcpiGenericInitiatorClass {
+        ObjectClass parent_class;
+} AcpiGenericInitiatorClass;
+
+#endif
qapi/qom.json +2 -0
index c53ef978ff..f726f5ea41 100644
@@ -794,6 +794,22 @@
 { 'struct': 'VfioUserServerProperties',
   'data': { 'socket': 'SocketAddress', 'device': 'str' } }
 
+##
+# @AcpiGenericInitiatorProperties:

qapi/qom.json line 808

Quoted 10 lines
qapi/qom.json +10 -0
+#
+# Properties for acpi-generic-initiator objects.
+#
+# @pci-dev: PCI device ID to be associated with the node
+#
+# @acpi-dev: ACPI device ID to be associated with the node
+#
+# @host-nodes: numa node list
+#
+# Since: 8.2

8.2 is in freeze, this is 9.0 material. Thanks,

Alex

Quoted 23 lines
qapi/qom.json +6 -0
+##
+{ 'struct': 'AcpiGenericInitiatorProperties',
+  'data': { '*pci-dev': 'str', '*acpi-dev': 'str', 'host-nodes': ['uint16'] } }
+
 ##
 # @RngProperties:
 #
@@ -911,6 +927,7 @@
 ##
 { 'enum': 'ObjectType',
   'data': [
+    'acpi-generic-initiator',
     'authz-list',
     'authz-listfile',
     'authz-pam',
@@ -981,6 +998,7 @@
             'id': 'str' },
   'discriminator': 'qom-type',
   'data': {
+      'acpi-generic-initiator':     'AcpiGenericInitiatorProperties',
       'authz-list':                 'AuthZListProperties',
       'authz-listfile':             'AuthZListFileProperties',
       'authz-pam':                  'AuthZPAMProperties',
#3 lore raw

Re: [PATCH v4 1/2] qom: new object to associate device to numa node

In reply to [email protected]
Quoted 62 lines · <[email protected]> writes:

From: Ankit Agrawal <[email protected]>

NVIDIA GPU's support MIG (Mult-Instance GPUs) feature [1], which allows partitioning of the GPU device resources (including device memory) into several (upto 8) isolated instances. Each of the partitioned memory needs a dedicated NUMA node to operate. The partitions are not fixed and they can be created/deleted at runtime.

Unfortunately Linux OS does not provide a means to dynamically create/destroy NUMA nodes and such feature implementation is not expected to be trivial. The nodes that OS discovers at the boot time while parsing SRAT remains fixed. So we utilize the Generic Initiator Affinity structures that allows association between nodes and devices. Multiple GI structures per BDF is possible, allowing creation of multiple nodes by exposing unique PXM in each of these structures.

Introduce a new acpi-generic-initiator object to allow host admin provide the device and the corresponding NUMA nodes. Qemu maintain this association and use this object to build the requisite GI Affinity Structure.

An admin can provide the range of nodes through a uint16 array host-nodes and link it to a device by providing its id. Currently, only PCI device is supported and an error is returned for acpi device. The following sample creates 8 nodes and link them to the PCI device dev0:

(quoted diff) +0 -10
-numa node,nodeid=2 \
-numa node,nodeid=3 \
-numa node,nodeid=4 \
-numa node,nodeid=5 \
-numa node,nodeid=6 \
-numa node,nodeid=7 \
-numa node,nodeid=8 \
-numa node,nodeid=9 \
-device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.0,addr=04.0,rombar=0,id=dev0 \
-object acpi-generic-initiator,id=gi0,pci-dev=dev0,host-nodes=2-9 \

[1] https://www.nvidia.com/en-in/technologies/multi-instance-gpu

---
 hw/acpi/acpi-generic-initiator.c         | 84 ++++++++++++++++++++++++
 hw/acpi/meson.build                      |  1 +
 include/hw/acpi/acpi-generic-initiator.h | 30 +++++++++
 qapi/qom.json                            | 18 +++++
 4 files changed, 133 insertions(+)
 create mode 100644 hw/acpi/acpi-generic-initiator.c
 create mode 100644 include/hw/acpi/acpi-generic-initiator.h
hw/acpi/acpi-generic-initiator.c new file +8 -0
new file mode 100644
index 0000000000..5ea51cb81e
@@ -0,0 +1,84 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved
+ */
+
+#include "qemu/osdep.h"
+#include "hw/qdev-properties.h"
+#include "qapi/error.h"

hw/acpi/acpi-generic-initiator.c line 18

Quoted 10 lines
hw/acpi/acpi-generic-initiator.c new file +10 -0
+#include "qapi/qapi-builtin-visit.h"
+#include "qapi/visitor.h"
+#include "qom/object_interfaces.h"
+#include "qom/object.h"
+#include "hw/qdev-core.h"
+#include "hw/vfio/vfio-common.h"
+#include "hw/vfio/pci.h"
+#include "hw/pci/pci_device.h"
+#include "sysemu/numa.h"
+#include "hw/acpi/acpi-generic-initiator.h"

Several #include are superfluous. This compiles for me:

  #include "qemu/osdep.h"
  #include "hw/acpi/acpi-generic-initiator.h"
  #include "hw/pci/pci_device.h"
  #include "qapi/error.h"
  #include "qapi/qapi-builtin-visit.h"
  #include "qapi/visitor.h"
  #include "qemu/error-report.h"
  #include "sysemu/numa.h"

Yes, the alphabetical order is intentional.

Quoted 19 lines
hw/acpi/acpi-generic-initiator.c new file +19 -0
+
+OBJECT_DEFINE_TYPE_WITH_INTERFACES(AcpiGenericInitiator, acpi_generic_initiator,
+                   ACPI_GENERIC_INITIATOR, OBJECT,
+                   { TYPE_USER_CREATABLE },
+                   { NULL })
+
+OBJECT_DECLARE_SIMPLE_TYPE(AcpiGenericInitiator, ACPI_GENERIC_INITIATOR)
+
+static void acpi_generic_initiator_init(Object *obj)
+{
+    AcpiGenericInitiator *gi = ACPI_GENERIC_INITIATOR(obj);
+    gi->device = NULL;
+    gi->nodelist = NULL;
+}
+
+static void acpi_generic_initiator_finalize(Object *obj)
+{
+    AcpiGenericInitiator *gi = ACPI_GENERIC_INITIATOR(obj);
+

hw/acpi/acpi-generic-initiator.c line 47

Quoted 10 lines
hw/acpi/acpi-generic-initiator.c new file +10 -0
+    g_free(gi->device);
+    qapi_free_uint16List(gi->nodelist);
+}
+
+static void acpi_generic_initiator_set_pci_device(Object *obj, const char *val,
+                                                  Error **errp)
+{
+    AcpiGenericInitiator *gi = ACPI_GENERIC_INITIATOR(obj);
+
+    gi->device = g_strdup(val);

The property is named "pci-dev", but the C member is called @device. Making developers remember this mapping is not nice. Suggest to rename to @pci_dev.

hw/acpi/acpi-generic-initiator.c line 54

Quoted 7 lines
hw/acpi/acpi-generic-initiator.c new file +7 -0
+}
+
+static void acpi_generic_initiator_set_acpi_device(Object *obj, const char *val,
+                                                   Error **errp)
+{
+    error_setg(errp, "Generic Initiator ACPI device not supported");
+}

Let's add the property when it actually works. More below at [*].

Quoted 7 lines
hw/acpi/acpi-generic-initiator.c new file +7 -0
+
+static void
+acpi_generic_initiator_set_host_nodes(Object *obj, Visitor *v, const char *name,
+                                      void *opaque, Error **errp)
+{
+    AcpiGenericInitiator *gi = ACPI_GENERIC_INITIATOR(obj);
+    uint16List *l;

hw/acpi/acpi-generic-initiator.c line 71

Quoted 10 lines
hw/acpi/acpi-generic-initiator.c new file +10 -0
+
+    visit_type_uint16List(v, name, &(gi->nodelist), errp);
+
+    for (l = gi->nodelist; l; l = l->next) {
+        if (l->value >= MAX_NODES) {
+            error_setg(errp, "Invalid host-nodes value: %d", l->value);
+            qapi_free_uint16List(gi->nodelist);
+            return;
+        }
+    }

Why not store the nodes in a bitset, like
host_memory_backend_set_host_nodes() does?

hw/acpi/acpi-generic-initiator.c line 79

Quoted 8 lines
hw/acpi/acpi-generic-initiator.c new file +8 -0
+}
+
+static void acpi_generic_initiator_class_init(ObjectClass *oc, void *data)
+{
+    object_class_property_add_str(oc, ACPI_GENERIC_INITIATOR_PCI_DEVICE_PROP,
+                                  NULL, acpi_generic_initiator_set_pci_device);
+    object_class_property_add_str(oc, ACPI_GENERIC_INITIATOR_ACPI_DEVICE_PROP,
+                                  NULL, acpi_generic_initiator_set_acpi_device);

[*] Drop this one.

Quoted 27 lines
hw/acpi/acpi-generic-initiator.c new file +5 -0
+    object_class_property_add(oc, ACPI_GENERIC_INITIATOR_HOSTNODE_PROP, "int",
+        NULL,
+        acpi_generic_initiator_set_host_nodes,
+        NULL, NULL);
+}
hw/acpi/meson.build +1 -0
index fc1b952379..2268589519 100644
@@ -1,5 +1,6 @@
 acpi_ss = ss.source_set()
 acpi_ss.add(files(
+  'acpi-generic-initiator.c',
   'acpi_interface.c',
   'aml-build.c',
   'bios-linker-loader.c',
include/hw/acpi/acpi-generic-initiator.h new file +5 -0
new file mode 100644
index 0000000000..db3ed02c80
@@ -0,0 +1,30 @@
+#ifndef ACPI_GENERIC_INITIATOR_H
+#define ACPI_GENERIC_INITIATOR_H
+
+#include "hw/mem/pc-dimm.h"
+#include "hw/acpi/bios-linker-loader.h"

include/hw/acpi/acpi-generic-initiator.h line 15

Quoted 10 lines
include/hw/acpi/acpi-generic-initiator.h new file +10 -0
+#include "qemu/uuid.h"
+#include "hw/acpi/aml-build.h"
+#include "qom/object.h"
+#include "qom/object_interfaces.h"
+
+#define TYPE_ACPI_GENERIC_INITIATOR "acpi-generic-initiator"
+
+#define ACPI_GENERIC_INITIATOR_PCI_DEVICE_PROP "pci-dev"
+#define ACPI_GENERIC_INITIATOR_ACPI_DEVICE_PROP "acpi-dev"
+#define ACPI_GENERIC_INITIATOR_HOSTNODE_PROP "host-nodes"

These three macros have exactly one use each. Get rid of them, please.

Quoted 21 lines
include/hw/acpi/acpi-generic-initiator.h new file +15 -0
+
+typedef struct AcpiGenericInitiator {
+    /* private */
+    Object parent;
+
+    /* public */
+    char *device;
+    uint16List *nodelist;
+} AcpiGenericInitiator;
+
+typedef struct AcpiGenericInitiatorClass {
+        ObjectClass parent_class;
+} AcpiGenericInitiatorClass;
+
+#endif
qapi/qom.json +0 -0
index c53ef978ff..f726f5ea41 100644
@@ -794,6 +794,22 @@
 { 'struct': 'VfioUserServerProperties',

qapi/qom.json line 804

Quoted 10 lines
qapi/qom.json +8 -0
   'data': { 'socket': 'SocketAddress', 'device': 'str' } }
 
+##
+# @AcpiGenericInitiatorProperties:
+#
+# Properties for acpi-generic-initiator objects.
+#
+# @pci-dev: PCI device ID to be associated with the node
+#
+# @acpi-dev: ACPI device ID to be associated with the node

[*] Drop this one.

qapi/qom.json line 808

Quoted 4 lines
qapi/qom.json +4 -0
+#
+# @host-nodes: numa node list
+#
+# Since: 8.2

9.0

qapi/qom.json line 811

Quoted 3 lines
qapi/qom.json +3 -0
+##
+{ 'struct': 'AcpiGenericInitiatorProperties',
+  'data': { '*pci-dev': 'str', '*acpi-dev': 'str', 'host-nodes': ['uint16'] } }

Long line. Better:

     'data': { '*pci-dev': 'str',
               '*acpi-dev': 'str',
               'host-nodes': ['uint16'] } }
Quoted 20 lines
qapi/qom.json +3 -0
+
 ##
 # @RngProperties:
 #
@@ -911,6 +927,7 @@
 ##
 { 'enum': 'ObjectType',
   'data': [
+    'acpi-generic-initiator',
     'authz-list',
     'authz-listfile',
     'authz-pam',
@@ -981,6 +998,7 @@
             'id': 'str' },
   'discriminator': 'qom-type',
   'data': {
+      'acpi-generic-initiator':     'AcpiGenericInitiatorProperties',
       'authz-list':                 'AuthZListProperties',
       'authz-listfile':             'AuthZListFileProperties',
       'authz-pam':                  'AuthZPAMProperties',
#4 lore raw

PATCH v4 2/2 hw/acpi: Implement the SRAT GI affinity structure

In reply to [email protected]

From: Ankit Agrawal <[email protected]>

ACPI spec provides a scheme to associate "Generic Initiators" [1] (e.g. heterogeneous processors and accelerators, GPUs, and I/O devices with integrated compute or DMA engines GPUs) with Proximity Domains. This is achieved using Generic Initiator Affinity Structure in SRAT. During bootup, Linux kernel parse the ACPI SRAT to determine the PXM ids and create a NUMA node for each unique PXM ID encountered. Qemu currently do not implement these structures while building SRAT.

Add GI structures while building VM ACPI SRAT. The association between devices and nodes are stored using acpi-generic-initiator object. Lookup presence of all such objects and use them to build these structures.

The structure needs a PCI device handle [2] that consists of the device BDF. The vfio-pci device corresponding to the acpi-generic-initiator object is located to determine the BDF.

[1] ACPI Spec 6.3, Section 5.2.16.6
[2] ACPI Spec 6.3, Table 5.80

---
 hw/acpi/acpi-generic-initiator.c         | 100 +++++++++++++++++++++++
 hw/arm/virt-acpi-build.c                 |   3 +
 include/hw/acpi/acpi-generic-initiator.h |  26 ++++++
 3 files changed, 129 insertions(+)
hw/acpi/acpi-generic-initiator.c +100 -0
index 5ea51cb81e..a9222438ec 100644
@@ -16,6 +16,7 @@
 #include "hw/pci/pci_device.h"
 #include "sysemu/numa.h"
 #include "hw/acpi/acpi-generic-initiator.h"
+#include "qemu/error-report.h"
 
 OBJECT_DEFINE_TYPE_WITH_INTERFACES(AcpiGenericInitiator, acpi_generic_initiator,
                    ACPI_GENERIC_INITIATOR, OBJECT,
@@ -82,3 +83,102 @@ static void acpi_generic_initiator_class_init(ObjectClass *oc, void *data)
         acpi_generic_initiator_set_host_nodes,
         NULL, NULL);
 }
+
+static int acpi_generic_initiator_list(Object *obj, void *opaque)
+{
+    GSList **list = opaque;
+
+    if (object_dynamic_cast(obj, TYPE_ACPI_GENERIC_INITIATOR)) {
+        *list = g_slist_append(*list, ACPI_GENERIC_INITIATOR(obj));
+    }
+
+    object_child_foreach(obj, acpi_generic_initiator_list, opaque);
+    return 0;
+}
+
+/*
+ * Identify Generic Initiator objects and link them into the list which is
+ * returned to the caller.
+ *
+ * Note: it is the caller's responsibility to free the list to avoid
+ * memory leak.
+ */
+static GSList *acpi_generic_initiator_get_list(void)
+{
+    GSList *list = NULL;
+
+    object_child_foreach(object_get_root(), acpi_generic_initiator_list, &list);
+    return list;
+}
+
+/*
+ * ACPI 6.3:
+ * Table 5-78 Generic Initiator Affinity Structure
+ */
+static
+void build_srat_generic_pci_initiator_affinity(GArray *table_data, int node,
+                                               PCIDeviceHandle *handle)
+{
+    uint8_t index;
+
+    build_append_int_noprefix(table_data, 5, 1);  /* Type */
+    build_append_int_noprefix(table_data, 32, 1); /* Length */
+    build_append_int_noprefix(table_data, 0, 1);  /* Reserved */
+    build_append_int_noprefix(table_data, 1, 1);  /* Device Handle Type: PCI */
+    build_append_int_noprefix(table_data, node, 4);  /* Proximity Domain */
+
+    /* Device Handle - PCI */
+    build_append_int_noprefix(table_data, handle->segment, 2);
+    build_append_int_noprefix(table_data, handle->bdf, 2);
+    for (index = 0; index < 12; index++) {
+        build_append_int_noprefix(table_data, 0, 1);
+    }
+
+    build_append_int_noprefix(table_data, GEN_AFFINITY_ENABLED, 4); /* Flags */
+    build_append_int_noprefix(table_data, 0, 4);     /* Reserved */
+}
+
+void build_srat_generic_pci_initiator(GArray *table_data)
+{
+    GSList *gi_list, *list = acpi_generic_initiator_get_list();
+    AcpiGenericInitiator *gi;
+
+    for (gi_list = list; gi_list; gi_list = gi_list->next) {
+        Object *o;
+        uint16List *l;
+        PCIDevice *pci_dev;
+        bool node_specified = false;
+
+        gi = gi_list->data;
+
+        /* User fails to provide a device. */
+        g_assert(gi->device);
+
+        o = object_resolve_path_type(gi->device, TYPE_PCI_DEVICE, NULL);
+        if (!o) {
+            error_printf("Specified device must be a PCI device.\n");
+            g_assert(o);
+        }
+        pci_dev = PCI_DEVICE(o);
+
+        for (l = gi->nodelist; l; l = l->next) {
+            PCIDeviceHandle dev_handle;
+            dev_handle.segment = 0;
+            dev_handle.bdf = PCI_BUILD_BDF(pci_bus_num(pci_get_bus(pci_dev)),
+                                                       pci_dev->devfn);
+            build_srat_generic_pci_initiator_affinity(table_data,
+                                                      l->value, &dev_handle);
+            node_specified = true;
+        }
+
+        if (!node_specified) {
+            error_report("Generic Initiator device 0:%x:%x.%x has no associated"
+                         " NUMA node.", pci_bus_num(pci_get_bus(pci_dev)),
+                         PCI_SLOT(pci_dev->devfn), PCI_FUNC(pci_dev->devfn));
+            error_printf("Specify NUMA node with -nodelist option.\n");
+            g_assert(node_specified);
+        }
+    }
+
+    g_slist_free(list);
+}
hw/arm/virt-acpi-build.c +3 -0
index 8bc35a483c..00d77327e0 100644
@@ -58,6 +58,7 @@
 #include "migration/vmstate.h"
 #include "hw/acpi/ghes.h"
 #include "hw/acpi/viot.h"
+#include "hw/acpi/acpi-generic-initiator.h"
 
 #define ARM_SPI_BASE 32
 
@@ -558,6 +559,8 @@ build_srat(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
         }
     }
 
+    build_srat_generic_pci_initiator(table_data);
+
     if (ms->nvdimms_state->is_enabled) {
         nvdimm_build_srat(table_data);
     }
include/hw/acpi/acpi-generic-initiator.h +26 -0
index db3ed02c80..6fdaf887cd 100644
@@ -27,4 +27,30 @@ typedef struct AcpiGenericInitiatorClass {
         ObjectClass parent_class;
 } AcpiGenericInitiatorClass;
 
+/*
+ * ACPI 6.3:
+ * Table 5-81 Flags – Generic Initiator Affinity Structure
+ */
+typedef enum {
+    GEN_AFFINITY_ENABLED = (1 << 0), /*
+                                      * If clear, the OSPM ignores the contents
+                                      * of the Generic Initiator/Port Affinity
+                                      * Structure. This allows system firmware
+                                      * to populate the SRAT with a static
+                                      * number of structures, but only enable
+                                      * them as necessary.
+                                      */
+} GenericAffinityFlags;
+
+/*
+ * ACPI 6.3:
+ * Table 5-80 Device Handle - PCI
+ */
+typedef struct PCIDeviceHandle {
+    uint16_t segment;
+    uint16_t bdf;
+} PCIDeviceHandle;
+
+void build_srat_generic_pci_initiator(GArray *table_data);
+
 #endif
Signature · 1 line
2.34.1
#5 lore raw

Re: [PATCH v4 2/2] hw/acpi: Implement the SRAT GI affinity structure

Quoted 127 lines · On Sun, 19 Nov 2023 18:31:11 +0530 <[email protected]> wrote:

From: Ankit Agrawal <[email protected]>

ACPI spec provides a scheme to associate "Generic Initiators" [1] (e.g. heterogeneous processors and accelerators, GPUs, and I/O devices with integrated compute or DMA engines GPUs) with Proximity Domains. This is achieved using Generic Initiator Affinity Structure in SRAT. During bootup, Linux kernel parse the ACPI SRAT to determine the PXM ids and create a NUMA node for each unique PXM ID encountered. Qemu currently do not implement these structures while building SRAT.

Add GI structures while building VM ACPI SRAT. The association between devices and nodes are stored using acpi-generic-initiator object. Lookup presence of all such objects and use them to build these structures.

The structure needs a PCI device handle [2] that consists of the device BDF. The vfio-pci device corresponding to the acpi-generic-initiator object is located to determine the BDF.

[1] ACPI Spec 6.3, Section 5.2.16.6
[2] ACPI Spec 6.3, Table 5.80

---
 hw/acpi/acpi-generic-initiator.c         | 100 +++++++++++++++++++++++
 hw/arm/virt-acpi-build.c                 |   3 +
 include/hw/acpi/acpi-generic-initiator.h |  26 ++++++
 3 files changed, 129 insertions(+)
hw/acpi/acpi-generic-initiator.c +84 -0
index 5ea51cb81e..a9222438ec 100644
@@ -16,6 +16,7 @@
 #include "hw/pci/pci_device.h"
 #include "sysemu/numa.h"
 #include "hw/acpi/acpi-generic-initiator.h"
+#include "qemu/error-report.h"
 
 OBJECT_DEFINE_TYPE_WITH_INTERFACES(AcpiGenericInitiator, acpi_generic_initiator,
                    ACPI_GENERIC_INITIATOR, OBJECT,
@@ -82,3 +83,102 @@ static void acpi_generic_initiator_class_init(ObjectClass *oc, void *data)
         acpi_generic_initiator_set_host_nodes,
         NULL, NULL);
 }
+
+static int acpi_generic_initiator_list(Object *obj, void *opaque)
+{
+    GSList **list = opaque;
+
+    if (object_dynamic_cast(obj, TYPE_ACPI_GENERIC_INITIATOR)) {
+        *list = g_slist_append(*list, ACPI_GENERIC_INITIATOR(obj));
+    }
+
+    object_child_foreach(obj, acpi_generic_initiator_list, opaque);
+    return 0;
+}
+
+/*
+ * Identify Generic Initiator objects and link them into the list which is
+ * returned to the caller.
+ *
+ * Note: it is the caller's responsibility to free the list to avoid
+ * memory leak.
+ */
+static GSList *acpi_generic_initiator_get_list(void)
+{
+    GSList *list = NULL;
+
+    object_child_foreach(object_get_root(), acpi_generic_initiator_list, &list);
+    return list;
+}
+
+/*
+ * ACPI 6.3:
+ * Table 5-78 Generic Initiator Affinity Structure
+ */
+static
+void build_srat_generic_pci_initiator_affinity(GArray *table_data, int node,
+                                               PCIDeviceHandle *handle)
+{
+    uint8_t index;
+
+    build_append_int_noprefix(table_data, 5, 1);  /* Type */
+    build_append_int_noprefix(table_data, 32, 1); /* Length */
+    build_append_int_noprefix(table_data, 0, 1);  /* Reserved */
+    build_append_int_noprefix(table_data, 1, 1);  /* Device Handle Type: PCI */
+    build_append_int_noprefix(table_data, node, 4);  /* Proximity Domain */
+
+    /* Device Handle - PCI */
+    build_append_int_noprefix(table_data, handle->segment, 2);
+    build_append_int_noprefix(table_data, handle->bdf, 2);
+    for (index = 0; index < 12; index++) {
+        build_append_int_noprefix(table_data, 0, 1);
+    }
+
+    build_append_int_noprefix(table_data, GEN_AFFINITY_ENABLED, 4); /* Flags */
+    build_append_int_noprefix(table_data, 0, 4);     /* Reserved */
+}
+
+void build_srat_generic_pci_initiator(GArray *table_data)
+{
+    GSList *gi_list, *list = acpi_generic_initiator_get_list();
+    AcpiGenericInitiator *gi;
+
+    for (gi_list = list; gi_list; gi_list = gi_list->next) {
+        Object *o;
+        uint16List *l;
+        PCIDevice *pci_dev;
+        bool node_specified = false;
+
+        gi = gi_list->data;
+
+        /* User fails to provide a device. */
+        g_assert(gi->device);
+
+        o = object_resolve_path_type(gi->device, TYPE_PCI_DEVICE, NULL);
+        if (!o) {
+            error_printf("Specified device must be a PCI device.\n");
+            g_assert(o);
+        }
+        pci_dev = PCI_DEVICE(o);
+
+        for (l = gi->nodelist; l; l = l->next) {
+            PCIDeviceHandle dev_handle;
+            dev_handle.segment = 0;
+            dev_handle.bdf = PCI_BUILD_BDF(pci_bus_num(pci_get_bus(pci_dev)),
+                                                       pci_dev->devfn);

hw/acpi/acpi-generic-initiator.c static void acpi_generic_initiator_class_init(ObjectClass *oc, void *data) · line 178

Quoted 10 lines
hw/acpi/acpi-generic-initiator.c +10 -0
+            build_srat_generic_pci_initiator_affinity(table_data,
+                                                      l->value, &dev_handle);
+            node_specified = true;
+        }
+
+        if (!node_specified) {
+            error_report("Generic Initiator device 0:%x:%x.%x has no associated"
+                         " NUMA node.", pci_bus_num(pci_get_bus(pci_dev)),
+                         PCI_SLOT(pci_dev->devfn), PCI_FUNC(pci_dev->devfn));
+            error_printf("Specify NUMA node with -nodelist option.\n");

No such option, -nodelist?

hw/acpi/acpi-generic-initiator.c static void acpi_generic_initiator_class_init(ObjectClass *oc, void *data) · line 179

Quoted 1 line
hw/acpi/acpi-generic-initiator.c +1 -0
+            g_assert(node_specified);

I won't claim expertise in QEMU error handling, but an assert is a pretty harsh way to handle failures. Thanks,

Alex

Quoted 61 lines
hw/acpi/acpi-generic-initiator.c +5 -0
+        }
+    }
+
+    g_slist_free(list);
+}
hw/arm/virt-acpi-build.c +3 -0
index 8bc35a483c..00d77327e0 100644
@@ -58,6 +58,7 @@
 #include "migration/vmstate.h"
 #include "hw/acpi/ghes.h"
 #include "hw/acpi/viot.h"
+#include "hw/acpi/acpi-generic-initiator.h"
 
 #define ARM_SPI_BASE 32
 
@@ -558,6 +559,8 @@ build_srat(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
         }
     }
 
+    build_srat_generic_pci_initiator(table_data);
+
     if (ms->nvdimms_state->is_enabled) {
         nvdimm_build_srat(table_data);
     }
include/hw/acpi/acpi-generic-initiator.h +26 -0
index db3ed02c80..6fdaf887cd 100644
@@ -27,4 +27,30 @@ typedef struct AcpiGenericInitiatorClass {
         ObjectClass parent_class;
 } AcpiGenericInitiatorClass;
 
+/*
+ * ACPI 6.3:
+ * Table 5-81 Flags – Generic Initiator Affinity Structure
+ */
+typedef enum {
+    GEN_AFFINITY_ENABLED = (1 << 0), /*
+                                      * If clear, the OSPM ignores the contents
+                                      * of the Generic Initiator/Port Affinity
+                                      * Structure. This allows system firmware
+                                      * to populate the SRAT with a static
+                                      * number of structures, but only enable
+                                      * them as necessary.
+                                      */
+} GenericAffinityFlags;
+
+/*
+ * ACPI 6.3:
+ * Table 5-80 Device Handle - PCI
+ */
+typedef struct PCIDeviceHandle {
+    uint16_t segment;
+    uint16_t bdf;
+} PCIDeviceHandle;
+
+void build_srat_generic_pci_initiator(GArray *table_data);
+
 #endif
#6 lore raw

Re: [PATCH v4 2/2] hw/acpi: Implement the SRAT GI affinity structure

In reply to [email protected]
Quoted 59 lines · <[email protected]> writes:

From: Ankit Agrawal <[email protected]>

ACPI spec provides a scheme to associate "Generic Initiators" [1] (e.g. heterogeneous processors and accelerators, GPUs, and I/O devices with integrated compute or DMA engines GPUs) with Proximity Domains. This is achieved using Generic Initiator Affinity Structure in SRAT. During bootup, Linux kernel parse the ACPI SRAT to determine the PXM ids and create a NUMA node for each unique PXM ID encountered. Qemu currently do not implement these structures while building SRAT.

Add GI structures while building VM ACPI SRAT. The association between devices and nodes are stored using acpi-generic-initiator object. Lookup presence of all such objects and use them to build these structures.

The structure needs a PCI device handle [2] that consists of the device BDF. The vfio-pci device corresponding to the acpi-generic-initiator object is located to determine the BDF.

[1] ACPI Spec 6.3, Section 5.2.16.6
[2] ACPI Spec 6.3, Table 5.80

---
 hw/acpi/acpi-generic-initiator.c         | 100 +++++++++++++++++++++++
 hw/arm/virt-acpi-build.c                 |   3 +
 include/hw/acpi/acpi-generic-initiator.h |  26 ++++++
 3 files changed, 129 insertions(+)
hw/acpi/acpi-generic-initiator.c +16 -0
index 5ea51cb81e..a9222438ec 100644
@@ -16,6 +16,7 @@
 #include "hw/pci/pci_device.h"
 #include "sysemu/numa.h"
 #include "hw/acpi/acpi-generic-initiator.h"
+#include "qemu/error-report.h"
 
 OBJECT_DEFINE_TYPE_WITH_INTERFACES(AcpiGenericInitiator, acpi_generic_initiator,
                    ACPI_GENERIC_INITIATOR, OBJECT,
@@ -82,3 +83,102 @@ static void acpi_generic_initiator_class_init(ObjectClass *oc, void *data)
         acpi_generic_initiator_set_host_nodes,
         NULL, NULL);
 }
+
+static int acpi_generic_initiator_list(Object *obj, void *opaque)
+{
+    GSList **list = opaque;
+
+    if (object_dynamic_cast(obj, TYPE_ACPI_GENERIC_INITIATOR)) {
+        *list = g_slist_append(*list, ACPI_GENERIC_INITIATOR(obj));
+    }
+
+    object_child_foreach(obj, acpi_generic_initiator_list, opaque);
+    return 0;
+}
+
+/*
+ * Identify Generic Initiator objects and link them into the list which is

hw/acpi/acpi-generic-initiator.c static void acpi_generic_initiator_class_init(ObjectClass *oc, void *data) · line 110

Quoted 10 lines
hw/acpi/acpi-generic-initiator.c +10 -0
+ * returned to the caller.
+ *
+ * Note: it is the caller's responsibility to free the list to avoid
+ * memory leak.
+ */
+static GSList *acpi_generic_initiator_get_list(void)
+{
+    GSList *list = NULL;
+
+    object_child_foreach(object_get_root(), acpi_generic_initiator_list, &list);

Long line.

hw/acpi/acpi-generic-initiator.c static void acpi_generic_initiator_class_init(ObjectClass *oc, void *data) · line 120

Quoted 10 lines
hw/acpi/acpi-generic-initiator.c +10 -0
+    return list;
+}
+
+/*
+ * ACPI 6.3:
+ * Table 5-78 Generic Initiator Affinity Structure
+ */
+static
+void build_srat_generic_pci_initiator_affinity(GArray *table_data, int node,
+                                               PCIDeviceHandle *handle)

Style nitpick: bad line break.

We traditionally format like

static void build_srat_generic_pci_initiator_affinity(GArray *table_data,
int node, PCIDeviceHandle *handle)

or, to avoid the long line

static void build_srat_generic_pci_initiator_affinity(GArray *table_data,
int node, PCIDeviceHandle *handle)

but there's also precedence for

static void build_srat_generic_pci_initiator_affinity(GArray *table_data, int node,
PCIDeviceHandle *handle)

Quoted 25 lines
hw/acpi/acpi-generic-initiator.c +25 -0
+{
+    uint8_t index;
+
+    build_append_int_noprefix(table_data, 5, 1);  /* Type */
+    build_append_int_noprefix(table_data, 32, 1); /* Length */
+    build_append_int_noprefix(table_data, 0, 1);  /* Reserved */
+    build_append_int_noprefix(table_data, 1, 1);  /* Device Handle Type: PCI */
+    build_append_int_noprefix(table_data, node, 4);  /* Proximity Domain */
+
+    /* Device Handle - PCI */
+    build_append_int_noprefix(table_data, handle->segment, 2);
+    build_append_int_noprefix(table_data, handle->bdf, 2);
+    for (index = 0; index < 12; index++) {
+        build_append_int_noprefix(table_data, 0, 1);
+    }
+
+    build_append_int_noprefix(table_data, GEN_AFFINITY_ENABLED, 4); /* Flags */
+    build_append_int_noprefix(table_data, 0, 4);     /* Reserved */
+}
+
+void build_srat_generic_pci_initiator(GArray *table_data)
+{
+    GSList *gi_list, *list = acpi_generic_initiator_get_list();
+    AcpiGenericInitiator *gi;
+

hw/acpi/acpi-generic-initiator.c static void acpi_generic_initiator_class_init(ObjectClass *oc, void *data) · line 155

Quoted 10 lines
hw/acpi/acpi-generic-initiator.c +10 -0
+    for (gi_list = list; gi_list; gi_list = gi_list->next) {
+        Object *o;
+        uint16List *l;
+        PCIDevice *pci_dev;
+        bool node_specified = false;
+
+        gi = gi_list->data;
+
+        /* User fails to provide a device. */
+        g_assert(gi->device);

Assertions are for programming errors, not for diagnosing or reporting user errors. Instead

           if (!gi->device) {
               error_report(...);
               exit(1);
           }

This assumes the function can only ever run duting initial startup. If that's not ensured, exit(1) is wrong, and you need to return failure instead, so the callers can do the right thing.

hw/acpi/acpi-generic-initiator.c static void acpi_generic_initiator_class_init(ObjectClass *oc, void *data) · line 160

Quoted 5 lines
hw/acpi/acpi-generic-initiator.c +5 -0
+
+        o = object_resolve_path_type(gi->device, TYPE_PCI_DEVICE, NULL);
+        if (!o) {
+            error_printf("Specified device must be a PCI device.\n");
+            g_assert(o);

Likewise.

Quoted 9 lines
hw/acpi/acpi-generic-initiator.c +9 -0
+        }
+        pci_dev = PCI_DEVICE(o);
+
+        for (l = gi->nodelist; l; l = l->next) {
+            PCIDeviceHandle dev_handle;
+            dev_handle.segment = 0;
+            dev_handle.bdf = PCI_BUILD_BDF(pci_bus_num(pci_get_bus(pci_dev)),
+                                                       pci_dev->devfn);
+            build_srat_generic_pci_initiator_affinity(table_data,

hw/acpi/acpi-generic-initiator.c static void acpi_generic_initiator_class_init(ObjectClass *oc, void *data) · line 179

Quoted 10 lines
hw/acpi/acpi-generic-initiator.c +10 -0
+                                                      l->value, &dev_handle);
+            node_specified = true;
+        }
+
+        if (!node_specified) {
+            error_report("Generic Initiator device 0:%x:%x.%x has no associated"
+                         " NUMA node.", pci_bus_num(pci_get_bus(pci_dev)),
+                         PCI_SLOT(pci_dev->devfn), PCI_FUNC(pci_dev->devfn));
+            error_printf("Specify NUMA node with -nodelist option.\n");
+            g_assert(node_specified);

Likewise.

Quoted 61 lines
hw/acpi/acpi-generic-initiator.c +5 -0
+        }
+    }
+
+    g_slist_free(list);
+}
hw/arm/virt-acpi-build.c +3 -0
index 8bc35a483c..00d77327e0 100644
@@ -58,6 +58,7 @@
 #include "migration/vmstate.h"
 #include "hw/acpi/ghes.h"
 #include "hw/acpi/viot.h"
+#include "hw/acpi/acpi-generic-initiator.h"
 
 #define ARM_SPI_BASE 32
 
@@ -558,6 +559,8 @@ build_srat(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
         }
     }
 
+    build_srat_generic_pci_initiator(table_data);
+
     if (ms->nvdimms_state->is_enabled) {
         nvdimm_build_srat(table_data);
     }
include/hw/acpi/acpi-generic-initiator.h +26 -0
index db3ed02c80..6fdaf887cd 100644
@@ -27,4 +27,30 @@ typedef struct AcpiGenericInitiatorClass {
         ObjectClass parent_class;
 } AcpiGenericInitiatorClass;
 
+/*
+ * ACPI 6.3:
+ * Table 5-81 Flags – Generic Initiator Affinity Structure
+ */
+typedef enum {
+    GEN_AFFINITY_ENABLED = (1 << 0), /*
+                                      * If clear, the OSPM ignores the contents
+                                      * of the Generic Initiator/Port Affinity
+                                      * Structure. This allows system firmware
+                                      * to populate the SRAT with a static
+                                      * number of structures, but only enable
+                                      * them as necessary.
+                                      */
+} GenericAffinityFlags;
+
+/*
+ * ACPI 6.3:
+ * Table 5-80 Device Handle - PCI
+ */
+typedef struct PCIDeviceHandle {
+    uint16_t segment;
+    uint16_t bdf;
+} PCIDeviceHandle;
+
+void build_srat_generic_pci_initiator(GArray *table_data);
+
 #endif