ACPI: Only create NUMA nodes from entries in SRAT or SRAT emulation.
Subsystems (9)
v2 cover ACPI: Only create NUMA nodes from entries in SRAT or SRAT emulation.
Here, I will use the term Proximity Domains for the ACPI description and NUMA Nodes for the in kernel representation.
ACPI 6.3 included a clarification that only Static Resource Allocation Structures in SRAT may define the existence of proximity domains (sec 5.2.16). This clarification closed a possible interpretation that other parts of ACPI (e.g. DSDT _PXM, NFIT etc) could define new proximity domains that were not also mentioned in SRAT structures.
In practice the kernel has never allowed this alternative interpretation as such nodes are only partially initialized. This is architecture specific but to take an example, on x86 alloc_node_data has not been called. Any use of them for node specific allocation, will result in a crash as the infrastructure to fallback to a node with memory is not setup.
We ran into a problem when enabling _PXM handling for PCI devices and found there were boards out there advertising devices in proximity domains that didn't exist [2].
The fix suggested in this series is to replace instances that should not 'create' new nodes with pxm_to_node. This function needs a some additional hardening against invalid inputs to make sure it is safe for use in these new callers.
Patch 1 Hardens pxm_to_node() against numa_off, and pxm entry being too large.
Patch 2-4 change the various callers not related to SRAT entries so that they set this parameter to false, so do not attempt to initialize a new NUMA node if the relevant one does not already exist.
Patch 5 is a function rename to reflect change in functionality of acpi_map_pxm_to_online_node() as it no longer creates a new map, but just does a lookup of existing maps.
Patch 6 covers the one place we do not allow the full flexibility defined in the ACPI spec. For SRAT GIC Interrupt Translation Service (ITS) Affinity Structures, on ARM64, the driver currently makes an additional pass of SRAT later in the boot than the one used to identify NUMA domains. Note, this currently means that an ITS placed in a proximity domain that is not defined by another SRAT structure will result in the a crash.
To avoid this crash with minimal changes we do not create new NUMA nodes based on this particular entry type. Any current platform trying to do this will not boot, so this is an improvement, if perhaps not a perfect solution.
[1] Note in ACPI Specification 6.3 5.2.16 System Resource Affinity Table (SRAT)
[2] https://patchwork.kernel.org/patch/10597777/
Thanks to Bjorn Helgaas for review of v1 and Barry Song for internal reviews that lead to a slightly different approach for this v2.
Changes since v1.
* Use pxm_to_node for what was previously the path using acpi_map_pxm_to_node with create=false. (Barry)
* Broke patch up into an initial noop stage followed by patches (Bjorn) to update each type of case in which partial creation of NUMA nodes is prevented.
* Added patch 5 to rename function to reflect change of functionality.
* Updated descriptions (now mostly in individual patches) inline with Bjorn's comments.
Jonathan Cameron (6):
ACPI: Add out of bounds and numa_off protections to pxm_to_node
ACPI: Do not create new NUMA domains from ACPI static tables that are
not SRAT
ACPI: Remove side effect of partly creating a node in
acpi_map_pxm_to_online_node
ACPI: rename acpi_map_pxm_to_online_node to pxm_to_online_node
ACPI: Remove side effect of partly creating a node in acpi_get_node irq-chip/gic-v3-its: Fix crash if ITS is in a proximity domain without
processor or memory
drivers/acpi/arm64/iort.c | 2 +- drivers/acpi/nfit/core.c | 6 ++---- drivers/acpi/numa/hmat.c | 4 ++-- drivers/acpi/numa/srat.c | 4 ++-- drivers/iommu/intel/dmar.c | 2 +- drivers/irqchip/irq-gic-v3-its.c | 7 ++++++- include/linux/acpi.h | 15 +++++++-------- 7 files changed, 21 insertions(+), 19 deletions(-)
Signature · 1 line
2.19.1
PATCH v2 1/6 ACPI: Add out of bounds and numa_off protections to pxm_to_node
The function should check the validity of the pxm value before using it to index the pxm_to_node_map array.
Whilst hardening this code may be good in general, the main intent here is to enable following patches that use this function to replace acpi_map_pxm_to_node for non SRAT usecases which should return NO_NUMA_NODE for PXM entries not matching with those in SRAT.
--- drivers/acpi/numa/srat.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/acpi/numa/srat.c +1 -1
@@ -31,7 +31,7 @@ int acpi_numa __initdata;int pxm_to_node (int pxm) { -if (pxm <0 ) +if (pxm <0 || pxm >=MAX_PXM_DOMAINS || numa_off)return NUMA_NO_NODE ;return pxm_to_node_map[pxm]; }
Signature · 1 line
2.19.1
RE: [PATCH v2 1/6] ACPI: Add out of bounds and numa_off protections to pxm_to_node
Quoted 21 lines
-----Original Message-----
From: Jonathan Cameron
Sent: Saturday, July 18, 2020 6:00 AM
To: [email protected]; [email protected]; [email protected]; [email protected]
Cc: Lorenzo Pieralisi <[email protected]>; Bjorn Helgaas <[email protected]>; [email protected]; [email protected]; Ingo Molnar <[email protected]>; [email protected]; Tony Luck <[email protected]>; Fenghua Yu <[email protected]>; Thomas Gleixner <[email protected]>; Linuxarm <[email protected]>; Dan Williams <[email protected]>; Song Bao Hua (Barry Song) <[email protected]>; Jonathan Cameron <[email protected]>
Subject: [PATCH v2 1/6] ACPI: Add out of bounds andnuma_offprotections topxm_to_nodeThe function should check the validity of the pxm value before using it to index the
pxm_to_node_maparray.
Quoted 6 lines
Whilst hardening this code may be good in general, the main intent here is to enable following patches that use this function to replace
acpi_map_pxm_to_nodefor non SRAT usecases which should returnNO_NUMA_NODEfor PXM entries not matching with those in SRAT.Signed-off-by Jonathan Cameron <[email protected]>
Quoted 19 lines
--- drivers/acpi/numa/srat.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/acpi/numa/srat.c +1
-2@@ -31,7 +31,7 @@ int acpi_numa __initdata;int pxm_to_node (int pxm) { -if (pxm <0 ) +if (pxm <0 || pxm >=MAX_PXM_DOMAINS || numa_off)return NUMA_NO_NODE ;return pxm_to_node_map[pxm]; } --2.19.1
PATCH v2 2/6 ACPI: Do not create new NUMA domains from ACPI static tables that are not SRAT
In reply to Jonathan CameronSeveral ACPI static tables contain references to proximity domains. ACPI 6.3 has clarified that only entries in SRAT may define a new domain (sec 5.2.16).
Those tables described in the ACPI spec have additional clarifying text.
NFIT: Table 5-132,
"Integer that represents the proximity domain to which the memory belongs.
This number must match with corresponding entry in the SRAT table."
HMAT: Table 5-145,
"... This number must match with the corresponding entry in the SRAT
table's processor affinity structure ... if the initiator is a processor, or the Generic Initiator Affinity Structure if the initiator is a generic initiator".
IORT and DMAR are defined by external specifications.
Intel Virtualization Technology for Directed I/O Rev 3.1 does not make any explicit statements, but the general SRAT statement above will still apply.
https://software.intel.com/sites/default/files/managed/c5/15/vt-directed-io-spec.pdf
IO Remapping Table, Platform Design Document rev D, also makes not explicit statement, but refers to ACPI SRAT table for more information and again the generic SRAT statement above applies.
https://developer.arm.com/documentation/den0049/d/
In conclusion, any proximity domain specified in these tables, should be a reference to a proximity domain also found in SRAT, and they should not be able to instantiate a new domain. Hence we switch to pxm_to_node() which will only return existing nodes.
--- drivers/acpi/arm64/iort.c | 2 +- drivers/acpi/nfit/core.c | 3 +-- drivers/acpi/numa/hmat.c | 2 +- drivers/iommu/intel/dmar.c | 2 +- 4 files changed, 4 insertions(+), 5 deletions(-)
drivers/acpi/arm64/iort.c +1 -1
@@ -1293,7 +1293,7 @@ static int __init arm_smmu_v3_set_proximity(struct device *dev, smmu = (struct acpi_iort_smmu_v3 *)node->node_data;if (smmu->flags &ACPI_IORT_SMMU_V3_PXM_VALID ) { -int dev_node =acpi_map_pxm_to_node (smmu->pxm); +int dev_node =pxm_to_node (smmu->pxm);if (dev_node !=NUMA_NO_NODE && !node_online (dev_node))return -EINVAL ;
drivers/acpi/nfit/core.c +1 -2
@@ -2947,8 +2947,7 @@ static int acpi_nfit_register_region(struct acpi_nfit_desc *acpi_desc,if (spa->flags &ACPI_NFIT_PROXIMITY_VALID ) { ndr_desc->numa_node =acpi_map_pxm_to_online_node ( spa->proximity_domain); - ndr_desc->target_node =acpi_map_pxm_to_node ( - spa->proximity_domain); + ndr_desc->target_node =pxm_to_node (spa->proximity_domain); }else { ndr_desc->numa_node =NUMA_NO_NODE ; ndr_desc->target_node =NUMA_NO_NODE ;
drivers/acpi/numa/hmat.c +1 -1
@@ -666,7 +666,7 @@ static void hmat_register_target_device(struct memory_target *target, pdev->dev.numa_node =acpi_map_pxm_to_online_node (target->memory_pxm); info = (struct memregion_info ) { - .target_node =acpi_map_pxm_to_node (target->memory_pxm), + .target_node =pxm_to_node (target->memory_pxm), }; rc =platform_device_add_data (pdev, &info,sizeof (info));if (rc <0 ) {
drivers/iommu/intel/dmar.c +1 -1
@@ -473,7 +473,7 @@ static int dmar_parse_one_rhsa(struct acpi_dmar_header *header, void *arg) rhsa = (struct acpi_dmar_rhsa *)header;for_each_drhd_unit (drhd) {if (drhd->reg_base_addr = rhsa->base_address) { -int node =acpi_map_pxm_to_node (rhsa->proximity_domain); +int node =pxm_to_node (rhsa->proximity_domain);if (!node_online (node)) node =NUMA_NO_NODE ;
Signature · 1 line
2.19.1
RE: [PATCH v2 2/6] ACPI: Do not create new NUMA domains from ACPI static tables that are not SRAT
Quoted 48 lines
-----Original Message-----
From: Jonathan Cameron
Sent: Saturday, July 18, 2020 6:00 AM
To: [email protected]; [email protected]; [email protected]; [email protected]
Cc: Lorenzo Pieralisi <[email protected]>; Bjorn Helgaas <[email protected]>; [email protected]; [email protected]; Ingo Molnar <[email protected]>; [email protected]; Tony Luck <[email protected]>; Fenghua Yu <[email protected]>; Thomas Gleixner <[email protected]>; Linuxarm <[email protected]>; Dan Williams <[email protected]>; Song Bao Hua (Barry Song) <[email protected]>; Jonathan Cameron <[email protected]>
Subject: [PATCH v2 2/6] ACPI: Do not create new NUMA domains from ACPI static tables that are not SRATSeveral ACPI static tables contain references to proximity domains. ACPI 6.3 has clarified that only entries in SRAT may define a new domain (sec 5.2.16).
Those tables described in the ACPI spec have additional clarifying text.
NFIT: Table 5-132,
"Integer that represents the proximity domain to which the memory belongs.
This number must match with corresponding entry in the SRAT table."HMAT: Table 5-145,
"... This number must match with the corresponding entry in the SRAT
table's processor affinity structure ... if the initiator is a processor, or the Generic Initiator Affinity Structure if the initiator is a generic initiator".IORT and DMAR are defined by external specifications.
Intel Virtualization Technology for Directed I/O Rev 3.1 does not make any explicit statements, but the general SRAT statement above will still apply.
https://software.intel.com/sites/default/files/managed/c5/15/vt-directed-io-s
pec.pdfIO Remapping Table, Platform Design Document rev D, also makes not explicit statement, but refers to ACPI SRAT table for more information and again the generic SRAT statement above applies.
https://developer.arm.com/documentation/den0049/d/
Quoted 8 lines
In conclusion, any proximity domain specified in these tables, should be a reference to a proximity domain also found in SRAT, and they should not be able
to instantiate a new domain. Hence we switch topxm_to_node()which will only
return existing nodes.Signed-off-by Jonathan Cameron <[email protected]>
Much better than v1 which was using a bool parameter "false" to stop acpi_map_pxm_to_node() from creating node.
And pxm_to_node() has been used by some other places as an API which is serving the transition from pxm to node.
Quoted 63 lines
--- drivers/acpi/arm64/iort.c | 2 +- drivers/acpi/nfit/core.c | 3 +-- drivers/acpi/numa/hmat.c | 2 +- drivers/iommu/intel/dmar.c | 2 +- 4 files changed, 4 insertions(+), 5 deletions(-)drivers/acpi/arm64/iort.c +1
-1@@ -1293,7 +1293,7 @@ static int __init arm_smmu_v3_set_proximity(struct device *dev, smmu = (struct acpi_iort_smmu_v3 *)node->node_data;if (smmu->flags &ACPI_IORT_SMMU_V3_PXM_VALID ) { -int dev_node =acpi_map_pxm_to_node (smmu->pxm); +int dev_node =pxm_to_node (smmu->pxm);if (dev_node !=NUMA_NO_NODE && !node_online (dev_node))drivers/acpi/arm64/iort.c +0
-0return -EINVAL ;drivers/acpi/nfit/core.c +1
-2@@ -2947,8 +2947,7 @@ static int acpi_nfit_register_region(struct acpi_nfit_desc *acpi_desc, if (spa->flags & ACPI_NFIT_PROXIMITY_VALID) { ndr_desc->numa_node =acpi_map_pxm_to_online_node ( spa->proximity_domain); - ndr_desc->target_node =acpi_map_pxm_to_node ( - spa->proximity_domain); + ndr_desc->target_node =pxm_to_node (spa->proximity_domain); } else { ndr_desc->numa_node =NUMA_NO_NODE ;drivers/acpi/nfit/core.c +0
-0ndr_desc->target_node =NUMA_NO_NODE ;drivers/acpi/numa/hmat.c +1
-1@@ -666,7 +666,7 @@ static void hmat_register_target_device(struct memory_target *target, pdev->dev.numa_node >acpi_map_pxm_to_online_node (target->memory_pxm); info = (struct memregion_info ) { - .target_node =acpi_map_pxm_to_node (target->memory_pxm), + .target_node =pxm_to_node (target->memory_pxm), }; rc =platform_device_add_data (pdev, &info,sizeof (info));drivers/acpi/numa/hmat.c +0
-0if (rc <0 ) {drivers/iommu/intel/dmar.c +1
-1@@ -473,7 +473,7 @@ static int dmar_parse_one_rhsa(struct acpi_dmar_header *header, void *arg) rhsa = (struct acpi_dmar_rhsa *)header;for_each_drhd_unit (drhd) {if (drhd->reg_base_addr = rhsa->base_address) { -int node =acpi_map_pxm_to_node (rhsa->proximity_domain); +int node =pxm_to_node (rhsa->proximity_domain);if (!node_online (node))
Quoted 3 lines
node =
NUMA_NO_NODE;
--
2.19.1
Thanks
Barry
PATCH v2 3/6 ACPI: Remove side effect of partly creating a node in acpi_map_pxm_to_online_node
In reply to Jonathan CameronWhilst this function will only return an online node, it can have the side effect of partially creating a new node. The existing comments suggest this is intentional, but the usecases of this function are related to NFIT and HMAT parsing, neither of which should be able to define new nodes.
One route by which the existing behaviour would cause a crash is to have a _PXM entry in ACPI DSDT attempt to place a device within this partly created proximity domain. A subsequent call to devm_kzalloc or similar would result in an attempt to allocate memory on a node for which zone lists have not been set up and a null pointer dereference.
We prevent such cases by switching to pxm_to_node() within acpi_map_pxm_to_online_node which cannot cause a new node to be partly created. If one would previously have been created we now return NO_NUMA_NODE. Documentation updated to reflect this change. We may want to think about renaming acpi_map_pxm_to_online_node to pxm_to_online_node to reflect this change.
--- include/linux/acpi.h | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
include/linux/acpi.h +3 -4
@@ -430,13 +430,12 @@ int acpi_get_node(acpi_handle handle);* ACPI device drivers, which are called after the NUMA initialization has * completed in the kernel, can call this interface to obtain their device * NUMA topology from ACPI tables. Such drivers do not have to deal with -* offline nodes. A node may be offline when a device proximity ID is -* unique, SRAT memory entry does not exist, or NUMA is disabled, ex. -* "numa=off" on x86. +* offline nodes. A node may be offline when SRAT memory entry does not exist, +* or NUMA is disabled, ex. "numa=off" on x86. */ static inline int acpi_map_pxm_to_online_node (int pxm) { -int node =acpi_map_pxm_to_node (pxm); +int node =pxm_to_node (pxm);return numa_map_to_online_node (node); }
Signature · 1 line
2.19.1
PATCH v2 4/6 ACPI: Rename acpi_map_pxm_to_online_node to pxm_to_online_node
In reply to Jonathan CameronAs this function is no longer allowed to create new mappings let us rename it to reflect this.
Note all nodes should already exist before any of the users of this function are called.
--- drivers/acpi/nfit/core.c | 3 +-- drivers/acpi/numa/hmat.c | 2 +- include/linux/acpi.h | 8 ++++---- 3 files changed, 6 insertions(+), 7 deletions(-)
drivers/acpi/nfit/core.c +1 -2
@@ -2945,8 +2945,7 @@ static int acpi_nfit_register_region(struct acpi_nfit_desc *acpi_desc, ndr_desc->provider_data = nfit_spa; ndr_desc->attr_groups = acpi_nfit_region_attribute_groups;if (spa->flags &ACPI_NFIT_PROXIMITY_VALID ) { - ndr_desc->numa_node =acpi_map_pxm_to_online_node ( - spa->proximity_domain); + ndr_desc->numa_node =pxm_to_online_node (spa->proximity_domain); ndr_desc->target_node =pxm_to_node (spa->proximity_domain); }else { ndr_desc->numa_node =NUMA_NO_NODE ;
drivers/acpi/numa/hmat.c +1 -1
@@ -664,7 +664,7 @@ static void hmat_register_target_device(struct memory_target *target, goto out_pdev; } - pdev->dev.numa_node =acpi_map_pxm_to_online_node (target->memory_pxm); + pdev->dev.numa_node =pxm_to_online_node (target->memory_pxm); info = (struct memregion_info ) { .target_node =pxm_to_node (target->memory_pxm), };
include/linux/acpi.h +4 -4
@@ -420,10 +420,10 @@ int acpi_map_pxm_to_node(int pxm);int acpi_get_node (acpi_handle handle);/** -* acpi_map_pxm_to_online_node - Map proximity ID to online node +* pxm_to_online_node - Map proximity ID to online node * @pxm: ACPI proximity ID * -* This is similar to acpi_map_pxm_to_node(), but always returns an online +* This is similar to pxm_to_node(), but always returns an online * node. When the mapped node from a given proximity ID is offline, it * looks up the node distance table and returns the nearest online node. *
@@ -433,14 +433,14 @@ int acpi_get_node(acpi_handle handle);* offline nodes. A node may be offline when SRAT memory entry does not exist, * or NUMA is disabled, ex. "numa=off" on x86. */ -static inline int acpi_map_pxm_to_online_node (int pxm) +static inline int pxm_to_online_node (int pxm) {int node =pxm_to_node (pxm);return numa_map_to_online_node (node); }#else -static inline int acpi_map_pxm_to_online_node (int pxm) +static inline int pxm_to_online_node (int pxm) {return 0 ; }
Signature · 1 line
2.19.1
PATCH v2 5/6 ACPI: Remove side effect of partly creating a node in acpi_get_node
In reply to Jonathan Cameronacpi_get_node calls acpi_get_pxm to evaluate the _PXM AML method for entries found in DSDT/SSDT. ACPI 6.3 sec 6.2.14 states
"_PXM evaluates to an integer that identifies a device as belonging to
a Proximity Domain defined in the System Resource Affinity Table (SRAT)."
Hence a _PXM method should not result in creation of a new NUMA node. Before this patch, _PXM could result in partial instantiation of NUMA node, missing elements such as zone lists. A call to devm_kzalloc for example results in a null pointer dereference.
This patch therefore replaces the acpi_map_pxm_to_node with a call to pxm_to_node.
--- drivers/acpi/numa/srat.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/acpi/numa/srat.c +1 -1
@@ -444,6 +444,6 @@ int acpi_get_node(acpi_handle handle) pxm =acpi_get_pxm (handle); -return acpi_map_pxm_to_node (pxm); +return pxm_to_node (pxm); }EXPORT_SYMBOL (acpi_get_node);
Signature · 1 line
2.19.1
RE: [PATCH v2 5/6] ACPI: Remove side effect of partly creating a node in acpi_get_node
Quoted 28 lines
-----Original Message-----
From: Jonathan Cameron
Sent: Saturday, July 18, 2020 6:00 AM
To: [email protected]; [email protected]; [email protected]; [email protected]
Cc: Lorenzo Pieralisi <[email protected]>; Bjorn Helgaas <[email protected]>; [email protected]; [email protected]; Ingo Molnar <[email protected]>; [email protected]; Tony Luck <[email protected]>; Fenghua Yu <[email protected]>; Thomas Gleixner <[email protected]>; Linuxarm <[email protected]>; Dan Williams <[email protected]>; Song Bao Hua (Barry Song) <[email protected]>; Jonathan Cameron <[email protected]>
Subject: [PATCH v2 5/6] ACPI: Remove side effect of partly creating a node inacpi_get_node
acpi_get_nodecallsacpi_get_pxmto evaluate the_PXMAML method for entries found in DSDT/SSDT. ACPI 6.3 sec 6.2.14 states
"_PXMevaluates to an integer that identifies a device as belonging to
a Proximity Domain defined in the System Resource Affinity Table (SRAT)."Hence a
_PXMmethod should not result in creation of a new NUMA node. Before this patch,_PXMcould result in partial instantiation of NUMA node, missing elements such as zone lists. A call todevm_kzallocfor example results in a null pointer dereference.
Quoted 4 lines
This patch therefore replaces the
acpi_map_pxm_to_nodewith a call topxm_to_node.Signed-off-by Jonathan Cameron <[email protected]>
Quoted 18 lines
--- drivers/acpi/numa/srat.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/acpi/numa/srat.c +1
-2@@ -444,6 +444,6 @@ int acpi_get_node(acpi_handle handle) pxm =acpi_get_pxm (handle); -return acpi_map_pxm_to_node (pxm); +return pxm_to_node (pxm); }EXPORT_SYMBOL (acpi_get_node); --2.19.1
PATCH v2 6/6 irq-chip/gic-v3-its: Fix crash if ITS is in a proximity domain without processor or memory
In reply to Jonathan CameronNote this crash is present before any of the patches in this series, but as explained below it is highly unlikely anyone is shipping a firmware that causes it. Tests were done using an overriden SRAT.
On ARM64, the gic-v3 driver directly parses SRAT to locate GIC Interrupt Translation Service (ITS) Affinity Structures. This is done much later in the boot than the parses of SRAT which identify proximity domains.
As a result, an ITS placed in a proximity domain that is not defined by another SRAT structure will result in a NUMA node that is not completely configured and a crash.
ITS [mem 0x202100000-0x20211ffff]
ITS@0x0000000202100000: Using ITS number 0
Unable to handle kernel paging request at virtual address 0000000000001a08
...
Call trace: __alloc_pages_nodemask+0xe8/0x338 alloc_pages_node.constprop.0+0x34/0x40 its_probe_one+0x2f8/0xb18 gic_acpi_parse_madt_its+0x108/0x150 acpi_table_parse_entries_array+0x17c/0x264 acpi_table_parse_entries+0x48/0x6c acpi_table_parse_madt+0x30/0x3c its_init+0x1c4/0x644 gic_init_bases+0x4b8/0x4ec gic_acpi_init+0x134/0x264 acpi_match_madt+0x4c/0x84 acpi_table_parse_entries_array+0x17c/0x264 acpi_table_parse_entries+0x48/0x6c acpi_table_parse_madt+0x30/0x3c __acpi_probe_device_table+0x8c/0xe8 irqchip_init+0x3c/0x48 init_IRQ+0xcc/0x100 start_kernel+0x33c/0x548
ACPI 6.3 allows any set of Affinity Structures in SRAT to define a proximity domain. However, as we do not see this crash, we can conclude that no firmware is currently placing an ITS in a node that is separate from those containing memory and / or processors.
We could modify the SRAT parsing behavior to identify the existence of Proximity Domains unique to the ITS structures, and handle them as a special case of a generic initiator (once support for those merges).
This patch avoids the complexity that would be needed to handle this corner case, by not allowing the ITS entry parsing code to instantiate new NUMA Nodes. If one is encountered that does not already exist, then NO_NUMA_NODE is assigned and a warning printed just as if the value had been greater than allowed NUMA Nodes.
"SRAT: Invalid NUMA node -1 in ITS affinity"
Whilst this does not provide the full flexibility allowed by ACPI, it does fix the problem. We can revisit a more sophisticated solution if needed by future platforms.
Change is simply to replace acpi_map_pxm_to_node with pxm_to_node reflecting the fact a new mapping is not created.
--- drivers/irqchip/irq-gic-v3-its.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/irqchip/irq-gic-v3-its.c +6 -1
@@ -5248,7 +5248,12 @@ static int __init gic_acpi_parse_srat_its(union acpi_subtable_headers *header,return -EINVAL ; } - node =acpi_map_pxm_to_node (its_affinity->proximity_domain); +/* +* Note that in theory a new proximity node could be created by this +* entry as it is an SRAT resource allocation structure. +* We do not currently support doing so. +*/ + node =pxm_to_node (its_affinity->proximity_domain);if (node =NUMA_NO_NODE || node >=MAX_NUMNODES ) {pr_err ("SRAT: Invalid NUMA node %d in ITS affinity\n" , node);
Signature · 1 line
2.19.1
Quoted 41 lines · Jonathan Cameron
Here, I will use the term Proximity Domains for the ACPI description and NUMA Nodes for the in kernel representation.
ACPI 6.3 included a clarification that only Static Resource Allocation Structures in SRAT may define the existence of proximity domains (sec 5.2.16). This clarification closed a possible interpretation that other parts of ACPI (e.g. DSDT
_PXM, NFIT etc) could define new proximity domains that were not also mentioned in SRAT structures.In practice the kernel has never allowed this alternative interpretation as such nodes are only partially initialized. This is architecture specific but to take an example, on x86
alloc_node_datahas not been called. Any use of them for node specific allocation, will result in a crash as the infrastructure to fallback to a node with memory is not setup.We ran into a problem when enabling
_PXMhandling for PCI devices and found there were boards out there advertising devices in proximity domains that didn't exist [2].The fix suggested in this series is to replace instances that should not 'create' new nodes with
pxm_to_node. This function needs a some additional hardening against invalid inputs to make sure it is safe for use in these new callers.Patch 1 Hardens
pxm_to_node()againstnuma_off, and pxm entry being too large.Patch 2-4 change the various callers not related to SRAT entries so that they set this parameter to false, so do not attempt to initialize a new NUMA node if the relevant one does not already exist.
Patch 5 is a function rename to reflect change in functionality of
acpi_map_pxm_to_online_node()as it no longer creates a new map, but just does a lookup of existing maps.Patch 6 covers the one place we do not allow the full flexibility defined in the ACPI spec. For SRAT GIC Interrupt Translation Service (ITS) Affinity Structures, on ARM64, the driver currently makes an additional pass of SRAT later in the boot than the one used to identify NUMA domains. Note, this currently means that an ITS placed in a proximity domain that is not defined by another SRAT structure will result in the a crash.
Quoted 3 lines
To avoid this crash with minimal changes we do not create new NUMA nodes based on this particular entry type. Any current platform trying to do this will not boot, so this is an improvement, if perhaps not a perfect solution.
Make sense to me,
Quoted 45 lines · On Sat, 18 Jul 2020 01:59:53 +0800 Jonathan Cameron…
Here, I will use the term Proximity Domains for the ACPI description and NUMA Nodes for the in kernel representation.
ACPI 6.3 included a clarification that only Static Resource Allocation Structures in SRAT may define the existence of proximity domains (sec 5.2.16). This clarification closed a possible interpretation that other parts of ACPI (e.g. DSDT
_PXM, NFIT etc) could define new proximity domains that were not also mentioned in SRAT structures.In practice the kernel has never allowed this alternative interpretation as such nodes are only partially initialized. This is architecture specific but to take an example, on x86
alloc_node_datahas not been called. Any use of them for node specific allocation, will result in a crash as the infrastructure to fallback to a node with memory is not setup.We ran into a problem when enabling
_PXMhandling for PCI devices and found there were boards out there advertising devices in proximity domains that didn't exist [2].The fix suggested in this series is to replace instances that should not 'create' new nodes with
pxm_to_node. This function needs a some additional hardening against invalid inputs to make sure it is safe for use in these new callers.Patch 1 Hardens
pxm_to_node()againstnuma_off, and pxm entry being too large.Patch 2-4 change the various callers not related to SRAT entries so that they set this parameter to false, so do not attempt to initialize a new NUMA node if the relevant one does not already exist.
Patch 5 is a function rename to reflect change in functionality of
acpi_map_pxm_to_online_node()as it no longer creates a new map, but just does a lookup of existing maps.Patch 6 covers the one place we do not allow the full flexibility defined in the ACPI spec. For SRAT GIC Interrupt Translation Service (ITS) Affinity Structures, on ARM64, the driver currently makes an additional pass of SRAT later in the boot than the one used to identify NUMA domains. Note, this currently means that an ITS placed in a proximity domain that is not defined by another SRAT structure will result in the a crash.
To avoid this crash with minimal changes we do not create new NUMA nodes based on this particular entry type. Any current platform trying to do this will not boot, so this is an improvement, if perhaps not a perfect solution.
Quoted 5 lines
[1] Note in ACPI Specification 6.3 5.2.16 System Resource Affinity Table (SRAT)
[2] https://patchwork.kernel.org/patch/10597777/Thanks to Bjorn Helgaas for review of v1 and Barry Song for internal reviews that lead to a slightly different approach for this v2.
Thanks Barry / Hanjun,
Anyone else have time to take a look?
I'm happy to bring it back after the merge window closes, but would like to know if people are happy with the general approach. I'm keen to finally be able to resolve that issue with _PXM and PCI.
Thanks,
Jonathan
Quoted 29 lines
Changes since v1.
* Usepxm_to_nodefor what was previously the path usingacpi_map_pxm_to_nodewith create=false. (Barry)
* Broke patch up into an initial noop stage followed by patches (Bjorn) to update each type of case in which partial creation of NUMA nodes is prevented.
* Added patch 5 to rename function to reflect change of functionality.
* Updated descriptions (now mostly in individual patches) inline with Bjorn's comments.Jonathan Cameron (6):
ACPI: Add out of bounds andnuma_offprotections topxm_to_node
ACPI: Do not create new NUMA domains from ACPI static tables that are
not SRAT
ACPI: Remove side effect of partly creating a node in
acpi_map_pxm_to_online_node
ACPI: renameacpi_map_pxm_to_online_nodetopxm_to_online_node
ACPI: Remove side effect of partly creating a node inacpi_get_nodeirq-chip/gic-v3-its: Fix crash if ITS is in a proximity domain without
processor or memorydrivers/acpi/arm64/iort.c | 2 +- drivers/acpi/nfit/core.c | 6 ++---- drivers/acpi/numa/hmat.c | 4 ++-- drivers/acpi/numa/srat.c | 4 ++-- drivers/iommu/intel/dmar.c | 2 +- drivers/irqchip/irq-gic-v3-its.c | 7 ++++++- include/linux/acpi.h | 15 +++++++-------- 7 files changed, 21 insertions(+), 19 deletions(-)