fix pmem RAM device when nid is NUMA_NO_NODE
cover fix pmem RAM device when nid is NUMA_NO_NODE
Background
==========
I once sent out the preparatory patches [1] but I dropped the last patch of using fallback nid with memory_add_physaddr_to_nid() due to some dependency.
After phys_addr_to_target_node() and memory_add_physaddr_to_nid() are stable now, it's time to fix the original bug on arm64 now.
Compared with the last version [2], this version rebases the patch to latest v5.14-rc3 (s/kmem_start/range.start)
Test
====
Tested on ThunderX2 host/qemu "-M virt" guest with a nvdimm device. The memblocks from the dax pmem device can be either hot-added or hot-removed on arm64 guest. Also passed the compilation test on x86.
[1] https://www.mail-archive.com/[email protected]/msg2228771.html
[2] https://lkml.org/lkml/2020/7/8/1546
Jia He (1):
device-dax: use fallback nid when numa_node is invalid
drivers/dax/kmem.c | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/1 device-dax: use fallback nid when numa_node is invalid
Previously, numa_off was set unconditionally in dummy_numa_init() even with a fake numa node. Then ACPI set node id as NUMA_NO_NODE(-1) after acpi_map_pxm_to_node() because it regards numa_off as turning off the numa node. Hence dev_dax->target_node is NUMA_NO_NODE on arm64 with fake numa.
Without this patch, pmem can't be probed as a RAM device on arm64 if
SRAT table isn't present:
$ndctl create-namespace -fe namespace0.0 --mode=devdax --map=dev -s 1g -a 64K kmem dax0.0: rejecting DAX region [mem 0x240400000-0x2bfffffff] with invalid node: -1
kmem: probe of dax0.0 failed with error -22
This fixes it by using fallback memory_add_physaddr_to_nid() as nid.
--- drivers/dax/kmem.c | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-)
drivers/dax/kmem.c +20 -16
@@ -46,20 +46,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)struct dax_kmem_data *data;int rc = -ENOMEM ;int i, mapped =0 ; -int numa_node; - -/* -* Ensure good NUMA information for the persistent memory. -* Without this check, there is a risk that slow memory -* could be mixed in a node with faster memory, causing -* unavoidable performance issues. -*/ - numa_node = dev_dax->target_node; -if (numa_node <0 ) { -dev_warn (dev,"rejecting DAX region with invalid node: %d\n" , - numa_node); -return -EINVAL ; - } +int numa_node = dev_dax->target_node, new_node; data =kzalloc (struct_size (data, res, dev_dax->nr_range),GFP_KERNEL );if (!data)
@@ -104,6 +91,20 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)*/ res->flags =IORESOURCE_SYSTEM_RAM ; +/* +* Ensure good NUMA information for the persistent memory. +* Without this check, there is a risk but not fatal that slow +* memory could be mixed in a node with faster memory, causing +* unavoidable performance issues. Furthermore, fallback node +* id can be used when numa_node is invalid. +*/ +if (numa_node <0 ) { + new_node =memory_add_physaddr_to_nid (range.start); +dev_info (dev,"changing nid from %d to %d for DAX region %pR\n" , + numa_node, new_node, res); + numa_node = new_node; + } +/* * Ensure that future kexec'd kernels will not treat * this as RAM automatically.
@@ -141,6 +142,7 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax)int i, success =0 ;struct device *dev = &dev_dax->dev;struct dax_kmem_data *data =dev_get_drvdata (dev); +int numa_node = dev_dax->target_node;/* * We have one shot for removing memory, if some memory blocks were not
@@ -156,8 +158,10 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax)if (rc)continue ; - rc =remove_memory (dev_dax->target_node, range.start, -range_len (&range)); +if (numa_node <0 ) + numa_node =memory_add_physaddr_to_nid (range.start); + + rc =remove_memory (numa_node, range.start,range_len (&range));if (rc ==0 ) {release_resource (data->res[i]);kfree (data->res[i]);
Signature · 1 line
2.17.1
Re: [PATCH] device-dax: use fallback nid when numa_node is invalid
Quoted 79 lines · Jia He
Previously,
numa_offwas set unconditionally indummy_numa_init()even with a fake numa node. Then ACPI set node id asNUMA_NO_NODE(-1)afteracpi_map_pxm_to_node()because it regardsnuma_offas turning off the numa node. Hencedev_dax->target_nodeisNUMA_NO_NODEon arm64 with fake numa.Without this patch, pmem can't be probed as a RAM device on arm64 if
SRAT table isn't present:
$ndctl create-namespace -fe namespace0.0 --mode=devdax --map=dev -s 1g -a 64K kmem dax0.0: rejecting DAX region [mem 0x240400000-0x2bfffffff] with invalid node: -1
kmem: probe of dax0.0 failed with error -22This fixes it by using fallback
memory_add_physaddr_to_nid()as nid.Suggested-by David Hildenbrand <[email protected]>Signed-off-by Jia He <[email protected]>--- drivers/dax/kmem.c | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-)drivers/dax/kmem.c +16
-14@@ -46,20 +46,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)struct dax_kmem_data *data;int rc = -ENOMEM ;int i, mapped =0 ; -int numa_node; - -/* -* Ensure good NUMA information for the persistent memory. -* Without this check, there is a risk that slow memory -* could be mixed in a node with faster memory, causing -* unavoidable performance issues. -*/ - numa_node = dev_dax->target_node; -if (numa_node <0 ) { -dev_warn (dev,"rejecting DAX region with invalid node: %d\n" , - numa_node); -return -EINVAL ; - } +int numa_node = dev_dax->target_node, new_node; data =kzalloc (struct_size (data, res, dev_dax->nr_range),GFP_KERNEL );if (!data)@@ -104,6 +91,20 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)*/ res->flags =IORESOURCE_SYSTEM_RAM ; +/* +* Ensure good NUMA information for the persistent memory. +* Without this check, there is a risk but not fatal that slow +* memory could be mixed in a node with faster memory, causing +* unavoidable performance issues. Furthermore, fallback node +* id can be used when numa_node is invalid. +*/ +if (numa_node <0 ) { + new_node =memory_add_physaddr_to_nid (range.start); +dev_info (dev,"changing nid from %d to %d for DAX region %pR\n" , + numa_node, new_node, res); + numa_node = new_node; + } +/* * Ensure that future kexec'd kernels will not treat * this as RAM automatically. @@ -141,6 +142,7 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax)int i, success =0 ;struct device *dev = &dev_dax->dev;struct dax_kmem_data *data =dev_get_drvdata (dev); +int numa_node = dev_dax->target_node;/* * We have one shot for removing memory, if some memory blocks were not @@ -156,8 +158,10 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax)if (rc)continue ;
drivers/dax/kmem.c static void dev_dax_kmem_remove(struct dev_dax *dev_dax) · line 167
Quoted 10 lines
drivers/dax/kmem.c +4
-2- rc =remove_memory (dev_dax->target_node, range.start, -range_len (&range)); +if (numa_node <0 ) + numa_node =memory_add_physaddr_to_nid (range.start); + + rc =remove_memory (numa_node, range.start,range_len (&range));if (rc ==0 ) {release_resource (data->res[i]);kfree (data->res[i]);
Note that this patch conflicts with:
https://lkml.kernel.org/r/[email protected]
But nothing fundamental. Determining a single NID is similar to how I'm handling it for ACPI:
https://lkml.kernel.org/r/[email protected]
Signature · 3 lines
Thanks, David / dhildenb
RE: [PATCH] device-dax: use fallback nid when numa_node is invalid
Hi David
Quoted 108 lines
-----Original Message-----
From: David Hildenbrand <[email protected]>
Sent: Thursday, July 29, 2021 4:17 AM
To: Justin He <[email protected]>; Dan Williams <[email protected]>; Vishal Verma <[email protected]>; Dave Jiang <[email protected]>
Cc: [email protected]; [email protected]; nd <[email protected]>
Subject: Re: [PATCH] device-dax: use fallback nid whennuma_nodeis invalidQuoted 9 lines · Jia He
Previously,
numa_offwas set unconditionally indummy_numa_init()even with a fake numa node. Then ACPI set node id asNUMA_NO_NODE(-1)afteracpi_map_pxm_to_node()because it regardsnuma_offas turning off the numa node. Hencedev_dax->target_nodeisNUMA_NO_NODEon arm64 with fake numa.Without this patch, pmem can't be probed as a RAM device on arm64 if SRAT table isn't present:
$ndctl create-namespace -fe namespace0.0 --mode=devdax --map=dev -s 1g-a 64K
Quoted 1 line
kmem dax0.0: rejecting DAX region [mem 0x240400000-0x2bfffffff] with
invalid node: -1
Quoted 25 lines
kmem: probe of dax0.0 failed with error -22
This fixes it by using fallback
memory_add_physaddr_to_nid()as nid.Suggested-by David Hildenbrand <[email protected]>Signed-off-by Jia He <[email protected]>--- drivers/dax/kmem.c | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-)drivers/dax/kmem.c +0
-7@@ -46,20 +46,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)struct dax_kmem_data *data;int rc = -ENOMEM ;int i, mapped =0 ; -int numa_node; - -/* -* Ensure good NUMA information for the persistent memory. -* Without this check, there is a risk that slow memory -* could be mixed in a node with faster memory, causing -* unavoidable performance issues. drivers/dax/kmem.c static int dev_dax_kmem_probe(struct dev_dax *dev_dax) · line 51
Quoted 10 lines
drivers/dax/kmem.c +1
-7-*/ - numa_node = dev_dax->target_node; -if (numa_node <0 ) { -dev_warn (dev,"rejecting DAX region with invalid node: %d\n" , - numa_node); -return -EINVAL ; - } +int numa_node = dev_dax->target_node, new_node; data = kzalloc(struct_size(data, res, dev_dax->nr_range),
GFP_KERNEL);Quoted 5 lines
drivers/dax/kmem.c +0
-0if (!data)@@ -104,6 +91,20 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)*/ res->flags =IORESOURCE_SYSTEM_RAM ;drivers/dax/kmem.c static int dev_dax_kmem_probe(struct dev_dax *dev_dax) · line 103
Quoted 10 lines
drivers/dax/kmem.c +10
-0+/* +* Ensure good NUMA information for the persistent memory. +* Without this check, there is a risk but not fatal that slow +* memory could be mixed in a node with faster memory, causing +* unavoidable performance issues. Furthermore, fallback node +* id can be used when numa_node is invalid. +*/ +if (numa_node <0 ) { + new_node =memory_add_physaddr_to_nid (range.start); + dev_info(dev, "changing nid from %d to %d for DAXregion %pR\n",
drivers/dax/kmem.c static void dev_dax_kmem_remove(struct dev_dax
Quoted 8 lines
drivers/dax/kmem.c +4
-0+ numa_node, new_node, res); + numa_node = new_node; + } +/* * Ensure that future kexec'd kernels will not treat * this as RAM automatically. @@ -141,6 +142,7 @@ static void dev_dax_kmem_remove(struct dev_dax
*dev_dax)drivers/dax/kmem.c static void dev_dax_kmem_remove(struct dev_dax *dev_dax)
Quoted 7 lines
drivers/dax/kmem.c +1
-0int i, success =0 ;struct device *dev = &dev_dax->dev;struct dax_kmem_data *data =dev_get_drvdata (dev); +int numa_node = dev_dax->target_node;/* * We have one shot for removing memory, if some memory blocks were not
drivers/dax/kmem.c static void dev_dax_kmem_remove(struct dev_dax
Quoted 1 line
drivers/dax/kmem.c +0
-0@@ -156,8 +158,10 @@ static void dev_dax_kmem_remove(struct dev_dax
*dev_dax)drivers/dax/kmem.c static void dev_dax_kmem_remove(struct dev_dax *dev_dax) · line 167
Quoted 13 lines
drivers/dax/kmem.c +4
-2if (rc)continue ; - rc =remove_memory (dev_dax->target_node, range.start, -range_len (&range)); +if (numa_node <0 ) + numa_node =memory_add_physaddr_to_nid (range.start); + + rc =remove_memory (numa_node, range.start,range_len (&range));if (rc ==0 ) {release_resource (data->res[i]);kfree (data->res[i]);Note that this patch conflicts with:
Quoted 7 lines
https://lkml.kernel.org/r/[email protected]
But nothing fundamental. Determining a single NID is similar to how I'm handling it for ACPI:
Okay, got it. Thanks for the reminder.
Seems my patch is not useful after your patch.
--
Cheers,
Justin (Jia He)
Re: [PATCH] device-dax: use fallback nid when numa_node is invalid
Hi Justin,
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Note that this patch conflicts with:
Quoted 9 lines
Quoted 5 lines
But nothing fundamental. Determining a single NID is similar to how I'm handling it for ACPI:
Okay, got it. Thanks for the reminder.
Seems my patch is not useful after your patch.
I think your patch still makes sense. With
https://lore.kernel.org/linux-acpi/[email protected]/
We'd have to detect the node id in the first loop instead.
Signature · 3 lines
Thanks, David / dhildenb
RE: [PATCH] device-dax: use fallback nid when numa_node is invalid
Hi David
Quoted 27 lines
-----Original Message-----
From: David Hildenbrand <[email protected]>
Sent: Thursday, July 29, 2021 3:59 PM
To: Justin He <[email protected]>; Dan Williams <[email protected]>; Vishal Verma <[email protected]>; Dave Jiang <[email protected]>
Cc: [email protected]; [email protected]; nd <[email protected]>
Subject: Re: [PATCH] device-dax: use fallback nid whennuma_nodeis invalidHi Justin,
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Note that this patch conflicts with:
Quoted 9 lines
Quoted 5 lines
But nothing fundamental. Determining a single NID is similar to how I'm handling it for ACPI:
Okay, got it. Thanks for the reminder.
Seems my patch is not useful after your patch.
Quoted 6 lines
I think your patch still makes sense. With
https://lore.kernel.org/linux-acpi/20210723125210.29987-7-
[email protected]/We'd have to detect the node id in the first loop instead.
Ok, I got your point. I will do that in v2.
Btw, sorry for commenting there about your patch 06 since I didn't subscribe lkml via this mailbox.
+ for (i = 0; i < dev_dax->nr_range; i++) {
+ struct range range;
+
+ rc = dax_kmem_range(dev_dax, i, &range);
+ if (rc) {
+ dev_info(dev, "mapping%d: %#llx-%#llx too small after alignment\n",
+ i, range.start, range.end);
+ continue;
+ }
+ total_len += range_len(&range);
+ }
You add an additional loop to get the total_len.
I wonder is it independent on 2nd loop?
If yes, why not merge the 2 loops into one?
Sorry if this question is too simple, I don't know too much about the background of your patch.
--
Cheers,
Justin (Jia He)
Re: [PATCH] device-dax: use fallback nid when numa_node is invalid
Quoted 34 lines · Justin He
Hi David
Quoted 32 lines
-----Original Message-----
From: David Hildenbrand <[email protected]>
Sent: Thursday, July 29, 2021 3:59 PM
To: Justin He <[email protected]>; Dan Williams <[email protected]>; Vishal Verma <[email protected]>; Dave Jiang <[email protected]>
Cc: [email protected]; [email protected]; nd <[email protected]>
Subject: Re: [PATCH] device-dax: use fallback nid whennuma_nodeis invalidHi Justin,
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Note that this patch conflicts with:
Quoted 9 lines
Quoted 5 lines
But nothing fundamental. Determining a single NID is similar to how I'm handling it for ACPI:
Okay, got it. Thanks for the reminder.
Seems my patch is not useful after your patch.I think your patch still makes sense. With
https://lore.kernel.org/linux-acpi/20210723125210.29987-7-
[email protected]/
Quoted 6 lines
Quoted 1 line
We'd have to detect the node id in the first loop instead.
Ok, I got your point. I will do that in v2.
Btw, sorry for commenting there about your patch 06 since I didn't subscribe lkml via this mailbox.
Sure, you really should subscribe :)
Quoted 11 lines
Quoted 1 line
(quoted diff) +1
-0+ for (i = 0; i < dev_dax->nr_range; i++) {(quoted diff) +10
-0+ struct range range; + + rc = dax_kmem_range(dev_dax, i, &range); + if (rc) { + dev_info(dev, "mapping%d: %#llx-%#llx too small after alignment\n", + i, range.start, range.end); + continue; + } + total_len += range_len(&range); + }
Quoted 5 lines
You add an additional loop to get the
total_len.
I wonder is it independent on 2nd loop?
If yes, why not merge the 2 loops into one?
Sorry if this question is too simple, I don't know too much about the background of your patch.
We need total_len to register the memory group. We need the memory group to add memory. Therefore, we need a second loop to calculate total_len upfront.
Signature · 3 lines
Thanks, David / dhildenb