Skip to contentSkip to conversation

Add device tree based NUMA support to Arm64

RFC PATCH In review Wei Chen 5 participants 196 messages Aug 11, 2021 – Sep 9, 2021 Page 2 of 2

lore
Share & tools
lore
#120 lore raw

RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

In reply to Wei Chen
Quoted 157 lines · Wei Chen

Hi Stefano,

Quoted 95 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 8:48
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

Review

Quoted 10 lines · Wei Chen

A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a new helper to parse this distance map.

---
 xen/arch/arm/numa_device_tree.c | 67 +++++++++++++++++++++++++++++++++
 1 file changed, 67 insertions(+)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt,

Quoted 4 lines

index bbe081dcd1..6e0d1d3d9f 100644

xen/arch/arm/numa_device_tree.c +0 -0
@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void *fdt,

int node,

Quoted 45 lines
xen/arch/arm/numa_device_tree.c +42 -0
     return 0;
 }
+
+/* Parse NUMA distance map v1 */
+int __init
+device_tree_parse_numa_distance_map_v1(const void *fdt, int node)
+{
+    const struct fdt_property *prop;
+    const __be32 *matrix;
+    int entry_count, len, i;
+
+    printk(XENLOG_INFO "NUMA: parsing numa-distance-map\n");
+
+    prop = fdt_get_property(fdt, node, "distance-matrix", &len);
+    if ( !prop )
+    {
+        printk(XENLOG_WARNING
+               "NUMA: No distance-matrix property in distance-map\n");
+
+        return -EINVAL;
+    }
+
+    if ( len % sizeof(uint32_t) != 0 )
+    {
+        printk(XENLOG_WARNING
+               "distance-matrix in node is not a multiple of u32\n");
+        return -EINVAL;
+    }
+
+    entry_count = len / sizeof(uint32_t);
+    if ( entry_count <= 0 )
+    {
+        printk(XENLOG_WARNING "NUMA: Invalid distance-matrix\n");
+
+        return -EINVAL;
+    }
+
+    matrix = (const __be32 *)prop->data;
+    for ( i = 0; i + 2 < entry_count; i += 3 )
+    {
+        uint32_t from, to, distance;
+
+        from = dt_read_number(matrix, 1);
+        matrix++;

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+        to = dt_read_number(matrix, 1);
+        matrix++;
+        distance = dt_read_number(matrix, 1);
+        matrix++;
+
+        if ( (from == to && distance != NUMA_LOCAL_DISTANCE) ||
+            (from != to && distance <= NUMA_LOCAL_DISTANCE) )
+        {
+            printk(XENLOG_WARNING
+                   "Invalid nodes' distance from node#%d to node#%d

= %d\n",

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5 -0
+                   from, to, distance);
+            return -EINVAL;
+        }
+
+        printk(XENLOG_INFO "NUMA: distance from node#%d to node#%d

= %d\n",

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 265

Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6 -0
+               from, to, distance);
+        numa_set_distance(from, to, distance);
+
+        /* Set default distance of node B->A same as A->B */
+        if (to > from)
+             numa_set_distance(to, from, distance);
Quoted 9 lines

I am a bit unsure about this last 2 lines: why calling numa_set_distance in the opposite direction only when to > from? Wouldn't it be OK to always do both:

numa_set_distance(from, to, distance);
numa_set_distance(to, from, distance);

?

I borrowed this code from Linux, but here is my understanding:

First, I read some notes in Documentation/devicetree/bindings/numa.txt
1. Each entry represents distance from first node to second node. The distances are equal in either direction.
2. distance-matrix should have entries in lexicographical ascending order of nodes.

Here is an example of distance-map node in DTB:
Sample#1, full list:

		distance-map {
			 compatible = "numa-distance-map-v1";
			 distance-matrix = <0 0  10>,
					   <0 1  20>,
					   <0 2  40>,
					   <0 3  20>,
					   <1 0  20>,
					   <1 1  10>,
					   <1 2  20>,
					   <1 3  40>,
					   <2 0  40>,
					   <2 1  20>,
					   <2 2  10>,
					   <2 3  20>,
					   <3 0  20>,
					   <3 1  40>,
					   <3 2  20>,
					   <3 3  10>;
		};

Call numa_set_distance when "to > from" will prevent Xen to call numa_set_distance(0, 1, 20) again when it's setting distance for <1 0 20>. But, numa_set_distance(1, 0, 20) will be call twice.

Normally, distance-map node will be optimized in following sample#2, all redundant entries are removed:
Sample#2, partial list:

		distance-map {
			 compatible = "numa-distance-map-v1";
			 distance-matrix = <0 0  10>,
					   <0 1  20>,
					   <0 2  40>,
					   <0 3  20>,
					   <1 1  10>,
					   <1 2  20>,
					   <1 3  40>,
					   <2 2  10>,
					   <2 3  20>,
					   <3 3  10>;
		};
Quoted 3 lines

There is not any "from > to" entry in the map. But using this partial map still can set all distances for all pairs. And numa_set_distance(1, 0, 20) will be only once.

I see. I can't find in Documentation/devicetree/bindings/numa.txt where it says that "from > to" nodes can be omitted. If it is not written down, then somebody could easily optimize it the opposite way:

distance-matrix = <0 0 10>,

 					   <1 0  20>,
 					   <2 0  40>,
 					   <3 0  20>,
 					   <1 1  10>,
 					   <2 1  20>,
 					   <3 1  40>,
 					   <2 2  10>,
 					   <3 2  20>,
 					   <3 3  10>;

I think the code in Xen should be resilient and able to cope with a device tree like the one you wrote or the one I wrote. From a code perspective, it should be very easy to do. If nothing else it would make Xen more resilient against buggy firmware.

Quoted 21 lines
Quoted 5 lines

But in any case, I have a different suggestion. The binding states that "distances are equal in either direction". Also it has an example where only one direction is expressed unfortunately (at the end of the document).

Oh, I should see this comment first, then I will not post above comment : )

Quoted 11 lines

So my suggestion is to parse it as follows:

- call numa_set_distance just once from
device_tree_parse_numa_distance_map_v1

- in numa_set_distance:
- set node_distance_map[from][to] = distance;
- check node_distance_map[to][from]
- if unset, node_distance_map[to][from] = distance;
- if already set to the same value, return success;
- if already set to a different value, return error;

Quoted 5 lines

I don't really like this implementation. I want the behavior of numa_set_distance just like the function name, do not include implicit operations. Otherwise, except the user read this function implementation before he use it, he probably doesn't know this function has done so many things.

You can leave numa_set_distance as-is without any implicit operations.

In that case, just call numa_set_distance twice from numa_set_distance for both from/to and to/from. numa_set_distance could return error is the entry was already set to a different value or success otherwise (also in the case it was already set to the same value). This would allow Xen to cope with both scenarios above.

author
#121 lore raw

RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

Hi Stefano,

Quoted 188 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Stefano Stabellini
Sent: 2021年9月1日 5:36
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

Quoted 7 lines · Wei Chen

Hi Stefano,

Quoted 5 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 8:48
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];

[email protected];

Quoted 2 lines
Quoted 2 lines

[email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to

parse

Quoted 10 lines
Quoted 10 lines

device tree NUMA distance map

Quoted 7 lines · Wei Chen

A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a new helper to parse this distance map.

---
 xen/arch/arm/numa_device_tree.c | 67

+++++++++++++++++++++++++++++++++

Quoted 8 lines
Quoted 8 lines

Review

Quoted 3 lines
 1 file changed, 67 insertions(+)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void

Quoted 4 lines

index bbe081dcd1..6e0d1d3d9f 100644

xen/arch/arm/numa_device_tree.c +0 -0
@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void

*fdt,

Quoted 12 lines
Quoted 12 lines

int node,

Quoted 11 lines
xen/arch/arm/numa_device_tree.c +8 -0
     return 0;
 }
+
+/* Parse NUMA distance map v1 */
+int __init
+device_tree_parse_numa_distance_map_v1(const void *fdt, int node)
+{
+    const struct fdt_property *prop;
+    const __be32 *matrix;
+    int entry_count, len, i;
Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+
+    printk(XENLOG_INFO "NUMA: parsing numa-distance-map\n");
+
+    prop = fdt_get_property(fdt, node, "distance-matrix", &len);
+    if ( !prop )
+    {
+        printk(XENLOG_WARNING
+               "NUMA: No distance-matrix property in distance-

map\n");

Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+
+        return -EINVAL;
+    }
+
+    if ( len % sizeof(uint32_t) != 0 )
+    {
+        printk(XENLOG_WARNING
+               "distance-matrix in node is not a multiple of

u32\n");

Quoted 35 lines
Quoted 35 lines
Quoted 18 lines
xen/arch/arm/numa_device_tree.c +18 -0
+        return -EINVAL;
+    }
+
+    entry_count = len / sizeof(uint32_t);
+    if ( entry_count <= 0 )
+    {
+        printk(XENLOG_WARNING "NUMA: Invalid distance-matrix\n");
+
+        return -EINVAL;
+    }
+
+    matrix = (const __be32 *)prop->data;
+    for ( i = 0; i + 2 < entry_count; i += 3 )
+    {
+        uint32_t from, to, distance;
+
+        from = dt_read_number(matrix, 1);
+        matrix++;

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+        to = dt_read_number(matrix, 1);
+        matrix++;
+        distance = dt_read_number(matrix, 1);
+        matrix++;
+
+        if ( (from == to && distance != NUMA_LOCAL_DISTANCE) ||
+            (from != to && distance <= NUMA_LOCAL_DISTANCE) )
+        {
+            printk(XENLOG_WARNING
+                   "Invalid nodes' distance from node#%d to node#%d

= %d\n",

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5 -0
+                   from, to, distance);
+            return -EINVAL;
+        }
+
+        printk(XENLOG_INFO "NUMA: distance from node#%d to node#%d

= %d\n",

Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 265

Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6 -0
+               from, to, distance);
+        numa_set_distance(from, to, distance);
+
+        /* Set default distance of node B->A same as A->B */
+        if (to > from)
+             numa_set_distance(to, from, distance);

I am a bit unsure about this last 2 lines: why calling

numa_set_distance

Quoted 38 lines
Quoted 8 lines

in the opposite direction only when to > from? Wouldn't it be OK to always do both:

numa_set_distance(from, to, distance);
numa_set_distance(to, from, distance);

?

I borrowed this code from Linux, but here is my understanding:

First, I read some notes in Documentation/devicetree/bindings/numa.txt
1. Each entry represents distance from first node to second node. The distances are equal in either direction.
2. distance-matrix should have entries in lexicographical ascending order of nodes.

Here is an example of distance-map node in DTB:
Sample#1, full list:

		distance-map {
			 compatible = "numa-distance-map-v1";
			 distance-matrix = <0 0  10>,
					   <0 1  20>,
					   <0 2  40>,
					   <0 3  20>,
					   <1 0  20>,
					   <1 1  10>,
					   <1 2  20>,
					   <1 3  40>,
					   <2 0  40>,
					   <2 1  20>,
					   <2 2  10>,
					   <2 3  20>,
					   <3 0  20>,
					   <3 1  40>,
					   <3 2  20>,
					   <3 3  10>;
		};
Quoted 2 lines

Call numa_set_distance when "to > from" will prevent Xen to call numa_set_distance(0, 1, 20) again when it's setting distance for <1 0

20>.

Quoted 19 lines

But, numa_set_distance(1, 0, 20) will be call twice.

Normally, distance-map node will be optimized in following sample#2, all redundant entries are removed:
Sample#2, partial list:

		distance-map {
			 compatible = "numa-distance-map-v1";
			 distance-matrix = <0 0  10>,
					   <0 1  20>,
					   <0 2  40>,
					   <0 3  20>,
					   <1 1  10>,
					   <1 2  20>,
					   <1 3  40>,
					   <2 2  10>,
					   <2 3  20>,
					   <3 3  10>;
		};
Quoted 1 line

There is not any "from > to" entry in the map. But using this partial

map

Quoted 1 line

still can set all distances for all pairs. And numa_set_distance(1, 0,

20)

Quoted 1 line

will be only once.

I see. I can't find in Documentation/devicetree/bindings/numa.txt where it says that "from > to" nodes can be omitted. If it is not written down, then somebody could easily optimize it the opposite way:

Review

Quoted 10 lines
(quoted diff) +0 -0
 			 distance-matrix = <0 0  10>,
 					   <1 0  20>,
 					   <2 0  40>,
 					   <3 0  20>,
 					   <1 1  10>,
 					   <2 1  20>,
 					   <3 1  40>,
 					   <2 2  10>,
 					   <3 2  20>,
 					   <3 3  10>;

Yes, you're right. Spec doesn't say opposite way is unallowed.

Quoted 6 lines

I think the code in Xen should be resilient and able to cope with a device tree like the one you wrote or the one I wrote. From a code perspective, it should be very easy to do. If nothing else it would make Xen more resilient against buggy firmware.

I don't disagree with that.

Quoted 29 lines
Quoted 1 line
Quoted 1 line

But in any case, I have a different suggestion. The binding states

that

Quoted 1 line
Quoted 1 line

"distances are equal in either direction". Also it has an example

where

Quoted 19 lines
Quoted 3 lines

only one direction is expressed unfortunately (at the end of the document).

Oh, I should see this comment first, then I will not post above comment : )

Quoted 11 lines

So my suggestion is to parse it as follows:

- call numa_set_distance just once from
device_tree_parse_numa_distance_map_v1

- in numa_set_distance:
- set node_distance_map[from][to] = distance;
- check node_distance_map[to][from]
- if unset, node_distance_map[to][from] = distance;
- if already set to the same value, return success;
- if already set to a different value, return error;

Quoted 5 lines

I don't really like this implementation. I want the behavior of numa_set_distance just like the function name, do not include implicit operations. Otherwise, except the user read this function implementation before he use it, he probably doesn't know this function has done so many things.

Quoted 4 lines

You can leave numa_set_distance as-is without any implicit operations.

In that case, just call numa_set_distance twice from numa_set_distance for both from/to and to/from. numa_set_distance could return error is

I am OK for the first sentence. But...

Quoted 2 lines

the entry was already set to a different value or success otherwise (also in the case it was already set to the same value). This would

... I prefer not to check the previous value. Subsequent numa_set_distance call will override previous calls. Keep numa_set_distance as simple as it can. And when you pass new data to numa_set_distance, it doesn't know whether the previous data was correct or the new data is correct. Only caller may have known.

Quoted 1 line

allow Xen to cope with both scenarios above.

#122 lore raw

RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

Quoted 248 lines · Wei Chen

Hi Stefano,

Quoted 188 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Stefano Stabellini
Sent: 2021年9月1日 5:36
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

Quoted 7 lines · Wei Chen

Hi Stefano,

Quoted 5 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 8:48
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];

[email protected];

Quoted 2 lines
Quoted 2 lines

[email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to

parse

Quoted 10 lines
Quoted 10 lines

device tree NUMA distance map

Quoted 7 lines · Wei Chen

A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a new helper to parse this distance map.

---
 xen/arch/arm/numa_device_tree.c | 67

+++++++++++++++++++++++++++++++++

Quoted 8 lines
Quoted 8 lines

Review

Quoted 3 lines
 1 file changed, 67 insertions(+)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void

Quoted 4 lines

index bbe081dcd1..6e0d1d3d9f 100644

xen/arch/arm/numa_device_tree.c +0 -0
@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void

*fdt,

Quoted 12 lines
Quoted 12 lines

int node,

Quoted 11 lines
xen/arch/arm/numa_device_tree.c +8 -0
     return 0;
 }
+
+/* Parse NUMA distance map v1 */
+int __init
+device_tree_parse_numa_distance_map_v1(const void *fdt, int node)
+{
+    const struct fdt_property *prop;
+    const __be32 *matrix;
+    int entry_count, len, i;
Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+
+    printk(XENLOG_INFO "NUMA: parsing numa-distance-map\n");
+
+    prop = fdt_get_property(fdt, node, "distance-matrix", &len);
+    if ( !prop )
+    {
+        printk(XENLOG_WARNING
+               "NUMA: No distance-matrix property in distance-

map\n");

Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+
+        return -EINVAL;
+    }
+
+    if ( len % sizeof(uint32_t) != 0 )
+    {
+        printk(XENLOG_WARNING
+               "distance-matrix in node is not a multiple of

u32\n");

Quoted 35 lines
Quoted 35 lines
Quoted 18 lines
xen/arch/arm/numa_device_tree.c +18 -0
+        return -EINVAL;
+    }
+
+    entry_count = len / sizeof(uint32_t);
+    if ( entry_count <= 0 )
+    {
+        printk(XENLOG_WARNING "NUMA: Invalid distance-matrix\n");
+
+        return -EINVAL;
+    }
+
+    matrix = (const __be32 *)prop->data;
+    for ( i = 0; i + 2 < entry_count; i += 3 )
+    {
+        uint32_t from, to, distance;
+
+        from = dt_read_number(matrix, 1);
+        matrix++;

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+        to = dt_read_number(matrix, 1);
+        matrix++;
+        distance = dt_read_number(matrix, 1);
+        matrix++;
+
+        if ( (from == to && distance != NUMA_LOCAL_DISTANCE) ||
+            (from != to && distance <= NUMA_LOCAL_DISTANCE) )
+        {
+            printk(XENLOG_WARNING
+                   "Invalid nodes' distance from node#%d to node#%d

= %d\n",

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5 -0
+                   from, to, distance);
+            return -EINVAL;
+        }
+
+        printk(XENLOG_INFO "NUMA: distance from node#%d to node#%d

= %d\n",

Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 265

Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6 -0
+               from, to, distance);
+        numa_set_distance(from, to, distance);
+
+        /* Set default distance of node B->A same as A->B */
+        if (to > from)
+             numa_set_distance(to, from, distance);

I am a bit unsure about this last 2 lines: why calling

numa_set_distance

Quoted 38 lines
Quoted 8 lines

in the opposite direction only when to > from? Wouldn't it be OK to always do both:

numa_set_distance(from, to, distance);
numa_set_distance(to, from, distance);

?

I borrowed this code from Linux, but here is my understanding:

First, I read some notes in Documentation/devicetree/bindings/numa.txt
1. Each entry represents distance from first node to second node. The distances are equal in either direction.
2. distance-matrix should have entries in lexicographical ascending order of nodes.

Here is an example of distance-map node in DTB:
Sample#1, full list:

		distance-map {
			 compatible = "numa-distance-map-v1";
			 distance-matrix = <0 0  10>,
					   <0 1  20>,
					   <0 2  40>,
					   <0 3  20>,
					   <1 0  20>,
					   <1 1  10>,
					   <1 2  20>,
					   <1 3  40>,
					   <2 0  40>,
					   <2 1  20>,
					   <2 2  10>,
					   <2 3  20>,
					   <3 0  20>,
					   <3 1  40>,
					   <3 2  20>,
					   <3 3  10>;
		};
Quoted 2 lines

Call numa_set_distance when "to > from" will prevent Xen to call numa_set_distance(0, 1, 20) again when it's setting distance for <1 0

20>.

Quoted 19 lines

But, numa_set_distance(1, 0, 20) will be call twice.

Normally, distance-map node will be optimized in following sample#2, all redundant entries are removed:
Sample#2, partial list:

		distance-map {
			 compatible = "numa-distance-map-v1";
			 distance-matrix = <0 0  10>,
					   <0 1  20>,
					   <0 2  40>,
					   <0 3  20>,
					   <1 1  10>,
					   <1 2  20>,
					   <1 3  40>,
					   <2 2  10>,
					   <2 3  20>,
					   <3 3  10>;
		};
Quoted 1 line

There is not any "from > to" entry in the map. But using this partial

map

Quoted 1 line

still can set all distances for all pairs. And numa_set_distance(1, 0,

20)

Quoted 1 line

will be only once.

I see. I can't find in Documentation/devicetree/bindings/numa.txt where it says that "from > to" nodes can be omitted. If it is not written down, then somebody could easily optimize it the opposite way:

Review

Quoted 10 lines
(quoted diff) +0 -0
 			 distance-matrix = <0 0  10>,
 					   <1 0  20>,
 					   <2 0  40>,
 					   <3 0  20>,
 					   <1 1  10>,
 					   <2 1  20>,
 					   <3 1  40>,
 					   <2 2  10>,
 					   <3 2  20>,
 					   <3 3  10>;

Yes, you're right. Spec doesn't say opposite way is unallowed.

Quoted 6 lines

I think the code in Xen should be resilient and able to cope with a device tree like the one you wrote or the one I wrote. From a code perspective, it should be very easy to do. If nothing else it would make Xen more resilient against buggy firmware.

I don't disagree with that.

Quoted 29 lines
Quoted 1 line
Quoted 1 line

But in any case, I have a different suggestion. The binding states

that

Quoted 1 line
Quoted 1 line

"distances are equal in either direction". Also it has an example

where

Quoted 19 lines
Quoted 3 lines

only one direction is expressed unfortunately (at the end of the document).

Oh, I should see this comment first, then I will not post above comment : )

Quoted 11 lines

So my suggestion is to parse it as follows:

- call numa_set_distance just once from
device_tree_parse_numa_distance_map_v1

- in numa_set_distance:
- set node_distance_map[from][to] = distance;
- check node_distance_map[to][from]
- if unset, node_distance_map[to][from] = distance;
- if already set to the same value, return success;
- if already set to a different value, return error;

Quoted 5 lines

I don't really like this implementation. I want the behavior of numa_set_distance just like the function name, do not include implicit operations. Otherwise, except the user read this function implementation before he use it, he probably doesn't know this function has done so many things.

Quoted 4 lines

You can leave numa_set_distance as-is without any implicit operations.

In that case, just call numa_set_distance twice from numa_set_distance for both from/to and to/from. numa_set_distance could return error is

I am OK for the first sentence. But...

Quoted 8 lines
Quoted 2 lines

the entry was already set to a different value or success otherwise (also in the case it was already set to the same value). This would

... I prefer not to check the previous value. Subsequent numa_set_distance call will override previous calls. Keep numa_set_distance as simple as it can. And when you pass new data to numa_set_distance, it doesn't know whether the previous data was correct or the new data is correct. Only caller may have known.

That might be OK but if not numa_set_distance then somebody else needs to check against overwriting previous values. That is to be able to spot bad device tree cases like:

0 1 20
1 0 40

author
#123 lore raw

RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

Hi Stefano,

Quoted 282 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月2日 0:22
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

Quoted 11 lines · Wei Chen

Hi Stefano,

Quoted 9 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Stefano Stabellini
Sent: 2021年9月1日 5:36
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to

parse

Quoted 25 lines
Quoted 25 lines

device tree NUMA distance map

Quoted 7 lines · Wei Chen

Hi Stefano,

Quoted 5 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 8:48
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];

[email protected];

Quoted 2 lines
Quoted 2 lines

[email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to

parse

Quoted 10 lines
Quoted 10 lines

device tree NUMA distance map

Quoted 7 lines · Wei Chen

A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a new helper to parse this distance map.

---
 xen/arch/arm/numa_device_tree.c | 67

+++++++++++++++++++++++++++++++++

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 3 lines
 1 file changed, 67 insertions(+)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const

Quoted 4 lines

index bbe081dcd1..6e0d1d3d9f 100644

xen/arch/arm/numa_device_tree.c +0 -0
@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const

void

Quoted 9 lines
Quoted 9 lines

*fdt,

Quoted 8 lines
Quoted 8 lines

int node,

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 7 lines
xen/arch/arm/numa_device_tree.c +4 -0
     return 0;
 }
+
+/* Parse NUMA distance map v1 */
+int __init
+device_tree_parse_numa_distance_map_v1(const void *fdt, int

node)

Quoted 22 lines
Quoted 22 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4 -0
+{
+    const struct fdt_property *prop;
+    const __be32 *matrix;
+    int entry_count, len, i;
Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+
+    printk(XENLOG_INFO "NUMA: parsing numa-distance-map\n");
+
+    prop = fdt_get_property(fdt, node, "distance-matrix", &len);
+    if ( !prop )
+    {
+        printk(XENLOG_WARNING
+               "NUMA: No distance-matrix property in distance-

map\n");

Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+
+        return -EINVAL;
+    }
+
+    if ( len % sizeof(uint32_t) != 0 )
+    {
+        printk(XENLOG_WARNING
+               "distance-matrix in node is not a multiple of

u32\n");

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 7 lines
xen/arch/arm/numa_device_tree.c +7 -0
+        return -EINVAL;
+    }
+
+    entry_count = len / sizeof(uint32_t);
+    if ( entry_count <= 0 )
+    {
+        printk(XENLOG_WARNING "NUMA: Invalid distance-

matrix\n");

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
xen/arch/arm/numa_device_tree.c +13 -0
+
+        return -EINVAL;
+    }
+
+    matrix = (const __be32 *)prop->data;
+    for ( i = 0; i + 2 < entry_count; i += 3 )
+    {
+        uint32_t from, to, distance;
+
+        from = dt_read_number(matrix, 1);
+        matrix++;
+        to = dt_read_number(matrix, 1);
+        matrix++;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+        distance = dt_read_number(matrix, 1);
+        matrix++;
+
+        if ( (from == to && distance != NUMA_LOCAL_DISTANCE) ||
+            (from != to && distance <= NUMA_LOCAL_DISTANCE) )
+        {
+            printk(XENLOG_WARNING
+                   "Invalid nodes' distance from node#%d to

node#%d

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

= %d\n",

Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5 -0
+                   from, to, distance);
+            return -EINVAL;
+        }
+
+        printk(XENLOG_INFO "NUMA: distance from node#%d to

node#%d

Quoted 11 lines
Quoted 11 lines
Quoted 9 lines
Quoted 9 lines

= %d\n",

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 265

Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6 -0
+               from, to, distance);
+        numa_set_distance(from, to, distance);
+
+        /* Set default distance of node B->A same as A->B */
+        if (to > from)
+             numa_set_distance(to, from, distance);

I am a bit unsure about this last 2 lines: why calling

numa_set_distance

Quoted 1 line
Quoted 1 line

in the opposite direction only when to > from? Wouldn't it be OK

to

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

always do both:

numa_set_distance(from, to, distance);
numa_set_distance(to, from, distance);

?

I borrowed this code from Linux, but here is my understanding:

First, I read some notes in

Documentation/devicetree/bindings/numa.txt

Quoted 27 lines
Quoted 27 lines
Quoted 27 lines

1. Each entry represents distance from first node to second node. The distances are equal in either direction.
2. distance-matrix should have entries in lexicographical ascending order of nodes.

Here is an example of distance-map node in DTB:
Sample#1, full list:

		distance-map {
			 compatible = "numa-distance-map-v1";
			 distance-matrix = <0 0  10>,
					   <0 1  20>,
					   <0 2  40>,
					   <0 3  20>,
					   <1 0  20>,
					   <1 1  10>,
					   <1 2  20>,
					   <1 3  40>,
					   <2 0  40>,
					   <2 1  20>,
					   <2 2  10>,
					   <2 3  20>,
					   <3 0  20>,
					   <3 1  40>,
					   <3 2  20>,
					   <3 3  10>;
		};
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Call numa_set_distance when "to > from" will prevent Xen to call numa_set_distance(0, 1, 20) again when it's setting distance for <1

0

Quoted 20 lines
Quoted 20 lines

20>.

Quoted 19 lines

But, numa_set_distance(1, 0, 20) will be call twice.

Normally, distance-map node will be optimized in following sample#2, all redundant entries are removed:
Sample#2, partial list:

		distance-map {
			 compatible = "numa-distance-map-v1";
			 distance-matrix = <0 0  10>,
					   <0 1  20>,
					   <0 2  40>,
					   <0 3  20>,
					   <1 1  10>,
					   <1 2  20>,
					   <1 3  40>,
					   <2 2  10>,
					   <2 3  20>,
					   <3 3  10>;
		};
Quoted 1 line
Quoted 1 line
Quoted 1 line

There is not any "from > to" entry in the map. But using this

partial

Quoted 2 lines
Quoted 2 lines

map

Quoted 1 line

still can set all distances for all pairs. And numa_set_distance(1,

0,

Quoted 4 lines
Quoted 4 lines

20)

Quoted 1 line

will be only once.

I see. I can't find in Documentation/devicetree/bindings/numa.txt

where

Quoted 15 lines
Quoted 4 lines

it says that "from > to" nodes can be omitted. If it is not written down, then somebody could easily optimize it the opposite way:

Review

Quoted 10 lines
(quoted diff) +0 -0
 			 distance-matrix = <0 0  10>,
 					   <1 0  20>,
 					   <2 0  40>,
 					   <3 0  20>,
 					   <1 1  10>,
 					   <2 1  20>,
 					   <3 1  40>,
 					   <2 2  10>,
 					   <3 2  20>,
 					   <3 3  10>;
Quoted 5 lines

Yes, you're right. Spec doesn't say opposite way is unallowed.

Quoted 3 lines

I think the code in Xen should be resilient and able to cope with a device tree like the one you wrote or the one I wrote. From a code perspective, it should be very easy to do. If nothing else it would

make

Quoted 40 lines
Quoted 3 lines

Xen more resilient against buggy firmware.

I don't disagree with that.

Quoted 29 lines
Quoted 1 line
Quoted 1 line

But in any case, I have a different suggestion. The binding states

that

Quoted 1 line
Quoted 1 line

"distances are equal in either direction". Also it has an example

where

Quoted 19 lines
Quoted 3 lines

only one direction is expressed unfortunately (at the end of the document).

Oh, I should see this comment first, then I will not post above comment : )

Quoted 11 lines

So my suggestion is to parse it as follows:

- call numa_set_distance just once from
device_tree_parse_numa_distance_map_v1

- in numa_set_distance:
- set node_distance_map[from][to] = distance;
- check node_distance_map[to][from]
- if unset, node_distance_map[to][from] = distance;
- if already set to the same value, return success;
- if already set to a different value, return error;

Quoted 5 lines

I don't really like this implementation. I want the behavior of numa_set_distance just like the function name, do not include implicit operations. Otherwise, except the user read this function implementation before he use it, he probably doesn't know this function has done so many things.

Quoted 4 lines

You can leave numa_set_distance as-is without any implicit operations.

In that case, just call numa_set_distance twice from numa_set_distance for both from/to and to/from. numa_set_distance could return error is

Quoted 6 lines

I am OK for the first sentence. But...

Quoted 2 lines

the entry was already set to a different value or success otherwise (also in the case it was already set to the same value). This would

... I prefer not to check the previous value. Subsequent

numa_set_distance

Quoted 4 lines

call will override previous calls. Keep numa_set_distance as simple as it can. And when you pass new data to numa_set_distance, it doesn't know whether the previous data was correct or the new data is correct. Only caller may have known.

Quoted 6 lines

That might be OK but if not numa_set_distance then somebody else needs to check against overwriting previous values. That is to be able to spot bad device tree cases like:

0 1 20
1 0 40

How about we check it still in NUMA distance parse function? Before setting the numa_set_distance for one pair nodes (e.g. a -> b), we can get its opposite way distance first.

distance_b_a = __node_distance(b, a); ==> get opposite way distance. if (distance_b_a == 0) ==> opposite way distance has not been set

{
    numa_set_distance(a, b, 20); ==> set both
    numa_set_distance(b, a, 20)
} else {
    if (distance_b_a == 20) ==> opposite way distance has been set
       numa_set_distance(a, b, 20); ==> set this way only
    else ===> opposite way distance has been set, but is unmatched
       // What can we do here?
       Panic the system? or Just warning users? Or choose the bigger
       distance for both ways?
       
       And distance_b_a == NUMA_NO_DISTANCE would be a special case
       here.
}
#124 lore raw

RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

Quoted 298 lines · Wei Chen

Hi Stefano,

Quoted 282 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月2日 0:22
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

Quoted 11 lines · Wei Chen

Hi Stefano,

Quoted 9 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Stefano Stabellini
Sent: 2021年9月1日 5:36
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to

parse

Quoted 25 lines
Quoted 25 lines

device tree NUMA distance map

Quoted 7 lines · Wei Chen

Hi Stefano,

Quoted 5 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 8:48
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];

[email protected];

Quoted 2 lines
Quoted 2 lines

[email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to

parse

Quoted 10 lines
Quoted 10 lines

device tree NUMA distance map

Quoted 7 lines · Wei Chen

A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a new helper to parse this distance map.

---
 xen/arch/arm/numa_device_tree.c | 67

+++++++++++++++++++++++++++++++++

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 3 lines
 1 file changed, 67 insertions(+)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const

Quoted 4 lines

index bbe081dcd1..6e0d1d3d9f 100644

xen/arch/arm/numa_device_tree.c +0 -0
@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const

void

Quoted 9 lines
Quoted 9 lines

*fdt,

Quoted 8 lines
Quoted 8 lines

int node,

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const · line 205

Quoted 7 lines
xen/arch/arm/numa_device_tree.c +4 -0
     return 0;
 }
+
+/* Parse NUMA distance map v1 */
+int __init
+device_tree_parse_numa_distance_map_v1(const void *fdt, int

node)

Quoted 22 lines
Quoted 22 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4 -0
+{
+    const struct fdt_property *prop;
+    const __be32 *matrix;
+    int entry_count, len, i;
Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const · line 217

Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+
+    printk(XENLOG_INFO "NUMA: parsing numa-distance-map\n");
+
+    prop = fdt_get_property(fdt, node, "distance-matrix", &len);
+    if ( !prop )
+    {
+        printk(XENLOG_WARNING
+               "NUMA: No distance-matrix property in distance-

map\n");

Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const · line 225

Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+
+        return -EINVAL;
+    }
+
+    if ( len % sizeof(uint32_t) != 0 )
+    {
+        printk(XENLOG_WARNING
+               "distance-matrix in node is not a multiple of

u32\n");

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const · line 232

Quoted 7 lines
xen/arch/arm/numa_device_tree.c +7 -0
+        return -EINVAL;
+    }
+
+    entry_count = len / sizeof(uint32_t);
+    if ( entry_count <= 0 )
+    {
+        printk(XENLOG_WARNING "NUMA: Invalid distance-

matrix\n");

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
xen/arch/arm/numa_device_tree.c +13 -0
+
+        return -EINVAL;
+    }
+
+    matrix = (const __be32 *)prop->data;
+    for ( i = 0; i + 2 < entry_count; i += 3 )
+    {
+        uint32_t from, to, distance;
+
+        from = dt_read_number(matrix, 1);
+        matrix++;
+        to = dt_read_number(matrix, 1);
+        matrix++;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const · line 253

Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+        distance = dt_read_number(matrix, 1);
+        matrix++;
+
+        if ( (from == to && distance != NUMA_LOCAL_DISTANCE) ||
+            (from != to && distance <= NUMA_LOCAL_DISTANCE) )
+        {
+            printk(XENLOG_WARNING
+                   "Invalid nodes' distance from node#%d to

node#%d

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

= %d\n",

Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5 -0
+                   from, to, distance);
+            return -EINVAL;
+        }
+
+        printk(XENLOG_INFO "NUMA: distance from node#%d to

node#%d

Quoted 11 lines
Quoted 11 lines
Quoted 9 lines
Quoted 9 lines

= %d\n",

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const · line 264

Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6 -0
+               from, to, distance);
+        numa_set_distance(from, to, distance);
+
+        /* Set default distance of node B->A same as A->B */
+        if (to > from)
+             numa_set_distance(to, from, distance);

I am a bit unsure about this last 2 lines: why calling

numa_set_distance

Quoted 1 line
Quoted 1 line

in the opposite direction only when to > from? Wouldn't it be OK

to

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

always do both:

numa_set_distance(from, to, distance);
numa_set_distance(to, from, distance);

?

I borrowed this code from Linux, but here is my understanding:

First, I read some notes in

Documentation/devicetree/bindings/numa.txt

Quoted 27 lines
Quoted 27 lines
Quoted 27 lines

1. Each entry represents distance from first node to second node. The distances are equal in either direction.
2. distance-matrix should have entries in lexicographical ascending order of nodes.

Here is an example of distance-map node in DTB:
Sample#1, full list:

		distance-map {
			 compatible = "numa-distance-map-v1";
			 distance-matrix = <0 0  10>,
					   <0 1  20>,
					   <0 2  40>,
					   <0 3  20>,
					   <1 0  20>,
					   <1 1  10>,
					   <1 2  20>,
					   <1 3  40>,
					   <2 0  40>,
					   <2 1  20>,
					   <2 2  10>,
					   <2 3  20>,
					   <3 0  20>,
					   <3 1  40>,
					   <3 2  20>,
					   <3 3  10>;
		};
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Call numa_set_distance when "to > from" will prevent Xen to call numa_set_distance(0, 1, 20) again when it's setting distance for <1

0

Quoted 20 lines
Quoted 20 lines

20>.

Quoted 19 lines

But, numa_set_distance(1, 0, 20) will be call twice.

Normally, distance-map node will be optimized in following sample#2, all redundant entries are removed:
Sample#2, partial list:

		distance-map {
			 compatible = "numa-distance-map-v1";
			 distance-matrix = <0 0  10>,
					   <0 1  20>,
					   <0 2  40>,
					   <0 3  20>,
					   <1 1  10>,
					   <1 2  20>,
					   <1 3  40>,
					   <2 2  10>,
					   <2 3  20>,
					   <3 3  10>;
		};
Quoted 1 line
Quoted 1 line
Quoted 1 line

There is not any "from > to" entry in the map. But using this

partial

Quoted 2 lines
Quoted 2 lines

map

Quoted 1 line

still can set all distances for all pairs. And numa_set_distance(1,

0,

Quoted 4 lines
Quoted 4 lines

20)

Quoted 1 line

will be only once.

I see. I can't find in Documentation/devicetree/bindings/numa.txt

where

Quoted 15 lines
Quoted 4 lines

it says that "from > to" nodes can be omitted. If it is not written down, then somebody could easily optimize it the opposite way:

Review

Quoted 10 lines
(quoted diff) +0 -0
 			 distance-matrix = <0 0  10>,
 					   <1 0  20>,
 					   <2 0  40>,
 					   <3 0  20>,
 					   <1 1  10>,
 					   <2 1  20>,
 					   <3 1  40>,
 					   <2 2  10>,
 					   <3 2  20>,
 					   <3 3  10>;
Quoted 5 lines

Yes, you're right. Spec doesn't say opposite way is unallowed.

Quoted 3 lines

I think the code in Xen should be resilient and able to cope with a device tree like the one you wrote or the one I wrote. From a code perspective, it should be very easy to do. If nothing else it would

make

Quoted 40 lines
Quoted 3 lines

Xen more resilient against buggy firmware.

I don't disagree with that.

Quoted 29 lines
Quoted 1 line
Quoted 1 line

But in any case, I have a different suggestion. The binding states

that

Quoted 1 line
Quoted 1 line

"distances are equal in either direction". Also it has an example

where

Quoted 19 lines
Quoted 3 lines

only one direction is expressed unfortunately (at the end of the document).

Oh, I should see this comment first, then I will not post above comment : )

Quoted 11 lines

So my suggestion is to parse it as follows:

- call numa_set_distance just once from
device_tree_parse_numa_distance_map_v1

- in numa_set_distance:
- set node_distance_map[from][to] = distance;
- check node_distance_map[to][from]
- if unset, node_distance_map[to][from] = distance;
- if already set to the same value, return success;
- if already set to a different value, return error;

Quoted 5 lines

I don't really like this implementation. I want the behavior of numa_set_distance just like the function name, do not include implicit operations. Otherwise, except the user read this function implementation before he use it, he probably doesn't know this function has done so many things.

Quoted 4 lines

You can leave numa_set_distance as-is without any implicit operations.

In that case, just call numa_set_distance twice from numa_set_distance for both from/to and to/from. numa_set_distance could return error is

Quoted 6 lines

I am OK for the first sentence. But...

Quoted 2 lines

the entry was already set to a different value or success otherwise (also in the case it was already set to the same value). This would

... I prefer not to check the previous value. Subsequent

numa_set_distance

Quoted 4 lines

call will override previous calls. Keep numa_set_distance as simple as it can. And when you pass new data to numa_set_distance, it doesn't know whether the previous data was correct or the new data is correct. Only caller may have known.

Quoted 6 lines

That might be OK but if not numa_set_distance then somebody else needs to check against overwriting previous values. That is to be able to spot bad device tree cases like:

0 1 20
1 0 40

How about we check it still in NUMA distance parse function? Before setting the numa_set_distance for one pair nodes (e.g. a -> b), we can get its opposite way distance first.

distance_b_a = __node_distance(b, a); ==> get opposite way distance. if (distance_b_a == 0) ==> opposite way distance has not been set

Quoted 14 lines
{
    numa_set_distance(a, b, 20); ==> set both
    numa_set_distance(b, a, 20)
} else {
    if (distance_b_a == 20) ==> opposite way distance has been set
       numa_set_distance(a, b, 20); ==> set this way only
    else ===> opposite way distance has been set, but is unmatched
       // What can we do here?
       Panic the system? or Just warning users? Or choose the bigger
       distance for both ways?
       
       And distance_b_a == NUMA_NO_DISTANCE would be a special case
       here.
}

Yes, I think something like this would work. If we detect an error in device tree I would probably print a warning and disable NUMA. It would also be acceptable to panic.

#125 lore raw

Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

In reply to Stefano Stabellini
Quoted 27 lines · Stefano Stabellini
Quoted 17 lines · Wei Chen
Quoted 10 lines

Stefano Stabellini
Sent: 2021年9月1日 5:36

Quoted 5 lines · Wei Chen

I don't really like this implementation. I want the behavior of numa_set_distance just like the function name, do not include implicit operations. Otherwise, except the user read this function implementation before he use it, he probably doesn't know this function has done so many things.

Quoted 4 lines

You can leave numa_set_distance as-is without any implicit operations.

In that case, just call numa_set_distance twice from numa_set_distance for both from/to and to/from. numa_set_distance could return error is

I am OK for the first sentence. But...

Quoted 8 lines
Quoted 2 lines

the entry was already set to a different value or success otherwise (also in the case it was already set to the same value). This would

... I prefer not to check the previous value. Subsequent numa_set_distance call will override previous calls. Keep numa_set_distance as simple as it can. And when you pass new data to numa_set_distance, it doesn't know whether the previous data was correct or the new data is correct. Only caller may have known.

Quoted 6 lines

That might be OK but if not numa_set_distance then somebody else needs to check against overwriting previous values. That is to be able to spot bad device tree cases like:

0 1 20
1 0 40

What's wrong with this? At least the ACPI spec cares to specifically permit this case:

"Except for the relative distance from a System Locality to itself,
each relative distance is stored twice in the matrix. This provides the capability to describe the scenario where the relative distances for the two directions between System Localities is different."

Jan

author
#126 lore raw

RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

Hi Jan,

Quoted 43 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年9月2日 14:00
To: Stefano Stabellini <[email protected]>; Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

Quoted 16 lines · Stefano Stabellini
Quoted 15 lines · Wei Chen
Quoted 14 lines

Stefano Stabellini
Sent: 2021年9月1日 5:36

Quoted 5 lines · Wei Chen

I don't really like this implementation. I want the behavior of numa_set_distance just like the function name, do not include implicit operations. Otherwise, except the user read this function implementation before he use it, he probably doesn't know this function has done so many things.

You can leave numa_set_distance as-is without any implicit operations.

In that case, just call numa_set_distance twice from numa_set_distance for both from/to and to/from. numa_set_distance could return error is

Quoted 6 lines
Quoted 6 lines

I am OK for the first sentence. But...

Quoted 2 lines

the entry was already set to a different value or success otherwise (also in the case it was already set to the same value). This would

... I prefer not to check the previous value. Subsequent

numa_set_distance

Quoted 9 lines
Quoted 4 lines

call will override previous calls. Keep numa_set_distance as simple as it can. And when you pass new data to numa_set_distance, it doesn't know whether the previous data was correct or the new data is correct. Only caller may have known.

That might be OK but if not numa_set_distance then somebody else needs to check against overwriting previous values. That is to be able to spot bad device tree cases like:

Quoted 6 lines
Quoted 2 lines

0 1 20
1 0 40

What's wrong with this? At least the ACPI spec cares to specifically permit this case:

This is because some notes description in device tree NUMA binding:
- distance-matrix
This property defines a matrix to describe the relative distances between all numa nodes.
It is represented as a list of node pairs and their relative distance.

Note:
1. Each entry represents distance from first node to second node. The distances are equal in either direction.
https://www.kernel.org/doc/Documentation/devicetree/bindings/numa.txt

So we treat this case is wrong in DT. But as you said ACPI allow such case, that means real machines allow different distances for one pair nodes in different directions.

Quoted 6 lines

"Except for the relative distance from a System Locality to itself,
each relative distance is stored twice in the matrix. This provides the capability to describe the scenario where the relative distances for the two directions between System Localities is different."

Jan

author
#127 lore raw

RFC PATCH 25/40 xen/arm: unified entry to parse all NUMA data from device tree

In reply to Wei Chen

In this API, we scan whole device tree to parse CPU node id, memory node id and distance-map. Though early_scan_node will invoke has a handler to process memory nodes. If we want to parse memory node id in this handler, we have to embeded NUMA parse code in this handler. But we still need to scan whole device tree to find CPU NUMA id and distance-map. In this case, we include memory NUMA id parse in this API too. Another benefit is that we have a unique entry for device tree NUMA data parse.

---
 xen/arch/arm/numa_device_tree.c | 31 ++++++++++++++++++++++++++++---
 xen/include/asm-arm/numa.h      |  1 +
 2 files changed, 29 insertions(+), 3 deletions(-)
xen/arch/arm/numa_device_tree.c +28 -3
index 6e0d1d3d9f..27ffb72f7b 100644
@@ -131,7 +131,8 @@ save_memblk:
 }
 
 /* Parse CPU NUMA node info */
-int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
+static int __init
+device_tree_parse_numa_cpu_node(const void *fdt, int node)
 {
     uint32_t nid;
 
@@ -147,7 +148,7 @@ int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
 }
 
 /* Parse memory node NUMA info */
-int __init
+static int __init
 device_tree_parse_numa_memory_node(const void *fdt, int node,
     const char *name, uint32_t addr_cells, uint32_t size_cells)
 {
@@ -202,7 +203,7 @@ device_tree_parse_numa_memory_node(const void *fdt, int node,
 }
 
 /* Parse NUMA distance map v1 */
-int __init
+static int __init
 device_tree_parse_numa_distance_map_v1(const void *fdt, int node)
 {
     const struct fdt_property *prop;
@@ -267,3 +268,27 @@ device_tree_parse_numa_distance_map_v1(const void *fdt, int node)
 
     return 0;
 }
+
+static int __init fdt_scan_numa_nodes(const void *fdt,
+                int node, const char *uname, int depth,
+                u32 address_cells, u32 size_cells, void *data)
+{
+    int ret = 0;
+
+    if ( fdt_node_check_type(fdt, node, "cpu") == 0 )
+        ret = device_tree_parse_numa_cpu_node(fdt, node);
+    else if ( fdt_node_check_type(fdt, node, "memory") == 0 )
+        ret = device_tree_parse_numa_memory_node(fdt, node, uname,
+                                address_cells, size_cells);
+    else if ( fdt_node_check_compatible(fdt, node,
+                                "numa-distance-map-v1") == 0 )
+        ret = device_tree_parse_numa_distance_map_v1(fdt, node);
+
+    return ret;
+}
+
+/* Initialize NUMA from device tree */
+int __init numa_device_tree_init(const void *fdt)
+{
+    return device_tree_for_each_node(fdt, 0, fdt_scan_numa_nodes, NULL);
+}
xen/include/asm-arm/numa.h +1 -0
index 756ad82d07..7a3588ac7f 100644
@@ -26,6 +26,7 @@ typedef u8 nodeid_t;
 extern s8 device_tree_numa;
 
 extern void numa_init(bool acpi_off);
+extern int numa_device_tree_init(const void *fdt);
 extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_t distance);
 
 /*
Signature · 1 line
2.25.1
#128 lore raw

Re: [XEN RFC PATCH 25/40] xen/arm: unified entry to parse all NUMA data from device tree

Quoted 59 lines · Wei Chen

In this API, we scan whole device tree to parse CPU node id, memory node id and distance-map. Though early_scan_node will invoke has a handler to process memory nodes. If we want to parse memory node id in this handler, we have to embeded NUMA parse code in this handler. But we still need to scan whole device tree to find CPU NUMA id and distance-map. In this case, we include memory NUMA id parse in this API too. Another benefit is that we have a unique entry for device tree NUMA data parse.

---
 xen/arch/arm/numa_device_tree.c | 31 ++++++++++++++++++++++++++++---
 xen/include/asm-arm/numa.h      |  1 +
 2 files changed, 29 insertions(+), 3 deletions(-)
xen/arch/arm/numa_device_tree.c +12 -3
index 6e0d1d3d9f..27ffb72f7b 100644
@@ -131,7 +131,8 @@ save_memblk:
 }
 
 /* Parse CPU NUMA node info */
-int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
+static int __init
+device_tree_parse_numa_cpu_node(const void *fdt, int node)
 {
     uint32_t nid;
 
@@ -147,7 +148,7 @@ int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
 }
 
 /* Parse memory node NUMA info */
-int __init
+static int __init
 device_tree_parse_numa_memory_node(const void *fdt, int node,
     const char *name, uint32_t addr_cells, uint32_t size_cells)
 {
@@ -202,7 +203,7 @@ device_tree_parse_numa_memory_node(const void *fdt, int node,
 }
 
 /* Parse NUMA distance map v1 */
-int __init
+static int __init
 device_tree_parse_numa_distance_map_v1(const void *fdt, int node)
 {
     const struct fdt_property *prop;
@@ -267,3 +268,27 @@ device_tree_parse_numa_distance_map_v1(const void *fdt, int node)
 
     return 0;
 }
+
+static int __init fdt_scan_numa_nodes(const void *fdt,
+                int node, const char *uname, int depth,
+                u32 address_cells, u32 size_cells, void *data)
+{
+    int ret = 0;
+
+    if ( fdt_node_check_type(fdt, node, "cpu") == 0 )

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_distance_map_v1(const void *fdt, int node) · line 288

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+        ret = device_tree_parse_numa_cpu_node(fdt, node);
+    else if ( fdt_node_check_type(fdt, node, "memory") == 0 )
+        ret = device_tree_parse_numa_memory_node(fdt, node, uname,
+                                address_cells, size_cells);
+    else if ( fdt_node_check_compatible(fdt, node,
+                                "numa-distance-map-v1") == 0 )
+        ret = device_tree_parse_numa_distance_map_v1(fdt, node);
+
+    return ret;
+}

Julien, do you have an opinion on whether it might be worth reusing the existing early_scan_node function for this to avoiding another full FDT scan (to avoid another call to device_tree_for_each_node)?

#129 lore raw

Re: [XEN RFC PATCH 25/40] xen/arm: unified entry to parse all NUMA data from device tree

Hi Stefano,

Quoted 66 lines · Stefano Stabellini
Quoted 65 lines · Wei Chen

In this API, we scan whole device tree to parse CPU node id, memory node id and distance-map. Though early_scan_node will invoke has a handler to process memory nodes. If we want to parse memory node id in this handler, we have to embeded NUMA parse code in this handler. But we still need to scan whole device tree to find CPU NUMA id and distance-map. In this case, we include memory NUMA id parse in this API too. Another benefit is that we have a unique entry for device tree NUMA data parse.

---
  xen/arch/arm/numa_device_tree.c | 31 ++++++++++++++++++++++++++++---
  xen/include/asm-arm/numa.h      |  1 +
  2 files changed, 29 insertions(+), 3 deletions(-)
xen/arch/arm/numa_device_tree.c +18 -3
index 6e0d1d3d9f..27ffb72f7b 100644
@@ -131,7 +131,8 @@ save_memblk:
  }
 
  /* Parse CPU NUMA node info */
-int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
+static int __init
+device_tree_parse_numa_cpu_node(const void *fdt, int node)
  {
      uint32_t nid;
 
@@ -147,7 +148,7 @@ int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
  }
 
  /* Parse memory node NUMA info */
-int __init
+static int __init
  device_tree_parse_numa_memory_node(const void *fdt, int node,
      const char *name, uint32_t addr_cells, uint32_t size_cells)
  {
@@ -202,7 +203,7 @@ device_tree_parse_numa_memory_node(const void *fdt, int node,
  }
 
  /* Parse NUMA distance map v1 */
-int __init
+static int __init
  device_tree_parse_numa_distance_map_v1(const void *fdt, int node)
  {
      const struct fdt_property *prop;
@@ -267,3 +268,27 @@ device_tree_parse_numa_distance_map_v1(const void *fdt, int node)
 
      return 0;
  }
+
+static int __init fdt_scan_numa_nodes(const void *fdt,
+                int node, const char *uname, int depth,
+                u32 address_cells, u32 size_cells, void *data)
+{
+    int ret = 0;
+
+    if ( fdt_node_check_type(fdt, node, "cpu") == 0 )
+        ret = device_tree_parse_numa_cpu_node(fdt, node);
+    else if ( fdt_node_check_type(fdt, node, "memory") == 0 )
+        ret = device_tree_parse_numa_memory_node(fdt, node, uname,
+                                address_cells, size_cells);
+    else if ( fdt_node_check_compatible(fdt, node,
+                                "numa-distance-map-v1") == 0 )
Quoted 8 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_distance_map_v1(const void *fdt, int node) · line 288

Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4 -0
+        ret = device_tree_parse_numa_distance_map_v1(fdt, node);
+
+    return ret;
+}

Julien, do you have an opinion on whether it might be worth reusing the existing early_scan_node function for this to avoiding another full FDT scan (to avoid another call to device_tree_for_each_node)?

I don't like the full FDT scan and actually drafted an e-mail in reply-to [1] to suggest parse all the NUMA information from early_scan_node().

However, we don't know whether ACPI or DT will be used at the time early_scan_node() is called. So we will need to revert any change which can make the code a little more awkward.

So I decided to drop my e-mail because I prefer the full DT scan for now. We can look at optimizing later if this becomes a pain point.

Cheers,

[1] https://lore.kernel.org/xen-devel/DB9PR08MB6857604B3D4B690F2B8832BD9EC79@DB9PR08MB6857.eurprd08.prod.outlook.com/

Signature · 1 line
Julien Grall
#130 lore raw

Re: [XEN RFC PATCH 25/40] xen/arm: unified entry to parse all NUMA data from device tree

Quoted 80 lines · Julien Grall

Hi Stefano,

Quoted 65 lines · Stefano Stabellini
Quoted 64 lines · Wei Chen

In this API, we scan whole device tree to parse CPU node id, memory node id and distance-map. Though early_scan_node will invoke has a handler to process memory nodes. If we want to parse memory node id in this handler, we have to embeded NUMA parse code in this handler. But we still need to scan whole device tree to find CPU NUMA id and distance-map. In this case, we include memory NUMA id parse in this API too. Another benefit is that we have a unique entry for device tree NUMA data parse.

---
  xen/arch/arm/numa_device_tree.c | 31 ++++++++++++++++++++++++++++---
  xen/include/asm-arm/numa.h      |  1 +
  2 files changed, 29 insertions(+), 3 deletions(-)
xen/arch/arm/numa_device_tree.c +2 -1
index 6e0d1d3d9f..27ffb72f7b 100644
@@ -131,7 +131,8 @@ save_memblk:
  }
    /* Parse CPU NUMA node info */
-int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
+static int __init
+device_tree_parse_numa_cpu_node(const void *fdt, int node)
  {
      uint32_t nid;
  @@ -147,7 +148,7 @@ int __init device_tree_parse_numa_cpu_node(const
 void *fdt, int node)

}

    /* Parse memory node NUMA info */
xen/arch/arm/numa_device_tree.c +16 -2
-int __init
+static int __init
  device_tree_parse_numa_memory_node(const void *fdt, int node,
      const char *name, uint32_t addr_cells, uint32_t size_cells)
  {
@@ -202,7 +203,7 @@ device_tree_parse_numa_memory_node(const void *fdt,
 int node,
  }
    /* Parse NUMA distance map v1 */
-int __init
+static int __init
  device_tree_parse_numa_distance_map_v1(const void *fdt, int node)
  {
      const struct fdt_property *prop;
@@ -267,3 +268,27 @@ device_tree_parse_numa_distance_map_v1(const void
 *fdt, int node)
        return 0;
  }
+
+static int __init fdt_scan_numa_nodes(const void *fdt,
+                int node, const char *uname, int depth,
+                u32 address_cells, u32 size_cells, void *data)
+{
+    int ret = 0;
+
+    if ( fdt_node_check_type(fdt, node, "cpu") == 0 )
+        ret = device_tree_parse_numa_cpu_node(fdt, node);
+    else if ( fdt_node_check_type(fdt, node, "memory") == 0 )
+        ret = device_tree_parse_numa_memory_node(fdt, node, uname,
+                                address_cells, size_cells);
+    else if ( fdt_node_check_compatible(fdt, node,
+                                "numa-distance-map-v1") == 0 )
Quoted 8 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_distance_map_v1(const void *fdt, int node) · line 288

Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4 -0
+        ret = device_tree_parse_numa_distance_map_v1(fdt, node);
+
+    return ret;
+}

Julien, do you have an opinion on whether it might be worth reusing the existing early_scan_node function for this to avoiding another full FDT scan (to avoid another call to device_tree_for_each_node)?

I don't like the full FDT scan and actually drafted an e-mail in reply-to [1] to suggest parse all the NUMA information from early_scan_node().

Quoted 6 lines

However, we don't know whether ACPI or DT will be used at the time early_scan_node() is called. So we will need to revert any change which can make the code a little more awkward.

So I decided to drop my e-mail because I prefer the full DT scan for now. We can look at optimizing later if this becomes a pain point.

Uhm, yes you are right.

We would have to move some of the logic to detect ACPI earlier to early_scan_node (e.g. xen/arch/arm/acpi/boot.c:dt_scan_depth1_nodes). That could actually be a good idea, but it is true that could be done with a separate patch later.

author
#131 lore raw

RE: [XEN RFC PATCH 25/40] xen/arm: unified entry to parse all NUMA data from device tree

Quoted 108 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月2日 2:31
To: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>; Wei Chen <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 25/40] xen/arm: unified entry to parse all NUMA data from device tree

Quoted 16 lines · Julien Grall

Hi Stefano,

Quoted 13 lines · Stefano Stabellini
Quoted 12 lines · Wei Chen

In this API, we scan whole device tree to parse CPU node id, memory node id and distance-map. Though early_scan_node will invoke has a handler to process memory nodes. If we want to parse memory node id in this handler, we have to embeded NUMA parse code in this handler. But we still need to scan whole device tree to find CPU NUMA id and distance-map. In this case, we include memory NUMA id parse in this API too. Another benefit is that we have a unique entry for device tree NUMA data parse.

---
  xen/arch/arm/numa_device_tree.c | 31 ++++++++++++++++++++++++++++-

--

Quoted 12 lines
Quoted 12 lines

xen/arch/arm/numa_device_tree.c save_memblk: · line 133

Quoted 12 lines
  xen/include/asm-arm/numa.h      |  1 +
  2 files changed, 29 insertions(+), 3 deletions(-)
xen/arch/arm/numa_device_tree.c +0 -1
index 6e0d1d3d9f..27ffb72f7b 100644
@@ -131,7 +131,8 @@ save_memblk:
  }
    /* Parse CPU NUMA node info */
-int __init device_tree_parse_numa_cpu_node(const void *fdt, int

node)

Quoted 5 lines
Quoted 5 lines

xen/arch/arm/numa_device_tree.c save_memblk:

Quoted 5 lines
xen/arch/arm/numa_device_tree.c +2 -0
+static int __init
+device_tree_parse_numa_cpu_node(const void *fdt, int node)
  {
      uint32_t nid;
  @@ -147,7 +148,7 @@ int __init

device_tree_parse_numa_cpu_node(const

Quoted 9 lines
Quoted 9 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void

Quoted 9 lines
xen/arch/arm/numa_device_tree.c +0 -0
 void *fdt, int node)

}

    /* Parse memory node NUMA info */
xen/arch/arm/numa_device_tree.c +1 -1
-int __init
+static int __init
  device_tree_parse_numa_memory_node(const void *fdt, int node,
      const char *name, uint32_t addr_cells, uint32_t size_cells)
  {
@@ -202,7 +203,7 @@ device_tree_parse_numa_memory_node(const void

*fdt,

Quoted 9 lines
Quoted 9 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_distance_map_v1(const

Quoted 9 lines
xen/arch/arm/numa_device_tree.c +1 -1
 int node,
  }
    /* Parse NUMA distance map v1 */
-int __init
+static int __init
  device_tree_parse_numa_distance_map_v1(const void *fdt, int node)
  {
      const struct fdt_property *prop;
@@ -267,3 +268,27 @@ device_tree_parse_numa_distance_map_v1(const

void

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
xen/arch/arm/numa_device_tree.c +12 -0
 *fdt, int node)
        return 0;
  }
+
+static int __init fdt_scan_numa_nodes(const void *fdt,
+                int node, const char *uname, int depth,
+                u32 address_cells, u32 size_cells, void *data)
+{
+    int ret = 0;
+
+    if ( fdt_node_check_type(fdt, node, "cpu") == 0 )
+        ret = device_tree_parse_numa_cpu_node(fdt, node);
+    else if ( fdt_node_check_type(fdt, node, "memory") == 0 )
+        ret = device_tree_parse_numa_memory_node(fdt, node, uname,
+                                address_cells, size_cells);
Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_distance_map_v1(const void *fdt, int node) · line 288

Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6 -0
+    else if ( fdt_node_check_compatible(fdt, node,
+                                "numa-distance-map-v1") == 0 )
+        ret = device_tree_parse_numa_distance_map_v1(fdt, node);
+
+    return ret;
+}

Julien, do you have an opinion on whether it might be worth reusing

the

Quoted 1 line
Quoted 1 line

existing early_scan_node function for this to avoiding another full

FDT

Quoted 3 lines
Quoted 1 line

scan (to avoid another call to device_tree_for_each_node)?

I don't like the full FDT scan and actually drafted an e-mail in reply-

to [1]

Quoted 4 lines

to suggest parse all the NUMA information from early_scan_node().

However, we don't know whether ACPI or DT will be used at the time early_scan_node() is called. So we will need to revert any change which

can

Quoted 3 lines

make the code a little more awkward.

So I decided to drop my e-mail because I prefer the full DT scan for now.

We

Quoted 8 lines
Quoted 1 line

can look at optimizing later if this becomes a pain point.

Uhm, yes you are right.

We would have to move some of the logic to detect ACPI earlier to early_scan_node (e.g. xen/arch/arm/acpi/boot.c:dt_scan_depth1_nodes). That could actually be a good idea, but it is true that could be done with a separate patch later.

Sounds good to me.

author
#132 lore raw

RFC PATCH 26/40 xen/arm: Add boot and secondary CPU to NUMA system

In reply to Wei Chen

When cpu boot up, we have add them to NUMA system. In current stage, we have not parsed the NUMA data, but we have created a fake NUMA node. So, in this patch, all CPU will be added to NUMA node#0. After the NUMA data has been parsed from device tree, the CPU will be added to correct NUMA node as the NUMA data described.

---
 xen/arch/arm/setup.c       | 6 ++++++
 xen/arch/arm/smpboot.c     | 6 ++++++
 xen/include/asm-arm/numa.h | 1 +
 3 files changed, 13 insertions(+)
xen/arch/arm/setup.c +6 -0
index 3c58d2d441..7531989f21 100644
@@ -918,6 +918,12 @@ void __init start_xen(unsigned long boot_phys_offset,
 
     processor_id();
 
+    /*
+     * If Xen is running on a NUMA off system, there will
+     * be a node#0 at least.
+     */
+    numa_add_cpu(0);
+
     smp_init_cpus();
     cpus = smp_get_max_cpus();
     printk(XENLOG_INFO "SMP: Allowing %u CPUs\n", cpus);
xen/arch/arm/smpboot.c +6 -0
index a1ee3146ef..aa78958c07 100644
@@ -358,6 +358,12 @@ void start_secondary(void)
      */
     smp_wmb();
 
+    /*
+     * If Xen is running on a NUMA off system, there will
+     * be a node#0 at least.
+     */
+    numa_add_cpu(cpuid);
+
     /* Now report this CPU is up */
     cpumask_set_cpu(cpuid, &cpu_online_map);
 
xen/include/asm-arm/numa.h +1 -0
index 7a3588ac7f..dd31324b0b 100644
@@ -59,6 +59,7 @@ extern mfn_t first_valid_mfn;
 #define __node_distance(a, b) (20)
 
 #define numa_init(x) do { } while (0)
+#define numa_add_cpu(x) do { } while (0)
 #define numa_set_node(x, y) do { } while (0)
 
 #endif
Signature · 1 line
2.25.1
#133 lore raw

Re: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA system

Hi Wei,

Quoted 36 lines · Wei Chen

When cpu boot up, we have add them to NUMA system. In current stage, we have not parsed the NUMA data, but we have created a fake NUMA node. So, in this patch, all CPU will be added to NUMA node#0. After the NUMA data has been parsed from device tree, the CPU will be added to correct NUMA node as the NUMA data described.

---
  xen/arch/arm/setup.c       | 6 ++++++
  xen/arch/arm/smpboot.c     | 6 ++++++
  xen/include/asm-arm/numa.h | 1 +
  3 files changed, 13 insertions(+)
xen/arch/arm/setup.c +6 -0
index 3c58d2d441..7531989f21 100644
@@ -918,6 +918,12 @@ void __init start_xen(unsigned long boot_phys_offset,
 
      processor_id();
 
+    /*
+     * If Xen is running on a NUMA off system, there will
+     * be a node#0 at least.
+     */
+    numa_add_cpu(0);
+
      smp_init_cpus();
      cpus = smp_get_max_cpus();
      printk(XENLOG_INFO "SMP: Allowing %u CPUs\n", cpus);
xen/arch/arm/smpboot.c +0 -0
index a1ee3146ef..aa78958c07 100644

xen/arch/arm/smpboot.c void start_secondary(void) · line 366

Quoted 9 lines
xen/arch/arm/smpboot.c +6 -0
@@ -358,6 +358,12 @@ void start_secondary(void)
       */
      smp_wmb();
 
+    /*
+     * If Xen is running on a NUMA off system, there will
+     * be a node#0 at least.
+     */
+    numa_add_cpu(cpuid);
+

On x86, numa_add_cpu() will be called before the pCPU is brought up. I am not quite too sure why we are doing it differently here. Can you clarify it?

xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 62

Quoted 12 lines
xen/arch/arm/smpboot.c +0 -0
      /* Now report this CPU is up */
      cpumask_set_cpu(cpuid, &cpu_online_map);
 
xen/include/asm-arm/numa.h +1 -0
index 7a3588ac7f..dd31324b0b 100644
@@ -59,6 +59,7 @@ extern mfn_t first_valid_mfn;
  #define __node_distance(a, b) (20)
 
  #define numa_init(x) do { } while (0)
+#define numa_add_cpu(x) do { } while (0)

This is a stubs for a common helper. So I think this wants to be moved in the !CONFIG_NUMA in xen/numa.h.

xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 65

Quoted 4 lines
xen/include/asm-arm/numa.h +0 -0
  #define numa_set_node(x, y) do { } while (0)
 
  #endif

Cheers,

Signature · 1 line
Julien Grall
author
#134 lore raw

RE: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA system

Hi Julien,

Quoted 52 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 0:58
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA system

Hi Wei,

xen/arch/arm/setup.c void __init start_xen(unsigned long

Quoted 19 lines · Wei Chen

When cpu boot up, we have add them to NUMA system. In current stage, we have not parsed the NUMA data, but we have created a fake NUMA node. So, in this patch, all CPU will be added to NUMA node#0. After the NUMA data has been parsed from device tree, the CPU will be added to correct NUMA node as the NUMA data described.

---
  xen/arch/arm/setup.c       | 6 ++++++
  xen/arch/arm/smpboot.c     | 6 ++++++
  xen/include/asm-arm/numa.h | 1 +
  3 files changed, 13 insertions(+)
xen/arch/arm/setup.c +0 -0
index 3c58d2d441..7531989f21 100644
@@ -918,6 +918,12 @@ void __init start_xen(unsigned long

boot_phys_offset,

Quoted 20 lines
xen/arch/arm/setup.c +6 -0
      processor_id();
 
+    /*
+     * If Xen is running on a NUMA off system, there will
+     * be a node#0 at least.
+     */
+    numa_add_cpu(0);
+
      smp_init_cpus();
      cpus = smp_get_max_cpus();
      printk(XENLOG_INFO "SMP: Allowing %u CPUs\n", cpus);
xen/arch/arm/smpboot.c +0 -0
index a1ee3146ef..aa78958c07 100644
@@ -358,6 +358,12 @@ void start_secondary(void)
       */
      smp_wmb();
 
Quoted 10 lines

xen/arch/arm/smpboot.c void start_secondary(void) · line 366

Quoted 6 lines
xen/arch/arm/smpboot.c +6 -0
+    /*
+     * If Xen is running on a NUMA off system, there will
+     * be a node#0 at least.
+     */
+    numa_add_cpu(cpuid);
+

On x86, numa_add_cpu() will be called before the pCPU is brought up. I am not quite too sure why we are doing it differently here. Can you clarify it?

Of course we can invoke numa_add_cpu before cpu_up as x86. But in my tests, I found when cpu bring up failed, this cpu still be add to NUMA. Although this does not affect the execution of the code (because CPU is offline), But I don't think adding a offline CPU to NUMA makes sense.

Quoted 11 lines
Quoted 11 lines
xen/arch/arm/smpboot.c +0 -0
      /* Now report this CPU is up */
      cpumask_set_cpu(cpuid, &cpu_online_map);
 
xen/include/asm-arm/numa.h +0 -0
index 7a3588ac7f..dd31324b0b 100644
@@ -59,6 +59,7 @@ extern mfn_t first_valid_mfn;
  #define __node_distance(a, b) (20)
 
Quoted 6 lines

xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 62

Quoted 2 lines
xen/include/asm-arm/numa.h +1 -0
  #define numa_init(x) do { } while (0)
+#define numa_add_cpu(x) do { } while (0)

This is a stubs for a common helper. So I think this wants to be moved in the !CONFIG_NUMA in xen/numa.h.

OK

Quoted 9 lines

xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 65

Quoted 4 lines
xen/include/asm-arm/numa.h +0 -0
  #define numa_set_node(x, y) do { } while (0)
 
  #endif

Cheers,

--
Julien Grall

#135 lore raw

Re: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA system

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 63 lines
Quoted 52 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 0:58
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA system

Hi Wei,

xen/arch/arm/setup.c void __init start_xen(unsigned long

Quoted 19 lines · Wei Chen

When cpu boot up, we have add them to NUMA system. In current stage, we have not parsed the NUMA data, but we have created a fake NUMA node. So, in this patch, all CPU will be added to NUMA node#0. After the NUMA data has been parsed from device tree, the CPU will be added to correct NUMA node as the NUMA data described.

---
   xen/arch/arm/setup.c       | 6 ++++++
   xen/arch/arm/smpboot.c     | 6 ++++++
   xen/include/asm-arm/numa.h | 1 +
   3 files changed, 13 insertions(+)
xen/arch/arm/setup.c +0 -0
index 3c58d2d441..7531989f21 100644
@@ -918,6 +918,12 @@ void __init start_xen(unsigned long

boot_phys_offset,

Quoted 20 lines
xen/arch/arm/setup.c +6 -0
       processor_id();
 
+    /*
+     * If Xen is running on a NUMA off system, there will
+     * be a node#0 at least.
+     */
+    numa_add_cpu(0);
+
       smp_init_cpus();
       cpus = smp_get_max_cpus();
       printk(XENLOG_INFO "SMP: Allowing %u CPUs\n", cpus);
xen/arch/arm/smpboot.c +0 -0
index a1ee3146ef..aa78958c07 100644
@@ -358,6 +358,12 @@ void start_secondary(void)
        */
       smp_wmb();
 
Quoted 10 lines

xen/arch/arm/smpboot.c void start_secondary(void) · line 366

Quoted 6 lines
xen/arch/arm/smpboot.c +6 -0
+    /*
+     * If Xen is running on a NUMA off system, there will
+     * be a node#0 at least.
+     */
+    numa_add_cpu(cpuid);
+

On x86, numa_add_cpu() will be called before the pCPU is brought up. I am not quite too sure why we are doing it differently here. Can you clarify it?

Quoted 4 lines

Of course we can invoke numa_add_cpu before cpu_up as x86. But in my tests, I found when cpu bring up failed, this cpu still be add to NUMA. Although this does not affect the execution of the code (because CPU is offline), But I don't think adding a offline CPU to NUMA makes sense.

Right, but again, why do you want to solve the problem on Arm and not x86? After all, NUMA is not architecture specific (in fact you move most of the code in common).

In fact, the risk, is someone may read arch/x86 and doesn't realize the CPU is not in the node until late on Arm.

So I think we should call numa_add_cpu() around the same place on all the architectures.

If you think the current position on x86 is not correct, then it should be changed at as well. However, I don't know the story behind the position of the call on x86. You may want to ask the x86 maintainers.

Cheers,

Signature · 1 line
Julien Grall
#136 lore raw

Re: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA system

Quoted 31 lines · Julien Grall
Quoted 21 lines · Wei Chen
Quoted 10 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 0:58

Quoted 6 lines · Wei Chen
xen/arch/arm/smpboot.c +0 -0
@@ -358,6 +358,12 @@ void start_secondary(void)
        */
       smp_wmb();
 
Quoted 10 lines

xen/arch/arm/smpboot.c void start_secondary(void) · line 366

Quoted 6 lines
xen/arch/arm/smpboot.c +6 -0
+    /*
+     * If Xen is running on a NUMA off system, there will
+     * be a node#0 at least.
+     */
+    numa_add_cpu(cpuid);
+

On x86, numa_add_cpu() will be called before the pCPU is brought up. I am not quite too sure why we are doing it differently here. Can you clarify it?

Quoted 4 lines

Of course we can invoke numa_add_cpu before cpu_up as x86. But in my tests, I found when cpu bring up failed, this cpu still be add to NUMA. Although this does not affect the execution of the code (because CPU is offline), But I don't think adding a offline CPU to NUMA makes sense.

Right, but again, why do you want to solve the problem on Arm and not x86? After all, NUMA is not architecture specific (in fact you move most of the code in common).

Quoted 5 lines

In fact, the risk, is someone may read arch/x86 and doesn't realize the CPU is not in the node until late on Arm.

So I think we should call numa_add_cpu() around the same place on all the architectures.

FWIW: +1

Jan

author
#137 lore raw

RE: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA system

Hi Jan, Julien,

Quoted 32 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月26日 17:40
To: Julien Grall <[email protected]>; Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA system

Quoted 22 lines · Julien Grall
Quoted 21 lines · Wei Chen
Quoted 20 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 0:58

xen/arch/arm/smpboot.c void start_secondary(void) · line 366

Quoted 12 lines · Wei Chen
xen/arch/arm/smpboot.c +6 -0
@@ -358,6 +358,12 @@ void start_secondary(void)
        */
       smp_wmb();
 
+    /*
+     * If Xen is running on a NUMA off system, there will
+     * be a node#0 at least.
+     */
+    numa_add_cpu(cpuid);
+

On x86, numa_add_cpu() will be called before the pCPU is brought up. I am not quite too sure why we are doing it differently here. Can you clarify it?

Quoted 11 lines
Quoted 1 line
Quoted 1 line

Of course we can invoke numa_add_cpu before cpu_up as x86. But in my

tests,

Quoted 1 line
Quoted 1 line

I found when cpu bring up failed, this cpu still be add to NUMA.

Although

Quoted 7 lines
Quoted 2 lines

this does not affect the execution of the code (because CPU is offline), But I don't think adding a offline CPU to NUMA makes sense.

Right, but again, why do you want to solve the problem on Arm and not x86? After all, NUMA is not architecture specific (in fact you move most of the code in common).

I am not very familiar with x86, so when I was composing this patch series, I always thought that if I could solve it inside Arm Arch, I would solve it inside Arm Arch. That seems a bit conservative, and inappropriate on solving this problem.

Quoted 7 lines
Quoted 5 lines

In fact, the risk, is someone may read arch/x86 and doesn't realize the CPU is not in the node until late on Arm.

So I think we should call numa_add_cpu() around the same place on all the architectures.

FWIW: +1

I agree. As Jan in this discussion. How about following current x86's numa_add_cpu behaviors in __start_xen, but add some code to revert numa_add_cpu when cpu_up failed (both Arm and x86)?

Quoted 2 lines

Jan

#138 lore raw

Re: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA system

Quoted 17 lines · Wei Chen
Quoted 8 lines

From: Jan Beulich <[email protected]>
Sent: 2021年8月26日 17:40

Quoted 4 lines · Julien Grall

Right, but again, why do you want to solve the problem on Arm and not x86? After all, NUMA is not architecture specific (in fact you move most of the code in common).

I am not very familiar with x86, so when I was composing this patch series, I always thought that if I could solve it inside Arm Arch, I would solve it inside Arm Arch. That seems a bit conservative, and inappropriate on solving this problem.

Quoted 3 lines
Quoted 3 lines

In fact, the risk, is someone may read arch/x86 and doesn't realize the CPU is not in the node until late on Arm.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

So I think we should call numa_add_cpu() around the same place on all the architectures.

FWIW: +1

I agree. As Jan in this discussion. How about following current x86's numa_add_cpu behaviors in __start_xen, but add some code to revert numa_add_cpu when cpu_up failed (both Arm and x86)?

Sure - if we don't clean up properly on x86 on a failure path, I'm all for having that fixed.

Jan

author
#139 lore raw

RFC PATCH 27/40 xen/arm: build CPU NUMA node map while creating cpu_logical_map

In reply to Wei Chen

Sometimes, CPU logical ID maybe different with physical CPU ID. Xen is using CPU logial ID for runtime usage, so we should use CPU logical ID to create map between NUMA node and CPU.

---
 xen/arch/arm/smpboot.c | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)
xen/arch/arm/smpboot.c +30 -1
index aa78958c07..dd5a45bffc 100644
@@ -121,7 +121,12 @@ static void __init dt_smp_init_cpus(void)
     {
         [0 ... NR_CPUS - 1] = MPIDR_INVALID
     };
+    static nodeid_t node_map[NR_CPUS] __initdata =
+    {
+        [0 ... NR_CPUS - 1] = NUMA_NO_NODE
+    };
     bool bootcpu_valid = false;
+    uint32_t nid = 0;
     int rc;
 
     mpidr = boot_cpu_data.mpidr.bits & MPIDR_HWID_MASK;
@@ -172,6 +177,26 @@ static void __init dt_smp_init_cpus(void)
             continue;
         }
 
+#ifdef CONFIG_DEVICE_TREE_NUMA
+        /*
+         *  When CONFIG_DEVICE_TREE_NUMA is set, try to fetch numa infomation
+         * from CPU dts node, otherwise the nid is always 0.
+         */
+        if ( !dt_property_read_u32(cpu, "numa-node-id", &nid) )
+        {
+            printk(XENLOG_WARNING
+                "cpu[%d] dts path: %s: doesn't have numa infomation!\n",
+                cpuidx, dt_node_full_name(cpu));
+            /*
+             * The the early stage of NUMA initialization, when Xen found any
+             * CPU dts node doesn't have numa-node-id info, the NUMA will be
+             * treated as off, all CPU will be set to a FAKE node 0. So if we
+             * get numa-node-id failed here, we should set nid to 0.
+             */
+            nid = 0;
+        }
+#endif
+
         /*
          * 8 MSBs must be set to 0 in the DT since the reg property
          * defines the MPIDR[23:0]
@@ -231,9 +256,12 @@ static void __init dt_smp_init_cpus(void)
         {
             printk("cpu%d init failed (hwid %"PRIregister"): %d\n", i, hwid, rc);
             tmp_map[i] = MPIDR_INVALID;
+            node_map[i] = NUMA_NO_NODE;
         }
-        else
+        else {
             tmp_map[i] = hwid;
+            node_map[i] = nid;
+        }
     }
 
     if ( !bootcpu_valid )
@@ -249,6 +277,7 @@ static void __init dt_smp_init_cpus(void)
             continue;
         cpumask_set_cpu(i, &cpu_possible_map);
         cpu_logical_map(i) = tmp_map[i];
+        numa_set_node(i, node_map[i]);
     }
 }
 
Signature · 1 line
2.25.1
#140 lore raw

Re: [XEN RFC PATCH 27/40] xen/arm: build CPU NUMA node map while creating cpu_logical_map

Hi Wei,

Quoted 3 lines · Wei Chen

Sometimes, CPU logical ID maybe different with physical CPU ID. Xen is using CPU logial ID for runtime usage, so we should use CPU logical ID to create map between NUMA node and CPU.

This commit message gives the impression that you are trying to fix a bug. However, what you are explaining is the reason why the code will use the logical ID rather than physical ID.

I think the commit message should explain what the patch is doing. You can then add an explanation why you are using the CPU logical ID. Something like "Note we storing the CPU logical ID because...".

Quoted 24 lines
---
  xen/arch/arm/smpboot.c | 31 ++++++++++++++++++++++++++++++-
  1 file changed, 30 insertions(+), 1 deletion(-)
xen/arch/arm/smpboot.c +5 -0
index aa78958c07..dd5a45bffc 100644
@@ -121,7 +121,12 @@ static void __init dt_smp_init_cpus(void)
      {
          [0 ... NR_CPUS - 1] = MPIDR_INVALID
      };
+    static nodeid_t node_map[NR_CPUS] __initdata =
+    {
+        [0 ... NR_CPUS - 1] = NUMA_NO_NODE
+    };
      bool bootcpu_valid = false;
+    uint32_t nid = 0;
      int rc;
 
      mpidr = boot_cpu_data.mpidr.bits & MPIDR_HWID_MASK;

xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 185

Quoted 9 lines
xen/arch/arm/smpboot.c +6 -0
@@ -172,6 +177,26 @@ static void __init dt_smp_init_cpus(void)
              continue;
          }
 
+#ifdef CONFIG_DEVICE_TREE_NUMA
+        /*
+         *  When CONFIG_DEVICE_TREE_NUMA is set, try to fetch numa infomation
+         * from CPU dts node, otherwise the nid is always 0.
+         */
+        if ( !dt_property_read_u32(cpu, "numa-node-id", &nid) )

You can avoid the #ifdef by writing:

if ( IS_ENABLED(CONFIG_DEVICE_TREE_NUMA) && ... )

However, I would using CONFIG_NUMA because this code is already DT specific. So we can shorten the name a bit.

xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 188

Quoted 3 lines
xen/arch/arm/smpboot.c +3 -0
+        {
+            printk(XENLOG_WARNING
+                "cpu[%d] dts path: %s: doesn't have numa infomation!\n",

s/information/information/

xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 191

Quoted 3 lines
xen/arch/arm/smpboot.c +3 -0
+                cpuidx, dt_node_full_name(cpu));
+            /*
+             * The the early stage of NUMA initialization, when Xen found any

s/The/During/?

Quoted 26 lines
xen/arch/arm/smpboot.c +12 -1
+             * CPU dts node doesn't have numa-node-id info, the NUMA will be
+             * treated as off, all CPU will be set to a FAKE node 0. So if we
+             * get numa-node-id failed here, we should set nid to 0.
+             */
+            nid = 0;
+        }
+#endif
+
          /*
           * 8 MSBs must be set to 0 in the DT since the reg property
           * defines the MPIDR[23:0]
@@ -231,9 +256,12 @@ static void __init dt_smp_init_cpus(void)
          {
              printk("cpu%d init failed (hwid %"PRIregister"): %d\n", i, hwid, rc);
              tmp_map[i] = MPIDR_INVALID;
+            node_map[i] = NUMA_NO_NODE;
          }
-        else
+        else {
              tmp_map[i] = hwid;
+            node_map[i] = nid;
+        }
      }
 
      if ( !bootcpu_valid )

xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 283

Quoted 7 lines
xen/arch/arm/smpboot.c +1 -0
@@ -249,6 +277,7 @@ static void __init dt_smp_init_cpus(void)
              continue;
          cpumask_set_cpu(i, &cpu_possible_map);
          cpu_logical_map(i) = tmp_map[i];
+        numa_set_node(i, node_map[i]);
      }
  }
   >

Cheers,

Signature · 1 line
Julien Grall
author
#141 lore raw

RE: [XEN RFC PATCH 27/40] xen/arm: build CPU NUMA node map while creating cpu_logical_map

Hi Julien,

Quoted 17 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 1:07
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [XEN RFC PATCH 27/40] xen/arm: build CPU NUMA node map while creating cpu_logical_map

Hi Wei,

Quoted 3 lines · Wei Chen

Sometimes, CPU logical ID maybe different with physical CPU ID. Xen is using CPU logial ID for runtime usage, so we should use CPU logical ID to create map between NUMA node and CPU.

Quoted 9 lines

This commit message gives the impression that you are trying to fix a bug. However, what you are explaining is the reason why the code will use the logical ID rather than physical ID.

I think the commit message should explain what the patch is doing. You can then add an explanation why you are using the CPU logical ID. Something like "Note we storing the CPU logical ID because...".

Ok

Quoted 35 lines
Quoted 24 lines
---
  xen/arch/arm/smpboot.c | 31 ++++++++++++++++++++++++++++++-
  1 file changed, 30 insertions(+), 1 deletion(-)
xen/arch/arm/smpboot.c +5 -0
index aa78958c07..dd5a45bffc 100644
@@ -121,7 +121,12 @@ static void __init dt_smp_init_cpus(void)
      {
          [0 ... NR_CPUS - 1] = MPIDR_INVALID
      };
+    static nodeid_t node_map[NR_CPUS] __initdata =
+    {
+        [0 ... NR_CPUS - 1] = NUMA_NO_NODE
+    };
      bool bootcpu_valid = false;
+    uint32_t nid = 0;
      int rc;
 
      mpidr = boot_cpu_data.mpidr.bits & MPIDR_HWID_MASK;

xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 182

Quoted 6 lines
xen/arch/arm/smpboot.c +3 -0
@@ -172,6 +177,26 @@ static void __init dt_smp_init_cpus(void)
              continue;
          }
 
+#ifdef CONFIG_DEVICE_TREE_NUMA
+        /*
+         *  When CONFIG_DEVICE_TREE_NUMA is set, try to fetch numa

infomation

xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 185

Quoted 3 lines
xen/arch/arm/smpboot.c +3 -0
+         * from CPU dts node, otherwise the nid is always 0.
+         */
+        if ( !dt_property_read_u32(cpu, "numa-node-id", &nid) )
Quoted 7 lines

You can avoid the #ifdef by writing:

if ( IS_ENABLED(CONFIG_DEVICE_TREE_NUMA) && ... )

However, I would using CONFIG_NUMA because this code is already DT specific. So we can shorten the name a bit.

OK

Quoted 5 lines

xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 188

Quoted 3 lines
xen/arch/arm/smpboot.c +3 -0
+        {
+            printk(XENLOG_WARNING
+                "cpu[%d] dts path: %s: doesn't have numa infomation!\n",

s/information/information/

OK

Quoted 7 lines

xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void)

Quoted 4 lines
xen/arch/arm/smpboot.c +3 -0
+                cpuidx, dt_node_full_name(cpu));
+            /*
+             * The the early stage of NUMA initialization, when Xen

found any

s/The/During/?

Oh, yes, I will fix it.

Quoted 42 lines

xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 192

Quoted 2 lines
xen/arch/arm/smpboot.c +1 -0
+             * CPU dts node doesn't have numa-node-id info, the NUMA

will be

xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 193

Quoted 1 line
xen/arch/arm/smpboot.c +1 -0
+             * treated as off, all CPU will be set to a FAKE node 0. So

if we

xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 257

Quoted 12 lines
xen/arch/arm/smpboot.c +6 -0
+             * get numa-node-id failed here, we should set nid to 0.
+             */
+            nid = 0;
+        }
+#endif
+
          /*
           * 8 MSBs must be set to 0 in the DT since the reg property
           * defines the MPIDR[23:0]
@@ -231,9 +256,12 @@ static void __init dt_smp_init_cpus(void)
          {
              printk("cpu%d init failed (hwid %"PRIregister"): %d\n", i,

hwid, rc);

Quoted 12 lines
xen/arch/arm/smpboot.c +4 -1
              tmp_map[i] = MPIDR_INVALID;
+            node_map[i] = NUMA_NO_NODE;
          }
-        else
+        else {
              tmp_map[i] = hwid;
+            node_map[i] = nid;
+        }
      }
 
      if ( !bootcpu_valid )

xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 283

Quoted 7 lines
xen/arch/arm/smpboot.c +1 -0
@@ -249,6 +277,7 @@ static void __init dt_smp_init_cpus(void)
              continue;
          cpumask_set_cpu(i, &cpu_possible_map);
          cpu_logical_map(i) = tmp_map[i];
+        numa_set_node(i, node_map[i]);
      }
  }
   >

Cheers,

--
Julien Grall

author
#142 lore raw

RFC PATCH 28/40 xen/x86: decouple nodes_cover_memory with E820 map

In reply to Wei Chen

We will reuse nodes_cover_memory for Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_number arch_get_memory_bank_range

Depends above two helpers, we make nodes_cover_memory become architecture independent.

---
 xen/arch/x86/numa.c    | 18 ++++++++++++++++++
 xen/arch/x86/srat.c    |  8 +++-----
 xen/include/xen/numa.h |  4 ++++
 3 files changed, 25 insertions(+), 5 deletions(-)
xen/arch/x86/numa.c +18 -0
index 6908738305..8b43be4aa7 100644
@@ -128,6 +128,24 @@ unsigned int __init arch_get_dma_bitsize(void)
                  + PAGE_SHIFT, 32);
 }
 
+uint32_t __init arch_meminfo_get_nr_bank(void)
+{
+	return e820.nr_map;
+}
+
+int __init arch_meminfo_get_ram_bank_range(int bank,
+	unsigned long long *start, unsigned long long *end)
+{
+	if (e820.map[bank].type != E820_RAM || !start || !end) {
+		return -1;
+	}
+
+	*start = e820.map[bank].addr;
+	*end = e820.map[bank].addr + e820.map[bank].size;
+
+	return 0;
+}
+
 static void dump_numa(unsigned char key)
 {
     s_time_t now = NOW();
xen/arch/x86/srat.c +3 -5
index 6d68b8a614..2298353846 100644
@@ -316,18 +316,16 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 static int __init nodes_cover_memory(void)
 {
 	int i;
+	uint32_t nr_banks = arch_meminfo_get_nr_bank();
 
-	for (i = 0; i < e820.nr_map; i++) {
+	for (i = 0; i < nr_banks; i++) {
 		int j, found;
 		unsigned long long start, end;
 
-		if (e820.map[i].type != E820_RAM) {
+		if (arch_meminfo_get_ram_bank_range(i, &start, &end)) {
 			continue;
 		}
 
-		start = e820.map[i].addr;
-		end = e820.map[i].addr + e820.map[i].size;
-
 		do {
 			found = 0;
 			for_each_node_mask(j, memory_nodes_parsed)
xen/include/xen/numa.h +4 -0
index 0475823b13..6d18059bcd 100644
@@ -89,6 +89,10 @@ static inline void clear_node_cpumask(int cpu)
 	cpumask_clear_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
 }
 
+extern uint32_t arch_meminfo_get_nr_bank(void);
+extern int arch_meminfo_get_ram_bank_range(int bank,
+    unsigned long long *start, unsigned long long *end);
+
 #endif /* CONFIG_NUMA */
 
 #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
#143 lore raw

Re: [XEN RFC PATCH 28/40] xen/x86: decouple nodes_cover_memory with E820 map

Quoted 8 lines · Wei Chen

We will reuse nodes_cover_memory for Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_number arch_get_memory_bank_range

Depends above two helpers, we make nodes_cover_memory become architecture independent.

You might want to note that the only change from an x86 perspective is the additional checks:

  !start || !end
Quoted 80 lines
---
 xen/arch/x86/numa.c    | 18 ++++++++++++++++++
 xen/arch/x86/srat.c    |  8 +++-----
 xen/include/xen/numa.h |  4 ++++
 3 files changed, 25 insertions(+), 5 deletions(-)
xen/arch/x86/numa.c +18 -0
index 6908738305..8b43be4aa7 100644
@@ -128,6 +128,24 @@ unsigned int __init arch_get_dma_bitsize(void)
                  + PAGE_SHIFT, 32);
 }
 
+uint32_t __init arch_meminfo_get_nr_bank(void)
+{
+	return e820.nr_map;
+}
+
+int __init arch_meminfo_get_ram_bank_range(int bank,
+	unsigned long long *start, unsigned long long *end)
+{
+	if (e820.map[bank].type != E820_RAM || !start || !end) {
+		return -1;
+	}
+
+	*start = e820.map[bank].addr;
+	*end = e820.map[bank].addr + e820.map[bank].size;
+
+	return 0;
+}
+
 static void dump_numa(unsigned char key)
 {
     s_time_t now = NOW();
xen/arch/x86/srat.c +3 -5
index 6d68b8a614..2298353846 100644
@@ -316,18 +316,16 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 static int __init nodes_cover_memory(void)
 {
 	int i;
+	uint32_t nr_banks = arch_meminfo_get_nr_bank();
 
-	for (i = 0; i < e820.nr_map; i++) {
+	for (i = 0; i < nr_banks; i++) {
 		int j, found;
 		unsigned long long start, end;
 
-		if (e820.map[i].type != E820_RAM) {
+		if (arch_meminfo_get_ram_bank_range(i, &start, &end)) {
 			continue;
 		}
 
-		start = e820.map[i].addr;
-		end = e820.map[i].addr + e820.map[i].size;
-
 		do {
 			found = 0;
 			for_each_node_mask(j, memory_nodes_parsed)
xen/include/xen/numa.h +4 -0
index 0475823b13..6d18059bcd 100644
@@ -89,6 +89,10 @@ static inline void clear_node_cpumask(int cpu)
 	cpumask_clear_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
 }
 
+extern uint32_t arch_meminfo_get_nr_bank(void);
+extern int arch_meminfo_get_ram_bank_range(int bank,
+    unsigned long long *start, unsigned long long *end);
+
 #endif /* CONFIG_NUMA */
 
 #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
author
#144 lore raw

RE: [XEN RFC PATCH 28/40] xen/x86: decouple nodes_cover_memory with E820 map

Hi Stefano,

Quoted 16 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 9:08
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 28/40] xen/x86: decouple nodes_cover_memory with E820 map

Quoted 6 lines · Wei Chen

We will reuse nodes_cover_memory for Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_number arch_get_memory_bank_range

Quoted 8 lines
Quoted 2 lines

Depends above two helpers, we make nodes_cover_memory become architecture independent.

You might want to note that the only change from an x86 perspective is the additional checks:

  !start || !end

Thanks, I will add it.

Quoted 82 lines

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct

Quoted 42 lines
---
 xen/arch/x86/numa.c    | 18 ++++++++++++++++++
 xen/arch/x86/srat.c    |  8 +++-----
 xen/include/xen/numa.h |  4 ++++
 3 files changed, 25 insertions(+), 5 deletions(-)
xen/arch/x86/numa.c +18 -0
index 6908738305..8b43be4aa7 100644
@@ -128,6 +128,24 @@ unsigned int __init arch_get_dma_bitsize(void)
                  + PAGE_SHIFT, 32);
 }
 
+uint32_t __init arch_meminfo_get_nr_bank(void)
+{
+	return e820.nr_map;
+}
+
+int __init arch_meminfo_get_ram_bank_range(int bank,
+	unsigned long long *start, unsigned long long *end)
+{
+	if (e820.map[bank].type != E820_RAM || !start || !end) {
+		return -1;
+	}
+
+	*start = e820.map[bank].addr;
+	*end = e820.map[bank].addr + e820.map[bank].size;
+
+	return 0;
+}
+
 static void dump_numa(unsigned char key)
 {
     s_time_t now = NOW();
xen/arch/x86/srat.c +0 -0
index 6d68b8a614..2298353846 100644
@@ -316,18 +316,16 @@ acpi_numa_memory_affinity_init(const struct

acpi_srat_mem_affinity *ma)

Quoted 39 lines
xen/arch/x86/srat.c +3 -5
 static int __init nodes_cover_memory(void)
 {
 	int i;
+	uint32_t nr_banks = arch_meminfo_get_nr_bank();
 
-	for (i = 0; i < e820.nr_map; i++) {
+	for (i = 0; i < nr_banks; i++) {
 		int j, found;
 		unsigned long long start, end;
 
-		if (e820.map[i].type != E820_RAM) {
+		if (arch_meminfo_get_ram_bank_range(i, &start, &end)) {
 			continue;
 		}
 
-		start = e820.map[i].addr;
-		end = e820.map[i].addr + e820.map[i].size;
-
 		do {
 			found = 0;
 			for_each_node_mask(j, memory_nodes_parsed)
xen/include/xen/numa.h +4 -1
index 0475823b13..6d18059bcd 100644
@@ -89,6 +89,10 @@ static inline void clear_node_cpumask(int cpu)
 	cpumask_clear_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
 }
 
+extern uint32_t arch_meminfo_get_nr_bank(void);
+extern int arch_meminfo_get_ram_bank_range(int bank,
+    unsigned long long *start, unsigned long long *end);
+
 #endif /* CONFIG_NUMA */
 
 #endif /* _XEN_NUMA_H */
--

2.25.1

author
#145 lore raw

RFC PATCH 29/40 xen/arm: implement Arm arch helpers Arm to get memory map info

In reply to Wei Chen

These two helpers are architecture APIs that are required by nodes_cover_memory.

---
 xen/arch/arm/numa.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
xen/arch/arm/numa.c +14 -0
index f61a8df645..6eebf8e8bc 100644
@@ -126,3 +126,17 @@ void __init numa_init(bool acpi_off)
     numa_initmem_init(PFN_UP(ram_start), PFN_DOWN(ram_end));
     return;
 }
+
+uint32_t __init arch_meminfo_get_nr_bank(void)
+{
+	return bootinfo.mem.nr_banks;
+}
+
+int __init arch_meminfo_get_ram_bank_range(int bank,
+	unsigned long long *start, unsigned long long *end)
+{
+	*start = bootinfo.mem.bank[bank].start;
+	*end = bootinfo.mem.bank[bank].start + bootinfo.mem.bank[bank].size;
+
+	return 0;
+}
Signature · 1 line
2.25.1
#146 lore raw

Re: [XEN RFC PATCH 29/40] xen/arm: implement Arm arch helpers Arm to get memory map info

Hi Wei,

Quoted 14 lines · Wei Chen

These two helpers are architecture APIs that are required by nodes_cover_memory.

---
  xen/arch/arm/numa.c | 14 ++++++++++++++
  1 file changed, 14 insertions(+)
xen/arch/arm/numa.c +0 -0
index f61a8df645..6eebf8e8bc 100644
@@ -126,3 +126,17 @@ void __init numa_init(bool acpi_off)
      numa_initmem_init(PFN_UP(ram_start), PFN_DOWN(ram_end));

xen/arch/arm/numa.c void __init numa_init(bool acpi_off) · line 136

Quoted 10 lines
xen/arch/arm/numa.c +8 -0
      return;
  }
+
+uint32_t __init arch_meminfo_get_nr_bank(void)
+{
+	return bootinfo.mem.nr_banks;
+}
+
+int __init arch_meminfo_get_ram_bank_range(int bank,
+	unsigned long long *start, unsigned long long *end)

They are physical address, so we should use "paddr_t" as on system such as 32-bit Arm, "unsigned long" is not enough to cover all the physical address.

As you change the type, I would also suggest to change the bank from an int to an unsigned int.

xen/arch/arm/numa.c void __init numa_init(bool acpi_off) · line 142

Quoted 7 lines
xen/arch/arm/numa.c +6 -0
+{
+	*start = bootinfo.mem.bank[bank].start;
+	*end = bootinfo.mem.bank[bank].start + bootinfo.mem.bank[bank].size;
+
+	return 0;
+}

Cheers,

Signature · 1 line
Julien Grall
author
#147 lore raw

RE: [XEN RFC PATCH 29/40] xen/arm: implement Arm arch helpers Arm to get memory map info

Hi Julien,

Quoted 41 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 1:10
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 29/40] xen/arm: implement Arm arch helpers Arm to get memory map info

Hi Wei,

Quoted 14 lines · Wei Chen

These two helpers are architecture APIs that are required by nodes_cover_memory.

---
  xen/arch/arm/numa.c | 14 ++++++++++++++
  1 file changed, 14 insertions(+)
xen/arch/arm/numa.c +0 -0
index f61a8df645..6eebf8e8bc 100644
@@ -126,3 +126,17 @@ void __init numa_init(bool acpi_off)
      numa_initmem_init(PFN_UP(ram_start), PFN_DOWN(ram_end));

xen/arch/arm/numa.c void __init numa_init(bool acpi_off) · line 136

Quoted 10 lines
xen/arch/arm/numa.c +8 -0
      return;
  }
+
+uint32_t __init arch_meminfo_get_nr_bank(void)
+{
+	return bootinfo.mem.nr_banks;
+}
+
+int __init arch_meminfo_get_ram_bank_range(int bank,
+	unsigned long long *start, unsigned long long *end)

They are physical address, so we should use "paddr_t" as on system such as 32-bit Arm, "unsigned long" is not enough to cover all the physical address.

Quoted 3 lines

As you change the type, I would also suggest to change the bank from an int to an unsigned int.

I will fix them in next version.

Quoted 12 lines

xen/arch/arm/numa.c void __init numa_init(bool acpi_off) · line 142

Quoted 7 lines
xen/arch/arm/numa.c +6 -0
+{
+	*start = bootinfo.mem.bank[bank].start;
+	*end = bootinfo.mem.bank[bank].start + bootinfo.mem.bank[bank].size;
+
+	return 0;
+}

Cheers,

--
Julien Grall

author
#148 lore raw

RFC PATCH 30/40 xen: move NUMA memory and CPU parsed nodemasks to common

In reply to Wei Chen

Both memory_nodes_parsed and processor_nodes_parsed are using for Arm and x86 to record parded NUMA memory and CPU. So we move them to common.

---
 xen/arch/arm/numa_device_tree.c | 2 --
 xen/arch/x86/srat.c             | 3 ---
 xen/common/numa.c               | 3 +++
 xen/include/xen/nodemask.h      | 2 ++
 4 files changed, 5 insertions(+), 5 deletions(-)
xen/arch/arm/numa_device_tree.c +0 -2
index 27ffb72f7b..f74b7f6427 100644
@@ -25,8 +25,6 @@
 #include <asm/setup.h>
 
 s8 device_tree_numa = 0;
-static nodemask_t processor_nodes_parsed __initdata;
-static nodemask_t memory_nodes_parsed __initdata;
 
 static int srat_disabled(void)
 {
xen/arch/x86/srat.c +0 -3
index 2298353846..dd3aa30843 100644
@@ -24,9 +24,6 @@
 
 static struct acpi_table_slit *__read_mostly acpi_slit;
 
-static nodemask_t memory_nodes_parsed __initdata;
-static nodemask_t processor_nodes_parsed __initdata;
-
 struct pxm2node {
 	unsigned pxm;
 	nodeid_t node;
xen/common/numa.c +3 -0
index 26c0006d04..79ab250543 100644
@@ -35,6 +35,9 @@ int num_node_memblks;
 struct node node_memblk_range[NR_NODE_MEMBLKS];
 nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
 
+nodemask_t memory_nodes_parsed __initdata;
+nodemask_t processor_nodes_parsed __initdata;
+
 bool numa_off;
 
 /*
xen/include/xen/nodemask.h +2 -0
index 1dd6c7458e..29ce5e28e7 100644
@@ -276,6 +276,8 @@ static inline int __cycle_node(int n, const nodemask_t *maskp, int nbits)
  */
 
 extern nodemask_t node_online_map;
+extern nodemask_t memory_nodes_parsed;
+extern nodemask_t processor_nodes_parsed;
 
 #if MAX_NUMNODES > 1
 #define num_online_nodes()	nodes_weight(node_online_map)
Signature · 1 line
2.25.1
#149 lore raw

Re: [XEN RFC PATCH 30/40] xen: move NUMA memory and CPU parsed nodemasks to common

Hi Wei,

Quoted 3 lines · Wei Chen

Both memory_nodes_parsed and processor_nodes_parsed are using for Arm and x86 to record parded NUMA memory and CPU. So we move them to common.

Looking at the usage, they both call:

numa_set...(..., bitmap)

So rather than exporting the two helpers, could we simply add helpers to abstract it?

Quoted 14 lines
---
  xen/arch/arm/numa_device_tree.c | 2 --
  xen/arch/x86/srat.c             | 3 ---
  xen/common/numa.c               | 3 +++
  xen/include/xen/nodemask.h      | 2 ++
  4 files changed, 5 insertions(+), 5 deletions(-)
xen/arch/arm/numa_device_tree.c +0 -0
index 27ffb72f7b..f74b7f6427 100644

xen/arch/arm/numa_device_tree.c line 29

Quoted 5 lines
xen/arch/arm/numa_device_tree.c +0 -2
@@ -25,8 +25,6 @@
  #include <asm/setup.h>
 
  s8 device_tree_numa = 0;
-static nodemask_t processor_nodes_parsed __initdata;
-static nodemask_t memory_nodes_parsed __initdata;

This is code that was introduced in a previous patch. In general, it is better to do the rework first and then add the new code. This makes easier to follow series as the code added is not changed.

Quoted 37 lines
xen/arch/arm/numa_device_tree.c +0 -0
  static int srat_disabled(void)
  {
xen/arch/x86/srat.c +0 -3
index 2298353846..dd3aa30843 100644
@@ -24,9 +24,6 @@
 
  static struct acpi_table_slit *__read_mostly acpi_slit;
 
-static nodemask_t memory_nodes_parsed __initdata;
-static nodemask_t processor_nodes_parsed __initdata;
-
  struct pxm2node {
  	unsigned pxm;
  	nodeid_t node;
xen/common/numa.c +3 -0
index 26c0006d04..79ab250543 100644
@@ -35,6 +35,9 @@ int num_node_memblks;
  struct node node_memblk_range[NR_NODE_MEMBLKS];
  nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
 
+nodemask_t memory_nodes_parsed __initdata;
+nodemask_t processor_nodes_parsed __initdata;
+
  bool numa_off;
 
  /*
xen/include/xen/nodemask.h +0 -0
index 1dd6c7458e..29ce5e28e7 100644

xen/include/xen/nodemask.h static inline int __cycle_node(int n, const nodemask_t *maskp, int nbits) · line 283

Quoted 8 lines
xen/include/xen/nodemask.h +2 -0
@@ -276,6 +276,8 @@ static inline int __cycle_node(int n, const nodemask_t *maskp, int nbits)
   */
 
  extern nodemask_t node_online_map;
+extern nodemask_t memory_nodes_parsed;
+extern nodemask_t processor_nodes_parsed;
 
  #if MAX_NUMNODES > 1
  #define num_online_nodes()	nodes_weight(node_online_map)

Cheers,

Signature · 1 line
Julien Grall
author
#150 lore raw

RE: [XEN RFC PATCH 30/40] xen: move NUMA memory and CPU parsed nodemasks to common

Hi Juilien,

Quoted 16 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 1:17
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 30/40] xen: move NUMA memory and CPU parsed nodemasks to common

Hi Wei,

Quoted 3 lines · Wei Chen

Both memory_nodes_parsed and processor_nodes_parsed are using for Arm and x86 to record parded NUMA memory and CPU. So we move them to common.

Quoted 7 lines

Looking at the usage, they both call:

numa_set...(..., bitmap)

So rather than exporting the two helpers, could we simply add helpers to abstract it?

I will try to fix it in next version.

Quoted 18 lines

Review

Quoted 11 lines
---
  xen/arch/arm/numa_device_tree.c | 2 --
  xen/arch/x86/srat.c             | 3 ---
  xen/common/numa.c               | 3 +++
  xen/include/xen/nodemask.h      | 2 ++
  4 files changed, 5 insertions(+), 5 deletions(-)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

Quoted 6 lines

index 27ffb72f7b..f74b7f6427 100644

xen/arch/arm/numa_device_tree.c +0 -0
@@ -25,8 +25,6 @@
  #include <asm/setup.h>
 
Quoted 8 lines

xen/arch/arm/numa_device_tree.c line 29

Quoted 3 lines
xen/arch/arm/numa_device_tree.c +0 -2
  s8 device_tree_numa = 0;
-static nodemask_t processor_nodes_parsed __initdata;
-static nodemask_t memory_nodes_parsed __initdata;

This is code that was introduced in a previous patch. In general, it is better to do the rework first and then add the new code. This makes easier to follow series as the code added is not changed.

Yes, I will fix it in next version.

Quoted 51 lines

xen/include/xen/nodemask.h static inline int __cycle_node(int n, const

Quoted 36 lines
xen/arch/arm/numa_device_tree.c +0 -0
  static int srat_disabled(void)
  {
xen/arch/x86/srat.c +0 -3
index 2298353846..dd3aa30843 100644
@@ -24,9 +24,6 @@
 
  static struct acpi_table_slit *__read_mostly acpi_slit;
 
-static nodemask_t memory_nodes_parsed __initdata;
-static nodemask_t processor_nodes_parsed __initdata;
-
  struct pxm2node {
  	unsigned pxm;
  	nodeid_t node;
xen/common/numa.c +3 -0
index 26c0006d04..79ab250543 100644
@@ -35,6 +35,9 @@ int num_node_memblks;
  struct node node_memblk_range[NR_NODE_MEMBLKS];
  nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
 
+nodemask_t memory_nodes_parsed __initdata;
+nodemask_t processor_nodes_parsed __initdata;
+
  bool numa_off;
 
  /*
xen/include/xen/nodemask.h +0 -0
index 1dd6c7458e..29ce5e28e7 100644
@@ -276,6 +276,8 @@ static inline int __cycle_node(int n, const

nodemask_t *maskp, int nbits)

xen/include/xen/nodemask.h static inline int __cycle_node(int n, const nodemask_t *maskp, int nbits) · line 283

Quoted 9 lines
xen/include/xen/nodemask.h +2 -0
   */
 
  extern nodemask_t node_online_map;
+extern nodemask_t memory_nodes_parsed;
+extern nodemask_t processor_nodes_parsed;
 
  #if MAX_NUMNODES > 1
  #define num_online_nodes()	nodes_weight(node_online_map)

Cheers,

--
Julien Grall

author
#151 lore raw

RFC PATCH 31/40 xen/x86: move nodes_cover_memory to common

In reply to Wei Chen

Not only ACPU NUMA, but also Arm device tree based NUMA will use nodes_cover_memory to do sanity check. So we move this function from arch/x86 to common.

---
 xen/arch/x86/srat.c    | 40 ----------------------------------------
 xen/common/numa.c      | 40 ++++++++++++++++++++++++++++++++++++++++
 xen/include/xen/numa.h |  1 +
 3 files changed, 41 insertions(+), 40 deletions(-)
xen/arch/x86/srat.c +0 -40
index dd3aa30843..dcebc7adec 100644
@@ -308,46 +308,6 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 	num_node_memblks++;
 }
 
-/* Sanity check to catch more bad SRATs (they are amazingly common).
-   Make sure the PXMs cover all memory. */
-static int __init nodes_cover_memory(void)
-{
-	int i;
-	uint32_t nr_banks = arch_meminfo_get_nr_bank();
-
-	for (i = 0; i < nr_banks; i++) {
-		int j, found;
-		unsigned long long start, end;
-
-		if (arch_meminfo_get_ram_bank_range(i, &start, &end)) {
-			continue;
-		}
-
-		do {
-			found = 0;
-			for_each_node_mask(j, memory_nodes_parsed)
-				if (start < nodes[j].end
-				    && end > nodes[j].start) {
-					if (start >= nodes[j].start) {
-						start = nodes[j].end;
-						found = 1;
-					}
-					if (end <= nodes[j].end) {
-						end = nodes[j].start;
-						found = 1;
-					}
-				}
-		} while (found && start < end);
-
-		if (start < end) {
-			printk(KERN_ERR "SRAT: No PXM for e820 range: "
-				"%016Lx - %016Lx\n", start, end);
-			return 0;
-		}
-	}
-	return 1;
-}
-
 void __init acpi_numa_arch_fixup(void) {}
 
 static uint64_t __initdata srat_region_mask;
xen/common/numa.c +40 -0
index 79ab250543..74960885a6 100644
@@ -193,6 +193,46 @@ void __init cutoff_node(int i, u64 start, u64 end)
 	}
 }
 
+/* Sanity check to catch more bad SRATs (they are amazingly common).
+   Make sure the PXMs cover all memory. */
+int __init nodes_cover_memory(void)
+{
+	int i;
+	uint32_t nr_banks = arch_meminfo_get_nr_bank();
+
+	for (i = 0; i < nr_banks; i++) {
+		int j, found;
+		unsigned long long start, end;
+
+		if (arch_meminfo_get_ram_bank_range(i, &start, &end)) {
+			continue;
+		}
+
+		do {
+			found = 0;
+			for_each_node_mask(j, memory_nodes_parsed)
+				if (start < nodes[j].end
+				    && end > nodes[j].start) {
+					if (start >= nodes[j].start) {
+						start = nodes[j].end;
+						found = 1;
+					}
+					if (end <= nodes[j].end) {
+						end = nodes[j].start;
+						found = 1;
+					}
+				}
+		} while (found && start < end);
+
+		if (start < end) {
+			printk(KERN_ERR "SRAT: No PXM for e820 range: "
+				"%016Lx - %016Lx\n", start, end);
+			return 0;
+		}
+	}
+	return 1;
+}
+
 void numa_add_cpu(int cpu)
 {
     cpumask_set_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
xen/include/xen/numa.h +1 -0
index 6d18059bcd..094ab904c9 100644
@@ -92,6 +92,7 @@ static inline void clear_node_cpumask(int cpu)
 extern uint32_t arch_meminfo_get_nr_bank(void);
 extern int arch_meminfo_get_ram_bank_range(int bank,
     unsigned long long *start, unsigned long long *end);
+extern int nodes_cover_memory(void);
 
 #endif /* CONFIG_NUMA */
 
Signature · 1 line
2.25.1
#152 lore raw

Re: [XEN RFC PATCH 31/40] xen/x86: move nodes_cover_memory to common

Quoted 94 lines · Wei Chen

Not only ACPU NUMA, but also Arm device tree based NUMA will use nodes_cover_memory to do sanity check. So we move this function from arch/x86 to common.

---
 xen/arch/x86/srat.c    | 40 ----------------------------------------
 xen/common/numa.c      | 40 ++++++++++++++++++++++++++++++++++++++++
 xen/include/xen/numa.h |  1 +
 3 files changed, 41 insertions(+), 40 deletions(-)
xen/arch/x86/srat.c +0 -40
index dd3aa30843..dcebc7adec 100644
@@ -308,46 +308,6 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 	num_node_memblks++;
 }
 
-/* Sanity check to catch more bad SRATs (they are amazingly common).
-   Make sure the PXMs cover all memory. */
-static int __init nodes_cover_memory(void)
-{
-	int i;
-	uint32_t nr_banks = arch_meminfo_get_nr_bank();
-
-	for (i = 0; i < nr_banks; i++) {
-		int j, found;
-		unsigned long long start, end;
-
-		if (arch_meminfo_get_ram_bank_range(i, &start, &end)) {
-			continue;
-		}
-
-		do {
-			found = 0;
-			for_each_node_mask(j, memory_nodes_parsed)
-				if (start < nodes[j].end
-				    && end > nodes[j].start) {
-					if (start >= nodes[j].start) {
-						start = nodes[j].end;
-						found = 1;
-					}
-					if (end <= nodes[j].end) {
-						end = nodes[j].start;
-						found = 1;
-					}
-				}
-		} while (found && start < end);
-
-		if (start < end) {
-			printk(KERN_ERR "SRAT: No PXM for e820 range: "
-				"%016Lx - %016Lx\n", start, end);
-			return 0;
-		}
-	}
-	return 1;
-}
-
 void __init acpi_numa_arch_fixup(void) {}
 
 static uint64_t __initdata srat_region_mask;
xen/common/numa.c +24 -0
index 79ab250543..74960885a6 100644
@@ -193,6 +193,46 @@ void __init cutoff_node(int i, u64 start, u64 end)
 	}
 }
 
+/* Sanity check to catch more bad SRATs (they are amazingly common).
+   Make sure the PXMs cover all memory. */
+int __init nodes_cover_memory(void)
+{
+	int i;
+	uint32_t nr_banks = arch_meminfo_get_nr_bank();
+
+	for (i = 0; i < nr_banks; i++) {
+		int j, found;
+		unsigned long long start, end;
+
+		if (arch_meminfo_get_ram_bank_range(i, &start, &end)) {
+			continue;
+		}
+
+		do {
+			found = 0;
+			for_each_node_mask(j, memory_nodes_parsed)
+				if (start < nodes[j].end
+				    && end > nodes[j].start) {
+					if (start >= nodes[j].start) {
+						start = nodes[j].end;
+						found = 1;
+					}

xen/common/numa.c void __init cutoff_node(int i, u64 start, u64 end) · line 229

Quoted 10 lines
xen/common/numa.c +10 -0
+					if (end <= nodes[j].end) {
+						end = nodes[j].start;
+						found = 1;
+					}
+				}
+		} while (found && start < end);
+
+		if (start < end) {
+			printk(KERN_ERR "SRAT: No PXM for e820 range: "
+				"%016Lx - %016Lx\n", start, end);

I don't know if you are already doing this in a later patch but the message shouldn't say e820 as it doesn't exist on all architecture. Maybe "for address range" or "for memory range" would suffice.

Normally we don't do change together with code movement but in this case I think it would be OK.

author
#153 lore raw

RE: [XEN RFC PATCH 31/40] xen/x86: move nodes_cover_memory to common

Hi Stefano,

Quoted 120 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 9:17
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 31/40] xen/x86: move nodes_cover_memory to common

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct

Quoted 16 lines · Wei Chen

Not only ACPU NUMA, but also Arm device tree based NUMA will use nodes_cover_memory to do sanity check. So we move this function from arch/x86 to common.

---
 xen/arch/x86/srat.c    | 40 ----------------------------------------
 xen/common/numa.c      | 40 ++++++++++++++++++++++++++++++++++++++++
 xen/include/xen/numa.h |  1 +
 3 files changed, 41 insertions(+), 40 deletions(-)
xen/arch/x86/srat.c +0 -0
index dd3aa30843..dcebc7adec 100644
@@ -308,46 +308,6 @@ acpi_numa_memory_affinity_init(const struct

acpi_srat_mem_affinity *ma)

Quoted 78 lines
xen/arch/x86/srat.c +0 -40
 	num_node_memblks++;
 }
 
-/* Sanity check to catch more bad SRATs (they are amazingly common).
-   Make sure the PXMs cover all memory. */
-static int __init nodes_cover_memory(void)
-{
-	int i;
-	uint32_t nr_banks = arch_meminfo_get_nr_bank();
-
-	for (i = 0; i < nr_banks; i++) {
-		int j, found;
-		unsigned long long start, end;
-
-		if (arch_meminfo_get_ram_bank_range(i, &start, &end)) {
-			continue;
-		}
-
-		do {
-			found = 0;
-			for_each_node_mask(j, memory_nodes_parsed)
-				if (start < nodes[j].end
-				    && end > nodes[j].start) {
-					if (start >= nodes[j].start) {
-						start = nodes[j].end;
-						found = 1;
-					}
-					if (end <= nodes[j].end) {
-						end = nodes[j].start;
-						found = 1;
-					}
-				}
-		} while (found && start < end);
-
-		if (start < end) {
-			printk(KERN_ERR "SRAT: No PXM for e820 range: "
-				"%016Lx - %016Lx\n", start, end);
-			return 0;
-		}
-	}
-	return 1;
-}
-
 void __init acpi_numa_arch_fixup(void) {}
 
 static uint64_t __initdata srat_region_mask;
xen/common/numa.c +24 -0
index 79ab250543..74960885a6 100644
@@ -193,6 +193,46 @@ void __init cutoff_node(int i, u64 start, u64 end)
 	}
 }
 
+/* Sanity check to catch more bad SRATs (they are amazingly common).
+   Make sure the PXMs cover all memory. */
+int __init nodes_cover_memory(void)
+{
+	int i;
+	uint32_t nr_banks = arch_meminfo_get_nr_bank();
+
+	for (i = 0; i < nr_banks; i++) {
+		int j, found;
+		unsigned long long start, end;
+
+		if (arch_meminfo_get_ram_bank_range(i, &start, &end)) {
+			continue;
+		}
+
+		do {
+			found = 0;
+			for_each_node_mask(j, memory_nodes_parsed)
+				if (start < nodes[j].end
+				    && end > nodes[j].start) {
+					if (start >= nodes[j].start) {
+						start = nodes[j].end;
+						found = 1;
+					}

xen/common/numa.c void __init cutoff_node(int i, u64 start, u64 end) · line 229

Quoted 10 lines
xen/common/numa.c +10 -0
+					if (end <= nodes[j].end) {
+						end = nodes[j].start;
+						found = 1;
+					}
+				}
+		} while (found && start < end);
+
+		if (start < end) {
+			printk(KERN_ERR "SRAT: No PXM for e820 range: "
+				"%016Lx - %016Lx\n", start, end);

I don't know if you are already doing this in a later patch but the message shouldn't say e820 as it doesn't exist on all architecture. Maybe "for address range" or "for memory range" would suffice.

Quoted 2 lines

Normally we don't do change together with code movement but in this case I think it would be OK.

OK, I will do it in next version.

author
#154 lore raw

RFC PATCH 32/40 xen/x86: make acpi_scan_nodes to be neutral

In reply to Wei Chen

The code in acpi_scan_nodes can be reused for device tree based NUMA. So we rename acpi_scan_nodes to numa_scan_nodes for a neutral function name. As acpi_numa variable is available in ACPU based NUMA system only, we use CONFIG_ACPI_NUMA to protect it.

---
 xen/arch/x86/srat.c        | 4 +++-
 xen/common/numa.c          | 2 +-
 xen/include/asm-x86/acpi.h | 2 +-
 3 files changed, 5 insertions(+), 3 deletions(-)
xen/arch/x86/srat.c +3 -1
index dcebc7adec..3d4d90a622 100644
@@ -362,7 +362,7 @@ void __init srat_parse_regions(u64 addr)
 }
 
 /* Use the information discovered above to actually set up the nodes. */
-int __init acpi_scan_nodes(u64 start, u64 end)
+int __init numa_scan_nodes(u64 start, u64 end)
 {
 	int i;
 	nodemask_t all_nodes_parsed;
@@ -371,8 +371,10 @@ int __init acpi_scan_nodes(u64 start, u64 end)
 	for (i = 0; i < MAX_NUMNODES; i++)
 		cutoff_node(i, start, end);
 
+#ifdef CONFIG_ACPI_NUMA
 	if (acpi_numa <= 0)
 		return -1;
+#endif
 
 	if (!nodes_cover_memory()) {
 		bad_srat();
xen/common/numa.c +1 -1
index 74960885a6..4152bbe83b 100644
@@ -330,7 +330,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
 #endif
 
 #ifdef CONFIG_ACPI_NUMA
-    if ( !numa_off && !acpi_scan_nodes((u64)start_pfn << PAGE_SHIFT,
+    if ( !numa_off && !numa_scan_nodes((u64)start_pfn << PAGE_SHIFT,
          (u64)end_pfn << PAGE_SHIFT) )
         return;
 #endif
xen/include/asm-x86/acpi.h +1 -1
index d347500a3c..33b71dfb3b 100644
@@ -102,7 +102,7 @@ extern unsigned long acpi_wakeup_address;
 #define ARCH_HAS_POWER_INIT	1
 
 extern s8 acpi_numa;
-extern int acpi_scan_nodes(u64 start, u64 end);
+extern int numa_scan_nodes(u64 start, u64 end);
 
 extern struct acpi_sleep_info acpi_sinfo;
 #define acpi_video_flags bootsym(video_flags)
Signature · 1 line
2.25.1
#155 lore raw

Re: [XEN RFC PATCH 32/40] xen/x86: make acpi_scan_nodes to be neutral

Hi Wei,

Quoted 26 lines · Wei Chen

The code in acpi_scan_nodes can be reused for device tree based NUMA. So we rename acpi_scan_nodes to numa_scan_nodes for a neutral function name. As acpi_numa variable is available in ACPU based NUMA system only, we use CONFIG_ACPI_NUMA to protect it.

---
  xen/arch/x86/srat.c        | 4 +++-
  xen/common/numa.c          | 2 +-
  xen/include/asm-x86/acpi.h | 2 +-
  3 files changed, 5 insertions(+), 3 deletions(-)
xen/arch/x86/srat.c +1 -1
index dcebc7adec..3d4d90a622 100644
@@ -362,7 +362,7 @@ void __init srat_parse_regions(u64 addr)
  }
 
  /* Use the information discovered above to actually set up the nodes. */
-int __init acpi_scan_nodes(u64 start, u64 end)
+int __init numa_scan_nodes(u64 start, u64 end)
  {
  	int i;
  	nodemask_t all_nodes_parsed;

xen/arch/x86/srat.c int __init acpi_scan_nodes(u64 start, u64 end) · line 377

Quoted 7 lines
xen/arch/x86/srat.c +2 -0
@@ -371,8 +371,10 @@ int __init acpi_scan_nodes(u64 start, u64 end)
  	for (i = 0; i < MAX_NUMNODES; i++)
  		cutoff_node(i, start, end);
 
+#ifdef CONFIG_ACPI_NUMA
  	if (acpi_numa <= 0)
  		return -1;
+#endif

Looking at the follow-up patches, I find a bit odd that there is a check for ACPI but there is none added for DT. Can you explain why?

However, I think this check is going to impair the work to support both ACPI and DT on Arm because acpi_numa would end up to be 0 so you would bail out here.

With that in mind, I think this check needs to either go away or replace by something there is firmware agnostic.

Cheers,

Signature · 1 line
Julien Grall
author
#156 lore raw

RE: [XEN RFC PATCH 32/40] xen/x86: make acpi_scan_nodes to be neutral

Hi Julien,

Quoted 42 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:09
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 32/40] xen/x86: make acpi_scan_nodes to be neutral

Hi Wei,

Quoted 17 lines · Wei Chen

The code in acpi_scan_nodes can be reused for device tree based NUMA. So we rename acpi_scan_nodes to numa_scan_nodes for a neutral function name. As acpi_numa variable is available in ACPU based NUMA system only, we use CONFIG_ACPI_NUMA to protect it.

---
  xen/arch/x86/srat.c        | 4 +++-
  xen/common/numa.c          | 2 +-
  xen/include/asm-x86/acpi.h | 2 +-
  3 files changed, 5 insertions(+), 3 deletions(-)
xen/arch/x86/srat.c +0 -0
index dcebc7adec..3d4d90a622 100644

xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr) · line 364

Quoted 3 lines
xen/arch/x86/srat.c +0 -0
@@ -362,7 +362,7 @@ void __init srat_parse_regions(u64 addr)
  }
 
  /* Use the information discovered above to actually set up the nodes.

*/

Quoted 9 lines
xen/arch/x86/srat.c +1 -1
-int __init acpi_scan_nodes(u64 start, u64 end)
+int __init numa_scan_nodes(u64 start, u64 end)
  {
  	int i;
  	nodemask_t all_nodes_parsed;
@@ -371,8 +371,10 @@ int __init acpi_scan_nodes(u64 start, u64 end)
  	for (i = 0; i < MAX_NUMNODES; i++)
  		cutoff_node(i, start, end);
 
Quoted 8 lines

xen/arch/x86/srat.c int __init acpi_scan_nodes(u64 start, u64 end) · line 377

Quoted 4 lines
xen/arch/x86/srat.c +2 -0
+#ifdef CONFIG_ACPI_NUMA
  	if (acpi_numa <= 0)
  		return -1;
+#endif

Looking at the follow-up patches, I find a bit odd that there is a check for ACPI but there is none added for DT. Can you explain why?

Oh, I forgot DT check. And simply to add DT check here seems not a good idea. Because once, when Arm support ACPI NUMA. CONFIG_ACPI_NUMA and CONFIG_DEVICE_TREE_NUMA can be selected at the same time. But only acpi_numa or dtb_numa can be > 0.

Quoted 6 lines

However, I think this check is going to impair the work to support both ACPI and DT on Arm because acpi_numa would end up to be 0 so you would bail out here.

With that in mind, I think this check needs to either go away or replace by something there is firmware agnostic.

Yes, we have discussed about something like fw_numa before.

Quoted 5 lines

Cheers,

--
Julien Grall

author
#157 lore raw

RFC PATCH 33/40 xen: export bad_srat and srat_disabled to extern

In reply to Wei Chen

These two functions are architecture implementation. But common code out of arch will invoke them, so we export them to extern.

---
 xen/arch/arm/numa_device_tree.c | 4 ++--
 xen/arch/x86/srat.c             | 2 +-
 xen/include/asm-x86/numa.h      | 1 -
 xen/include/xen/numa.h          | 2 ++
 4 files changed, 5 insertions(+), 4 deletions(-)
xen/arch/arm/numa_device_tree.c +2 -2
index f74b7f6427..f7f2eeebc3 100644
@@ -26,12 +26,12 @@
 
 s8 device_tree_numa = 0;
 
-static int srat_disabled(void)
+int srat_disabled(void)
 {
     return numa_off || device_tree_numa < 0;
 }
 
-static __init void bad_srat(void)
+__init void bad_srat(void)
 {
     printk(KERN_ERR "DT: NUMA information is not used.\n");
     device_tree_numa = -1;
xen/arch/x86/srat.c +1 -1
index 3d4d90a622..c979939fdd 100644
@@ -97,7 +97,7 @@ nodeid_t setup_node(unsigned pxm)
 	return node;
 }
 
-static __init void bad_srat(void)
+__init void bad_srat(void)
 {
 	int i;
 	printk(KERN_ERR "SRAT: SRAT not used.\n");
xen/include/asm-x86/numa.h +0 -1
index e0253c20b7..e63869135c 100644
@@ -13,7 +13,6 @@ extern nodeid_t pxm_to_node(unsigned int pxm);
 
 #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
 
-extern int srat_disabled(void);
 extern nodeid_t setup_node(unsigned int pxm);
 extern void srat_detect_node(int cpu);
 
xen/include/xen/numa.h +2 -0
index 094ab904c9..490381bd13 100644
@@ -75,6 +75,8 @@ extern int compute_hash_shift(struct node *nodes, int numnodes,
 extern int conflicting_memblks(u64 start, u64 end);
 extern void cutoff_node(int i, u64 start, u64 end);
 extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
+extern int srat_disabled(void);
+extern void bad_srat(void);
 
 extern void numa_init_array(void);
 extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
Signature · 1 line
2.25.1
author
#158 lore raw

RFC PATCH 34/40 xen: move numa_scan_nodes from x86 to common

In reply to Wei Chen

After the previous patches preparations, numa_scan_nodes can be used by Arm and x86. So we move this function from x86 to common. As node_cover_memory will not be used cross files, we restore its static attribute in this patch.

---
 xen/arch/x86/srat.c        | 52 ------------------------------------
 xen/common/numa.c          | 54 +++++++++++++++++++++++++++++++++++++-
 xen/include/asm-x86/acpi.h |  3 ---
 xen/include/xen/numa.h     |  3 ++-
 4 files changed, 55 insertions(+), 57 deletions(-)
xen/arch/x86/srat.c +0 -52
index c979939fdd..c9f019c307 100644
@@ -361,58 +361,6 @@ void __init srat_parse_regions(u64 addr)
 	pfn_pdx_hole_setup(mask >> PAGE_SHIFT);
 }
 
-/* Use the information discovered above to actually set up the nodes. */
-int __init numa_scan_nodes(u64 start, u64 end)
-{
-	int i;
-	nodemask_t all_nodes_parsed;
-
-	/* First clean up the node list */
-	for (i = 0; i < MAX_NUMNODES; i++)
-		cutoff_node(i, start, end);
-
-#ifdef CONFIG_ACPI_NUMA
-	if (acpi_numa <= 0)
-		return -1;
-#endif
-
-	if (!nodes_cover_memory()) {
-		bad_srat();
-		return -1;
-	}
-
-	memnode_shift = compute_hash_shift(node_memblk_range, num_node_memblks,
-				memblk_nodeid);
-
-	if (memnode_shift < 0) {
-		printk(KERN_ERR
-		     "SRAT: No NUMA node hash function found. Contact maintainer\n");
-		bad_srat();
-		return -1;
-	}
-
-	nodes_or(all_nodes_parsed, memory_nodes_parsed, processor_nodes_parsed);
-
-	/* Finally register nodes */
-	for_each_node_mask(i, all_nodes_parsed)
-	{
-		u64 size = nodes[i].end - nodes[i].start;
-		if ( size == 0 )
-			printk(KERN_WARNING "SRAT: Node %u has no memory. "
-			       "BIOS Bug or mis-configured hardware?\n", i);
-
-		setup_node_bootmem(i, nodes[i].start, nodes[i].end);
-	}
-	for (i = 0; i < nr_cpu_ids; i++) {
-		if (cpu_to_node[i] == NUMA_NO_NODE)
-			continue;
-		if (!nodemask_test(cpu_to_node[i], &processor_nodes_parsed))
-			numa_set_node(i, NUMA_NO_NODE);
-	}
-	numa_init_array();
-	return 0;
-}
-
 static unsigned node_to_pxm(nodeid_t n)
 {
 	unsigned i;
xen/common/numa.c +53 -1

66 lines not on this page · raw message

xen/include/asm-x86/acpi.h +0 -3
index 33b71dfb3b..2140461ff3 100644
@@ -101,9 +101,6 @@ extern unsigned long acpi_wakeup_address;
 
 #define ARCH_HAS_POWER_INIT	1
 
-extern s8 acpi_numa;
-extern int numa_scan_nodes(u64 start, u64 end);
-
 extern struct acpi_sleep_info acpi_sinfo;
 #define acpi_video_flags bootsym(video_flags)
 struct xenpf_enter_acpi_sleep;
xen/include/xen/numa.h +2 -1
index 490381bd13..b9b5d1ad88 100644
@@ -81,8 +81,10 @@ extern void bad_srat(void);
 extern void numa_init_array(void);
 extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
 extern void numa_set_node(int cpu, nodeid_t node);
+extern int numa_scan_nodes(u64 start, u64 end);
 extern bool numa_off;
 extern int numa_fake;
+extern s8 acpi_numa;
 
 extern void setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end);
 
@@ -94,7 +96,6 @@ static inline void clear_node_cpumask(int cpu)
 extern uint32_t arch_meminfo_get_nr_bank(void);
 extern int arch_meminfo_get_ram_bank_range(int bank,
     unsigned long long *start, unsigned long long *end);
-extern int nodes_cover_memory(void);
 
 #endif /* CONFIG_NUMA */
 
Signature · 1 line
2.25.1
#159 lore raw

Re: [XEN RFC PATCH 34/40] xen: move numa_scan_nodes from x86 to common

Hi Wei,

Quoted 19 lines · Wei Chen
xen/include/asm-x86/acpi.h +0 -3
index 33b71dfb3b..2140461ff3 100644
@@ -101,9 +101,6 @@ extern unsigned long acpi_wakeup_address;
 
  #define ARCH_HAS_POWER_INIT	1
 
-extern s8 acpi_numa;
-extern int numa_scan_nodes(u64 start, u64 end);
-
  extern struct acpi_sleep_info acpi_sinfo;
  #define acpi_video_flags bootsym(video_flags)
  struct xenpf_enter_acpi_sleep;
xen/include/xen/numa.h +0 -0
index 490381bd13..b9b5d1ad88 100644

xen/include/xen/numa.h extern void bad_srat(void); · line 84

Quoted 4 lines
xen/include/xen/numa.h +1 -0
@@ -81,8 +81,10 @@ extern void bad_srat(void);
  extern void numa_init_array(void);
  extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
  extern void numa_set_node(int cpu, nodeid_t node);
+extern int numa_scan_nodes(u64 start, u64 end);

AFAICT, by the end of the series, the function is only called by the common code. So this should be static.

Cheers,

Signature · 1 line
Julien Grall
author
#160 lore raw

RE: [XEN RFC PATCH 34/40] xen: move numa_scan_nodes from x86 to common

Hi Julien,

Quoted 34 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:14
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 34/40] xen: move numa_scan_nodes from x86 to common

Hi Wei,

Quoted 19 lines · Wei Chen
xen/include/asm-x86/acpi.h +0 -3
index 33b71dfb3b..2140461ff3 100644
@@ -101,9 +101,6 @@ extern unsigned long acpi_wakeup_address;
 
  #define ARCH_HAS_POWER_INIT	1
 
-extern s8 acpi_numa;
-extern int numa_scan_nodes(u64 start, u64 end);
-
  extern struct acpi_sleep_info acpi_sinfo;
  #define acpi_video_flags bootsym(video_flags)
  struct xenpf_enter_acpi_sleep;
xen/include/xen/numa.h +0 -0
index 490381bd13..b9b5d1ad88 100644

xen/include/xen/numa.h extern void bad_srat(void); · line 82

Quoted 2 lines
xen/include/xen/numa.h +0 -0
@@ -81,8 +81,10 @@ extern void bad_srat(void);
  extern void numa_init_array(void);
  extern void numa_initmem_init(unsigned long start_pfn, unsigned long

end_pfn);

Quoted 6 lines

xen/include/xen/numa.h extern void bad_srat(void); · line 84

Quoted 2 lines
xen/include/xen/numa.h +1 -0
  extern void numa_set_node(int cpu, nodeid_t node);
+extern int numa_scan_nodes(u64 start, u64 end);

AFAICT, by the end of the series, the function is only called by the common code. So this should be static.

OK

Quoted 4 lines

Cheers,

--
Julien Grall

#161 lore raw

Re: [XEN RFC PATCH 34/40] xen: move numa_scan_nodes from x86 to common

In reply to Wei Chen
Quoted 122 lines · Wei Chen

After the previous patches preparations, numa_scan_nodes can be used by Arm and x86. So we move this function from x86 to common. As node_cover_memory will not be used cross files, we restore its static attribute in this patch.

---
 xen/arch/x86/srat.c        | 52 ------------------------------------
 xen/common/numa.c          | 54 +++++++++++++++++++++++++++++++++++++-
 xen/include/asm-x86/acpi.h |  3 ---
 xen/include/xen/numa.h     |  3 ++-
 4 files changed, 55 insertions(+), 57 deletions(-)
xen/arch/x86/srat.c +0 -52
index c979939fdd..c9f019c307 100644
@@ -361,58 +361,6 @@ void __init srat_parse_regions(u64 addr)
 	pfn_pdx_hole_setup(mask >> PAGE_SHIFT);
 }
 
-/* Use the information discovered above to actually set up the nodes. */
-int __init numa_scan_nodes(u64 start, u64 end)
-{
-	int i;
-	nodemask_t all_nodes_parsed;
-
-	/* First clean up the node list */
-	for (i = 0; i < MAX_NUMNODES; i++)
-		cutoff_node(i, start, end);
-
-#ifdef CONFIG_ACPI_NUMA
-	if (acpi_numa <= 0)
-		return -1;
-#endif
-
-	if (!nodes_cover_memory()) {
-		bad_srat();
-		return -1;
-	}
-
-	memnode_shift = compute_hash_shift(node_memblk_range, num_node_memblks,
-				memblk_nodeid);
-
-	if (memnode_shift < 0) {
-		printk(KERN_ERR
-		     "SRAT: No NUMA node hash function found. Contact maintainer\n");
-		bad_srat();
-		return -1;
-	}
-
-	nodes_or(all_nodes_parsed, memory_nodes_parsed, processor_nodes_parsed);
-
-	/* Finally register nodes */
-	for_each_node_mask(i, all_nodes_parsed)
-	{
-		u64 size = nodes[i].end - nodes[i].start;
-		if ( size == 0 )
-			printk(KERN_WARNING "SRAT: Node %u has no memory. "
-			       "BIOS Bug or mis-configured hardware?\n", i);
-
-		setup_node_bootmem(i, nodes[i].start, nodes[i].end);
-	}
-	for (i = 0; i < nr_cpu_ids; i++) {
-		if (cpu_to_node[i] == NUMA_NO_NODE)
-			continue;
-		if (!nodemask_test(cpu_to_node[i], &processor_nodes_parsed))
-			numa_set_node(i, NUMA_NO_NODE);
-	}
-	numa_init_array();
-	return 0;
-}
-
 static unsigned node_to_pxm(nodeid_t n)
 {
 	unsigned i;
xen/common/numa.c +30 -1
index 4152bbe83b..8ca13e27d1 100644
@@ -195,7 +195,7 @@ void __init cutoff_node(int i, u64 start, u64 end)
 
 /* Sanity check to catch more bad SRATs (they are amazingly common).
    Make sure the PXMs cover all memory. */
-int __init nodes_cover_memory(void)
+static int __init nodes_cover_memory(void)
 {
 	int i;
 	uint32_t nr_banks = arch_meminfo_get_nr_bank();
@@ -271,6 +271,58 @@ void __init numa_init_array(void)
     }
 }
 
+/* Use the information discovered above to actually set up the nodes. */
+int __init numa_scan_nodes(u64 start, u64 end)
+{
+	int i;
+	nodemask_t all_nodes_parsed;
+
+	/* First clean up the node list */
+	for (i = 0; i < MAX_NUMNODES; i++)
+		cutoff_node(i, start, end);
+
+#ifdef CONFIG_ACPI_NUMA
+	if (acpi_numa <= 0)
+		return -1;
+#endif
+
+	if (!nodes_cover_memory()) {
+		bad_srat();
+		return -1;
+	}
+
+	memnode_shift = compute_hash_shift(node_memblk_range, num_node_memblks,
+				memblk_nodeid);
+
+	if (memnode_shift < 0) {
+		printk(KERN_ERR
+		     "SRAT: No NUMA node hash function found. Contact maintainer\n");
+		bad_srat();
+		return -1;
+	}

xen/common/numa.c void __init numa_init_array(void) · line 312

Quoted 10 lines
xen/common/numa.c +10 -0
+
+	nodes_or(all_nodes_parsed, memory_nodes_parsed, processor_nodes_parsed);
+
+	/* Finally register nodes */
+	for_each_node_mask(i, all_nodes_parsed)
+	{
+		u64 size = nodes[i].end - nodes[i].start;
+		if ( size == 0 )
+			printk(KERN_WARNING "SRAT: Node %u has no memory. "
+			       "BIOS Bug or mis-configured hardware?\n", i);

Not all archs have a BIOS so I'd say "firmware bug". Like last time, we usually don't do code changes together with code movement, but in this case it might be OK. I am also happy with a separate patch to adjust the two comments (this one and the one in the previous patch).

author
#162 lore raw

RE: [XEN RFC PATCH 34/40] xen: move numa_scan_nodes from x86 to common

Hi Stefano,

Quoted 142 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 9:27
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 34/40] xen: move numa_scan_nodes from x86 to common

Quoted 18 lines · Wei Chen

After the previous patches preparations, numa_scan_nodes can be used by Arm and x86. So we move this function from x86 to common. As node_cover_memory will not be used cross files, we restore its static attribute in this patch.

---
 xen/arch/x86/srat.c        | 52 ------------------------------------
 xen/common/numa.c          | 54 +++++++++++++++++++++++++++++++++++++-
 xen/include/asm-x86/acpi.h |  3 ---
 xen/include/xen/numa.h     |  3 ++-
 4 files changed, 55 insertions(+), 57 deletions(-)
xen/arch/x86/srat.c +0 -0
index c979939fdd..c9f019c307 100644

xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr) · line 364

Quoted 4 lines
xen/arch/x86/srat.c +0 -1
@@ -361,58 +361,6 @@ void __init srat_parse_regions(u64 addr)
 	pfn_pdx_hole_setup(mask >> PAGE_SHIFT);
 }
 
-/* Use the information discovered above to actually set up the nodes.

*/

Quoted 10 lines
xen/arch/x86/srat.c +0 -10
-int __init numa_scan_nodes(u64 start, u64 end)
-{
-	int i;
-	nodemask_t all_nodes_parsed;
-
-	/* First clean up the node list */
-	for (i = 0; i < MAX_NUMNODES; i++)
-		cutoff_node(i, start, end);
-
-#ifdef CONFIG_ACPI_NUMA

xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr)

Quoted 10 lines
xen/arch/x86/srat.c +0 -10
-	if (acpi_numa <= 0)
-		return -1;
-#endif
-
-	if (!nodes_cover_memory()) {
-		bad_srat();
-		return -1;
-	}
-
-	memnode_shift = compute_hash_shift(node_memblk_range,

num_node_memblks,

xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr)

Quoted 5 lines
xen/arch/x86/srat.c +0 -5
-				memblk_nodeid);
-
-	if (memnode_shift < 0) {
-		printk(KERN_ERR
-		     "SRAT: No NUMA node hash function found. Contact

maintainer\n");

xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr)

Quoted 5 lines
xen/arch/x86/srat.c +0 -5
-		bad_srat();
-		return -1;
-	}
-
-	nodes_or(all_nodes_parsed, memory_nodes_parsed,

processor_nodes_parsed);

Quoted 38 lines
xen/arch/x86/srat.c +0 -21
-
-	/* Finally register nodes */
-	for_each_node_mask(i, all_nodes_parsed)
-	{
-		u64 size = nodes[i].end - nodes[i].start;
-		if ( size == 0 )
-			printk(KERN_WARNING "SRAT: Node %u has no memory. "
-			       "BIOS Bug or mis-configured hardware?\n", i);
-
-		setup_node_bootmem(i, nodes[i].start, nodes[i].end);
-	}
-	for (i = 0; i < nr_cpu_ids; i++) {
-		if (cpu_to_node[i] == NUMA_NO_NODE)
-			continue;
-		if (!nodemask_test(cpu_to_node[i], &processor_nodes_parsed))
-			numa_set_node(i, NUMA_NO_NODE);
-	}
-	numa_init_array();
-	return 0;
-}
-
 static unsigned node_to_pxm(nodeid_t n)
 {
 	unsigned i;
xen/common/numa.c +1 -1
index 4152bbe83b..8ca13e27d1 100644
@@ -195,7 +195,7 @@ void __init cutoff_node(int i, u64 start, u64 end)
 
 /* Sanity check to catch more bad SRATs (they are amazingly common).
    Make sure the PXMs cover all memory. */
-int __init nodes_cover_memory(void)
+static int __init nodes_cover_memory(void)
 {
 	int i;
 	uint32_t nr_banks = arch_meminfo_get_nr_bank();

xen/common/numa.c void __init numa_init_array(void) · line 274

Quoted 4 lines
xen/common/numa.c +1 -0
@@ -271,6 +271,58 @@ void __init numa_init_array(void)
     }
 }
 
+/* Use the information discovered above to actually set up the nodes.

*/

Quoted 10 lines
xen/common/numa.c +10 -0
+int __init numa_scan_nodes(u64 start, u64 end)
+{
+	int i;
+	nodemask_t all_nodes_parsed;
+
+	/* First clean up the node list */
+	for (i = 0; i < MAX_NUMNODES; i++)
+		cutoff_node(i, start, end);
+
+#ifdef CONFIG_ACPI_NUMA

xen/common/numa.c void __init numa_init_array(void)

Quoted 10 lines
xen/common/numa.c +10 -0
+	if (acpi_numa <= 0)
+		return -1;
+#endif
+
+	if (!nodes_cover_memory()) {
+		bad_srat();
+		return -1;
+	}
+
+	memnode_shift = compute_hash_shift(node_memblk_range,

num_node_memblks,

xen/common/numa.c void __init numa_init_array(void)

Quoted 5 lines
xen/common/numa.c +5 -0
+				memblk_nodeid);
+
+	if (memnode_shift < 0) {
+		printk(KERN_ERR
+		     "SRAT: No NUMA node hash function found. Contact

maintainer\n");

xen/common/numa.c void __init numa_init_array(void)

Quoted 5 lines
xen/common/numa.c +5 -0
+		bad_srat();
+		return -1;
+	}
+
+	nodes_or(all_nodes_parsed, memory_nodes_parsed,

processor_nodes_parsed);

Quoted 13 lines

xen/common/numa.c void __init numa_init_array(void) · line 312

Quoted 8 lines
xen/common/numa.c +8 -0
+
+	/* Finally register nodes */
+	for_each_node_mask(i, all_nodes_parsed)
+	{
+		u64 size = nodes[i].end - nodes[i].start;
+		if ( size == 0 )
+			printk(KERN_WARNING "SRAT: Node %u has no memory. "
+			       "BIOS Bug or mis-configured hardware?\n", i);

Not all archs have a BIOS so I'd say "firmware bug". Like last time, we usually don't do code changes together with code movement, but in this case it might be OK. I am also happy with a separate patch to adjust the two comments (this one and the one in the previous patch).

OK, I will do it.

author
#163 lore raw

RFC PATCH 35/40 xen: enable numa_scan_nodes for device tree based NUMA

In reply to Wei Chen

Now, we can use the same function for ACPI and device tree based NUMA to scan memory nodes.

---
 xen/common/numa.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
xen/common/numa.c +1 -1
index 8ca13e27d1..d15c2fc311 100644
@@ -381,7 +381,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
         return;
 #endif
 
-#ifdef CONFIG_ACPI_NUMA
+#if defined(CONFIG_ACPI_NUMA) || defined(CONFIG_DEVICE_TREE_NUMA)
     if ( !numa_off && !numa_scan_nodes((u64)start_pfn << PAGE_SHIFT,
          (u64)end_pfn << PAGE_SHIFT) )
         return;
Signature · 1 line
2.25.1
#164 lore raw

Re: [XEN RFC PATCH 35/40] xen: enable numa_scan_nodes for device tree based NUMA

Hi Wei,

Quoted 13 lines · Wei Chen

Now, we can use the same function for ACPI and device tree based NUMA to scan memory nodes.

---
  xen/common/numa.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
xen/common/numa.c +0 -0
index 8ca13e27d1..d15c2fc311 100644

xen/common/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 384

Quoted 5 lines
xen/common/numa.c +1 -1
@@ -381,7 +381,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
          return;
  #endif
 
-#ifdef CONFIG_ACPI_NUMA
+#if defined(CONFIG_ACPI_NUMA) || defined(CONFIG_DEVICE_TREE_NUMA)

numa.c is only built when CONFIG_NUMA is set. I don't think CONFIG_NUMA will ever set if neither CONFIG_ACPI_NUMA or CONFIG_DEVICE_TREE_NUMA is set. So do we actually need this #ifdef?

xen/common/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 387

Quoted 4 lines
xen/common/numa.c +0 -0
      if ( !numa_off && !numa_scan_nodes((u64)start_pfn << PAGE_SHIFT,
           (u64)end_pfn << PAGE_SHIFT) )
          return;

Cheers,

Signature · 1 line
Julien Grall
author
#165 lore raw

RE: [XEN RFC PATCH 35/40] xen: enable numa_scan_nodes for device tree based NUMA

Hi Julien,

Quoted 29 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:19
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 35/40] xen: enable numa_scan_nodes for device tree based NUMA

Hi Wei,

xen/common/numa.c void __init numa_initmem_init(unsigned long

Quoted 13 lines · Wei Chen

Now, we can use the same function for ACPI and device tree based NUMA to scan memory nodes.

---
  xen/common/numa.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
xen/common/numa.c +0 -0
index 8ca13e27d1..d15c2fc311 100644
@@ -381,7 +381,7 @@ void __init numa_initmem_init(unsigned long

start_pfn, unsigned long end_pfn)

Quoted 3 lines
xen/common/numa.c +0 -0
          return;
  #endif
 
Quoted 7 lines

xen/common/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 384

Quoted 2 lines
xen/common/numa.c +1 -1
-#ifdef CONFIG_ACPI_NUMA
+#if defined(CONFIG_ACPI_NUMA) || defined(CONFIG_DEVICE_TREE_NUMA)

numa.c is only built when CONFIG_NUMA is set. I don't think CONFIG_NUMA will ever set if neither CONFIG_ACPI_NUMA or CONFIG_DEVICE_TREE_NUMA is set. So do we actually need this #ifdef?

Yes, you're right. This check should be removed.

Quoted 9 lines

xen/common/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 387

Quoted 4 lines
xen/common/numa.c +0 -0
      if ( !numa_off && !numa_scan_nodes((u64)start_pfn << PAGE_SHIFT,
           (u64)end_pfn << PAGE_SHIFT) )
          return;

Cheers,

--
Julien Grall

author
#166 lore raw

RFC PATCH 36/40 xen/arm: keep guest still be NUMA unware

In reply to Wei Chen

We have not wanted to make Xen guest be NUMA aware in this patch series. So in this patch, Xen will skip NUMA distance matrix node and skip the numa-node-id property in CPU node and memory node, when Xen is creating guest device tree binary.

---
 xen/arch/arm/domain_build.c | 6 ++++++
 1 file changed, 6 insertions(+)
xen/arch/arm/domain_build.c +6 -0
index cf341f349f..e62fa761bd 100644
@@ -584,6 +584,10 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo,
                 continue;
         }
 
+        /* Guest is numa unaware in current stage */
+        if ( dt_property_name_is_equal(prop, "numa-node-id") )
+            continue;
+
         res = fdt_property(kinfo->fdt, prop->name, prop_data, prop_len);
 
         if ( res )
@@ -1454,6 +1458,8 @@ static int __init handle_node(struct domain *d, struct kernel_info *kinfo,
         DT_MATCH_TYPE("memory"),
         /* The memory mapped timer is not supported by Xen. */
         DT_MATCH_COMPATIBLE("arm,armv7-timer-mem"),
+        /* Numa info doesn't need to be exposed to Domain-0 */
+        DT_MATCH_COMPATIBLE("numa-distance-map-v1"),
         { /* sentinel */ },
     };
     static const struct dt_device_match timer_matches[] __initconst =
Signature · 1 line
2.25.1
#167 lore raw

Re: [XEN RFC PATCH 36/40] xen/arm: keep guest still be NUMA unware

Hi Wei,

Quoted 2 lines · Wei Chen

We have not wanted to make Xen guest be NUMA aware in this patch series.

The concept of patch series ceases to exist once we merge the code. So about how:

"The NUMA information provided in the host Device-Tree are only for Xen. For dom0, we want to hide them as they may be different (for now, dom0 is still not aware of NUMA".

Quoted 3 lines

So in this patch, Xen will skip NUMA distance matrix node and skip the numa-node-id property in CPU node and memory node, when Xen is creating guest device tree binary.

The CPU and memory nodes are recreated from scratch for the domain. So we already skip the property numa-node-id. However...

Quoted 10 lines
---
  xen/arch/arm/domain_build.c | 6 ++++++
  1 file changed, 6 insertions(+)
xen/arch/arm/domain_build.c +0 -0
index cf341f349f..e62fa761bd 100644

xen/arch/arm/domain_build.c static int __init write_properties(struct domain *d, struct kernel_info *kinfo, · line 589

Quoted 6 lines
xen/arch/arm/domain_build.c +3 -0
@@ -584,6 +584,10 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo,
                  continue;
          }
 
+        /* Guest is numa unaware in current stage */
+        if ( dt_property_name_is_equal(prop, "numa-node-id") )
+            continue;

... your code is doing more than skipping the property for the two nodes you mentionned. Can the property exists in other nodes?

Quoted 6 lines
xen/arch/arm/domain_build.c +1 -0
+
          res = fdt_property(kinfo->fdt, prop->name, prop_data, prop_len);
 
          if ( res )

xen/arch/arm/domain_build.c static int __init handle_node(struct domain *d, struct kernel_info *kinfo, · line 1465

Quoted 8 lines
xen/arch/arm/domain_build.c +2 -0
@@ -1454,6 +1458,8 @@ static int __init handle_node(struct domain *d, struct kernel_info *kinfo,
          DT_MATCH_TYPE("memory"),
          /* The memory mapped timer is not supported by Xen. */
          DT_MATCH_COMPATIBLE("arm,armv7-timer-mem"),
+        /* Numa info doesn't need to be exposed to Domain-0 */
+        DT_MATCH_COMPATIBLE("numa-distance-map-v1"),
          { /* sentinel */ },
      };
      static const struct dt_device_match timer_matches[] __initconst =

Cheers,

Signature · 1 line
Julien Grall
author
#168 lore raw

RE: [XEN RFC PATCH 36/40] xen/arm: keep guest still be NUMA unware

Hi Julien,

Quoted 15 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:28
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 36/40] xen/arm: keep guest still be NUMA unware

Hi Wei,

Quoted 2 lines · Wei Chen

We have not wanted to make Xen guest be NUMA aware in this patch series.

Quoted 7 lines

The concept of patch series ceases to exist once we merge the code. So about how:

"The NUMA information provided in the host Device-Tree are only for Xen. For dom0, we want to hide them as they may be different (for now, dom0 is still not aware of NUMA".

Thanks, I will do it.

Quoted 21 lines
Quoted 3 lines

So in this patch, Xen will skip NUMA distance matrix node and skip the numa-node-id property in CPU node and memory node, when Xen is creating guest device tree binary.

The CPU and memory nodes are recreated from scratch for the domain. So we already skip the property numa-node-id. However...

xen/arch/arm/domain_build.c static int __init write_properties(struct domain *d,

Quoted 10 lines
---
  xen/arch/arm/domain_build.c | 6 ++++++
  1 file changed, 6 insertions(+)
xen/arch/arm/domain_build.c +0 -0
index cf341f349f..e62fa761bd 100644
@@ -584,6 +584,10 @@ static int __init write_properties(struct domain *d,

struct kernel_info *kinfo,

Quoted 3 lines
xen/arch/arm/domain_build.c +0 -0
                  continue;
          }
 
Quoted 6 lines

xen/arch/arm/domain_build.c static int __init write_properties(struct domain *d, struct kernel_info *kinfo, · line 589

Quoted 3 lines
xen/arch/arm/domain_build.c +3 -0
+        /* Guest is numa unaware in current stage */
+        if ( dt_property_name_is_equal(prop, "numa-node-id") )
+            continue;

... your code is doing more than skipping the property for the two nodes you mentionned. Can the property exists in other nodes?

Some devices like PCIe may have numa-node-id. In the future maybe more device can have NUMA property.

Quoted 22 lines

xen/arch/arm/domain_build.c static int __init write_properties(struct domain *d, · line 591

Quoted 3 lines
xen/arch/arm/domain_build.c +1 -0
+
          res = fdt_property(kinfo->fdt, prop->name, prop_data,

prop_len);

xen/arch/arm/domain_build.c static int __init handle_node(struct domain *d,

Quoted 3 lines
xen/arch/arm/domain_build.c +0 -0
          if ( res )
@@ -1454,6 +1458,8 @@ static int __init handle_node(struct domain *d,

struct kernel_info *kinfo,

xen/arch/arm/domain_build.c static int __init handle_node(struct domain *d, struct kernel_info *kinfo, · line 1465

Quoted 9 lines
xen/arch/arm/domain_build.c +2 -0
          DT_MATCH_TYPE("memory"),
          /* The memory mapped timer is not supported by Xen. */
          DT_MATCH_COMPATIBLE("arm,armv7-timer-mem"),
+        /* Numa info doesn't need to be exposed to Domain-0 */
+        DT_MATCH_COMPATIBLE("numa-distance-map-v1"),
          { /* sentinel */ },
      };
      static const struct dt_device_match timer_matches[] __initconst =

Cheers,

--
Julien Grall

author
#169 lore raw

RFC PATCH 37/40 xen: introduce an arch helper to do NUMA init failed fallback

In reply to Wei Chen

When Xen initialize NUMA failed, some architectures may need to do fallback actions. For example, in device tree based NUMA, Arm need to reset the distance between any two nodes.

---
 xen/arch/arm/numa.c        | 13 +++++++++++++
 xen/common/numa.c          |  3 +++
 xen/include/asm-arm/numa.h |  1 +
 xen/include/asm-x86/numa.h |  6 ++++++
 4 files changed, 23 insertions(+)
xen/arch/arm/numa.c +13 -0
index 6eebf8e8bc..2a18c97470 100644
@@ -140,3 +140,16 @@ int __init arch_meminfo_get_ram_bank_range(int bank,
 
 	return 0;
 }
+
+void __init arch_numa_init_failed_fallback(void)
+{
+    int i, j;
+
+    /* Reset all node distance to remote_distance */
+    for ( i = 0; i < MAX_NUMNODES; i++ ) {
+        for ( j = 0; j < MAX_NUMNODES; j++ ) {
+            numa_set_distance(i, j,
+                (i == j) ? NUMA_LOCAL_DISTANCE : NUMA_REMOTE_DISTANCE);
+        }
+    }
+}
xen/common/numa.c +3 -0
index d15c2fc311..88f1594127 100644
@@ -405,4 +405,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
     cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
     setup_node_bootmem(0, (u64)start_pfn << PAGE_SHIFT,
                     (u64)end_pfn << PAGE_SHIFT);
+
+    /* architecture specified fallback operations */
+    arch_numa_init_failed_fallback();
 }
xen/include/asm-arm/numa.h +1 -0
index dd31324b0b..a3982a94b6 100644
@@ -28,6 +28,7 @@ extern s8 device_tree_numa;
 extern void numa_init(bool acpi_off);
 extern int numa_device_tree_init(const void *fdt);
 extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_t distance);
+extern void arch_numa_init_failed_fallback(void);
 
 /*
  * Temporary for fake NUMA node, when CPU, memory and distance
xen/include/asm-x86/numa.h +6 -0
index e63869135c..26280b0f3a 100644
@@ -22,4 +22,10 @@ extern void init_cpu_to_node(void);
 void srat_parse_regions(u64 addr);
 unsigned int arch_get_dma_bitsize(void);
 
+/* Dummy function for numa init failed in numa_initmem_init */
+static inline void arch_numa_init_failed_fallback(void)
+{
+    return;
+}
+
 #endif
Signature · 1 line
2.25.1
#170 lore raw

Re: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallback

Hi,

Quoted 3 lines · Wei Chen

When Xen initialize NUMA failed, some architectures may need to do fallback actions. For example, in device tree based NUMA, Arm need to reset the distance between any two nodes.

From the description here, I don't understand why we need to reset the distance for Arm but not x86. In fact...

Quoted 20 lines
---
  xen/arch/arm/numa.c        | 13 +++++++++++++
  xen/common/numa.c          |  3 +++
  xen/include/asm-arm/numa.h |  1 +
  xen/include/asm-x86/numa.h |  6 ++++++
  4 files changed, 23 insertions(+)
xen/arch/arm/numa.c +3 -0
index 6eebf8e8bc..2a18c97470 100644
@@ -140,3 +140,16 @@ int __init arch_meminfo_get_ram_bank_range(int bank,
 
  	return 0;
  }
+
+void __init arch_numa_init_failed_fallback(void)
+{

xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int bank, · line 155

Quoted 10 lines
xen/arch/arm/numa.c +10 -0
+    int i, j;
+
+    /* Reset all node distance to remote_distance */
+    for ( i = 0; i < MAX_NUMNODES; i++ ) {
+        for ( j = 0; j < MAX_NUMNODES; j++ ) {
+            numa_set_distance(i, j,
+                (i == j) ? NUMA_LOCAL_DISTANCE : NUMA_REMOTE_DISTANCE);
+        }
+    }
+}

... this implementation looks fairly generic. So can you explain why we need it on Arm but not x86?

Quoted 29 lines
xen/common/numa.c +3 -0
index d15c2fc311..88f1594127 100644
@@ -405,4 +405,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
      cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
      setup_node_bootmem(0, (u64)start_pfn << PAGE_SHIFT,
                      (u64)end_pfn << PAGE_SHIFT);
+
+    /* architecture specified fallback operations */
+    arch_numa_init_failed_fallback();
  }
xen/include/asm-arm/numa.h +1 -0
index dd31324b0b..a3982a94b6 100644
@@ -28,6 +28,7 @@ extern s8 device_tree_numa;
  extern void numa_init(bool acpi_off);
  extern int numa_device_tree_init(const void *fdt);
  extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_t distance);
+extern void arch_numa_init_failed_fallback(void);
 
  /*
   * Temporary for fake NUMA node, when CPU, memory and distance
xen/include/asm-x86/numa.h +0 -0
index e63869135c..26280b0f3a 100644

xen/include/asm-x86/numa.h extern void init_cpu_to_node(void); · line 28

Quoted 7 lines
xen/include/asm-x86/numa.h +4 -0
@@ -22,4 +22,10 @@ extern void init_cpu_to_node(void);
  void srat_parse_regions(u64 addr);
  unsigned int arch_get_dma_bitsize(void);
 
+/* Dummy function for numa init failed in numa_initmem_init */
+static inline void arch_numa_init_failed_fallback(void)
+{
+    return;

NIT: The return is pointless.

xen/include/asm-x86/numa.h extern void init_cpu_to_node(void); · line 31

Quoted 4 lines
xen/include/asm-x86/numa.h +2 -0
+}
+
  #endif

Cheers,

Signature · 1 line
Julien Grall
author
#171 lore raw

RE: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallback

Hi Julien,

Quoted 44 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:30
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallback

Hi,

Quoted 3 lines · Wei Chen

When Xen initialize NUMA failed, some architectures may need to do fallback actions. For example, in device tree based NUMA, Arm need to reset the distance between any two nodes.

From the description here, I don't understand why we need to reset the
distance for Arm but not x86. In fact...

Quoted 25 lines
---
  xen/arch/arm/numa.c        | 13 +++++++++++++
  xen/common/numa.c          |  3 +++
  xen/include/asm-arm/numa.h |  1 +
  xen/include/asm-x86/numa.h |  6 ++++++
  4 files changed, 23 insertions(+)
xen/arch/arm/numa.c +8 -0
index 6eebf8e8bc..2a18c97470 100644
@@ -140,3 +140,16 @@ int __init arch_meminfo_get_ram_bank_range(int bank,
 
  	return 0;
  }
+
+void __init arch_numa_init_failed_fallback(void)
+{
+    int i, j;
+
+    /* Reset all node distance to remote_distance */
+    for ( i = 0; i < MAX_NUMNODES; i++ ) {
+        for ( j = 0; j < MAX_NUMNODES; j++ ) {
Quoted 9 lines

xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int bank, · line 155

Quoted 5 lines
xen/arch/arm/numa.c +5 -0
+            numa_set_distance(i, j,
+                (i == j) ? NUMA_LOCAL_DISTANCE : NUMA_REMOTE_DISTANCE);
+        }
+    }
+}

... this implementation looks fairly generic. So can you explain why we need it on Arm but not x86?

This implementation is DT only, for x86, it's using acpi_slit. For now, I am not quit sure ACPI need to do fallback or not. Or say in another way, I don't know how to implement the fallback for ACPI. I planned to solve it in Arm ACPI version NUMA, so I left an empty helper for x86.

@Jan Beulich Could you give me some suggestion about x86 fallback?

Quoted 34 lines

xen/common/numa.c void __init numa_initmem_init(unsigned long

Quoted 5 lines
xen/common/numa.c +0 -0
index d15c2fc311..88f1594127 100644
@@ -405,4 +405,7 @@ void __init numa_initmem_init(unsigned long

start_pfn, unsigned long end_pfn)

Quoted 12 lines
xen/common/numa.c +3 -0
      cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
      setup_node_bootmem(0, (u64)start_pfn << PAGE_SHIFT,
                      (u64)end_pfn << PAGE_SHIFT);
+
+    /* architecture specified fallback operations */
+    arch_numa_init_failed_fallback();
  }
xen/include/asm-arm/numa.h +0 -0
index dd31324b0b..a3982a94b6 100644

xen/include/asm-arm/numa.h extern s8 device_tree_numa; · line 30

Quoted 3 lines
xen/include/asm-arm/numa.h +0 -0
@@ -28,6 +28,7 @@ extern s8 device_tree_numa;
  extern void numa_init(bool acpi_off);
  extern int numa_device_tree_init(const void *fdt);
  extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_t

distance);

Quoted 12 lines
xen/include/asm-arm/numa.h +1 -0
+extern void arch_numa_init_failed_fallback(void);
 
  /*
   * Temporary for fake NUMA node, when CPU, memory and distance
xen/include/asm-x86/numa.h +0 -0
index e63869135c..26280b0f3a 100644
@@ -22,4 +22,10 @@ extern void init_cpu_to_node(void);
  void srat_parse_regions(u64 addr);
  unsigned int arch_get_dma_bitsize(void);
 
Quoted 7 lines

xen/include/asm-x86/numa.h extern void init_cpu_to_node(void); · line 28

Quoted 4 lines
xen/include/asm-x86/numa.h +4 -0
+/* Dummy function for numa init failed in numa_initmem_init */
+static inline void arch_numa_init_failed_fallback(void)
+{
+    return;

NIT: The return is pointless.

OK

Quoted 9 lines

xen/include/asm-x86/numa.h extern void init_cpu_to_node(void); · line 31

Quoted 4 lines
xen/include/asm-x86/numa.h +2 -0
+}
+
  #endif

Cheers,

--
Julien Grall

author
#172 lore raw

RE: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallback

Hi Julien, Jan

Quoted 68 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Wei Chen
Sent: 2021年8月28日 11:09
To: Julien Grall <[email protected]>; [email protected]; [email protected]; Jan Beulich <[email protected]>
Cc: Bertrand Marquis <[email protected]>
Subject: RE: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallback

Hi Julien,

Quoted 28 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:30
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallback

Hi,

Quoted 3 lines · Wei Chen

When Xen initialize NUMA failed, some architectures may need to do fallback actions. For example, in device tree based NUMA, Arm need to reset the distance between any two nodes.

From the description here, I don't understand why we need to reset the
distance for Arm but not x86. In fact...

Quoted 9 lines
---
  xen/arch/arm/numa.c        | 13 +++++++++++++
  xen/common/numa.c          |  3 +++
  xen/include/asm-arm/numa.h |  1 +
  xen/include/asm-x86/numa.h |  6 ++++++
  4 files changed, 23 insertions(+)
Quoted 5 lines

xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int

Quoted 5 lines
xen/arch/arm/numa.c +0 -0
index 6eebf8e8bc..2a18c97470 100644
@@ -140,3 +140,16 @@ int __init arch_meminfo_get_ram_bank_range(int

bank,

Quoted 13 lines

xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int bank,

Quoted 13 lines
xen/arch/arm/numa.c +10 -0
  	return 0;
  }
+
+void __init arch_numa_init_failed_fallback(void)
+{
+    int i, j;
+
+    /* Reset all node distance to remote_distance */
+    for ( i = 0; i < MAX_NUMNODES; i++ ) {
+        for ( j = 0; j < MAX_NUMNODES; j++ ) {
+            numa_set_distance(i, j,
+                (i == j) ? NUMA_LOCAL_DISTANCE :

NUMA_REMOTE_DISTANCE);

Quoted 7 lines

xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int bank, · line 155

Quoted 3 lines
xen/arch/arm/numa.c +3 -0
+        }
+    }
+}

... this implementation looks fairly generic. So can you explain why we need it on Arm but not x86?

Quoted 9 lines

This implementation is DT only, for x86, it's using acpi_slit. For now, I am not quit sure ACPI need to do fallback or not. Or say in another way, I don't know how to implement the fallback for ACPI. I planned to solve it in Arm ACPI version NUMA, so I left an empty helper for x86.

@Jan Beulich Could you give me some suggestion about x86 fallback?

I have a quick look into Linux. When Arch do numa init failed, the numa_free_distance will be invoked to revert numa_distance.

Quoted 53 lines
Quoted 34 lines

xen/common/numa.c void __init numa_initmem_init(unsigned long

Quoted 5 lines
xen/common/numa.c +0 -0
index d15c2fc311..88f1594127 100644
@@ -405,4 +405,7 @@ void __init numa_initmem_init(unsigned long

start_pfn, unsigned long end_pfn)

Quoted 12 lines
xen/common/numa.c +3 -0
      cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
      setup_node_bootmem(0, (u64)start_pfn << PAGE_SHIFT,
                      (u64)end_pfn << PAGE_SHIFT);
+
+    /* architecture specified fallback operations */
+    arch_numa_init_failed_fallback();
  }
xen/include/asm-arm/numa.h +0 -0
index dd31324b0b..a3982a94b6 100644

xen/include/asm-arm/numa.h extern s8 device_tree_numa; · line 30

Quoted 3 lines
xen/include/asm-arm/numa.h +0 -0
@@ -28,6 +28,7 @@ extern s8 device_tree_numa;
  extern void numa_init(bool acpi_off);
  extern int numa_device_tree_init(const void *fdt);
  extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_t

distance);

Quoted 12 lines
xen/include/asm-arm/numa.h +1 -0
+extern void arch_numa_init_failed_fallback(void);
 
  /*
   * Temporary for fake NUMA node, when CPU, memory and distance
xen/include/asm-x86/numa.h +0 -0
index e63869135c..26280b0f3a 100644
@@ -22,4 +22,10 @@ extern void init_cpu_to_node(void);
  void srat_parse_regions(u64 addr);
  unsigned int arch_get_dma_bitsize(void);
 
Quoted 7 lines

xen/include/asm-x86/numa.h extern void init_cpu_to_node(void); · line 28

Quoted 4 lines
xen/include/asm-x86/numa.h +4 -0
+/* Dummy function for numa init failed in numa_initmem_init */
+static inline void arch_numa_init_failed_fallback(void)
+{
+    return;

NIT: The return is pointless.

OK

Quoted 9 lines

xen/include/asm-x86/numa.h extern void init_cpu_to_node(void); · line 31

Quoted 4 lines
xen/include/asm-x86/numa.h +2 -0
+}
+
  #endif

Cheers,

--
Julien Grall

#173 lore raw

Re: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallback

Quoted 39 lines · Wei Chen
Quoted 39 lines

From: Xen-devel <[email protected]> On Behalf Of Wei Chen
Sent: 2021年8月28日 11:09

Quoted 6 lines

From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:30

xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int

Quoted 3 lines
xen/arch/arm/numa.c +0 -0
@@ -140,3 +140,16 @@ int __init arch_meminfo_get_ram_bank_range(int

bank,

Quoted 13 lines

xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int bank,

Quoted 13 lines
xen/arch/arm/numa.c +10 -0
  	return 0;
  }
+
+void __init arch_numa_init_failed_fallback(void)
+{
+    int i, j;
+
+    /* Reset all node distance to remote_distance */
+    for ( i = 0; i < MAX_NUMNODES; i++ ) {
+        for ( j = 0; j < MAX_NUMNODES; j++ ) {
+            numa_set_distance(i, j,
+                (i == j) ? NUMA_LOCAL_DISTANCE :

NUMA_REMOTE_DISTANCE);

Quoted 7 lines

xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int bank, · line 155

Quoted 3 lines
xen/arch/arm/numa.c +3 -0
+        }
+    }
+}

... this implementation looks fairly generic. So can you explain why we need it on Arm but not x86?

This implementation is DT only, for x86, it's using acpi_slit. For now, I am not quit sure ACPI need to do fallback or not. Or say in another way, I don't know how to implement the fallback for ACPI. I planned to solve it in Arm ACPI version NUMA, so I left an empty helper for x86.

Quoted 6 lines
Quoted 3 lines

@Jan Beulich Could you give me some suggestion about x86 fallback?

I have a quick look into Linux. When Arch do numa init failed, the numa_free_distance will be invoked to revert numa_distance.

Does this matter in the first place? Don't we fall back to single node mode, in which case the sole entry of the distance table will say "local" anyway?

Jan

author
#174 lore raw

RE: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallback

Hi Jan,

Quoted 55 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月30日 17:52
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; Julien Grall <[email protected]>; [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallback

Quoted 1 line · Wei Chen
Quoted 1 line

From: Xen-devel <[email protected]> On Behalf Of

Wei

Quoted 24 lines
Quoted 24 lines

Chen
Sent: 2021年8月28日 11:09

Quoted 6 lines

From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:30

xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int

Quoted 3 lines
xen/arch/arm/numa.c +0 -0
@@ -140,3 +140,16 @@ int __init arch_meminfo_get_ram_bank_range(int

bank,

Quoted 13 lines

xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int bank,

Quoted 13 lines
xen/arch/arm/numa.c +10 -0
  	return 0;
  }
+
+void __init arch_numa_init_failed_fallback(void)
+{
+    int i, j;
+
+    /* Reset all node distance to remote_distance */
+    for ( i = 0; i < MAX_NUMNODES; i++ ) {
+        for ( j = 0; j < MAX_NUMNODES; j++ ) {
+            numa_set_distance(i, j,
+                (i == j) ? NUMA_LOCAL_DISTANCE :

NUMA_REMOTE_DISTANCE);

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int bank, · line 155

Quoted 3 lines
xen/arch/arm/numa.c +3 -0
+        }
+    }
+}

... this implementation looks fairly generic. So can you explain why

we

Quoted 13 lines
Quoted 12 lines
Quoted 2 lines

need it on Arm but not x86?

This implementation is DT only, for x86, it's using acpi_slit. For now, I am not quit sure ACPI need to do fallback or not. Or say in another way, I don't know how to implement the fallback for ACPI. I planned to solve it in Arm ACPI version NUMA, so I left an empty helper for x86.

@Jan Beulich Could you give me some suggestion about x86 fallback?

Quoted 7 lines
Quoted 2 lines

I have a quick look into Linux. When Arch do numa init failed, the numa_free_distance will be invoked to revert numa_distance.

Does this matter in the first place? Don't we fall back to single node mode, in which case the sole entry of the distance table will say "local" anyway?

Thank you for providing another way of thinking. Yes, once NUMA init is failed, the system will fall back to single node mode. If we call __node_distance normally, we will not pass two different nodes to this function. Even if we don't revert the values in distance table, we will not trigger the condition of "node_a != node_b". We will always get "local" from __node_distance.

But for closed-loop of code, I tend to revert data when initialization is failed.

Cheers,
Wei Chen

Quoted 1 line

Jan

author
#175 lore raw

RFC PATCH 38/40 xen/arm: enable device tree based NUMA in system init

In reply to Wei Chen

Everything is ready, we can remove the fake NUMA node and depends on device tree to create NUMA system.

---
 xen/arch/arm/numa.c        | 45 ++++++++++++++++++++++----------------
 xen/include/asm-arm/numa.h |  7 ------
 2 files changed, 26 insertions(+), 26 deletions(-)
xen/arch/arm/numa.c +26 -19
index 2a18c97470..3b04220e60 100644
@@ -18,6 +18,7 @@
  *
  */
 #include <xen/init.h>
+#include <xen/device_tree.h>
 #include <xen/nodemask.h>
 #include <xen/numa.h>
 #include <xen/pfn.h>
@@ -83,28 +84,34 @@ void __init numa_init(bool acpi_off)
     paddr_t ram_size = 0;
     paddr_t ram_end = 0;
 
-    printk(XENLOG_WARNING
-        "NUMA has not been supported yet, NUMA off!\n");
-    /* Arm NUMA has not been implemented until this patch */
-    numa_off = true;
+    /* NUMA has been turned off through Xen parameters */
+    if ( numa_off )
+        goto mem_init;
 
-    /*
-     * Set all cpu_to_node mapping to 0, this will make cpu_to_node
-     * function return 0 as previous fake cpu_to_node API.
-     */
-    for ( idx = 0; idx < NR_CPUS; idx++ )
-        cpu_to_node[idx] = 0;
-
-    /*
-     * Make node_to_cpumask, node_spanned_pages and node_start_pfn
-     * return as previous fake APIs.
-     */
-    for ( idx = 0; idx < MAX_NUMNODES; idx++ ) {
-        node_to_cpumask[idx] = cpu_online_map;
-        node_spanned_pages(idx) = (max_page - mfn_x(first_valid_mfn));
-        node_start_pfn(idx) = (mfn_x(first_valid_mfn));
+    /* Initialize NUMA from device tree when system is not ACPI booted */
+    if ( acpi_off )
+    {
+#ifdef CONFIG_DEVICE_TREE_NUMA
+        int ret = numa_device_tree_init(device_tree_flattened);
+        if ( !ret )
+            goto mem_init;
+        printk(XENLOG_WARNING
+               "Init NUMA from device tree failed, ret=%d\n", ret);
+#else
+        printk(XENLOG_WARNING
+               "CONFIG_DEVICE_TREE_NUMA is not set, NUMA off!\n");
+#endif
+        numa_off = true;
+    }
+    else
+    {
+        /* We don't support NUMA for ACPI boot currently */
+        printk(XENLOG_WARNING
+               "ACPI NUMA has not been supported yet, NUMA off!\n");
+        numa_off = true;
     }
 
+mem_init:
     /*
      * Find the minimal and maximum address of RAM, NUMA will
      * build a memory to node mapping table for the whole range.
xen/include/asm-arm/numa.h +0 -7
index a3982a94b6..425eb9aede 100644
@@ -30,13 +30,6 @@ extern int numa_device_tree_init(const void *fdt);
 extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_t distance);
 extern void arch_numa_init_failed_fallback(void);
 
-/*
- * Temporary for fake NUMA node, when CPU, memory and distance
- * matrix will be read from DTB or ACPI SRAT. The following
- * symbols will be removed.
- */
-extern mfn_t first_valid_mfn;
-
 #else
 
 /* Fake one node for now. See also node_online_map. */
Signature · 1 line
2.25.1
#176 lore raw

Re: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system init

Hi Wei,

Quoted 2 lines · Wei Chen

Everything is ready, we can remove the fake NUMA node and depends on device tree to create NUMA system.

So you just added code a few patches before that are now completely rewritten. Can you please re-order this series so it doesn't happen?

This may mean that CONFIG_NUMA is only selected until late in this series.

Cheers,

Signature · 1 line
Julien Grall
author
#177 lore raw

RE: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system init

Hi Julien,

Quoted 11 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:33
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system init

Hi Wei,

Quoted 9 lines
Quoted 2 lines · Wei Chen

Everything is ready, we can remove the fake NUMA node and depends on device tree to create NUMA system.

So you just added code a few patches before that are now completely rewritten. Can you please re-order this series so it doesn't happen?

This may mean that CONFIG_NUMA is only selected until late in this series.

Why I did like this is because my original concerns are:
1. When I introduced the CONFIG_NUMA. Users will be using a code base on this commit by accident.
2. If users select CONFIG_NUMA, but not all NUMA data are not initialized properly. The system may not work properly.
3. So I created the fake node to initialize the NUMA data, before we parser real data from DTB.
4. In this case, user can work well with CONFIG_NUMA is enabled, without this series is completed.

It seems I thought too much. If these concerns are not necessary. I am OK to re-order this series.

Quoted 4 lines

Cheers,

--
Julien Grall

#178 lore raw

Re: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system init

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 19 lines
Quoted 19 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:33
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system init

Hi Wei,

Quoted 2 lines · Wei Chen

Everything is ready, we can remove the fake NUMA node and depends on device tree to create NUMA system.

So you just added code a few patches before that are now completely rewritten. Can you please re-order this series so it doesn't happen?

Quoted 8 lines
Quoted 2 lines

This may mean that CONFIG_NUMA is only selected until late in this series.

Why I did like this is because my original concerns are:
1. When I introduced the CONFIG_NUMA. Users will be using a code base on this commit by accident.
2. If users select CONFIG_NUMA, but not all NUMA data are not initialized properly. The system may not work properly.

We have to make sure we don't break any existing use case when writing a new feature. However, a user should not expect a new feature to work it is using a random commit in the middle of the series.

This is also why I suggested that maybe CONFIG_NUMA is only selected for Arm towards the end of the series. So you reduce the risk of someone selecting it.

Quoted 4 lines

3. So I created the fake node to initialize the NUMA data, before we parser real data from DTB.
4. In this case, user can work well with CONFIG_NUMA is enabled, without this series is completed.

The flip side is you are adding more load on the reviewers because there are extra code. The series is already quite big (40 patches), any way to ease the review will definitely be appreciated.

Another possible way to ease the review is to move the patch that rework/move code at the beginning of the series and leave the Arm part for the second part of the series. This way, we can start to merge the series without waiting for the Arm bits to be completed.

Cheers,

Signature · 1 line
Julien Grall
author
#179 lore raw

RE: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system init

Hi Julien,

Quoted 47 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月28日 18:45
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system init

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 8 lines
Quoted 8 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:33
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 38/40] xen/arm: enable device tree based

NUMA

Quoted 4 lines
Quoted 4 lines

in system init

Hi Wei,

Quoted 8 lines
Quoted 8 lines · Wei Chen
Quoted 2 lines

Everything is ready, we can remove the fake NUMA node and depends on device tree to create NUMA system.

So you just added code a few patches before that are now completely rewritten. Can you please re-order this series so it doesn't happen?

This may mean that CONFIG_NUMA is only selected until late in this

series.

Quoted 6 lines
Quoted 1 line

Why I did like this is because my original concerns are:
1. When I introduced the CONFIG_NUMA. Users will be using a code base on this commit by accident.
2. If users select CONFIG_NUMA, but not all NUMA data are not

initialized

Quoted 1 line

properly. The system may not work properly.

Quoted 8 lines

We have to make sure we don't break any existing use case when writing a new feature. However, a user should not expect a new feature to work it is using a random commit in the middle of the series.

This is also why I suggested that maybe CONFIG_NUMA is only selected for Arm towards the end of the series. So you reduce the risk of someone selecting it.

Thanks for this clarification.

Quoted 6 lines
Quoted 1 line

3. So I created the fake node to initialize the NUMA data, before we

parser

Quoted 3 lines

real data from DTB.
4. In this case, user can work well with CONFIG_NUMA is enabled, without this series is completed.

Quoted 9 lines

The flip side is you are adding more load on the reviewers because there are extra code. The series is already quite big (40 patches), any way to ease the review will definitely be appreciated.

Another possible way to ease the review is to move the patch that rework/move code at the beginning of the series and leave the Arm part for the second part of the series. This way, we can start to merge the series without waiting for the Arm bits to be completed.

Yes, I will try to re-order the patches in this way in next version.

Quoted 4 lines

Cheers,

--
Julien Grall

#180 lore raw

Re: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system init

In reply to Wei Chen
Quoted 50 lines · Wei Chen

Everything is ready, we can remove the fake NUMA node and depends on device tree to create NUMA system.

---
 xen/arch/arm/numa.c        | 45 ++++++++++++++++++++++----------------
 xen/include/asm-arm/numa.h |  7 ------
 2 files changed, 26 insertions(+), 26 deletions(-)
xen/arch/arm/numa.c +6 -19
index 2a18c97470..3b04220e60 100644
@@ -18,6 +18,7 @@
  *
  */
 #include <xen/init.h>
+#include <xen/device_tree.h>
 #include <xen/nodemask.h>
 #include <xen/numa.h>
 #include <xen/pfn.h>
@@ -83,28 +84,34 @@ void __init numa_init(bool acpi_off)
     paddr_t ram_size = 0;
     paddr_t ram_end = 0;
 
-    printk(XENLOG_WARNING
-        "NUMA has not been supported yet, NUMA off!\n");
-    /* Arm NUMA has not been implemented until this patch */
-    numa_off = true;
+    /* NUMA has been turned off through Xen parameters */
+    if ( numa_off )
+        goto mem_init;
 
-    /*
-     * Set all cpu_to_node mapping to 0, this will make cpu_to_node
-     * function return 0 as previous fake cpu_to_node API.
-     */
-    for ( idx = 0; idx < NR_CPUS; idx++ )
-        cpu_to_node[idx] = 0;
-
-    /*
-     * Make node_to_cpumask, node_spanned_pages and node_start_pfn
-     * return as previous fake APIs.
-     */
-    for ( idx = 0; idx < MAX_NUMNODES; idx++ ) {
-        node_to_cpumask[idx] = cpu_online_map;
-        node_spanned_pages(idx) = (max_page - mfn_x(first_valid_mfn));
-        node_start_pfn(idx) = (mfn_x(first_valid_mfn));
+    /* Initialize NUMA from device tree when system is not ACPI booted */
+    if ( acpi_off )

xen/arch/arm/numa.c void __init numa_init(bool acpi_off) · line 102

Quoted 10 lines
xen/arch/arm/numa.c +10 -0
+    {
+#ifdef CONFIG_DEVICE_TREE_NUMA
+        int ret = numa_device_tree_init(device_tree_flattened);
+        if ( !ret )
+            goto mem_init;
+        printk(XENLOG_WARNING
+               "Init NUMA from device tree failed, ret=%d\n", ret);
+#else
+        printk(XENLOG_WARNING
+               "CONFIG_DEVICE_TREE_NUMA is not set, NUMA off!\n");

I don't think we want to see this warning every time at boot when CONFIG_DEVICE_TREE_NUMA is off. I'd set it to XENLOG_DEBUG or remove it.

Also given that we have many stub functions in xen/include/asm-arm/numa.h already, maybe we could also have a stub function for numa_device_tree_init so that we won'd need an #ifdef CONFIG_DEVICE_TREE_NUMA here.

Quoted 36 lines
xen/arch/arm/numa.c +10 -0
+#endif
+        numa_off = true;
+    }
+    else
+    {
+        /* We don't support NUMA for ACPI boot currently */
+        printk(XENLOG_WARNING
+               "ACPI NUMA has not been supported yet, NUMA off!\n");
+        numa_off = true;
     }
 
+mem_init:
     /*
      * Find the minimal and maximum address of RAM, NUMA will
      * build a memory to node mapping table for the whole range.
xen/include/asm-arm/numa.h +0 -7
index a3982a94b6..425eb9aede 100644
@@ -30,13 +30,6 @@ extern int numa_device_tree_init(const void *fdt);
 extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_t distance);
 extern void arch_numa_init_failed_fallback(void);
 
-/*
- * Temporary for fake NUMA node, when CPU, memory and distance
- * matrix will be read from DTB or ACPI SRAT. The following
- * symbols will be removed.
- */
-extern mfn_t first_valid_mfn;
-
 #else
 
 /* Fake one node for now. See also node_online_map. */
Signature · 1 line
2.25.1
author
#181 lore raw

RE: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system init

Hi Stefano,

Quoted 60 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 9:51
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system init

Quoted 39 lines · Wei Chen

Everything is ready, we can remove the fake NUMA node and depends on device tree to create NUMA system.

---
 xen/arch/arm/numa.c        | 45 ++++++++++++++++++++++----------------
 xen/include/asm-arm/numa.h |  7 ------
 2 files changed, 26 insertions(+), 26 deletions(-)
xen/arch/arm/numa.c +4 -10
index 2a18c97470..3b04220e60 100644
@@ -18,6 +18,7 @@
  *
  */
 #include <xen/init.h>
+#include <xen/device_tree.h>
 #include <xen/nodemask.h>
 #include <xen/numa.h>
 #include <xen/pfn.h>
@@ -83,28 +84,34 @@ void __init numa_init(bool acpi_off)
     paddr_t ram_size = 0;
     paddr_t ram_end = 0;
 
-    printk(XENLOG_WARNING
-        "NUMA has not been supported yet, NUMA off!\n");
-    /* Arm NUMA has not been implemented until this patch */
-    numa_off = true;
+    /* NUMA has been turned off through Xen parameters */
+    if ( numa_off )
+        goto mem_init;
 
-    /*
-     * Set all cpu_to_node mapping to 0, this will make cpu_to_node
-     * function return 0 as previous fake cpu_to_node API.
-     */
-    for ( idx = 0; idx < NR_CPUS; idx++ )
-        cpu_to_node[idx] = 0;

xen/arch/arm/numa.c void __init numa_init(bool acpi_off) · line 91

Quoted 10 lines
xen/arch/arm/numa.c +1 -9
-
-    /*
-     * Make node_to_cpumask, node_spanned_pages and node_start_pfn
-     * return as previous fake APIs.
-     */
-    for ( idx = 0; idx < MAX_NUMNODES; idx++ ) {
-        node_to_cpumask[idx] = cpu_online_map;
-        node_spanned_pages(idx) = (max_page - mfn_x(first_valid_mfn));
-        node_start_pfn(idx) = (mfn_x(first_valid_mfn));
+    /* Initialize NUMA from device tree when system is not ACPI booted

*/

Quoted 15 lines

xen/arch/arm/numa.c void __init numa_init(bool acpi_off) · line 102

Quoted 11 lines
xen/arch/arm/numa.c +11 -0
+    if ( acpi_off )
+    {
+#ifdef CONFIG_DEVICE_TREE_NUMA
+        int ret = numa_device_tree_init(device_tree_flattened);
+        if ( !ret )
+            goto mem_init;
+        printk(XENLOG_WARNING
+               "Init NUMA from device tree failed, ret=%d\n", ret);
+#else
+        printk(XENLOG_WARNING
+               "CONFIG_DEVICE_TREE_NUMA is not set, NUMA off!\n");

I don't think we want to see this warning every time at boot when CONFIG_DEVICE_TREE_NUMA is off. I'd set it to XENLOG_DEBUG or remove it.

OK

Quoted 5 lines

Also given that we have many stub functions in xen/include/asm-arm/numa.h already, maybe we could also have a stub function for numa_device_tree_init so that we won'd need an #ifdef CONFIG_DEVICE_TREE_NUMA here.

Yes, it's a good idea. I will do it.

Quoted 38 lines
Quoted 21 lines
xen/arch/arm/numa.c +10 -0
+#endif
+        numa_off = true;
+    }
+    else
+    {
+        /* We don't support NUMA for ACPI boot currently */
+        printk(XENLOG_WARNING
+               "ACPI NUMA has not been supported yet, NUMA off!\n");
+        numa_off = true;
     }
 
+mem_init:
     /*
      * Find the minimal and maximum address of RAM, NUMA will
      * build a memory to node mapping table for the whole range.
xen/include/asm-arm/numa.h +0 -0
index a3982a94b6..425eb9aede 100644

xen/include/asm-arm/numa.h extern int numa_device_tree_init(const void *fdt); · line 30

Quoted 1 line
xen/include/asm-arm/numa.h +0 -0
@@ -30,13 +30,6 @@ extern int numa_device_tree_init(const void *fdt);
 extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_t

distance);

Quoted 15 lines
xen/include/asm-arm/numa.h +0 -8
 extern void arch_numa_init_failed_fallback(void);
 
-/*
- * Temporary for fake NUMA node, when CPU, memory and distance
- * matrix will be read from DTB or ACPI SRAT. The following
- * symbols will be removed.
- */
-extern mfn_t first_valid_mfn;
-
 #else
 
 /* Fake one node for now. See also node_online_map. */
--

2.25.1

author
#182 lore raw

RFC PATCH 39/40 xen/x86: move numa_setup to common to support NUMA switch in command line

In reply to Wei Chen

Xen x86 has created a command line parameter "numa" as NUMA switch for user to turn on/off NUMA. As device tree based NUMA has been enabled for Arm, this parameter can be reused by Arm. So in this patch, we move this parameter to common.

---
 xen/arch/x86/numa.c    | 34 ----------------------------------
 xen/common/numa.c      | 35 ++++++++++++++++++++++++++++++++++-
 xen/include/xen/numa.h |  1 -
 3 files changed, 34 insertions(+), 36 deletions(-)
xen/arch/x86/numa.c +0 -34
index 8b43be4aa7..380d8ed6fd 100644
@@ -11,7 +11,6 @@
 #include <xen/nodemask.h>
 #include <xen/numa.h>
 #include <xen/keyhandler.h>
-#include <xen/param.h>
 #include <xen/time.h>
 #include <xen/smp.h>
 #include <xen/pfn.h>
@@ -19,9 +18,6 @@
 #include <xen/sched.h>
 #include <xen/softirq.h>
 
-static int numa_setup(const char *s);
-custom_param("numa", numa_setup);
-
 #ifndef Dprintk
 #define Dprintk(x...)
 #endif
@@ -50,35 +46,6 @@ void numa_set_node(int cpu, nodeid_t node)
     cpu_to_node[cpu] = node;
 }
 
-/* [numa=off] */
-static __init int numa_setup(const char *opt)
-{
-    if ( !strncmp(opt,"off",3) )
-        numa_off = true;
-    else if ( !strncmp(opt,"on",2) )
-        numa_off = false;
-#ifdef CONFIG_NUMA_EMU
-    else if ( !strncmp(opt, "fake=", 5) )
-    {
-        numa_off = false;
-        numa_fake = simple_strtoul(opt+5,NULL,0);
-        if ( numa_fake >= MAX_NUMNODES )
-            numa_fake = MAX_NUMNODES;
-    }
-#endif
-#ifdef CONFIG_ACPI_NUMA
-    else if ( !strncmp(opt,"noacpi",6) )
-    {
-        numa_off = false;
-        acpi_numa = -1;
-    }
-#endif
-    else
-        return -EINVAL;
-
-    return 0;
-} 
-
 /*
  * Setup early cpu_to_node.
  *
@@ -287,4 +254,3 @@ static __init int register_numa_trigger(void)
     return 0;
 }
 __initcall(register_numa_trigger);
-
xen/common/numa.c +34 -1
index 88f1594127..c98eb8d571 100644
@@ -14,8 +14,12 @@
 #include <xen/smp.h>
 #include <xen/pfn.h>
 #include <xen/sched.h>
+#include <xen/param.h>
 #include <asm/acpi.h>
 
+static int numa_setup(const char *s);
+custom_param("numa", numa_setup);
+
 struct node_data node_data[MAX_NUMNODES];
 
 /* Mapping from pdx to node id */
@@ -324,7 +328,7 @@ int __init numa_scan_nodes(u64 start, u64 end)
 }
 
 #ifdef CONFIG_NUMA_EMU
-int numa_fake __initdata = 0;
+static int numa_fake __initdata = 0;
 
 /* Numa emulation */
 static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
@@ -409,3 +413,32 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
     /* architecture specified fallback operations */
     arch_numa_init_failed_fallback();
 }
+
+/* [numa=off] */
+static __init int numa_setup(const char *opt)
+{
+    if ( !strncmp(opt,"off",3) )
+        numa_off = true;
+    else if ( !strncmp(opt,"on",2) )
+        numa_off = false;
+#ifdef CONFIG_NUMA_EMU
+    else if ( !strncmp(opt, "fake=", 5) )
+    {
+        numa_off = false;
+        numa_fake = simple_strtoul(opt+5,NULL,0);
+        if ( numa_fake >= MAX_NUMNODES )
+            numa_fake = MAX_NUMNODES;
+    }
+#endif
+#ifdef CONFIG_ACPI_NUMA
+    else if ( !strncmp(opt,"noacpi",6) )
+    {
+        numa_off = false;
+        acpi_numa = -1;
+    }
+#endif
+    else
+        return -EINVAL;
+
+    return 0;
+}
xen/include/xen/numa.h +0 -1
index b9b5d1ad88..c647fef736 100644
@@ -83,7 +83,6 @@ extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
 extern void numa_set_node(int cpu, nodeid_t node);
 extern int numa_scan_nodes(u64 start, u64 end);
 extern bool numa_off;
-extern int numa_fake;
 extern s8 acpi_numa;
 
 extern void setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end);
Signature · 1 line
2.25.1
#183 lore raw

Re: [XEN RFC PATCH 39/40] xen/x86: move numa_setup to common to support NUMA switch in command line

Hi Wei,

Quoted 102 lines · Wei Chen

Xen x86 has created a command line parameter "numa" as NUMA switch for user to turn on/off NUMA. As device tree based NUMA has been enabled for Arm, this parameter can be reused by Arm. So in this patch, we move this parameter to common.

---
  xen/arch/x86/numa.c    | 34 ----------------------------------
  xen/common/numa.c      | 35 ++++++++++++++++++++++++++++++++++-
  xen/include/xen/numa.h |  1 -
  3 files changed, 34 insertions(+), 36 deletions(-)
xen/arch/x86/numa.c +0 -34
index 8b43be4aa7..380d8ed6fd 100644
@@ -11,7 +11,6 @@
  #include <xen/nodemask.h>
  #include <xen/numa.h>
  #include <xen/keyhandler.h>
-#include <xen/param.h>
  #include <xen/time.h>
  #include <xen/smp.h>
  #include <xen/pfn.h>
@@ -19,9 +18,6 @@
  #include <xen/sched.h>
  #include <xen/softirq.h>
 
-static int numa_setup(const char *s);
-custom_param("numa", numa_setup);
-
  #ifndef Dprintk
  #define Dprintk(x...)
  #endif
@@ -50,35 +46,6 @@ void numa_set_node(int cpu, nodeid_t node)
      cpu_to_node[cpu] = node;
  }
 
-/* [numa=off] */
-static __init int numa_setup(const char *opt)
-{
-    if ( !strncmp(opt,"off",3) )
-        numa_off = true;
-    else if ( !strncmp(opt,"on",2) )
-        numa_off = false;
-#ifdef CONFIG_NUMA_EMU
-    else if ( !strncmp(opt, "fake=", 5) )
-    {
-        numa_off = false;
-        numa_fake = simple_strtoul(opt+5,NULL,0);
-        if ( numa_fake >= MAX_NUMNODES )
-            numa_fake = MAX_NUMNODES;
-    }
-#endif
-#ifdef CONFIG_ACPI_NUMA
-    else if ( !strncmp(opt,"noacpi",6) )
-    {
-        numa_off = false;
-        acpi_numa = -1;
-    }
-#endif
-    else
-        return -EINVAL;
-
-    return 0;
-}
-
  /*
   * Setup early cpu_to_node.
   *
@@ -287,4 +254,3 @@ static __init int register_numa_trigger(void)
      return 0;
  }
  __initcall(register_numa_trigger);
-
xen/common/numa.c +5 -1
index 88f1594127..c98eb8d571 100644
@@ -14,8 +14,12 @@
  #include <xen/smp.h>
  #include <xen/pfn.h>
  #include <xen/sched.h>
+#include <xen/param.h>
  #include <asm/acpi.h>
 
+static int numa_setup(const char *s);
+custom_param("numa", numa_setup);
+
  struct node_data node_data[MAX_NUMNODES];
 
  /* Mapping from pdx to node id */
@@ -324,7 +328,7 @@ int __init numa_scan_nodes(u64 start, u64 end)
  }
 
  #ifdef CONFIG_NUMA_EMU
-int numa_fake __initdata = 0;
+static int numa_fake __initdata = 0;
 
  /* Numa emulation */
  static int __init numa_emulation(u64 start_pfn, u64 end_pfn)

xen/common/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 417

Quoted 5 lines
xen/common/numa.c +2 -0
@@ -409,3 +413,32 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
      /* architecture specified fallback operations */
      arch_numa_init_failed_fallback();
  }
+
+/* [numa=off] */

The documentation also needs be be updated to reflect that facts this option is not architecture-agnostic.

Quoted 12 lines
xen/common/numa.c +12 -0
+static __init int numa_setup(const char *opt)
+{
+    if ( !strncmp(opt,"off",3) )
+        numa_off = true;
+    else if ( !strncmp(opt,"on",2) )
+        numa_off = false;
+#ifdef CONFIG_NUMA_EMU
+    else if ( !strncmp(opt, "fake=", 5) )
+    {
+        numa_off = false;
+        numa_fake = simple_strtoul(opt+5,NULL,0);
+        if ( numa_fake >= MAX_NUMNODES )

xen/common/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 439

Quoted 10 lines
xen/common/numa.c +10 -0
+            numa_fake = MAX_NUMNODES;
+    }
+#endif
+#ifdef CONFIG_ACPI_NUMA
+    else if ( !strncmp(opt,"noacpi",6) )
+    {
+        numa_off = false;
+        acpi_numa = -1;
+    }
+#endif

Looking at this code, I am not quite too sure to understand the difference between between "numa=noacpi" and "numa=off".

In fact, I am tempted to say this option should disappear because this is odd to have a firmware specific option just for ACPI but not DT. Even if we have one for each, this makes things a bit more complicated for the admin.

Quoted 11 lines
xen/common/numa.c +5 -0
+    else
+        return -EINVAL;
+
+    return 0;
+}
xen/include/xen/numa.h +0 -0
index b9b5d1ad88..c647fef736 100644

xen/include/xen/numa.h extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn); · line 88

Quoted 7 lines
xen/include/xen/numa.h +0 -1
@@ -83,7 +83,6 @@ extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
  extern void numa_set_node(int cpu, nodeid_t node);
  extern int numa_scan_nodes(u64 start, u64 end);
  extern bool numa_off;
-extern int numa_fake;
  extern s8 acpi_numa;
 
  extern void setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end);

Cheers,

Signature · 1 line
Julien Grall
author
#184 lore raw

RE: [XEN RFC PATCH 39/40] xen/x86: move numa_setup to common to support NUMA switch in command line

Hi Julien,

Quoted 115 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:38
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 39/40] xen/x86: move numa_setup to common to support NUMA switch in command line

Hi Wei,

xen/common/numa.c void __init numa_initmem_init(unsigned long

Quoted 102 lines · Wei Chen

Xen x86 has created a command line parameter "numa" as NUMA switch for user to turn on/off NUMA. As device tree based NUMA has been enabled for Arm, this parameter can be reused by Arm. So in this patch, we move this parameter to common.

---
  xen/arch/x86/numa.c    | 34 ----------------------------------
  xen/common/numa.c      | 35 ++++++++++++++++++++++++++++++++++-
  xen/include/xen/numa.h |  1 -
  3 files changed, 34 insertions(+), 36 deletions(-)
xen/arch/x86/numa.c +0 -34
index 8b43be4aa7..380d8ed6fd 100644
@@ -11,7 +11,6 @@
  #include <xen/nodemask.h>
  #include <xen/numa.h>
  #include <xen/keyhandler.h>
-#include <xen/param.h>
  #include <xen/time.h>
  #include <xen/smp.h>
  #include <xen/pfn.h>
@@ -19,9 +18,6 @@
  #include <xen/sched.h>
  #include <xen/softirq.h>
 
-static int numa_setup(const char *s);
-custom_param("numa", numa_setup);
-
  #ifndef Dprintk
  #define Dprintk(x...)
  #endif
@@ -50,35 +46,6 @@ void numa_set_node(int cpu, nodeid_t node)
      cpu_to_node[cpu] = node;
  }
 
-/* [numa=off] */
-static __init int numa_setup(const char *opt)
-{
-    if ( !strncmp(opt,"off",3) )
-        numa_off = true;
-    else if ( !strncmp(opt,"on",2) )
-        numa_off = false;
-#ifdef CONFIG_NUMA_EMU
-    else if ( !strncmp(opt, "fake=", 5) )
-    {
-        numa_off = false;
-        numa_fake = simple_strtoul(opt+5,NULL,0);
-        if ( numa_fake >= MAX_NUMNODES )
-            numa_fake = MAX_NUMNODES;
-    }
-#endif
-#ifdef CONFIG_ACPI_NUMA
-    else if ( !strncmp(opt,"noacpi",6) )
-    {
-        numa_off = false;
-        acpi_numa = -1;
-    }
-#endif
-    else
-        return -EINVAL;
-
-    return 0;
-}
-
  /*
   * Setup early cpu_to_node.
   *
@@ -287,4 +254,3 @@ static __init int register_numa_trigger(void)
      return 0;
  }
  __initcall(register_numa_trigger);
-
xen/common/numa.c +5 -1
index 88f1594127..c98eb8d571 100644
@@ -14,8 +14,12 @@
  #include <xen/smp.h>
  #include <xen/pfn.h>
  #include <xen/sched.h>
+#include <xen/param.h>
  #include <asm/acpi.h>
 
+static int numa_setup(const char *s);
+custom_param("numa", numa_setup);
+
  struct node_data node_data[MAX_NUMNODES];
 
  /* Mapping from pdx to node id */
@@ -324,7 +328,7 @@ int __init numa_scan_nodes(u64 start, u64 end)
  }
 
  #ifdef CONFIG_NUMA_EMU
-int numa_fake __initdata = 0;
+static int numa_fake __initdata = 0;
 
  /* Numa emulation */
  static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
@@ -409,3 +413,32 @@ void __init numa_initmem_init(unsigned long

start_pfn, unsigned long end_pfn)

Quoted 9 lines

xen/common/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 417

Quoted 5 lines
xen/common/numa.c +2 -0
      /* architecture specified fallback operations */
      arch_numa_init_failed_fallback();
  }
+
+/* [numa=off] */

The documentation also needs be be updated to reflect that facts this option is not architecture-agnostic.

Ok, I will update the relate document in next version.

Quoted 26 lines
Quoted 12 lines
xen/common/numa.c +12 -0
+static __init int numa_setup(const char *opt)
+{
+    if ( !strncmp(opt,"off",3) )
+        numa_off = true;
+    else if ( !strncmp(opt,"on",2) )
+        numa_off = false;
+#ifdef CONFIG_NUMA_EMU
+    else if ( !strncmp(opt, "fake=", 5) )
+    {
+        numa_off = false;
+        numa_fake = simple_strtoul(opt+5,NULL,0);
+        if ( numa_fake >= MAX_NUMNODES )

xen/common/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 439

Quoted 10 lines
xen/common/numa.c +10 -0
+            numa_fake = MAX_NUMNODES;
+    }
+#endif
+#ifdef CONFIG_ACPI_NUMA
+    else if ( !strncmp(opt,"noacpi",6) )
+    {
+        numa_off = false;
+        acpi_numa = -1;
+    }
+#endif

Looking at this code, I am not quite too sure to understand the difference between between "numa=noacpi" and "numa=off".

Quoted 5 lines

In fact, I am tempted to say this option should disappear because this is odd to have a firmware specific option just for ACPI but not DT. Even if we have one for each, this makes things a bit more complicated for the admin.

Yes, I agree. I would consider a proper way to address it in next version. If x86 maintainers can give some background of these two options would be better.

Quoted 24 lines

xen/include/xen/numa.h extern void numa_initmem_init(unsigned long start_pfn,

Quoted 10 lines
xen/common/numa.c +5 -0
+    else
+        return -EINVAL;
+
+    return 0;
+}
xen/include/xen/numa.h +0 -0
index b9b5d1ad88..c647fef736 100644
@@ -83,7 +83,6 @@ extern void numa_initmem_init(unsigned long start_pfn,

unsigned long end_pfn);

xen/include/xen/numa.h extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn); · line 88

Quoted 8 lines
xen/include/xen/numa.h +0 -1
  extern void numa_set_node(int cpu, nodeid_t node);
  extern int numa_scan_nodes(u64 start, u64 end);
  extern bool numa_off;
-extern int numa_fake;
  extern s8 acpi_numa;
 
  extern void setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end);

Cheers,

--
Julien Grall

#185 lore raw

Re: [XEN RFC PATCH 39/40] xen/x86: move numa_setup to common to support NUMA switch in command line

In reply to Wei Chen
Quoted 71 lines · Wei Chen

Xen x86 has created a command line parameter "numa" as NUMA switch for user to turn on/off NUMA. As device tree based NUMA has been enabled for Arm, this parameter can be reused by Arm. So in this patch, we move this parameter to common.

---
 xen/arch/x86/numa.c    | 34 ----------------------------------
 xen/common/numa.c      | 35 ++++++++++++++++++++++++++++++++++-
 xen/include/xen/numa.h |  1 -
 3 files changed, 34 insertions(+), 36 deletions(-)
xen/arch/x86/numa.c +0 -33
index 8b43be4aa7..380d8ed6fd 100644
@@ -11,7 +11,6 @@
 #include <xen/nodemask.h>
 #include <xen/numa.h>
 #include <xen/keyhandler.h>
-#include <xen/param.h>
 #include <xen/time.h>
 #include <xen/smp.h>
 #include <xen/pfn.h>
@@ -19,9 +18,6 @@
 #include <xen/sched.h>
 #include <xen/softirq.h>
 
-static int numa_setup(const char *s);
-custom_param("numa", numa_setup);
-
 #ifndef Dprintk
 #define Dprintk(x...)
 #endif
@@ -50,35 +46,6 @@ void numa_set_node(int cpu, nodeid_t node)
     cpu_to_node[cpu] = node;
 }
 
-/* [numa=off] */
-static __init int numa_setup(const char *opt)
-{
-    if ( !strncmp(opt,"off",3) )
-        numa_off = true;
-    else if ( !strncmp(opt,"on",2) )
-        numa_off = false;
-#ifdef CONFIG_NUMA_EMU
-    else if ( !strncmp(opt, "fake=", 5) )
-    {
-        numa_off = false;
-        numa_fake = simple_strtoul(opt+5,NULL,0);
-        if ( numa_fake >= MAX_NUMNODES )
-            numa_fake = MAX_NUMNODES;
-    }
-#endif
-#ifdef CONFIG_ACPI_NUMA
-    else if ( !strncmp(opt,"noacpi",6) )
-    {
-        numa_off = false;
-        acpi_numa = -1;
-    }
-#endif
-    else
-        return -EINVAL;
-
-    return 0;
-} 
-
 /*
  * Setup early cpu_to_node.
  *

xen/arch/x86/numa.c static __init int register_numa_trigger(void) · line 290

Quoted 4 lines
xen/arch/x86/numa.c +0 -1
@@ -287,4 +254,3 @@ static __init int register_numa_trigger(void)
     return 0;
 }
 __initcall(register_numa_trigger);
-

spurious change

author
#186 lore raw

RE: [XEN RFC PATCH 39/40] xen/x86: move numa_setup to common to support NUMA switch in command line

Hi Stefano,

Quoted 79 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 9:53
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 39/40] xen/x86: move numa_setup to common to support NUMA switch in command line

Quoted 69 lines · Wei Chen

Xen x86 has created a command line parameter "numa" as NUMA switch for user to turn on/off NUMA. As device tree based NUMA has been enabled for Arm, this parameter can be reused by Arm. So in this patch, we move this parameter to common.

---
 xen/arch/x86/numa.c    | 34 ----------------------------------
 xen/common/numa.c      | 35 ++++++++++++++++++++++++++++++++++-
 xen/include/xen/numa.h |  1 -
 3 files changed, 34 insertions(+), 36 deletions(-)
xen/arch/x86/numa.c +0 -33
index 8b43be4aa7..380d8ed6fd 100644
@@ -11,7 +11,6 @@
 #include <xen/nodemask.h>
 #include <xen/numa.h>
 #include <xen/keyhandler.h>
-#include <xen/param.h>
 #include <xen/time.h>
 #include <xen/smp.h>
 #include <xen/pfn.h>
@@ -19,9 +18,6 @@
 #include <xen/sched.h>
 #include <xen/softirq.h>
 
-static int numa_setup(const char *s);
-custom_param("numa", numa_setup);
-
 #ifndef Dprintk
 #define Dprintk(x...)
 #endif
@@ -50,35 +46,6 @@ void numa_set_node(int cpu, nodeid_t node)
     cpu_to_node[cpu] = node;
 }
 
-/* [numa=off] */
-static __init int numa_setup(const char *opt)
-{
-    if ( !strncmp(opt,"off",3) )
-        numa_off = true;
-    else if ( !strncmp(opt,"on",2) )
-        numa_off = false;
-#ifdef CONFIG_NUMA_EMU
-    else if ( !strncmp(opt, "fake=", 5) )
-    {
-        numa_off = false;
-        numa_fake = simple_strtoul(opt+5,NULL,0);
-        if ( numa_fake >= MAX_NUMNODES )
-            numa_fake = MAX_NUMNODES;
-    }
-#endif
-#ifdef CONFIG_ACPI_NUMA
-    else if ( !strncmp(opt,"noacpi",6) )
-    {
-        numa_off = false;
-        acpi_numa = -1;
-    }
-#endif
-    else
-        return -EINVAL;
-
-    return 0;
-}
-
 /*
  * Setup early cpu_to_node.
Quoted 8 lines

xen/arch/x86/numa.c static __init int register_numa_trigger(void) · line 290

Quoted 6 lines
xen/arch/x86/numa.c +0 -1
  *
@@ -287,4 +254,3 @@ static __init int register_numa_trigger(void)
     return 0;
 }
 __initcall(register_numa_trigger);
-

spurious change

got it.

author
#187 lore raw

RFC PATCH 40/40 xen/x86: move dump_numa info hotkey to common

In reply to Wei Chen

As device tree based NUMA has been enabled for Arm, so not only x86 needs to dump numa info through hotkey, but also Arm can use this hotkey to dump numa info.

In this patch, we move this hotkey to common. Arm can use it to dump its numa information:

(XEN) key 'u' (ascii '75') => dump NUMA info (XEN) 'u' pressed -> dumping numa info (now = 8805901249990) (XEN) NODE0 start->524288 size->520192 free->257673 (XEN) NODE1 start->8912896 size->524288 free->499676 (XEN) CPU0...1 -> NODE0
(XEN) CPU2...3 -> NODE1
(XEN) Memory location of each domain:
(XEN) Domain 0 (total: 262144):
(XEN) Node 0: 262144
(XEN) Node 1: 0

---
 xen/arch/x86/numa.c | 142 -------------------------------------------
 xen/common/numa.c   | 144 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 144 insertions(+), 142 deletions(-)
xen/arch/x86/numa.c +0 -142
index 380d8ed6fd..322801cb17 100644
@@ -112,145 +112,3 @@ int __init arch_meminfo_get_ram_bank_range(int bank,
 
 	return 0;
 }
-
-static void dump_numa(unsigned char key)
-{
-    s_time_t now = NOW();
-    unsigned int i, j, n;
-    struct domain *d;
-    struct page_info *page;
-    unsigned int page_num_node[MAX_NUMNODES];
-    const struct vnuma_info *vnuma;
-
-    printk("'%c' pressed -> dumping numa info (now = %"PRI_stime")\n", key,
-           now);
-
-    for_each_online_node ( i )
-    {
-        paddr_t pa = pfn_to_paddr(node_start_pfn(i) + 1);
-
-        printk("NODE%u start->%lu size->%lu free->%lu\n",
-               i, node_start_pfn(i), node_spanned_pages(i),
-               avail_node_heap_pages(i));
-        /* sanity check phys_to_nid() */
-        if ( phys_to_nid(pa) != i )
-            printk("phys_to_nid(%"PRIpaddr") -> %d should be %u\n",
-                   pa, phys_to_nid(pa), i);
-    }
-
-    j = cpumask_first(&cpu_online_map);
-    n = 0;
-    for_each_online_cpu ( i )
-    {
-        if ( i != j + n || cpu_to_node[j] != cpu_to_node[i] )
-        {
-            if ( n > 1 )
-                printk("CPU%u...%u -> NODE%d\n", j, j + n - 1, cpu_to_node[j]);
-            else
-                printk("CPU%u -> NODE%d\n", j, cpu_to_node[j]);
-            j = i;
-            n = 1;
-        }
-        else
-            ++n;
-    }
-    if ( n > 1 )
-        printk("CPU%u...%u -> NODE%d\n", j, j + n - 1, cpu_to_node[j]);
-    else
-        printk("CPU%u -> NODE%d\n", j, cpu_to_node[j]);
-
-    rcu_read_lock(&domlist_read_lock);
-
-    printk("Memory location of each domain:\n");
-    for_each_domain ( d )
-    {
-        process_pending_softirqs();
-
-        printk("Domain %u (total: %u):\n", d->domain_id, domain_tot_pages(d));
-
-        for_each_online_node ( i )
-            page_num_node[i] = 0;
-
-        spin_lock(&d->page_alloc_lock);
-        page_list_for_each(page, &d->page_list)
-        {
-            i = phys_to_nid(page_to_maddr(page));
-            page_num_node[i]++;
-        }
-        spin_unlock(&d->page_alloc_lock);
-
-        for_each_online_node ( i )
-            printk("    Node %u: %u\n", i, page_num_node[i]);
-
-        if ( !read_trylock(&d->vnuma_rwlock) )
-            continue;
-
-        if ( !d->vnuma )
-        {
-            read_unlock(&d->vnuma_rwlock);
-            continue;
-        }
-
-        vnuma = d->vnuma;
-        printk("     %u vnodes, %u vcpus, guest physical layout:\n",
-               vnuma->nr_vnodes, d->max_vcpus);
-        for ( i = 0; i < vnuma->nr_vnodes; i++ )
-        {
-            unsigned int start_cpu = ~0U;
-
-            if ( vnuma->vnode_to_pnode[i] == NUMA_NO_NODE )
-                printk("       %3u: pnode ???,", i);
-            else
-                printk("       %3u: pnode %3u,", i, vnuma->vnode_to_pnode[i]);
-
-            printk(" vcpus ");
-
-            for ( j = 0; j < d->max_vcpus; j++ )
-            {
-                if ( !(j & 0x3f) )
-                    process_pending_softirqs();
-
-                if ( vnuma->vcpu_to_vnode[j] == i )
-                {
-                    if ( start_cpu == ~0U )
-                    {
-                        printk("%d", j);
-                        start_cpu = j;
-                    }
-                }
-                else if ( start_cpu != ~0U )
-                {
-                    if ( j - 1 != start_cpu )
-                        printk("-%d ", j - 1);
-                    else
-                        printk(" ");
-                    start_cpu = ~0U;
-                }
-            }
-
-            if ( start_cpu != ~0U  && start_cpu != j - 1 )
-                printk("-%d", j - 1);
-
-            printk("\n");
-
-            for ( j = 0; j < vnuma->nr_vmemranges; j++ )
-            {
-                if ( vnuma->vmemrange[j].nid == i )
-                    printk("           %016"PRIx64" - %016"PRIx64"\n",
-                           vnuma->vmemrange[j].start,
-                           vnuma->vmemrange[j].end);
-            }
-        }
-
-        read_unlock(&d->vnuma_rwlock);
-    }
-
-    rcu_read_unlock(&domlist_read_lock);
-}
-
-static __init int register_numa_trigger(void)
-{
-    register_keyhandler('u', dump_numa, "dump NUMA info", 1);
-    return 0;
-}
-__initcall(register_numa_trigger);
xen/common/numa.c +144 -0

154 lines not on this page · raw message

Signature · 1 line
2.25.1
#188 lore raw
In reply to Wei Chen
Quoted 36 lines · Wei Chen

Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0

Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf xen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI in Kconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Move numa_add_cpu_node to common
xen/x86: Move NR_NODE_MEMBLKS macro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Move numa_initmem_init to common
xen/arm: introduce numa_set_node for Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introduce device_tree_numa as a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creating cpu_logical_map xen/x86: decouple nodes_cover_memory with E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: move nodes_cover_memory to common
xen/x86: make acpi_scan_nodes to be neutral
xen: export bad_srat and srat_disabled to extern

Quoted 8 lines

xen: move numa_scan_nodes from x86 to common
xen: enable numa_scan_nodes for device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: move numa_setup to common to support NUMA switch in command
line
xen/x86: move dump_numa info hotkey to common

May I please ask that you follow patch submission guidelines, in that you send patches To: the list and Cc: relevant people. Furthermore I doubt that I need to be on Cc: for all 40 of the patches.

Thanks and regards,
Jan

author
#189 lore raw

Hi Jan,

Quoted 55 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月11日 18:42
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 00/40] Add device tree based NUMA support to Arm64

Quoted 36 lines · Wei Chen

Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0

Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf xen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI in Kconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Move numa_add_cpu_node to common
xen/x86: Move NR_NODE_MEMBLKS macro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Move numa_initmem_init to common
xen/arm: introduce numa_set_node for Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introduce device_tree_numa as a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creating cpu_logical_map xen/x86: decouple nodes_cover_memory with E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: move nodes_cover_memory to common
xen/x86: make acpi_scan_nodes to be neutral
xen: export bad_srat and srat_disabled to extern

Quoted 8 lines

xen: move numa_scan_nodes from x86 to common
xen: enable numa_scan_nodes for device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: move numa_setup to common to support NUMA switch in command
line
xen/x86: move dump_numa info hotkey to common

Quoted 4 lines

May I please ask that you follow patch submission guidelines, in that you send patches To: the list and Cc: relevant people. Furthermore I doubt that I need to be on Cc: for all 40 of the patches.

Thanks for your reminder. Before I sent this series, I had paid sometime to consider CC or TO you, I found you are in the X86 Arch, x86 memory management maintainer lists. And in this patch series, I have done some changes that affects x86, so I added you in TO list. Obviously, my understanding had some mistake. I will add you to CC list in next version.

Quoted 2 lines

Thanks and regards,
Jan

#190 lore raw
Quoted 54 lines · Wei Chen
Quoted 49 lines

From: Jan Beulich <[email protected]>
Sent: 2021年8月11日 18:42

Quoted 36 lines · Wei Chen

Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0

Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf xen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI in Kconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Move numa_add_cpu_node to common
xen/x86: Move NR_NODE_MEMBLKS macro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Move numa_initmem_init to common
xen/arm: introduce numa_set_node for Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introduce device_tree_numa as a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creating cpu_logical_map xen/x86: decouple nodes_cover_memory with E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: move nodes_cover_memory to common
xen/x86: make acpi_scan_nodes to be neutral
xen: export bad_srat and srat_disabled to extern

Quoted 8 lines

xen: move numa_scan_nodes from x86 to common
xen: enable numa_scan_nodes for device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: move numa_setup to common to support NUMA switch in command
line
xen/x86: move dump_numa info hotkey to common

Quoted 4 lines

May I please ask that you follow patch submission guidelines, in that you send patches To: the list and Cc: relevant people. Furthermore I doubt that I need to be on Cc: for all 40 of the patches.

Quoted 6 lines

Thanks for your reminder. Before I sent this series, I had paid sometime to consider CC or TO you, I found you are in the X86 Arch, x86 memory management maintainer lists. And in this patch series, I have done some changes that affects x86, so I added you in TO list. Obviously, my understanding had some mistake. I will add you to CC list in next version.

And then on a patch-by-patch basis please, unless you see a specific need to also Cc my on certain Arm-only patches. Thanks.

Jan

#191 lore raw
In reply to Wei Chen

Hi Wei,

Quoted 11 lines · Wei Chen

Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.

Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.

All the platforms you mention here are servers (so mainly ACPI). However, this series is adding DT support.

Could you outline the long term plan for DT? Is it going to be used on production HW?

Cheers,

Signature · 1 line
Julien Grall
#192 lore raw

Hi Julien,

Quoted 16 lines · Julien Grall

Hi Wei,

Quoted 10 lines · Wei Chen

Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.
Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.

Quoted 3 lines

All the platforms you mention here are servers (so mainly ACPI). However, this series is adding DT support.

Could you outline the long term plan for DT? Is it going to be used on production HW?

Yes we are already and will continue to use this on production HW.
Some embedded hardware will have some usage of NUMA (as some embedded functions do require lots of computing power). We are doing experiments of that right now using those patches.

Cheers
Bertrand

Quoted 5 lines

Cheers,

Signature · 1 line
Julien Grall
#193 lore raw
Quoted 1 line · Bertrand Marquis

Hi Julien,

Hi Bertrand,

Quoted 16 lines
Quoted 16 lines · Julien Grall

Hi Wei,

Quoted 10 lines · Wei Chen

Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.
Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.

Quoted 6 lines
Quoted 3 lines

All the platforms you mention here are servers (so mainly ACPI). However, this series is adding DT support.

Could you outline the long term plan for DT? Is it going to be used on production HW?

Yes we are already and will continue to use this on production HW.
Some embedded hardware will have some usage of NUMA (as some embedded functions do require lots of computing power).

Interesting! Thank you for the clarifications.

Quoted 1 line

We are doing experiments of that right now using those patches.

Cheers,

Signature · 1 line
Julien Grall
#194 lore raw
In reply to Wei Chen

Thanks for the big contribution!

I just wanted to let you know that the series passed all the gitlab-ci build tests without issues.

The runtime tests originally failed due to unrelated problems (there was a Debian testing upgrade that broke Gitlab-CI.) I fix the underlying issue and restarted the failed tests and now they passed.

This is the pipeline:
https://gitlab.com/xen-project/patchew/xen/-/pipelines/351484940

There are still two runtime x86 tests that fail but I don't think the failures are related to your series.

Quoted 120 lines · Wei Chen

Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.

Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.

So now we have a new problem. When Xen is running on these Arm devices, Xen still treat them as single node SMP systems. The NUMA affinity capability of Xen memory allocation and scheduler becomes meaningless. Because they rely on input data that does not reflect real NUMA layout.

Xen still think the access time for all of the memory is the same for all CPUs. However, Xen may allocate memory to a VM from different NUMA nodes with different access speeds. This difference can be amplified in workloads inside VM, causing performance instability and timeouts.

So in this patch series, we implement a set of NUMA API to use device tree to describe the NUMA layout. We reuse most of the code of x86 NUMA to create and maintain the mapping between memory and CPU, create the matrix between any two NUMA nodes. Except ACPI and some x86 specified code, we have moved other code to common. In next stage, when we implement ACPI based NUMA for Arm64, we may move the ACPI NUMA code to common too, but in current stage, we keep it as x86 only.

This patch serires has been tested and booted well on one Arm64 NUMA machine and one HPE x86 NUMA machine.

Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0

Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf xen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI in Kconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Move numa_add_cpu_node to common
xen/x86: Move NR_NODE_MEMBLKS macro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Move numa_initmem_init to common
xen/arm: introduce numa_set_node for Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introduce device_tree_numa as a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creating cpu_logical_map xen/x86: decouple nodes_cover_memory with E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: move nodes_cover_memory to common
xen/x86: make acpi_scan_nodes to be neutral
xen: export bad_srat and srat_disabled to extern
xen: move numa_scan_nodes from x86 to common
xen: enable numa_scan_nodes for device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: move numa_setup to common to support NUMA switch in command
line
xen/x86: move dump_numa info hotkey to common

 tools/libs/util/libxlu_pci.c    |   3 +-
 xen/arch/arm/Kconfig            |  10 +
 xen/arch/arm/Makefile           |   2 +
 xen/arch/arm/arm64/head.S       |   9 +-
 xen/arch/arm/bootfdt.c          |   8 +-
 xen/arch/arm/domain_build.c     |  17 +-
 xen/arch/arm/efi/efi-boot.h     |  25 --
 xen/arch/arm/numa.c             | 162 +++++++++
 xen/arch/arm/numa_device_tree.c | 292 ++++++++++++++++
 xen/arch/arm/platform.c         |   4 +-
 xen/arch/arm/setup.c            |  14 +
 xen/arch/arm/smpboot.c          |  37 +-
 xen/arch/x86/Kconfig            |   2 +-
 xen/arch/x86/numa.c             | 421 +----------------------
 xen/arch/x86/srat.c             | 147 +-------
 xen/common/Kconfig              |   3 +
 xen/common/Makefile             |   1 +
 xen/common/libfdt/fdt_ro.c      |  15 +
 xen/common/numa.c               | 588 ++++++++++++++++++++++++++++++++
 xen/common/page_alloc.c         |   2 +-
 xen/drivers/acpi/Kconfig        |   3 +-
 xen/drivers/acpi/Makefile       |   2 +-
 xen/include/asm-arm/numa.h      |  33 ++
 xen/include/asm-arm/setup.h     |   6 +
 xen/include/asm-x86/acpi.h      |   4 -
 xen/include/asm-x86/config.h    |   1 -
 xen/include/asm-x86/numa.h      |  65 +---
 xen/include/asm-x86/setup.h     |   1 -
 xen/include/xen/libfdt/libfdt.h |  25 ++
 xen/include/xen/nodemask.h      |   2 +
 xen/include/xen/numa.h          |  80 +++++
 31 files changed, 1325 insertions(+), 659 deletions(-)
 create mode 100644 xen/arch/arm/numa.c
 create mode 100644 xen/arch/arm/numa_device_tree.c
 create mode 100644 xen/common/numa.c
Signature · 1 line
2.25.1
author
#195 lore raw

Hi Stefano,

Quoted 19 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月26日 8:09
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]
Subject: Re: [XEN RFC PATCH 00/40] Add device tree based NUMA support to Arm64

Thanks for the big contribution!

I just wanted to let you know that the series passed all the gitlab-ci build tests without issues.

The runtime tests originally failed due to unrelated problems (there was a Debian testing upgrade that broke Gitlab-CI.) I fix the underlying issue and restarted the failed tests and now they passed.

Quoted 7 lines

This is the pipeline:
https://gitlab.com/xen-project/patchew/xen/-/pipelines/351484940

There are still two runtime x86 tests that fail but I don't think the failures are related to your series.

Thanks for testing this series : )

Quoted 121 lines · Wei Chen
Quoted 120 lines

Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.

Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.

So now we have a new problem. When Xen is running on these Arm devices, Xen still treat them as single node SMP systems. The NUMA affinity capability of Xen memory allocation and scheduler becomes meaningless. Because they rely on input data that does not reflect real NUMA layout.

Xen still think the access time for all of the memory is the same for all CPUs. However, Xen may allocate memory to a VM from different NUMA nodes with different access speeds. This difference can be amplified in workloads inside VM, causing performance instability and timeouts.

So in this patch series, we implement a set of NUMA API to use device tree to describe the NUMA layout. We reuse most of the code of x86 NUMA to create and maintain the mapping between memory and CPU, create the matrix between any two NUMA nodes. Except ACPI and some x86 specified code, we have moved other code to common. In next stage, when we implement ACPI based NUMA for Arm64, we may move the ACPI NUMA code to common too, but in current stage, we keep it as x86 only.

This patch serires has been tested and booted well on one Arm64 NUMA machine and one HPE x86 NUMA machine.

Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0

Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf xen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI in Kconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Move numa_add_cpu_node to common
xen/x86: Move NR_NODE_MEMBLKS macro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Move numa_initmem_init to common
xen/arm: introduce numa_set_node for Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introduce device_tree_numa as a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creating cpu_logical_map xen/x86: decouple nodes_cover_memory with E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: move nodes_cover_memory to common
xen/x86: make acpi_scan_nodes to be neutral
xen: export bad_srat and srat_disabled to extern
xen: move numa_scan_nodes from x86 to common
xen: enable numa_scan_nodes for device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: move numa_setup to common to support NUMA switch in command
line
xen/x86: move dump_numa info hotkey to common

 tools/libs/util/libxlu_pci.c    |   3 +-
 xen/arch/arm/Kconfig            |  10 +
 xen/arch/arm/Makefile           |   2 +
 xen/arch/arm/arm64/head.S       |   9 +-
 xen/arch/arm/bootfdt.c          |   8 +-
 xen/arch/arm/domain_build.c     |  17 +-
 xen/arch/arm/efi/efi-boot.h     |  25 --
 xen/arch/arm/numa.c             | 162 +++++++++
 xen/arch/arm/numa_device_tree.c | 292 ++++++++++++++++
 xen/arch/arm/platform.c         |   4 +-
 xen/arch/arm/setup.c            |  14 +
 xen/arch/arm/smpboot.c          |  37 +-
 xen/arch/x86/Kconfig            |   2 +-
 xen/arch/x86/numa.c             | 421 +----------------------
 xen/arch/x86/srat.c             | 147 +-------
 xen/common/Kconfig              |   3 +
 xen/common/Makefile             |   1 +
 xen/common/libfdt/fdt_ro.c      |  15 +
 xen/common/numa.c               | 588 ++++++++++++++++++++++++++++++++
 xen/common/page_alloc.c         |   2 +-
 xen/drivers/acpi/Kconfig        |   3 +-
 xen/drivers/acpi/Makefile       |   2 +-
 xen/include/asm-arm/numa.h      |  33 ++
 xen/include/asm-arm/setup.h     |   6 +
 xen/include/asm-x86/acpi.h      |   4 -
 xen/include/asm-x86/config.h    |   1 -
 xen/include/asm-x86/numa.h      |  65 +---
 xen/include/asm-x86/setup.h     |   1 -
 xen/include/xen/libfdt/libfdt.h |  25 ++
 xen/include/xen/nodemask.h      |   2 +
 xen/include/xen/numa.h          |  80 +++++
 31 files changed, 1325 insertions(+), 659 deletions(-)
 create mode 100644 xen/arch/arm/numa.c
 create mode 100644 xen/arch/arm/numa_device_tree.c
 create mode 100644 xen/common/numa.c

--
2.25.1