Add device tree based NUMA support to Arm64
RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map
In reply to Wei ChenQuoted 157 lines · Wei Chen
Hi Stefano,
Quoted 95 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 8:48
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance mapReview
Quoted 10 lines · Wei Chen
A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a new helper to parse this distance map.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 67 +++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cxen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt,
Quoted 4 lines
index
bbe081dcd1..6e0d1d3d9f100644xen/arch/arm/numa_device_tree.c +0
-0@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void *fdt,int node,
Quoted 45 lines
xen/arch/arm/numa_device_tree.c +42
-0return 0 ; } + +/* Parse NUMA distance map v1 */ +int __init +device_tree_parse_numa_distance_map_v1 (const void *fdt,int node) +{ +const struct fdt_property *prop; +const __be32 *matrix; +int entry_count, len, i; + +printk (XENLOG_INFO "NUMA: parsing numa-distance-map\n" ); + + prop =fdt_get_property (fdt, node,"distance-matrix" , &len); +if ( !prop ) + { +printk (XENLOG_WARNING +"NUMA: No distance-matrix property in distance-map\n" ); + +return -EINVAL ; + } + +if ( len %sizeof (uint32_t ) !=0 ) + { +printk (XENLOG_WARNING +"distance-matrix in node is not a multiple of u32\n" ); +return -EINVAL ; + } + + entry_count = len /sizeof (uint32_t ); +if ( entry_count <=0 ) + { +printk (XENLOG_WARNING "NUMA: Invalid distance-matrix\n" ); + +return -EINVAL ; + } + + matrix = (const __be32 *)prop->data; +for ( i =0 ; i +2 < entry_count; i +=3 ) + { +uint32_t from, to, distance; + + from =dt_read_number (matrix,1 ); + matrix++;xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ to =dt_read_number (matrix,1 ); + matrix++; + distance =dt_read_number (matrix,1 ); + matrix++; + + if ( (from == to && distance != NUMA_LOCAL_DISTANCE) || + (from != to && distance <= NUMA_LOCAL_DISTANCE) ) + { + printk(XENLOG_WARNING + "Invalid nodes' distance from node#%d to node#%d= %d\n",
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5
-0+ from, to, distance); +return -EINVAL ; + } + + printk(XENLOG_INFO "NUMA: distance from node#%d to node#%d= %d\n",
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 265
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6
-0+ from, to, distance); +numa_set_distance (from, to, distance); + +/* Set default distance of node B->A same as A->B */ +if (to > from) +numa_set_distance (to, from, distance);Quoted 9 lines
I am a bit unsure about this last 2 lines: why calling
numa_set_distancein the opposite direction only when to > from? Wouldn't it be OK to always do both:
numa_set_distance(from, to, distance);numa_set_distance(to, from, distance);?
I borrowed this code from Linux, but here is my understanding:
First, I read some notes in
Documentation/devicetree/bindings/numa.txt
1. Each entry represents distance from first node to second node. The distances are equal in either direction.
2. distance-matrix should have entries in lexicographical ascending order of nodes.Here is an example of distance-map node in DTB:
Sample#1, full list:distance-map { compatible = "numa-distance-map-v1"; distance-matrix = <0 0 10>, <0 1 20>, <0 2 40>, <0 3 20>, <1 0 20>, <1 1 10>, <1 2 20>, <1 3 40>, <2 0 40>, <2 1 20>, <2 2 10>, <2 3 20>, <3 0 20>, <3 1 40>, <3 2 20>, <3 3 10>; };Call
numa_set_distancewhen "to > from" will prevent Xen to callnuma_set_distance(0, 1, 20)again when it's setting distance for <1 0 20>. But,numa_set_distance(1, 0, 20)will be call twice.Normally, distance-map node will be optimized in following sample#2, all redundant entries are removed:
Sample#2, partial list:distance-map { compatible = "numa-distance-map-v1"; distance-matrix = <0 0 10>, <0 1 20>, <0 2 40>, <0 3 20>, <1 1 10>, <1 2 20>, <1 3 40>, <2 2 10>, <2 3 20>, <3 3 10>; };
Quoted 3 lines
There is not any "from > to" entry in the map. But using this partial map still can set all distances for all pairs. And
numa_set_distance(1, 0, 20)will be only once.
I see. I can't find in Documentation/devicetree/bindings/numa.txt where it says that "from > to" nodes can be omitted. If it is not written down, then somebody could easily optimize it the opposite way:
distance-matrix = <0 0 10>,
<1 0 20>, <2 0 40>, <3 0 20>, <1 1 10>, <2 1 20>, <3 1 40>, <2 2 10>, <3 2 20>, <3 3 10>;
I think the code in Xen should be resilient and able to cope with a device tree like the one you wrote or the one I wrote. From a code perspective, it should be very easy to do. If nothing else it would make Xen more resilient against buggy firmware.
Quoted 21 lines
Quoted 5 lines
But in any case, I have a different suggestion. The binding states that "distances are equal in either direction". Also it has an example where only one direction is expressed unfortunately (at the end of the document).
Oh, I should see this comment first, then I will not post above comment : )
Quoted 11 lines
So my suggestion is to parse it as follows:
- call
numa_set_distancejust once from
device_tree_parse_numa_distance_map_v1- in
numa_set_distance:
- set node_distance_map[from][to] = distance;
- check node_distance_map[to][from]
- if unset, node_distance_map[to][from] = distance;
- if already set to the same value, return success;
- if already set to a different value, return error;
Quoted 5 lines
I don't really like this implementation. I want the behavior of
numa_set_distancejust like the function name, do not include implicit operations. Otherwise, except the user read this function implementation before he use it, he probably doesn't know this function has done so many things.
You can leave numa_set_distance as-is without any implicit operations.
In that case, just call numa_set_distance twice from numa_set_distance for both from/to and to/from. numa_set_distance could return error is the entry was already set to a different value or success otherwise (also in the case it was already set to the same value). This would allow Xen to cope with both scenarios above.
RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map
Hi Stefano,
Quoted 188 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Stefano Stabellini
Sent: 2021年9月1日 5:36
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance mapQuoted 7 lines · Wei Chen
Hi Stefano,
Quoted 5 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 8:48
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];Quoted 2 lines
Quoted 2 lines
[email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper toparse
Quoted 10 lines
Quoted 10 lines
device tree NUMA distance map
Quoted 7 lines · Wei Chen
A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a new helper to parse this distance map.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 67+++++++++++++++++++++++++++++++++
Quoted 8 lines
Quoted 8 lines
Review
Quoted 3 lines
1 file changed, 67 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cxen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void
Quoted 4 lines
index
bbe081dcd1..6e0d1d3d9f100644xen/arch/arm/numa_device_tree.c +0
-0@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void*fdt,
Quoted 12 lines
Quoted 12 lines
int node,
Quoted 11 lines
xen/arch/arm/numa_device_tree.c +8
-0return 0 ; } + +/* Parse NUMA distance map v1 */ +int __init +device_tree_parse_numa_distance_map_v1 (const void *fdt,int node) +{ +const struct fdt_property *prop; +const __be32 *matrix; +int entry_count, len, i;Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+ +printk (XENLOG_INFO "NUMA: parsing numa-distance-map\n" ); + + prop =fdt_get_property (fdt, node,"distance-matrix" , &len); + if ( !prop ) + { + printk(XENLOG_WARNING + "NUMA: No distance-matrix property in distance-map\n");
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+ +return -EINVAL ; + } + + if ( len % sizeof(uint32_t) != 0 ) + { + printk(XENLOG_WARNING + "distance-matrix in node is not a multiple ofu32\n");
Quoted 35 lines
Quoted 35 lines
Quoted 18 lines
xen/arch/arm/numa_device_tree.c +18
-0+return -EINVAL ; + } + + entry_count = len /sizeof (uint32_t ); +if ( entry_count <=0 ) + { +printk (XENLOG_WARNING "NUMA: Invalid distance-matrix\n" ); + +return -EINVAL ; + } + + matrix = (const __be32 *)prop->data; +for ( i =0 ; i +2 < entry_count; i +=3 ) + { +uint32_t from, to, distance; + + from =dt_read_number (matrix,1 ); + matrix++;xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ to =dt_read_number (matrix,1 ); + matrix++; + distance =dt_read_number (matrix,1 ); + matrix++; + + if ( (from == to && distance != NUMA_LOCAL_DISTANCE) || + (from != to && distance <= NUMA_LOCAL_DISTANCE) ) + { + printk(XENLOG_WARNING + "Invalid nodes' distance from node#%d to node#%d= %d\n",
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5
-0+ from, to, distance); +return -EINVAL ; + } + + printk(XENLOG_INFO "NUMA: distance from node#%d to node#%d= %d\n",
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 265
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6
-0+ from, to, distance); +numa_set_distance (from, to, distance); + +/* Set default distance of node B->A same as A->B */ +if (to > from) +numa_set_distance (to, from, distance);I am a bit unsure about this last 2 lines: why calling
numa_set_distanceQuoted 38 lines
Quoted 8 lines
in the opposite direction only when to > from? Wouldn't it be OK to always do both:
numa_set_distance(from, to, distance);numa_set_distance(to, from, distance);?
I borrowed this code from Linux, but here is my understanding:
First, I read some notes in
Documentation/devicetree/bindings/numa.txt
1. Each entry represents distance from first node to second node. The distances are equal in either direction.
2. distance-matrix should have entries in lexicographical ascending order of nodes.Here is an example of distance-map node in DTB:
Sample#1, full list:distance-map { compatible = "numa-distance-map-v1"; distance-matrix = <0 0 10>, <0 1 20>, <0 2 40>, <0 3 20>, <1 0 20>, <1 1 10>, <1 2 20>, <1 3 40>, <2 0 40>, <2 1 20>, <2 2 10>, <2 3 20>, <3 0 20>, <3 1 40>, <3 2 20>, <3 3 10>; };Quoted 2 lines
Call
numa_set_distancewhen "to > from" will prevent Xen to callnuma_set_distance(0, 1, 20)again when it's setting distance for <1 020>.
Quoted 19 lines
But,
numa_set_distance(1, 0, 20)will be call twice.Normally, distance-map node will be optimized in following sample#2, all redundant entries are removed:
Sample#2, partial list:distance-map { compatible = "numa-distance-map-v1"; distance-matrix = <0 0 10>, <0 1 20>, <0 2 40>, <0 3 20>, <1 1 10>, <1 2 20>, <1 3 40>, <2 2 10>, <2 3 20>, <3 3 10>; };Quoted 1 line
There is not any "from > to" entry in the map. But using this partial
map
Quoted 1 line
still can set all distances for all pairs. And numa_set_distance(1, 0,
20)
Quoted 1 line
will be only once.
I see. I can't find in
Documentation/devicetree/bindings/numa.txtwhere it says that "from > to" nodes can be omitted. If it is not written down, then somebody could easily optimize it the opposite way:
Review
Quoted 10 lines
(quoted diff) +0
-0distance-matrix = <0 0 10>, <1 0 20>, <2 0 40>, <3 0 20>, <1 1 10>, <2 1 20>, <3 1 40>, <2 2 10>, <3 2 20>, <3 3 10>;
Yes, you're right. Spec doesn't say opposite way is unallowed.
Quoted 6 lines
I think the code in Xen should be resilient and able to cope with a device tree like the one you wrote or the one I wrote. From a code perspective, it should be very easy to do. If nothing else it would make Xen more resilient against buggy firmware.
I don't disagree with that.
Quoted 29 lines
Quoted 1 line
Quoted 1 line
But in any case, I have a different suggestion. The binding states
that
Quoted 1 line
Quoted 1 line
"distances are equal in either direction". Also it has an example
where
Quoted 19 lines
Quoted 3 lines
only one direction is expressed unfortunately (at the end of the document).
Oh, I should see this comment first, then I will not post above comment : )
Quoted 11 lines
So my suggestion is to parse it as follows:
- call
numa_set_distancejust once from
device_tree_parse_numa_distance_map_v1- in
numa_set_distance:
- set node_distance_map[from][to] = distance;
- check node_distance_map[to][from]
- if unset, node_distance_map[to][from] = distance;
- if already set to the same value, return success;
- if already set to a different value, return error;Quoted 5 lines
I don't really like this implementation. I want the behavior of
numa_set_distancejust like the function name, do not include implicit operations. Otherwise, except the user read this function implementation before he use it, he probably doesn't know this function has done so many things.
Quoted 4 lines
You can leave
numa_set_distanceas-is without any implicit operations.In that case, just call
numa_set_distancetwice fromnuma_set_distancefor both from/to and to/from.numa_set_distancecould return error is
I am OK for the first sentence. But...
Quoted 2 lines
the entry was already set to a different value or success otherwise (also in the case it was already set to the same value). This would
... I prefer not to check the previous value. Subsequent numa_set_distance call will override previous calls. Keep numa_set_distance as simple as it can. And when you pass new data to numa_set_distance, it doesn't know whether the previous data was correct or the new data is correct. Only caller may have known.
Quoted 1 line
allow Xen to cope with both scenarios above.
RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map
Quoted 248 lines · Wei Chen
Hi Stefano,
Quoted 188 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Stefano Stabellini
Sent: 2021年9月1日 5:36
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance mapQuoted 7 lines · Wei Chen
Hi Stefano,
Quoted 5 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 8:48
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];Quoted 2 lines
Quoted 2 lines
[email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper toparse
Quoted 10 lines
Quoted 10 lines
device tree NUMA distance map
Quoted 7 lines · Wei Chen
A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a new helper to parse this distance map.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 67+++++++++++++++++++++++++++++++++
Quoted 8 lines
Quoted 8 lines
Review
Quoted 3 lines
1 file changed, 67 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cxen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void
Quoted 4 lines
index
bbe081dcd1..6e0d1d3d9f100644xen/arch/arm/numa_device_tree.c +0
-0@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void*fdt,
Quoted 12 lines
Quoted 12 lines
int node,
Quoted 11 lines
xen/arch/arm/numa_device_tree.c +8
-0return 0 ; } + +/* Parse NUMA distance map v1 */ +int __init +device_tree_parse_numa_distance_map_v1 (const void *fdt,int node) +{ +const struct fdt_property *prop; +const __be32 *matrix; +int entry_count, len, i;Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+ +printk (XENLOG_INFO "NUMA: parsing numa-distance-map\n" ); + + prop =fdt_get_property (fdt, node,"distance-matrix" , &len); + if ( !prop ) + { + printk(XENLOG_WARNING + "NUMA: No distance-matrix property in distance-map\n");
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+ +return -EINVAL ; + } + + if ( len % sizeof(uint32_t) != 0 ) + { + printk(XENLOG_WARNING + "distance-matrix in node is not a multiple ofu32\n");
Quoted 35 lines
Quoted 35 lines
Quoted 18 lines
xen/arch/arm/numa_device_tree.c +18
-0+return -EINVAL ; + } + + entry_count = len /sizeof (uint32_t ); +if ( entry_count <=0 ) + { +printk (XENLOG_WARNING "NUMA: Invalid distance-matrix\n" ); + +return -EINVAL ; + } + + matrix = (const __be32 *)prop->data; +for ( i =0 ; i +2 < entry_count; i +=3 ) + { +uint32_t from, to, distance; + + from =dt_read_number (matrix,1 ); + matrix++;xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ to =dt_read_number (matrix,1 ); + matrix++; + distance =dt_read_number (matrix,1 ); + matrix++; + + if ( (from == to && distance != NUMA_LOCAL_DISTANCE) || + (from != to && distance <= NUMA_LOCAL_DISTANCE) ) + { + printk(XENLOG_WARNING + "Invalid nodes' distance from node#%d to node#%d= %d\n",
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5
-0+ from, to, distance); +return -EINVAL ; + } + + printk(XENLOG_INFO "NUMA: distance from node#%d to node#%d= %d\n",
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 265
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6
-0+ from, to, distance); +numa_set_distance (from, to, distance); + +/* Set default distance of node B->A same as A->B */ +if (to > from) +numa_set_distance (to, from, distance);I am a bit unsure about this last 2 lines: why calling
numa_set_distanceQuoted 38 lines
Quoted 8 lines
in the opposite direction only when to > from? Wouldn't it be OK to always do both:
numa_set_distance(from, to, distance);numa_set_distance(to, from, distance);?
I borrowed this code from Linux, but here is my understanding:
First, I read some notes in
Documentation/devicetree/bindings/numa.txt
1. Each entry represents distance from first node to second node. The distances are equal in either direction.
2. distance-matrix should have entries in lexicographical ascending order of nodes.Here is an example of distance-map node in DTB:
Sample#1, full list:distance-map { compatible = "numa-distance-map-v1"; distance-matrix = <0 0 10>, <0 1 20>, <0 2 40>, <0 3 20>, <1 0 20>, <1 1 10>, <1 2 20>, <1 3 40>, <2 0 40>, <2 1 20>, <2 2 10>, <2 3 20>, <3 0 20>, <3 1 40>, <3 2 20>, <3 3 10>; };Quoted 2 lines
Call
numa_set_distancewhen "to > from" will prevent Xen to callnuma_set_distance(0, 1, 20)again when it's setting distance for <1 020>.
Quoted 19 lines
But,
numa_set_distance(1, 0, 20)will be call twice.Normally, distance-map node will be optimized in following sample#2, all redundant entries are removed:
Sample#2, partial list:distance-map { compatible = "numa-distance-map-v1"; distance-matrix = <0 0 10>, <0 1 20>, <0 2 40>, <0 3 20>, <1 1 10>, <1 2 20>, <1 3 40>, <2 2 10>, <2 3 20>, <3 3 10>; };Quoted 1 line
There is not any "from > to" entry in the map. But using this partial
map
Quoted 1 line
still can set all distances for all pairs. And numa_set_distance(1, 0,
20)
Quoted 1 line
will be only once.
I see. I can't find in
Documentation/devicetree/bindings/numa.txtwhere it says that "from > to" nodes can be omitted. If it is not written down, then somebody could easily optimize it the opposite way:Review
Quoted 10 lines
(quoted diff) +0
-0distance-matrix = <0 0 10>, <1 0 20>, <2 0 40>, <3 0 20>, <1 1 10>, <2 1 20>, <3 1 40>, <2 2 10>, <3 2 20>, <3 3 10>;Yes, you're right. Spec doesn't say opposite way is unallowed.
Quoted 6 lines
I think the code in Xen should be resilient and able to cope with a device tree like the one you wrote or the one I wrote. From a code perspective, it should be very easy to do. If nothing else it would make Xen more resilient against buggy firmware.
I don't disagree with that.
Quoted 29 lines
Quoted 1 line
Quoted 1 line
But in any case, I have a different suggestion. The binding states
that
Quoted 1 line
Quoted 1 line
"distances are equal in either direction". Also it has an example
where
Quoted 19 lines
Quoted 3 lines
only one direction is expressed unfortunately (at the end of the document).
Oh, I should see this comment first, then I will not post above comment : )
Quoted 11 lines
So my suggestion is to parse it as follows:
- call
numa_set_distancejust once from
device_tree_parse_numa_distance_map_v1- in
numa_set_distance:
- set node_distance_map[from][to] = distance;
- check node_distance_map[to][from]
- if unset, node_distance_map[to][from] = distance;
- if already set to the same value, return success;
- if already set to a different value, return error;Quoted 5 lines
I don't really like this implementation. I want the behavior of
numa_set_distancejust like the function name, do not include implicit operations. Otherwise, except the user read this function implementation before he use it, he probably doesn't know this function has done so many things.Quoted 4 lines
You can leave
numa_set_distanceas-is without any implicit operations.In that case, just call
numa_set_distancetwice fromnuma_set_distancefor both from/to and to/from.numa_set_distancecould return error isI am OK for the first sentence. But...
Quoted 8 lines
Quoted 2 lines
the entry was already set to a different value or success otherwise (also in the case it was already set to the same value). This would
... I prefer not to check the previous value. Subsequent
numa_set_distancecall will override previous calls. Keepnuma_set_distanceas simple as it can. And when you pass new data tonuma_set_distance, it doesn't know whether the previous data was correct or the new data is correct. Only caller may have known.
That might be OK but if not numa_set_distance then somebody else needs to check against overwriting previous values. That is to be able to spot bad device tree cases like:
0 1 20
1 0 40
RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map
Hi Stefano,
Quoted 282 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月2日 0:22
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance mapQuoted 11 lines · Wei Chen
Hi Stefano,
Quoted 9 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Stefano Stabellini
Sent: 2021年9月1日 5:36
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper toparse
Quoted 25 lines
Quoted 25 lines
device tree NUMA distance map
Quoted 7 lines · Wei Chen
Hi Stefano,
Quoted 5 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 8:48
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];Quoted 2 lines
Quoted 2 lines
[email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper toparse
Quoted 10 lines
Quoted 10 lines
device tree NUMA distance map
Quoted 7 lines · Wei Chen
A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a new helper to parse this distance map.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 67+++++++++++++++++++++++++++++++++
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Review
Quoted 3 lines
1 file changed, 67 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cxen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const
Quoted 4 lines
index
bbe081dcd1..6e0d1d3d9f100644xen/arch/arm/numa_device_tree.c +0
-0@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(constvoid
Quoted 9 lines
Quoted 9 lines
*fdt,
Quoted 8 lines
Quoted 8 lines
int node,
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 7 lines
xen/arch/arm/numa_device_tree.c +4
-0return 0 ; } + +/* Parse NUMA distance map v1 */ +int __init +device_tree_parse_numa_distance_map_v1(const void *fdt, intnode)
Quoted 22 lines
Quoted 22 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4
-0+{ +const struct fdt_property *prop; +const __be32 *matrix; +int entry_count, len, i;Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+ +printk (XENLOG_INFO "NUMA: parsing numa-distance-map\n" ); + + prop =fdt_get_property (fdt, node,"distance-matrix" , &len); + if ( !prop ) + { + printk(XENLOG_WARNING + "NUMA: No distance-matrix property in distance-map\n");
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+ +return -EINVAL ; + } + + if ( len % sizeof(uint32_t) != 0 ) + { + printk(XENLOG_WARNING + "distance-matrix in node is not a multiple ofu32\n");
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 7 lines
xen/arch/arm/numa_device_tree.c +7
-0+return -EINVAL ; + } + + entry_count = len /sizeof (uint32_t ); + if ( entry_count <= 0 ) + { + printk(XENLOG_WARNING "NUMA: Invalid distance-matrix\n");
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
xen/arch/arm/numa_device_tree.c +13
-0+ +return -EINVAL ; + } + + matrix = (const __be32 *)prop->data; +for ( i =0 ; i +2 < entry_count; i +=3 ) + { +uint32_t from, to, distance; + + from =dt_read_number (matrix,1 ); + matrix++; + to =dt_read_number (matrix,1 ); + matrix++;Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+ distance =dt_read_number (matrix,1 ); + matrix++; + + if ( (from == to && distance != NUMA_LOCAL_DISTANCE) || + (from != to && distance <= NUMA_LOCAL_DISTANCE) ) + { + printk(XENLOG_WARNING + "Invalid nodes' distance from node#%d tonode#%d
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
= %d\n",
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5
-0+ from, to, distance); +return -EINVAL ; + } + + printk(XENLOG_INFO "NUMA: distance from node#%d tonode#%d
Quoted 11 lines
Quoted 11 lines
Quoted 9 lines
Quoted 9 lines
= %d\n",
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 265
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6
-0+ from, to, distance); +numa_set_distance (from, to, distance); + +/* Set default distance of node B->A same as A->B */ +if (to > from) +numa_set_distance (to, from, distance);I am a bit unsure about this last 2 lines: why calling
numa_set_distanceQuoted 1 line
Quoted 1 line
in the opposite direction only when to > from? Wouldn't it be OK
to
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
always do both:
numa_set_distance(from, to, distance);numa_set_distance(to, from, distance);?
I borrowed this code from Linux, but here is my understanding:
First, I read some notes in
Documentation/devicetree/bindings/numa.txtQuoted 27 lines
Quoted 27 lines
Quoted 27 lines
1. Each entry represents distance from first node to second node. The distances are equal in either direction.
2. distance-matrix should have entries in lexicographical ascending order of nodes.Here is an example of distance-map node in DTB:
Sample#1, full list:distance-map { compatible = "numa-distance-map-v1"; distance-matrix = <0 0 10>, <0 1 20>, <0 2 40>, <0 3 20>, <1 0 20>, <1 1 10>, <1 2 20>, <1 3 40>, <2 0 40>, <2 1 20>, <2 2 10>, <2 3 20>, <3 0 20>, <3 1 40>, <3 2 20>, <3 3 10>; };Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Call
numa_set_distancewhen "to > from" will prevent Xen to callnuma_set_distance(0, 1, 20)again when it's setting distance for <10
Quoted 20 lines
Quoted 20 lines
20>.
Quoted 19 lines
But,
numa_set_distance(1, 0, 20)will be call twice.Normally, distance-map node will be optimized in following sample#2, all redundant entries are removed:
Sample#2, partial list:distance-map { compatible = "numa-distance-map-v1"; distance-matrix = <0 0 10>, <0 1 20>, <0 2 40>, <0 3 20>, <1 1 10>, <1 2 20>, <1 3 40>, <2 2 10>, <2 3 20>, <3 3 10>; };Quoted 1 line
Quoted 1 line
Quoted 1 line
There is not any "from > to" entry in the map. But using this
partial
Quoted 2 lines
Quoted 2 lines
map
Quoted 1 line
still can set all distances for all pairs. And numa_set_distance(1,
0,
Quoted 4 lines
Quoted 4 lines
20)
Quoted 1 line
will be only once.
I see. I can't find in
Documentation/devicetree/bindings/numa.txtwhere
Quoted 15 lines
Quoted 4 lines
it says that "from > to" nodes can be omitted. If it is not written down, then somebody could easily optimize it the opposite way:
Review
Quoted 10 lines
(quoted diff) +0
-0distance-matrix = <0 0 10>, <1 0 20>, <2 0 40>, <3 0 20>, <1 1 10>, <2 1 20>, <3 1 40>, <2 2 10>, <3 2 20>, <3 3 10>;Quoted 5 lines
Yes, you're right. Spec doesn't say opposite way is unallowed.
Quoted 3 lines
I think the code in Xen should be resilient and able to cope with a device tree like the one you wrote or the one I wrote. From a code perspective, it should be very easy to do. If nothing else it would
make
Quoted 40 lines
Quoted 3 lines
Xen more resilient against buggy firmware.
I don't disagree with that.
Quoted 29 lines
Quoted 1 line
Quoted 1 line
But in any case, I have a different suggestion. The binding states
that
Quoted 1 line
Quoted 1 line
"distances are equal in either direction". Also it has an example
where
Quoted 19 lines
Quoted 3 lines
only one direction is expressed unfortunately (at the end of the document).
Oh, I should see this comment first, then I will not post above comment : )
Quoted 11 lines
So my suggestion is to parse it as follows:
- call
numa_set_distancejust once from
device_tree_parse_numa_distance_map_v1- in
numa_set_distance:
- set node_distance_map[from][to] = distance;
- check node_distance_map[to][from]
- if unset, node_distance_map[to][from] = distance;
- if already set to the same value, return success;
- if already set to a different value, return error;Quoted 5 lines
I don't really like this implementation. I want the behavior of
numa_set_distancejust like the function name, do not include implicit operations. Otherwise, except the user read this function implementation before he use it, he probably doesn't know this function has done so many things.Quoted 4 lines
You can leave
numa_set_distanceas-is without any implicit operations.In that case, just call
numa_set_distancetwice fromnuma_set_distancefor both from/to and to/from.numa_set_distancecould return error isQuoted 6 lines
I am OK for the first sentence. But...
Quoted 2 lines
the entry was already set to a different value or success otherwise (also in the case it was already set to the same value). This would
... I prefer not to check the previous value. Subsequent
numa_set_distanceQuoted 4 lines
call will override previous calls. Keep
numa_set_distanceas simple as it can. And when you pass new data tonuma_set_distance, it doesn't know whether the previous data was correct or the new data is correct. Only caller may have known.
Quoted 6 lines
That might be OK but if not
numa_set_distancethen somebody else needs to check against overwriting previous values. That is to be able to spot bad device tree cases like:0 1 20
1 0 40
How about we check it still in NUMA distance parse function? Before setting the numa_set_distance for one pair nodes (e.g. a -> b), we can get its opposite way distance first.
distance_b_a = __node_distance(b, a); ==> get opposite way distance. if (distance_b_a == 0) ==> opposite way distance has not been set
{
numa_set_distance(a, b, 20); ==> set both
numa_set_distance(b, a, 20)
} else {
if (distance_b_a == 20) ==> opposite way distance has been set
numa_set_distance(a, b, 20); ==> set this way only
else ===> opposite way distance has been set, but is unmatched
// What can we do here?
Panic the system? or Just warning users? Or choose the bigger
distance for both ways?
And distance_b_a == NUMA_NO_DISTANCE would be a special case
here.
}RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map
Quoted 298 lines · Wei Chen
Hi Stefano,
Quoted 282 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月2日 0:22
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance mapQuoted 11 lines · Wei Chen
Hi Stefano,
Quoted 9 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Stefano Stabellini
Sent: 2021年9月1日 5:36
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper toparse
Quoted 25 lines
Quoted 25 lines
device tree NUMA distance map
Quoted 7 lines · Wei Chen
Hi Stefano,
Quoted 5 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 8:48
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];Quoted 2 lines
Quoted 2 lines
[email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper toparse
Quoted 10 lines
Quoted 10 lines
device tree NUMA distance map
Quoted 7 lines · Wei Chen
A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a new helper to parse this distance map.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 67+++++++++++++++++++++++++++++++++
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Review
Quoted 3 lines
1 file changed, 67 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cxen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const
Quoted 4 lines
index
bbe081dcd1..6e0d1d3d9f100644xen/arch/arm/numa_device_tree.c +0
-0@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(constvoid
Quoted 9 lines
Quoted 9 lines
*fdt,
Quoted 8 lines
Quoted 8 lines
int node,
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const · line 205
Quoted 7 lines
xen/arch/arm/numa_device_tree.c +4
-0return 0 ; } + +/* Parse NUMA distance map v1 */ +int __init +device_tree_parse_numa_distance_map_v1(const void *fdt, intnode)
Quoted 22 lines
Quoted 22 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4
-0+{ +const struct fdt_property *prop; +const __be32 *matrix; +int entry_count, len, i;Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const · line 217
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+ +printk (XENLOG_INFO "NUMA: parsing numa-distance-map\n" ); + + prop =fdt_get_property (fdt, node,"distance-matrix" , &len); + if ( !prop ) + { + printk(XENLOG_WARNING + "NUMA: No distance-matrix property in distance-map\n");
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const · line 225
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+ +return -EINVAL ; + } + + if ( len % sizeof(uint32_t) != 0 ) + { + printk(XENLOG_WARNING + "distance-matrix in node is not a multiple ofu32\n");
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const · line 232
Quoted 7 lines
xen/arch/arm/numa_device_tree.c +7
-0+return -EINVAL ; + } + + entry_count = len /sizeof (uint32_t ); + if ( entry_count <= 0 ) + { + printk(XENLOG_WARNING "NUMA: Invalid distance-matrix\n");
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
xen/arch/arm/numa_device_tree.c +13
-0+ +return -EINVAL ; + } + + matrix = (const __be32 *)prop->data; +for ( i =0 ; i +2 < entry_count; i +=3 ) + { +uint32_t from, to, distance; + + from =dt_read_number (matrix,1 ); + matrix++; + to =dt_read_number (matrix,1 ); + matrix++;Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const · line 253
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+ distance =dt_read_number (matrix,1 ); + matrix++; + + if ( (from == to && distance != NUMA_LOCAL_DISTANCE) || + (from != to && distance <= NUMA_LOCAL_DISTANCE) ) + { + printk(XENLOG_WARNING + "Invalid nodes' distance from node#%d tonode#%d
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
= %d\n",
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5
-0+ from, to, distance); +return -EINVAL ; + } + + printk(XENLOG_INFO "NUMA: distance from node#%d tonode#%d
Quoted 11 lines
Quoted 11 lines
Quoted 9 lines
Quoted 9 lines
= %d\n",
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const · line 264
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6
-0+ from, to, distance); +numa_set_distance (from, to, distance); + +/* Set default distance of node B->A same as A->B */ +if (to > from) +numa_set_distance (to, from, distance);I am a bit unsure about this last 2 lines: why calling
numa_set_distanceQuoted 1 line
Quoted 1 line
in the opposite direction only when to > from? Wouldn't it be OK
to
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
always do both:
numa_set_distance(from, to, distance);numa_set_distance(to, from, distance);?
I borrowed this code from Linux, but here is my understanding:
First, I read some notes in
Documentation/devicetree/bindings/numa.txtQuoted 27 lines
Quoted 27 lines
Quoted 27 lines
1. Each entry represents distance from first node to second node. The distances are equal in either direction.
2. distance-matrix should have entries in lexicographical ascending order of nodes.Here is an example of distance-map node in DTB:
Sample#1, full list:distance-map { compatible = "numa-distance-map-v1"; distance-matrix = <0 0 10>, <0 1 20>, <0 2 40>, <0 3 20>, <1 0 20>, <1 1 10>, <1 2 20>, <1 3 40>, <2 0 40>, <2 1 20>, <2 2 10>, <2 3 20>, <3 0 20>, <3 1 40>, <3 2 20>, <3 3 10>; };Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Call
numa_set_distancewhen "to > from" will prevent Xen to callnuma_set_distance(0, 1, 20)again when it's setting distance for <10
Quoted 20 lines
Quoted 20 lines
20>.
Quoted 19 lines
But,
numa_set_distance(1, 0, 20)will be call twice.Normally, distance-map node will be optimized in following sample#2, all redundant entries are removed:
Sample#2, partial list:distance-map { compatible = "numa-distance-map-v1"; distance-matrix = <0 0 10>, <0 1 20>, <0 2 40>, <0 3 20>, <1 1 10>, <1 2 20>, <1 3 40>, <2 2 10>, <2 3 20>, <3 3 10>; };Quoted 1 line
Quoted 1 line
Quoted 1 line
There is not any "from > to" entry in the map. But using this
partial
Quoted 2 lines
Quoted 2 lines
map
Quoted 1 line
still can set all distances for all pairs. And numa_set_distance(1,
0,
Quoted 4 lines
Quoted 4 lines
20)
Quoted 1 line
will be only once.
I see. I can't find in
Documentation/devicetree/bindings/numa.txtwhere
Quoted 15 lines
Quoted 4 lines
it says that "from > to" nodes can be omitted. If it is not written down, then somebody could easily optimize it the opposite way:
Review
Quoted 10 lines
(quoted diff) +0
-0distance-matrix = <0 0 10>, <1 0 20>, <2 0 40>, <3 0 20>, <1 1 10>, <2 1 20>, <3 1 40>, <2 2 10>, <3 2 20>, <3 3 10>;Quoted 5 lines
Yes, you're right. Spec doesn't say opposite way is unallowed.
Quoted 3 lines
I think the code in Xen should be resilient and able to cope with a device tree like the one you wrote or the one I wrote. From a code perspective, it should be very easy to do. If nothing else it would
make
Quoted 40 lines
Quoted 3 lines
Xen more resilient against buggy firmware.
I don't disagree with that.
Quoted 29 lines
Quoted 1 line
Quoted 1 line
But in any case, I have a different suggestion. The binding states
that
Quoted 1 line
Quoted 1 line
"distances are equal in either direction". Also it has an example
where
Quoted 19 lines
Quoted 3 lines
only one direction is expressed unfortunately (at the end of the document).
Oh, I should see this comment first, then I will not post above comment : )
Quoted 11 lines
So my suggestion is to parse it as follows:
- call
numa_set_distancejust once from
device_tree_parse_numa_distance_map_v1- in
numa_set_distance:
- set node_distance_map[from][to] = distance;
- check node_distance_map[to][from]
- if unset, node_distance_map[to][from] = distance;
- if already set to the same value, return success;
- if already set to a different value, return error;Quoted 5 lines
I don't really like this implementation. I want the behavior of
numa_set_distancejust like the function name, do not include implicit operations. Otherwise, except the user read this function implementation before he use it, he probably doesn't know this function has done so many things.Quoted 4 lines
You can leave
numa_set_distanceas-is without any implicit operations.In that case, just call
numa_set_distancetwice fromnuma_set_distancefor both from/to and to/from.numa_set_distancecould return error isQuoted 6 lines
I am OK for the first sentence. But...
Quoted 2 lines
the entry was already set to a different value or success otherwise (also in the case it was already set to the same value). This would
... I prefer not to check the previous value. Subsequent
numa_set_distanceQuoted 4 lines
call will override previous calls. Keep
numa_set_distanceas simple as it can. And when you pass new data tonuma_set_distance, it doesn't know whether the previous data was correct or the new data is correct. Only caller may have known.Quoted 6 lines
That might be OK but if not
numa_set_distancethen somebody else needs to check against overwriting previous values. That is to be able to spot bad device tree cases like:0 1 20
1 0 40How about we check it still in NUMA distance parse function? Before setting the
numa_set_distancefor one pair nodes (e.g. a -> b), we can get its opposite way distance first.
distance_b_a=__node_distance(b, a); ==> get opposite way distance. if (distance_b_a== 0) ==> opposite way distance has not been set
Quoted 14 lines
{
numa_set_distance(a, b, 20); ==> set both
numa_set_distance(b, a, 20)
} else {
if (distance_b_a == 20) ==> opposite way distance has been set
numa_set_distance(a, b, 20); ==> set this way only
else ===> opposite way distance has been set, but is unmatched
// What can we do here?
Panic the system? or Just warning users? Or choose the bigger
distance for both ways?
And distance_b_a == NUMA_NO_DISTANCE would be a special case
here.
}Yes, I think something like this would work. If we detect an error in device tree I would probably print a warning and disable NUMA. It would also be acceptable to panic.
Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map
In reply to Stefano StabelliniQuoted 27 lines · Stefano Stabellini
Quoted 17 lines · Wei Chen
Quoted 10 lines
Stefano Stabellini
Sent: 2021年9月1日 5:36Quoted 5 lines · Wei Chen
I don't really like this implementation. I want the behavior of
numa_set_distancejust like the function name, do not include implicit operations. Otherwise, except the user read this function implementation before he use it, he probably doesn't know this function has done so many things.Quoted 4 lines
You can leave
numa_set_distanceas-is without any implicit operations.In that case, just call
numa_set_distancetwice fromnuma_set_distancefor both from/to and to/from.numa_set_distancecould return error isI am OK for the first sentence. But...
Quoted 8 lines
Quoted 2 lines
the entry was already set to a different value or success otherwise (also in the case it was already set to the same value). This would
... I prefer not to check the previous value. Subsequent
numa_set_distancecall will override previous calls. Keepnuma_set_distanceas simple as it can. And when you pass new data tonuma_set_distance, it doesn't know whether the previous data was correct or the new data is correct. Only caller may have known.
Quoted 6 lines
That might be OK but if not
numa_set_distancethen somebody else needs to check against overwriting previous values. That is to be able to spot bad device tree cases like:0 1 20
1 0 40
What's wrong with this? At least the ACPI spec cares to specifically permit this case:
"Except for the relative distance from a System Locality to itself,
each relative distance is stored twice in the matrix. This provides the capability to describe the scenario where the relative distances for the two directions between System Localities is different."
Jan
RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map
Hi Jan,
Quoted 43 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年9月2日 14:00
To: Stefano Stabellini <[email protected]>; Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance mapQuoted 16 lines · Stefano Stabellini
Quoted 15 lines · Wei Chen
Quoted 14 lines
Stefano Stabellini
Sent: 2021年9月1日 5:36Quoted 5 lines · Wei Chen
I don't really like this implementation. I want the behavior of
numa_set_distancejust like the function name, do not include implicit operations. Otherwise, except the user read this function implementation before he use it, he probably doesn't know this function has done so many things.You can leave
numa_set_distanceas-is without any implicit operations.In that case, just call
numa_set_distancetwice fromnuma_set_distancefor both from/to and to/from.numa_set_distancecould return error isQuoted 6 lines
Quoted 6 lines
I am OK for the first sentence. But...
Quoted 2 lines
the entry was already set to a different value or success otherwise (also in the case it was already set to the same value). This would
... I prefer not to check the previous value. Subsequent
numa_set_distanceQuoted 9 lines
Quoted 4 lines
call will override previous calls. Keep
numa_set_distanceas simple as it can. And when you pass new data tonuma_set_distance, it doesn't know whether the previous data was correct or the new data is correct. Only caller may have known.That might be OK but if not
numa_set_distancethen somebody else needs to check against overwriting previous values. That is to be able to spot bad device tree cases like:
Quoted 6 lines
Quoted 2 lines
0 1 20
1 0 40What's wrong with this? At least the ACPI spec cares to specifically permit this case:
This is because some notes description in device tree NUMA binding:
- distance-matrix
This property defines a matrix to describe the relative distances between all numa nodes.
It is represented as a list of node pairs and their relative distance.
Note:
1. Each entry represents distance from first node to second node. The distances are equal in either direction.
https://www.kernel.org/doc/Documentation/devicetree/bindings/numa.txt
So we treat this case is wrong in DT. But as you said ACPI allow such case, that means real machines allow different distances for one pair nodes in different directions.
Quoted 6 lines
"Except for the relative distance from a System Locality to itself,
each relative distance is stored twice in the matrix. This provides the capability to describe the scenario where the relative distances for the two directions between System Localities is different."Jan
RFC PATCH 25/40 xen/arm: unified entry to parse all NUMA data from device tree
In reply to Wei ChenIn this API, we scan whole device tree to parse CPU node id, memory node id and distance-map. Though early_scan_node will invoke has a handler to process memory nodes. If we want to parse memory node id in this handler, we have to embeded NUMA parse code in this handler. But we still need to scan whole device tree to find CPU NUMA id and distance-map. In this case, we include memory NUMA id parse in this API too. Another benefit is that we have a unique entry for device tree NUMA data parse.
--- xen/arch/arm/numa_device_tree.c | 31 ++++++++++++++++++++++++++++--- xen/include/asm-arm/numa.h | 1 + 2 files changed, 29 insertions(+), 3 deletions(-)
xen/arch/arm/numa_device_tree.c +28 -3
@@ -131,7 +131,8 @@ save_memblk: }/* Parse CPU NUMA node info */ -int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +static int __init +device_tree_parse_numa_cpu_node (const void *fdt,int node) {uint32_t nid;
@@ -147,7 +148,7 @@ int __init device_tree_parse_numa_cpu_node(const void *fdt, int node) }/* Parse memory node NUMA info */ -int __init +static int __init device_tree_parse_numa_memory_node (const void *fdt,int node,const char *name,uint32_t addr_cells,uint32_t size_cells) {
@@ -202,7 +203,7 @@ device_tree_parse_numa_memory_node(const void *fdt, int node, }/* Parse NUMA distance map v1 */ -int __init +static int __init device_tree_parse_numa_distance_map_v1 (const void *fdt,int node) {const struct fdt_property *prop;
@@ -267,3 +268,27 @@ device_tree_parse_numa_distance_map_v1(const void *fdt, int node)return 0 ; } + +static int __init fdt_scan_numa_nodes (const void *fdt, +int node,const char *uname,int depth, +u32 address_cells,u32 size_cells,void *data) +{ +int ret =0 ; + +if (fdt_node_check_type (fdt, node,"cpu" ) ==0 ) + ret =device_tree_parse_numa_cpu_node (fdt, node); +else if (fdt_node_check_type (fdt, node,"memory" ) ==0 ) + ret =device_tree_parse_numa_memory_node (fdt, node, uname, + address_cells, size_cells); +else if (fdt_node_check_compatible (fdt, node, +"numa-distance-map-v1" ) ==0 ) + ret =device_tree_parse_numa_distance_map_v1 (fdt, node); + +return ret; +} + +/* Initialize NUMA from device tree */ +int __init numa_device_tree_init (const void *fdt) +{ +return device_tree_for_each_node (fdt,0 , fdt_scan_numa_nodes,NULL ); +}
xen/include/asm-arm/numa.h +1 -0
@@ -26,6 +26,7 @@ typedef u8 nodeid_t;extern s8 device_tree_numa;extern void numa_init (bool acpi_off); +extern int numa_device_tree_init (const void *fdt);extern void numa_set_distance (nodeid_t from,nodeid_t to,uint32_t distance);/*
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 25/40] xen/arm: unified entry to parse all NUMA data from device tree
Quoted 59 lines · Wei Chen
In this API, we scan whole device tree to parse CPU node id, memory node id and distance-map. Though
early_scan_nodewill invoke has a handler to process memory nodes. If we want to parse memory node id in this handler, we have to embeded NUMA parse code in this handler. But we still need to scan whole device tree to find CPU NUMA id and distance-map. In this case, we include memory NUMA id parse in this API too. Another benefit is that we have a unique entry for device tree NUMA data parse.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 31 ++++++++++++++++++++++++++++--- xen/include/asm-arm/numa.h | 1 + 2 files changed, 29 insertions(+), 3 deletions(-)xen/arch/arm/numa_device_tree.c +12
-3@@ -131,7 +131,8 @@ save_memblk: }/* Parse CPU NUMA node info */ -int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +static int __init +device_tree_parse_numa_cpu_node (const void *fdt,int node) {uint32_t nid;@@ -147,7 +148,7 @@ int __init device_tree_parse_numa_cpu_node(const void *fdt, int node) }/* Parse memory node NUMA info */ -int __init +static int __init device_tree_parse_numa_memory_node (const void *fdt,int node,const char *name,uint32_t addr_cells,uint32_t size_cells) {@@ -202,7 +203,7 @@ device_tree_parse_numa_memory_node(const void *fdt, int node, }/* Parse NUMA distance map v1 */ -int __init +static int __init device_tree_parse_numa_distance_map_v1 (const void *fdt,int node) {const struct fdt_property *prop;@@ -267,3 +268,27 @@ device_tree_parse_numa_distance_map_v1(const void *fdt, int node)return 0 ; } + +static int __init fdt_scan_numa_nodes (const void *fdt, +int node,const char *uname,int depth, +u32 address_cells,u32 size_cells,void *data) +{ +int ret =0 ; + +if (fdt_node_check_type (fdt, node,"cpu" ) ==0 )
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_distance_map_v1(const void *fdt, int node) · line 288
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ ret =device_tree_parse_numa_cpu_node (fdt, node); + else if ( fdt_node_check_type(fdt, node, "memory") == 0 ) + ret =device_tree_parse_numa_memory_node (fdt, node, uname, + address_cells, size_cells); + else if ( fdt_node_check_compatible(fdt, node, + "numa-distance-map-v1") == 0 ) + ret =device_tree_parse_numa_distance_map_v1 (fdt, node); + +return ret; +}
Julien, do you have an opinion on whether it might be worth reusing the existing early_scan_node function for this to avoiding another full FDT scan (to avoid another call to device_tree_for_each_node)?
Re: [XEN RFC PATCH 25/40] xen/arm: unified entry to parse all NUMA data from device tree
Hi Stefano,
Quoted 66 lines · Stefano Stabellini
Quoted 65 lines · Wei Chen
In this API, we scan whole device tree to parse CPU node id, memory node id and distance-map. Though
early_scan_nodewill invoke has a handler to process memory nodes. If we want to parse memory node id in this handler, we have to embeded NUMA parse code in this handler. But we still need to scan whole device tree to find CPU NUMA id and distance-map. In this case, we include memory NUMA id parse in this API too. Another benefit is that we have a unique entry for device tree NUMA data parse.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 31 ++++++++++++++++++++++++++++--- xen/include/asm-arm/numa.h | 1 + 2 files changed, 29 insertions(+), 3 deletions(-)xen/arch/arm/numa_device_tree.c +18
-3@@ -131,7 +131,8 @@ save_memblk: }/* Parse CPU NUMA node info */ -int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +static int __init +device_tree_parse_numa_cpu_node (const void *fdt,int node) {uint32_t nid;@@ -147,7 +148,7 @@ int __init device_tree_parse_numa_cpu_node(const void *fdt, int node) }/* Parse memory node NUMA info */ -int __init +static int __init device_tree_parse_numa_memory_node (const void *fdt,int node,const char *name,uint32_t addr_cells,uint32_t size_cells) {@@ -202,7 +203,7 @@ device_tree_parse_numa_memory_node(const void *fdt, int node, }/* Parse NUMA distance map v1 */ -int __init +static int __init device_tree_parse_numa_distance_map_v1 (const void *fdt,int node) {const struct fdt_property *prop;@@ -267,3 +268,27 @@ device_tree_parse_numa_distance_map_v1(const void *fdt, int node)return 0 ; } + +static int __init fdt_scan_numa_nodes (const void *fdt, +int node,const char *uname,int depth, +u32 address_cells,u32 size_cells,void *data) +{ +int ret =0 ; + +if (fdt_node_check_type (fdt, node,"cpu" ) ==0 ) + ret =device_tree_parse_numa_cpu_node (fdt, node); +else if (fdt_node_check_type (fdt, node,"memory" ) ==0 ) + ret =device_tree_parse_numa_memory_node (fdt, node, uname, + address_cells, size_cells); +else if (fdt_node_check_compatible (fdt, node, +"numa-distance-map-v1" ) ==0 )
Quoted 8 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_distance_map_v1(const void *fdt, int node) · line 288
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4
-0+ ret =device_tree_parse_numa_distance_map_v1 (fdt, node); + +return ret; +}Julien, do you have an opinion on whether it might be worth reusing the existing
early_scan_nodefunction for this to avoiding another full FDT scan (to avoid another call todevice_tree_for_each_node)?
I don't like the full FDT scan and actually drafted an e-mail in reply-to [1] to suggest parse all the NUMA information from early_scan_node().
However, we don't know whether ACPI or DT will be used at the time early_scan_node() is called. So we will need to revert any change which can make the code a little more awkward.
So I decided to drop my e-mail because I prefer the full DT scan for now. We can look at optimizing later if this becomes a pain point.
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 25/40] xen/arm: unified entry to parse all NUMA data from device tree
Quoted 80 lines · Julien Grall
Hi Stefano,
Quoted 65 lines · Stefano Stabellini
Quoted 64 lines · Wei Chen
In this API, we scan whole device tree to parse CPU node id, memory node id and distance-map. Though
early_scan_nodewill invoke has a handler to process memory nodes. If we want to parse memory node id in this handler, we have to embeded NUMA parse code in this handler. But we still need to scan whole device tree to find CPU NUMA id and distance-map. In this case, we include memory NUMA id parse in this API too. Another benefit is that we have a unique entry for device tree NUMA data parse.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 31 ++++++++++++++++++++++++++++--- xen/include/asm-arm/numa.h | 1 + 2 files changed, 29 insertions(+), 3 deletions(-)xen/arch/arm/numa_device_tree.c +2
-1@@ -131,7 +131,8 @@ save_memblk: }/* Parse CPU NUMA node info */ -int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +static int __init +device_tree_parse_numa_cpu_node (const void *fdt,int node) {uint32_t nid; @@ -147,7 +148,7 @@ int __init device_tree_parse_numa_cpu_node(constvoid *fdt,int node)}
/* Parse memory node NUMA info */xen/arch/arm/numa_device_tree.c +16
-2-int __init +static int __init device_tree_parse_numa_memory_node (const void *fdt,int node,const char *name,uint32_t addr_cells,uint32_t size_cells) {@@ -202,7 +203,7 @@ device_tree_parse_numa_memory_node(const void *fdt, int node, }/* Parse NUMA distance map v1 */ -int __init +static int __init device_tree_parse_numa_distance_map_v1 (const void *fdt,int node) {const struct fdt_property *prop;@@ -267,3 +268,27 @@ device_tree_parse_numa_distance_map_v1(const void *fdt, int node)return 0 ; } + +static int __init fdt_scan_numa_nodes (const void *fdt, +int node,const char *uname,int depth, +u32 address_cells,u32 size_cells,void *data) +{ +int ret =0 ; + +if (fdt_node_check_type (fdt, node,"cpu" ) ==0 ) + ret =device_tree_parse_numa_cpu_node (fdt, node); +else if (fdt_node_check_type (fdt, node,"memory" ) ==0 ) + ret =device_tree_parse_numa_memory_node (fdt, node, uname, + address_cells, size_cells); +else if (fdt_node_check_compatible (fdt, node, +"numa-distance-map-v1" ) ==0 )Quoted 8 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_distance_map_v1(const void *fdt, int node) · line 288
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4
-0+ ret =device_tree_parse_numa_distance_map_v1 (fdt, node); + +return ret; +}Julien, do you have an opinion on whether it might be worth reusing the existing
early_scan_nodefunction for this to avoiding another full FDT scan (to avoid another call todevice_tree_for_each_node)?I don't like the full FDT scan and actually drafted an e-mail in reply-to [1] to suggest parse all the NUMA information from
early_scan_node().
Quoted 6 lines
However, we don't know whether ACPI or DT will be used at the time
early_scan_node()is called. So we will need to revert any change which can make the code a little more awkward.So I decided to drop my e-mail because I prefer the full DT scan for now. We can look at optimizing later if this becomes a pain point.
Uhm, yes you are right.
We would have to move some of the logic to detect ACPI earlier to early_scan_node (e.g. xen/arch/arm/acpi/boot.c:dt_scan_depth1_nodes). That could actually be a good idea, but it is true that could be done with a separate patch later.
RE: [XEN RFC PATCH 25/40] xen/arm: unified entry to parse all NUMA data from device tree
Quoted 108 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月2日 2:31
To: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>; Wei Chen <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 25/40] xen/arm: unified entry to parse all NUMA data from device treeQuoted 16 lines · Julien Grall
Hi Stefano,
Quoted 13 lines · Stefano Stabellini
Quoted 12 lines · Wei Chen
In this API, we scan whole device tree to parse CPU node id, memory node id and distance-map. Though
early_scan_nodewill invoke has a handler to process memory nodes. If we want to parse memory node id in this handler, we have to embeded NUMA parse code in this handler. But we still need to scan whole device tree to find CPU NUMA id and distance-map. In this case, we include memory NUMA id parse in this API too. Another benefit is that we have a unique entry for device tree NUMA data parse.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 31 ++++++++++++++++++++++++++++---
Quoted 12 lines
Quoted 12 lines
xen/arch/arm/numa_device_tree.c save_memblk: · line 133
Quoted 12 lines
xen/include/asm-arm/numa.h | 1 + 2 files changed, 29 insertions(+), 3 deletions(-)xen/arch/arm/numa_device_tree.c +0
-1@@ -131,7 +131,8 @@ save_memblk: }/* Parse CPU NUMA node info */ -int __init device_tree_parse_numa_cpu_node(const void *fdt, intnode)
Quoted 5 lines
Quoted 5 lines
xen/arch/arm/numa_device_tree.c save_memblk:
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +2
-0+static int __init +device_tree_parse_numa_cpu_node (const void *fdt,int node) {uint32_t nid; @@ -147,7 +148,7 @@ int __initdevice_tree_parse_numa_cpu_node(const
Quoted 9 lines
Quoted 9 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void
Quoted 9 lines
xen/arch/arm/numa_device_tree.c +0
-0void *fdt, int node)}
/* Parse memory node NUMA info */xen/arch/arm/numa_device_tree.c +1
-1-int __init +static int __init device_tree_parse_numa_memory_node (const void *fdt,int node,const char *name,uint32_t addr_cells,uint32_t size_cells) {@@ -202,7 +203,7 @@ device_tree_parse_numa_memory_node(const void*fdt,
Quoted 9 lines
Quoted 9 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_distance_map_v1(const
Quoted 9 lines
xen/arch/arm/numa_device_tree.c +1
-1int node, }/* Parse NUMA distance map v1 */ -int __init +static int __init device_tree_parse_numa_distance_map_v1 (const void *fdt,int node) {const struct fdt_property *prop;@@ -267,3 +268,27 @@ device_tree_parse_numa_distance_map_v1(constvoid
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
xen/arch/arm/numa_device_tree.c +12
-0*fdt, int node)return 0 ; } + +static int __init fdt_scan_numa_nodes (const void *fdt, +int node,const char *uname,int depth, +u32 address_cells,u32 size_cells,void *data) +{ +int ret =0 ; + +if (fdt_node_check_type (fdt, node,"cpu" ) ==0 ) + ret =device_tree_parse_numa_cpu_node (fdt, node); +else if (fdt_node_check_type (fdt, node,"memory" ) ==0 ) + ret =device_tree_parse_numa_memory_node (fdt, node, uname, + address_cells, size_cells);Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_distance_map_v1(const void *fdt, int node) · line 288
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6
-0+else if (fdt_node_check_compatible (fdt, node, +"numa-distance-map-v1" ) ==0 ) + ret =device_tree_parse_numa_distance_map_v1 (fdt, node); + +return ret; +}Julien, do you have an opinion on whether it might be worth reusing
the
Quoted 1 line
Quoted 1 line
existing
early_scan_nodefunction for this to avoiding another fullFDT
Quoted 3 lines
Quoted 1 line
scan (to avoid another call to
device_tree_for_each_node)?I don't like the full FDT scan and actually drafted an e-mail in reply-
to [1]
Quoted 4 lines
to suggest parse all the NUMA information from
early_scan_node().However, we don't know whether ACPI or DT will be used at the time
early_scan_node()is called. So we will need to revert any change whichcan
Quoted 3 lines
make the code a little more awkward.
So I decided to drop my e-mail because I prefer the full DT scan for now.
We
Quoted 8 lines
Quoted 1 line
can look at optimizing later if this becomes a pain point.
Uhm, yes you are right.
We would have to move some of the logic to detect ACPI earlier to
early_scan_node(e.g. xen/arch/arm/acpi/boot.c:dt_scan_depth1_nodes). That could actually be a good idea, but it is true that could be done with a separate patch later.
Sounds good to me.
RFC PATCH 26/40 xen/arm: Add boot and secondary CPU to NUMA system
In reply to Wei ChenWhen cpu boot up, we have add them to NUMA system. In current stage, we have not parsed the NUMA data, but we have created a fake NUMA node. So, in this patch, all CPU will be added to NUMA node#0. After the NUMA data has been parsed from device tree, the CPU will be added to correct NUMA node as the NUMA data described.
--- xen/arch/arm/setup.c | 6 ++++++ xen/arch/arm/smpboot.c | 6 ++++++ xen/include/asm-arm/numa.h | 1 + 3 files changed, 13 insertions(+)
xen/arch/arm/setup.c +6 -0
@@ -918,6 +918,12 @@ void __init start_xen(unsigned long boot_phys_offset,processor_id (); +/* +* If Xen is running on a NUMA off system, there will +* be a node#0 at least. +*/ +numa_add_cpu (0 ); +smp_init_cpus (); cpus =smp_get_max_cpus ();printk (XENLOG_INFO "SMP: Allowing %u CPUs\n" , cpus);
xen/arch/arm/smpboot.c +6 -0
@@ -358,6 +358,12 @@ void start_secondary(void)*/ smp_wmb (); +/* +* If Xen is running on a NUMA off system, there will +* be a node#0 at least. +*/ +numa_add_cpu (cpuid); +/* Now report this CPU is up */ cpumask_set_cpu (cpuid, &cpu_online_map);
xen/include/asm-arm/numa.h +1 -0
@@ -59,6 +59,7 @@ extern mfn_t first_valid_mfn;#define __node_distance (a, b) (20)#define numa_init (x) do { } while (0) +#define numa_add_cpu (x) do { } while (0)#define numa_set_node (x, y) do { } while (0)#endif
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA system
Hi Wei,
Quoted 36 lines · Wei Chen
When cpu boot up, we have add them to NUMA system. In current stage, we have not parsed the NUMA data, but we have created a fake NUMA node. So, in this patch, all CPU will be added to NUMA node#0. After the NUMA data has been parsed from device tree, the CPU will be added to correct NUMA node as the NUMA data described.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/setup.c | 6 ++++++ xen/arch/arm/smpboot.c | 6 ++++++ xen/include/asm-arm/numa.h | 1 + 3 files changed, 13 insertions(+)xen/arch/arm/setup.c +6
-0@@ -918,6 +918,12 @@ void __init start_xen(unsigned long boot_phys_offset,processor_id (); +/* +* If Xen is running on a NUMA off system, there will +* be a node#0 at least. +*/ +numa_add_cpu (0 ); +smp_init_cpus (); cpus =smp_get_max_cpus ();printk (XENLOG_INFO "SMP: Allowing %u CPUs\n" , cpus);xen/arch/arm/smpboot.c +0
-0
xen/arch/arm/smpboot.c void start_secondary(void) · line 366
Quoted 9 lines
xen/arch/arm/smpboot.c +6
-0@@ -358,6 +358,12 @@ void start_secondary(void)*/ smp_wmb (); +/* +* If Xen is running on a NUMA off system, there will +* be a node#0 at least. +*/ +numa_add_cpu (cpuid); +
On x86, numa_add_cpu() will be called before the pCPU is brought up. I am not quite too sure why we are doing it differently here. Can you clarify it?
xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 62
Quoted 12 lines
xen/arch/arm/smpboot.c +0
-0/* Now report this CPU is up */ cpumask_set_cpu (cpuid, &cpu_online_map);xen/include/asm-arm/numa.h +1
-0@@ -59,6 +59,7 @@ extern mfn_t first_valid_mfn;#define __node_distance (a, b) (20)#define numa_init (x) do { } while (0) +#define numa_add_cpu (x) do { } while (0)
This is a stubs for a common helper. So I think this wants to be moved in the !CONFIG_NUMA in xen/numa.h.
xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 65
Quoted 4 lines
xen/include/asm-arm/numa.h +0
-0#define numa_set_node (x, y) do { } while (0)#endif
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA system
Hi Julien,
Quoted 52 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 0:58
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA systemHi Wei,
xen/arch/arm/setup.c void __init start_xen(unsigned long
Quoted 19 lines · Wei Chen
When cpu boot up, we have add them to NUMA system. In current stage, we have not parsed the NUMA data, but we have created a fake NUMA node. So, in this patch, all CPU will be added to NUMA node#0. After the NUMA data has been parsed from device tree, the CPU will be added to correct NUMA node as the NUMA data described.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/setup.c | 6 ++++++ xen/arch/arm/smpboot.c | 6 ++++++ xen/include/asm-arm/numa.h | 1 + 3 files changed, 13 insertions(+)xen/arch/arm/setup.c +0
-0@@ -918,6 +918,12 @@ void __init start_xen(unsigned long
boot_phys_offset,Quoted 20 lines
xen/arch/arm/setup.c +6
-0processor_id (); +/* +* If Xen is running on a NUMA off system, there will +* be a node#0 at least. +*/ +numa_add_cpu (0 ); +smp_init_cpus (); cpus =smp_get_max_cpus ();printk (XENLOG_INFO "SMP: Allowing %u CPUs\n" , cpus);xen/arch/arm/smpboot.c +0
-0@@ -358,6 +358,12 @@ void start_secondary(void)*/ smp_wmb ();
Quoted 10 lines
xen/arch/arm/smpboot.c void start_secondary(void) · line 366
Quoted 6 lines
xen/arch/arm/smpboot.c +6
-0+/* +* If Xen is running on a NUMA off system, there will +* be a node#0 at least. +*/ +numa_add_cpu (cpuid); +On x86,
numa_add_cpu()will be called before the pCPU is brought up. I am not quite too sure why we are doing it differently here. Can you clarify it?
Of course we can invoke numa_add_cpu before cpu_up as x86. But in my tests, I found when cpu bring up failed, this cpu still be add to NUMA. Although this does not affect the execution of the code (because CPU is offline), But I don't think adding a offline CPU to NUMA makes sense.
Quoted 11 lines
Quoted 11 lines
xen/arch/arm/smpboot.c +0
-0/* Now report this CPU is up */ cpumask_set_cpu (cpuid, &cpu_online_map);xen/include/asm-arm/numa.h +0
-0@@ -59,6 +59,7 @@ extern mfn_t first_valid_mfn;#define __node_distance (a, b) (20)
Quoted 6 lines
xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 62
Quoted 2 lines
xen/include/asm-arm/numa.h +1
-0#define numa_init (x) do { } while (0) +#define numa_add_cpu (x) do { } while (0)This is a stubs for a common helper. So I think this wants to be moved in the !CONFIG_NUMA in
xen/numa.h.
OK
Quoted 9 lines
xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 65
Quoted 4 lines
xen/include/asm-arm/numa.h +0
-0#define numa_set_node (x, y) do { } while (0)#endif Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA system
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 63 lines
Quoted 52 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 0:58
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA systemHi Wei,
xen/arch/arm/setup.c void __init start_xen(unsigned long
Quoted 19 lines · Wei Chen
When cpu boot up, we have add them to NUMA system. In current stage, we have not parsed the NUMA data, but we have created a fake NUMA node. So, in this patch, all CPU will be added to NUMA node#0. After the NUMA data has been parsed from device tree, the CPU will be added to correct NUMA node as the NUMA data described.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/setup.c | 6 ++++++ xen/arch/arm/smpboot.c | 6 ++++++ xen/include/asm-arm/numa.h | 1 + 3 files changed, 13 insertions(+)xen/arch/arm/setup.c +0
-0@@ -918,6 +918,12 @@ void __init start_xen(unsigned long
boot_phys_offset,Quoted 20 lines
xen/arch/arm/setup.c +6
-0processor_id (); +/* +* If Xen is running on a NUMA off system, there will +* be a node#0 at least. +*/ +numa_add_cpu (0 ); +smp_init_cpus (); cpus =smp_get_max_cpus ();printk (XENLOG_INFO "SMP: Allowing %u CPUs\n" , cpus);xen/arch/arm/smpboot.c +0
-0@@ -358,6 +358,12 @@ void start_secondary(void)*/ smp_wmb ();Quoted 10 lines
xen/arch/arm/smpboot.c void start_secondary(void) · line 366
Quoted 6 lines
xen/arch/arm/smpboot.c +6
-0+/* +* If Xen is running on a NUMA off system, there will +* be a node#0 at least. +*/ +numa_add_cpu (cpuid); +On x86,
numa_add_cpu()will be called before the pCPU is brought up. I am not quite too sure why we are doing it differently here. Can you clarify it?
Quoted 4 lines
Of course we can invoke
numa_add_cpubeforecpu_upas x86. But in my tests, I found when cpu bring up failed, this cpu still be add to NUMA. Although this does not affect the execution of the code (because CPU is offline), But I don't think adding a offline CPU to NUMA makes sense.
Right, but again, why do you want to solve the problem on Arm and not x86? After all, NUMA is not architecture specific (in fact you move most of the code in common).
In fact, the risk, is someone may read arch/x86 and doesn't realize the CPU is not in the node until late on Arm.
So I think we should call numa_add_cpu() around the same place on all the architectures.
If you think the current position on x86 is not correct, then it should be changed at as well. However, I don't know the story behind the position of the call on x86. You may want to ask the x86 maintainers.
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA system
Quoted 31 lines · Julien Grall
Quoted 21 lines · Wei Chen
Quoted 10 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 0:58Quoted 6 lines · Wei Chen
xen/arch/arm/smpboot.c +0
-0@@ -358,6 +358,12 @@ void start_secondary(void)*/ smp_wmb ();Quoted 10 lines
xen/arch/arm/smpboot.c void start_secondary(void) · line 366
Quoted 6 lines
xen/arch/arm/smpboot.c +6
-0+/* +* If Xen is running on a NUMA off system, there will +* be a node#0 at least. +*/ +numa_add_cpu (cpuid); +On x86,
numa_add_cpu()will be called before the pCPU is brought up. I am not quite too sure why we are doing it differently here. Can you clarify it?Quoted 4 lines
Of course we can invoke
numa_add_cpubeforecpu_upas x86. But in my tests, I found when cpu bring up failed, this cpu still be add to NUMA. Although this does not affect the execution of the code (because CPU is offline), But I don't think adding a offline CPU to NUMA makes sense.Right, but again, why do you want to solve the problem on Arm and not x86? After all, NUMA is not architecture specific (in fact you move most of the code in common).
Quoted 5 lines
In fact, the risk, is someone may read
arch/x86and doesn't realize the CPU is not in the node until late on Arm.So I think we should call
numa_add_cpu()around the same place on all the architectures.
FWIW: +1
Jan
RE: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA system
Hi Jan, Julien,
Quoted 32 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月26日 17:40
To: Julien Grall <[email protected]>; Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA systemQuoted 22 lines · Julien Grall
Quoted 21 lines · Wei Chen
Quoted 20 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 0:58xen/arch/arm/smpboot.c void start_secondary(void) · line 366
Quoted 12 lines · Wei Chen
xen/arch/arm/smpboot.c +6
-0@@ -358,6 +358,12 @@ void start_secondary(void)*/ smp_wmb (); +/* +* If Xen is running on a NUMA off system, there will +* be a node#0 at least. +*/ +numa_add_cpu (cpuid); +On x86,
numa_add_cpu()will be called before the pCPU is brought up. I am not quite too sure why we are doing it differently here. Can you clarify it?
Quoted 11 lines
Quoted 1 line
Quoted 1 line
Of course we can invoke
numa_add_cpubeforecpu_upas x86. But in mytests,
Quoted 1 line
Quoted 1 line
I found when cpu bring up failed, this cpu still be add to NUMA.
Although
Quoted 7 lines
Quoted 2 lines
this does not affect the execution of the code (because CPU is offline), But I don't think adding a offline CPU to NUMA makes sense.
Right, but again, why do you want to solve the problem on Arm and not x86? After all, NUMA is not architecture specific (in fact you move most of the code in common).
I am not very familiar with x86, so when I was composing this patch series, I always thought that if I could solve it inside Arm Arch, I would solve it inside Arm Arch. That seems a bit conservative, and inappropriate on solving this problem.
Quoted 7 lines
Quoted 5 lines
In fact, the risk, is someone may read
arch/x86and doesn't realize the CPU is not in the node until late on Arm.So I think we should call
numa_add_cpu()around the same place on all the architectures.FWIW: +1
I agree. As Jan in this discussion. How about following current x86's numa_add_cpu behaviors in __start_xen, but add some code to revert numa_add_cpu when cpu_up failed (both Arm and x86)?
Quoted 2 lines
Jan
Re: [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA system
Quoted 17 lines · Wei Chen
Quoted 8 lines
From: Jan Beulich <[email protected]>
Sent: 2021年8月26日 17:40Quoted 4 lines · Julien Grall
Right, but again, why do you want to solve the problem on Arm and not x86? After all, NUMA is not architecture specific (in fact you move most of the code in common).
I am not very familiar with x86, so when I was composing this patch series, I always thought that if I could solve it inside Arm Arch, I would solve it inside Arm Arch. That seems a bit conservative, and inappropriate on solving this problem.
Quoted 3 lines
Quoted 3 lines
In fact, the risk, is someone may read
arch/x86and doesn't realize the CPU is not in the node until late on Arm.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
So I think we should call
numa_add_cpu()around the same place on all the architectures.FWIW: +1
I agree. As Jan in this discussion. How about following current x86's
numa_add_cpubehaviors in__start_xen, but add some code to revertnuma_add_cpuwhencpu_upfailed (both Arm and x86)?
Sure - if we don't clean up properly on x86 on a failure path, I'm all for having that fixed.
Jan
RFC PATCH 27/40 xen/arm: build CPU NUMA node map while creating cpu_logical_map
In reply to Wei ChenSometimes, CPU logical ID maybe different with physical CPU ID. Xen is using CPU logial ID for runtime usage, so we should use CPU logical ID to create map between NUMA node and CPU.
--- xen/arch/arm/smpboot.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)
xen/arch/arm/smpboot.c +30 -1
@@ -121,7 +121,12 @@ static void __init dt_smp_init_cpus(void) { [0 ... NR_CPUS - 1] = MPIDR_INVALID }; +static nodeid_t node_map[NR_CPUS ]__initdata = + { + [0 ...NR_CPUS -1 ] =NUMA_NO_NODE + };bool bootcpu_valid = false; +uint32_t nid =0 ;int rc; mpidr = boot_cpu_data.mpidr.bits &MPIDR_HWID_MASK ;
@@ -172,6 +177,26 @@ static void __init dt_smp_init_cpus(void)continue ; } +#ifdef CONFIG_DEVICE_TREE_NUMA +/* +* When CONFIG_DEVICE_TREE_NUMA is set, try to fetch numa infomation +* from CPU dts node, otherwise the nid is always 0. +*/ +if ( !dt_property_read_u32 (cpu,"numa-node-id" , &nid) ) + { +printk (XENLOG_WARNING +"cpu[%d] dts path: %s: doesn't have numa infomation!\n" , + cpuidx,dt_node_full_name (cpu)); +/* +* The the early stage of NUMA initialization, when Xen found any +* CPU dts node doesn't have numa-node-id info, the NUMA will be +* treated as off, all CPU will be set to a FAKE node 0. So if we +* get numa-node-id failed here, we should set nid to 0. +*/ + nid =0 ; + } +#endif +/* * 8 MSBs must be set to 0 in the DT since the reg property * defines the MPIDR[23:0]
@@ -231,9 +256,12 @@ static void __init dt_smp_init_cpus(void) {printk ("cpu%d init failed (hwid %" PRIregister"): %d\n" , i, hwid, rc); tmp_map[i] =MPIDR_INVALID ; + node_map[i] =NUMA_NO_NODE ; } - else + else { tmp_map[i] = hwid; + node_map[i] = nid; + } }if ( !bootcpu_valid )
@@ -249,6 +277,7 @@ static void __init dt_smp_init_cpus(void)continue ;cpumask_set_cpu (i, &cpu_possible_map);cpu_logical_map (i) = tmp_map[i]; +numa_set_node (i, node_map[i]); } }
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 27/40] xen/arm: build CPU NUMA node map while creating cpu_logical_map
Hi Wei,
Quoted 3 lines · Wei Chen
Sometimes, CPU logical ID maybe different with physical CPU ID. Xen is using CPU logial ID for runtime usage, so we should use CPU logical ID to create map between NUMA node and CPU.
This commit message gives the impression that you are trying to fix a bug. However, what you are explaining is the reason why the code will use the logical ID rather than physical ID.
I think the commit message should explain what the patch is doing. You can then add an explanation why you are using the CPU logical ID. Something like "Note we storing the CPU logical ID because...".
Quoted 24 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/smpboot.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)xen/arch/arm/smpboot.c +5
-0@@ -121,7 +121,12 @@ static void __init dt_smp_init_cpus(void) { [0 ... NR_CPUS - 1] = MPIDR_INVALID }; +static nodeid_t node_map[NR_CPUS ]__initdata = + { + [0 ...NR_CPUS -1 ] =NUMA_NO_NODE + };bool bootcpu_valid = false; +uint32_t nid =0 ;int rc; mpidr = boot_cpu_data.mpidr.bits &MPIDR_HWID_MASK ;
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 185
Quoted 9 lines
xen/arch/arm/smpboot.c +6
-0@@ -172,6 +177,26 @@ static void __init dt_smp_init_cpus(void)continue ; } +#ifdef CONFIG_DEVICE_TREE_NUMA +/* +* When CONFIG_DEVICE_TREE_NUMA is set, try to fetch numa infomation +* from CPU dts node, otherwise the nid is always 0. +*/ +if ( !dt_property_read_u32 (cpu,"numa-node-id" , &nid) )
You can avoid the #ifdef by writing:
if ( IS_ENABLED(CONFIG_DEVICE_TREE_NUMA) && ... )
However, I would using CONFIG_NUMA because this code is already DT specific. So we can shorten the name a bit.
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 188
Quoted 3 lines
xen/arch/arm/smpboot.c +3
-0+ { + printk(XENLOG_WARNING + "cpu[%d] dts path: %s: doesn't have numa infomation!\n",
s/information/information/
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 191
Quoted 3 lines
xen/arch/arm/smpboot.c +3
-0+ cpuidx, dt_node_full_name(cpu)); +/* +* The the early stage of NUMA initialization, when Xen found any
s/The/During/?
Quoted 26 lines
xen/arch/arm/smpboot.c +12
-1+* CPU dts node doesn't have numa-node-id info, the NUMA will be +* treated as off, all CPU will be set to a FAKE node 0. So if we +* get numa-node-id failed here, we should set nid to 0. +*/ + nid = 0; + } +#endif +/* * 8 MSBs must be set to 0 in the DT since the reg property * defines the MPIDR[23:0] @@ -231,9 +256,12 @@ static void __init dt_smp_init_cpus(void) {printk ("cpu%d init failed (hwid %" PRIregister"): %d\n" , i, hwid, rc); tmp_map[i] =MPIDR_INVALID ; + node_map[i] =NUMA_NO_NODE ; } - else + else { tmp_map[i] = hwid; + node_map[i] = nid; + } }if ( !bootcpu_valid )
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 283
Quoted 7 lines
xen/arch/arm/smpboot.c +1
-0@@ -249,6 +277,7 @@ static void __init dt_smp_init_cpus(void)continue ;cpumask_set_cpu (i, &cpu_possible_map);cpu_logical_map (i) = tmp_map[i]; +numa_set_node (i, node_map[i]); } } >
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 27/40] xen/arm: build CPU NUMA node map while creating cpu_logical_map
Hi Julien,
Quoted 17 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 1:07
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [XEN RFC PATCH 27/40] xen/arm: build CPU NUMA node map while creatingcpu_logical_mapHi Wei,
Quoted 3 lines · Wei Chen
Sometimes, CPU logical ID maybe different with physical CPU ID. Xen is using CPU logial ID for runtime usage, so we should use CPU logical ID to create map between NUMA node and CPU.
Quoted 9 lines
This commit message gives the impression that you are trying to fix a bug. However, what you are explaining is the reason why the code will use the logical ID rather than physical ID.
I think the commit message should explain what the patch is doing. You can then add an explanation why you are using the CPU logical ID. Something like "Note we storing the CPU logical ID because...".
Ok
Quoted 35 lines
Quoted 24 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/smpboot.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)xen/arch/arm/smpboot.c +5
-0@@ -121,7 +121,12 @@ static void __init dt_smp_init_cpus(void) { [0 ... NR_CPUS - 1] = MPIDR_INVALID }; +static nodeid_t node_map[NR_CPUS ]__initdata = + { + [0 ...NR_CPUS -1 ] =NUMA_NO_NODE + };bool bootcpu_valid = false; +uint32_t nid =0 ;int rc; mpidr = boot_cpu_data.mpidr.bits &MPIDR_HWID_MASK ;xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 182
Quoted 6 lines
xen/arch/arm/smpboot.c +3
-0@@ -172,6 +177,26 @@ static void __init dt_smp_init_cpus(void)continue ; } +#ifdef CONFIG_DEVICE_TREE_NUMA +/* +* When CONFIG_DEVICE_TREE_NUMA is set, try to fetch numa infomation
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 185
Quoted 3 lines
xen/arch/arm/smpboot.c +3
-0+* from CPU dts node, otherwise the nid is always 0. +*/ +if ( !dt_property_read_u32 (cpu,"numa-node-id" , &nid) )
Quoted 7 lines
You can avoid the #ifdef by writing:
if (
IS_ENABLED(CONFIG_DEVICE_TREE_NUMA)&& ... )However, I would using
CONFIG_NUMAbecause this code is already DT specific. So we can shorten the name a bit.
OK
Quoted 5 lines
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 188
Quoted 3 lines
xen/arch/arm/smpboot.c +3
-0+ { + printk(XENLOG_WARNING + "cpu[%d] dts path: %s: doesn't have numa infomation!\n",s/information/information/
OK
Quoted 7 lines
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void)
Quoted 4 lines
xen/arch/arm/smpboot.c +3
-0+ cpuidx, dt_node_full_name(cpu)); +/* +* The the early stage of NUMA initialization, when Xen found any
s/The/During/?
Oh, yes, I will fix it.
Quoted 42 lines
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 192
Quoted 2 lines
xen/arch/arm/smpboot.c +1
-0+* CPU dts node doesn't have numa-node-id info, the NUMA will be
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 193
Quoted 1 line
xen/arch/arm/smpboot.c +1
-0+* treated as off, all CPU will be set to a FAKE node 0. So if we
xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 257
Quoted 12 lines
xen/arch/arm/smpboot.c +6
-0+* get numa-node-id failed here, we should set nid to 0. +*/ + nid = 0; + } +#endif +/* * 8 MSBs must be set to 0 in the DT since the reg property * defines the MPIDR[23:0] @@ -231,9 +256,12 @@ static void __init dt_smp_init_cpus(void) { printk("cpu%d init failed (hwid %"PRIregister"): %d\n", i,hwid, rc);
Quoted 12 lines
xen/arch/arm/smpboot.c +4
-1tmp_map[i] =MPIDR_INVALID ; + node_map[i] =NUMA_NO_NODE ; } -else +else { tmp_map[i] = hwid; + node_map[i] = nid; + } }if ( !bootcpu_valid )xen/arch/arm/smpboot.c static void __init dt_smp_init_cpus(void) · line 283
Quoted 7 lines
xen/arch/arm/smpboot.c +1
-0@@ -249,6 +277,7 @@ static void __init dt_smp_init_cpus(void)continue ;cpumask_set_cpu (i, &cpu_possible_map);cpu_logical_map (i) = tmp_map[i]; +numa_set_node (i, node_map[i]); } } >Cheers,
--
Julien Grall
RFC PATCH 28/40 xen/x86: decouple nodes_cover_memory with E820 map
In reply to Wei ChenWe will reuse nodes_cover_memory for Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_number arch_get_memory_bank_range
Depends above two helpers, we make nodes_cover_memory become architecture independent.
--- xen/arch/x86/numa.c | 18 ++++++++++++++++++ xen/arch/x86/srat.c | 8 +++----- xen/include/xen/numa.h | 4 ++++ 3 files changed, 25 insertions(+), 5 deletions(-)
xen/arch/x86/numa.c +18 -0
@@ -128,6 +128,24 @@ unsigned int __init arch_get_dma_bitsize(void) + PAGE_SHIFT, 32); } +uint32_t __init arch_meminfo_get_nr_bank (void ) +{ +return e820.nr_map; +} + +int __init arch_meminfo_get_ram_bank_range (int bank, +unsigned long long *start,unsigned long long *end) +{ +if (e820.map[bank].type !=E820_RAM || !start || !end) { +return -1 ; + } + + *start = e820.map[bank].addr; + *end = e820.map[bank].addr + e820.map[bank].size; + +return 0 ; +} +static void dump_numa (unsigned char key) {s_time_t now =NOW ();
xen/arch/x86/srat.c +3 -5
@@ -316,18 +316,16 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)static int __init nodes_cover_memory (void ) {int i; +uint32_t nr_banks =arch_meminfo_get_nr_bank (); -for (i =0 ; i < e820.nr_map; i++) { +for (i =0 ; i < nr_banks; i++) {int j, found;unsigned long long start, end; -if (e820.map[i].type !=E820_RAM ) { +if (arch_meminfo_get_ram_bank_range (i, &start, &end)) {continue ; } - start = e820.map[i].addr; - end = e820.map[i].addr + e820.map[i].size; - do { found = 0; for_each_node_mask(j, memory_nodes_parsed)
xen/include/xen/numa.h +4 -0
@@ -89,6 +89,10 @@ static inline void clear_node_cpumask(int cpu)cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } +extern uint32_t arch_meminfo_get_nr_bank (void ); +extern int arch_meminfo_get_ram_bank_range (int bank, +unsigned long long *start,unsigned long long *end); +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 28/40] xen/x86: decouple nodes_cover_memory with E820 map
Quoted 8 lines · Wei Chen
We will reuse
nodes_cover_memoryfor Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_numberarch_get_memory_bank_rangeDepends above two helpers, we make
nodes_cover_memorybecome architecture independent.
You might want to note that the only change from an x86 perspective is the additional checks:
!start || !end
Quoted 80 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 18 ++++++++++++++++++ xen/arch/x86/srat.c | 8 +++----- xen/include/xen/numa.h | 4 ++++ 3 files changed, 25 insertions(+), 5 deletions(-)xen/arch/x86/numa.c +18
-0@@ -128,6 +128,24 @@ unsigned int __init arch_get_dma_bitsize(void) + PAGE_SHIFT, 32); } +uint32_t __init arch_meminfo_get_nr_bank (void ) +{ +return e820.nr_map; +} + +int __init arch_meminfo_get_ram_bank_range (int bank, +unsigned long long *start,unsigned long long *end) +{ +if (e820.map[bank].type !=E820_RAM || !start || !end) { +return -1 ; + } + + *start = e820.map[bank].addr; + *end = e820.map[bank].addr + e820.map[bank].size; + +return 0 ; +} +static void dump_numa (unsigned char key) {s_time_t now =NOW ();xen/arch/x86/srat.c +3
-5@@ -316,18 +316,16 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)static int __init nodes_cover_memory (void ) {int i; +uint32_t nr_banks =arch_meminfo_get_nr_bank (); -for (i =0 ; i < e820.nr_map; i++) { +for (i =0 ; i < nr_banks; i++) {int j, found;unsigned long long start, end; -if (e820.map[i].type !=E820_RAM ) { +if (arch_meminfo_get_ram_bank_range (i, &start, &end)) {continue ; } - start = e820.map[i].addr; - end = e820.map[i].addr + e820.map[i].size; - do { found = 0; for_each_node_mask(j, memory_nodes_parsed)xen/include/xen/numa.h +4
-0@@ -89,6 +89,10 @@ static inline void clear_node_cpumask(int cpu)cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } +extern uint32_t arch_meminfo_get_nr_bank (void ); +extern int arch_meminfo_get_ram_bank_range (int bank, +unsigned long long *start,unsigned long long *end); +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */ Signature · 1 line
2.25.1
RE: [XEN RFC PATCH 28/40] xen/x86: decouple nodes_cover_memory with E820 map
Hi Stefano,
Quoted 16 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 9:08
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 28/40] xen/x86: decouplenodes_cover_memorywith E820 mapQuoted 6 lines · Wei Chen
We will reuse
nodes_cover_memoryfor Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_numberarch_get_memory_bank_range
Quoted 8 lines
Quoted 2 lines
Depends above two helpers, we make
nodes_cover_memorybecome architecture independent.You might want to note that the only change from an x86 perspective is the additional checks:
!start || !end
Thanks, I will add it.
Quoted 82 lines
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct
Quoted 42 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 18 ++++++++++++++++++ xen/arch/x86/srat.c | 8 +++----- xen/include/xen/numa.h | 4 ++++ 3 files changed, 25 insertions(+), 5 deletions(-)xen/arch/x86/numa.c +18
-0@@ -128,6 +128,24 @@ unsigned int __init arch_get_dma_bitsize(void) + PAGE_SHIFT, 32); } +uint32_t __init arch_meminfo_get_nr_bank (void ) +{ +return e820.nr_map; +} + +int __init arch_meminfo_get_ram_bank_range (int bank, +unsigned long long *start,unsigned long long *end) +{ +if (e820.map[bank].type !=E820_RAM || !start || !end) { +return -1 ; + } + + *start = e820.map[bank].addr; + *end = e820.map[bank].addr + e820.map[bank].size; + +return 0 ; +} +static void dump_numa (unsigned char key) {s_time_t now =NOW ();xen/arch/x86/srat.c +0
-0@@ -316,18 +316,16 @@ acpi_numa_memory_affinity_init(const struct
acpi_srat_mem_affinity*ma)Quoted 39 lines
xen/arch/x86/srat.c +3
-5static int __init nodes_cover_memory (void ) {int i; +uint32_t nr_banks =arch_meminfo_get_nr_bank (); -for (i =0 ; i < e820.nr_map; i++) { +for (i =0 ; i < nr_banks; i++) {int j, found;unsigned long long start, end; -if (e820.map[i].type !=E820_RAM ) { +if (arch_meminfo_get_ram_bank_range (i, &start, &end)) {continue ; } - start = e820.map[i].addr; - end = e820.map[i].addr + e820.map[i].size; - do { found = 0; for_each_node_mask(j, memory_nodes_parsed)xen/include/xen/numa.h +4
-1@@ -89,6 +89,10 @@ static inline void clear_node_cpumask(int cpu)cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } +extern uint32_t arch_meminfo_get_nr_bank (void ); +extern int arch_meminfo_get_ram_bank_range (int bank, +unsigned long long *start,unsigned long long *end); +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */ --2.25.1
RFC PATCH 29/40 xen/arm: implement Arm arch helpers Arm to get memory map info
In reply to Wei ChenThese two helpers are architecture APIs that are required by nodes_cover_memory.
--- xen/arch/arm/numa.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
xen/arch/arm/numa.c +14 -0
@@ -126,3 +126,17 @@ void __init numa_init(bool acpi_off)numa_initmem_init (PFN_UP (ram_start),PFN_DOWN (ram_end));return ; } + +uint32_t __init arch_meminfo_get_nr_bank (void ) +{ +return bootinfo.mem.nr_banks; +} + +int __init arch_meminfo_get_ram_bank_range (int bank, +unsigned long long *start,unsigned long long *end) +{ + *start = bootinfo.mem.bank[bank].start; + *end = bootinfo.mem.bank[bank].start + bootinfo.mem.bank[bank].size; + +return 0 ; +}
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 29/40] xen/arm: implement Arm arch helpers Arm to get memory map info
Hi Wei,
Quoted 14 lines · Wei Chen
These two helpers are architecture APIs that are required by
nodes_cover_memory.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)xen/arch/arm/numa.c +0
-0@@ -126,3 +126,17 @@ void __init numa_init(bool acpi_off)numa_initmem_init (PFN_UP (ram_start),PFN_DOWN (ram_end));
xen/arch/arm/numa.c void __init numa_init(bool acpi_off) · line 136
Quoted 10 lines
xen/arch/arm/numa.c +8
-0return ; } + +uint32_t __init arch_meminfo_get_nr_bank (void ) +{ +return bootinfo.mem.nr_banks; +} + +int __init arch_meminfo_get_ram_bank_range (int bank, +unsigned long long *start,unsigned long long *end)
They are physical address, so we should use "paddr_t" as on system such as 32-bit Arm, "unsigned long" is not enough to cover all the physical address.
As you change the type, I would also suggest to change the bank from an int to an unsigned int.
xen/arch/arm/numa.c void __init numa_init(bool acpi_off) · line 142
Quoted 7 lines
xen/arch/arm/numa.c +6
-0+{ + *start = bootinfo.mem.bank[bank].start; + *end = bootinfo.mem.bank[bank].start + bootinfo.mem.bank[bank].size; + +return 0 ; +}
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 29/40] xen/arm: implement Arm arch helpers Arm to get memory map info
Hi Julien,
Quoted 41 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 1:10
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 29/40] xen/arm: implement Arm arch helpers Arm to get memory map infoHi Wei,
Quoted 14 lines · Wei Chen
These two helpers are architecture APIs that are required by
nodes_cover_memory.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)xen/arch/arm/numa.c +0
-0@@ -126,3 +126,17 @@ void __init numa_init(bool acpi_off)numa_initmem_init (PFN_UP (ram_start),PFN_DOWN (ram_end));xen/arch/arm/numa.c void __init numa_init(bool acpi_off) · line 136
Quoted 10 lines
xen/arch/arm/numa.c +8
-0return ; } + +uint32_t __init arch_meminfo_get_nr_bank (void ) +{ +return bootinfo.mem.nr_banks; +} + +int __init arch_meminfo_get_ram_bank_range (int bank, +unsigned long long *start,unsigned long long *end)They are physical address, so we should use "
paddr_t" as on system such as 32-bit Arm, "unsigned long" is not enough to cover all the physical address.
Quoted 3 lines
As you change the type, I would also suggest to change the bank from an int to an unsigned int.
I will fix them in next version.
Quoted 12 lines
xen/arch/arm/numa.c void __init numa_init(bool acpi_off) · line 142
Quoted 7 lines
xen/arch/arm/numa.c +6
-0+{ + *start = bootinfo.mem.bank[bank].start; + *end = bootinfo.mem.bank[bank].start + bootinfo.mem.bank[bank].size; + +return 0 ; +}Cheers,
--
Julien Grall
RFC PATCH 30/40 xen: move NUMA memory and CPU parsed nodemasks to common
In reply to Wei ChenBoth memory_nodes_parsed and processor_nodes_parsed are using for Arm and x86 to record parded NUMA memory and CPU. So we move them to common.
--- xen/arch/arm/numa_device_tree.c | 2 -- xen/arch/x86/srat.c | 3 --- xen/common/numa.c | 3 +++ xen/include/xen/nodemask.h | 2 ++ 4 files changed, 5 insertions(+), 5 deletions(-)
xen/arch/arm/numa_device_tree.c +0 -2
@@ -25,8 +25,6 @@#include <asm/setup.h> s8 device_tree_numa =0 ; -static nodemask_t processor_nodes_parsed__initdata ; -static nodemask_t memory_nodes_parsed__initdata ;static int srat_disabled (void ) {
xen/arch/x86/srat.c +0 -3
@@ -24,9 +24,6 @@static struct acpi_table_slit *__read_mostly acpi_slit; -static nodemask_t memory_nodes_parsed__initdata ; -static nodemask_t processor_nodes_parsed__initdata ; -struct pxm2node {unsigned pxm;nodeid_t node;
xen/common/numa.c +3 -0
@@ -35,6 +35,9 @@ int num_node_memblks;struct node node_memblk_range[NR_NODE_MEMBLKS ];nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; +nodemask_t memory_nodes_parsed__initdata ; +nodemask_t processor_nodes_parsed__initdata ; +bool numa_off;/*
xen/include/xen/nodemask.h +2 -0
@@ -276,6 +276,8 @@ static inline int __cycle_node(int n, const nodemask_t *maskp, int nbits)*/ extern nodemask_t node_online_map; +extern nodemask_t memory_nodes_parsed; +extern nodemask_t processor_nodes_parsed;#if MAX_NUMNODES >1 #define num_online_nodes () nodes_weight(node_online_map)
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 30/40] xen: move NUMA memory and CPU parsed nodemasks to common
Hi Wei,
Quoted 3 lines · Wei Chen
Both
memory_nodes_parsedandprocessor_nodes_parsedare using for Arm and x86 to record parded NUMA memory and CPU. So we move them to common.
Looking at the usage, they both call:
numa_set...(..., bitmap)
So rather than exporting the two helpers, could we simply add helpers to abstract it?
Quoted 14 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 2 -- xen/arch/x86/srat.c | 3 --- xen/common/numa.c | 3 +++ xen/include/xen/nodemask.h | 2 ++ 4 files changed, 5 insertions(+), 5 deletions(-)xen/arch/arm/numa_device_tree.c +0
-0
xen/arch/arm/numa_device_tree.c line 29
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +0
-2@@ -25,8 +25,6 @@#include <asm/setup.h> s8 device_tree_numa =0 ; -static nodemask_t processor_nodes_parsed__initdata ; -static nodemask_t memory_nodes_parsed__initdata ;
This is code that was introduced in a previous patch. In general, it is better to do the rework first and then add the new code. This makes easier to follow series as the code added is not changed.
Quoted 37 lines
xen/arch/arm/numa_device_tree.c +0
-0static int srat_disabled (void ) {xen/arch/x86/srat.c +0
-3@@ -24,9 +24,6 @@static struct acpi_table_slit *__read_mostly acpi_slit; -static nodemask_t memory_nodes_parsed__initdata ; -static nodemask_t processor_nodes_parsed__initdata ; -struct pxm2node {unsigned pxm;nodeid_t node;xen/common/numa.c +3
-0@@ -35,6 +35,9 @@ int num_node_memblks;struct node node_memblk_range[NR_NODE_MEMBLKS ];nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; +nodemask_t memory_nodes_parsed__initdata ; +nodemask_t processor_nodes_parsed__initdata ; +bool numa_off;/* xen/include/xen/nodemask.h +0
-0
xen/include/xen/nodemask.h static inline int __cycle_node(int n, const nodemask_t *maskp, int nbits) · line 283
Quoted 8 lines
xen/include/xen/nodemask.h +2
-0@@ -276,6 +276,8 @@ static inline int __cycle_node(int n, const nodemask_t *maskp, int nbits)*/ extern nodemask_t node_online_map; +extern nodemask_t memory_nodes_parsed; +extern nodemask_t processor_nodes_parsed;#if MAX_NUMNODES >1 #define num_online_nodes () nodes_weight(node_online_map)
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 30/40] xen: move NUMA memory and CPU parsed nodemasks to common
Hi Juilien,
Quoted 16 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 1:17
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 30/40] xen: move NUMA memory and CPU parsed nodemasks to commonHi Wei,
Quoted 3 lines · Wei Chen
Both
memory_nodes_parsedandprocessor_nodes_parsedare using for Arm and x86 to record parded NUMA memory and CPU. So we move them to common.
Quoted 7 lines
Looking at the usage, they both call:
numa_set...(..., bitmap)
So rather than exporting the two helpers, could we simply add helpers to abstract it?
I will try to fix it in next version.
Quoted 18 lines
Review
Quoted 11 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 2 -- xen/arch/x86/srat.c | 3 --- xen/common/numa.c | 3 +++ xen/include/xen/nodemask.h | 2 ++ 4 files changed, 5 insertions(+), 5 deletions(-)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cQuoted 6 lines
index
27ffb72f7b..f74b7f6427100644xen/arch/arm/numa_device_tree.c +0
-0@@ -25,8 +25,6 @@#include <asm/setup.h>
Quoted 8 lines
xen/arch/arm/numa_device_tree.c line 29
Quoted 3 lines
xen/arch/arm/numa_device_tree.c +0
-2s8 device_tree_numa =0 ; -static nodemask_t processor_nodes_parsed__initdata ; -static nodemask_t memory_nodes_parsed__initdata ;This is code that was introduced in a previous patch. In general, it is better to do the rework first and then add the new code. This makes easier to follow series as the code added is not changed.
Yes, I will fix it in next version.
Quoted 51 lines
xen/include/xen/nodemask.h static inline int __cycle_node(int n, const
Quoted 36 lines
xen/arch/arm/numa_device_tree.c +0
-0static int srat_disabled (void ) {xen/arch/x86/srat.c +0
-3@@ -24,9 +24,6 @@static struct acpi_table_slit *__read_mostly acpi_slit; -static nodemask_t memory_nodes_parsed__initdata ; -static nodemask_t processor_nodes_parsed__initdata ; -struct pxm2node {unsigned pxm;nodeid_t node;xen/common/numa.c +3
-0@@ -35,6 +35,9 @@ int num_node_memblks;struct node node_memblk_range[NR_NODE_MEMBLKS ];nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; +nodemask_t memory_nodes_parsed__initdata ; +nodemask_t processor_nodes_parsed__initdata ; +bool numa_off;/* xen/include/xen/nodemask.h +0
-0@@ -276,6 +276,8 @@ static inline int __cycle_node(int n, const
nodemask_t*maskp, int nbits)xen/include/xen/nodemask.h static inline int __cycle_node(int n, const nodemask_t *maskp, int nbits) · line 283
Quoted 9 lines
xen/include/xen/nodemask.h +2
-0*/ extern nodemask_t node_online_map; +extern nodemask_t memory_nodes_parsed; +extern nodemask_t processor_nodes_parsed;#if MAX_NUMNODES >1 #define num_online_nodes () nodes_weight(node_online_map)Cheers,
--
Julien Grall
RFC PATCH 31/40 xen/x86: move nodes_cover_memory to common
In reply to Wei ChenNot only ACPU NUMA, but also Arm device tree based NUMA will use nodes_cover_memory to do sanity check. So we move this function from arch/x86 to common.
--- xen/arch/x86/srat.c | 40 ---------------------------------------- xen/common/numa.c | 40 ++++++++++++++++++++++++++++++++++++++++ xen/include/xen/numa.h | 1 + 3 files changed, 41 insertions(+), 40 deletions(-)
xen/arch/x86/srat.c +0 -40
@@ -308,46 +308,6 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) num_node_memblks++; } -/* Sanity check to catch more bad SRATs (they are amazingly common). -Make sure the PXMs cover all memory. */ -static int __init nodes_cover_memory (void ) -{ -int i; -uint32_t nr_banks =arch_meminfo_get_nr_bank (); - -for (i =0 ; i < nr_banks; i++) { -int j, found; -unsigned long long start, end; - -if (arch_meminfo_get_ram_bank_range (i, &start, &end)) { -continue ; - } - -do { - found =0 ; -for_each_node_mask (j, memory_nodes_parsed) -if (start < nodes[j].end - && end > nodes[j].start) { -if (start >= nodes[j].start) { - start = nodes[j].end; - found =1 ; - } -if (end <= nodes[j].end) { - end = nodes[j].start; - found =1 ; - } - } - }while (found && start < end); - -if (start < end) { -printk (KERN_ERR "SRAT: No PXM for e820 range: " -"%016Lx - %016Lx\n" , start, end); -return 0 ; - } - } -return 1 ; -} -void __init acpi_numa_arch_fixup (void ) {}static uint64_t __initdata srat_region_mask;
xen/common/numa.c +40 -0
@@ -193,6 +193,46 @@ void __init cutoff_node(int i, u64 start, u64 end) } } +/* Sanity check to catch more bad SRATs (they are amazingly common). +Make sure the PXMs cover all memory. */ +int __init nodes_cover_memory (void ) +{ +int i; +uint32_t nr_banks =arch_meminfo_get_nr_bank (); + +for (i =0 ; i < nr_banks; i++) { +int j, found; +unsigned long long start, end; + +if (arch_meminfo_get_ram_bank_range (i, &start, &end)) { +continue ; + } + +do { + found =0 ; +for_each_node_mask (j, memory_nodes_parsed) +if (start < nodes[j].end + && end > nodes[j].start) { +if (start >= nodes[j].start) { + start = nodes[j].end; + found =1 ; + } +if (end <= nodes[j].end) { + end = nodes[j].start; + found =1 ; + } + } + }while (found && start < end); + +if (start < end) { +printk (KERN_ERR "SRAT: No PXM for e820 range: " +"%016Lx - %016Lx\n" , start, end); +return 0 ; + } + } +return 1 ; +} +void numa_add_cpu (int cpu) {cpumask_set_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]);
xen/include/xen/numa.h +1 -0
@@ -92,6 +92,7 @@ static inline void clear_node_cpumask(int cpu)extern uint32_t arch_meminfo_get_nr_bank (void );extern int arch_meminfo_get_ram_bank_range (int bank,unsigned long long *start,unsigned long long *end); +extern int nodes_cover_memory (void );#endif /* CONFIG_NUMA */
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 31/40] xen/x86: move nodes_cover_memory to common
Quoted 94 lines · Wei Chen
Not only ACPU NUMA, but also Arm device tree based NUMA will use
nodes_cover_memoryto do sanity check. So we move this function fromarch/x86to common.Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/srat.c | 40 ---------------------------------------- xen/common/numa.c | 40 ++++++++++++++++++++++++++++++++++++++++ xen/include/xen/numa.h | 1 + 3 files changed, 41 insertions(+), 40 deletions(-)xen/arch/x86/srat.c +0
-40@@ -308,46 +308,6 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) num_node_memblks++; } -/* Sanity check to catch more bad SRATs (they are amazingly common). -Make sure the PXMs cover all memory. */ -static int __init nodes_cover_memory (void ) -{ -int i; -uint32_t nr_banks =arch_meminfo_get_nr_bank (); - -for (i =0 ; i < nr_banks; i++) { -int j, found; -unsigned long long start, end; - -if (arch_meminfo_get_ram_bank_range (i, &start, &end)) { -continue ; - } - -do { - found =0 ; -for_each_node_mask (j, memory_nodes_parsed) -if (start < nodes[j].end - && end > nodes[j].start) { -if (start >= nodes[j].start) { - start = nodes[j].end; - found =1 ; - } -if (end <= nodes[j].end) { - end = nodes[j].start; - found =1 ; - } - } - }while (found && start < end); - -if (start < end) { -printk (KERN_ERR "SRAT: No PXM for e820 range: " -"%016Lx - %016Lx\n" , start, end); -return 0 ; - } - } -return 1 ; -} -void __init acpi_numa_arch_fixup (void ) {}static uint64_t __initdata srat_region_mask;xen/common/numa.c +24
-0@@ -193,6 +193,46 @@ void __init cutoff_node(int i, u64 start, u64 end) } } +/* Sanity check to catch more bad SRATs (they are amazingly common). +Make sure the PXMs cover all memory. */ +int __init nodes_cover_memory (void ) +{ +int i; +uint32_t nr_banks =arch_meminfo_get_nr_bank (); + +for (i =0 ; i < nr_banks; i++) { +int j, found; +unsigned long long start, end; + +if (arch_meminfo_get_ram_bank_range (i, &start, &end)) { +continue ; + } + + do { + found = 0; + for_each_node_mask(j, memory_nodes_parsed) + if (start < nodes[j].end + && end > nodes[j].start) { + if (start >= nodes[j].start) { + start = nodes[j].end; + found = 1; + }
xen/common/numa.c void __init cutoff_node(int i, u64 start, u64 end) · line 229
Quoted 10 lines
xen/common/numa.c +10
-0+if (end <= nodes[j].end) { + end = nodes[j].start; + found =1 ; + } + } + }while (found && start < end); + +if (start < end) { +printk (KERN_ERR "SRAT: No PXM for e820 range: " +"%016Lx - %016Lx\n" , start, end);
I don't know if you are already doing this in a later patch but the message shouldn't say e820 as it doesn't exist on all architecture. Maybe "for address range" or "for memory range" would suffice.
Normally we don't do change together with code movement but in this case I think it would be OK.
RE: [XEN RFC PATCH 31/40] xen/x86: move nodes_cover_memory to common
Hi Stefano,
Quoted 120 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 9:17
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 31/40] xen/x86: movenodes_cover_memoryto commonxen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct
Quoted 16 lines · Wei Chen
Not only ACPU NUMA, but also Arm device tree based NUMA will use
nodes_cover_memoryto do sanity check. So we move this function fromarch/x86to common.Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/srat.c | 40 ---------------------------------------- xen/common/numa.c | 40 ++++++++++++++++++++++++++++++++++++++++ xen/include/xen/numa.h | 1 + 3 files changed, 41 insertions(+), 40 deletions(-)xen/arch/x86/srat.c +0
-0@@ -308,46 +308,6 @@ acpi_numa_memory_affinity_init(const struct
acpi_srat_mem_affinity*ma)Quoted 78 lines
xen/arch/x86/srat.c +0
-40num_node_memblks++; } -/* Sanity check to catch more bad SRATs (they are amazingly common). -Make sure the PXMs cover all memory. */ -static int __init nodes_cover_memory (void ) -{ -int i; -uint32_t nr_banks =arch_meminfo_get_nr_bank (); - -for (i =0 ; i < nr_banks; i++) { -int j, found; -unsigned long long start, end; - -if (arch_meminfo_get_ram_bank_range (i, &start, &end)) { -continue ; - } - -do { - found =0 ; -for_each_node_mask (j, memory_nodes_parsed) -if (start < nodes[j].end - && end > nodes[j].start) { -if (start >= nodes[j].start) { - start = nodes[j].end; - found =1 ; - } -if (end <= nodes[j].end) { - end = nodes[j].start; - found =1 ; - } - } - }while (found && start < end); - -if (start < end) { -printk (KERN_ERR "SRAT: No PXM for e820 range: " -"%016Lx - %016Lx\n" , start, end); -return 0 ; - } - } -return 1 ; -} -void __init acpi_numa_arch_fixup (void ) {}static uint64_t __initdata srat_region_mask;xen/common/numa.c +24
-0@@ -193,6 +193,46 @@ void __init cutoff_node(int i, u64 start, u64 end) } } +/* Sanity check to catch more bad SRATs (they are amazingly common). +Make sure the PXMs cover all memory. */ +int __init nodes_cover_memory (void ) +{ +int i; +uint32_t nr_banks =arch_meminfo_get_nr_bank (); + +for (i =0 ; i < nr_banks; i++) { +int j, found; +unsigned long long start, end; + +if (arch_meminfo_get_ram_bank_range (i, &start, &end)) { +continue ; + } + + do { + found = 0; + for_each_node_mask(j, memory_nodes_parsed) + if (start < nodes[j].end + && end > nodes[j].start) { + if (start >= nodes[j].start) { + start = nodes[j].end; + found = 1; + }xen/common/numa.c void __init cutoff_node(int i, u64 start, u64 end) · line 229
Quoted 10 lines
xen/common/numa.c +10
-0+if (end <= nodes[j].end) { + end = nodes[j].start; + found =1 ; + } + } + }while (found && start < end); + +if (start < end) { +printk (KERN_ERR "SRAT: No PXM for e820 range: " +"%016Lx - %016Lx\n" , start, end);I don't know if you are already doing this in a later patch but the message shouldn't say e820 as it doesn't exist on all architecture. Maybe "for address range" or "for memory range" would suffice.
Quoted 2 lines
Normally we don't do change together with code movement but in this case I think it would be OK.
OK, I will do it in next version.
RFC PATCH 32/40 xen/x86: make acpi_scan_nodes to be neutral
In reply to Wei ChenThe code in acpi_scan_nodes can be reused for device tree based NUMA. So we rename acpi_scan_nodes to numa_scan_nodes for a neutral function name. As acpi_numa variable is available in ACPU based NUMA system only, we use CONFIG_ACPI_NUMA to protect it.
--- xen/arch/x86/srat.c | 4 +++- xen/common/numa.c | 2 +- xen/include/asm-x86/acpi.h | 2 +- 3 files changed, 5 insertions(+), 3 deletions(-)
xen/arch/x86/srat.c +3 -1
@@ -362,7 +362,7 @@ void __init srat_parse_regions(u64 addr) }/* Use the information discovered above to actually set up the nodes. */ -int __init acpi_scan_nodes (u64 start,u64 end) +int __init numa_scan_nodes (u64 start,u64 end) {int i;nodemask_t all_nodes_parsed;
@@ -371,8 +371,10 @@ int __init acpi_scan_nodes(u64 start, u64 end)for (i =0 ; i <MAX_NUMNODES ; i++)cutoff_node (i, start, end); +#ifdef CONFIG_ACPI_NUMA if (acpi_numa <=0 )return -1 ; +#endif if (!nodes_cover_memory ()) {bad_srat ();
xen/common/numa.c +1 -1
@@ -330,7 +330,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)#endif #ifdef CONFIG_ACPI_NUMA -if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , +if ( !numa_off && !numa_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , (u64 )end_pfn <<PAGE_SHIFT ) )return ;#endif
xen/include/asm-x86/acpi.h +1 -1
@@ -102,7 +102,7 @@ extern unsigned long acpi_wakeup_address;#define ARCH_HAS_POWER_INIT 1extern s8 acpi_numa; -extern int acpi_scan_nodes (u64 start,u64 end); +extern int numa_scan_nodes (u64 start,u64 end);extern struct acpi_sleep_info acpi_sinfo;#define acpi_video_flags bootsym(video_flags)
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 32/40] xen/x86: make acpi_scan_nodes to be neutral
Hi Wei,
Quoted 26 lines · Wei Chen
The code in
acpi_scan_nodescan be reused for device tree based NUMA. So we renameacpi_scan_nodestonuma_scan_nodesfor a neutral function name. Asacpi_numavariable is available in ACPU based NUMA system only, we useCONFIG_ACPI_NUMAto protect it.Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/srat.c | 4 +++- xen/common/numa.c | 2 +- xen/include/asm-x86/acpi.h | 2 +- 3 files changed, 5 insertions(+), 3 deletions(-)xen/arch/x86/srat.c +1
-1@@ -362,7 +362,7 @@ void __init srat_parse_regions(u64 addr) }/* Use the information discovered above to actually set up the nodes. */ -int __init acpi_scan_nodes (u64 start,u64 end) +int __init numa_scan_nodes (u64 start,u64 end) {int i;nodemask_t all_nodes_parsed;
xen/arch/x86/srat.c int __init acpi_scan_nodes(u64 start, u64 end) · line 377
Quoted 7 lines
xen/arch/x86/srat.c +2
-0@@ -371,8 +371,10 @@ int __init acpi_scan_nodes(u64 start, u64 end)for (i =0 ; i <MAX_NUMNODES ; i++)cutoff_node (i, start, end); +#ifdef CONFIG_ACPI_NUMA if (acpi_numa <=0 )return -1 ; +#endif
Looking at the follow-up patches, I find a bit odd that there is a check for ACPI but there is none added for DT. Can you explain why?
However, I think this check is going to impair the work to support both ACPI and DT on Arm because acpi_numa would end up to be 0 so you would bail out here.
With that in mind, I think this check needs to either go away or replace by something there is firmware agnostic.
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 32/40] xen/x86: make acpi_scan_nodes to be neutral
Hi Julien,
Quoted 42 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:09
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 32/40] xen/x86: makeacpi_scan_nodesto be neutralHi Wei,
Quoted 17 lines · Wei Chen
The code in
acpi_scan_nodescan be reused for device tree based NUMA. So we renameacpi_scan_nodestonuma_scan_nodesfor a neutral function name. Asacpi_numavariable is available in ACPU based NUMA system only, we useCONFIG_ACPI_NUMAto protect it.Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/srat.c | 4 +++- xen/common/numa.c | 2 +- xen/include/asm-x86/acpi.h | 2 +- 3 files changed, 5 insertions(+), 3 deletions(-)xen/arch/x86/srat.c +0
-0xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr) · line 364
Quoted 3 lines
xen/arch/x86/srat.c +0
-0@@ -362,7 +362,7 @@ void __init srat_parse_regions(u64 addr) }/* Use the information discovered above to actually set up the nodes. */
Quoted 9 lines
xen/arch/x86/srat.c +1
-1-int __init acpi_scan_nodes (u64 start,u64 end) +int __init numa_scan_nodes (u64 start,u64 end) {int i;nodemask_t all_nodes_parsed;@@ -371,8 +371,10 @@ int __init acpi_scan_nodes(u64 start, u64 end)for (i =0 ; i <MAX_NUMNODES ; i++)cutoff_node (i, start, end);
Quoted 8 lines
xen/arch/x86/srat.c int __init acpi_scan_nodes(u64 start, u64 end) · line 377
Quoted 4 lines
xen/arch/x86/srat.c +2
-0+#ifdef CONFIG_ACPI_NUMA if (acpi_numa <=0 )return -1 ; +#endif Looking at the follow-up patches, I find a bit odd that there is a check for ACPI but there is none added for DT. Can you explain why?
Oh, I forgot DT check. And simply to add DT check here seems not a good idea. Because once, when Arm support ACPI NUMA. CONFIG_ACPI_NUMA and CONFIG_DEVICE_TREE_NUMA can be selected at the same time. But only acpi_numa or dtb_numa can be > 0.
Quoted 6 lines
However, I think this check is going to impair the work to support both ACPI and DT on Arm because
acpi_numawould end up to be 0 so you would bail out here.With that in mind, I think this check needs to either go away or replace by something there is firmware agnostic.
Yes, we have discussed about something like fw_numa before.
Quoted 5 lines
Cheers,
--
Julien Grall
RFC PATCH 33/40 xen: export bad_srat and srat_disabled to extern
In reply to Wei ChenThese two functions are architecture implementation. But common code out of arch will invoke them, so we export them to extern.
--- xen/arch/arm/numa_device_tree.c | 4 ++-- xen/arch/x86/srat.c | 2 +- xen/include/asm-x86/numa.h | 1 - xen/include/xen/numa.h | 2 ++ 4 files changed, 5 insertions(+), 4 deletions(-)
xen/arch/arm/numa_device_tree.c +2 -2
@@ -26,12 +26,12 @@s8 device_tree_numa =0 ; -static int srat_disabled (void ) +int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; } -static __init void bad_srat (void ) +__init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ;
xen/arch/x86/srat.c +1 -1
@@ -97,7 +97,7 @@ nodeid_t setup_node(unsigned pxm)return node; } -static __init void bad_srat (void ) +__init void bad_srat (void ) {int i;printk (KERN_ERR "SRAT: SRAT not used.\n" );
xen/include/asm-x86/numa.h +0 -1
@@ -13,7 +13,6 @@ extern nodeid_t pxm_to_node(unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT)) -extern int srat_disabled (void );extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu);
xen/include/xen/numa.h +2 -0
@@ -75,6 +75,8 @@ extern int compute_hash_shift(struct node *nodes, int numnodes,extern int conflicting_memblks (u64 start,u64 end);extern void cutoff_node (int i,u64 start,u64 end);extern int valid_numa_range (u64 start,u64 end,nodeid_t node); +extern int srat_disabled (void ); +extern void bad_srat (void );extern void numa_init_array (void );extern void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn);
Signature · 1 line
2.25.1
RFC PATCH 34/40 xen: move numa_scan_nodes from x86 to common
In reply to Wei ChenAfter the previous patches preparations, numa_scan_nodes can be used by Arm and x86. So we move this function from x86 to common. As node_cover_memory will not be used cross files, we restore its static attribute in this patch.
--- xen/arch/x86/srat.c | 52 ------------------------------------ xen/common/numa.c | 54 +++++++++++++++++++++++++++++++++++++- xen/include/asm-x86/acpi.h | 3 --- xen/include/xen/numa.h | 3 ++- 4 files changed, 55 insertions(+), 57 deletions(-)
xen/arch/x86/srat.c +0 -52
@@ -361,58 +361,6 @@ void __init srat_parse_regions(u64 addr)pfn_pdx_hole_setup (mask >>PAGE_SHIFT ); } -/* Use the information discovered above to actually set up the nodes. */ -int __init numa_scan_nodes (u64 start,u64 end) -{ -int i; -nodemask_t all_nodes_parsed; - -/* First clean up the node list */ -for (i =0 ; i <MAX_NUMNODES ; i++) -cutoff_node (i, start, end); - -#ifdef CONFIG_ACPI_NUMA -if (acpi_numa <=0 ) -return -1 ; -#endif - -if (!nodes_cover_memory ()) { -bad_srat (); -return -1 ; - } - - memnode_shift =compute_hash_shift (node_memblk_range, num_node_memblks, - memblk_nodeid); - -if (memnode_shift <0 ) { -printk (KERN_ERR -"SRAT: No NUMA node hash function found. Contact maintainer\n" ); -bad_srat (); -return -1 ; - } - -nodes_or (all_nodes_parsed, memory_nodes_parsed, processor_nodes_parsed); - -/* Finally register nodes */ -for_each_node_mask (i, all_nodes_parsed) - { -u64 size = nodes[i].end - nodes[i].start; -if ( size ==0 ) -printk (KERN_WARNING "SRAT: Node %u has no memory. " -"BIOS Bug or mis-configured hardware?\n" , i); - -setup_node_bootmem (i, nodes[i].start, nodes[i].end); - } -for (i =0 ; i < nr_cpu_ids; i++) { -if (cpu_to_node[i] ==NUMA_NO_NODE ) -continue ; -if (!nodemask_test (cpu_to_node[i], &processor_nodes_parsed)) -numa_set_node (i,NUMA_NO_NODE ); - } -numa_init_array (); -return 0 ; -} -static unsigned node_to_pxm (nodeid_t n) {unsigned i;
xen/common/numa.c +53 -1
66 lines not on this page · raw message
xen/include/asm-x86/acpi.h +0 -3
@@ -101,9 +101,6 @@ extern unsigned long acpi_wakeup_address;#define ARCH_HAS_POWER_INIT 1 -extern s8 acpi_numa; -extern int numa_scan_nodes (u64 start,u64 end); -extern struct acpi_sleep_info acpi_sinfo;#define acpi_video_flags bootsym(video_flags)struct xenpf_enter_acpi_sleep ;
xen/include/xen/numa.h +2 -1
@@ -81,8 +81,10 @@ extern void bad_srat(void);extern void numa_init_array (void );extern void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn);extern void numa_set_node (int cpu,nodeid_t node); +extern int numa_scan_nodes (u64 start,u64 end);extern bool numa_off;extern int numa_fake; +extern s8 acpi_numa;extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end);
@@ -94,7 +96,6 @@ static inline void clear_node_cpumask(int cpu)extern uint32_t arch_meminfo_get_nr_bank (void );extern int arch_meminfo_get_ram_bank_range (int bank,unsigned long long *start,unsigned long long *end); -extern int nodes_cover_memory (void );#endif /* CONFIG_NUMA */
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 34/40] xen: move numa_scan_nodes from x86 to common
Hi Wei,
Quoted 19 lines · Wei Chen
xen/include/asm-x86/acpi.h +0
-3@@ -101,9 +101,6 @@ extern unsigned long acpi_wakeup_address;#define ARCH_HAS_POWER_INIT 1 -extern s8 acpi_numa; -extern int numa_scan_nodes (u64 start,u64 end); -extern struct acpi_sleep_info acpi_sinfo;#define acpi_video_flags bootsym(video_flags)struct xenpf_enter_acpi_sleep ;xen/include/xen/numa.h +0
-0
xen/include/xen/numa.h extern void bad_srat(void); · line 84
Quoted 4 lines
xen/include/xen/numa.h +1
-0@@ -81,8 +81,10 @@ extern void bad_srat(void);extern void numa_init_array (void );extern void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn);extern void numa_set_node (int cpu,nodeid_t node); +extern int numa_scan_nodes (u64 start,u64 end);
AFAICT, by the end of the series, the function is only called by the common code. So this should be static.
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 34/40] xen: move numa_scan_nodes from x86 to common
Hi Julien,
Quoted 34 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:14
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 34/40] xen: movenuma_scan_nodesfrom x86 to commonHi Wei,
Quoted 19 lines · Wei Chen
xen/include/asm-x86/acpi.h +0
-3@@ -101,9 +101,6 @@ extern unsigned long acpi_wakeup_address;#define ARCH_HAS_POWER_INIT 1 -extern s8 acpi_numa; -extern int numa_scan_nodes (u64 start,u64 end); -extern struct acpi_sleep_info acpi_sinfo;#define acpi_video_flags bootsym(video_flags)struct xenpf_enter_acpi_sleep ;xen/include/xen/numa.h +0
-0xen/include/xen/numa.h extern void bad_srat(void); · line 82
Quoted 2 lines
xen/include/xen/numa.h +0
-0@@ -81,8 +81,10 @@ extern void bad_srat(void);extern void numa_init_array (void ); extern void numa_initmem_init(unsigned long start_pfn, unsigned long
end_pfn);
Quoted 6 lines
xen/include/xen/numa.h extern void bad_srat(void); · line 84
Quoted 2 lines
xen/include/xen/numa.h +1
-0extern void numa_set_node (int cpu,nodeid_t node); +extern int numa_scan_nodes (u64 start,u64 end);AFAICT, by the end of the series, the function is only called by the common code. So this should be static.
OK
Quoted 4 lines
Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 34/40] xen: move numa_scan_nodes from x86 to common
In reply to Wei ChenQuoted 122 lines · Wei Chen
After the previous patches preparations,
numa_scan_nodescan be used by Arm and x86. So we move this function from x86 to common. Asnode_cover_memorywill not be used cross files, we restore its static attribute in this patch.Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/srat.c | 52 ------------------------------------ xen/common/numa.c | 54 +++++++++++++++++++++++++++++++++++++- xen/include/asm-x86/acpi.h | 3 --- xen/include/xen/numa.h | 3 ++- 4 files changed, 55 insertions(+), 57 deletions(-)xen/arch/x86/srat.c +0
-52@@ -361,58 +361,6 @@ void __init srat_parse_regions(u64 addr)pfn_pdx_hole_setup (mask >>PAGE_SHIFT ); } -/* Use the information discovered above to actually set up the nodes. */ -int __init numa_scan_nodes (u64 start,u64 end) -{ -int i; -nodemask_t all_nodes_parsed; - -/* First clean up the node list */ -for (i =0 ; i <MAX_NUMNODES ; i++) -cutoff_node (i, start, end); - -#ifdef CONFIG_ACPI_NUMA -if (acpi_numa <=0 ) -return -1 ; -#endif - -if (!nodes_cover_memory ()) { -bad_srat (); -return -1 ; - } - - memnode_shift =compute_hash_shift (node_memblk_range, num_node_memblks, - memblk_nodeid); - -if (memnode_shift <0 ) { -printk (KERN_ERR -"SRAT: No NUMA node hash function found. Contact maintainer\n" ); -bad_srat (); -return -1 ; - } - -nodes_or (all_nodes_parsed, memory_nodes_parsed, processor_nodes_parsed); - -/* Finally register nodes */ -for_each_node_mask (i, all_nodes_parsed) - { -u64 size = nodes[i].end - nodes[i].start; -if ( size ==0 ) -printk (KERN_WARNING "SRAT: Node %u has no memory. " -"BIOS Bug or mis-configured hardware?\n" , i); - -setup_node_bootmem (i, nodes[i].start, nodes[i].end); - } -for (i =0 ; i < nr_cpu_ids; i++) { -if (cpu_to_node[i] ==NUMA_NO_NODE ) -continue ; -if (!nodemask_test (cpu_to_node[i], &processor_nodes_parsed)) -numa_set_node (i,NUMA_NO_NODE ); - } -numa_init_array (); -return 0 ; -} -static unsigned node_to_pxm (nodeid_t n) {unsigned i;xen/common/numa.c +30
-1@@ -195,7 +195,7 @@ void __init cutoff_node(int i, u64 start, u64 end)/* Sanity check to catch more bad SRATs (they are amazingly common). Make sure the PXMs cover all memory. */ -int __init nodes_cover_memory (void ) +static int __init nodes_cover_memory (void ) {int i;uint32_t nr_banks =arch_meminfo_get_nr_bank ();@@ -271,6 +271,58 @@ void __init numa_init_array(void) } } +/* Use the information discovered above to actually set up the nodes. */ +int __init numa_scan_nodes (u64 start,u64 end) +{ +int i; +nodemask_t all_nodes_parsed; + +/* First clean up the node list */ +for (i =0 ; i <MAX_NUMNODES ; i++) +cutoff_node (i, start, end); + +#ifdef CONFIG_ACPI_NUMA +if (acpi_numa <=0 ) +return -1 ; +#endif + +if (!nodes_cover_memory ()) { +bad_srat (); +return -1 ; + } + + memnode_shift =compute_hash_shift (node_memblk_range, num_node_memblks, + memblk_nodeid); + +if (memnode_shift <0 ) { +printk (KERN_ERR +"SRAT: No NUMA node hash function found. Contact maintainer\n" ); +bad_srat (); +return -1 ; + }
xen/common/numa.c void __init numa_init_array(void) · line 312
Quoted 10 lines
xen/common/numa.c +10
-0+ +nodes_or (all_nodes_parsed, memory_nodes_parsed, processor_nodes_parsed); + +/* Finally register nodes */ +for_each_node_mask (i, all_nodes_parsed) + { +u64 size = nodes[i].end - nodes[i].start; +if ( size ==0 ) +printk (KERN_WARNING "SRAT: Node %u has no memory. " +"BIOS Bug or mis-configured hardware?\n" , i);
Not all archs have a BIOS so I'd say "firmware bug". Like last time, we usually don't do code changes together with code movement, but in this case it might be OK. I am also happy with a separate patch to adjust the two comments (this one and the one in the previous patch).
RE: [XEN RFC PATCH 34/40] xen: move numa_scan_nodes from x86 to common
Hi Stefano,
Quoted 142 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 9:27
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 34/40] xen: movenuma_scan_nodesfrom x86 to commonQuoted 18 lines · Wei Chen
After the previous patches preparations,
numa_scan_nodescan be used by Arm and x86. So we move this function from x86 to common. Asnode_cover_memorywill not be used cross files, we restore its static attribute in this patch.Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/srat.c | 52 ------------------------------------ xen/common/numa.c | 54 +++++++++++++++++++++++++++++++++++++- xen/include/asm-x86/acpi.h | 3 --- xen/include/xen/numa.h | 3 ++- 4 files changed, 55 insertions(+), 57 deletions(-)xen/arch/x86/srat.c +0
-0xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr) · line 364
Quoted 4 lines
xen/arch/x86/srat.c +0
-1@@ -361,58 +361,6 @@ void __init srat_parse_regions(u64 addr)pfn_pdx_hole_setup (mask >>PAGE_SHIFT ); } -/* Use the information discovered above to actually set up the nodes. */
Quoted 10 lines
xen/arch/x86/srat.c +0
-10-int __init numa_scan_nodes (u64 start,u64 end) -{ -int i; -nodemask_t all_nodes_parsed; - -/* First clean up the node list */ -for (i =0 ; i <MAX_NUMNODES ; i++) -cutoff_node (i, start, end); - -#ifdef CONFIG_ACPI_NUMA xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr)
Quoted 10 lines
xen/arch/x86/srat.c +0
-10-if (acpi_numa <=0 ) -return -1 ; -#endif - -if (!nodes_cover_memory ()) { -bad_srat (); -return -1 ; - } - - memnode_shift = compute_hash_shift(node_memblk_range,
num_node_memblks,xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr)
Quoted 5 lines
xen/arch/x86/srat.c +0
-5- memblk_nodeid); - - if (memnode_shift < 0) { - printk(KERN_ERR - "SRAT: No NUMA node hash function found. Contactmaintainer\n");
xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr)
Quoted 5 lines
xen/arch/x86/srat.c +0
-5-bad_srat (); -return -1 ; - } - - nodes_or(all_nodes_parsed, memory_nodes_parsed,
processor_nodes_parsed);Quoted 38 lines
xen/arch/x86/srat.c +0
-21- -/* Finally register nodes */ -for_each_node_mask (i, all_nodes_parsed) - { -u64 size = nodes[i].end - nodes[i].start; -if ( size ==0 ) -printk (KERN_WARNING "SRAT: Node %u has no memory. " -"BIOS Bug or mis-configured hardware?\n" , i); - -setup_node_bootmem (i, nodes[i].start, nodes[i].end); - } -for (i =0 ; i < nr_cpu_ids; i++) { -if (cpu_to_node[i] ==NUMA_NO_NODE ) -continue ; -if (!nodemask_test (cpu_to_node[i], &processor_nodes_parsed)) -numa_set_node (i,NUMA_NO_NODE ); - } -numa_init_array (); -return 0 ; -} -static unsigned node_to_pxm (nodeid_t n) {unsigned i;xen/common/numa.c +1
-1@@ -195,7 +195,7 @@ void __init cutoff_node(int i, u64 start, u64 end)/* Sanity check to catch more bad SRATs (they are amazingly common). Make sure the PXMs cover all memory. */ -int __init nodes_cover_memory (void ) +static int __init nodes_cover_memory (void ) {int i;uint32_t nr_banks =arch_meminfo_get_nr_bank ();xen/common/numa.c void __init numa_init_array(void) · line 274
Quoted 4 lines
xen/common/numa.c +1
-0@@ -271,6 +271,58 @@ void __init numa_init_array(void) } } +/* Use the information discovered above to actually set up the nodes. */
Quoted 10 lines
xen/common/numa.c +10
-0+int __init numa_scan_nodes (u64 start,u64 end) +{ +int i; +nodemask_t all_nodes_parsed; + +/* First clean up the node list */ +for (i =0 ; i <MAX_NUMNODES ; i++) +cutoff_node (i, start, end); + +#ifdef CONFIG_ACPI_NUMA xen/common/numa.c void __init numa_init_array(void)
Quoted 10 lines
xen/common/numa.c +10
-0+if (acpi_numa <=0 ) +return -1 ; +#endif + +if (!nodes_cover_memory ()) { +bad_srat (); +return -1 ; + } + + memnode_shift = compute_hash_shift(node_memblk_range,
num_node_memblks,xen/common/numa.c void __init numa_init_array(void)
Quoted 5 lines
xen/common/numa.c +5
-0+ memblk_nodeid); + + if (memnode_shift < 0) { + printk(KERN_ERR + "SRAT: No NUMA node hash function found. Contactmaintainer\n");
xen/common/numa.c void __init numa_init_array(void)
Quoted 5 lines
xen/common/numa.c +5
-0+bad_srat (); +return -1 ; + } + + nodes_or(all_nodes_parsed, memory_nodes_parsed,
processor_nodes_parsed);
Quoted 13 lines
xen/common/numa.c void __init numa_init_array(void) · line 312
Quoted 8 lines
xen/common/numa.c +8
-0+ +/* Finally register nodes */ +for_each_node_mask (i, all_nodes_parsed) + { +u64 size = nodes[i].end - nodes[i].start; +if ( size ==0 ) +printk (KERN_WARNING "SRAT: Node %u has no memory. " +"BIOS Bug or mis-configured hardware?\n" , i);Not all archs have a BIOS so I'd say "firmware bug". Like last time, we usually don't do code changes together with code movement, but in this case it might be OK. I am also happy with a separate patch to adjust the two comments (this one and the one in the previous patch).
OK, I will do it.
RFC PATCH 35/40 xen: enable numa_scan_nodes for device tree based NUMA
In reply to Wei ChenNow, we can use the same function for ACPI and device tree based NUMA to scan memory nodes.
--- xen/common/numa.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
xen/common/numa.c +1 -1
@@ -381,7 +381,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)return ;#endif -#ifdef CONFIG_ACPI_NUMA +#if defined(CONFIG_ACPI_NUMA ) || defined(CONFIG_DEVICE_TREE_NUMA )if ( !numa_off && !numa_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , (u64 )end_pfn <<PAGE_SHIFT ) )return ;
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 35/40] xen: enable numa_scan_nodes for device tree based NUMA
Hi Wei,
Quoted 13 lines · Wei Chen
Now, we can use the same function for ACPI and device tree based NUMA to scan memory nodes.
Signed-off-by Wei Chen <[email protected]>--- xen/common/numa.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)xen/common/numa.c +0
-0
xen/common/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 384
Quoted 5 lines
xen/common/numa.c +1
-1@@ -381,7 +381,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)return ;#endif -#ifdef CONFIG_ACPI_NUMA +#if defined(CONFIG_ACPI_NUMA ) || defined(CONFIG_DEVICE_TREE_NUMA )
numa.c is only built when CONFIG_NUMA is set. I don't think CONFIG_NUMA will ever set if neither CONFIG_ACPI_NUMA or CONFIG_DEVICE_TREE_NUMA is set. So do we actually need this #ifdef?
xen/common/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 387
Quoted 4 lines
xen/common/numa.c +0
-0if ( !numa_off && !numa_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , (u64 )end_pfn <<PAGE_SHIFT ) )return ;
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 35/40] xen: enable numa_scan_nodes for device tree based NUMA
Hi Julien,
Quoted 29 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:19
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 35/40] xen: enablenuma_scan_nodesfor device tree based NUMAHi Wei,
xen/common/numa.c void __init numa_initmem_init(unsigned long
Quoted 13 lines · Wei Chen
Now, we can use the same function for ACPI and device tree based NUMA to scan memory nodes.
Signed-off-by Wei Chen <[email protected]>--- xen/common/numa.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)xen/common/numa.c +0
-0@@ -381,7 +381,7 @@ void __init numa_initmem_init(unsigned long
start_pfn, unsigned longend_pfn)Quoted 3 lines
xen/common/numa.c +0
-0return ;#endif
Quoted 7 lines
xen/common/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 384
Quoted 2 lines
xen/common/numa.c +1
-1-#ifdef CONFIG_ACPI_NUMA +#if defined(CONFIG_ACPI_NUMA ) || defined(CONFIG_DEVICE_TREE_NUMA )
numa.cis only built whenCONFIG_NUMAis set. I don't thinkCONFIG_NUMAwill ever set if neitherCONFIG_ACPI_NUMAorCONFIG_DEVICE_TREE_NUMAis set. So do we actually need this #ifdef?
Yes, you're right. This check should be removed.
Quoted 9 lines
xen/common/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 387
Quoted 4 lines
xen/common/numa.c +0
-0if ( !numa_off && !numa_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , (u64 )end_pfn <<PAGE_SHIFT ) )return ;Cheers,
--
Julien Grall
RFC PATCH 36/40 xen/arm: keep guest still be NUMA unware
In reply to Wei ChenWe have not wanted to make Xen guest be NUMA aware in this patch series. So in this patch, Xen will skip NUMA distance matrix node and skip the numa-node-id property in CPU node and memory node, when Xen is creating guest device tree binary.
--- xen/arch/arm/domain_build.c | 6 ++++++ 1 file changed, 6 insertions(+)
xen/arch/arm/domain_build.c +6 -0
@@ -584,6 +584,10 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo,continue ; } +/* Guest is numa unaware in current stage */ +if (dt_property_name_is_equal (prop,"numa-node-id" ) ) +continue ; + res =fdt_property (kinfo->fdt, prop->name, prop_data, prop_len);if ( res )
@@ -1454,6 +1458,8 @@ static int __init handle_node(struct domain *d, struct kernel_info *kinfo,DT_MATCH_TYPE ("memory" ),/* The memory mapped timer is not supported by Xen. */ DT_MATCH_COMPATIBLE ("arm,armv7-timer-mem" ), +/* Numa info doesn't need to be exposed to Domain-0 */ + DT_MATCH_COMPATIBLE("numa-distance-map-v1"), { /* sentinel */ }, }; static const struct dt_device_match timer_matches[] __initconst =
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 36/40] xen/arm: keep guest still be NUMA unware
Hi Wei,
Quoted 2 lines · Wei Chen
We have not wanted to make Xen guest be NUMA aware in this patch series.
The concept of patch series ceases to exist once we merge the code. So about how:
"The NUMA information provided in the host Device-Tree are only for Xen. For dom0, we want to hide them as they may be different (for now, dom0 is still not aware of NUMA".
Quoted 3 lines
So in this patch, Xen will skip NUMA distance matrix node and skip the numa-node-id property in CPU node and memory node, when Xen is creating guest device tree binary.
The CPU and memory nodes are recreated from scratch for the domain. So we already skip the property numa-node-id. However...
Quoted 10 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/domain_build.c | 6 ++++++ 1 file changed, 6 insertions(+)xen/arch/arm/domain_build.c +0
-0
xen/arch/arm/domain_build.c static int __init write_properties(struct domain *d, struct kernel_info *kinfo, · line 589
Quoted 6 lines
xen/arch/arm/domain_build.c +3
-0@@ -584,6 +584,10 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo,continue ; } +/* Guest is numa unaware in current stage */ +if (dt_property_name_is_equal (prop,"numa-node-id" ) ) +continue ;
... your code is doing more than skipping the property for the two nodes you mentionned. Can the property exists in other nodes?
Quoted 6 lines
xen/arch/arm/domain_build.c +1
-0+ res =fdt_property (kinfo->fdt, prop->name, prop_data, prop_len);if ( res )
xen/arch/arm/domain_build.c static int __init handle_node(struct domain *d, struct kernel_info *kinfo, · line 1465
Quoted 8 lines
xen/arch/arm/domain_build.c +2
-0@@ -1454,6 +1458,8 @@ static int __init handle_node(struct domain *d, struct kernel_info *kinfo,DT_MATCH_TYPE ("memory" ),/* The memory mapped timer is not supported by Xen. */ DT_MATCH_COMPATIBLE ("arm,armv7-timer-mem" ), +/* Numa info doesn't need to be exposed to Domain-0 */ + DT_MATCH_COMPATIBLE("numa-distance-map-v1"), { /* sentinel */ }, }; static const struct dt_device_match timer_matches[] __initconst =
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 36/40] xen/arm: keep guest still be NUMA unware
Hi Julien,
Quoted 15 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:28
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 36/40] xen/arm: keep guest still be NUMA unwareHi Wei,
Quoted 2 lines · Wei Chen
We have not wanted to make Xen guest be NUMA aware in this patch series.
Quoted 7 lines
The concept of patch series ceases to exist once we merge the code. So about how:
"The NUMA information provided in the host Device-Tree are only for Xen. For dom0, we want to hide them as they may be different (for now, dom0 is still not aware of NUMA".
Thanks, I will do it.
Quoted 21 lines
Quoted 3 lines
So in this patch, Xen will skip NUMA distance matrix node and skip the numa-node-id property in CPU node and memory node, when Xen is creating guest device tree binary.
The CPU and memory nodes are recreated from scratch for the domain. So we already skip the property numa-node-id. However...
xen/arch/arm/domain_build.c static int __init write_properties(struct domain *d,
Quoted 10 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/domain_build.c | 6 ++++++ 1 file changed, 6 insertions(+)xen/arch/arm/domain_build.c +0
-0@@ -584,6 +584,10 @@ static int __init write_properties(struct domain *d,struct
kernel_info*kinfo,Quoted 3 lines
xen/arch/arm/domain_build.c +0
-0continue ; }
Quoted 6 lines
xen/arch/arm/domain_build.c static int __init write_properties(struct domain *d, struct kernel_info *kinfo, · line 589
Quoted 3 lines
xen/arch/arm/domain_build.c +3
-0+/* Guest is numa unaware in current stage */ +if (dt_property_name_is_equal (prop,"numa-node-id" ) ) +continue ;... your code is doing more than skipping the property for the two nodes you mentionned. Can the property exists in other nodes?
Some devices like PCIe may have numa-node-id. In the future maybe more device can have NUMA property.
Quoted 22 lines
xen/arch/arm/domain_build.c static int __init write_properties(struct domain *d, · line 591
Quoted 3 lines
xen/arch/arm/domain_build.c +1
-0+ res = fdt_property(kinfo->fdt, prop->name, prop_data,
prop_len);xen/arch/arm/domain_build.c static int __init handle_node(struct domain *d,
Quoted 3 lines
xen/arch/arm/domain_build.c +0
-0if ( res )@@ -1454,6 +1458,8 @@ static int __init handle_node(struct domain *d,struct
kernel_info*kinfo,xen/arch/arm/domain_build.c static int __init handle_node(struct domain *d, struct kernel_info *kinfo, · line 1465
Quoted 9 lines
xen/arch/arm/domain_build.c +2
-0DT_MATCH_TYPE ("memory" ),/* The memory mapped timer is not supported by Xen. */ DT_MATCH_COMPATIBLE ("arm,armv7-timer-mem" ), +/* Numa info doesn't need to be exposed to Domain-0 */ + DT_MATCH_COMPATIBLE("numa-distance-map-v1"), { /* sentinel */ }, }; static const struct dt_device_match timer_matches[] __initconst =Cheers,
--
Julien Grall
RFC PATCH 37/40 xen: introduce an arch helper to do NUMA init failed fallback
In reply to Wei ChenWhen Xen initialize NUMA failed, some architectures may need to do fallback actions. For example, in device tree based NUMA, Arm need to reset the distance between any two nodes.
--- xen/arch/arm/numa.c | 13 +++++++++++++ xen/common/numa.c | 3 +++ xen/include/asm-arm/numa.h | 1 + xen/include/asm-x86/numa.h | 6 ++++++ 4 files changed, 23 insertions(+)
xen/arch/arm/numa.c +13 -0
@@ -140,3 +140,16 @@ int __init arch_meminfo_get_ram_bank_range(int bank,return 0 ; } + +void __init arch_numa_init_failed_fallback (void ) +{ +int i, j; + +/* Reset all node distance to remote_distance */ +for ( i =0 ; i <MAX_NUMNODES ; i++ ) { +for ( j =0 ; j <MAX_NUMNODES ; j++ ) { +numa_set_distance (i, j, + (i == j) ?NUMA_LOCAL_DISTANCE :NUMA_REMOTE_DISTANCE ); + } + } +}
xen/common/numa.c +3 -0
@@ -405,4 +405,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 ));setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , (u64 )end_pfn <<PAGE_SHIFT ); + +/* architecture specified fallback operations */ +arch_numa_init_failed_fallback (); }
xen/include/asm-arm/numa.h +1 -0
@@ -28,6 +28,7 @@ extern s8 device_tree_numa;extern void numa_init (bool acpi_off);extern int numa_device_tree_init (const void *fdt);extern void numa_set_distance (nodeid_t from,nodeid_t to,uint32_t distance); +extern void arch_numa_init_failed_fallback (void );/* * Temporary for fake NUMA node, when CPU, memory and distance
xen/include/asm-x86/numa.h +6 -0
@@ -22,4 +22,10 @@ extern void init_cpu_to_node(void);void srat_parse_regions (u64 addr);unsigned int arch_get_dma_bitsize (void ); +/* Dummy function for numa init failed in numa_initmem_init */ +static inline void arch_numa_init_failed_fallback (void ) +{ +return ; +} +#endif
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallback
Hi,
Quoted 3 lines · Wei Chen
When Xen initialize NUMA failed, some architectures may need to do fallback actions. For example, in device tree based NUMA, Arm need to reset the distance between any two nodes.
From the description here, I don't understand why we need to reset the distance for Arm but not x86. In fact...
Quoted 20 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa.c | 13 +++++++++++++ xen/common/numa.c | 3 +++ xen/include/asm-arm/numa.h | 1 + xen/include/asm-x86/numa.h | 6 ++++++ 4 files changed, 23 insertions(+)xen/arch/arm/numa.c +3
-0@@ -140,3 +140,16 @@ int __init arch_meminfo_get_ram_bank_range(int bank,return 0 ; } + +void __init arch_numa_init_failed_fallback (void ) +{
xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int bank, · line 155
Quoted 10 lines
xen/arch/arm/numa.c +10
-0+int i, j; + +/* Reset all node distance to remote_distance */ +for ( i =0 ; i <MAX_NUMNODES ; i++ ) { +for ( j =0 ; j <MAX_NUMNODES ; j++ ) { +numa_set_distance (i, j, + (i == j) ?NUMA_LOCAL_DISTANCE :NUMA_REMOTE_DISTANCE ); + } + } +}
... this implementation looks fairly generic. So can you explain why we need it on Arm but not x86?
Quoted 29 lines
xen/common/numa.c +3
-0@@ -405,4 +405,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 ));setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , (u64 )end_pfn <<PAGE_SHIFT ); + +/* architecture specified fallback operations */ +arch_numa_init_failed_fallback (); }xen/include/asm-arm/numa.h +1
-0@@ -28,6 +28,7 @@ extern s8 device_tree_numa;extern void numa_init (bool acpi_off);extern int numa_device_tree_init (const void *fdt);extern void numa_set_distance (nodeid_t from,nodeid_t to,uint32_t distance); +extern void arch_numa_init_failed_fallback (void );/* * Temporary for fake NUMA node, when CPU, memory and distance xen/include/asm-x86/numa.h +0
-0
xen/include/asm-x86/numa.h extern void init_cpu_to_node(void); · line 28
Quoted 7 lines
xen/include/asm-x86/numa.h +4
-0@@ -22,4 +22,10 @@ extern void init_cpu_to_node(void);void srat_parse_regions (u64 addr);unsigned int arch_get_dma_bitsize (void ); +/* Dummy function for numa init failed in numa_initmem_init */ +static inline void arch_numa_init_failed_fallback (void ) +{ +return ;
NIT: The return is pointless.
xen/include/asm-x86/numa.h extern void init_cpu_to_node(void); · line 31
Quoted 4 lines
xen/include/asm-x86/numa.h +2
-0+} +#endif
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallback
Hi Julien,
Quoted 44 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:30
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallbackHi,
Quoted 3 lines · Wei Chen
When Xen initialize NUMA failed, some architectures may need to do fallback actions. For example, in device tree based NUMA, Arm need to reset the distance between any two nodes.
From the description here, I don't understand why we need to reset the
distance for Arm but not x86. In fact...Quoted 25 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa.c | 13 +++++++++++++ xen/common/numa.c | 3 +++ xen/include/asm-arm/numa.h | 1 + xen/include/asm-x86/numa.h | 6 ++++++ 4 files changed, 23 insertions(+)xen/arch/arm/numa.c +8
-0@@ -140,3 +140,16 @@ int __init arch_meminfo_get_ram_bank_range(int bank,return 0 ; } + +void __init arch_numa_init_failed_fallback (void ) +{ +int i, j; + +/* Reset all node distance to remote_distance */ +for ( i =0 ; i <MAX_NUMNODES ; i++ ) { +for ( j =0 ; j <MAX_NUMNODES ; j++ ) {
Quoted 9 lines
xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int bank, · line 155
Quoted 5 lines
xen/arch/arm/numa.c +5
-0+numa_set_distance (i, j, + (i == j) ?NUMA_LOCAL_DISTANCE :NUMA_REMOTE_DISTANCE ); + } + } +}... this implementation looks fairly generic. So can you explain why we need it on Arm but not x86?
This implementation is DT only, for x86, it's using acpi_slit. For now, I am not quit sure ACPI need to do fallback or not. Or say in another way, I don't know how to implement the fallback for ACPI. I planned to solve it in Arm ACPI version NUMA, so I left an empty helper for x86.
@Jan Beulich Could you give me some suggestion about x86 fallback?
Quoted 34 lines
xen/common/numa.c void __init numa_initmem_init(unsigned long
Quoted 5 lines
xen/common/numa.c +0
-0@@ -405,4 +405,7 @@ void __init numa_initmem_init(unsigned long
start_pfn, unsigned longend_pfn)Quoted 12 lines
xen/common/numa.c +3
-0cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 ));setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , (u64 )end_pfn <<PAGE_SHIFT ); + +/* architecture specified fallback operations */ +arch_numa_init_failed_fallback (); }xen/include/asm-arm/numa.h +0
-0xen/include/asm-arm/numa.h extern s8 device_tree_numa; · line 30
Quoted 3 lines
xen/include/asm-arm/numa.h +0
-0@@ -28,6 +28,7 @@ extern s8 device_tree_numa;extern void numa_init (bool acpi_off);extern int numa_device_tree_init (const void *fdt); extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_tdistance);
Quoted 12 lines
xen/include/asm-arm/numa.h +1
-0+extern void arch_numa_init_failed_fallback (void );/* * Temporary for fake NUMA node, when CPU, memory and distance xen/include/asm-x86/numa.h +0
-0@@ -22,4 +22,10 @@ extern void init_cpu_to_node(void);void srat_parse_regions (u64 addr);unsigned int arch_get_dma_bitsize (void );
Quoted 7 lines
xen/include/asm-x86/numa.h extern void init_cpu_to_node(void); · line 28
Quoted 4 lines
xen/include/asm-x86/numa.h +4
-0+/* Dummy function for numa init failed in numa_initmem_init */ +static inline void arch_numa_init_failed_fallback (void ) +{ +return ;NIT: The return is pointless.
OK
Quoted 9 lines
xen/include/asm-x86/numa.h extern void init_cpu_to_node(void); · line 31
Quoted 4 lines
xen/include/asm-x86/numa.h +2
-0+} +#endif Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallback
Hi Julien, Jan
Quoted 68 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Wei Chen
Sent: 2021年8月28日 11:09
To: Julien Grall <[email protected]>; [email protected]; [email protected]; Jan Beulich <[email protected]>
Cc: Bertrand Marquis <[email protected]>
Subject: RE: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallbackHi Julien,
Quoted 28 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:30
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallbackHi,
Quoted 3 lines · Wei Chen
When Xen initialize NUMA failed, some architectures may need to do fallback actions. For example, in device tree based NUMA, Arm need to reset the distance between any two nodes.
From the description here, I don't understand why we need to reset the
distance for Arm but not x86. In fact...Quoted 9 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa.c | 13 +++++++++++++ xen/common/numa.c | 3 +++ xen/include/asm-arm/numa.h | 1 + xen/include/asm-x86/numa.h | 6 ++++++ 4 files changed, 23 insertions(+)Quoted 5 lines
xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int
Quoted 5 lines
xen/arch/arm/numa.c +0
-0@@ -140,3 +140,16 @@ int __init arch_meminfo_get_ram_bank_range(intbank,
Quoted 13 lines
xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int bank,
Quoted 13 lines
xen/arch/arm/numa.c +10
-0return 0 ; } + +void __init arch_numa_init_failed_fallback (void ) +{ +int i, j; + +/* Reset all node distance to remote_distance */ +for ( i =0 ; i <MAX_NUMNODES ; i++ ) { +for ( j =0 ; j <MAX_NUMNODES ; j++ ) { + numa_set_distance(i, j, + (i == j) ? NUMA_LOCAL_DISTANCE :
NUMA_REMOTE_DISTANCE);Quoted 7 lines
xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int bank, · line 155
Quoted 3 lines
xen/arch/arm/numa.c +3
-0+ } + } +}... this implementation looks fairly generic. So can you explain why we need it on Arm but not x86?
Quoted 9 lines
This implementation is DT only, for x86, it's using
acpi_slit. For now, I am not quit sure ACPI need to do fallback or not. Or say in another way, I don't know how to implement the fallback for ACPI. I planned to solve it in Arm ACPI version NUMA, so I left an empty helper for x86.@Jan Beulich Could you give me some suggestion about x86 fallback?
I have a quick look into Linux. When Arch do numa init failed, the numa_free_distance will be invoked to revert numa_distance.
Quoted 53 lines
Quoted 34 lines
xen/common/numa.c void __init numa_initmem_init(unsigned long
Quoted 5 lines
xen/common/numa.c +0
-0@@ -405,4 +405,7 @@ void __init numa_initmem_init(unsigned long
start_pfn, unsigned longend_pfn)Quoted 12 lines
xen/common/numa.c +3
-0cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 ));setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , (u64 )end_pfn <<PAGE_SHIFT ); + +/* architecture specified fallback operations */ +arch_numa_init_failed_fallback (); }xen/include/asm-arm/numa.h +0
-0xen/include/asm-arm/numa.h extern s8 device_tree_numa; · line 30
Quoted 3 lines
xen/include/asm-arm/numa.h +0
-0@@ -28,6 +28,7 @@ extern s8 device_tree_numa;extern void numa_init (bool acpi_off);extern int numa_device_tree_init (const void *fdt); extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_tdistance);
Quoted 12 lines
xen/include/asm-arm/numa.h +1
-0+extern void arch_numa_init_failed_fallback (void );/* * Temporary for fake NUMA node, when CPU, memory and distance xen/include/asm-x86/numa.h +0
-0@@ -22,4 +22,10 @@ extern void init_cpu_to_node(void);void srat_parse_regions (u64 addr);unsigned int arch_get_dma_bitsize (void );Quoted 7 lines
xen/include/asm-x86/numa.h extern void init_cpu_to_node(void); · line 28
Quoted 4 lines
xen/include/asm-x86/numa.h +4
-0+/* Dummy function for numa init failed in numa_initmem_init */ +static inline void arch_numa_init_failed_fallback (void ) +{ +return ;NIT: The return is pointless.
OK
Quoted 9 lines
xen/include/asm-x86/numa.h extern void init_cpu_to_node(void); · line 31
Quoted 4 lines
xen/include/asm-x86/numa.h +2
-0+} +#endif Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallback
Quoted 39 lines · Wei Chen
Quoted 39 lines
From: Xen-devel <[email protected]> On Behalf Of Wei Chen
Sent: 2021年8月28日 11:09Quoted 6 lines
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:30xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int
Quoted 3 lines
xen/arch/arm/numa.c +0
-0@@ -140,3 +140,16 @@ int __init arch_meminfo_get_ram_bank_range(intbank,
Quoted 13 lines
xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int bank,
Quoted 13 lines
xen/arch/arm/numa.c +10
-0return 0 ; } + +void __init arch_numa_init_failed_fallback (void ) +{ +int i, j; + +/* Reset all node distance to remote_distance */ +for ( i =0 ; i <MAX_NUMNODES ; i++ ) { +for ( j =0 ; j <MAX_NUMNODES ; j++ ) { + numa_set_distance(i, j, + (i == j) ? NUMA_LOCAL_DISTANCE :
NUMA_REMOTE_DISTANCE);Quoted 7 lines
xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int bank, · line 155
Quoted 3 lines
xen/arch/arm/numa.c +3
-0+ } + } +}... this implementation looks fairly generic. So can you explain why we need it on Arm but not x86?
This implementation is DT only, for x86, it's using
acpi_slit. For now, I am not quit sure ACPI need to do fallback or not. Or say in another way, I don't know how to implement the fallback for ACPI. I planned to solve it in Arm ACPI version NUMA, so I left an empty helper for x86.
Quoted 6 lines
Quoted 3 lines
@Jan Beulich Could you give me some suggestion about x86 fallback?
I have a quick look into Linux. When Arch do numa init failed, the
numa_free_distancewill be invoked to revertnuma_distance.
Does this matter in the first place? Don't we fall back to single node mode, in which case the sole entry of the distance table will say "local" anyway?
Jan
RE: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallback
Hi Jan,
Quoted 55 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月30日 17:52
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; Julien Grall <[email protected]>; [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallbackQuoted 1 line · Wei Chen
Quoted 1 line
From: Xen-devel <[email protected]> On Behalf Of
Wei
Quoted 24 lines
Quoted 24 lines
Chen
Sent: 2021年8月28日 11:09Quoted 6 lines
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:30xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int
Quoted 3 lines
xen/arch/arm/numa.c +0
-0@@ -140,3 +140,16 @@ int __init arch_meminfo_get_ram_bank_range(intbank,
Quoted 13 lines
xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int bank,
Quoted 13 lines
xen/arch/arm/numa.c +10
-0return 0 ; } + +void __init arch_numa_init_failed_fallback (void ) +{ +int i, j; + +/* Reset all node distance to remote_distance */ +for ( i =0 ; i <MAX_NUMNODES ; i++ ) { +for ( j =0 ; j <MAX_NUMNODES ; j++ ) { + numa_set_distance(i, j, + (i == j) ? NUMA_LOCAL_DISTANCE :
NUMA_REMOTE_DISTANCE);Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
xen/arch/arm/numa.c int __init arch_meminfo_get_ram_bank_range(int bank, · line 155
Quoted 3 lines
xen/arch/arm/numa.c +3
-0+ } + } +}... this implementation looks fairly generic. So can you explain why
we
Quoted 13 lines
Quoted 12 lines
Quoted 2 lines
need it on Arm but not x86?
This implementation is DT only, for x86, it's using
acpi_slit. For now, I am not quit sure ACPI need to do fallback or not. Or say in another way, I don't know how to implement the fallback for ACPI. I planned to solve it in Arm ACPI version NUMA, so I left an empty helper for x86.@Jan Beulich Could you give me some suggestion about x86 fallback?
Quoted 7 lines
Quoted 2 lines
I have a quick look into Linux. When Arch do numa init failed, the
numa_free_distancewill be invoked to revertnuma_distance.Does this matter in the first place? Don't we fall back to single node mode, in which case the sole entry of the distance table will say "local" anyway?
Thank you for providing another way of thinking. Yes, once NUMA init is failed, the system will fall back to single node mode. If we call __node_distance normally, we will not pass two different nodes to this function. Even if we don't revert the values in distance table, we will not trigger the condition of "node_a != node_b". We will always get "local" from __node_distance.
But for closed-loop of code, I tend to revert data when initialization is failed.
Cheers,
Wei Chen
Quoted 1 line
Jan
RFC PATCH 38/40 xen/arm: enable device tree based NUMA in system init
In reply to Wei ChenEverything is ready, we can remove the fake NUMA node and depends on device tree to create NUMA system.
--- xen/arch/arm/numa.c | 45 ++++++++++++++++++++++---------------- xen/include/asm-arm/numa.h | 7 ------ 2 files changed, 26 insertions(+), 26 deletions(-)
xen/arch/arm/numa.c +26 -19
@@ -18,6 +18,7 @@* */ #include <xen/init.h> +#include <xen/device_tree.h> #include <xen/nodemask.h> #include <xen/numa.h> #include <xen/pfn.h>
@@ -83,28 +84,34 @@ void __init numa_init(bool acpi_off)paddr_t ram_size =0 ;paddr_t ram_end =0 ; -printk (XENLOG_WARNING -"NUMA has not been supported yet, NUMA off!\n" ); -/* Arm NUMA has not been implemented until this patch */ - numa_off = true; +/* NUMA has been turned off through Xen parameters */ +if ( numa_off ) + goto mem_init; -/* -* Set all cpu_to_node mapping to 0, this will make cpu_to_node -* function return 0 as previous fake cpu_to_node API. -*/ -for ( idx =0 ; idx <NR_CPUS ; idx++ ) - cpu_to_node[idx] =0 ; - -/* -* Make node_to_cpumask, node_spanned_pages and node_start_pfn -* return as previous fake APIs. -*/ -for ( idx =0 ; idx <MAX_NUMNODES ; idx++ ) { - node_to_cpumask[idx] = cpu_online_map; -node_spanned_pages (idx) = (max_page -mfn_x (first_valid_mfn)); -node_start_pfn (idx) = (mfn_x (first_valid_mfn)); +/* Initialize NUMA from device tree when system is not ACPI booted */ +if ( acpi_off ) + { +#ifdef CONFIG_DEVICE_TREE_NUMA +int ret =numa_device_tree_init (device_tree_flattened); +if ( !ret ) + goto mem_init; +printk (XENLOG_WARNING +"Init NUMA from device tree failed, ret=%d\n" , ret); +#else +printk (XENLOG_WARNING +"CONFIG_DEVICE_TREE_NUMA is not set, NUMA off!\n" ); +#endif + numa_off = true; + } +else + { +/* We don't support NUMA for ACPI boot currently */ +printk (XENLOG_WARNING +"ACPI NUMA has not been supported yet, NUMA off!\n" ); + numa_off = true; } +mem_init:/* * Find the minimal and maximum address of RAM, NUMA will * build a memory to node mapping table for the whole range.
xen/include/asm-arm/numa.h +0 -7
@@ -30,13 +30,6 @@ extern int numa_device_tree_init(const void *fdt);extern void numa_set_distance (nodeid_t from,nodeid_t to,uint32_t distance);extern void arch_numa_init_failed_fallback (void ); -/* -* Temporary for fake NUMA node, when CPU, memory and distance -* matrix will be read from DTB or ACPI SRAT. The following -* symbols will be removed. -*/ -extern mfn_t first_valid_mfn; -#else /* Fake one node for now. See also node_online_map. */
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system init
Hi Wei,
Quoted 2 lines · Wei Chen
Everything is ready, we can remove the fake NUMA node and depends on device tree to create NUMA system.
So you just added code a few patches before that are now completely rewritten. Can you please re-order this series so it doesn't happen?
This may mean that CONFIG_NUMA is only selected until late in this series.
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system init
Hi Julien,
Quoted 11 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:33
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system initHi Wei,
Quoted 9 lines
Quoted 2 lines · Wei Chen
Everything is ready, we can remove the fake NUMA node and depends on device tree to create NUMA system.
So you just added code a few patches before that are now completely rewritten. Can you please re-order this series so it doesn't happen?
This may mean that
CONFIG_NUMAis only selected until late in this series.
Why I did like this is because my original concerns are:
1. When I introduced the CONFIG_NUMA. Users will be using a code base on this commit by accident.
2. If users select CONFIG_NUMA, but not all NUMA data are not initialized properly. The system may not work properly.
3. So I created the fake node to initialize the NUMA data, before we parser real data from DTB.
4. In this case, user can work well with CONFIG_NUMA is enabled, without this series is completed.
It seems I thought too much. If these concerns are not necessary. I am OK to re-order this series.
Quoted 4 lines
Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system init
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 19 lines
Quoted 19 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:33
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system initHi Wei,
Quoted 2 lines · Wei Chen
Everything is ready, we can remove the fake NUMA node and depends on device tree to create NUMA system.
So you just added code a few patches before that are now completely rewritten. Can you please re-order this series so it doesn't happen?
Quoted 8 lines
Quoted 2 lines
This may mean that
CONFIG_NUMAis only selected until late in this series.Why I did like this is because my original concerns are:
1. When I introduced theCONFIG_NUMA. Users will be using a code base on this commit by accident.
2. If users selectCONFIG_NUMA, but not all NUMA data are not initialized properly. The system may not work properly.
We have to make sure we don't break any existing use case when writing a new feature. However, a user should not expect a new feature to work it is using a random commit in the middle of the series.
This is also why I suggested that maybe CONFIG_NUMA is only selected for Arm towards the end of the series. So you reduce the risk of someone selecting it.
Quoted 4 lines
3. So I created the fake node to initialize the NUMA data, before we parser real data from DTB.
4. In this case, user can work well withCONFIG_NUMAis enabled, without this series is completed.
The flip side is you are adding more load on the reviewers because there are extra code. The series is already quite big (40 patches), any way to ease the review will definitely be appreciated.
Another possible way to ease the review is to move the patch that rework/move code at the beginning of the series and leave the Arm part for the second part of the series. This way, we can start to merge the series without waiting for the Arm bits to be completed.
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system init
Hi Julien,
Quoted 47 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月28日 18:45
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system initQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 8 lines
Quoted 8 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:33
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 38/40] xen/arm: enable device tree basedNUMA
Quoted 4 lines
Quoted 4 lines
in system init
Hi Wei,
Quoted 8 lines
Quoted 8 lines · Wei Chen
Quoted 2 lines
Everything is ready, we can remove the fake NUMA node and depends on device tree to create NUMA system.
So you just added code a few patches before that are now completely rewritten. Can you please re-order this series so it doesn't happen?
This may mean that
CONFIG_NUMAis only selected until late in thisseries.
Quoted 6 lines
Quoted 1 line
Why I did like this is because my original concerns are:
1. When I introduced theCONFIG_NUMA. Users will be using a code base on this commit by accident.
2. If users selectCONFIG_NUMA, but not all NUMA data are notinitialized
Quoted 1 line
properly. The system may not work properly.
Quoted 8 lines
We have to make sure we don't break any existing use case when writing a new feature. However, a user should not expect a new feature to work it is using a random commit in the middle of the series.
This is also why I suggested that maybe
CONFIG_NUMAis only selected for Arm towards the end of the series. So you reduce the risk of someone selecting it.
Thanks for this clarification.
Quoted 6 lines
Quoted 1 line
3. So I created the fake node to initialize the NUMA data, before we
parser
Quoted 3 lines
real data from DTB.
4. In this case, user can work well withCONFIG_NUMAis enabled, without this series is completed.
Quoted 9 lines
The flip side is you are adding more load on the reviewers because there are extra code. The series is already quite big (40 patches), any way to ease the review will definitely be appreciated.
Another possible way to ease the review is to move the patch that rework/move code at the beginning of the series and leave the Arm part for the second part of the series. This way, we can start to merge the series without waiting for the Arm bits to be completed.
Yes, I will try to re-order the patches in this way in next version.
Quoted 4 lines
Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system init
In reply to Wei ChenQuoted 50 lines · Wei Chen
Everything is ready, we can remove the fake NUMA node and depends on device tree to create NUMA system.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa.c | 45 ++++++++++++++++++++++---------------- xen/include/asm-arm/numa.h | 7 ------ 2 files changed, 26 insertions(+), 26 deletions(-)xen/arch/arm/numa.c +6
-19@@ -18,6 +18,7 @@* */ #include <xen/init.h> +#include <xen/device_tree.h> #include <xen/nodemask.h> #include <xen/numa.h> #include <xen/pfn.h> @@ -83,28 +84,34 @@ void __init numa_init(bool acpi_off)paddr_t ram_size =0 ;paddr_t ram_end =0 ; -printk (XENLOG_WARNING -"NUMA has not been supported yet, NUMA off!\n" ); -/* Arm NUMA has not been implemented until this patch */ - numa_off = true; +/* NUMA has been turned off through Xen parameters */ +if ( numa_off ) + goto mem_init; -/* -* Set all cpu_to_node mapping to 0, this will make cpu_to_node -* function return 0 as previous fake cpu_to_node API. -*/ -for ( idx =0 ; idx <NR_CPUS ; idx++ ) - cpu_to_node[idx] =0 ; - -/* -* Make node_to_cpumask, node_spanned_pages and node_start_pfn -* return as previous fake APIs. -*/ -for ( idx =0 ; idx <MAX_NUMNODES ; idx++ ) { - node_to_cpumask[idx] = cpu_online_map; -node_spanned_pages (idx) = (max_page -mfn_x (first_valid_mfn)); -node_start_pfn (idx) = (mfn_x (first_valid_mfn)); +/* Initialize NUMA from device tree when system is not ACPI booted */ +if ( acpi_off )
xen/arch/arm/numa.c void __init numa_init(bool acpi_off) · line 102
Quoted 10 lines
xen/arch/arm/numa.c +10
-0+ { +#ifdef CONFIG_DEVICE_TREE_NUMA +int ret =numa_device_tree_init (device_tree_flattened); +if ( !ret ) + goto mem_init; +printk (XENLOG_WARNING +"Init NUMA from device tree failed, ret=%d\n" , ret); +#else +printk (XENLOG_WARNING +"CONFIG_DEVICE_TREE_NUMA is not set, NUMA off!\n" );
I don't think we want to see this warning every time at boot when CONFIG_DEVICE_TREE_NUMA is off. I'd set it to XENLOG_DEBUG or remove it.
Also given that we have many stub functions in xen/include/asm-arm/numa.h already, maybe we could also have a stub function for numa_device_tree_init so that we won'd need an #ifdef CONFIG_DEVICE_TREE_NUMA here.
Quoted 36 lines
xen/arch/arm/numa.c +10
-0+#endif + numa_off = true; + } +else + { +/* We don't support NUMA for ACPI boot currently */ +printk (XENLOG_WARNING +"ACPI NUMA has not been supported yet, NUMA off!\n" ); + numa_off = true; } +mem_init:/* * Find the minimal and maximum address of RAM, NUMA will * build a memory to node mapping table for the whole range. xen/include/asm-arm/numa.h +0
-7@@ -30,13 +30,6 @@ extern int numa_device_tree_init(const void *fdt);extern void numa_set_distance (nodeid_t from,nodeid_t to,uint32_t distance);extern void arch_numa_init_failed_fallback (void ); -/* -* Temporary for fake NUMA node, when CPU, memory and distance -* matrix will be read from DTB or ACPI SRAT. The following -* symbols will be removed. -*/ -extern mfn_t first_valid_mfn; -#else /* Fake one node for now. See also node_online_map. */ Signature · 1 line
2.25.1
RE: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system init
Hi Stefano,
Quoted 60 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 9:51
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system initQuoted 39 lines · Wei Chen
Everything is ready, we can remove the fake NUMA node and depends on device tree to create NUMA system.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa.c | 45 ++++++++++++++++++++++---------------- xen/include/asm-arm/numa.h | 7 ------ 2 files changed, 26 insertions(+), 26 deletions(-)xen/arch/arm/numa.c +4
-10@@ -18,6 +18,7 @@* */ #include <xen/init.h> +#include <xen/device_tree.h> #include <xen/nodemask.h> #include <xen/numa.h> #include <xen/pfn.h> @@ -83,28 +84,34 @@ void __init numa_init(bool acpi_off)paddr_t ram_size =0 ;paddr_t ram_end =0 ; -printk (XENLOG_WARNING -"NUMA has not been supported yet, NUMA off!\n" ); -/* Arm NUMA has not been implemented until this patch */ - numa_off = true; +/* NUMA has been turned off through Xen parameters */ +if ( numa_off ) + goto mem_init; -/* -* Set all cpu_to_node mapping to 0, this will make cpu_to_node -* function return 0 as previous fake cpu_to_node API. -*/ -for ( idx =0 ; idx <NR_CPUS ; idx++ ) - cpu_to_node[idx] =0 ;xen/arch/arm/numa.c void __init numa_init(bool acpi_off) · line 91
Quoted 10 lines
xen/arch/arm/numa.c +1
-9- -/* -* Make node_to_cpumask, node_spanned_pages and node_start_pfn -* return as previous fake APIs. -*/ -for ( idx =0 ; idx <MAX_NUMNODES ; idx++ ) { - node_to_cpumask[idx] = cpu_online_map; -node_spanned_pages (idx) = (max_page -mfn_x (first_valid_mfn)); -node_start_pfn (idx) = (mfn_x (first_valid_mfn)); +/* Initialize NUMA from device tree when system is not ACPI booted */
Quoted 15 lines
xen/arch/arm/numa.c void __init numa_init(bool acpi_off) · line 102
Quoted 11 lines
xen/arch/arm/numa.c +11
-0+if ( acpi_off ) + { +#ifdef CONFIG_DEVICE_TREE_NUMA +int ret =numa_device_tree_init (device_tree_flattened); +if ( !ret ) + goto mem_init; +printk (XENLOG_WARNING +"Init NUMA from device tree failed, ret=%d\n" , ret); +#else +printk (XENLOG_WARNING +"CONFIG_DEVICE_TREE_NUMA is not set, NUMA off!\n" );I don't think we want to see this warning every time at boot when
CONFIG_DEVICE_TREE_NUMAis off. I'd set it toXENLOG_DEBUGor remove it.
OK
Quoted 5 lines
Also given that we have many stub functions in
xen/include/asm-arm/numa.halready, maybe we could also have a stub function fornuma_device_tree_initso that we won'd need an #ifdefCONFIG_DEVICE_TREE_NUMAhere.
Yes, it's a good idea. I will do it.
Quoted 38 lines
Quoted 21 lines
xen/arch/arm/numa.c +10
-0+#endif + numa_off = true; + } +else + { +/* We don't support NUMA for ACPI boot currently */ +printk (XENLOG_WARNING +"ACPI NUMA has not been supported yet, NUMA off!\n" ); + numa_off = true; } +mem_init:/* * Find the minimal and maximum address of RAM, NUMA will * build a memory to node mapping table for the whole range. xen/include/asm-arm/numa.h +0
-0xen/include/asm-arm/numa.h extern int numa_device_tree_init(const void *fdt); · line 30
Quoted 1 line
xen/include/asm-arm/numa.h +0
-0@@ -30,13 +30,6 @@ extern int numa_device_tree_init(const void *fdt); extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_tdistance);
Quoted 15 lines
xen/include/asm-arm/numa.h +0
-8extern void arch_numa_init_failed_fallback (void ); -/* -* Temporary for fake NUMA node, when CPU, memory and distance -* matrix will be read from DTB or ACPI SRAT. The following -* symbols will be removed. -*/ -extern mfn_t first_valid_mfn; -#else /* Fake one node for now. See also node_online_map. */ --2.25.1
RFC PATCH 39/40 xen/x86: move numa_setup to common to support NUMA switch in command line
In reply to Wei ChenXen x86 has created a command line parameter "numa" as NUMA switch for user to turn on/off NUMA. As device tree based NUMA has been enabled for Arm, this parameter can be reused by Arm. So in this patch, we move this parameter to common.
--- xen/arch/x86/numa.c | 34 ---------------------------------- xen/common/numa.c | 35 ++++++++++++++++++++++++++++++++++- xen/include/xen/numa.h | 1 - 3 files changed, 34 insertions(+), 36 deletions(-)
xen/arch/x86/numa.c +0 -34
@@ -11,7 +11,6 @@#include <xen/nodemask.h> #include <xen/numa.h> #include <xen/keyhandler.h> -#include <xen/param.h> #include <xen/time.h> #include <xen/smp.h> #include <xen/pfn.h>
@@ -19,9 +18,6 @@#include <xen/sched.h> #include <xen/softirq.h> -static int numa_setup (const char *s); -custom_param ("numa" , numa_setup); -#ifndef Dprintk #define Dprintk(x...)#endif
@@ -50,35 +46,6 @@ void numa_set_node(int cpu, nodeid_t node) cpu_to_node[cpu] = node; } -/* [numa=off] */ -static __init int numa_setup (const char *opt) -{ -if ( !strncmp (opt,"off" ,3 ) ) - numa_off = true; -else if ( !strncmp (opt,"on" ,2 ) ) - numa_off = false; -#ifdef CONFIG_NUMA_EMU - else if ( !strncmp(opt, "fake=", 5) ) - { - numa_off = false; - numa_fake =simple_strtoul (opt+5 ,NULL ,0 ); -if ( numa_fake >=MAX_NUMNODES ) - numa_fake =MAX_NUMNODES ; - } -#endif -#ifdef CONFIG_ACPI_NUMA - else if ( !strncmp(opt,"noacpi",6) ) - { - numa_off = false; - acpi_numa =-1 ; - } -#endif - else - return -EINVAL; - -return 0 ; -} -/* * Setup early cpu_to_node. *
@@ -287,4 +254,3 @@ static __init int register_numa_trigger(void)return 0 ; }__initcall (register_numa_trigger); -
xen/common/numa.c +34 -1
@@ -14,8 +14,12 @@#include <xen/smp.h> #include <xen/pfn.h> #include <xen/sched.h> +#include <xen/param.h> #include <asm/acpi.h> +static int numa_setup (const char *s); +custom_param ("numa" , numa_setup); +struct node_data node_data[MAX_NUMNODES ];/* Mapping from pdx to node id */
@@ -324,7 +328,7 @@ int __init numa_scan_nodes(u64 start, u64 end) }#ifdef CONFIG_NUMA_EMU -int numa_fake__initdata =0 ; +static int numa_fake__initdata =0 ;/* Numa emulation */ static int __init numa_emulation (u64 start_pfn,u64 end_pfn)
@@ -409,3 +413,32 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)/* architecture specified fallback operations */ arch_numa_init_failed_fallback (); } + +/* [numa=off] */ +static __init int numa_setup (const char *opt) +{ +if ( !strncmp (opt,"off" ,3 ) ) + numa_off = true; +else if ( !strncmp (opt,"on" ,2 ) ) + numa_off = false; +#ifdef CONFIG_NUMA_EMU + else if ( !strncmp(opt, "fake=", 5) ) + { + numa_off = false; + numa_fake =simple_strtoul (opt+5 ,NULL ,0 ); +if ( numa_fake >=MAX_NUMNODES ) + numa_fake =MAX_NUMNODES ; + } +#endif +#ifdef CONFIG_ACPI_NUMA + else if ( !strncmp(opt,"noacpi",6) ) + { + numa_off = false; + acpi_numa =-1 ; + } +#endif + else + return -EINVAL; + +return 0 ; +}
xen/include/xen/numa.h +0 -1
@@ -83,7 +83,6 @@ extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);extern void numa_set_node (int cpu,nodeid_t node);extern int numa_scan_nodes (u64 start,u64 end);extern bool numa_off; -extern int numa_fake;extern s8 acpi_numa;extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end);
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 39/40] xen/x86: move numa_setup to common to support NUMA switch in command line
Hi Wei,
Quoted 102 lines · Wei Chen
Xen x86 has created a command line parameter "numa" as NUMA switch for user to turn on/off NUMA. As device tree based NUMA has been enabled for Arm, this parameter can be reused by Arm. So in this patch, we move this parameter to common.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 34 ---------------------------------- xen/common/numa.c | 35 ++++++++++++++++++++++++++++++++++- xen/include/xen/numa.h | 1 - 3 files changed, 34 insertions(+), 36 deletions(-)xen/arch/x86/numa.c +0
-34@@ -11,7 +11,6 @@#include <xen/nodemask.h> #include <xen/numa.h> #include <xen/keyhandler.h> -#include <xen/param.h> #include <xen/time.h> #include <xen/smp.h> #include <xen/pfn.h> @@ -19,9 +18,6 @@#include <xen/sched.h> #include <xen/softirq.h> -static int numa_setup (const char *s); -custom_param ("numa" , numa_setup); -#ifndef Dprintk #define Dprintk(x...)#endif @@ -50,35 +46,6 @@ void numa_set_node(int cpu, nodeid_t node) cpu_to_node[cpu] = node; } -/* [numa=off] */ -static __init int numa_setup (const char *opt) -{ -if ( !strncmp (opt,"off" ,3 ) ) - numa_off = true; -else if ( !strncmp (opt,"on" ,2 ) ) - numa_off = false; -#ifdef CONFIG_NUMA_EMU - else if ( !strncmp(opt, "fake=", 5) ) - { - numa_off = false; - numa_fake =simple_strtoul (opt+5 ,NULL ,0 ); -if ( numa_fake >=MAX_NUMNODES ) - numa_fake =MAX_NUMNODES ; - } -#endif -#ifdef CONFIG_ACPI_NUMA - else if ( !strncmp(opt,"noacpi",6) ) - { - numa_off = false; - acpi_numa =-1 ; - } -#endif - else - return -EINVAL; - -return 0 ; -} -/* * Setup early cpu_to_node. * @@ -287,4 +254,3 @@ static __init int register_numa_trigger(void)return 0 ; }__initcall (register_numa_trigger); -xen/common/numa.c +5
-1@@ -14,8 +14,12 @@#include <xen/smp.h> #include <xen/pfn.h> #include <xen/sched.h> +#include <xen/param.h> #include <asm/acpi.h> +static int numa_setup (const char *s); +custom_param ("numa" , numa_setup); +struct node_data node_data[MAX_NUMNODES ];/* Mapping from pdx to node id */ @@ -324,7 +328,7 @@ int __init numa_scan_nodes(u64 start, u64 end) }#ifdef CONFIG_NUMA_EMU -int numa_fake__initdata =0 ; +static int numa_fake__initdata =0 ;/* Numa emulation */ static int __init numa_emulation (u64 start_pfn,u64 end_pfn)
xen/common/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 417
Quoted 5 lines
xen/common/numa.c +2
-0@@ -409,3 +413,32 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)/* architecture specified fallback operations */ arch_numa_init_failed_fallback (); } + +/* [numa=off] */
The documentation also needs be be updated to reflect that facts this option is not architecture-agnostic.
Quoted 12 lines
xen/common/numa.c +12
-0+static __init int numa_setup (const char *opt) +{ +if ( !strncmp (opt,"off" ,3 ) ) + numa_off = true; +else if ( !strncmp (opt,"on" ,2 ) ) + numa_off = false; +#ifdef CONFIG_NUMA_EMU + else if ( !strncmp(opt, "fake=", 5) ) + { + numa_off = false; + numa_fake =simple_strtoul (opt+5 ,NULL ,0 ); +if ( numa_fake >=MAX_NUMNODES )
xen/common/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 439
Quoted 10 lines
xen/common/numa.c +10
-0+ numa_fake =MAX_NUMNODES ; + } +#endif +#ifdef CONFIG_ACPI_NUMA + else if ( !strncmp(opt,"noacpi",6) ) + { + numa_off = false; + acpi_numa =-1 ; + } +#endif
Looking at this code, I am not quite too sure to understand the difference between between "numa=noacpi" and "numa=off".
In fact, I am tempted to say this option should disappear because this is odd to have a firmware specific option just for ACPI but not DT. Even if we have one for each, this makes things a bit more complicated for the admin.
Quoted 11 lines
xen/common/numa.c +5
-0+else +return -EINVAL ; + +return 0 ; +}xen/include/xen/numa.h +0
-0
xen/include/xen/numa.h extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn); · line 88
Quoted 7 lines
xen/include/xen/numa.h +0
-1@@ -83,7 +83,6 @@ extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);extern void numa_set_node (int cpu,nodeid_t node);extern int numa_scan_nodes (u64 start,u64 end);extern bool numa_off; -extern int numa_fake;extern s8 acpi_numa;extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end);
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 39/40] xen/x86: move numa_setup to common to support NUMA switch in command line
Hi Julien,
Quoted 115 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月27日 22:38
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 39/40] xen/x86: movenuma_setupto common to support NUMA switch in command lineHi Wei,
xen/common/numa.c void __init numa_initmem_init(unsigned long
Quoted 102 lines · Wei Chen
Xen x86 has created a command line parameter "numa" as NUMA switch for user to turn on/off NUMA. As device tree based NUMA has been enabled for Arm, this parameter can be reused by Arm. So in this patch, we move this parameter to common.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 34 ---------------------------------- xen/common/numa.c | 35 ++++++++++++++++++++++++++++++++++- xen/include/xen/numa.h | 1 - 3 files changed, 34 insertions(+), 36 deletions(-)xen/arch/x86/numa.c +0
-34@@ -11,7 +11,6 @@#include <xen/nodemask.h> #include <xen/numa.h> #include <xen/keyhandler.h> -#include <xen/param.h> #include <xen/time.h> #include <xen/smp.h> #include <xen/pfn.h> @@ -19,9 +18,6 @@#include <xen/sched.h> #include <xen/softirq.h> -static int numa_setup (const char *s); -custom_param ("numa" , numa_setup); -#ifndef Dprintk #define Dprintk(x...)#endif @@ -50,35 +46,6 @@ void numa_set_node(int cpu, nodeid_t node) cpu_to_node[cpu] = node; } -/* [numa=off] */ -static __init int numa_setup (const char *opt) -{ -if ( !strncmp (opt,"off" ,3 ) ) - numa_off = true; -else if ( !strncmp (opt,"on" ,2 ) ) - numa_off = false; -#ifdef CONFIG_NUMA_EMU - else if ( !strncmp(opt, "fake=", 5) ) - { - numa_off = false; - numa_fake =simple_strtoul (opt+5 ,NULL ,0 ); -if ( numa_fake >=MAX_NUMNODES ) - numa_fake =MAX_NUMNODES ; - } -#endif -#ifdef CONFIG_ACPI_NUMA - else if ( !strncmp(opt,"noacpi",6) ) - { - numa_off = false; - acpi_numa =-1 ; - } -#endif - else - return -EINVAL; - -return 0 ; -} -/* * Setup early cpu_to_node. * @@ -287,4 +254,3 @@ static __init int register_numa_trigger(void)return 0 ; }__initcall (register_numa_trigger); -xen/common/numa.c +5
-1@@ -14,8 +14,12 @@#include <xen/smp.h> #include <xen/pfn.h> #include <xen/sched.h> +#include <xen/param.h> #include <asm/acpi.h> +static int numa_setup (const char *s); +custom_param ("numa" , numa_setup); +struct node_data node_data[MAX_NUMNODES ];/* Mapping from pdx to node id */ @@ -324,7 +328,7 @@ int __init numa_scan_nodes(u64 start, u64 end) }#ifdef CONFIG_NUMA_EMU -int numa_fake__initdata =0 ; +static int numa_fake__initdata =0 ;/* Numa emulation */ static int __init numa_emulation (u64 start_pfn,u64 end_pfn)@@ -409,3 +413,32 @@ void __init numa_initmem_init(unsigned long
start_pfn, unsigned longend_pfn)
Quoted 9 lines
xen/common/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 417
Quoted 5 lines
xen/common/numa.c +2
-0/* architecture specified fallback operations */ arch_numa_init_failed_fallback (); } + +/* [numa=off] */ The documentation also needs be be updated to reflect that facts this option is not architecture-agnostic.
Ok, I will update the relate document in next version.
Quoted 26 lines
Quoted 12 lines
xen/common/numa.c +12
-0+static __init int numa_setup (const char *opt) +{ +if ( !strncmp (opt,"off" ,3 ) ) + numa_off = true; +else if ( !strncmp (opt,"on" ,2 ) ) + numa_off = false; +#ifdef CONFIG_NUMA_EMU + else if ( !strncmp(opt, "fake=", 5) ) + { + numa_off = false; + numa_fake =simple_strtoul (opt+5 ,NULL ,0 ); +if ( numa_fake >=MAX_NUMNODES )xen/common/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 439
Quoted 10 lines
xen/common/numa.c +10
-0+ numa_fake =MAX_NUMNODES ; + } +#endif +#ifdef CONFIG_ACPI_NUMA + else if ( !strncmp(opt,"noacpi",6) ) + { + numa_off = false; + acpi_numa =-1 ; + } +#endif Looking at this code, I am not quite too sure to understand the difference between between "numa=noacpi" and "numa=off".
Quoted 5 lines
In fact, I am tempted to say this option should disappear because this is odd to have a firmware specific option just for ACPI but not DT. Even if we have one for each, this makes things a bit more complicated for the admin.
Yes, I agree. I would consider a proper way to address it in next version. If x86 maintainers can give some background of these two options would be better.
Quoted 24 lines
xen/include/xen/numa.h extern void numa_initmem_init(unsigned long start_pfn,
Quoted 10 lines
xen/common/numa.c +5
-0+else +return -EINVAL ; + +return 0 ; +}xen/include/xen/numa.h +0
-0@@ -83,7 +83,6 @@ extern void numa_initmem_init(unsigned long start_pfn,unsigned long
end_pfn);xen/include/xen/numa.h extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn); · line 88
Quoted 8 lines
xen/include/xen/numa.h +0
-1extern void numa_set_node (int cpu,nodeid_t node);extern int numa_scan_nodes (u64 start,u64 end);extern bool numa_off; -extern int numa_fake;extern s8 acpi_numa;extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end);Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 39/40] xen/x86: move numa_setup to common to support NUMA switch in command line
In reply to Wei ChenQuoted 71 lines · Wei Chen
Xen x86 has created a command line parameter "numa" as NUMA switch for user to turn on/off NUMA. As device tree based NUMA has been enabled for Arm, this parameter can be reused by Arm. So in this patch, we move this parameter to common.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 34 ---------------------------------- xen/common/numa.c | 35 ++++++++++++++++++++++++++++++++++- xen/include/xen/numa.h | 1 - 3 files changed, 34 insertions(+), 36 deletions(-)xen/arch/x86/numa.c +0
-33@@ -11,7 +11,6 @@#include <xen/nodemask.h> #include <xen/numa.h> #include <xen/keyhandler.h> -#include <xen/param.h> #include <xen/time.h> #include <xen/smp.h> #include <xen/pfn.h> @@ -19,9 +18,6 @@#include <xen/sched.h> #include <xen/softirq.h> -static int numa_setup (const char *s); -custom_param ("numa" , numa_setup); -#ifndef Dprintk #define Dprintk(x...)#endif @@ -50,35 +46,6 @@ void numa_set_node(int cpu, nodeid_t node) cpu_to_node[cpu] = node; } -/* [numa=off] */ -static __init int numa_setup (const char *opt) -{ -if ( !strncmp (opt,"off" ,3 ) ) - numa_off = true; -else if ( !strncmp (opt,"on" ,2 ) ) - numa_off = false; -#ifdef CONFIG_NUMA_EMU - else if ( !strncmp(opt, "fake=", 5) ) - { - numa_off = false; - numa_fake =simple_strtoul (opt+5 ,NULL ,0 ); -if ( numa_fake >=MAX_NUMNODES ) - numa_fake =MAX_NUMNODES ; - } -#endif -#ifdef CONFIG_ACPI_NUMA - else if ( !strncmp(opt,"noacpi",6) ) - { - numa_off = false; - acpi_numa =-1 ; - } -#endif - else - return -EINVAL; - -return 0 ; -} -/* * Setup early cpu_to_node. *
xen/arch/x86/numa.c static __init int register_numa_trigger(void) · line 290
Quoted 4 lines
xen/arch/x86/numa.c +0
-1@@ -287,4 +254,3 @@ static __init int register_numa_trigger(void)return 0 ; }__initcall (register_numa_trigger); -
spurious change
RE: [XEN RFC PATCH 39/40] xen/x86: move numa_setup to common to support NUMA switch in command line
Hi Stefano,
Quoted 79 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 9:53
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 39/40] xen/x86: movenuma_setupto common to support NUMA switch in command lineQuoted 69 lines · Wei Chen
Xen x86 has created a command line parameter "numa" as NUMA switch for user to turn on/off NUMA. As device tree based NUMA has been enabled for Arm, this parameter can be reused by Arm. So in this patch, we move this parameter to common.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 34 ---------------------------------- xen/common/numa.c | 35 ++++++++++++++++++++++++++++++++++- xen/include/xen/numa.h | 1 - 3 files changed, 34 insertions(+), 36 deletions(-)xen/arch/x86/numa.c +0
-33@@ -11,7 +11,6 @@#include <xen/nodemask.h> #include <xen/numa.h> #include <xen/keyhandler.h> -#include <xen/param.h> #include <xen/time.h> #include <xen/smp.h> #include <xen/pfn.h> @@ -19,9 +18,6 @@#include <xen/sched.h> #include <xen/softirq.h> -static int numa_setup (const char *s); -custom_param ("numa" , numa_setup); -#ifndef Dprintk #define Dprintk(x...)#endif @@ -50,35 +46,6 @@ void numa_set_node(int cpu, nodeid_t node) cpu_to_node[cpu] = node; } -/* [numa=off] */ -static __init int numa_setup (const char *opt) -{ -if ( !strncmp (opt,"off" ,3 ) ) - numa_off = true; -else if ( !strncmp (opt,"on" ,2 ) ) - numa_off = false; -#ifdef CONFIG_NUMA_EMU - else if ( !strncmp(opt, "fake=", 5) ) - { - numa_off = false; - numa_fake =simple_strtoul (opt+5 ,NULL ,0 ); -if ( numa_fake >=MAX_NUMNODES ) - numa_fake =MAX_NUMNODES ; - } -#endif -#ifdef CONFIG_ACPI_NUMA - else if ( !strncmp(opt,"noacpi",6) ) - { - numa_off = false; - acpi_numa =-1 ; - } -#endif - else - return -EINVAL; - -return 0 ; -} -/* * Setup early cpu_to_node.
Quoted 8 lines
xen/arch/x86/numa.c static __init int register_numa_trigger(void) · line 290
Quoted 6 lines
xen/arch/x86/numa.c +0
-1* @@ -287,4 +254,3 @@ static __init int register_numa_trigger(void)return 0 ; }__initcall (register_numa_trigger); -spurious change
got it.
RFC PATCH 40/40 xen/x86: move dump_numa info hotkey to common
In reply to Wei ChenAs device tree based NUMA has been enabled for Arm, so not only x86 needs to dump numa info through hotkey, but also Arm can use this hotkey to dump numa info.
In this patch, we move this hotkey to common. Arm can use it to dump its numa information:
(XEN) key 'u' (ascii '75') => dump NUMA info (XEN) 'u' pressed -> dumping numa info (now = 8805901249990) (XEN) NODE0 start->524288 size->520192 free->257673 (XEN) NODE1 start->8912896 size->524288 free->499676 (XEN) CPU0...1 -> NODE0
(XEN) CPU2...3 -> NODE1
(XEN) Memory location of each domain:
(XEN) Domain 0 (total: 262144):
(XEN) Node 0: 262144
(XEN) Node 1: 0
--- xen/arch/x86/numa.c | 142 ------------------------------------------- xen/common/numa.c | 144 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 144 insertions(+), 142 deletions(-)
xen/arch/x86/numa.c +0 -142
@@ -112,145 +112,3 @@ int __init arch_meminfo_get_ram_bank_range(int bank,return 0 ; } - -static void dump_numa (unsigned char key) -{ -s_time_t now =NOW (); -unsigned int i, j, n; -struct domain *d; -struct page_info *page; -unsigned int page_num_node[MAX_NUMNODES ]; -const struct vnuma_info *vnuma; - -printk ("'%c' pressed -> dumping numa info (now = %" PRI_stime")\n" , key, - now); - -for_each_online_node ( i ) - { -paddr_t pa =pfn_to_paddr (node_start_pfn (i) +1 ); - -printk ("NODE%u start->%lu size->%lu free->%lu\n" , - i,node_start_pfn (i),node_spanned_pages (i), -avail_node_heap_pages (i)); -/* sanity check phys_to_nid() */ -if (phys_to_nid (pa) != i ) -printk ("phys_to_nid(%" PRIpaddr") -> %d should be %u\n" , - pa,phys_to_nid (pa), i); - } - - j =cpumask_first (&cpu_online_map); - n =0 ; -for_each_online_cpu ( i ) - { -if ( i != j + n || cpu_to_node[j] != cpu_to_node[i] ) - { -if ( n >1 ) -printk ("CPU%u...%u -> NODE%d\n" , j, j + n -1 , cpu_to_node[j]); -else -printk ("CPU%u -> NODE%d\n" , j, cpu_to_node[j]); - j = i; - n =1 ; - } -else - ++n; - } -if ( n >1 ) -printk ("CPU%u...%u -> NODE%d\n" , j, j + n -1 , cpu_to_node[j]); -else -printk ("CPU%u -> NODE%d\n" , j, cpu_to_node[j]); - -rcu_read_lock (&domlist_read_lock); - -printk ("Memory location of each domain:\n" ); -for_each_domain ( d ) - { -process_pending_softirqs (); - -printk ("Domain %u (total: %u):\n" , d->domain_id,domain_tot_pages (d)); - -for_each_online_node ( i ) - page_num_node[i] =0 ; - -spin_lock (&d->page_alloc_lock); -page_list_for_each (page, &d->page_list) - { - i =phys_to_nid (page_to_maddr (page)); - page_num_node[i]++; - } -spin_unlock (&d->page_alloc_lock); - - for_each_online_node ( i ) - printk(" Node %u: %u\n", i, page_num_node[i]); - -if ( !read_trylock (&d->vnuma_rwlock) ) -continue ; - -if ( !d->vnuma ) - { -read_unlock (&d->vnuma_rwlock); -continue ; - } - - vnuma = d->vnuma; -printk (" %u vnodes, %u vcpus, guest physical layout:\n" , - vnuma->nr_vnodes, d->max_vcpus); -for ( i =0 ; i < vnuma->nr_vnodes; i++ ) - { -unsigned int start_cpu = ~0U ; - -if ( vnuma->vnode_to_pnode[i] ==NUMA_NO_NODE ) -printk (" %3u: pnode ???," , i); -else -printk (" %3u: pnode %3u," , i, vnuma->vnode_to_pnode[i]); - -printk (" vcpus " ); - -for ( j =0 ; j < d->max_vcpus; j++ ) - { -if ( !(j &0x3f ) ) -process_pending_softirqs (); - -if ( vnuma->vcpu_to_vnode[j] == i ) - { -if ( start_cpu == ~0U ) - { -printk ("%d" , j); - start_cpu = j; - } - } -else if ( start_cpu != ~0U ) - { -if ( j -1 != start_cpu ) -printk ("-%d " , j -1 ); -else -printk (" " ); - start_cpu = ~0U ; - } - } - -if ( start_cpu != ~0U && start_cpu != j -1 ) -printk ("-%d" , j -1 ); - -printk ("\n" ); - -for ( j =0 ; j < vnuma->nr_vmemranges; j++ ) - { -if ( vnuma->vmemrange[j].nid == i ) -printk (" %016" PRIx64" - %016" PRIx64"\n" , - vnuma->vmemrange[j].start, - vnuma->vmemrange[j].end); - } - } - -read_unlock (&d->vnuma_rwlock); - } - -rcu_read_unlock (&domlist_read_lock); -} - -static __init int register_numa_trigger (void ) -{ -register_keyhandler ('u' , dump_numa,"dump NUMA info" ,1 ); -return 0 ; -} -__initcall (register_numa_trigger);
xen/common/numa.c +144 -0
154 lines not on this page · raw message
Signature · 1 line
2.25.1
Quoted 36 lines · Wei Chen
Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fixlowmem_bitsizewhenarch_get_dma_bitsizereturn 0Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized forxlu_pci_parse_bdfxen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI inKconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Movenuma_add_cpu_nodeto common
xen/x86: MoveNR_NODE_MEMBLKSmacro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Movenuma_initmem_initto common
xen/arm: introducenuma_set_nodefor Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introducedevice_tree_numaas a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creatingcpu_logical_mapxen/x86: decouplenodes_cover_memorywith E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: movenodes_cover_memoryto common
xen/x86: makeacpi_scan_nodesto be neutral
xen: exportbad_sratandsrat_disabledto extern
Quoted 8 lines
xen: move
numa_scan_nodesfrom x86 to common
xen: enablenuma_scan_nodesfor device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: movenuma_setupto common to support NUMA switch in command
line
xen/x86: movedump_numainfo hotkey to common
May I please ask that you follow patch submission guidelines, in that you send patches To: the list and Cc: relevant people. Furthermore I doubt that I need to be on Cc: for all 40 of the patches.
Thanks and regards,
Jan
Hi Jan,
Quoted 55 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月11日 18:42
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 00/40] Add device tree based NUMA support to Arm64Quoted 36 lines · Wei Chen
Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fixlowmem_bitsizewhenarch_get_dma_bitsizereturn 0Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized forxlu_pci_parse_bdfxen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI inKconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Movenuma_add_cpu_nodeto common
xen/x86: MoveNR_NODE_MEMBLKSmacro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Movenuma_initmem_initto common
xen/arm: introducenuma_set_nodefor Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introducedevice_tree_numaas a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creatingcpu_logical_mapxen/x86: decouplenodes_cover_memorywith E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: movenodes_cover_memoryto common
xen/x86: makeacpi_scan_nodesto be neutral
xen: exportbad_sratandsrat_disabledto externQuoted 8 lines
xen: move
numa_scan_nodesfrom x86 to common
xen: enablenuma_scan_nodesfor device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: movenuma_setupto common to support NUMA switch in command
line
xen/x86: movedump_numainfo hotkey to common
Quoted 4 lines
May I please ask that you follow patch submission guidelines, in that you send patches To: the list and Cc: relevant people. Furthermore I doubt that I need to be on Cc: for all 40 of the patches.
Thanks for your reminder. Before I sent this series, I had paid sometime to consider CC or TO you, I found you are in the X86 Arch, x86 memory management maintainer lists. And in this patch series, I have done some changes that affects x86, so I added you in TO list. Obviously, my understanding had some mistake. I will add you to CC list in next version.
Quoted 2 lines
Thanks and regards,
Jan
Quoted 54 lines · Wei Chen
Quoted 49 lines
From: Jan Beulich <[email protected]>
Sent: 2021年8月11日 18:42Quoted 36 lines · Wei Chen
Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fixlowmem_bitsizewhenarch_get_dma_bitsizereturn 0Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized forxlu_pci_parse_bdfxen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI inKconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Movenuma_add_cpu_nodeto common
xen/x86: MoveNR_NODE_MEMBLKSmacro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Movenuma_initmem_initto common
xen/arm: introducenuma_set_nodefor Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introducedevice_tree_numaas a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creatingcpu_logical_mapxen/x86: decouplenodes_cover_memorywith E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: movenodes_cover_memoryto common
xen/x86: makeacpi_scan_nodesto be neutral
xen: exportbad_sratandsrat_disabledto externQuoted 8 lines
xen: move
numa_scan_nodesfrom x86 to common
xen: enablenuma_scan_nodesfor device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: movenuma_setupto common to support NUMA switch in command
line
xen/x86: movedump_numainfo hotkey to commonQuoted 4 lines
May I please ask that you follow patch submission guidelines, in that you send patches To: the list and Cc: relevant people. Furthermore I doubt that I need to be on Cc: for all 40 of the patches.
Quoted 6 lines
Thanks for your reminder. Before I sent this series, I had paid sometime to consider CC or TO you, I found you are in the X86 Arch, x86 memory management maintainer lists. And in this patch series, I have done some changes that affects x86, so I added you in TO list. Obviously, my understanding had some mistake. I will add you to CC list in next version.
And then on a patch-by-patch basis please, unless you see a specific need to also Cc my on certain Arm-only patches. Thanks.
Jan
Hi Wei,
Quoted 11 lines · Wei Chen
Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.
Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.
All the platforms you mention here are servers (so mainly ACPI). However, this series is adding DT support.
Could you outline the long term plan for DT? Is it going to be used on production HW?
Cheers,
Signature · 1 line
Julien Grall
Hi Julien,
Quoted 16 lines · Julien Grall
Hi Wei,
Quoted 10 lines · Wei Chen
Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.
Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.
Quoted 3 lines
All the platforms you mention here are servers (so mainly ACPI). However, this series is adding DT support.
Could you outline the long term plan for DT? Is it going to be used on production HW?
Yes we are already and will continue to use this on production HW.
Some embedded hardware will have some usage of NUMA (as some embedded functions do require lots of computing power). We are doing experiments of that right now using those patches.
Cheers
Bertrand
Quoted 5 lines
Cheers,
Signature · 1 line
Julien Grall
Quoted 1 line · Bertrand Marquis
Hi Julien,
Hi Bertrand,
Quoted 16 lines
Quoted 16 lines · Julien Grall
Hi Wei,
Quoted 10 lines · Wei Chen
Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.
Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.
Quoted 6 lines
Quoted 3 lines
All the platforms you mention here are servers (so mainly ACPI). However, this series is adding DT support.
Could you outline the long term plan for DT? Is it going to be used on production HW?
Yes we are already and will continue to use this on production HW.
Some embedded hardware will have some usage of NUMA (as some embedded functions do require lots of computing power).
Interesting! Thank you for the clarifications.
Quoted 1 line
We are doing experiments of that right now using those patches.
Cheers,
Signature · 1 line
Julien Grall
Thanks for the big contribution!
I just wanted to let you know that the series passed all the gitlab-ci build tests without issues.
The runtime tests originally failed due to unrelated problems (there was a Debian testing upgrade that broke Gitlab-CI.) I fix the underlying issue and restarted the failed tests and now they passed.
This is the pipeline:
https://gitlab.com/xen-project/patchew/xen/-/pipelines/351484940
There are still two runtime x86 tests that fail but I don't think the failures are related to your series.
Quoted 120 lines · Wei Chen
Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.
Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.
So now we have a new problem. When Xen is running on these Arm devices, Xen still treat them as single node SMP systems. The NUMA affinity capability of Xen memory allocation and scheduler becomes meaningless. Because they rely on input data that does not reflect real NUMA layout.
Xen still think the access time for all of the memory is the same for all CPUs. However, Xen may allocate memory to a VM from different NUMA nodes with different access speeds. This difference can be amplified in workloads inside VM, causing performance instability and timeouts.
So in this patch series, we implement a set of NUMA API to use device tree to describe the NUMA layout. We reuse most of the code of x86 NUMA to create and maintain the mapping between memory and CPU, create the matrix between any two NUMA nodes. Except ACPI and some x86 specified code, we have moved other code to common. In next stage, when we implement ACPI based NUMA for Arm64, we may move the ACPI NUMA code to common too, but in current stage, we keep it as x86 only.
This patch serires has been tested and booted well on one Arm64 NUMA machine and one HPE x86 NUMA machine.
Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fixlowmem_bitsizewhenarch_get_dma_bitsizereturn 0Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized forxlu_pci_parse_bdfxen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI inKconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Movenuma_add_cpu_nodeto common
xen/x86: MoveNR_NODE_MEMBLKSmacro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Movenuma_initmem_initto common
xen/arm: introducenuma_set_nodefor Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introducedevice_tree_numaas a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creatingcpu_logical_mapxen/x86: decouplenodes_cover_memorywith E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: movenodes_cover_memoryto common
xen/x86: makeacpi_scan_nodesto be neutral
xen: exportbad_sratandsrat_disabledto extern
xen: movenuma_scan_nodesfrom x86 to common
xen: enablenuma_scan_nodesfor device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: movenuma_setupto common to support NUMA switch in command
line
xen/x86: movedump_numainfo hotkey to commontools/libs/util/libxlu_pci.c | 3 +- xen/arch/arm/Kconfig | 10 + xen/arch/arm/Makefile | 2 + xen/arch/arm/arm64/head.S | 9 +- xen/arch/arm/bootfdt.c | 8 +- xen/arch/arm/domain_build.c | 17 +- xen/arch/arm/efi/efi-boot.h | 25 -- xen/arch/arm/numa.c | 162 +++++++++ xen/arch/arm/numa_device_tree.c | 292 ++++++++++++++++ xen/arch/arm/platform.c | 4 +- xen/arch/arm/setup.c | 14 + xen/arch/arm/smpboot.c | 37 +- xen/arch/x86/Kconfig | 2 +- xen/arch/x86/numa.c | 421 +---------------------- xen/arch/x86/srat.c | 147 +------- xen/common/Kconfig | 3 + xen/common/Makefile | 1 + xen/common/libfdt/fdt_ro.c | 15 + xen/common/numa.c | 588 ++++++++++++++++++++++++++++++++ xen/common/page_alloc.c | 2 +- xen/drivers/acpi/Kconfig | 3 +- xen/drivers/acpi/Makefile | 2 +- xen/include/asm-arm/numa.h | 33 ++ xen/include/asm-arm/setup.h | 6 + xen/include/asm-x86/acpi.h | 4 - xen/include/asm-x86/config.h | 1 - xen/include/asm-x86/numa.h | 65 +--- xen/include/asm-x86/setup.h | 1 - xen/include/xen/libfdt/libfdt.h | 25 ++ xen/include/xen/nodemask.h | 2 + xen/include/xen/numa.h | 80 +++++ 31 files changed, 1325 insertions(+), 659 deletions(-) create mode 100644 xen/arch/arm/numa.c create mode 100644 xen/arch/arm/numa_device_tree.c create mode 100644 xen/common/numa.cSignature · 1 line
2.25.1
Hi Stefano,
Quoted 19 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月26日 8:09
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]
Subject: Re: [XEN RFC PATCH 00/40] Add device tree based NUMA support to Arm64Thanks for the big contribution!
I just wanted to let you know that the series passed all the gitlab-ci build tests without issues.
The runtime tests originally failed due to unrelated problems (there was a Debian testing upgrade that broke Gitlab-CI.) I fix the underlying issue and restarted the failed tests and now they passed.
Quoted 7 lines
This is the pipeline:
https://gitlab.com/xen-project/patchew/xen/-/pipelines/351484940There are still two runtime x86 tests that fail but I don't think the failures are related to your series.
Thanks for testing this series : )
Quoted 121 lines · Wei Chen
Quoted 120 lines
Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.
Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.
So now we have a new problem. When Xen is running on these Arm devices, Xen still treat them as single node SMP systems. The NUMA affinity capability of Xen memory allocation and scheduler becomes meaningless. Because they rely on input data that does not reflect real NUMA layout.
Xen still think the access time for all of the memory is the same for all CPUs. However, Xen may allocate memory to a VM from different NUMA nodes with different access speeds. This difference can be amplified in workloads inside VM, causing performance instability and timeouts.
So in this patch series, we implement a set of NUMA API to use device tree to describe the NUMA layout. We reuse most of the code of x86 NUMA to create and maintain the mapping between memory and CPU, create the matrix between any two NUMA nodes. Except ACPI and some x86 specified code, we have moved other code to common. In next stage, when we implement ACPI based NUMA for Arm64, we may move the ACPI NUMA code to common too, but in current stage, we keep it as x86 only.
This patch serires has been tested and booted well on one Arm64 NUMA machine and one HPE x86 NUMA machine.
Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fixlowmem_bitsizewhenarch_get_dma_bitsizereturn 0Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized forxlu_pci_parse_bdfxen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI inKconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Movenuma_add_cpu_nodeto common
xen/x86: MoveNR_NODE_MEMBLKSmacro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Movenuma_initmem_initto common
xen/arm: introducenuma_set_nodefor Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introducedevice_tree_numaas a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creatingcpu_logical_mapxen/x86: decouplenodes_cover_memorywith E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: movenodes_cover_memoryto common
xen/x86: makeacpi_scan_nodesto be neutral
xen: exportbad_sratandsrat_disabledto extern
xen: movenuma_scan_nodesfrom x86 to common
xen: enablenuma_scan_nodesfor device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: movenuma_setupto common to support NUMA switch in command
line
xen/x86: movedump_numainfo hotkey to commontools/libs/util/libxlu_pci.c | 3 +- xen/arch/arm/Kconfig | 10 + xen/arch/arm/Makefile | 2 + xen/arch/arm/arm64/head.S | 9 +- xen/arch/arm/bootfdt.c | 8 +- xen/arch/arm/domain_build.c | 17 +- xen/arch/arm/efi/efi-boot.h | 25 -- xen/arch/arm/numa.c | 162 +++++++++ xen/arch/arm/numa_device_tree.c | 292 ++++++++++++++++ xen/arch/arm/platform.c | 4 +- xen/arch/arm/setup.c | 14 + xen/arch/arm/smpboot.c | 37 +- xen/arch/x86/Kconfig | 2 +- xen/arch/x86/numa.c | 421 +---------------------- xen/arch/x86/srat.c | 147 +------- xen/common/Kconfig | 3 + xen/common/Makefile | 1 + xen/common/libfdt/fdt_ro.c | 15 + xen/common/numa.c | 588 ++++++++++++++++++++++++++++++++ xen/common/page_alloc.c | 2 +- xen/drivers/acpi/Kconfig | 3 +- xen/drivers/acpi/Makefile | 2 +- xen/include/asm-arm/numa.h | 33 ++ xen/include/asm-arm/setup.h | 6 + xen/include/asm-x86/acpi.h | 4 - xen/include/asm-x86/config.h | 1 - xen/include/asm-x86/numa.h | 65 +--- xen/include/asm-x86/setup.h | 1 - xen/include/xen/libfdt/libfdt.h | 25 ++ xen/include/xen/nodemask.h | 2 + xen/include/xen/numa.h | 80 +++++ 31 files changed, 1325 insertions(+), 659 deletions(-) create mode 100644 xen/arch/arm/numa.c create mode 100644 xen/arch/arm/numa_device_tree.c create mode 100644 xen/common/numa.c--
2.25.1