Device tree based NUMA support for Arm - Part#2
v4 cover Device tree based NUMA support for Arm - Part#2
(The Arm device tree based NUMA support patch set contains 35 patches. In order to make stuff easier for reviewers, I split them into 3 parts:
1. Preparation. I have re-sorted the patch series. And moved independent patches to the head of the series - merged in [1]
2. Move generically usable code from x86 to common - this series.
3. Add new code to support Arm.
This series only contains the second part patches. As the whole NUMA series has been reviewed for 1 round in [2], so this series would be v3)
Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.
Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system.
So now we have a new problem. When Xen is running on these Arm devices, Xen still treat them as single node SMP systems. The NUMA affinity capability of Xen memory allocation and scheduler becomes meaningless. Because they rely on input data that does not reflect real NUMA layout.
Xen still think the access time for all of the memory is the same for all CPUs. However, Xen may allocate memory to a VM from different NUMA nodes with different access speeds. This difference can be amplified in workloads inside VM, causing performance instability and timeouts.
So in this patch series, we implement a set of NUMA API to use device tree to describe the NUMA layout. We reuse most of the code of x86 NUMA to create and maintain the mapping between memory and CPU, create the matrix between any two NUMA nodes. Except ACPI and some x86 specified code, we have moved other code to common. In next stage, when we implement ACPI based NUMA for Arm64, we may move the ACPI NUMA code to common too, but in current stage, we keep it as x86 only.
This patch serires has been tested and booted well on one Arm64 NUMA machine and one HPE x86 NUMA machine.
[1] https://lists.xenproject.org/archives/html/xen-devel/2022-06/msg00499.html
[2] https://lists.xenproject.org/archives/html/xen-devel/2021-09/msg01903.html
---
v3 -> v4:
1. Add init_as_disable as arch_numa_disabled parameter in the patche where use it.
2. Drop unnecessary "else" from arch_numa_setup, and fix its indentation.
3. Restore compute_hash_shift's return value to int.
4. Remove unnecessary parentheses for macros.
5. Use unsigned int for proper variables.
6. Fix some code-style.
7. Move arch_get_ram_range function comment to header file.
8. Use bool for found, and add a new "err" for the return value of arch_get_ram_range.
9. Use -ENODATA instead of -EINVAL for non-RAM type ranges.
10. Use bool as return value for functions that only return
0/1 or 0/-EINVAL.
11. Move mem_hotplug to a proper place in mm.h
12. Remove useless "size" in numa_scan_nodes.
13. Add CONFIG_HAS_NUMA_NODE_FWID to gate print the mapping between node id and architectural node id (fw node id).
v2 -> v3:
1. Drop enumeration of numa status.
2. Use helpers to get/update acpi_numa.
3. Insert spaces among parameters of strncmp in numa_setup.
4. Drop helpers to access mem_hotplug. Export mem_hotplug for all arch.
5. Remove acpi.h from common/numa.c.
6. Rename acpi_scan_nodes to numa_scan_nodes.
7. Replace u8 by uint8_t for memnodemap.
8. Use unsigned int for memnode_shift and adjust related functions (compute_hash_shift, populate_memnodemap) to use correct types for return values or parameters.
9. Use nodeid_t for nodeid and node numbers.
10. Use __read_mostly and __ro_after_init for appropriate variables.
11. Adjust the __read_mostly and __initdata location for some variables.
12. Convert from plain int to unsigned for cpuid and other proper
13. Remove unnecessary change items in history.
14. Rename arch_get_memory_map to arch_get_ram_range.
15. Use -ENOENT instead of -ENODEV to indicate end of memory map.
16. Add description to code comment that arch_get_ram_range returns RAM range in [start, end) format.
17. Rename bad_srat to numa_fw_bad.
18. Rename node_to_pxm to numa_node_to_arch_nid.
19. Merge patch#7 and #8 into patch#6.
20. Move NR_NODE_MEMBLKS from x86/acpi.h to common/numa.h
22. Use 2-64 for node range.
v1 -> v2:
1. Refine the commit messages of several patches.
2. Merge v1 patch#9,10 into one patch. Introduce the new functions in the same patch that this patch will be used first time.
3. Fold if ( end > mem_hotplug ) to mem_hotplug_update_boundary, in this case, we can drop mem_hotplug_boundary.
4. Remove fw_numa, use enumeration to replace numa_off and acpi_numa.
5. Correct return value of srat_disabled.
6. Introduce numa_enabled_with_firmware.
7. Refine the justification of using !node_data[nid].node_spanned_pages.
8. Use ASSERT to replace VIRTUAL_BUG_ON in phys_to_nid.
9. Adjust the conditional express for ASSERT.
10. Move MAX_NUMNODES from xen/numa.h to asm/numa.h for x86.
11. Use conditional macro to gate MAX_NUMNODES for other architectures.
12. Use arch_get_memory_map to replace arch_get_memory_bank_range and arch_get_memory_bank_number.
13. Remove the !start || !end check, because caller guarantee these two pointers will not be NULL.
14. Add code comment for numa_update_node_memblks to explain:
Assumes all memory regions belonging to a single node
are in one chunk. Holes between them will be included
in the node.
15. Merge this single patch instead of serval patches to move x86 SRAT code to common.
16. Export node_to_pxm to keep pxm information in NUMA scan nodes error messages.
17. Change the code style to target file's Xen code-style.
18. Adjust some __init and __initdata for some functions and variables.
19. Replace CONFIG_ACPI_NUMA by CONFIG_NUMA. Replace "SRAT" texts.
20. Turn numa_scan_nodes to static.
21. Change NR_NUMA_NODES upper bound from 4095 to 255.
Wei Chen (6):
xen/x86: Provide helpers for common code to access acpi_numa xen/x86: move generically usable NUMA code from x86 to common xen/x86: Use ASSERT instead of VIRTUAL_BUG_ON for phys_to_nid xen/x86: use arch_get_ram_range to get information from E820 map xen/x86: move NUMA scan nodes codes from x86 to common
xen: introduce a Kconfig option to configure NUMA nodes number
xen/arch/Kconfig | 11 + xen/arch/x86/include/asm/acpi.h | 2 - xen/arch/x86/include/asm/mm.h | 2 - xen/arch/x86/include/asm/numa.h | 61 +-- xen/arch/x86/include/asm/setup.h | 1 - xen/arch/x86/mm.c | 2 - xen/arch/x86/numa.c | 438 +----------------- xen/arch/x86/smpboot.c | 2 +- xen/arch/x86/srat.c | 311 ++----------- xen/common/Makefile | 1 + xen/common/numa.c | 759 +++++++++++++++++++++++++++++++ xen/common/page_alloc.c | 2 + xen/drivers/acpi/Kconfig | 1 + xen/include/xen/mm.h | 2 + xen/include/xen/numa.h | 97 +++- 15 files changed, 917 insertions(+), 775 deletions(-) create mode 100644 xen/common/numa.c
Signature · 1 line
2.25.1
PATCH v4 1/6 xen/x86: Provide helpers for common code to access acpi_numa
acpi_numa is a specific NUMA switch for ACPI NUMA implementation. Other NUMA implementation may not need this switch. But this switch is not only used by ACPI code, it is also used directly in some general NUMA logic code. So far this hasn't caused any problem because Xen only has x86 implementing ACPI NUMA, but now Arm is implementing device tree based NUMA. Accesssing acpi_numa directly in some functions will be a block of reusing NUMA common code. It is also difficult for us to replace it with a new generic switch, because it is hard to prove that the new switch states can guarantee the original code will work correctly.
So in this patch, we provide two helpers for common code to update and get states of acpi_numa. And other new NUMA implementations just need to provide the same helpers for common code. In this case, the generic NUMA logic code can be reused by all NUMA implementations.
---
v3 -> v4:
1. Drop parameter from arch_numa_disabled, the parameter will be introduced in later patch where use it.
2. Drop unnecessary "else" from arch_numa_setup, and fix its indentation.
v2 -> v3:
1. Drop enumeration of numa status.
2. Use helpers to get/update acpi_numa.
3. Insert spaces among parameters of strncmp in numa_setup. v1 -> v2:
1. Remove fw_numa.
2. Use enumeration to replace numa_off and acpi_numa.
3. Correct return value of srat_disabled.
4. Introduce numa_enabled_with_firmware.
--- xen/arch/x86/include/asm/numa.h | 5 +++-- xen/arch/x86/numa.c | 38 ++++++++++++++++++++++----------- 2 files changed, 28 insertions(+), 15 deletions(-)
xen/arch/x86/include/asm/numa.h +3 -2
@@ -32,8 +32,9 @@ extern void numa_add_cpu(int cpu);extern void numa_init_array (void );extern bool numa_off; - -extern int srat_disabled (void ); +extern int arch_numa_setup (const char *opt); +extern bool arch_numa_disabled (void ); +extern bool srat_disabled (void );extern void numa_set_node (int cpu,nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu);
xen/arch/x86/numa.c +25 -13
@@ -50,9 +50,28 @@ nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };bool numa_off;s8 acpi_numa =0 ; -int srat_disabled (void ) +int __init arch_numa_setup (const char *opt) { -return numa_off || acpi_numa <0 ; +#ifdef CONFIG_ACPI_NUMA +if ( !strncmp (opt,"noacpi" ,6 ) ) + { + numa_off = false; + acpi_numa =-1 ; +return 0 ; + } +#endif + +return -EINVAL ; +} + +bool arch_numa_disabled (void ) +{ +return acpi_numa <0 ; +} + +bool srat_disabled (void ) +{ +return numa_off ||arch_numa_disabled (); }/*
@@ -291,28 +310,21 @@ void numa_set_node(int cpu, nodeid_t node)/* [numa=off] */ static int __init cf_check numa_setup(const char *opt) { -if ( !strncmp (opt,"off" ,3 ) ) +if ( !strncmp (opt,"off" ,3 ) ) numa_off = true; -else if ( !strncmp (opt,"on" ,2 ) ) +else if ( !strncmp (opt,"on" ,2 ) ) numa_off = false;#ifdef CONFIG_NUMA_EMU else if ( !strncmp(opt, "fake=", 5) ) { numa_off = false; - numa_fake =simple_strtoul (opt+5 ,NULL ,0 ); + numa_fake =simple_strtoul (opt +5 ,NULL ,0 );if ( numa_fake >=MAX_NUMNODES ) numa_fake =MAX_NUMNODES ; } -#endif -#ifdef CONFIG_ACPI_NUMA - else if ( !strncmp(opt,"noacpi",6) ) - { - numa_off = false; - acpi_numa =-1 ; - }#endif else - return -EINVAL; + return arch_numa_setup(opt);return 0 ; }
Signature · 1 line
2.25.1
Re: [PATCH v4 1/6] xen/x86: Provide helpers for common code to access acpi_numa
Quoted 10 lines · Wei Chen
acpi_numais a specific NUMA switch for ACPI NUMA implementation. Other NUMA implementation may not need this switch. But this switch is not only used by ACPI code, it is also used directly in some general NUMA logic code. So far this hasn't caused any problem because Xen only has x86 implementing ACPI NUMA, but now Arm is implementing device tree based NUMA. Accesssingacpi_numadirectly in some functions will be a block of reusing NUMA common code. It is also difficult for us to replace it with a new generic switch, because it is hard to prove that the new switch states can guarantee the original code will work correctly.
Quoted 6 lines
So in this patch, we provide two helpers for common code to update and get states of
acpi_numa. And other new NUMA implementations just need to provide the same helpers for common code. In this case, the generic NUMA logic code can be reused by all NUMA implementations.Signed-off-by Wei Chen <[email protected]>
PATCH v4 2/6 xen/x86: move generically usable NUMA code from x86 to common
In reply to Wei ChenThere are some codes in x86/numa.c can be shared by common architectures to implememnt NUMA support. Just like some variables and functions to check and store NUMA memory map. And some variables and functions to do NUMA initialization.
In this patch, we move them to common/numa.c and xen/numa.h and use the CONFIG_NUMA to gate them for non-NUMA supported architectures. As the target header file is Xen-style, so we trim some spaces and replace tabs for the codes that has been moved to xen/numa.h at the same time.
As acpi_scan_nodes has been used in a common function, it doesn't make sense to use acpi_xxx in common code, so we rename it to numa_scan_nodes in this patch too. After that if we still use CONFIG_ACPI_NUMA in to gate numa_scan_nodes in numa_initmem_init, that doesn't make sense. As CONFIG_NUMA will be selected by CONFIG_ACPI_NUMA for x86. So, we replace CONFIG_ACPI_NUMA by CONFIG_NUMA to gate numa_scan_nodes.
As arch_numa_disabled has been implememnted for ACPI NUMA, we can rename srat_disabled to numa_disabled and move it to common code as well.
---
v3 -> v4:
1. Restore compute_hash_shift's return value to int.
2. Remove unnecessary parentheses for macros.
3. Use unsigned int for proper variables.
4. Fix some code-style.
v2 -> v3:
1. Remove acpi.h from common/numa.c.
2. Rename acpi_scan_nodes to numa_scan_nodes.
3. Replace u8 by uint8_t for memnodemap.
4. Use unsigned int for memnode_shift and adjust related functions (compute_hash_shift, populate_memnodemap) to use correct types for return values or parameters.
5. Use nodeid_t for nodeid and node numbers.
6. Use __read_mostly and __ro_after_init for appropriate variables.
7. Adjust the __read_mostly and __initdata location for some variables.
8. convert from plain int to unsigned for cpuid and other proper variables.
9. Use __attribute_pure__ instead of __attribute__((pure)).
10. Replace CONFIG_ACPI_NUMA by CONFIG_NUMA in numa_initmem_init.
11. Add const for some functions' parameters.
12. Move srat_disabled to common code with new name numa_disabled.
13. Fix some spaces code-style for numa_emulation.
14. Change from int to unsigned int for numa_fake.
v1 -> v2:
1. New patch in v2.
--- xen/arch/x86/include/asm/acpi.h | 1 - xen/arch/x86/include/asm/numa.h | 57 +--- xen/arch/x86/include/asm/setup.h | 1 - xen/arch/x86/numa.c | 430 +----------------------------- xen/arch/x86/smpboot.c | 2 +- xen/arch/x86/srat.c | 8 +- xen/common/Makefile | 1 + xen/common/numa.c | 442 +++++++++++++++++++++++++++++++ xen/include/xen/numa.h | 67 +++++ 9 files changed, 517 insertions(+), 492 deletions(-) create mode 100644 xen/common/numa.c
xen/arch/x86/include/asm/acpi.h +0 -1
@@ -102,7 +102,6 @@ extern unsigned long acpi_wakeup_address;#define ARCH_HAS_POWER_INIT 1extern s8 acpi_numa; -extern int acpi_scan_nodes (u64 start,u64 end);#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)extern struct acpi_sleep_info acpi_sinfo;
xen/arch/x86/include/asm/numa.h +1 -56
73 lines not on this page · raw message
xen/arch/x86/include/asm/setup.h +0 -1
@@ -20,7 +20,6 @@ void early_time_init(void);void set_nr_cpu_ids (unsigned int max_cpus); -void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn);void arch_init_memory (void );void subarch_init_memory (void );
xen/arch/x86/numa.c +1 -429
462 lines not on this page · raw message
xen/arch/x86/smpboot.c +1 -1
@@ -1350,7 +1350,7 @@ int cpu_add(uint32_t apic_id, uint32_t acpi_id, uint32_t pxm) x86_acpiid_to_apicid[acpi_id] = apic_id; -if ( !srat_disabled () ) +if ( !numa_disabled () ) {nodeid_t node =setup_node (pxm);
xen/arch/x86/srat.c +4 -4
@@ -238,7 +238,7 @@ acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *pa)unsigned pxm;nodeid_t node; -if (srat_disabled ()) +if (numa_disabled ())return ;if (pa->header.length <sizeof (struct acpi_srat_x2apic_cpu_affinity )) {bad_srat ();
@@ -274,7 +274,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)unsigned pxm;nodeid_t node; -if (srat_disabled ()) +if (numa_disabled ())return ;if (pa->header.length !=sizeof (struct acpi_srat_cpu_affinity )) {bad_srat ();
@@ -313,7 +313,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)nodeid_t node;unsigned int i; -if (srat_disabled ()) +if (numa_disabled ())return ;if (ma->header.length !=sizeof (struct acpi_srat_mem_affinity )) {bad_srat ();
@@ -520,7 +520,7 @@ void __init srat_parse_regions(paddr_t addr) }/* Use the information discovered above to actually set up the nodes. */ -int __init acpi_scan_nodes (paddr_t start,paddr_t end) +int __init numa_scan_nodes (paddr_t start,paddr_t end) {int i;nodemask_t all_nodes_parsed;
xen/common/Makefile +1 -0
@@ -26,6 +26,7 @@ obj-$(CONFIG_MEM_ACCESS) += mem_access.o obj-y += memory.o obj-y += multicall.o obj-y += notifier.o +obj-$(CONFIG_NUMA) += numa.o obj-y += page_alloc.o obj-$(CONFIG_HAS_PDX) += pdx.o obj-$(CONFIG_PERF_COUNTERS) += perfc.o
xen/common/numa.c new file +442 -0
442 lines not on this page · raw message
xen/include/xen/numa.h +67 -0
71 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH v4 2/6] xen/x86: move generically usable NUMA code from x86 to common
Quoted 15 lines · Wei Chen
xen/common/numa.c new file +12
-0@@ -0,0 +1,442 @@ +/* +* Generic VM initialization for NUMA setups. +* Copyright 2002,2003 Andi Kleen, SuSE Labs. +* Adapted for Xen: Ryan Harper <[email protected]> +*/ + +#include <xen/init.h> +#include <xen/keyhandler.h> +#include <xen/mm.h> +#include <xen/nodemask.h> +#include <xen/numa.h> +#include <xen/param.h>
xen/common/numa.c line 22
Quoted 10 lines
xen/common/numa.c new file +10
-0+#include <xen/sched.h> +#include <xen/softirq.h> + +struct node_data __ro_after_init node_data[MAX_NUMNODES ]; + +/* Mapping from pdx to node id */ +unsigned int __ro_after_init memnode_shift; +unsigned long __ro_after_init memnodemapsize; +uint8_t *__ro_after_init memnodemap; +static uint8_t __ro_after_init _memnodemap[64 ];
These last two want to use nodeid_t instead of uint8_t. Originally the latter used typeof(*memnodemap) for (I think - iirc it was me who made it that way) this reason: That way correcting memnodemap's type would have propagated without the need for further adjustments.
xen/common/numa.c line 32
Quoted 9 lines
xen/common/numa.c new file +9
-0+nodeid_t __read_mostly cpu_to_node[NR_CPUS ] = { + [0 ...NR_CPUS -1 ] =NUMA_NO_NODE +}; + +cpumask_t __read_mostly node_to_cpumask[MAX_NUMNODES ]; + +nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } }; + +bool __read_mostly numa_off;
The v3 review discussing this possibly becoming __ro_after_init didn't really finish (you didn't reply to my latest request there), but you also didn't change the attribute. Please clarify.
xen/common/numa.c line 48
Quoted 3 lines
xen/common/numa.c new file +3
-0+static int __init populate_memnodemap (const struct node *nodes, +unsigned int numnodes,unsigned int shift, +nodeid_t *nodeids)
Can't this be pointer-to-const, and then also in the caller?
xen/common/numa.c line 103
Quoted 5 lines
xen/common/numa.c new file +5
-0+static unsigned int __init extract_lsb_from_nodes (const struct node *nodes, +nodeid_t numnodes) +{ +unsigned int i; +nodeid_t nodes_used =0 ;
This once again is a variable which doesn't really hold a node ID (but instead is a counter), and hence would better be unsigned int (see ./CODING_STYLE).
Quoted 8 lines
xen/common/numa.c new file +8
-0+unsigned long spdx, epdx; +unsigned long bitfield =0 , memtop =0 ; + +for ( i =0 ; i < numnodes; i++ ) + { + spdx =paddr_to_pdx (nodes[i].start); + epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; +if ( spdx >= epdx )
xen/common/numa.c line 121
Quoted 10 lines
xen/common/numa.c new file +10
-0+continue ; + bitfield |= spdx; + nodes_used++; +if ( epdx > memtop ) + memtop = epdx; + } +if ( nodes_used <=1 ) + i =BITS_PER_LONG -1 ; +else + i =find_first_bit (&bitfield,sizeof (unsigned long )*8 );
Please add the missing blanks around * .
xen/common/numa.c line 123
Quoted 2 lines
xen/common/numa.c new file +2
-0+ memnodemapsize = (memtop >> i) +1 ; +return i;
Please add the missing blank line before the (main) return statement of the function.
xen/common/numa.c line 127
Quoted 2 lines
xen/common/numa.c new file +2
-0+int __init compute_hash_shift (const struct node *nodes, +nodeid_t numnodes,nodeid_t *nodeids)
While I agree that nodeid_t can hold all necessary values, I still don't think a cound should be expressed by nodeid_t. As above - see ./CODING_STYLE.
xen/common/numa.c line 136
Quoted 9 lines
xen/common/numa.c new file +9
-0+{ +unsigned int shift; + + shift =extract_lsb_from_nodes (nodes, numnodes); +if ( memnodemapsize <=ARRAY_SIZE (_memnodemap) ) + memnodemap = _memnodemap; +else if (allocate_cachealigned_memnodemap () ) +return -1 ; +printk (KERN_DEBUG "NUMA: Using %d for the hash shift.\n" , shift);
This wants to be %u now. I'd also like to ask to drop the full stop at this occasion.
xen/common/numa.c line 142
Quoted 5 lines
xen/common/numa.c new file +5
-0+if (populate_memnodemap (nodes, numnodes, shift, nodeids) !=1 ) + { +printk (KERN_INFO "Your memory is not aligned you need to " +"rebuild your hypervisor with a bigger NODEMAPSIZE " +"shift=%d\n" , shift);
Again %u please.
xen/common/numa.c line 150
Quoted 2 lines
xen/common/numa.c new file +2
-0+/* initialize NODE_DATA given nodeid and start/end */ +void __init setup_node_bootmem (nodeid_t nodeid,paddr_t start,paddr_t end)
Please capitalize the first letter of the comment (see ./CODING_STYLE).
xen/common/numa.c line 172
Quoted 12 lines
xen/common/numa.c new file +12
-0+void __init numa_init_array (void ) +{ +unsigned int i; +nodeid_t rr; + +/* +* There are unfortunately some poorly designed mainboards around +* that only connect memory to a single CPU. This breaks the 1:1 cpu->node +* mapping. To avoid this fill in the mapping for all possible +* CPUs, as the number of CPUs is not known yet. +* We round robin the existing nodes. +*/
Along with the style correction re-flowing of the text would have been nice, such the lines aren't wrapped seemingly randomly without utilizing permitted line length.
Quoted 12 lines
xen/common/numa.c new file +12
-0+void __init numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn) +{ +unsigned int i; +paddr_t start =pfn_to_paddr (start_pfn); +paddr_t end =pfn_to_paddr (end_pfn); + +#ifdef CONFIG_NUMA_EMU +if ( numa_fake && !numa_emulation (start_pfn, end_pfn) ) +return ; +#endif + +#ifdef CONFIG_NUMA
xen/common/numa.c line 254
Quoted 10 lines
xen/common/numa.c new file +10
-0+if ( !numa_off && !numa_scan_nodes (start, end) ) +return ; +#endif + +printk (KERN_INFO "%s\n" , + numa_off ?"NUMA turned off" :"No NUMA configuration found" ); + +printk (KERN_INFO "Faking a node at %" PRIpaddr"-%" PRIpaddr"\n" , + start, end); +/* setup dummy node covering all memory */
Please again capitalize the first character of the comment.
xen/common/numa.c line 307
Quoted 6 lines
xen/common/numa.c new file +6
-0+static void cf_check dump_numa(unsigned char key) +{ +s_time_t now =NOW (); +unsigned int i, j, n; +struct domain *d; +struct page_info *page;
Along with the various other style corrections perhaps add const here?
Quoted 4 lines
xen/common/numa.c new file +4
-0+unsigned int page_num_node[MAX_NUMNODES ]; +const struct vnuma_info *vnuma; + + printk("'%c' pressed -> dumping numa info (now = %"PRI_stime")\n", key,
xen/common/numa.c line 321
Quoted 10 lines
xen/common/numa.c new file +10
-0+ now); + +for_each_online_node ( i ) + { +paddr_t pa =pfn_to_paddr (node_start_pfn (i) +1 ); + +printk ("NODE%u start->%lu size->%lu free->%lu\n" , + i,node_start_pfn (i),node_spanned_pages (i), +avail_node_heap_pages (i)); +/* sanity check phys_to_nid() */
First char of comment again.
Jan
Re: [PATCH v4 2/6] xen/x86: move generically usable NUMA code from x86 to common
Hi Jan,
Quoted 23 lines · Jan Beulich
Quoted 22 lines · Wei Chen
xen/common/numa.c new file +19
-0@@ -0,0 +1,442 @@ +/* +* Generic VM initialization for NUMA setups. +* Copyright 2002,2003 Andi Kleen, SuSE Labs. +* Adapted for Xen: Ryan Harper <[email protected]> +*/ + +#include <xen/init.h> +#include <xen/keyhandler.h> +#include <xen/mm.h> +#include <xen/nodemask.h> +#include <xen/numa.h> +#include <xen/param.h> +#include <xen/sched.h> +#include <xen/softirq.h> + +struct node_data __ro_after_init node_data[MAX_NUMNODES ]; + +/* Mapping from pdx to node id */ +unsigned int __ro_after_init memnode_shift;
Quoted 9 lines
xen/common/numa.c line 22
Quoted 3 lines
xen/common/numa.c new file +3
-0+unsigned long __ro_after_init memnodemapsize; +uint8_t *__ro_after_init memnodemap; +static uint8_t __ro_after_init _memnodemap[64 ];These last two want to use
nodeid_tinstead ofuint8_t. Originally the latter usedtypeof(*memnodemap)for (I think - iirc it was me who made it that way) this reason: That way correcting memnodemap's type would have propagated without the need for further adjustments.
Thanks for this info, should I need to restore it to use "typeof(*memnodemap)" in next version ?
Quoted 5 lines
Quoted 5 lines
xen/common/numa.c new file +5
-0+nodeid_t __read_mostly cpu_to_node[NR_CPUS ] = { + [0 ...NR_CPUS -1 ] =NUMA_NO_NODE +}; + +cpumask_t __read_mostly node_to_cpumask[MAX_NUMNODES ];
Quoted 9 lines
xen/common/numa.c line 32
Quoted 4 lines
xen/common/numa.c new file +4
-0+ +nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } }; + +bool __read_mostly numa_off;The v3 review discussing this possibly becoming
__ro_after_initdidn't really finish (you didn't reply to my latest request there), but you also didn't change the attribute. Please clarify.
I think I had answered your question by:
Quoted 2 lines
Quoted 2 lines
I think yes, it will be used in
numa_disabledandnuma_disabledwill be called incpu_add."
And you replied me with:
Quoted 3 lines
In the original code I cannot spot such a path - can you please point out how exactly you see
numa_disabled()reachable fromcpu_add()? I'm clearly overlooking something ..."
But there is a time difference here, your reply was sent after I sent v3, maybe you didn't notice it
About the new question:cpu_add will call srat_disabled, srat_disabled will access numa_off. srat_disabled is a function without __init.
Quoted 6 lines
xen/common/numa.c line 48
Quoted 3 lines
xen/common/numa.c new file +3
-0+static int __init populate_memnodemap (const struct node *nodes, +unsigned int numnodes,unsigned int shift, +nodeid_t *nodeids)Can't this be pointer-to-const, and then also in the caller?
Yes, it's possible, I will fix it.
Quoted 10 lines
xen/common/numa.c line 103
Quoted 5 lines
xen/common/numa.c new file +5
-0+static unsigned int __init extract_lsb_from_nodes (const struct node *nodes, +nodeid_t numnodes) +{ +unsigned int i; +nodeid_t nodes_used =0 ;This once again is a variable which doesn't really hold a node ID (but instead is a counter), and hence would better be unsigned int (see ./CODING_STYLE).
Ok.
Quoted 19 lines
Quoted 8 lines
xen/common/numa.c new file +8
-0+unsigned long spdx, epdx; +unsigned long bitfield =0 , memtop =0 ; + +for ( i =0 ; i < numnodes; i++ ) + { + spdx =paddr_to_pdx (nodes[i].start); + epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; +if ( spdx >= epdx )xen/common/numa.c line 121
Quoted 10 lines
xen/common/numa.c new file +10
-0+continue ; + bitfield |= spdx; + nodes_used++; +if ( epdx > memtop ) + memtop = epdx; + } +if ( nodes_used <=1 ) + i =BITS_PER_LONG -1 ; +else + i =find_first_bit (&bitfield,sizeof (unsigned long )*8 );
Quoted 2 lines
Please add the missing blanks around * .
Ok.
Quoted 6 lines
xen/common/numa.c line 123
Quoted 2 lines
xen/common/numa.c new file +2
-0+ memnodemapsize = (memtop >> i) +1 ; +return i;Please add the missing blank line before the (main) return statement of the function.
I'll fix him and other places, if there are any.
Quoted 7 lines
xen/common/numa.c line 127
Quoted 2 lines
xen/common/numa.c new file +2
-0+int __init compute_hash_shift (const struct node *nodes, +nodeid_t numnodes,nodeid_t *nodeids)While I agree that
nodeid_tcan hold all necessary values, I still don't think a cound should be expressed bynodeid_t. As above - see ./CODING_STYLE.
Ok.
Quoted 4 lines
Quoted 4 lines
xen/common/numa.c new file +4
-0+{ +unsigned int shift; + + shift =extract_lsb_from_nodes (nodes, numnodes);
Quoted 8 lines
xen/common/numa.c line 136
Quoted 5 lines
xen/common/numa.c new file +5
-0+if ( memnodemapsize <=ARRAY_SIZE (_memnodemap) ) + memnodemap = _memnodemap; +else if (allocate_cachealigned_memnodemap () ) +return -1 ; +printk (KERN_DEBUG "NUMA: Using %d for the hash shift.\n" , shift);This wants to be %u now. I'd also like to ask to drop the full stop at this occasion.
Ok, that makes sense.
Quoted 9 lines
xen/common/numa.c line 142
Quoted 6 lines
xen/common/numa.c new file +5
-0+if (populate_memnodemap (nodes, numnodes, shift, nodeids) !=1 ) + { +printk (KERN_INFO "Your memory is not aligned you need to " +"rebuild your hypervisor with a bigger NODEMAPSIZE " +"shift=%d\n" , shift);Again %u please.
Ack.
Quoted 5 lines
xen/common/numa.c line 150
Quoted 2 lines
xen/common/numa.c new file +2
-0+/* initialize NODE_DATA given nodeid and start/end */ +void __init setup_node_bootmem (nodeid_t nodeid,paddr_t start,paddr_t end)Please capitalize the first letter of the comment (see ./CODING_STYLE).
Ok.
Quoted 8 lines
Quoted 8 lines
xen/common/numa.c new file +8
-0+void __init numa_init_array (void ) +{ +unsigned int i; +nodeid_t rr; + +/* +* There are unfortunately some poorly designed mainboards around +* that only connect memory to a single CPU. This breaks the 1:1 cpu->node
Quoted 9 lines
xen/common/numa.c line 172
Quoted 4 lines
xen/common/numa.c new file +4
-0+* mapping. To avoid this fill in the mapping for all possible +* CPUs, as the number of CPUs is not known yet. +* We round robin the existing nodes. +*/ Along with the style correction re-flowing of the text would have been nice, such the lines aren't wrapped seemingly randomly without utilizing permitted line length.
I will adjust it.
Quoted 16 lines
Quoted 16 lines
xen/common/numa.c new file +16
-0+void __init numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn) +{ +unsigned int i; +paddr_t start =pfn_to_paddr (start_pfn); +paddr_t end =pfn_to_paddr (end_pfn); + +#ifdef CONFIG_NUMA_EMU +if ( numa_fake && !numa_emulation (start_pfn, end_pfn) ) +return ; +#endif + +#ifdef CONFIG_NUMA +if ( !numa_off && !numa_scan_nodes (start, end) ) +return ; +#endif +
Quoted 9 lines
xen/common/numa.c line 254
Quoted 6 lines
xen/common/numa.c new file +6
-0+printk (KERN_INFO "%s\n" , + numa_off ?"NUMA turned off" :"No NUMA configuration found" ); + +printk (KERN_INFO "Faking a node at %" PRIpaddr"-%" PRIpaddr"\n" , + start, end); +/* setup dummy node covering all memory */ Please again capitalize the first character of the comment.
Ok.
Quoted 9 lines
xen/common/numa.c line 307
Quoted 6 lines
xen/common/numa.c new file +6
-0+static void cf_check dump_numa(unsigned char key) +{ +s_time_t now =NOW (); +unsigned int i, j, n; +struct domain *d; +struct page_info *page;Along with the various other style corrections perhaps add const here?
I will add it.
Quoted 8 lines
Quoted 8 lines
xen/common/numa.c new file +8
-0+unsigned int page_num_node[MAX_NUMNODES ]; +const struct vnuma_info *vnuma; + +printk ("'%c' pressed -> dumping numa info (now = %" PRI_stime")\n" , key, + now); + +for_each_online_node ( i ) + {
Quoted 9 lines
xen/common/numa.c line 321
Quoted 6 lines
xen/common/numa.c new file +6
-0+paddr_t pa =pfn_to_paddr (node_start_pfn (i) +1 ); + +printk ("NODE%u start->%lu size->%lu free->%lu\n" , + i,node_start_pfn (i),node_spanned_pages (i), +avail_node_heap_pages (i)); +/* sanity check phys_to_nid() */ First char of comment again.
Ok.
Thanks.
Wei Chen
Quoted 1 line
Jan
Re: [PATCH v4 2/6] xen/x86: move generically usable NUMA code from x86 to common
Quoted 34 lines · Wei Chen
Quoted 23 lines · Jan Beulich
Quoted 22 lines · Wei Chen
xen/common/numa.c new file +19
-0@@ -0,0 +1,442 @@ +/* +* Generic VM initialization for NUMA setups. +* Copyright 2002,2003 Andi Kleen, SuSE Labs. +* Adapted for Xen: Ryan Harper <[email protected]> +*/ + +#include <xen/init.h> +#include <xen/keyhandler.h> +#include <xen/mm.h> +#include <xen/nodemask.h> +#include <xen/numa.h> +#include <xen/param.h> +#include <xen/sched.h> +#include <xen/softirq.h> + +struct node_data __ro_after_init node_data[MAX_NUMNODES ]; + +/* Mapping from pdx to node id */ +unsigned int __ro_after_init memnode_shift;Quoted 9 lines
xen/common/numa.c line 22
Quoted 3 lines
xen/common/numa.c new file +3
-0+unsigned long __ro_after_init memnodemapsize; +uint8_t *__ro_after_init memnodemap; +static uint8_t __ro_after_init _memnodemap[64 ];These last two want to use
nodeid_tinstead ofuint8_t. Originally the latter usedtypeof(*memnodemap)for (I think - iirc it was me who made it that way) this reason: That way correcting memnodemap's type would have propagated without the need for further adjustments.
Quoted 2 lines
Thanks for this info, should I need to restore it to use "
typeof(*memnodemap)" in next version ?
That would be more in line with the original code, but it's not strictly necessary once nodeid_t if properly used for these variables. I'd leave it up to you as long as you switch to nodeid_t.
Quoted 19 lines
Quoted 5 lines
Quoted 5 lines
xen/common/numa.c new file +5
-0+nodeid_t __read_mostly cpu_to_node[NR_CPUS ] = { + [0 ...NR_CPUS -1 ] =NUMA_NO_NODE +}; + +cpumask_t __read_mostly node_to_cpumask[MAX_NUMNODES ];Quoted 9 lines
xen/common/numa.c line 32
Quoted 4 lines
xen/common/numa.c new file +4
-0+ +nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } }; + +bool __read_mostly numa_off;The v3 review discussing this possibly becoming
__ro_after_initdidn't really finish (you didn't reply to my latest request there), but you also didn't change the attribute. Please clarify.I think I had answered your question by:
Quoted 2 lines
Quoted 2 lines
I think yes, it will be used in
numa_disabledandnuma_disabledwill be called incpu_add."
Quoted 7 lines
And you replied me with:
Quoted 3 lines
In the original code I cannot spot such a path - can you please point out how exactly you see
numa_disabled()reachable fromcpu_add()? I'm clearly overlooking something ..."But there is a time difference here, your reply was sent after I sent v3, maybe you didn't notice it
Which suggests you might better have waited with sending v3 until the discussion had settled.
Quoted 3 lines
About the new question:
cpu_addwill callsrat_disabled,srat_disabledwill accessnuma_off.srat_disabledis a function without__init.
But the request wasn't to make the variable __initdata. That would be wrong of course. Since srat_disabled() only reads numa_off, __ro_after_init does look usable to me.
Jan
RE: [PATCH v4 2/6] xen/x86: move generically usable NUMA code from x86 to common
Hi Jan,
Quoted 46 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年9月8日 19:42
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [PATCH v4 2/6] xen/x86: move generically usable NUMA code from x86 to commonQuoted 34 lines · Wei Chen
Quoted 32 lines · Jan Beulich
Quoted 15 lines · Wei Chen
xen/common/numa.c new file +12
-0@@ -0,0 +1,442 @@ +/* +* Generic VM initialization for NUMA setups. +* Copyright 2002,2003 Andi Kleen, SuSE Labs. +* Adapted for Xen: Ryan Harper <[email protected]> +*/ + +#include <xen/init.h> +#include <xen/keyhandler.h> +#include <xen/mm.h> +#include <xen/nodemask.h> +#include <xen/numa.h> +#include <xen/param.h> xen/common/numa.c line 22
Quoted 10 lines
xen/common/numa.c new file +10
-0+#include <xen/sched.h> +#include <xen/softirq.h> + +struct node_data __ro_after_init node_data[MAX_NUMNODES ]; + +/* Mapping from pdx to node id */ +unsigned int __ro_after_init memnode_shift; +unsigned long __ro_after_init memnodemapsize; +uint8_t *__ro_after_init memnodemap; +static uint8_t __ro_after_init _memnodemap[64 ];These last two want to use
nodeid_tinstead ofuint8_t. Originally the latter usedtypeof(*memnodemap)for (I think - iirc it was me who made it that way) this reason: That way correcting memnodemap's type would have propagated without the need for further adjustments.
Quoted 7 lines
Quoted 2 lines
Thanks for this info, should I need to restore it to use "
typeof(*memnodemap)" in next version ?That would be more in line with the original code, but it's not strictly necessary once
nodeid_tif properly used for these variables. I'd leave it up to you as long as you switch tonodeid_t.
Ok, I will think more about it in next version.
Quoted 30 lines
Quoted 19 lines
Quoted 5 lines
Quoted 5 lines
xen/common/numa.c new file +5
-0+nodeid_t __read_mostly cpu_to_node[NR_CPUS ] = { + [0 ...NR_CPUS -1 ] =NUMA_NO_NODE +}; + +cpumask_t __read_mostly node_to_cpumask[MAX_NUMNODES ];Quoted 9 lines
xen/common/numa.c line 32
Quoted 4 lines
xen/common/numa.c new file +4
-0+ +nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } }; + +bool __read_mostly numa_off;The v3 review discussing this possibly becoming
__ro_after_initdidn't really finish (you didn't reply to my latest request there), but you also didn't change the attribute. Please clarify.I think I had answered your question by:
Quoted 2 lines
Quoted 2 lines
I think yes, it will be used in
numa_disabledandnuma_disabledwill be called incpu_add."Quoted 7 lines
And you replied me with:
Quoted 3 lines
In the original code I cannot spot such a path - can you please point out how exactly you see
numa_disabled()reachable fromcpu_add()? I'm clearly overlooking something ..."But there is a time difference here, your reply was sent after I sent v3, maybe you didn't notice it
Which suggests you might better have waited with sending v3 until the discussion had settled.
Quoted 8 lines
Quoted 3 lines
About the new question:
cpu_addwill callsrat_disabled,srat_disabledwill accessnuma_off.srat_disabledis a function without__init.But the request wasn't to make the variable
__initdata. That would be wrong of course. Sincesrat_disabled()only readsnuma_off,__ro_after_initdoes look usable to me.
Oh, yes, you're right. I had thought wrong. I will correct this.
Cheers,
Wei Chen.
Quoted 1 line
Jan
PATCH v4 3/6 xen/x86: Use ASSERT instead of VIRTUAL_BUG_ON for phys_to_nid
In reply to Wei ChenVIRTUAL_BUG_ON is an empty macro used in phys_to_nid. This results in two lines of error-checking code in phys_to_nid that is not actually working and causing two compilation errors:
1. error: "MAX_NUMNODES" undeclared (first use in this function). This is because in the common header file, "MAX_NUMNODES" is defined after the common header file includes the ARCH header file, where phys_to_nid has attempted to use "MAX_NUMNODES". This error was resolved after we moved the phys_to_nid from x86 ARCH header file to common header file.
2. error: wrong type argument to unary exclamation mark. This is because, the error-checking code contains !node_data[nid]. But node_data is a data structure variable, it's not a pointer.
So, in this patch, we use ASSERT instead of VIRTUAL_BUG_ON to enable the two lines of error-checking code. And fix the left compilation errors by replacing !node_data[nid] to !node_data[nid].node_spanned_pages. Although NUMA allows one node can only have CPUs but without any memory. And node with 0 bytes of memory might have an entry in memnodemap[] theoretically. But that doesn't mean phys_to_nid can find any valid address from a node with 0 bytes memory.
---
v3 -> v4:
no change.
v2 -> v3:
1. Remove unnecessary change items in history.
2. Add Acked-by.
v1 -> v2:
1. Use ASSERT to replace VIRTUAL_BUG_ON in phys_to_nid.
2. Adjust the conditional express for ASSERT.
3. Refine the justification of using !node_data[nid].node_spanned_pages.
--- xen/include/xen/numa.h | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
xen/include/xen/numa.h +2 -4
@@ -36,8 +36,6 @@ struct node {extern int compute_hash_shift (const struct node *nodes,nodeid_t numnodes,nodeid_t *nodeids); -#define VIRTUAL_BUG_ON (x) -extern bool numa_off;extern void numa_add_cpu (unsigned int cpu);
@@ -70,9 +68,9 @@ extern struct node_data node_data[]; static inline nodeid_t __attribute_pure__ phys_to_nid(paddr_t addr) {nodeid_t nid; -VIRTUAL_BUG_ON ((paddr_to_pdx (addr) >> memnode_shift) >= memnodemapsize); +ASSERT ((paddr_to_pdx (addr) >> memnode_shift) < memnodemapsize); nid = memnodemap[paddr_to_pdx (addr) >> memnode_shift]; -VIRTUAL_BUG_ON (nid >=MAX_NUMNODES || !node_data[nid]); +ASSERT (nid <MAX_NUMNODES && node_data[nid].node_spanned_pages);return nid; }
Signature · 1 line
2.25.1
PATCH v4 4/6 xen/x86: use arch_get_ram_range to get information from E820 map
In reply to Wei ChenThe sanity check of nodes_cover_memory is also a requirement of other architectures that support NUMA. But now, the code of nodes_cover_memory is tied to the x86 E820. In this case, we introduce arch_get_ram_range to decouple architecture specific memory map from this function. This means, other architectures like Arm can also use it to check its node and memory coverage from bootmem info.
Depends arch_get_ram_range, we make nodes_cover_memory become architecture independent. We also use neutral words to replace SRAT and E820 in the print message of this function. This will to make the massage seems more common.
As arch_get_ram_range use unsigned int for index, we also adjust the index in nodes_cover_memory from int to unsigned int.
---
v3 -> v4:
1. Move function comment to header file.
2. Use bool for found, and add a new "err" for the return value of arch_get_ram_range.
3. Use -ENODATA instead of -EINVAL for non-RAM type ranges. v2 -> v3:
1. Rename arch_get_memory_map to arch_get_ram_range.
2. Use -ENOENT instead of -ENODEV to indicate end of memory map.
3. Add description to code comment that arch_get_ram_range returns RAM range in [start, end) format.
v1 -> v2:
1. Use arch_get_memory_map to replace arch_get_memory_bank_range and arch_get_memory_bank_number.
2. Remove the !start || !end check, because caller guarantee these two pointers will not be NULL.
--- xen/arch/x86/numa.c | 15 +++++++++++++++ xen/arch/x86/srat.c | 30 ++++++++++++++++++------------ xen/include/xen/numa.h | 13 +++++++++++++ 3 files changed, 46 insertions(+), 12 deletions(-)
xen/arch/x86/numa.c +15 -0
@@ -9,6 +9,7 @@#include <xen/nodemask.h> #include <xen/numa.h> #include <asm/acpi.h> +#include <asm/e820.h> #ifndef Dprintk #define Dprintk(x...)
@@ -93,3 +94,17 @@ unsigned int __init arch_get_dma_bitsize(void)flsl (node_start_pfn (node) +node_spanned_pages (node) /4 -1 ) + PAGE_SHIFT, 32); } + +int __init arch_get_ram_range (unsigned int idx,paddr_t *start,paddr_t *end) +{ +if ( idx >= e820.nr_map ) +return -ENOENT ; + +if ( e820.map[idx].type !=E820_RAM ) +return -ENODATA ; + + *start = e820.map[idx].addr; + *end = *start + e820.map[idx].size; + +return 0 ; +}
xen/arch/x86/srat.c +18 -12
@@ -428,37 +428,43 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)Make sure the PXMs cover all memory. */ static int __init nodes_cover_memory (void ) { -int i; +unsigned int i; -for (i =0 ; i < e820.nr_map; i++) { -int j, found; +for (i =0 ; ; i++) { +int err; +unsigned int j; +bool found;paddr_t start, end; -if (e820.map[i].type !=E820_RAM ) { -continue ; - } +/* Try to loop memory map from index 0 to end to get RAM ranges. */ + err =arch_get_ram_range (i, &start, &end); - start = e820.map[i].addr; - end = e820.map[i].addr + e820.map[i].size; +/* Reach the end of arch's memory map */ +if (err == -ENOENT ) +break ; + +/* Index relate entry is not RAM, skip it. */ +if (err) +continue ;do { - found =0 ; + found = false;for_each_node_mask (j, memory_nodes_parsed)if (start < nodes[j].end && end > nodes[j].start) {if (start >= nodes[j].start) { start = nodes[j].end; - found =1 ; + found = true; }if (end <= nodes[j].end) { end = nodes[j].start; - found =1 ; + found = true; } } }while (found && start < end);if (start < end) { -printk (KERN_ERR "SRAT: No PXM for e820 range: " +printk (KERN_ERR "NUMA: No NODE for RAM range: " "[%" PRIpaddr", %" PRIpaddr"]\n" , start, end -1 );return 0 ; }
xen/include/xen/numa.h +13 -0
@@ -81,6 +81,19 @@ static inline nodeid_t __attribute_pure__ phys_to_nid(paddr_t addr)#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ NODE_DATA(nid)->node_spanned_pages) +/* +* This function provides the ability for caller to get one RAM entry +* from architectural memory map by index. +* +* This function will return zero if it can return a proper RAM entry. +* otherwise it will return -ENOENT for out of scope index, or return +* -ENODATA for non-RAM type memory entry. +* +* Note: the range is exclusive at the end, e.g. [start, end). +*/ +extern int arch_get_ram_range (unsigned int idx, +paddr_t *start,paddr_t *end); +#endif #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
Re: [PATCH v4 4/6] xen/x86: use arch_get_ram_range to get information from E820 map
Quoted 13 lines · Wei Chen
The sanity check of
nodes_cover_memoryis also a requirement of other architectures that support NUMA. But now, the code ofnodes_cover_memoryis tied to the x86 E820. In this case, we introducearch_get_ram_rangeto decouple architecture specific memory map from this function. This means, other architectures like Arm can also use it to check its node and memory coverage from bootmem info.Depends
arch_get_ram_range, we makenodes_cover_memorybecome architecture independent. We also use neutral words to replace SRAT and E820 in the print message of this function. This will to make the massage seems more common.
Quoted 4 lines
As
arch_get_ram_rangeuse unsigned int for index, we also adjust the index innodes_cover_memoryfrom int to unsigned int.Signed-off-by Wei Chen <[email protected]>
albeit still with a couple of suggestions:
Quoted 18 lines
xen/arch/x86/srat.c +5
-4@@ -428,37 +428,43 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)Make sure the PXMs cover all memory. */ static int __init nodes_cover_memory (void ) { -int i; +unsigned int i; -for (i =0 ; i < e820.nr_map; i++) { -int j, found; +for (i =0 ; ; i++) { +int err; +unsigned int j; +bool found;paddr_t start, end; -if (e820.map[i].type !=E820_RAM ) {
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 444
Quoted 10 lines
xen/arch/x86/srat.c +5
-4-continue ; - } +/* Try to loop memory map from index 0 to end to get RAM ranges. */ + err =arch_get_ram_range (i, &start, &end); - start = e820.map[i].addr; - end = e820.map[i].addr + e820.map[i].size; +/* Reach the end of arch's memory map */ +if (err == -ENOENT ) +break ;
Such a comment ahead of an if() is often better put as a question, e.g. "Reached the end of the memory map?" here or, if you dislike using a question, "Exit the loop at the end of the memory map".
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 448
Quoted 3 lines
xen/arch/x86/srat.c +3
-0+/* Index relate entry is not RAM, skip it. */ +if (err) +continue ;
And then perhaps "Skip non-RAM entries" here.
xen/include/xen/numa.h static inline nodeid_t __attribute_pure__ phys_to_nid(paddr_t addr) · line 90
Quoted 13 lines
xen/include/xen/numa.h +7
-0@@ -81,6 +81,19 @@ static inline nodeid_t __attribute_pure__ phys_to_nid(paddr_t addr)#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ NODE_DATA(nid)->node_spanned_pages) +/* +* This function provides the ability for caller to get one RAM entry +* from architectural memory map by index. +* +* This function will return zero if it can return a proper RAM entry. +* otherwise it will return -ENOENT for out of scope index, or return +* -ENODATA for non-RAM type memory entry.
The way you've implemented things, -ENODATA isn't special anymore, so better wouldn't be called out as special here. May I suggest to at least insert "e.g." in front of it? (An alternative would be to check for -ENODATA in nodes_cover_memory() again, followed by ASSERT(!err).)
xen/include/xen/numa.h static inline nodeid_t __attribute_pure__ phys_to_nid(paddr_t addr) · line 92
Quoted 1 line
xen/include/xen/numa.h +1
-0+* Note: the range is exclusive at the end, e.g. [start, end).
Perhaps better [*start, *end) to match ...
xen/include/xen/numa.h static inline nodeid_t __attribute_pure__ phys_to_nid(paddr_t addr) · line 95
Quoted 3 lines
xen/include/xen/numa.h +3
-0+*/ +extern int arch_get_ram_range (unsigned int idx, +paddr_t *start,paddr_t *end);
... this?
Jan
RE: [PATCH v4 4/6] xen/x86: use arch_get_ram_range to get information from E820 map
Hi Jan,
Quoted 25 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年9月8日 20:14
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [PATCH v4 4/6] xen/x86: usearch_get_ram_rangeto get information from E820 mapQuoted 13 lines · Wei Chen
The sanity check of
nodes_cover_memoryis also a requirement of other architectures that support NUMA. But now, the code ofnodes_cover_memoryis tied to the x86 E820. In this case, we introducearch_get_ram_rangeto decouple architecture specific memory map from this function. This means, other architectures like Arm can also use it to check its node and memory coverage from bootmem info.Depends
arch_get_ram_range, we makenodes_cover_memorybecome architecture independent. We also use neutral words to replace SRAT and E820 in the print message of this function. This will to make the massage seems more common.
Quoted 8 lines
Quoted 4 lines
As
arch_get_ram_rangeuse unsigned int for index, we also adjust the index innodes_cover_memoryfrom int to unsigned int.Signed-off-by Wei Chen <[email protected]>Reviewed-by Jan Beulich <[email protected]>albeit still with a couple of suggestions:
Thanks, I will adjust the code comments to address your suggestions.
Quoted 72 lines
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct
Quoted 3 lines
xen/arch/x86/srat.c +0
-0@@ -428,37 +428,43 @@ acpi_numa_memory_affinity_init(const struct
acpi_srat_mem_affinity*ma)Quoted 8 lines
xen/arch/x86/srat.c +1
-3Make sure the PXMs cover all memory. */ static int __init nodes_cover_memory (void ) { -int i; +unsigned int i; -for (i =0 ; i < e820.nr_map; i++) { -int j, found;xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
Quoted 10 lines
xen/arch/x86/srat.c +5
-3+for (i =0 ; ; i++) { +int err; +unsigned int j; +bool found;paddr_t start, end; -if (e820.map[i].type !=E820_RAM ) { -continue ; - } +/* Try to loop memory map from index 0 to end to get RAM ranges. */
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 444
Quoted 7 lines
xen/arch/x86/srat.c +4
-2+ err =arch_get_ram_range (i, &start, &end); - start = e820.map[i].addr; - end = e820.map[i].addr + e820.map[i].size; +/* Reach the end of arch's memory map */ +if (err == -ENOENT ) +break ;Such a comment ahead of an
if()is often better put as a question, e.g. "Reached the end of the memory map?" here or, if you dislike using a question, "Exit the loop at the end of the memory map".xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 448
Quoted 3 lines
xen/arch/x86/srat.c +3
-0+/* Index relate entry is not RAM, skip it. */ +if (err) +continue ;And then perhaps "Skip non-RAM entries" here.
xen/include/xen/numa.h static inline nodeid_t __attribute_pure__
Quoted 3 lines
xen/include/xen/numa.h +0
-0@@ -81,6 +81,19 @@ static inline nodeid_t __attribute_pure__
phys_to_nid(paddr_t addr)xen/include/xen/numa.h static inline nodeid_t __attribute_pure__ phys_to_nid(paddr_t addr) · line 90
Quoted 10 lines
xen/include/xen/numa.h +7
-0#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ NODE_DATA(nid)->node_spanned_pages) +/* +* This function provides the ability for caller to get one RAM entry +* from architectural memory map by index. +* +* This function will return zero if it can return a proper RAM entry. +* otherwise it will return -ENOENT for out of scope index, or return +* -ENODATA for non-RAM type memory entry. The way you've implemented things,
-ENODATAisn't special anymore, so better wouldn't be called out as special here. May I suggest to at least insert "e.g." in front of it? (An alternative would be to check for-ENODATAinnodes_cover_memory()again, followed by ASSERT(!err).)xen/include/xen/numa.h static inline nodeid_t __attribute_pure__ phys_to_nid(paddr_t addr) · line 92
Quoted 1 line
xen/include/xen/numa.h +1
-0+* Note: the range is exclusive at the end, e.g. [start, end). Perhaps better [*start, *end) to match ...
xen/include/xen/numa.h static inline nodeid_t __attribute_pure__ phys_to_nid(paddr_t addr) · line 95
Quoted 3 lines
xen/include/xen/numa.h +3
-0+*/ +extern int arch_get_ram_range (unsigned int idx, +paddr_t *start,paddr_t *end);... this?
Jan
PATCH v4 5/6 xen/x86: move NUMA scan nodes codes from x86 to common
In reply to Wei Chenx86 has implemented a set of codes to scan NUMA nodes. These codes will parse NUMA memory and processor information from ACPI SRAT table. But except some ACPI specific codes, most of the scan codes like memory blocks validation, node memory range updates and some sanity check can be reused by other NUMA implementation.
So in this patch, we move some variables and related functions for NUMA memory and processor to common as library. At the same time, numa_set_processor_nodes_parsed has been introduced for ACPI specific code to update processor parsing results. With this helper, we can reuse most of NUMA memory affinity init code from ACPI. As bad_srat and node_to_pxm functions have been used in common code to do architectural fallback and node to architectural node info translation. But it doesn't make sense to reuse the functions names in common code, we have rename them to neutral names as well.
PXM is an ACPI specific item, we can't use it in common code directly. As an alternative, we extend the parameters of numa_update_node_memblks. The caller can pass the PXM as print messages' prefix or as architectural node id. And we introduced a CONFIG_HAS_NUMA_NODE_FWID to prevent print the mapping between node id and architectural node id for those architectures do not have architectural node id. In this case, we do not need to retain a lot of per-arch code but still can print architectural log messages for different NUMA implementations.
mem_hotplug also has been accessing by common code, except x86, other architectures like Arm will also want to implement memory hotplug in future. We export mem_hotplug to common will not bring any harm for Arm and we also can reduce some per-arch helpers to access mem_hotplug.
As asm/acpi.h has been removed from common/numa.c, we have to move NR_NODE_MEMBLKS from asm/acpi.h to xen/numa.h in this patch as well.
---
v3 -> v4:
1. Use bool as return value for functions that only return 0/1 or 0/-EINVAL.
2. Move mem_hotplug to a proper place in mm.h
3. Remove useless "size" in numa_scan_nodes.
4. Use unsigned int or const for proper variables.
5. Fix code-style.
6. Add init_as_disable as arch_numa_disabled parameter.
7. Add CONFIG_HAS_NUMA_NODE_FWID to gate print the mapping between node id and architectural node id (fw node id).
v2 -> v3:
1. Add __ro_after_init to proper variables.
2. Rename bad_srat to numa_fw_bad.
3. Rename node_to_pxm to numa_node_to_arch_nid.
4. Merge patch#7 and #8 into this patch.
5. Correct int to unsigned int in proper places.
6. Move NR_NODE_MEMBLKS from x86/acpi.h to common/numa.h
7. Drop helpers to access mem_hotplug, we export mem_hotplug from x86/mm.c to common/page_alloc.c
v1 -> v2:
1. Add code comment for numa_update_node_memblks to explain:
Assumes all memory regions belonging to a single node are in one chunk. Holes between them will be included in the node.
2. Merge this single patch instead of serval patches to move x86 SRAT code to common.
3. Export node_to_pxm to keep pxm information in NUMA scan nodes error messages.
4. Change the code style to target file's Xen code-style.
5. Adjust some __init and __initdata for some functions and variables.
6. Merge two patches into this patch:
1. replace CONFIG_ACPI_NUMA by CONFIG_NUMA.
2. replace "SRAT" texts.
7. Turn numa_scan_nodes to static.
--- xen/arch/x86/include/asm/acpi.h | 1 - xen/arch/x86/include/asm/mm.h | 2 - xen/arch/x86/include/asm/numa.h | 3 +- xen/arch/x86/mm.c | 2 - xen/arch/x86/numa.c | 7 +- xen/arch/x86/srat.c | 311 +++---------------------------- xen/common/numa.c | 321 +++++++++++++++++++++++++++++++- xen/common/page_alloc.c | 2 + xen/drivers/acpi/Kconfig | 1 + xen/include/xen/mm.h | 2 + xen/include/xen/numa.h | 12 +- 11 files changed, 363 insertions(+), 301 deletions(-)
xen/arch/x86/include/asm/acpi.h +0 -1
@@ -102,7 +102,6 @@ extern unsigned long acpi_wakeup_address;#define ARCH_HAS_POWER_INIT 1extern s8 acpi_numa; -#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)extern struct acpi_sleep_info acpi_sinfo;#define acpi_video_flags bootsym(video_flags)
xen/arch/x86/include/asm/mm.h +0 -2
@@ -474,8 +474,6 @@ static inline int get_page_and_type(struct page_info *page,ASSERT (((_p)->count_info & PGC_count_mask) !=0 ); \ASSERT (page_get_owner (_p) == (_d)) -extern paddr_t mem_hotplug; -/****************************************************************************** * With shadow pagetables, the different kinds of address start * to get get confusing.
xen/arch/x86/include/asm/numa.h +1 -2
@@ -10,6 +10,7 @@ typedef u8 nodeid_t;extern int srat_rev;extern nodeid_t pxm_to_node (unsigned int pxm); +extern unsigned int numa_node_to_arch_nid (nodeid_t n);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
@@ -22,8 +23,6 @@ extern void init_cpu_to_node(void);#define arch_want_default_dmazone () (num_online_nodes() > 1) -extern int valid_numa_range (paddr_t start,paddr_t end,nodeid_t node); -void srat_parse_regions (paddr_t addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );
xen/arch/x86/mm.c +0 -2
@@ -157,8 +157,6 @@ l1_pgentry_t __section(".bss.page_aligned") __aligned(PAGE_SIZE) l1_pgentry_t __section(".bss.page_aligned") __aligned(PAGE_SIZE) l1_fixmap_x[L1_PAGETABLE_ENTRIES ]; -paddr_t __read_mostly mem_hotplug; -/* Frame table size in pages. */ unsigned long max_page;unsigned long total_pages;
xen/arch/x86/numa.c +5 -2
@@ -41,9 +41,12 @@ int __init arch_numa_setup(const char *opt)return -EINVAL ; } -bool arch_numa_disabled (void ) +bool arch_numa_disabled (bool init_as_disable) { -return acpi_numa <0 ; +if ( !init_as_disable ) +return acpi_numa <0 ; + +return acpi_numa <=0 ; }/*
xen/arch/x86/srat.c +23 -288
402 lines not on this page · raw message
xen/common/numa.c +319 -2
339 lines not on this page · raw message
xen/common/page_alloc.c +2 -0
@@ -159,6 +159,8 @@#define PGT_TYPE_INFO_INITIALIZER 0#endif +paddr_t __read_mostly mem_hotplug; +/* * Comma-separated list of hexadecimal page numbers containing bad bytes. * e.g. 'badpage=0x3f45,0x8a321'.
xen/drivers/acpi/Kconfig +1 -0
@@ -7,4 +7,5 @@ config ACPI_LEGACY_TABLES_LOOKUP config ACPI_NUMA bool + select HAS_NUMA_NODE_FWID select NUMA
xen/include/xen/mm.h +2 -0
@@ -148,6 +148,8 @@ int assign_page(/* Dump info to serial console */ void arch_dump_shared_mem_info (void ); +extern paddr_t mem_hotplug; +/* * Extra fault info types which are used to further describe * the source of an access violation.
xen/include/xen/numa.h +10 -2
@@ -11,6 +11,7 @@#define NUMA_NO_DISTANCE 0xFF#define MAX_NUMNODES (1 << NODES_SHIFT) +#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)#define vcpu_to_node (v) (cpu_to_node((v)->processor))
@@ -42,10 +43,10 @@ extern void numa_add_cpu(unsigned int cpu);extern void numa_init_array (void );extern void numa_set_node (unsigned int cpu,nodeid_t node);extern void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn); -extern int numa_scan_nodes (paddr_t start,paddr_t end); +extern void numa_fw_bad (void );extern int arch_numa_setup (const char *opt); -extern bool arch_numa_disabled (void ); +extern bool arch_numa_disabled (bool init_as_disable);extern void setup_node_bootmem (nodeid_t nodeid,paddr_t start,paddr_t end);static inline void clear_node_cpumask (unsigned int cpu)
@@ -93,6 +94,13 @@ static inline nodeid_t __attribute_pure__ phys_to_nid(paddr_t addr)*/ extern int arch_get_ram_range (unsigned int idx,paddr_t *start,paddr_t *end); +extern bool valid_numa_range (paddr_t start,paddr_t end,nodeid_t node); +extern bool numa_memblks_available (void ); +extern bool numa_update_node_memblks (nodeid_t node,unsigned int arch_nid, +paddr_t start,paddr_t size, +const char *prefix, +bool hotplug); +extern void numa_set_processor_nodes_parsed (nodeid_t node);#endif
Signature · 1 line
2.25.1
Re: [PATCH v4 5/6] xen/x86: move NUMA scan nodes codes from x86 to common
xen/arch/x86/numa.c int __init arch_numa_setup(const char *opt) · line 44
Quoted 8 lines · Wei Chen
xen/arch/x86/numa.c +1
-1@@ -41,9 +41,12 @@ int __init arch_numa_setup(const char *opt)return -EINVAL ; } -bool arch_numa_disabled (void ) +bool arch_numa_disabled (bool init_as_disable)
I'm afraid my question as to the meaning of the name of the parameter has remained unanswered.
Quoted 18 lines
xen/arch/x86/numa.c +1
-5@@ -306,32 +218,27 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)void __init acpi_numa_memory_affinity_init (const struct acpi_srat_mem_affinity *ma) { -struct node *nd; -paddr_t nd_start, nd_end; -paddr_t start, end;unsigned pxm;nodeid_t node; -unsigned int i;if (numa_disabled ())return ;if (ma->header.length !=sizeof (struct acpi_srat_mem_affinity )) { -bad_srat (); +numa_fw_bad ();return ; }
xen/arch/x86/numa.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) · line 236
Quoted 10 lines
xen/arch/x86/numa.c +3
-3if (!(ma->flags &ACPI_SRAT_MEM_ENABLED ))return ; - start = ma->base_address; - end = start + ma->length;/* Supplement the heuristics in l1tf_calculations(). */ - l1tf_safe_maddr =max (l1tf_safe_maddr,ROUNDUP (end,PAGE_SIZE )); + l1tf_safe_maddr =max (l1tf_safe_maddr, +ROUNDUP (ma->base_address + ma->length, +PAGE_SIZE ));
Indentation:
l1tf_safe_maddr = max(l1tf_safe_maddr, ROUNDUP(ma->base_address + ma->length, PAGE_SIZE));
Quoted 10 lines
xen/arch/x86/numa.c +5
-1@@ -33,7 +48,309 @@ bool __read_mostly numa_off;bool numa_disabled (void ) { -return numa_off ||arch_numa_disabled (); +return numa_off ||arch_numa_disabled (false); +} + +void __init numa_set_processor_nodes_parsed (nodeid_t node) +{
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 65
Quoted 10 lines
xen/arch/x86/numa.c +10
-0+node_set (node, processor_nodes_parsed); +} + +bool valid_numa_range (paddr_t start,paddr_t end,nodeid_t node) +{ +unsigned int i; + +for ( i =0 ; i < num_node_memblks; i++ ) + { +struct node *nd = &node_memblk_range[i];
const? (This is particularly relevant with __ro_after_init.)
Quoted 9 lines
xen/arch/x86/numa.c +9
-0+bool __init numa_update_node_memblks (nodeid_t node,unsigned int arch_nid, +paddr_t start,paddr_t size, +const char *prefix, +bool hotplug) +{ +unsigned int i; +paddr_t end = start + size; +paddr_t nd_start = start; +paddr_t nd_end = end;
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 84
Quoted 10 lines
xen/arch/x86/numa.c +10
-0+struct node *nd = &nodes[node]; + +/* +* For the node that already has some memory blocks, we will +* expand the node memory range temporarily to check memory +* interleaves with other nodes. We will not use this node +* temp memory range to check overlaps, because it will mask +* the overlaps in same node. +* +* Node with 0 bytes memory doesn't need this expandsion.
Mind taking the opportunity and drop the 'd' from "expansion"?
Quoted 7 lines
xen/arch/x86/numa.c +7
-0+*/ +if ( nd->start != nd->end ) + { +if ( nd_start > nd->start ) + nd_start = nd->start; + +if ( nd_end < nd->end )
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 101
Quoted 10 lines
xen/arch/x86/numa.c +10
-0+ nd_end = nd->end; + } + +/* It is fine to add this area to the nodes data it will be used later */ +switch (conflicting_memblks (node, start, end, nd_start, nd_end, &i) ) + { +case OVERLAP : +if ( memblk_nodeid[i] == node ) + { +bool mismatch = !(hotplug) != !test_bit (i, memblk_hotplug);
No need to parenthesize "hotplug" here.
Quoted 8 lines
xen/arch/x86/numa.c +8
-0+printk ("%sNUMA: %s %u [%" PRIpaddr", %" PRIpaddr"] overlaps with itself [%" PRIpaddr", %" PRIpaddr"]\n" , + mismatch ?KERN_ERR :KERN_WARNING , prefix, + arch_nid, start, end -1 , + node_memblk_range[i].start, node_memblk_range[i].end -1 ); +if ( mismatch ) +return false; +break ; + }
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 119
Quoted 10 lines
xen/arch/x86/numa.c +10
-0+ +printk (KERN_ERR +"NUMA: %s %u [%" PRIpaddr", %" PRIpaddr"] overlaps with %s %u [%" PRIpaddr", %" PRIpaddr"]\n" , + prefix, arch_nid, start, end -1 , prefix, +numa_node_to_arch_nid (memblk_nodeid[i]), + node_memblk_range[i].start, node_memblk_range[i].end -1 ); +return false; + + +case INTERLEAVE :
Please don't add double blank lines anywhere (original code didn't have these); there's at least one more instance below.
Quoted 15 lines
xen/arch/x86/numa.c +15
-0+static int __init nodes_cover_memory (void ) +{ +unsigned int i; + +for ( i =0 ; ; i++ ) + { +int err; +bool found; +unsigned int j; +paddr_t start, end; + +/* Try to loop memory map from index 0 to end to get RAM ranges. */ + err =arch_get_ram_range (i, &start, &end); + +/* Reach the end of arch's memory map */
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 144
Quoted 10 lines
xen/arch/x86/numa.c +10
-0+if ( err == -ENOENT ) +break ; + +/* Index relate entry is not RAM, skip it. */ +if ( err ) +continue ; + + do { + found = false; + for_each_node_mask( j, memory_nodes_parsed )
Please be consistent with style for constructs like this one: Either you consider for_each_node_mask a pseudo-keyword (along the lines of for(;;)), then there's a blank missing ahead of the opening parenthesis. Or you consider this an ordinary identifier (i.e. the function-like macro that it is), then there shouldn't be blanks immediately inside the parentheses. (Same issue elsewhere.)
Quoted 16 lines
xen/arch/x86/numa.c +16
-0+if ( start < nodes[j].end + && end > nodes[j].start ) + { +if ( start >= nodes[j].start ) + { + start = nodes[j].end; + found = true; + } + +if ( end <= nodes[j].end ) + { + end = nodes[j].start; + found = true; + } + } + }while ( found && start < end );
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 170
Quoted 10 lines
xen/arch/x86/numa.c +10
-0+ +if ( start < end ) + { +printk (KERN_ERR "NUMA: No node for RAM range: " +"[%" PRIpaddr", %" PRIpaddr"]\n" , start, end -1 ); +return 0 ; + } + } +return 1 ; +}
Seeing the two returns (and no further ones in the function) - did you not mean to also switch to bool/true/false here?
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 172
Quoted 2 lines
xen/arch/x86/numa.c +2
-0+/* Use the information discovered above to actually set up the nodes. */ +static bool __init numa_scan_nodes (paddr_t start,paddr_t end)
Is "above" in the comment actually still accurate? Aiui the discovery is now in a different CU. Then perhaps "Use discovered information to actually set up the nodes."
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 183
Quoted 11 lines
xen/arch/x86/numa.c +11
-0+{ +unsigned int i; +nodemask_t all_nodes_parsed; + +/* First clean up the node list */ +for ( i =0 ; i <MAX_NUMNODES ; i++ ) +cutoff_node (i, start, end); + +/* When numa is on with good firmware, we can do numa scan nodes. */ +if (arch_numa_disabled (true) ) +return false;
Btw - the comment here doesn't help me figure your choice of "init_as_disabled". The wording towards the end is also a little odd, considering we're already in numa_scan_nodes(). Which further points out that really there's no scanning here, just processing, so maybe the earlier patch wants to rename the function to numa_process_nodes()?
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 193
Quoted 10 lines
xen/arch/x86/numa.c +10
-0+if ( !nodes_cover_memory () ) + { +numa_fw_bad (); +return false; + } + + memnode_shift =compute_hash_shift (node_memblk_range, num_node_memblks, + memblk_nodeid); + +if ( memnode_shift <0 )
As previously pointed out: As of patch 2 memnode_shift is unsigned, so this comparison is always false (and the latest Coverity will point this out). You can't get away here without using an intermediate (signed, i.e. plain int) variable.
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 206
Quoted 13 lines
xen/arch/x86/numa.c +13
-0+ { +printk (KERN_ERR +"NUMA: No NUMA node hash function found. Contact maintainer\n" ); +numa_fw_bad (); +return false; + } + +nodes_or (all_nodes_parsed, memory_nodes_parsed, processor_nodes_parsed); + +/* Finally register nodes */ +for_each_node_mask ( i, all_nodes_parsed ) + { +if ( nodes[i].end - nodes[i].start ==0 )
nodes[i].end == nodes[i].start ?
Quoted 5 lines
xen/arch/x86/numa.c +5
-0+printk (KERN_INFO "NUMA: node %u has no memory\n" , i); + +setup_node_bootmem (i, nodes[i].start, nodes[i].end); + } +
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 221
Quoted 10 lines
xen/arch/x86/numa.c +9
-0+for ( i =0 ; i < nr_cpu_ids; i++ ) + { +if ( cpu_to_node[i] ==NUMA_NO_NODE ) +continue ; +if ( !nodemask_test (cpu_to_node[i], &processor_nodes_parsed) ) +numa_set_node (i,NUMA_NO_NODE ); + } +numa_init_array (); +return true; }
While you said you'd check elsewhere as well, just to be sure: Please add a blank line before the function's main "return". And perhaps another one between loop and function call.
xen/drivers/acpi/Kconfig config ACPI_LEGACY_TABLES_LOOKUP · line 10
Quoted 7 lines
xen/drivers/acpi/Kconfig +1
-0@@ -7,4 +7,5 @@ config ACPI_LEGACY_TABLES_LOOKUP config ACPI_NUMA bool + select HAS_NUMA_NODE_FWID
Are you selecting an option here which doesn't exist anywhere? Or am I overlooking where this new option is being added?
Jan
RE: [PATCH v4 5/6] xen/x86: move NUMA scan nodes codes from x86 to common
Hi Jan,
Quoted 18 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年9月8日 21:03
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [PATCH v4 5/6] xen/x86: move NUMA scan nodes codes from x86 to commonQuoted 6 lines · Wei Chen
xen/arch/x86/numa.c +0
-0@@ -41,9 +41,12 @@ int __init arch_numa_setup(const char *opt)return -EINVAL ; }
Quoted 6 lines
xen/arch/x86/numa.c int __init arch_numa_setup(const char *opt) · line 44
Quoted 2 lines
xen/arch/x86/numa.c +1
-1-bool arch_numa_disabled (void ) +bool arch_numa_disabled (bool init_as_disable)I'm afraid my question as to the meaning of the name of the parameter has remained unanswered.
Sorry, I might missed some contents of your reply in v3. The name of this parameter has been bothering me for a long time, and now this is actually quite awkward. The origin of this parameter is because the current NUMA implementation will make different judgments under different usage conditions when using acpi_numa. In acpi_scan_nodes, it uses acpi_numa <= 0 as the condition for judging that ACPI NUMA is turned off. But only use acpi_numa < 0 as condition in srat_disabled and elsewhere. I use this parameter in the hope that we can keep the same semantics as the original code without changing the code of the caller.
Quoted 32 lines
xen/arch/x86/numa.c acpi_numa_processor_affinity_init(const struct
Quoted 1 line
xen/arch/x86/numa.c +0
-0@@ -306,32 +218,27 @@ acpi_numa_processor_affinity_init(const struct
acpi_srat_cpu_affinity*pa)Quoted 17 lines
xen/arch/x86/numa.c +1
-5void __init acpi_numa_memory_affinity_init (const struct acpi_srat_mem_affinity *ma) { -struct node *nd; -paddr_t nd_start, nd_end; -paddr_t start, end;unsigned pxm;nodeid_t node; -unsigned int i;if (numa_disabled ())return ;if (ma->header.length !=sizeof (struct acpi_srat_mem_affinity )) { -bad_srat (); +numa_fw_bad ();return ; }xen/arch/x86/numa.c acpi_numa_processor_affinity_init(const struct · line 236
Quoted 10 lines
xen/arch/x86/numa.c +3
-3if (!(ma->flags &ACPI_SRAT_MEM_ENABLED ))return ; - start = ma->base_address; - end = start + ma->length;/* Supplement the heuristics in l1tf_calculations(). */ - l1tf_safe_maddr =max (l1tf_safe_maddr,ROUNDUP (end,PAGE_SIZE )); + l1tf_safe_maddr =max (l1tf_safe_maddr, +ROUNDUP (ma->base_address + ma->length, +PAGE_SIZE ));Indentation:
Quoted 4 lines
l1tf_safe_maddr = max(l1tf_safe_maddr, ROUNDUP(ma->base_address + ma->length, PAGE_SIZE));
Ok.
Quoted 14 lines
Quoted 14 lines
xen/arch/x86/numa.c +9
-1@@ -33,7 +48,309 @@ bool __read_mostly numa_off;bool numa_disabled (void ) { -return numa_off ||arch_numa_disabled (); +return numa_off ||arch_numa_disabled (false); +} + +void __init numa_set_processor_nodes_parsed (nodeid_t node) +{ +node_set (node, processor_nodes_parsed); +} + +bool valid_numa_range (paddr_t start,paddr_t end,nodeid_t node)
Quoted 9 lines
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 65
Quoted 6 lines
xen/arch/x86/numa.c +6
-0+{ +unsigned int i; + +for ( i =0 ; i < num_node_memblks; i++ ) + { +struct node *nd = &node_memblk_range[i];const? (This is particularly relevant with
__ro_after_init.)
Yes, I will fix it.
Quoted 14 lines
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 66
Quoted 1 line
xen/arch/x86/numa.c +1
-0+bool __init numa_update_node_memblks(nodeid_t node, unsigned int
arch_nid,Quoted 12 lines
xen/arch/x86/numa.c +12
-0+ paddr_t start, paddr_t size, + const char *prefix, + bool hotplug) +{ + unsigned int i; +paddr_t end = start + size; +paddr_t nd_start = start; +paddr_t nd_end = end; +struct node *nd = &nodes[node]; + +/* +* For the node that already has some memory blocks, we will
Quoted 9 lines
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 84
Quoted 6 lines
xen/arch/x86/numa.c +6
-0+* expand the node memory range temporarily to check memory +* interleaves with other nodes. We will not use this node +* temp memory range to check overlaps, because it will mask +* the overlaps in same node. +* +* Node with 0 bytes memory doesn't need this expandsion. Mind taking the opportunity and drop the 'd' from "expansion"?
Ok.
Quoted 13 lines
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 95
Quoted 11 lines
xen/arch/x86/numa.c +11
-0+*/ +if ( nd->start != nd->end ) + { +if ( nd_start > nd->start ) + nd_start = nd->start; + +if ( nd_end < nd->end ) + nd_end = nd->end; + } + +/* It is fine to add this area to the nodes data it will be used later */
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 96
Quoted 1 line
xen/arch/x86/numa.c +1
-0+ switch ( conflicting_memblks(node, start, end, nd_start, nd_end,
Quoted 9 lines
&i) )
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 101
Quoted 5 lines
xen/arch/x86/numa.c +5
-0+ { +case OVERLAP : +if ( memblk_nodeid[i] == node ) + { +bool mismatch = !(hotplug) != !test_bit (i, memblk_hotplug);No need to parenthesize "hotplug" here.
Ok
Quoted 17 lines
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 102
Quoted 1 line
xen/arch/x86/numa.c +1
-0+ printk("%sNUMA: %s %u [%"PRIpaddr", %"PRIpaddr"] overlapswith itself [%"PRIpaddr", %"PRIpaddr"]\n",
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 105
Quoted 3 lines
xen/arch/x86/numa.c +3
-0+ mismatch ?KERN_ERR :KERN_WARNING , prefix, + arch_nid, start, end -1 , + node_memblk_range[i].start, node_memblk_range[i].end- 1);
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 112
Quoted 7 lines
xen/arch/x86/numa.c +7
-0+if ( mismatch ) +return false; +break ; + } + +printk (KERN_ERR + "NUMA: %s %u [%"PRIpaddr", %"PRIpaddr"] overlapswith %s %u [%"PRIpaddr", %"PRIpaddr"]\n",
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 115
Quoted 3 lines
xen/arch/x86/numa.c +3
-0+ prefix, arch_nid, start, end -1 , prefix, +numa_node_to_arch_nid (memblk_nodeid[i]), + node_memblk_range[i].start, node_memblk_range[i].end -
Quoted 9 lines
1);
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 119
Quoted 4 lines
xen/arch/x86/numa.c +4
-0+return false; + + +case INTERLEAVE :Please don't add double blank lines anywhere (original code didn't have these); there's at least one more instance below.
I will check the code and fix them.
Quoted 25 lines
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 131
Quoted 12 lines
xen/arch/x86/numa.c +12
-0+static int __init nodes_cover_memory (void ) +{ +unsigned int i; + +for ( i =0 ; ; i++ ) + { +int err; +bool found; +unsigned int j; +paddr_t start, end; + +/* Try to loop memory map from index 0 to end to get RAM ranges. */
Quoted 12 lines
xen/arch/x86/numa.c +12
-0+ err =arch_get_ram_range (i, &start, &end); + +/* Reach the end of arch's memory map */ +if ( err == -ENOENT ) +break ; + +/* Index relate entry is not RAM, skip it. */ +if ( err ) +continue ; + + do { + found = false;
Quoted 9 lines
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 144
Quoted 1 line
xen/arch/x86/numa.c +1
-0+for_each_node_mask ( j, memory_nodes_parsed )Please be consistent with style for constructs like this one: Either you consider
for_each_node_maska pseudo-keyword (along the lines of for(;;)), then there's a blank missing ahead of the opening parenthesis. Or you consider this an ordinary identifier (i.e. the function-like macro that it is), then there shouldn't be blanks immediately inside the parentheses. (Same issue elsewhere.)
I will check the code and fix them.
Quoted 21 lines
Quoted 21 lines
xen/arch/x86/numa.c +21
-0+if ( start < nodes[j].end + && end > nodes[j].start ) + { +if ( start >= nodes[j].start ) + { + start = nodes[j].end; + found = true; + } + +if ( end <= nodes[j].end ) + { + end = nodes[j].start; + found = true; + } + } + }while ( found && start < end ); + +if ( start < end ) + { +printk (KERN_ERR "NUMA: No node for RAM range: " +"[%" PRIpaddr", %" PRIpaddr"]\n" , start, end -1 );
Quoted 9 lines
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 170
Quoted 5 lines
xen/arch/x86/numa.c +5
-0+return 0 ; + } + } +return 1 ; +}Seeing the two returns (and no further ones in the function) - did you not mean to also switch to bool/true/false here?
Ok, I will switch the return value to bool and find if there still are some other functions that can switch to bool.
Quoted 8 lines
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 171
Quoted 1 line
xen/arch/x86/numa.c +1
-0+/* Use the information discovered above to actually set up the nodes. */
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 172
Quoted 1 line
xen/arch/x86/numa.c +1
-0+static bool __init numa_scan_nodes (paddr_t start,paddr_t end)Is "above" in the comment actually still accurate? Aiui the discovery is now in a different CU. Then perhaps "Use discovered information to actually set up the nodes."
Ok.
Quoted 13 lines
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 181
Quoted 9 lines
xen/arch/x86/numa.c +9
-0+{ +unsigned int i; +nodemask_t all_nodes_parsed; + +/* First clean up the node list */ +for ( i =0 ; i <MAX_NUMNODES ; i++ ) +cutoff_node (i, start, end); + +/* When numa is on with good firmware, we can do numa scan nodes. */
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 183
Quoted 2 lines
xen/arch/x86/numa.c +2
-0+if (arch_numa_disabled (true) ) +return false;
Quoted 7 lines
Btw - the comment here doesn't help me figure your choice of "
init_as_disabled". The wording towards the end is also a little odd, considering we're already innuma_scan_nodes(). Which further points out that really there's no scanning here, just processing, so maybe the earlier patch wants to rename the function tonuma_process_nodes()?
Yes, scan will make some confusion, your suggestion make sense, I will fix it in next version.
Quoted 8 lines
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 190
Quoted 7 lines
xen/arch/x86/numa.c +7
-0+if ( !nodes_cover_memory () ) + { +numa_fw_bad (); +return false; + } + + memnode_shift = compute_hash_shift(node_memblk_range,
num_node_memblks,
Quoted 9 lines
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 193
Quoted 3 lines
xen/arch/x86/numa.c +3
-0+ memblk_nodeid); + +if ( memnode_shift <0 )As previously pointed out: As of patch 2
memnode_shiftis unsigned, so this comparison is always false (and the latest Coverity will point this out). You can't get away here without using an intermediate (signed, i.e. plain int) variable.
Yes, you're right, I will introduce a new int variable for return value checking.
Quoted 9 lines
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 196
Quoted 3 lines
xen/arch/x86/numa.c +3
-0+ { + printk(KERN_ERR + "NUMA: No NUMA node hash function found. Contactmaintainer\n");
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 201
Quoted 5 lines
xen/arch/x86/numa.c +5
-0+numa_fw_bad (); +return false; + } + + nodes_or(all_nodes_parsed, memory_nodes_parsed,
Quoted 9 lines
processor_nodes_parsed);xen/arch/x86/numa.c bool __read_mostly numa_off; · line 206
Quoted 5 lines
xen/arch/x86/numa.c +5
-0+ +/* Finally register nodes */ +for_each_node_mask ( i, all_nodes_parsed ) + { +if ( nodes[i].end - nodes[i].start ==0 )nodes[i].end == nodes[i].start ?
Yes.
Quoted 11 lines
Quoted 11 lines
xen/arch/x86/numa.c +11
-0+printk (KERN_INFO "NUMA: node %u has no memory\n" , i); + +setup_node_bootmem (i, nodes[i].start, nodes[i].end); + } + +for ( i =0 ; i < nr_cpu_ids; i++ ) + { +if ( cpu_to_node[i] ==NUMA_NO_NODE ) +continue ; +if ( !nodemask_test (cpu_to_node[i], &processor_nodes_parsed) ) +numa_set_node (i,NUMA_NO_NODE );
Quoted 9 lines
xen/arch/x86/numa.c bool __read_mostly numa_off; · line 221
Quoted 4 lines
xen/arch/x86/numa.c +3
-0+ } +numa_init_array (); +return true; }While you said you'd check elsewhere as well, just to be sure: Please add a blank line before the function's main "return". And perhaps another one between loop and function call.
Ok.
Quoted 11 lines
xen/drivers/acpi/Kconfig config ACPI_LEGACY_TABLES_LOOKUP · line 10
Quoted 7 lines
xen/drivers/acpi/Kconfig +1
-0@@ -7,4 +7,5 @@ config ACPI_LEGACY_TABLES_LOOKUP config ACPI_NUMA bool + select HAS_NUMA_NODE_FWIDAre you selecting an option here which doesn't exist anywhere? Or am I overlooking where this new option is being added?
Yes, this is a new Kconfig option. Should I need to introduce in a separate patch?
Cheers,
Wei Chen
Quoted 1 line
Jan
Re: [PATCH v4 5/6] xen/x86: move NUMA scan nodes codes from x86 to common
Quoted 18 lines · Wei Chen
Quoted 11 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年9月8日 21:03Quoted 6 lines · Wei Chen
xen/arch/x86/numa.c +0
-0@@ -41,9 +41,12 @@ int __init arch_numa_setup(const char *opt)return -EINVAL ; }Quoted 6 lines
xen/arch/x86/numa.c int __init arch_numa_setup(const char *opt) · line 44
Quoted 2 lines
xen/arch/x86/numa.c +1
-1-bool arch_numa_disabled (void ) +bool arch_numa_disabled (bool init_as_disable)I'm afraid my question as to the meaning of the name of the parameter has remained unanswered.
Quoted 9 lines
Sorry, I might missed some contents of your reply in v3. The name of this parameter has been bothering me for a long time, and now this is actually quite awkward. The origin of this parameter is because the current NUMA implementation will make different judgments under different usage conditions when using
acpi_numa. Inacpi_scan_nodes, it usesacpi_numa<= 0 as the condition for judging that ACPI NUMA is turned off. But only useacpi_numa< 0 as condition insrat_disabledand elsewhere. I use this parameter in the hope that we can keep the same semantics as the original code without changing the code of the caller.
The difference is "bad only" vs "bad or no data". Maybe that's easier to express via two functions - arch_numa_disabled() (checking <= 0) and arch_numa_broken() (checking < 0)? With a single function I guess the name of the parameter would always be clumsy at best. Unless someone has a good idea for a suitable name ...
Quoted 12 lines
Quoted 11 lines
xen/drivers/acpi/Kconfig config ACPI_LEGACY_TABLES_LOOKUP · line 10
Quoted 7 lines
xen/drivers/acpi/Kconfig +1
-0@@ -7,4 +7,5 @@ config ACPI_LEGACY_TABLES_LOOKUP config ACPI_NUMA bool + select HAS_NUMA_NODE_FWIDAre you selecting an option here which doesn't exist anywhere? Or am I overlooking where this new option is being added?
Quoted 2 lines
Yes, this is a new
Kconfigoption. Should I need to introduce in a separate patch?
I don't think that'll need to be in a separate patch; it can simply be another hunk in the one here, adding the needed 2 lines (plus a blank one) to, presumably, common/Kconfig.
Jan
RE: [PATCH v4 5/6] xen/x86: move NUMA scan nodes codes from x86 to common
Hi Jan,
Quoted 42 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年9月9日 0:02
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [PATCH v4 5/6] xen/x86: move NUMA scan nodes codes from x86 to commonQuoted 11 lines · Wei Chen
Quoted 11 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年9月8日 21:03Quoted 6 lines · Wei Chen
xen/arch/x86/numa.c +0
-0@@ -41,9 +41,12 @@ int __init arch_numa_setup(const char *opt)return -EINVAL ; }Quoted 4 lines
Quoted 4 lines
xen/arch/x86/numa.c int __init arch_numa_setup(const char *opt) · line 44
Quoted 2 lines
xen/arch/x86/numa.c +1
-1-bool arch_numa_disabled (void ) +bool arch_numa_disabled (bool init_as_disable)I'm afraid my question as to the meaning of the name of the parameter
has
Quoted 4 lines
Quoted 2 lines
remained unanswered.
Sorry, I might missed some contents of your reply in v3. The name of
this
Quoted 1 line
parameter has been bothering me for a long time, and now this is
actually
Quoted 3 lines
quite awkward. The origin of this parameter is because the current NUMA implementation will make different judgments under different usage conditions when using
acpi_numa. Inacpi_scan_nodes, it usesacpi_numa<= 0
Quoted 3 lines
as the condition for judging that ACPI NUMA is turned off. But only use
acpi_numa< 0 as condition insrat_disabledand elsewhere. I use this parameter in the hope that we can keep the same semantics as the
Quoted 9 lines
original
Quoted 1 line
code without changing the code of the caller.
The difference is "bad only" vs "bad or no data". Maybe that's easier to express via two functions -
arch_numa_disabled()(checking <= 0) andarch_numa_broken()(checking < 0)? With a single function I guess the name of the parameter would always be clumsy at best. Unless someone has a good idea for a suitable name ...
Yes, I can't find a good name for the parameter, so break into two functions would be better, I will do it in next version.
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
xen/drivers/acpi/Kconfig config ACPI_LEGACY_TABLES_LOOKUP · line 10
Quoted 7 lines
xen/drivers/acpi/Kconfig +1
-0@@ -7,4 +7,5 @@ config ACPI_LEGACY_TABLES_LOOKUP config ACPI_NUMA bool + select HAS_NUMA_NODE_FWIDAre you selecting an option here which doesn't exist anywhere? Or am I overlooking where this new option is being added?
Quoted 6 lines
Quoted 2 lines
Yes, this is a new
Kconfigoption. Should I need to introduce in a separate patch?I don't think that'll need to be in a separate patch; it can simply be another hunk in the one here, adding the needed 2 lines (plus a blank one) to, presumably, common/Kconfig.
Ok.
Thanks,
Wei Chen
Quoted 2 lines
Jan
PATCH v4 6/6 xen: introduce a Kconfig option to configure NUMA nodes number
In reply to Wei ChenCurrently the maximum number of NUMA nodes is a hardcoded value. This provides little flexibility unless changing the code.
Introduce a new Kconfig option to change the maximum number of NUMA nodes conveniently. Also considering that not all architectures support NUMA, this Kconfig option is only visible on NUMA enabled architectures. Architectures not supporting NUMA still use 1 for MAX_NUMNODES.
As NODES_SHIFT is currently unused, we're taking this opportunity to remove it.
---
v3 -> v4:
1. Update the commit log to follow Jan's suggestion.
2. Add Ack-by.
v2 -> v3:
1. Fix indent.
2. Use 2-64 for node range.
v1 -> v2:
1. Add NODES_SHIFT remove message in commit log
2. Change NR_NUMA_NODES upper bound from 4095 to 255.
--- xen/arch/Kconfig | 11 +++++++++++ xen/arch/x86/include/asm/numa.h | 2 -- xen/include/xen/numa.h | 11 ++++++----- 3 files changed, 17 insertions(+), 7 deletions(-)
xen/arch/Kconfig +11 -0
@@ -17,3 +17,14 @@ config NR_CPUS For CPU cores which support Simultaneous Multi-Threading or similar technologies, this the number of logical threads which Xen will support. + +config NR_NUMA_NODES + int "Maximum number of NUMA nodes supported" + range 2 64 + default "64" + depends on NUMA + help + Controls the build-time size of various arrays and bitmaps + associated with multiple-nodes management. It is the upper bound of + the number of NUMA nodes that the scheduler, memory allocation and + other NUMA-aware components can handle.
xen/arch/x86/include/asm/numa.h +0 -2
@@ -3,8 +3,6 @@#include <xen/cpumask.h> -#define NODES_SHIFT 6 -typedef u8 nodeid_t ;extern int srat_rev;
xen/include/xen/numa.h +6 -5
@@ -3,14 +3,15 @@#include <asm/numa.h> -#ifndef NODES_SHIFT -#define NODES_SHIFT 0 -#endif -#define NUMA_NO_NODE 0xFF#define NUMA_NO_DISTANCE 0xFF -#define MAX_NUMNODES (1 << NODES_SHIFT) +#ifdef CONFIG_NR_NUMA_NODES +#define MAX_NUMNODES CONFIG_NR_NUMA_NODES +#else +#define MAX_NUMNODES 1 +#endif +#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)#define vcpu_to_node (v) (cpu_to_node((v)->processor))
Signature · 1 line
2.25.1