powerpc: Fix warning reported by verify_cpu_node_mapping()
RFC PATCH v3 1/3 powerpc: Fix warning reported by verify_cpu_node_mapping()
With commit 2fabf084b6ad ("powerpc: reorder per-cpu NUMA information's initialization"), during boottime, cpu_numa_callback() is called earlier(before their online) for each cpu, and verify_cpu_node_mapping() uses cpu_to_node() to check whether siblings are in the same node.
It skips the checking for siblings that are not online yet. So the only check done here is for the bootcpu, which is online at that time. But the per-cpu numa_node cpu_to_node() uses hasn't been set up yet (which will be set up in smp_prepare_cpus()).
So I saw something like following reported:
[ 0.000000] CPU thread siblings 1/2/3 and 0 don't belong to the same
node!
As we don't actually do the checking during this early stage, so maybe we could directly call numa_setup_cpu() in do_init_bootmem().
--- arch/powerpc/mm/numa.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/powerpc/mm/numa.c +1 -2
@@ -1128,8 +1128,7 @@ void __init do_init_bootmem(void)* early in boot, cf. smp_prepare_cpus(). */ for_each_possible_cpu (cpu) { -cpu_numa_callback (&ppc64_numa_nb,CPU_UP_PREPARE , - (void *)(unsigned long )cpu); +numa_setup_cpu ((unsigned long )cpu); } }
Signature · 1 line
1.9.1
RFC PATCH v3 2/3 powerpc: Only set numa node information for present cpus at boottime
As Nish suggested, it makes more sense to init the numa node informatiion for present cpus at boottime, which could also avoid WARN_ON(1) in numa_setup_cpu().
With this change, we also need to change the smp_prepare_cpus() to set up numa information only on present cpus.
For those possible, but not present cpus, their numa information will be set up after they are started, as the original code did before commit 2fabf084b6ad.
--- arch/powerpc/kernel/smp.c | 10 ++++++++-- arch/powerpc/mm/numa.c | 2 +- 2 files changed, 9 insertions(+), 3 deletions(-)
arch/powerpc/kernel/smp.c +8 -2
@@ -379,8 +379,11 @@ void __init smp_prepare_cpus(unsigned int max_cpus)/* * numa_node_id() works after this. */ -set_cpu_numa_node (cpu, numa_cpu_lookup_table[cpu]); -set_cpu_numa_mem (cpu,local_memory_node (numa_cpu_lookup_table[cpu])); +if (cpu_present (cpu)) { +set_cpu_numa_node (cpu, numa_cpu_lookup_table[cpu]); +set_cpu_numa_mem (cpu, +local_memory_node (numa_cpu_lookup_table[cpu])); + } }cpumask_set_cpu (boot_cpuid,cpu_sibling_mask (boot_cpuid));
@@ -728,6 +731,9 @@ void start_secondary(void *unused) }traverse_core_siblings (cpu, true); +set_numa_node (numa_cpu_lookup_table[cpu]); +set_numa_mem (local_memory_node (numa_cpu_lookup_table[cpu])); +smp_wmb ();notify_cpu_starting (cpu);set_cpu_online (cpu, true);
arch/powerpc/mm/numa.c +1 -1
@@ -1127,7 +1127,7 @@ void __init do_init_bootmem(void)* even before we online them, so that we can use cpu_to_{node,mem} * early in boot, cf. smp_prepare_cpus(). */ -for_each_possible_cpu (cpu) { +for_each_present_cpu (cpu) {numa_setup_cpu ((unsigned long )cpu); } }
Signature · 1 line
1.9.1
Re: [RFC PATCH v3 2/3] powerpc: Only set numa node information for present cpus at boottime
Quoted 7 lines · Li Zhong
As Nish suggested, it makes more sense to init the numa node informatiion for present cpus at boottime, which could also avoid
WARN_ON(1)innuma_setup_cpu().With this change, we also need to change the
smp_prepare_cpus()to set up numa information only on present cpus.
Quoted 7 lines
For those possible, but not present cpus, their numa information will be set up after they are started, as the original code did before commit
2fabf084b6ad.Cc Nishanth Aravamudan <[email protected]>Cc Nathan Fontenot <[email protected]>Signed-off-by Li Zhong <[email protected]>
Quoted 49 lines
--- arch/powerpc/kernel/smp.c | 10 ++++++++-- arch/powerpc/mm/numa.c | 2 +- 2 files changed, 9 insertions(+), 3 deletions(-)arch/powerpc/kernel/smp.c +8
-2@@ -379,8 +379,11 @@ void __init smp_prepare_cpus(unsigned int max_cpus)/* * numa_node_id() works after this. */ -set_cpu_numa_node (cpu, numa_cpu_lookup_table[cpu]); -set_cpu_numa_mem (cpu,local_memory_node (numa_cpu_lookup_table[cpu])); +if (cpu_present (cpu)) { +set_cpu_numa_node (cpu, numa_cpu_lookup_table[cpu]); +set_cpu_numa_mem (cpu, +local_memory_node (numa_cpu_lookup_table[cpu])); + } }cpumask_set_cpu (boot_cpuid,cpu_sibling_mask (boot_cpuid));@@ -728,6 +731,9 @@ void start_secondary(void *unused) }traverse_core_siblings (cpu, true); +set_numa_node (numa_cpu_lookup_table[cpu]); +set_numa_mem (local_memory_node (numa_cpu_lookup_table[cpu])); +smp_wmb ();notify_cpu_starting (cpu);set_cpu_online (cpu, true);arch/powerpc/mm/numa.c +1
-1@@ -1127,7 +1127,7 @@ void __init do_init_bootmem(void)* even before we online them, so that we can use cpu_to_{node,mem} * early in boot, cf. smp_prepare_cpus(). */ -for_each_possible_cpu (cpu) { +for_each_present_cpu (cpu) {numa_setup_cpu ((unsigned long )cpu); } }Signature · 1 line
1.9.1
Cyril Bur
Tested-byRe: [RFC PATCH v3 2/3] powerpc: Only set numa node information for present cpus at boottime
Quoted 4 lines · Nishanth Aravamudan
Quoted 3 lines · Li Zhong
As Nish suggested, it makes more sense to init the numa node informatiion for present cpus at boottime, which could also avoid
WARN_ON(1)innuma_setup_cpu().
Hit this on a Power8 LPAR. With the patchset applied the warnings no longer present.
Quoted 8 lines
Quoted 8 lines
With this change, we also need to change the
smp_prepare_cpus()to set up numa information only on present cpus.For those possible, but not present cpus, their numa information will be set up after they are started, as the original code did before commit
2fabf084b6ad.
Quoted 5 lines
Quoted 3 lines
Cc Nishanth Aravamudan <[email protected]>Cc Nathan Fontenot <[email protected]>Signed-off-by Li Zhong <[email protected]>
Quoted 55 lines
Quoted 47 lines
--- arch/powerpc/kernel/smp.c | 10 ++++++++-- arch/powerpc/mm/numa.c | 2 +- 2 files changed, 9 insertions(+), 3 deletions(-)arch/powerpc/kernel/smp.c +8
-2@@ -379,8 +379,11 @@ void __init smp_prepare_cpus(unsigned int max_cpus)/* * numa_node_id() works after this. */ -set_cpu_numa_node (cpu, numa_cpu_lookup_table[cpu]); -set_cpu_numa_mem (cpu,local_memory_node (numa_cpu_lookup_table[cpu])); +if (cpu_present (cpu)) { +set_cpu_numa_node (cpu, numa_cpu_lookup_table[cpu]); +set_cpu_numa_mem (cpu, +local_memory_node (numa_cpu_lookup_table[cpu])); + } }cpumask_set_cpu (boot_cpuid,cpu_sibling_mask (boot_cpuid));@@ -728,6 +731,9 @@ void start_secondary(void *unused) }traverse_core_siblings (cpu, true); +set_numa_node (numa_cpu_lookup_table[cpu]); +set_numa_mem (local_memory_node (numa_cpu_lookup_table[cpu])); +smp_wmb ();notify_cpu_starting (cpu);set_cpu_online (cpu, true);arch/powerpc/mm/numa.c +1
-2@@ -1127,7 +1127,7 @@ void __init do_init_bootmem(void)* even before we online them, so that we can use cpu_to_{node,mem} * early in boot, cf. smp_prepare_cpus(). */ -for_each_possible_cpu (cpu) { +for_each_present_cpu (cpu) {numa_setup_cpu ((unsigned long )cpu); } } --Quoted 2 lines
1.9.1
_______________________________________________
Linuxppc-dev mailing list
[email protected]
https://lists.ozlabs.org/listinfo/linuxppc-dev
RFC PATCH v3 3/3 powerpc: some changes in numa_setup_cpu()
In reply to Li Zhongthis patches changes some error handling logics in numa_setup_cpu(), when cpu node is not found, so:
if the cpu is possible, but not present, -1 is kept in numa_cpu_lookup_table, so later, if the cpu is added, we could set correct numa information for it.
if the cpu is present, then we set the first online node to numa_cpu_lookup_table instead of 0 ( in case 0 might not be an online node? )
--- arch/powerpc/mm/numa.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)
arch/powerpc/mm/numa.c +8 -6
@@ -538,7 +538,7 @@ static int of_drconf_to_nid_single(struct of_drconf_cell *drmem,*/ static int numa_setup_cpu (unsigned long lcpu) { -int nid; +int nid =-1 ;struct device_node *cpu;/*
@@ -555,19 +555,21 @@ static int numa_setup_cpu(unsigned long lcpu)if (!cpu) {WARN_ON (1 ); - nid =0 ; - goto out; +if (cpu_present (lcpu)) + goto out_present; +else + goto out; } nid =of_node_to_nid_single (cpu); +out_present:if (nid <0 || !node_online (nid)) nid = first_online_node; -out: -map_cpu_to_node (lcpu, nid); +map_cpu_to_node (lcpu, nid);of_node_put (cpu); - +out:return nid; }
Signature · 1 line
1.9.1
Re: [RFC PATCH v3 3/3] powerpc: some changes in numa_setup_cpu()
Quoted 8 lines · Li Zhong
this patches changes some error handling logics in
numa_setup_cpu(), when cpu node is not found, so:if the cpu is possible, but not present, -1 is kept in
numa_cpu_lookup_table, so later, if the cpu is added, we could set correct numa information for it.if the cpu is present, then we set the first online node to
numa_cpu_lookup_tableinstead of 0 ( in case 0 might not be an online node? )
Not currently possible (Node 0 is always online), but I'm working on changing that :)
Quoted 4 lines
Cc Nishanth Aravamudan <[email protected]>Cc Nathan Fontenot <[email protected]>Signed-off-by Li Zhong <[email protected]>
Quoted 47 lines
--- arch/powerpc/mm/numa.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)arch/powerpc/mm/numa.c +8
-6@@ -538,7 +538,7 @@ static int of_drconf_to_nid_single(struct of_drconf_cell *drmem,*/ static int numa_setup_cpu (unsigned long lcpu) { -int nid; +int nid =-1 ;struct device_node *cpu;/* @@ -555,19 +555,21 @@ static int numa_setup_cpu(unsigned long lcpu)if (!cpu) {WARN_ON (1 ); - nid =0 ; - goto out; +if (cpu_present (lcpu)) + goto out_present; +else + goto out; } nid =of_node_to_nid_single (cpu); +out_present:if (nid <0 || !node_online (nid)) nid = first_online_node; -out: -map_cpu_to_node (lcpu, nid); +map_cpu_to_node (lcpu, nid);of_node_put (cpu); - +out:return nid; }Signature · 1 line
1.9.1
Quoted 12 lines · Li Zhong
With commit
2fabf084b6ad("powerpc: reorder per-cpu NUMA information's initialization"), during boottime,cpu_numa_callback()is called earlier(before their online) for each cpu, andverify_cpu_node_mapping()usescpu_to_node()to check whether siblings are in the same node.It skips the checking for siblings that are not online yet. So the only check done here is for the bootcpu, which is online at that time. But the per-cpu
numa_nodecpu_to_node()uses hasn't been set up yet (which will be set up insmp_prepare_cpus()).So I saw something like following reported:
[ 0.000000] CPU thread siblings 1/2/3 and 0 don't belong to the same
Quoted 8 lines
node!
As we don't actually do the checking during this early stage, so maybe we could directly call
numa_setup_cpu()indo_init_bootmem().Cc Nishanth Aravamudan <[email protected]>Cc Nathan Fontenot <[email protected]>Signed-off-by Li Zhong <[email protected]>
Quoted 21 lines
--- arch/powerpc/mm/numa.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/powerpc/mm/numa.c +1
-2@@ -1128,8 +1128,7 @@ void __init do_init_bootmem(void)* early in boot, cf. smp_prepare_cpus(). */ for_each_possible_cpu (cpu) { -cpu_numa_callback (&ppc64_numa_nb,CPU_UP_PREPARE , - (void *)(unsigned long )cpu); +numa_setup_cpu ((unsigned long )cpu); } }Signature · 1 line
1.9.1
Ben & Michael,
What's the status of these patches?
Thanks,
Nish
Quoted 41 lines · Li Zhong
With commit
2fabf084b6ad("powerpc: reorder per-cpu NUMA information's initialization"), during boottime,cpu_numa_callback()is called earlier(before their online) for each cpu, andverify_cpu_node_mapping()usescpu_to_node()to check whether siblings are in the same node.It skips the checking for siblings that are not online yet. So the only check done here is for the bootcpu, which is online at that time. But the per-cpu
numa_nodecpu_to_node()uses hasn't been set up yet (which will be set up insmp_prepare_cpus()).So I saw something like following reported:
[ 0.000000] CPU thread siblings 1/2/3 and 0 don't belong to the samenode!
As we don't actually do the checking during this early stage, so maybe we could directly call
numa_setup_cpu()indo_init_bootmem().Cc Nishanth Aravamudan <[email protected]>Cc Nathan Fontenot <[email protected]>Signed-off-by Li Zhong <[email protected]>--- arch/powerpc/mm/numa.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/powerpc/mm/numa.c +1
-2@@ -1128,8 +1128,7 @@ void __init do_init_bootmem(void)* early in boot, cf. smp_prepare_cpus(). */ for_each_possible_cpu (cpu) { -cpu_numa_callback (&ppc64_numa_nb,CPU_UP_PREPARE , - (void *)(unsigned long )cpu); +numa_setup_cpu ((unsigned long )cpu); } }Signature · 1 line
1.9.1
Quoted 3 lines · Nishanth Aravamudan
Ben & Michael,
What's the status of these patches?
Waiting for somebody to review them ? :-)
Cheers,
Ben.
Quoted 45 lines
Thanks,
NishQuoted 41 lines · Li Zhong
With commit
2fabf084b6ad("powerpc: reorder per-cpu NUMA information's initialization"), during boottime,cpu_numa_callback()is called earlier(before their online) for each cpu, andverify_cpu_node_mapping()usescpu_to_node()to check whether siblings are in the same node.It skips the checking for siblings that are not online yet. So the only check done here is for the bootcpu, which is online at that time. But the per-cpu
numa_nodecpu_to_node()uses hasn't been set up yet (which will be set up insmp_prepare_cpus()).So I saw something like following reported:
[ 0.000000] CPU thread siblings 1/2/3 and 0 don't belong to the samenode!
As we don't actually do the checking during this early stage, so maybe we could directly call
numa_setup_cpu()indo_init_bootmem().Cc Nishanth Aravamudan <[email protected]>Cc Nathan Fontenot <[email protected]>Signed-off-by Li Zhong <[email protected]>--- arch/powerpc/mm/numa.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/powerpc/mm/numa.c +1
-2@@ -1128,8 +1128,7 @@ void __init do_init_bootmem(void)* early in boot, cf. smp_prepare_cpus(). */ for_each_possible_cpu (cpu) { -cpu_numa_callback (&ppc64_numa_nb,CPU_UP_PREPARE , - (void *)(unsigned long )cpu); +numa_setup_cpu ((unsigned long )cpu); } }Signature · 1 line
1.9.1
Quoted 6 lines · Benjamin Herrenschmidt
Quoted 3 lines · Nishanth Aravamudan
Ben & Michael,
What's the status of these patches?
Waiting for somebody to review them ? :-)
Heh, I acked them about a month ago, I can give a stronger Reviewed-by, if you prefer?
-Nish
Quoted 10 lines · Nishanth Aravamudan
Quoted 6 lines · Benjamin Herrenschmidt
Quoted 3 lines · Nishanth Aravamudan
Ben & Michael,
What's the status of these patches?
Waiting for somebody to review them ? :-)
Heh, I acked them about a month ago, I can give a stronger Reviewed-by, if you prefer?
Nah, nothing other than having me or mpe pick them up. I've been letting Michael handle this merge window, so maybe he should take it :-)
Cheers,
Ben.
Michael Ellerman
reviewerQuoted 3 lines · Nishanth Aravamudan
Ben & Michael,
What's the status of these patches?
Been in my next for a week :)
https://git.kernel.org/cgit/linux/kernel/git/mpe/linux.git/log/?h=next
cheers
Quoted 8 lines · Michael Ellerman
Quoted 3 lines · Nishanth Aravamudan
Ben & Michael,
What's the status of these patches?
Been in my next for a week :)
https://git.kernel.org/cgit/linux/kernel/git/mpe/linux.git/log/?h=next
Ah ok, thanks -- I wasn't following your tree, my fault. Do we want these to go back to 3.17-stable, as they fix some annoying splats during boot (non-fatal afaict, though)?
-Nish
Michael Ellerman
reviewerQuoted 11 lines · Nishanth Aravamudan
Quoted 8 lines · Michael Ellerman
Quoted 3 lines · Nishanth Aravamudan
Ben & Michael,
What's the status of these patches?
Been in my next for a week :)
https://git.kernel.org/cgit/linux/kernel/git/mpe/linux.git/log/?h=next
Ah ok, thanks -- I wasn't following your tree, my fault.
Not really your fault, I hadn't announced my trees existence :)
Quoted 2 lines
Do we want these to go back to 3.17-stable, as they fix some annoying splats during boot (non-fatal afaict, though)?
Up to you really, I don't know how often/bad they were. I haven't added CC stable tags to the commits, so if you want them in stable you should send them explicitly.
cheers
Quoted 13 lines · Michael Ellerman
Quoted 11 lines · Nishanth Aravamudan
Quoted 8 lines · Michael Ellerman
Quoted 3 lines · Nishanth Aravamudan
Ben & Michael,
What's the status of these patches?
Been in my next for a week :)
https://git.kernel.org/cgit/linux/kernel/git/mpe/linux.git/log/?h=next
Ah ok, thanks -- I wasn't following your tree, my fault.
Quoted 8 lines
Not really your fault, I hadn't announced my trees existence :)
Quoted 2 lines
Do we want these to go back to 3.17-stable, as they fix some annoying splats during boot (non-fatal afaict, though)?
Up to you really, I don't know how often/bad they were. I haven't added CC stable tags to the commits, so if you want them in stable you should send them explicitly.
I think they occur every boot, unconditionally, on pseries. Doesn't prevent boot, just really noisy. I think it'd be good to get them into -stable.
Li Zhong, can you push them once they get sent upstream?
Thanks,
Nish
Quoted 23 lines · Nishanth Aravamudan
Quoted 13 lines · Michael Ellerman
Quoted 11 lines · Nishanth Aravamudan
Quoted 8 lines · Michael Ellerman
Quoted 3 lines · Nishanth Aravamudan
Ben & Michael,
What's the status of these patches?
Been in my next for a week :)
https://git.kernel.org/cgit/linux/kernel/git/mpe/linux.git/log/?h=next
Ah ok, thanks -- I wasn't following your tree, my fault.
Quoted 8 lines
Not really your fault, I hadn't announced my trees existence :)
Quoted 2 lines
Do we want these to go back to 3.17-stable, as they fix some annoying splats during boot (non-fatal afaict, though)?
Up to you really, I don't know how often/bad they were. I haven't added CC stable tags to the commits, so if you want them in stable you should send them explicitly.
Quoted 5 lines
I think they occur every boot, unconditionally, on pseries. Doesn't prevent boot, just really noisy. I think it'd be good to get them into -stable.
Li Zhong, can you push them once they get sent upstream?
Ok, I will send these patches to stable mailing list after it is merged.
Thanks, Zhong
Quoted 3 lines
Thanks,
Nish
Quoted 23 lines · Nishanth Aravamudan
Quoted 13 lines · Michael Ellerman
Quoted 11 lines · Nishanth Aravamudan
Quoted 8 lines · Michael Ellerman
Quoted 3 lines · Nishanth Aravamudan
Ben & Michael,
What's the status of these patches?
Been in my next for a week :)
https://git.kernel.org/cgit/linux/kernel/git/mpe/linux.git/log/?h=next
Ah ok, thanks -- I wasn't following your tree, my fault.
Quoted 8 lines
Not really your fault, I hadn't announced my trees existence :)
Quoted 2 lines
Do we want these to go back to 3.17-stable, as they fix some annoying splats during boot (non-fatal afaict, though)?
Up to you really, I don't know how often/bad they were. I haven't added CC stable tags to the commits, so if you want them in stable you should send them explicitly.
Quoted 5 lines
I think they occur every boot, unconditionally, on pseries. Doesn't prevent boot, just really noisy. I think it'd be good to get them into -stable.
Li Zhong, can you push them once they get sent upstream?
I guess I only need to send the first two patches to stable?
Thanks, Zhong
Quoted 3 lines
Thanks,
Nish
Michael Ellerman
reviewerQuoted 24 lines · Li Zhong
Quoted 23 lines · Nishanth Aravamudan
Quoted 21 lines · Michael Ellerman
Quoted 11 lines · Nishanth Aravamudan
Quoted 8 lines · Michael Ellerman
Quoted 3 lines · Nishanth Aravamudan
Ben & Michael,
What's the status of these patches?
Been in my next for a week :)
https://git.kernel.org/cgit/linux/kernel/git/mpe/linux.git/log/?h=next
Ah ok, thanks -- I wasn't following your tree, my fault.
Not really your fault, I hadn't announced my trees existence :)
Quoted 2 lines
Do we want these to go back to 3.17-stable, as they fix some annoying splats during boot (non-fatal afaict, though)?
Up to you really, I don't know how often/bad they were. I haven't added CC stable tags to the commits, so if you want them in stable you should send them explicitly.
Quoted 7 lines
Quoted 5 lines
I think they occur every boot, unconditionally, on pseries. Doesn't prevent boot, just really noisy. I think it'd be good to get them into -stable.
Li Zhong, can you push them once they get sent upstream?
I guess I only need to send the first two patches to stable?
Probably. It's not clear from the changelog how serious a problem it fixes.
See Documentation/stable_kernel_rules.txt
cheers
Quoted 29 lines · Michael Ellerman
Quoted 28 lines · Li Zhong
Quoted 27 lines · Nishanth Aravamudan
Quoted 13 lines · Michael Ellerman
Quoted 11 lines · Nishanth Aravamudan
Quoted 8 lines · Michael Ellerman
Quoted 3 lines · Nishanth Aravamudan
Ben & Michael,
What's the status of these patches?
Been in my next for a week :)
https://git.kernel.org/cgit/linux/kernel/git/mpe/linux.git/log/?h=next
Ah ok, thanks -- I wasn't following your tree, my fault.
Quoted 8 lines
Not really your fault, I hadn't announced my trees existence :)
Quoted 2 lines
Do we want these to go back to 3.17-stable, as they fix some annoying splats during boot (non-fatal afaict, though)?
Up to you really, I don't know how often/bad they were. I haven't added CC stable tags to the commits, so if you want them in stable you should send them explicitly.
I think they occur every boot, unconditionally, on pseries. Doesn't prevent boot, just really noisy. I think it'd be good to get them into -stable.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Li Zhong, can you push them once they get sent upstream?
I guess I only need to send the first two patches to stable?
Probably. It's not clear from the changelog how serious a problem it fixes.
See
Documentation/stable_kernel_rules.txt
OK, I'll send the first two to stable.
Thanks, Zhong
Quoted 5 lines
cheers