Skip to contentSkip to conversation

riscv: smp: set CPU 0 possible in setup_smp()

PATCH In review Paul Sherman 3 participants 6 messages Jul 14 – Jul 15 RISC-V

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 riscv: smp: set CPU 0 possible in setup_smp()

setup_smp() calls set_cpu_possible() for CPUs 1..nr_cpu_ids-1 but never for CPU 0 (the boot CPU). x86 handles this via init_cpu_possible(cpumask_of(0)); RISC-V has no equivalent.

Without CPU 0 in cpu_possible_mask, rcu_init_one()'s for_each_possible_cpu() loop skips it, leaving rdp->mynode=NULL. rcutree_prepare_cpu() then dereferences NULL and hangs.

Exposed on Sophgo SG2042 (64-hart, 4-NUMA) with Linux 7.2-rc3.

Fixes a4166aec1130 ("riscv: Deduplicate code in setup_smp()")
---
 arch/riscv/kernel/smpboot.c | 9 +++++++++
 1 file changed, 9 insertions(+)
arch/riscv/kernel/smpboot.c +9 -0
index f6ef57930b50..8d3a437c0f30 100644
@@ -171,6 +171,15 @@ void __init setup_smp(void)
 	for (cpuid = 1; cpuid < nr_cpu_ids; cpuid++)
 		if (cpuid_to_hartid_map(cpuid) != INVALID_HARTID)
 			set_cpu_possible(cpuid, true);
+
+	/*
+	 * Boot CPU (cpuid=0) is never set possible by the loop above.
+	 * x86 and arm64 call init_cpu_possible(cpumask_of(0)) explicitly;
+	 * RISC-V was missing this. Without it, rcu_init_one()'s
+	 * for_each_possible_cpu loop skips CPU 0, leaving rdp->mynode=NULL,
+	 * causing rcutree_prepare_cpu() to dereference NULL and hang.
+	 */
+	set_cpu_possible(0, true);
 }
 
 static int start_secondary_cpu(int cpu, struct task_struct *tidle)
Signature · 7 lines
2.53.0


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
maintainer
#1 lore raw
Quoted 4 lines · Paul Sherman

setup_smp() calls set_cpu_possible() for CPUs 1..nr_cpu_ids-1 but never for CPU 0 (the boot CPU). x86 handles this via init_cpu_possible(cpumask_of(0)); RISC-V has no equivalent.

Quoted 8 lines

Without CPU 0 in cpu_possible_mask, rcu_init_one()'s for_each_possible_cpu() loop skips it, leaving rdp->mynode=NULL. rcutree_prepare_cpu() then dereferences NULL and hangs.

Exposed on Sophgo SG2042 (64-hart, 4-NUMA) with Linux 7.2-rc3.

Fixes a4166aec1130 ("riscv: Deduplicate code in setup_smp()")

Thanks, queued for v7.2-rc.

- Paul

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#2 lore raw
Quoted 10 lines · Paul Walmsley
Quoted 8 lines · Paul Sherman

setup_smp() calls set_cpu_possible() for CPUs 1..nr_cpu_ids-1 but never for CPU 0 (the boot CPU). x86 handles this via init_cpu_possible(cpumask_of(0)); RISC-V has no equivalent.

Without CPU 0 in cpu_possible_mask, rcu_init_one()'s for_each_possible_cpu() loop skips it, leaving rdp->mynode=NULL. rcutree_prepare_cpu() then dereferences NULL and hangs.

Quoted 5 lines
Quoted 4 lines

Exposed on Sophgo SG2042 (64-hart, 4-NUMA) with Linux 7.2-rc3.

Fixes a4166aec1130 ("riscv: Deduplicate code in setup_smp()")

Thanks, queued for v7.2-rc.

I hope it's not too late to object, but I don't see how this could happen. This is what happens in start_kernel():

    assembly boot code
    -> start_kernel()
       -> boot_cpu_init();
          -> int cpu = smp_processor_id();
             /* cpu = 0 because INIT_THREAD_INFO(tsk) leaves it out */
          
             /* Mark the boot cpu "present", "online" etc for SMP and UP case */
             set_cpu_online(cpu, true);
             set_cpu_active(cpu, true);
             set_cpu_present(cpu, true);
             set_cpu_possible(cpu, true); /* <---- HERE */
       -> ...
       -> setup_arch(&command_line);
          -> setup_smp()
             -> for (cpuid = 1; cpuid < nr_cpu_ids; cpuid++)
                	if (cpuid_to_hartid_map(cpuid) != INVALID_HARTID)
                		set_cpu_possible(cpuid, true);

So CPU 0 is already in cpu_possible_mask. This is also what I see by adding prints to setup_smp().

* Paul Sherman: How did you reproduce this? Is your boot CPU not cpu 0 (note: not the same as hart 0)?

Vivian "dramforever" Wang

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#3 lore raw

Vivian, Paul,

Thank you for the precise analysis. You are correct for mainline, boot_cpu_init() calls set_cpu_possible(0, true) before setup_smp().

The hang was observed on a development tree based on Linux 7.0-rc2 with substantial percpu/SLUB allocator changes layered on top for large NUMA geometry (4 nodes, 128GB). In that context I instrumented rcu_init_one() and rcutree_prepare_cpu() via SBI ecall probes (no JTAG/denugger available on this hardware) and observed:

Za ffffffff81035000 <- per_cpu_ptr(&rcu_data, 0) in rcu_init_one
Zn 0000000000000000 <- mynode was NULL at write time

rdp ffffffff81035000 <- same address at rcutree_prepare_cpu
mynode = NULL <- confirmed NULL dereference

The for_each_possible_cpu loop in rcu_init_one was skipping CPU 0 because cpu_possible_mask had not yet been populated at that point in my tree's initialization order. The additional allocator changes may have altered the timing.

I am rebasing onto 7.2-rc3. If the condition cannot be reproduced there, please feel free to drop the patch.

The probe methodology above uses M-mode SBI ecalls as a poor man's logic analyzer; happy to share the infrastructure if useful.

Paul Sherman

Quoted 40 lines · Vivian Wang
Quoted 10 lines · Paul Walmsley
Quoted 8 lines · Paul Sherman

setup_smp() calls set_cpu_possible() for CPUs 1..nr_cpu_ids-1 but never for CPU 0 (the boot CPU). x86 handles this via init_cpu_possible(cpumask_of(0)); RISC-V has no equivalent.

Without CPU 0 in cpu_possible_mask, rcu_init_one()'s for_each_possible_cpu() loop skips it, leaving rdp->mynode=NULL. rcutree_prepare_cpu() then dereferences NULL and hangs.

Quoted 5 lines
Quoted 4 lines

Exposed on Sophgo SG2042 (64-hart, 4-NUMA) with Linux 7.2-rc3.

Fixes a4166aec1130 ("riscv: Deduplicate code in setup_smp()")

Thanks, queued for v7.2-rc.

I hope it's not too late to object, but I don't see how this could happen. This is what happens in start_kernel():

    assembly boot code
    -> start_kernel()
       -> boot_cpu_init();
          -> int cpu = smp_processor_id();
             /* cpu = 0 because INIT_THREAD_INFO(tsk) leaves it out */

             /* Mark the boot cpu "present", "online" etc for SMP and UP case */
             set_cpu_online(cpu, true);
             set_cpu_active(cpu, true);
             set_cpu_present(cpu, true);
             set_cpu_possible(cpu, true); /* <---- HERE */
       -> ...
       -> setup_arch(&command_line);
          -> setup_smp()
             -> for (cpuid = 1; cpuid < nr_cpu_ids; cpuid++)
                        if (cpuid_to_hartid_map(cpuid) != INVALID_HARTID)
                                set_cpu_possible(cpuid, true);
Quoted 8 lines

So CPU 0 is already in cpu_possible_mask. This is also what I see by adding prints to setup_smp().

* Paul Sherman: How did you reproduce this? Is your boot CPU not cpu 0 (note: not the same as hart 0)?

Vivian "dramforever" Wang

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#4 lore raw

On 7/15/26 12:51, Paul Sherman wrote:

[ Your reply was here - I moved it to the bottom. Please do not top-post. See section "Use trimmed interleaved replies in email discussions" in Documentation/process/submitting-patches.rst for how email replies are expected to be formatted on LKML. ]

Quoted 45 lines · Vivian Wang
Quoted 44 lines · Paul Walmsley
Quoted 11 lines · Paul Sherman
Quoted 9 lines

setup_smp() calls set_cpu_possible() for CPUs 1..nr_cpu_ids-1 but never for CPU 0 (the boot CPU). x86 handles this via init_cpu_possible(cpumask_of(0)); RISC-V has no equivalent.

Without CPU 0 in cpu_possible_mask, rcu_init_one()'s for_each_possible_cpu() loop skips it, leaving rdp->mynode=NULL. rcutree_prepare_cpu() then dereferences NULL and hangs.

Quoted 5 lines
Quoted 4 lines

Exposed on Sophgo SG2042 (64-hart, 4-NUMA) with Linux 7.2-rc3.

Fixes a4166aec1130 ("riscv: Deduplicate code in setup_smp()")

Thanks, queued for v7.2-rc.

I hope it's not too late to object, but I don't see how this could happen. This is what happens in start_kernel():

    assembly boot code
    -> start_kernel()
       -> boot_cpu_init();
          -> int cpu = smp_processor_id();
             /* cpu = 0 because INIT_THREAD_INFO(tsk) leaves it out */

             /* Mark the boot cpu "present", "online" etc for SMP and UP case */
             set_cpu_online(cpu, true);
             set_cpu_active(cpu, true);
             set_cpu_present(cpu, true);
             set_cpu_possible(cpu, true); /* <---- HERE */
       -> ...
       -> setup_arch(&command_line);
          -> setup_smp()
             -> for (cpuid = 1; cpuid < nr_cpu_ids; cpuid++)
                        if (cpuid_to_hartid_map(cpuid) != INVALID_HARTID)
                                set_cpu_possible(cpuid, true);

So CPU 0 is already in cpu_possible_mask. This is also what I see by adding prints to setup_smp().

* Paul Sherman: How did you reproduce this? Is your boot CPU not cpu 0 (note: not the same as hart 0)?

Quoted 6 lines
Quoted 2 lines

Vivian "dramforever" Wang

Vivian, Paul,

Thank you for the precise analysis. You are correct for mainline, boot_cpu_init() calls set_cpu_possible(0, true) before setup_smp().

Okay, if it's not a bug in mainline, it's not a bug in mainline.

Quoted 10 lines

The hang was observed on a development tree based on Linux 7.0-rc2 with substantial percpu/SLUB allocator changes layered on top for large NUMA geometry (4 nodes, 128GB).

[...]

The for_each_possible_cpu loop in rcu_init_one was skipping CPU 0 because cpu_possible_mask had not yet been populated at that point in my tree's initialization order. The additional allocator changes may have altered the timing.

I don't know what you're tree is doing, obviously, but I want to note that boot_cpu_init() comes *before* setup_smp() in start_kernel(), and indeed both come *extremely* early. So maybe all you need to do is, when moving stuff around, to also keep boot_cpu_init() before setup_smp(), which comes (way) before rcu_init()?

Quoted 2 lines

I am rebasing onto 7.2-rc3. If the condition cannot be reproduced there, please feel free to drop the patch.

Given that neither visual analysis (see above) nor runtime testing (we have SG2042 machines in ISCAS running 7.0.x and 7.1.x without running into this) can confirm that this bug exists, I'm concluding that, unless further convincing evidence otherwise comes up, this is a bug you introduced in your tree and does not fixing for mainline.

Quoted 2 lines

The probe methodology above uses M-mode SBI ecalls as a poor man's logic analyzer; happy to share the infrastructure if useful.

If you can reproduce this on 7.2-rc3 *without* your stuff on top, that would be useful. Otherwise, nobody else knows what the bug even is...

Vivian "dramforever" Wang

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

maintainer
#5 lore raw
Quoted 5 lines · Vivian Wang

Given that neither visual analysis (see above) nor runtime testing (we have SG2042 machines in ISCAS running 7.0.x and 7.1.x without running into this) can confirm that this bug exists, I'm concluding that, unless further convincing evidence otherwise comes up, this is a bug you introduced in your tree and does not fixing for mainline.

Thanks for taking a closer look, Vivian; I'll drop this patch for now until either Paul or someone else can reproduce this on mainline.

- Paul

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv