Skip to contentSkip to conversation

x86: make reboot task only run on the appropriate processor

PATCH In review Baoquan He 5 participants 16 messages Nov 5, 2013 – Nov 11, 2013

Subsystems (5)
lore
Share & tools
lore

Baoquan He

author
#0 lore raw

PATCH 1/1 x86: make reboot task only run on the appropriate processor

Currently system always reboot after below message when execute "kexec -e".

[    0.572119] smpboot: Booting Node   0, Processors  #   1 OK

In commit 1b3a5d02ee070c8f9943333b9b6370f486601e0f, reboot= handling was moved to kerne/reboot.c. However, the code to migrate current thread to reboot cpu was removed. That cause this incorrect kexec behavior.

Now add that code block back.

---
 arch/x86/kernel/reboot.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
arch/x86/kernel/reboot.c +15 -0
index 7e920bf..3049de9 100644
@@ -551,6 +551,21 @@ void native_machine_shutdown(void)
 {
 	/* Stop the cpus and apics */
 #ifdef CONFIG_SMP
+       /* The boot cpu is always logical cpu 0 */
+       int reboot_cpu_id = 0;
+
+       /* See if there has been given a command line override */
+       if ((reboot_cpu != -1) && (reboot_cpu < nr_cpu_ids) &&
+               cpu_online(reboot_cpu))
+               reboot_cpu_id = reboot_cpu;
+
+       /* Make certain the cpu I'm about to reboot on is online */
+       if (!cpu_online(reboot_cpu_id))
+               reboot_cpu_id = smp_processor_id();
+
+       /* Make certain I only run on the appropriate processor */
+       set_cpus_allowed_ptr(current, cpumask_of(reboot_cpu_id));
+
 	/*
 	 * Stop all of the others. Also disable the local irq to
 	 * not receive the per-cpu timer interrupt which may trigger
Signature · 7 lines
1.8.3.1


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#1 lore raw
Quoted 4 lines · Baoquan He

Currently system always reboot after below message when execute "kexec -e".

[    0.572119] smpboot: Booting Node   0, Processors  #   1 OK

So is it same problem were we reboot on non-boot cpu and sending INIT to boot cpu in second kernel brings down the machine?

I think for x86, it makes sense to reboot on boot cpu.

Thanks
Vivek

Quoted 42 lines

In commit 1b3a5d02ee070c8f9943333b9b6370f486601e0f, reboot= handling was moved to kerne/reboot.c. However, the code to migrate current thread to reboot cpu was removed. That cause this incorrect kexec behavior.

Now add that code block back.

---
 arch/x86/kernel/reboot.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
arch/x86/kernel/reboot.c +15 -0
index 7e920bf..3049de9 100644
@@ -551,6 +551,21 @@ void native_machine_shutdown(void)
 {
 	/* Stop the cpus and apics */
 #ifdef CONFIG_SMP
+       /* The boot cpu is always logical cpu 0 */
+       int reboot_cpu_id = 0;
+
+       /* See if there has been given a command line override */
+       if ((reboot_cpu != -1) && (reboot_cpu < nr_cpu_ids) &&
+               cpu_online(reboot_cpu))
+               reboot_cpu_id = reboot_cpu;
+
+       /* Make certain the cpu I'm about to reboot on is online */
+       if (!cpu_online(reboot_cpu_id))
+               reboot_cpu_id = smp_processor_id();
+
+       /* Make certain I only run on the appropriate processor */
+       set_cpus_allowed_ptr(current, cpumask_of(reboot_cpu_id));
+
 	/*
 	 * Stop all of the others. Also disable the local irq to
 	 * not receive the per-cpu timer interrupt which may trigger
Signature · 1 line
1.8.3.1

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#2 lore raw

We should, yes, if cpu 0 is available.

Quoted 59 lines · Vivek Goyal
Quoted 1 line · Baoquan He

Currently system always reboot after below message when execute

"kexec -e".

Quoted 3 lines
[    0.572119] smpboot: Booting Node   0, Processors  #   1 OK

So is it same problem were we reboot on non-boot cpu and sending INIT to boot cpu in second kernel brings down the machine?

I think for x86, it makes sense to reboot on boot cpu.

Thanks
Vivek

Quoted 1 line

In commit 1b3a5d02ee070c8f9943333b9b6370f486601e0f, reboot= handling

was

Quoted 1 line

moved to kerne/reboot.c. However, the code to migrate current thread

to

Quoted 40 lines

reboot cpu was removed. That cause this incorrect kexec behavior.

Now add that code block back.

---
 arch/x86/kernel/reboot.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
arch/x86/kernel/reboot.c +15 -0
index 7e920bf..3049de9 100644
@@ -551,6 +551,21 @@ void native_machine_shutdown(void)
 {
 	/* Stop the cpus and apics */
 #ifdef CONFIG_SMP
+       /* The boot cpu is always logical cpu 0 */
+       int reboot_cpu_id = 0;
+
+       /* See if there has been given a command line override */
+       if ((reboot_cpu != -1) && (reboot_cpu < nr_cpu_ids) &&
+               cpu_online(reboot_cpu))
+               reboot_cpu_id = reboot_cpu;
+
+       /* Make certain the cpu I'm about to reboot on is online */
+       if (!cpu_online(reboot_cpu_id))
+               reboot_cpu_id = smp_processor_id();
+
+       /* Make certain I only run on the appropriate processor */
+       set_cpus_allowed_ptr(current, cpumask_of(reboot_cpu_id));
+
 	/*
 	 * Stop all of the others. Also disable the local irq to
 	 * not receive the per-cpu timer interrupt which may trigger
Signature · 1 line
1.8.3.1
Signature · 6 lines
Sent from my mobile phone.  Please pardon brevity and lack of formatting.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
author
#3 lore raw
In reply to Vivek Goyal
Quoted 8 lines · Vivek Goyal
Quoted 4 lines · Baoquan He

Currently system always reboot after below message when execute "kexec -e".

[    0.572119] smpboot: Booting Node   0, Processors  #   1 OK

So is it same problem were we reboot on non-boot cpu and sending INIT to boot cpu in second kernel brings down the machine?

They are different. Kdump hang when crash didn't happen on bsp, then multiple CPUs are brought up, the bsp will go through the boot strap code after INIT IPI message received. However in this case, executing "kexec -e" will make system reboot and bring up only 1 CPU, then go back to BIOS and reboot.

In kdump, nmi is sent to shutdown other CPUs. In this shutdown, cleanup is done and finally processor is halt. And if crash didn't happen on bsp, during system startup INIT sent to bsp will cause system hang.

However, in kexec reboot, a REBOOT_VECTOR interrupt is sent to all other CPUs. In commit 4ef702c10b5df18ab04921fc252c26421d4d6c75, Andi imported this IPI interrupt vector. Andi said it has been given the highest priority.

It's still not very clear why this happened without that code block. Seems interrupt disable is very doubtful.

Quoted 11 lines

I think for x86, it makes sense to reboot on boot cpu.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#4 lore raw
In reply to Vivek Goyal
Quoted 8 lines · Vivek Goyal
Quoted 4 lines · Baoquan He

Currently system always reboot after below message when execute "kexec -e".

[    0.572119] smpboot: Booting Node   0, Processors  #   1 OK

So is it same problem were we reboot on non-boot cpu and sending INIT to boot cpu in second kernel brings down the machine?

I was wrong, they are the same problem. With multiple CPUs, by setting affinity to execute crash or kexec on CPUn(n!==0), they all reboot after the same message as below and then reboot through BIOS. It should be the BSP got a INIT IPI message and make system hang.

[    0.572119] smpboot: Booting Node   0, Processors  #   1 OK
Quoted 45 lines

I think for x86, it makes sense to reboot on boot cpu.

Thanks
Vivek

Quoted 39 lines

In commit 1b3a5d02ee070c8f9943333b9b6370f486601e0f, reboot= handling was moved to kerne/reboot.c. However, the code to migrate current thread to reboot cpu was removed. That cause this incorrect kexec behavior.

Now add that code block back.

---
 arch/x86/kernel/reboot.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
arch/x86/kernel/reboot.c +15 -0
index 7e920bf..3049de9 100644
@@ -551,6 +551,21 @@ void native_machine_shutdown(void)
 {
 	/* Stop the cpus and apics */
 #ifdef CONFIG_SMP
+       /* The boot cpu is always logical cpu 0 */
+       int reboot_cpu_id = 0;
+
+       /* See if there has been given a command line override */
+       if ((reboot_cpu != -1) && (reboot_cpu < nr_cpu_ids) &&
+               cpu_online(reboot_cpu))
+               reboot_cpu_id = reboot_cpu;
+
+       /* Make certain the cpu I'm about to reboot on is online */
+       if (!cpu_online(reboot_cpu_id))
+               reboot_cpu_id = smp_processor_id();
+
+       /* Make certain I only run on the appropriate processor */
+       set_cpus_allowed_ptr(current, cpumask_of(reboot_cpu_id));
+
 	/*
 	 * Stop all of the others. Also disable the local irq to
Quoted 8 lines
Quoted 3 lines
arch/x86/kernel/reboot.c +0 -0
 	 * not receive the per-cpu timer interrupt which may trigger
Signature · 1 line
1.8.3.1

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#5 lore raw
In reply to Baoquan He
Quoted 9 lines · Baoquan He

Currently system always reboot after below message when execute "kexec -e".

[    0.572119] smpboot: Booting Node   0, Processors  #   1 OK

In commit 1b3a5d02ee070c8f9943333b9b6370f486601e0f, reboot= handling was moved to kerne/reboot.c. However, the code to migrate current thread to reboot cpu was removed. That cause this incorrect kexec behavior.

Now add that code block back.

quote a mail from Thomas Renninger <[email protected]>:
Answer his questions here.

Quoted 7 lines
Quoted 4 lines

For the smp boot issue I think there's nothing to do with dracut.

Can you try below patch?
https://lkml.org/lkml/2013/11/5/88

That seem to have helped, thanks!

Quoted 7 lines

Feel free to add:

The problem existed in 3.11 already?
I had the problem with a 3.11 kernel and tried a
3.12 kernel with this patch now.

It's a regression between 3.10 ~ 3.11

Quoted 5 lines

It would be great if:
Cc: [email protected]
is added as well then.

I also like it to be in stable.

Thanks
Dave

#6 lore raw
In reply to Baoquan He
Quoted 8 lines · Baoquan He

Currently system always reboot after below message when execute "kexec -e".

[    0.572119] smpboot: Booting Node   0, Processors  #   1 OK

In commit 1b3a5d02ee070c8f9943333b9b6370f486601e0f, reboot= handling was moved to kerne/reboot.c. However, the code to migrate current thread to reboot cpu was removed. That cause this incorrect kexec behavior.

Quoted 6 lines

Now add that code block back.

Hi Bao,

This patch fixes the issue for me too. I noticed that we have generic function migrate_to_reboot_cpu() to achieve what we want and rest of the reboot paths are using it. So how about using that function. I wrote the new patch below. It works for me. Can you please give it a try.

Thanks
Vivek

kexec: migrate to reboot cpu

Commit 1b3a5d02ee070c8f9943333b9b6370f486601e0f moved reboot= handling to generic code. In the process it also removed the code in native_machine_shutdown() which are moving reboot process to reboot_cpu/cpu0.

I guess that thought must have been that all reboot paths are calling migrate_to_reboot_cpu(), so we don't need this special handling. But kexec reboot path (kernel_kexec()) is not calling migrate_to_reboot_cpu() so above change broke kexec. Now reboot can happen on non-boot cpu and when INIT is sent in second kerneo to bring up BP, it brings down the machine.

So start calling migrate_to_reboot_cpu() in kexec reboot path to avoid this problem.

---
 include/linux/reboot.h |    1 +
 kernel/kexec.c         |    1 +
 kernel/reboot.c        |    2 +-
 3 files changed, 3 insertions(+), 1 deletion(-)
kernel/reboot.c +1 -1
Index: linux-2.6/kernel/reboot.c
===================================================================
@@ -104,7 +104,7 @@ int unregister_reboot_notifier(struct no
 }
 EXPORT_SYMBOL(unregister_reboot_notifier);
 
-static void migrate_to_reboot_cpu(void)
+void migrate_to_reboot_cpu(void)
 {
 	/* The boot cpu is always logical cpu 0 */
 	int cpu = reboot_cpu;
include/linux/reboot.h +1 -0
Index: linux-2.6/include/linux/reboot.h
===================================================================
@@ -43,6 +43,7 @@ extern int unregister_reboot_notifier(st
  * Architecture-specific implementations of sys_reboot commands.
  */
 
+extern void migrate_to_reboot_cpu(void);
 extern void machine_restart(char *cmd);
 extern void machine_halt(void);
 extern void machine_power_off(void);
kernel/kexec.c +1 -0
Index: linux-2.6/kernel/kexec.c
===================================================================
@@ -1676,6 +1676,7 @@ int kernel_kexec(void)
 #endif
 	{
 		kernel_restart_prepare(NULL);
+		migrate_to_reboot_cpu();
 		printk(KERN_EMERG "Starting new kernel\n");
 		machine_shutdown();
 	}

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#7 lore raw
Quoted 12 lines · Vivek Goyal

Hi Bao,

This patch fixes the issue for me too. I noticed that we have generic function migrate_to_reboot_cpu() to achieve what we want and rest of the reboot paths are using it. So how about using that function. I wrote the new patch below. It works for me. Can you please give it a try.

Thanks
Vivek

Is this path exercised for the kdump flow? migrate_to_reboot_cpu() is unsafe in that case.

kernel/kexec.c int kernel_kexec(void) · line 1682

Quoted 13 lines

Index: linux-2.6/kernel/kexec.c
===================================================================

kernel/kexec.c +1 -0
@@ -1676,6 +1676,7 @@ int kernel_kexec(void)
 #endif
 	{
 		kernel_restart_prepare(NULL);
+		migrate_to_reboot_cpu();
 		printk(KERN_EMERG "Starting new kernel\n");
 		machine_shutdown();
 	}

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#8 lore raw
Quoted 10 lines · H. Peter Anvin
Quoted 9 lines · Vivek Goyal

Hi Bao,

This patch fixes the issue for me too. I noticed that we have generic function migrate_to_reboot_cpu() to achieve what we want and rest of the reboot paths are using it. So how about using that function. I wrote the new patch below. It works for me. Can you please give it a try.

Quoted 6 lines
Quoted 3 lines

Thanks
Vivek

Is this path exercised for the kdump flow? migrate_to_reboot_cpu() is unsafe in that case.

kdump path should not be affected by this change as it uses crash_kexec() instead of kerenl_kexec() for its entry. And crash_kexec() path does not call migrate_to_reboot_cpu().

Thanks
Vivek

Quoted 14 lines
Quoted 7 lines

Index: linux-2.6/kernel/kexec.c
===================================================================

kernel/kexec.c int kernel_kexec(void) · line 1682

Quoted 7 lines
kernel/kexec.c +1 -0
@@ -1676,6 +1676,7 @@ int kernel_kexec(void)
 #endif
 	{
 		kernel_restart_prepare(NULL);
+		migrate_to_reboot_cpu();
 		printk(KERN_EMERG "Starting new kernel\n");
 		machine_shutdown();
 	}

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#9 lore raw
In reply to H. Peter Anvin
Quoted 10 lines · H. Peter Anvin
Quoted 9 lines · Vivek Goyal

Hi Bao,

This patch fixes the issue for me too. I noticed that we have generic function migrate_to_reboot_cpu() to achieve what we want and rest of the reboot paths are using it. So how about using that function. I wrote the new patch below. It works for me. Can you please give it a try.

Quoted 6 lines
Quoted 3 lines

Thanks
Vivek

Is this path exercised for the kdump flow? migrate_to_reboot_cpu() is unsafe in that case.

Hi Peter,

Can you please consider queuing up this patch for next release.

Thanks
Vivek

Quoted 14 lines
Quoted 7 lines

Index: linux-2.6/kernel/kexec.c
===================================================================

kernel/kexec.c int kernel_kexec(void) · line 1682

Quoted 7 lines
kernel/kexec.c +1 -0
@@ -1676,6 +1676,7 @@ int kernel_kexec(void)
 #endif
 	{
 		kernel_restart_prepare(NULL);
+		migrate_to_reboot_cpu();
 		printk(KERN_EMERG "Starting new kernel\n");
 		machine_shutdown();
 	}

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#10 lore raw

Yes, unless there is a better path for it since it is not x86-specific. I am fine taking it, though.

Quoted 43 lines · Vivek Goyal
Quoted 5 lines · H. Peter Anvin
Quoted 4 lines · Vivek Goyal

Hi Bao,

This patch fixes the issue for me too. I noticed that we have

generic

Quoted 1 line
Quoted 1 line

function migrate_to_reboot_cpu() to achieve what we want and rest

of

Quoted 9 lines
Quoted 7 lines

the reboot paths are using it. So how about using that function. I wrote the new patch below. It works for me. Can you please give it a try.

Thanks
Vivek

Is this path exercised for the kdump flow? migrate_to_reboot_cpu()

is

Quoted 1 line

unsafe in that case.

Hi Peter,

Can you please consider queuing up this patch for next release.

Thanks
Vivek

Quoted 4 lines
Quoted 4 lines

Index: linux-2.6/kernel/kexec.c
===================================================================
--- linux-2.6.orig/kernel/kexec.c 2013-10-16 00:30:50.000000000

-0400

Quoted 10 lines
Quoted 10 lines

+++ linux-2.6/kernel/kexec.c 2013-11-08 21:34:02.492072375 -0500

kernel/kexec.c +1 -0
@@ -1676,6 +1676,7 @@ int kernel_kexec(void)
 #endif
 	{
 		kernel_restart_prepare(NULL);
+		migrate_to_reboot_cpu();
 		printk(KERN_EMERG "Starting new kernel\n");
 		machine_shutdown();
 	}
Signature · 6 lines
Sent from my mobile phone.  Please pardon brevity and lack of formatting.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#11 lore raw
Quoted 1 line · H. Peter Anvin

Yes, unless there is a better path for it since it is not x86-specific. I am fine taking it, though.

Generally Andrew Morton routes kexec/kdump patches through his tree. Andrew, do you think it should go through your tree?

Thanks
Vivek

Quoted 35 lines
Quoted 33 lines · Vivek Goyal
Quoted 5 lines · H. Peter Anvin
Quoted 4 lines · Vivek Goyal

Hi Bao,

This patch fixes the issue for me too. I noticed that we have

generic

Quoted 1 line
Quoted 1 line

function migrate_to_reboot_cpu() to achieve what we want and rest

of

Quoted 9 lines
Quoted 7 lines

the reboot paths are using it. So how about using that function. I wrote the new patch below. It works for me. Can you please give it a try.

Thanks
Vivek

Is this path exercised for the kdump flow? migrate_to_reboot_cpu()

is

Quoted 1 line

unsafe in that case.

Hi Peter,

Can you please consider queuing up this patch for next release.

Thanks
Vivek

Quoted 4 lines
Quoted 4 lines

Index: linux-2.6/kernel/kexec.c
===================================================================
--- linux-2.6.orig/kernel/kexec.c 2013-10-16 00:30:50.000000000

-0400

Quoted 13 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

+++ linux-2.6/kernel/kexec.c 2013-11-08 21:34:02.492072375 -0500

kernel/kexec.c +1 -0
@@ -1676,6 +1676,7 @@ int kernel_kexec(void)
 #endif
 	{
 		kernel_restart_prepare(NULL);
+		migrate_to_reboot_cpu();
 		printk(KERN_EMERG "Starting new kernel\n");
 		machine_shutdown();
 	}
Signature · 1 line
Sent from my mobile phone.  Please pardon brevity and lack of formatting.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

reviewer Acked-by
#12 lore raw
Quoted 8 lines · Vivek Goyal
Quoted 1 line · H. Peter Anvin

Yes, unless there is a better path for it since it is not x86-specific. I am fine taking it, though.

Generally Andrew Morton routes kexec/kdump patches through his tree. Andrew, do you think it should go through your tree?

Thanks
Vivek

If so, you can have my:

-hpa

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#13 lore raw
In reply to Vivek Goyal
Quoted 10 lines · Vivek Goyal

On Tue, Nov 05, 2013 at 05:16:07PM +0800, Baoquan He wrote:

Hi Bao,

This patch fixes the issue for me too. I noticed that we have generic function migrate_to_reboot_cpu() to achieve what we want and rest of the reboot paths are using it. So how about using that function. I wrote the new patch below. It works for me. Can you please give it a try.

Seems there's something wrong with this function, that error happened when I executed the 2nd "kexec -e". That means executing "kexec -e" in kexec-ed kernel.

Tomorrow I will ask people around take more tests. We need set affinity to test this. E.g always execute :

taskset -c 1 -sh "kexec -e"

Quoted 65 lines

Thanks
Vivek

kexec: migrate to reboot cpu

Commit 1b3a5d02ee070c8f9943333b9b6370f486601e0f moved reboot= handling to generic code. In the process it also removed the code in native_machine_shutdown() which are moving reboot process to reboot_cpu/cpu0.

I guess that thought must have been that all reboot paths are calling migrate_to_reboot_cpu(), so we don't need this special handling. But kexec reboot path (kernel_kexec()) is not calling migrate_to_reboot_cpu() so above change broke kexec. Now reboot can happen on non-boot cpu and when INIT is sent in second kerneo to bring up BP, it brings down the machine.

So start calling migrate_to_reboot_cpu() in kexec reboot path to avoid this problem.

---
 include/linux/reboot.h |    1 +
 kernel/kexec.c         |    1 +
 kernel/reboot.c        |    2 +-
 3 files changed, 3 insertions(+), 1 deletion(-)

Index: linux-2.6/kernel/reboot.c
===================================================================

kernel/reboot.c +1 -1
@@ -104,7 +104,7 @@ int unregister_reboot_notifier(struct no
 }
 EXPORT_SYMBOL(unregister_reboot_notifier);
 
-static void migrate_to_reboot_cpu(void)
+void migrate_to_reboot_cpu(void)
 {
 	/* The boot cpu is always logical cpu 0 */
 	int cpu = reboot_cpu;

Index: linux-2.6/include/linux/reboot.h
===================================================================

include/linux/reboot.h +1 -0
@@ -43,6 +43,7 @@ extern int unregister_reboot_notifier(st
  * Architecture-specific implementations of sys_reboot commands.
  */
 
+extern void migrate_to_reboot_cpu(void);
 extern void machine_restart(char *cmd);
 extern void machine_halt(void);
 extern void machine_power_off(void);

Index: linux-2.6/kernel/kexec.c
===================================================================

kernel/kexec.c +1 -0
@@ -1676,6 +1676,7 @@ int kernel_kexec(void)
 #endif
 	{
 		kernel_restart_prepare(NULL);
+		migrate_to_reboot_cpu();
 		printk(KERN_EMERG "Starting new kernel\n");
 		machine_shutdown();
 	}
author
#14 lore raw
Quoted 11 lines · Baoquan He
Quoted 10 lines · Vivek Goyal

On Tue, Nov 05, 2013 at 05:16:07PM +0800, Baoquan He wrote:

Hi Bao,

This patch fixes the issue for me too. I noticed that we have generic function migrate_to_reboot_cpu() to achieve what we want and rest of the reboot paths are using it. So how about using that function. I wrote the new patch below. It works for me. Can you please give it a try.

Hi Vivek,

Your patch works, my test script has wrong typo.

Baoquan
Thanks a lot

Quoted 60 lines

Seems there's something wrong with this function, that error happened when I executed the 2nd "kexec -e". That means executing "kexec -e" in kexec-ed kernel.

Tomorrow I will ask people around take more tests. We need set affinity to test this. E.g always execute :

taskset -c 1 -sh "kexec -e"

Quoted 49 lines

Thanks
Vivek

kexec: migrate to reboot cpu

Commit 1b3a5d02ee070c8f9943333b9b6370f486601e0f moved reboot= handling to generic code. In the process it also removed the code in native_machine_shutdown() which are moving reboot process to reboot_cpu/cpu0.

I guess that thought must have been that all reboot paths are calling migrate_to_reboot_cpu(), so we don't need this special handling. But kexec reboot path (kernel_kexec()) is not calling migrate_to_reboot_cpu() so above change broke kexec. Now reboot can happen on non-boot cpu and when INIT is sent in second kerneo to bring up BP, it brings down the machine.

So start calling migrate_to_reboot_cpu() in kexec reboot path to avoid this problem.

---
 include/linux/reboot.h |    1 +
 kernel/kexec.c         |    1 +
 kernel/reboot.c        |    2 +-
 3 files changed, 3 insertions(+), 1 deletion(-)

Index: linux-2.6/kernel/reboot.c
===================================================================

kernel/reboot.c +1 -1
@@ -104,7 +104,7 @@ int unregister_reboot_notifier(struct no
 }
 EXPORT_SYMBOL(unregister_reboot_notifier);
 
-static void migrate_to_reboot_cpu(void)
+void migrate_to_reboot_cpu(void)
 {
 	/* The boot cpu is always logical cpu 0 */
 	int cpu = reboot_cpu;

Index: linux-2.6/include/linux/reboot.h
===================================================================

include/linux/reboot.h +0 -0
@@ -43,6 +43,7 @@ extern int unregister_reboot_notifier(st
  * Architecture-specific implementations of sys_reboot commands.
  */
 
Quoted 16 lines
Quoted 9 lines
include/linux/reboot.h +1 -0
+extern void migrate_to_reboot_cpu(void);
 extern void machine_restart(char *cmd);
 extern void machine_halt(void);
 extern void machine_power_off(void);

Index: linux-2.6/kernel/kexec.c
===================================================================

kernel/kexec.c int kernel_kexec(void) · line 1682

Quoted 7 lines
kernel/kexec.c +1 -0
@@ -1676,6 +1676,7 @@ int kernel_kexec(void)
 #endif
 	{
 		kernel_restart_prepare(NULL);
+		migrate_to_reboot_cpu();
 		printk(KERN_EMERG "Starting new kernel\n");
 		machine_shutdown();
 	}

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Tested-by
#15 lore raw
In reply to Vivek Goyal
Quoted 16 lines · Vivek Goyal
Quoted 8 lines · Baoquan He

Currently system always reboot after below message when execute "kexec -e".

[    0.572119] smpboot: Booting Node   0, Processors  #   1 OK

In commit 1b3a5d02ee070c8f9943333b9b6370f486601e0f, reboot= handling was moved to kerne/reboot.c. However, the code to migrate current thread to reboot cpu was removed. That cause this incorrect kexec behavior.

Quoted 6 lines

Now add that code block back.

Quoted 7 lines

Hi Bao,

This patch fixes the issue for me too. I noticed that we have generic function migrate_to_reboot_cpu() to achieve what we want and rest of the reboot paths are using it. So how about using that function. I wrote the new patch below. It works for me. Can you please give it a try.

Hi, Vivek

I confirm this patch works for me, even in a nested kexec scenario. Feel free to add these:

Thanks
WANG Chao

Quoted 58 lines

Thanks
Vivek

kexec: migrate to reboot cpu

Commit 1b3a5d02ee070c8f9943333b9b6370f486601e0f moved reboot= handling to generic code. In the process it also removed the code in native_machine_shutdown() which are moving reboot process to reboot_cpu/cpu0.

I guess that thought must have been that all reboot paths are calling migrate_to_reboot_cpu(), so we don't need this special handling. But kexec reboot path (kernel_kexec()) is not calling migrate_to_reboot_cpu() so above change broke kexec. Now reboot can happen on non-boot cpu and when INIT is sent in second kerneo to bring up BP, it brings down the machine.

So start calling migrate_to_reboot_cpu() in kexec reboot path to avoid this problem.

---
 include/linux/reboot.h |    1 +
 kernel/kexec.c         |    1 +
 kernel/reboot.c        |    2 +-
 3 files changed, 3 insertions(+), 1 deletion(-)

Index: linux-2.6/kernel/reboot.c
===================================================================

kernel/reboot.c +1 -1
@@ -104,7 +104,7 @@ int unregister_reboot_notifier(struct no
 }
 EXPORT_SYMBOL(unregister_reboot_notifier);
 
-static void migrate_to_reboot_cpu(void)
+void migrate_to_reboot_cpu(void)
 {
 	/* The boot cpu is always logical cpu 0 */
 	int cpu = reboot_cpu;

Index: linux-2.6/include/linux/reboot.h
===================================================================

include/linux/reboot.h +1 -0
@@ -43,6 +43,7 @@ extern int unregister_reboot_notifier(st
  * Architecture-specific implementations of sys_reboot commands.
  */
 
+extern void migrate_to_reboot_cpu(void);
 extern void machine_restart(char *cmd);
 extern void machine_halt(void);
 extern void machine_power_off(void);

Index: linux-2.6/kernel/kexec.c
===================================================================

kernel/kexec.c int kernel_kexec(void) · line 1682

Quoted 7 lines
kernel/kexec.c +1 -0
@@ -1676,6 +1676,7 @@ int kernel_kexec(void)
 #endif
 	{
 		kernel_restart_prepare(NULL);
+		migrate_to_reboot_cpu();
 		printk(KERN_EMERG "Starting new kernel\n");
 		machine_shutdown();
 	}

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec