Skip to contentSkip to conversation

MIPS: kexec/kdump: Fix smp reboot and other issues

PATCH Superseded Dengcheng Zhu 2 participants 17 messages Sep 5, 2018 – Sep 7, 2018

Subsystems (4)

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v4 cover MIPS: kexec/kdump: Fix smp reboot and other issues

The issues are mentioned in patches 1/4/5/6. I will update kdump documentation for MIPS if the series gets accepted. Testing has been done on single core i6500/Boston with IOCU, dual core i6500 without IOCU, and dual core interAptiv without IOCU.

Changes:

v4 - v3:
* In patch #1, idle_task_exit() is moved out from play_dead() to its sole caller arch_cpu_idle_dead(). So no interface change of play_dead().
* In patch #6, the kexec_prepare method for the Generic platform is defined as uhi_machine_kexec_prepare() for all platforms using UHI boot protocol.

v3 - v2:
* Code style changes according to scripts/checkpatch.pl --strict. Patch #6, like before, still has a warning message reminding if MAINTAINERS needs updating. But it does NOT involve a maintainer change.
* Add LIBFDT to CPU_LOONGSON3 for default_machine_kexec_prepare().

v2 - v1:
* Tested on MIPS32R2 platform in addition to MIPS64R6.
* Added patches #5 and #6.
* In patch #2, removed the unnecessary inclusion of asm/mipsmtregs.h

Dengcheng Zhu (6):
MIPS: Make play_dead() work for kexec
MIPS: kexec: Let the new kernel handle all CPUs
MIPS: kexec: Deprecate (relocated_)kexec_smp_wait
MIPS: kexec: Do not flush system wide caches in machine_kexec()
MIPS: kexec: Relax memory restriction
MIPS: kexec: Use prepare method from Generic for UHI platforms

 arch/mips/Kconfig                     |  4 ++
 arch/mips/cavium-octeon/setup.c       |  2 +-
 arch/mips/cavium-octeon/smp.c         | 34 ++++++-----
 arch/mips/generic/Makefile            |  1 -
 arch/mips/generic/kexec.c             | 44 --------------
 arch/mips/include/asm/kexec.h         | 11 ++--
 arch/mips/include/asm/smp.h           |  4 +-
 arch/mips/kernel/crash.c              |  4 +-
 arch/mips/kernel/machine_kexec.c      | 82 +++++++++++++++++++++++----
 arch/mips/kernel/process.c            |  2 +
 arch/mips/kernel/relocate_kernel.S    | 39 -------------
 arch/mips/kernel/smp-bmips.c          |  8 ++-
 arch/mips/kernel/smp-cps.c            | 49 +++++++++-------
 arch/mips/loongson64/loongson-3/smp.c | 82 ++++++++++++++-------------
 14 files changed, 183 insertions(+), 183 deletions(-)
 delete mode 100644 arch/mips/generic/kexec.c
Signature · 1 line
2.17.1
#1 lore raw

PATCH v4 1/6 MIPS: Make play_dead() work for kexec

Extract play_dead() from CONFIG_HOTPLUG_CPU and share with CONFIG_KEXEC. Meanwhile, move idle_task_exit() out to arch_cpu_idle_dead() because it's not meant for kexec.

This is needed to correctly support SMP new kernel in kexec. Before doing this, all non-crashing CPUs are waiting for the reboot signal (kexec_ready_to_reboot) to jump to kexec_start_address in kexec_smp_wait(), which creates some problems like incorrect CPU topology upon booting from new UP kernel, sluggish performance in MT environment during and after reboot, new SMP kernel not able to bring up secondary CPU etc.

It would make sense to let the new (SMP) kernel manage all CPUs, instead of crashing CPU booting from reboot_code_buffer whereas others jumping to kexec_start_address. To do this, play_dead() is well suitable for preparing a boot environment for the new kernel.

---
Changes:

* idle_task_exit() is moved out from play_dead() to its sole caller arch_cpu_idle_dead(). So no interface change of play_dead().

 arch/mips/cavium-octeon/smp.c         | 34 ++++++-----
 arch/mips/include/asm/smp.h           |  4 +-
 arch/mips/kernel/process.c            |  2 +
 arch/mips/kernel/smp-bmips.c          |  8 ++-
 arch/mips/kernel/smp-cps.c            | 49 +++++++++-------
 arch/mips/loongson64/loongson-3/smp.c | 82 ++++++++++++++-------------
 6 files changed, 99 insertions(+), 80 deletions(-)
arch/mips/cavium-octeon/smp.c +19 -15
index 75e7c8625659..19dea9622fe5 100644
@@ -280,11 +280,29 @@ static void octeon_smp_finish(void)
 	local_irq_enable();
 }
 
-#ifdef CONFIG_HOTPLUG_CPU
+#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_KEXEC)
 
 /* State of each CPU. */
 DEFINE_PER_CPU(int, cpu_state);
 
+void play_dead(void)
+{
+	int cpu = cpu_number_map(cvmx_get_core_num());
+
+	octeon_processor_boot = 0xff;
+	per_cpu(cpu_state, cpu) = CPU_DEAD;
+
+	/* make the change visible before going off */
+	mb();
+
+	while (1)	/* core will be reset here */
+		;
+}
+
+#endif /* CONFIG_HOTPLUG_CPU || CONFIG_KEXEC */
+
+#ifdef CONFIG_HOTPLUG_CPU
+
 static int octeon_cpu_disable(void)
 {
 	unsigned int cpu = smp_processor_id();
@@ -343,20 +361,6 @@ static void octeon_cpu_die(unsigned int cpu)
 	cvmx_write_csr(CVMX_CIU_PP_RST, 0);
 }
 
-void play_dead(void)
-{
-	int cpu = cpu_number_map(cvmx_get_core_num());
-
-	idle_task_exit();
-	octeon_processor_boot = 0xff;
-	per_cpu(cpu_state, cpu) = CPU_DEAD;
-
-	mb();
-
-	while (1)	/* core will be reset here */
-		;
-}
-
 static void start_after_reset(void)
 {
 	kernel_entry(0, 0, 0);	/* set a2 = 0 for secondary core */
arch/mips/include/asm/smp.h +3 -1
index 88ebd83b3bf9..cad5e78889c0 100644
@@ -77,8 +77,10 @@ static inline void __cpu_die(unsigned int cpu)
 
 	mp_ops->cpu_die(cpu);
 }
+#endif
 
-extern void play_dead(void);
+#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_KEXEC)
+void play_dead(void);
 #endif
 
 /*
arch/mips/kernel/process.c +2 -0
index 8d85046adcc8..8e5868362e55 100644
@@ -14,6 +14,7 @@
 #include <linux/sched/debug.h>
 #include <linux/sched/task.h>
 #include <linux/sched/task_stack.h>
+#include <linux/sched/hotplug.h>
 #include <linux/tick.h>
 #include <linux/kernel.h>
 #include <linux/mm.h>
@@ -53,6 +54,7 @@
 #ifdef CONFIG_HOTPLUG_CPU
 void arch_cpu_idle_dead(void)
 {
+	idle_task_exit();
 	play_dead();
 }
 #endif
arch/mips/kernel/smp-bmips.c +5 -3
index 159e83add4bb..284e49c8222f 100644
@@ -381,10 +381,12 @@ static void bmips_cpu_die(unsigned int cpu)
 {
 }
 
+#endif /* CONFIG_HOTPLUG_CPU */
+
+#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_KEXEC)
+
 void __ref play_dead(void)
 {
-	idle_task_exit();
-
 	/* flush data cache */
 	_dma_cache_wback_inv(0, ~0);
 
@@ -409,7 +411,7 @@ void __ref play_dead(void)
 	: : : "memory");
 }
 
-#endif /* CONFIG_HOTPLUG_CPU */
+#endif /* CONFIG_HOTPLUG_CPU || CONFIG_KEXEC */
 
 const struct plat_smp_ops bmips43xx_smp_ops = {
 	.smp_setup		= bmips_smp_setup,
arch/mips/kernel/smp-cps.c +27 -22

67 lines not on this page · raw message

arch/mips/loongson64/loongson-3/smp.c +43 -39

100 lines not on this page · raw message

Signature · 1 line
2.17.1
#2 lore raw

PATCH v4 2/6 MIPS: kexec: Let the new kernel handle all CPUs

In reply to Dengcheng Zhu

Use play_dead() to prepare an environment for the new kernel. Do not rely on non-crashing CPUs jumping to kexec_start_address.

---
Changes:

* Use play_dead() without an extra parameter.

 arch/mips/cavium-octeon/setup.c  |  2 +-
 arch/mips/include/asm/kexec.h    |  3 ++-
 arch/mips/kernel/crash.c         |  4 +++-
 arch/mips/kernel/machine_kexec.c | 28 ++++++++++++++++++++++++----
 4 files changed, 30 insertions(+), 7 deletions(-)
arch/mips/cavium-octeon/setup.c +1 -1
index a8034d0dcade..3a1f7fa42413 100644
@@ -95,7 +95,7 @@ static void octeon_kexec_smp_down(void *ignored)
 	"	sync						\n"
 	"	synci	($0)					\n");
 
-	relocated_kexec_smp_wait(NULL);
+	kexec_smp_reboot();
 }
 #endif
 
arch/mips/include/asm/kexec.h +2 -1
index 493a3cc7c39a..a8e0bfc0da19 100644
@@ -39,12 +39,13 @@ extern unsigned long kexec_args[4];
 extern int (*_machine_kexec_prepare)(struct kimage *);
 extern void (*_machine_kexec_shutdown)(void);
 extern void (*_machine_crash_shutdown)(struct pt_regs *regs);
-extern void default_machine_crash_shutdown(struct pt_regs *regs);
+void default_machine_crash_shutdown(struct pt_regs *regs);
 #ifdef CONFIG_SMP
 extern const unsigned char kexec_smp_wait[];
 extern unsigned long secondary_kexec_args[4];
 extern void (*relocated_kexec_smp_wait) (void *);
 extern atomic_t kexec_ready_to_reboot;
+void kexec_smp_reboot(void);
 extern void (*_crash_smp_send_stop)(void);
 #endif
 #endif
arch/mips/kernel/crash.c +3 -1
index d455363d51c3..6d4320067cff 100644
@@ -43,7 +43,9 @@ static void crash_shutdown_secondary(void *passed_regs)
 
 	while (!atomic_read(&kexec_ready_to_reboot))
 		cpu_relax();
-	relocated_kexec_smp_wait(NULL);
+
+	kexec_smp_reboot();
+
 	/* NOTREACHED */
 }
 
arch/mips/kernel/machine_kexec.c +24 -4
index 8b574bcd39ba..900475ae256d 100644
@@ -19,6 +19,10 @@ extern const size_t relocate_new_kernel_size;
 extern unsigned long kexec_start_address;
 extern unsigned long kexec_indirection_page;
 
+static unsigned long reboot_code_buffer;
+
+typedef void (*noretfun_t)(void) __noreturn;
+
 int (*_machine_kexec_prepare)(struct kimage *) = NULL;
 void (*_machine_kexec_shutdown)(void) = NULL;
 void (*_machine_crash_shutdown)(struct pt_regs *regs) = NULL;
@@ -26,6 +30,20 @@ void (*_machine_crash_shutdown)(struct pt_regs *regs) = NULL;
 void (*relocated_kexec_smp_wait) (void *);
 atomic_t kexec_ready_to_reboot = ATOMIC_INIT(0);
 void (*_crash_smp_send_stop)(void) = NULL;
+
+void kexec_smp_reboot(void)
+{
+	if (smp_processor_id() > 0) {
+		/*
+		 * Instead of cpu_relax() or wait, this is needed for kexec
+		 * smp reboot. Kdump usually doesn't require an smp new
+		 * kernel, but kexec may do.
+		 */
+		play_dead();
+	} else {
+		((noretfun_t)reboot_code_buffer)();
+	}
+}
 #endif
 
 static void kexec_image_info(const struct kimage *kimage)
@@ -79,12 +97,9 @@ machine_crash_shutdown(struct pt_regs *regs)
 		default_machine_crash_shutdown(regs);
 }
 
-typedef void (*noretfun_t)(void) __noreturn;
-
 void
 machine_kexec(struct kimage *image)
 {
-	unsigned long reboot_code_buffer;
 	unsigned long entry;
 	unsigned long *ptr;
 
@@ -132,6 +147,11 @@ machine_kexec(struct kimage *image)
 		(void *)(kexec_smp_wait - relocate_new_kernel);
 	smp_wmb();
 	atomic_set(&kexec_ready_to_reboot, 1);
-#endif
+
+	kexec_smp_reboot();
+
+	/* NOT REACHED */
+#else
 	((noretfun_t) reboot_code_buffer)();
+#endif
 }
Signature · 1 line
2.17.1
#3 lore raw

PATCH v4 3/6 MIPS: kexec: Deprecate (relocated_)kexec_smp_wait

In reply to Dengcheng Zhu

Since we've changed the SMP boot mechanism (by having the new kernel handle all CPUs), now remove (relocated_)kexec_smp_wait.

---
 arch/mips/include/asm/kexec.h      |  2 --
 arch/mips/kernel/machine_kexec.c   |  5 ----
 arch/mips/kernel/relocate_kernel.S | 39 ------------------------------
 3 files changed, 46 deletions(-)
arch/mips/include/asm/kexec.h +0 -2
index a8e0bfc0da19..33f31a8bd8d3 100644
@@ -41,9 +41,7 @@ extern void (*_machine_kexec_shutdown)(void);
 extern void (*_machine_crash_shutdown)(struct pt_regs *regs);
 void default_machine_crash_shutdown(struct pt_regs *regs);
 #ifdef CONFIG_SMP
-extern const unsigned char kexec_smp_wait[];
 extern unsigned long secondary_kexec_args[4];
-extern void (*relocated_kexec_smp_wait) (void *);
 extern atomic_t kexec_ready_to_reboot;
 void kexec_smp_reboot(void);
 extern void (*_crash_smp_send_stop)(void);
arch/mips/kernel/machine_kexec.c +0 -5
index 900475ae256d..baffc7113204 100644
@@ -27,7 +27,6 @@ int (*_machine_kexec_prepare)(struct kimage *) = NULL;
 void (*_machine_kexec_shutdown)(void) = NULL;
 void (*_machine_crash_shutdown)(struct pt_regs *regs) = NULL;
 #ifdef CONFIG_SMP
-void (*relocated_kexec_smp_wait) (void *);
 atomic_t kexec_ready_to_reboot = ATOMIC_INIT(0);
 void (*_crash_smp_send_stop)(void) = NULL;
 
@@ -142,10 +141,6 @@ machine_kexec(struct kimage *image)
 	printk("Bye ...\n");
 	__flush_cache_all();
 #ifdef CONFIG_SMP
-	/* All secondary cpus now may jump to kexec_wait cycle */
-	relocated_kexec_smp_wait = reboot_code_buffer +
-		(void *)(kexec_smp_wait - relocate_new_kernel);
-	smp_wmb();
 	atomic_set(&kexec_ready_to_reboot, 1);
 
 	kexec_smp_reboot();
arch/mips/kernel/relocate_kernel.S +0 -39
index c6bbf2165051..14e0eaf12306 100644
@@ -100,45 +100,6 @@ done:
 	j		s1
 	END(relocate_new_kernel)
 
-#ifdef CONFIG_SMP
-/*
- * Other CPUs should wait until code is relocated and
- * then start at entry (?) point.
- */
-LEAF(kexec_smp_wait)
-	PTR_L		a0, s_arg0
-	PTR_L		a1, s_arg1
-	PTR_L		a2, s_arg2
-	PTR_L		a3, s_arg3
-	PTR_L		s1, kexec_start_address
-
-	/* Non-relocated address works for args and kexec_start_address ( old
-	 * kernel is not overwritten). But we need relocated address of
-	 * kexec_flag.
-	 */
-
-	bal		1f
-1:	move		t1,ra;
-	PTR_LA		t2,1b
-	PTR_LA		t0,kexec_flag
-	PTR_SUB		t0,t0,t2;
-	PTR_ADD		t0,t1,t0;
-
-1:	LONG_L		s0, (t0)
-	bne		s0, zero,1b
-
-#ifdef CONFIG_CPU_CAVIUM_OCTEON
-	.set push
-	.set noreorder
-	synci		0($0)
-	.set pop
-#else
-	sync
-#endif
-	j		s1
-	END(kexec_smp_wait)
-#endif
-
 #ifdef __mips64
        /* all PTR's must be aligned to 8 byte in 64-bit mode */
        .align  3
Signature · 1 line
2.17.1
#4 lore raw

PATCH v4 4/6 MIPS: kexec: Do not flush system wide caches in machine_kexec()

In reply to Dengcheng Zhu

Instead of __flush_cache_all(), simply flush local icache range. In systems without IOCU, flushing system wide caches require sending IPIs. But other CPUs have disabled local IRQs waiting for the reboot signal. It will then cause system hang.

This patch fixes this problem.

---
 arch/mips/kernel/machine_kexec.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
arch/mips/kernel/machine_kexec.c +10 -1
index baffc7113204..c2119e448490 100644
@@ -139,7 +139,16 @@ machine_kexec(struct kimage *image)
 
 	printk("Will call new kernel at %08lx\n", image->start);
 	printk("Bye ...\n");
-	__flush_cache_all();
+	/*
+	 * __flush_cache_all() is expensive but unnecessary. More
+	 * importantly, it could freeze the system as it may need to send
+	 * IPIs, whereas other CPUs have been waiting for the reboot signal
+	 * (kexec_ready_to_reboot) with local irqs disabled, because
+	 * machine_crash_shutdown() has been called prior to entering
+	 * this function - machine_kexec().
+	 */
+	local_flush_icache_range(reboot_code_buffer,
+				 reboot_code_buffer + relocate_new_kernel_size);
 #ifdef CONFIG_SMP
 	atomic_set(&kexec_ready_to_reboot, 1);
 
Signature · 1 line
2.17.1
#5 lore raw

PATCH v4 5/6 MIPS: kexec: Relax memory restriction

In reply to Dengcheng Zhu

We can rely on the system kernel and the dump capture kernel themselves in memory usage.

Being restrictive with 512MB limit may cause kexec tool failure on some platforms.

---
 arch/mips/include/asm/kexec.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
arch/mips/include/asm/kexec.h +3 -3
index 33f31a8bd8d3..2583e6a201e3 100644
@@ -12,11 +12,11 @@
 #include <asm/stacktrace.h>
 
 /* Maximum physical address we can use pages from */
-#define KEXEC_SOURCE_MEMORY_LIMIT (0x20000000)
+#define KEXEC_SOURCE_MEMORY_LIMIT (-1UL)
 /* Maximum address we can reach in physical address mode */
-#define KEXEC_DESTINATION_MEMORY_LIMIT (0x20000000)
+#define KEXEC_DESTINATION_MEMORY_LIMIT (-1UL)
  /* Maximum address we can use for the control code buffer */
-#define KEXEC_CONTROL_MEMORY_LIMIT (0x20000000)
+#define KEXEC_CONTROL_MEMORY_LIMIT (-1UL)
 /* Reserve 3*4096 bytes for board-specific info */
 #define KEXEC_CONTROL_PAGE_SIZE (4096 + 3*4096)
 
Signature · 1 line
2.17.1
#6 lore raw

PATCH v4 6/6 MIPS: kexec: Use prepare method from Generic for UHI platforms

In reply to Dengcheng Zhu

The kexec_prepare method for the Generic platform should be applicable to others using UHI boot protocol. In arch/mips/Kconfig, "select UHI_BOOT" is needed for such platforms.

If the platform has its own method for _machine_kexec_prepare(), that one will be used.

---
Changes:

* The kexec_prepare method for the Generic platform is defined as uhi_machine_kexec_prepare() for all platforms using UHI boot protocol.

 arch/mips/Kconfig                |  4 +++
 arch/mips/generic/Makefile       |  1 -
 arch/mips/generic/kexec.c        | 44 --------------------------------
 arch/mips/kernel/machine_kexec.c | 38 ++++++++++++++++++++++++++-
 4 files changed, 41 insertions(+), 46 deletions(-)
 delete mode 100644 arch/mips/generic/kexec.c
arch/mips/Kconfig +4 -0
index 08c10c518f83..260f2ee083ac 100644
@@ -129,6 +129,7 @@ config MIPS_GENERIC
 	select USB_UHCI_BIG_ENDIAN_DESC if CPU_BIG_ENDIAN
 	select USB_UHCI_BIG_ENDIAN_MMIO if CPU_BIG_ENDIAN
 	select USE_OF
+	select UHI_BOOT
 	help
 	  Select this to build a kernel which aims to support multiple boards,
 	  generally using a flattened device tree passed from the bootloader
@@ -2907,6 +2908,9 @@ config USE_OF
 	select OF_EARLY_FLATTREE
 	select IRQ_DOMAIN
 
+config UHI_BOOT
+	bool
+
 config BUILTIN_DTB
 	bool
 
arch/mips/generic/Makefile +0 -1
index d03a36f869a4..181aa1335419 100644
@@ -15,5 +15,4 @@ obj-y += proc.o
 obj-$(CONFIG_YAMON_DT_SHIM)		+= yamon-dt.o
 obj-$(CONFIG_LEGACY_BOARD_SEAD3)	+= board-sead3.o
 obj-$(CONFIG_LEGACY_BOARD_OCELOT)	+= board-ocelot.o
-obj-$(CONFIG_KEXEC)			+= kexec.o
 obj-$(CONFIG_VIRT_BOARD_RANCHU)		+= board-ranchu.o
arch/mips/generic/kexec.c deleted +0 -44

44 lines not on this page · raw message

arch/mips/kernel/machine_kexec.c +37 -1
index c2119e448490..209196b195c5 100644
@@ -9,6 +9,7 @@
 #include <linux/kexec.h>
 #include <linux/mm.h>
 #include <linux/delay.h>
+#include <linux/libfdt.h>
 
 #include <asm/cacheflush.h>
 #include <asm/page.h>
@@ -23,7 +24,6 @@ static unsigned long reboot_code_buffer;
 
 typedef void (*noretfun_t)(void) __noreturn;
 
-int (*_machine_kexec_prepare)(struct kimage *) = NULL;
 void (*_machine_kexec_shutdown)(void) = NULL;
 void (*_machine_crash_shutdown)(struct pt_regs *regs) = NULL;
 #ifdef CONFIG_SMP
@@ -65,6 +65,42 @@ static void kexec_image_info(const struct kimage *kimage)
 	}
 }
 
+#ifdef CONFIG_UHI_BOOT
+static int uhi_machine_kexec_prepare(struct kimage *kimage)
+{
+	int i;
+
+	/*
+	 * In case DTB file is not passed to the new kernel, a flat device
+	 * tree will be created by kexec tool. It holds modified command
+	 * line for the new kernel.
+	 */
+	for (i = 0; i < kimage->nr_segments; i++) {
+		struct fdt_header fdt;
+
+		if (kimage->segment[i].memsz <= sizeof(fdt))
+			continue;
+
+		if (copy_from_user(&fdt, kimage->segment[i].buf, sizeof(fdt)))
+			continue;
+
+		if (fdt_check_header(&fdt))
+			continue;
+
+		kexec_args[0] = -2;
+		kexec_args[1] = (unsigned long)
+			phys_to_virt((unsigned long)kimage->segment[i].mem);
+		break;
+	}
+
+	return 0;
+}
+
+int (*_machine_kexec_prepare)(struct kimage *) = uhi_machine_kexec_prepare;
+#else
+int (*_machine_kexec_prepare)(struct kimage *) = NULL;
+#endif /* CONFIG_UHI_BOOT */
+
 int
 machine_kexec_prepare(struct kimage *kimage)
 {
Signature · 1 line
2.17.1
#7 lore raw
In reply to Dengcheng Zhu

Hi Dengcheng,

Quoted 5 lines · Dengcheng Zhu

The issues are mentioned in patches 1/4/5/6. I will update kdump documentation for MIPS if the series gets accepted. Testing has been done on single core i6500/Boston with IOCU, dual core i6500 without IOCU, and dual core interAptiv without IOCU.

Quoted 7 lines

Changes:

v4 - v3:
* In patch #1, idle_task_exit() is moved out from play_dead() to its sole caller arch_cpu_idle_dead(). So no interface change of play_dead().
* In patch #6, the kexec_prepare method for the Generic platform is defined as uhi_machine_kexec_prepare() for all platforms using UHI boot protocol.

Thanks! I've applied patches 1,2,3,5 to a test branch with a few changes:

git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux.git test-kexec

https://git.kernel.org/pub/scm/linux/kernel/git/mips/linux.git/log/?h=test-kexec

I didn't apply patch 4 because I'm not sure it's correct & I believe the changes in the branch above should take care of it - CPUs that reach kexec_this_cpu() are maked offline, so they shouldn't be IPI'd by __flush_cache_all(). The CPU that runs machine_kexec() should still flush its dcache (& the L2), and then CPU 0 invalidates its icache in kexec_this_cpu() prior to jumping into reboot_code_buffer.

I'm also still not sure about patch 6 - since no platforms besides the arch/mips/generic/ make use of the UHI boot code yet I think it'd be best to leave as-is. If we do ever need to use it from another platform then we can deal with the problem then. If an out of tree platform needs to use it then for now it could copy generic_kexec_prepare() and deal with removing the duplication when it heads upstream.

Could you take a look & let me know if you see any problems?

Thanks,
Paul

#8 lore raw

Hi Paul,

Quoted 18 lines · Paul Burton

Hi Dengcheng,

Quoted 5 lines · Dengcheng Zhu

The issues are mentioned in patches 1/4/5/6. I will update kdump documentation for MIPS if the series gets accepted. Testing has been done on single core i6500/Boston with IOCU, dual core i6500 without IOCU, and dual core interAptiv without IOCU.

Quoted 7 lines

Changes:

v4 - v3:
* In patch #1, idle_task_exit() is moved out from play_dead() to its sole caller arch_cpu_idle_dead(). So no interface change of play_dead().
* In patch #6, the kexec_prepare method for the Generic platform is defined as uhi_machine_kexec_prepare() for all platforms using UHI boot protocol.

Thanks! I've applied patches 1,2,3,5 to a test branch with a few changes:

Quoted 8 lines

git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux.git test-kexec

https://git.kernel.org/pub/scm/linux/kernel/git/mips/linux.git/log/?h=test-kexec

I didn't apply patch 4 because I'm not sure it's correct & I believe the changes in the branch above should take care of it - CPUs that reach kexec_this_cpu() are maked offline, so they shouldn't be IPI'd by __flush_cache_all().

I believe patch 4 is necessary. As mentioned in the code comment and patch comment of that patch, machine_crash_shutdown() is called prior to machine_kexec() in the kdump sequence. So other CPUs have disabled local IRQs waiting for the reboot signal.

In fact, in kexec_this_cpu() [you renamed and modified kexec_smp_reboot()], the added marking CPU offline will cause system hang (tested). This is because it will change how play_dead() will work.

Quoted 10 lines

The CPU that runs machine_kexec() should still
flush its dcache (& the L2), and then CPU 0 invalidates its icache in kexec_this_cpu() prior to jumping into reboot_code_buffer.

I'm also still not sure about patch 6 - since no platforms besides the arch/mips/generic/ make use of the UHI boot code yet I think it'd be best to leave as-is. If we do ever need to use it from another platform then we can deal with the problem then. If an out of tree platform needs to use it then for now it could copy generic_kexec_prepare() and deal with removing the duplication when it heads upstream.

Understood. It really depends on how this problem is viewed. If patch #6 is considered creating a framework for future UHI platforms, then it has the following facts:

* It doesn't create code redundancy. I mean, it does not add unnecessary code to the kernel.

* Out of tree platforms will get access to this functionality by a simple "select UHI_BOOT". When the kernel developer of an out-of-tree platform wants to make kexec work, they will naturally look at machine_kexec.c, where they will find this UHI stuff, obviously telling them to do "select UHI_BOOT". Otherwise, unless they google onto this discussion thread, it's harder to know the solution to the kexec_args related problem hides in the code of another platform (Generic).

* It simplifies work if the out of tree platform wants to upstream.

Quoted 5 lines

Could you take a look & let me know if you see any problems?

Thanks,
Paul

Thanks,

Dengcheng

---------------------------------------------------------------------------

*From:* Paul Burton <mailto:[email protected]>
*Sent:* Wednesday, September 05, 2018 3:54PM
*To:* Dengcheng Zhu <mailto:[email protected]>
*Cc:* Pburton <mailto:[email protected]>, Ralf <mailto:[email protected]>, Linux-mips <mailto:[email protected]>, Rachel.mozes <mailto:[email protected]> *Subject:* Re: [PATCH v4 0/6] MIPS: kexec/kdump: Fix smp reboot and other issues

Hi Dengcheng,

Quoted 5 lines · Dengcheng Zhu

The issues are mentioned in patches 1/4/5/6. I will update kdump documentation for MIPS if the series gets accepted. Testing has been done on single core i6500/Boston with IOCU, dual core i6500 without IOCU, and dual core interAptiv without IOCU.

Quoted 7 lines

Changes:

v4 - v3:
* In patch #1, idle_task_exit() is moved out from play_dead() to its sole caller arch_cpu_idle_dead(). So no interface change of play_dead().
* In patch #6, the kexec_prepare method for the Generic platform is defined as uhi_machine_kexec_prepare() for all platforms using UHI boot protocol.

Thanks! I've applied patches 1,2,3,5 to a test branch with a few changes:

git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux.git test-kexec

https://git.kernel.org/pub/scm/linux/kernel/git/mips/linux.git/log/?h=test-kexec

I didn't apply patch 4 because I'm not sure it's correct & I believe the changes in the branch above should take care of it - CPUs that reach kexec_this_cpu() are maked offline, so they shouldn't be IPI'd by __flush_cache_all(). The CPU that runs machine_kexec() should still flush its dcache (& the L2), and then CPU 0 invalidates its icache in kexec_this_cpu() prior to jumping into reboot_code_buffer.

I'm also still not sure about patch 6 - since no platforms besides the arch/mips/generic/ make use of the UHI boot code yet I think it'd be best to leave as-is. If we do ever need to use it from another platform then we can deal with the problem then. If an out of tree platform needs to use it then for now it could copy generic_kexec_prepare() and deal with removing the duplication when it heads upstream.

Could you take a look & let me know if you see any problems?

Thanks,
Paul

#9 lore raw

Hi Dengcheng,

Quoted 5 lines · Dengcheng Zhu
Quoted 4 lines

I didn't apply patch 4 because I'm not sure it's correct & I believe the changes in the branch above should take care of it - CPUs that reach kexec_this_cpu() are maked offline, so they shouldn't be IPI'd by __flush_cache_all().

Quoted 8 lines

I believe patch 4 is necessary. As mentioned in the code comment and patch comment of that patch, machine_crash_shutdown() is called prior to machine_kexec() in the kdump sequence. So other CPUs have disabled local IRQs waiting for the reboot signal.

In fact, in kexec_this_cpu() [you renamed and modified kexec_smp_reboot()], the added marking CPU offline will cause system hang (tested). This is because it will change how play_dead() will work.

So the problem is that patch 4 isn't really right either, and I suspect the hang you mention probably shows a problem with the whole play_dead() approach from patches 1 & 2 too.

On the cache flushing, what we really need is for stores performed by the CPU running machine_kexec() via its dcache to be observed by the icache of the CPU that jumps into reboot_code_buffer, which is always CPU 0 after patch 2. Using local_flush_icache_range() will only ensure that the icache of the CPU running machine_kexec() observes the changes made by that same CPU, and does nothing with CPU 0 unless they happen to be the same CPU or siblings sharing caches.

Consider I6x00 CPUs for example - patch 4 may *almost* work OK because both cpu_has_ic_fills_f_dc & cpu_icache_snoops_remote_store are true, so when the machine_kexec() CPU writes to reboot_code_buffer CPU 0's icache will observe that & will fill from dcache without needing it to be written back to L2. It's still not quite right because if CPU 0's icache already contained any valid lines within reboot_code_buffer (which could happen speculatively) then it'll execute stale/garbage code.

The hang you mention is likely down to the fact that play_dead(), at least the smp-cps.c implementation of it, is written for the CPU hotplug infrastructure which will only operate on one CPU at a time. The loop looking for a sibling CPU just isn't going to be safe to run on multiple CPUs in parallel like patch 2 does. That's true of either your original patch or my modification - it's just that my modified patch will cause siblings to race towards powering off the core, whilst your original will cause them to all halt themselves & never power off the core.

Which inclines me to return to my earlier thought that perhaps we shouldn't use play_dead() for this at all.

Quoted 26 lines
Quoted 10 lines

The CPU that runs machine_kexec() should still
flush its dcache (& the L2), and then CPU 0 invalidates its icache in kexec_this_cpu() prior to jumping into reboot_code_buffer.

I'm also still not sure about patch 6 - since no platforms besides the arch/mips/generic/ make use of the UHI boot code yet I think it'd be best to leave as-is. If we do ever need to use it from another platform then we can deal with the problem then. If an out of tree platform needs to use it then for now it could copy generic_kexec_prepare() and deal with removing the duplication when it heads upstream.

Understood. It really depends on how this problem is viewed. If patch #6 is considered creating a framework for future UHI platforms, then it has the following facts:

* It doesn't create code redundancy. I mean, it does not add unnecessary code to the kernel.

* Out of tree platforms will get access to this functionality by a simple "select UHI_BOOT". When the kernel developer of an out-of-tree platform wants to make kexec work, they will naturally look at machine_kexec.c, where they will find this UHI stuff, obviously telling them to do "select UHI_BOOT". Otherwise, unless they google onto this discussion thread, it's harder to know the solution to the kexec_args related problem hides in the code of another platform (Generic).

Quoted 1 line

* It simplifies work if the out of tree platform wants to upstream.

Well, ideally future platforms will just use arch/mips/generic rather than adding more platform/board code at all. Right now the only reason not to is if a system has a memory map that doesn't fit nicely, which should be resolved at some point by mapping the kernel which will allow the generic kernel to better support differing physical address maps.

So I hope we don't add more platform code anyway, which would make all this moot. I certainly don't want to encourage adding more by explicitly making it easier to do. And if there does come a point where we need to add more platform code for some good reason then we can deal with this at that point rather than speculating now.

For out of tree platforms, I don't think that copying the generic_kexec_prepare() function is particularly onerous anyway, and should be trivial to remove if such code is submitted upstream. Whilst I wouldn't want to make submitting code upstream more difficult, it's likely to need some amount of rework if submitted upstream anyway so this doesn't seem like a big deal.

I don't think upstream should be particularly concerned with making life easy for out of tree code - if one wants the benefit of their code being taken into account upstream, then one should submit one's code upstream. In the paraphrased words of my pastor at a marriage prep class - no benefits (except those already conferred by free access to open source software) without commitment.

Thanks,
Paul

#10 lore raw

Hi Paul,

Regarding cache flushing in machine_kexec(), I recommend simply removing __flush_cache_all() as CPU0 will do local icache flush before jumping to reboot_code_buffer, and other CPUs don't jump at all.

Re: marking CPU offline in kexec_this_cpu(), it's probably good NOT doing it either, as the system is going to reboot from CPU0 soon. HALT is good enough, no need to gate core power. As to whether it's safe running play_dead() in parallel, it shouldn't be a problem, as the loop is based on cpu online mask (which we don't mark offline as mentioned above) -- The CPU will simply halt itself. For non-MT CPUs, play_dead() does make sense as well, as it's supposed to stop this CPU's execution, getting ready to reboot from CPU0.

On UHI stuff, if it's true "future platforms will just use arch/mips/generic", then certainly no need of getting code out of Generic. My feeling is the customer is NOT doing so, according to "I fixed it by moving the code to another file". Other MIPS systems may be like this as well. I understand your "upstream-or-copy-code" point, but the only thing, which IMO meaningfully justifies patch 6 *for_such_cases*, is already said:

-----------------

When the kernel developer of an out-of-tree platform
wants to make kexec work, they will naturally look at machine_kexec.c, where they will find this UHI stuff, obviously telling them to do "select UHI_BOOT". Otherwise, unless they google onto this discussion thread, it's harder to know the solution to the kexec_args related problem hides in the code of another platform (Generic).

-----------------

In other words, it may take considerable efforts to realize "copying the generic_kexec_prepare() function" can solve the problem. Copying code itself is certainly not onerous.

Regards,

Dengcheng

Quoted 99 lines · Paul Burton

Hi Dengcheng,

Quoted 4 lines · Dengcheng Zhu
Quoted 4 lines

I didn't apply patch 4 because I'm not sure it's correct & I believe the changes in the branch above should take care of it - CPUs that reach kexec_this_cpu() are maked offline, so they shouldn't be IPI'd by __flush_cache_all().

Quoted 8 lines

I believe patch 4 is necessary. As mentioned in the code comment and patch comment of that patch, machine_crash_shutdown() is called prior to machine_kexec() in the kdump sequence. So other CPUs have disabled local IRQs waiting for the reboot signal.

In fact, in kexec_this_cpu() [you renamed and modified kexec_smp_reboot()], the added marking CPU offline will cause system hang (tested). This is because it will change how play_dead() will work.

So the problem is that patch 4 isn't really right either, and I suspect the hang you mention probably shows a problem with the whole play_dead() approach from patches 1 & 2 too.

On the cache flushing, what we really need is for stores performed by the CPU running machine_kexec() via its dcache to be observed by the icache of the CPU that jumps into reboot_code_buffer, which is always CPU 0 after patch 2. Using local_flush_icache_range() will only ensure that the icache of the CPU running machine_kexec() observes the changes made by that same CPU, and does nothing with CPU 0 unless they happen to be the same CPU or siblings sharing caches.

Consider I6x00 CPUs for example - patch 4 may *almost* work OK because both cpu_has_ic_fills_f_dc & cpu_icache_snoops_remote_store are true, so when the machine_kexec() CPU writes to reboot_code_buffer CPU 0's icache will observe that & will fill from dcache without needing it to be written back to L2. It's still not quite right because if CPU 0's icache already contained any valid lines within reboot_code_buffer (which could happen speculatively) then it'll execute stale/garbage code.

The hang you mention is likely down to the fact that play_dead(), at least the smp-cps.c implementation of it, is written for the CPU hotplug infrastructure which will only operate on one CPU at a time. The loop looking for a sibling CPU just isn't going to be safe to run on multiple CPUs in parallel like patch 2 does. That's true of either your original patch or my modification - it's just that my modified patch will cause siblings to race towards powering off the core, whilst your original will cause them to all halt themselves & never power off the core.

Which inclines me to return to my earlier thought that perhaps we shouldn't use play_dead() for this at all.

Quoted 25 lines
Quoted 10 lines

The CPU that runs machine_kexec() should still
flush its dcache (& the L2), and then CPU 0 invalidates its icache in kexec_this_cpu() prior to jumping into reboot_code_buffer.

I'm also still not sure about patch 6 - since no platforms besides the arch/mips/generic/ make use of the UHI boot code yet I think it'd be best to leave as-is. If we do ever need to use it from another platform then we can deal with the problem then. If an out of tree platform needs to use it then for now it could copy generic_kexec_prepare() and deal with removing the duplication when it heads upstream.

Understood. It really depends on how this problem is viewed. If patch #6 is considered creating a framework for future UHI platforms, then it has the following facts:

* It doesn't create code redundancy. I mean, it does not add unnecessary code to the kernel.

* Out of tree platforms will get access to this functionality by a simple "select UHI_BOOT". When the kernel developer of an out-of-tree platform wants to make kexec work, they will naturally look at machine_kexec.c, where they will find this UHI stuff, obviously telling them to do "select UHI_BOOT". Otherwise, unless they google onto this discussion thread, it's harder to know the solution to the kexec_args related problem hides in the code of another platform (Generic).

Quoted 1 line

* It simplifies work if the out of tree platform wants to upstream.

Well, ideally future platforms will just use arch/mips/generic rather than adding more platform/board code at all. Right now the only reason not to is if a system has a memory map that doesn't fit nicely, which should be resolved at some point by mapping the kernel which will allow the generic kernel to better support differing physical address maps.

So I hope we don't add more platform code anyway, which would make all this moot. I certainly don't want to encourage adding more by explicitly making it easier to do. And if there does come a point where we need to add more platform code for some good reason then we can deal with this at that point rather than speculating now.

For out of tree platforms, I don't think that copying the generic_kexec_prepare() function is particularly onerous anyway, and should be trivial to remove if such code is submitted upstream. Whilst I wouldn't want to make submitting code upstream more difficult, it's likely to need some amount of rework if submitted upstream anyway so this doesn't seem like a big deal.

I don't think upstream should be particularly concerned with making life easy for out of tree code - if one wants the benefit of their code being taken into account upstream, then one should submit one's code upstream. In the paraphrased words of my pastor at a marriage prep class - no benefits (except those already conferred by free access to open source software) without commitment.

Quoted 2 lines

Thanks,
Paul

Dengcheng

---------------------------------------------------------------------------

*From:* Paul Burton <mailto:[email protected]>
*Sent:* Thursday, September 06, 2018 1:34PM
*To:* Dengcheng Zhu <mailto:[email protected]>
*Cc:* Pburton <mailto:[email protected]>, Ralf <mailto:[email protected]>, Linux-mips <mailto:[email protected]>, Rachel.mozes <mailto:[email protected]> *Subject:* Re: [PATCH v4 0/6] MIPS: kexec/kdump: Fix smp reboot and other issues

Hi Dengcheng,

Quoted 4 lines · Dengcheng Zhu
Quoted 4 lines

I didn't apply patch 4 because I'm not sure it's correct & I believe the changes in the branch above should take care of it - CPUs that reach kexec_this_cpu() are maked offline, so they shouldn't be IPI'd by __flush_cache_all().

Quoted 8 lines

I believe patch 4 is necessary. As mentioned in the code comment and patch comment of that patch, machine_crash_shutdown() is called prior to machine_kexec() in the kdump sequence. So other CPUs have disabled local IRQs waiting for the reboot signal.

In fact, in kexec_this_cpu() [you renamed and modified kexec_smp_reboot()], the added marking CPU offline will cause system hang (tested). This is because it will change how play_dead() will work.

So the problem is that patch 4 isn't really right either, and I suspect the hang you mention probably shows a problem with the whole play_dead() approach from patches 1 & 2 too.

On the cache flushing, what we really need is for stores performed by the CPU running machine_kexec() via its dcache to be observed by the icache of the CPU that jumps into reboot_code_buffer, which is always CPU 0 after patch 2. Using local_flush_icache_range() will only ensure that the icache of the CPU running machine_kexec() observes the changes made by that same CPU, and does nothing with CPU 0 unless they happen to be the same CPU or siblings sharing caches.

Consider I6x00 CPUs for example - patch 4 may *almost* work OK because both cpu_has_ic_fills_f_dc & cpu_icache_snoops_remote_store are true, so when the machine_kexec() CPU writes to reboot_code_buffer CPU 0's icache will observe that & will fill from dcache without needing it to be written back to L2. It's still not quite right because if CPU 0's icache already contained any valid lines within reboot_code_buffer (which could happen speculatively) then it'll execute stale/garbage code.

The hang you mention is likely down to the fact that play_dead(), at least the smp-cps.c implementation of it, is written for the CPU hotplug infrastructure which will only operate on one CPU at a time. The loop looking for a sibling CPU just isn't going to be safe to run on multiple CPUs in parallel like patch 2 does. That's true of either your original patch or my modification - it's just that my modified patch will cause siblings to race towards powering off the core, whilst your original will cause them to all halt themselves & never power off the core.

Which inclines me to return to my earlier thought that perhaps we shouldn't use play_dead() for this at all.

Quoted 25 lines
Quoted 10 lines

The CPU that runs machine_kexec() should still
flush its dcache (& the L2), and then CPU 0 invalidates its icache in kexec_this_cpu() prior to jumping into reboot_code_buffer.

I'm also still not sure about patch 6 - since no platforms besides the arch/mips/generic/ make use of the UHI boot code yet I think it'd be best to leave as-is. If we do ever need to use it from another platform then we can deal with the problem then. If an out of tree platform needs to use it then for now it could copy generic_kexec_prepare() and deal with removing the duplication when it heads upstream.

Understood. It really depends on how this problem is viewed. If patch #6 is considered creating a framework for future UHI platforms, then it has the following facts:

* It doesn't create code redundancy. I mean, it does not add unnecessary code to the kernel.

* Out of tree platforms will get access to this functionality by a simple "select UHI_BOOT". When the kernel developer of an out-of-tree platform wants to make kexec work, they will naturally look at machine_kexec.c, where they will find this UHI stuff, obviously telling them to do "select UHI_BOOT". Otherwise, unless they google onto this discussion thread, it's harder to know the solution to the kexec_args related problem hides in the code of another platform (Generic).

Quoted 1 line

* It simplifies work if the out of tree platform wants to upstream.

Well, ideally future platforms will just use arch/mips/generic rather than adding more platform/board code at all. Right now the only reason not to is if a system has a memory map that doesn't fit nicely, which should be resolved at some point by mapping the kernel which will allow the generic kernel to better support differing physical address maps.

So I hope we don't add more platform code anyway, which would make all this moot. I certainly don't want to encourage adding more by explicitly making it easier to do. And if there does come a point where we need to add more platform code for some good reason then we can deal with this at that point rather than speculating now.

For out of tree platforms, I don't think that copying the generic_kexec_prepare() function is particularly onerous anyway, and should be trivial to remove if such code is submitted upstream. Whilst I wouldn't want to make submitting code upstream more difficult, it's likely to need some amount of rework if submitted upstream anyway so this doesn't seem like a big deal.

I don't think upstream should be particularly concerned with making life easy for out of tree code - if one wants the benefit of their code being taken into account upstream, then one should submit one's code upstream. In the paraphrased words of my pastor at a marriage prep class - no benefits (except those already conferred by free access to open source software) without commitment.

Thanks,
Paul

#11 lore raw

Hi Dengcheng,

Quoted 3 lines · Dengcheng Zhu

Regarding cache flushing in machine_kexec(), I recommend simply removing __flush_cache_all() as CPU0 will do local icache flush before jumping to reboot_code_buffer, and other CPUs don't jump at all.

Unfortunately that doesn't work - what if CPU 1 runs machine_kexec() & writes to reboot_code_buffer, then CPU 0 fetches stale/garbage code from L2 into its icache?

The icache flush I added in kexec_this_cpu() isn't enough in all systems unless the CPU that wrote the code writes it back far enough for a remote CPU to observe. The existing __flush_cache_all() is one heavy-handed way to achieve that.

Quoted 2 lines

Re: marking CPU offline in kexec_this_cpu(), it's probably good NOT doing it either, as the system is going to reboot from CPU0 soon.

...but the new kernel will have no knowledge of whether the old kernel had CPUs marked online or offline, so I don't follow the argument?

Marking CPUs offline does have the advantage that we won't try to IPI them. This just makes perfect sense to me, and note that both arch/arm & arch/x86 offline CPUs during kexec too (I *really* like the way arch/arm just uses disable_nonboot_cpus() to go through the usual hotplug sequence in the non-crash case).

Quoted 7 lines

HALT is good enough, no need to gate core power.
As to whether it's safe running play_dead() in parallel, it shouldn't be a problem, as the loop is based
on cpu online mask (which we don't mark offline as mentioned above) -- The CPU will simply halt itself. For non-MT CPUs, play_dead() does make sense as well, as it's supposed to stop this CPU's execution, getting ready to reboot from CPU0.

That feels like relying on play_dead() accidentally working rather than doing something it's designed for though, and it would come at the expense of not marking CPUs offline & the associated fragility of needing to avoid anything that might IPI them (like some cache flush functions, as we've seen). I'd much rather we figure out a way to do this without all that.

One option might be to add something like arch/x86's stop_other_cpus & crash_stop_other_cpus callbacks in struct plat_smp_ops, which we can then implement as appropriate. We'd hopefully still reuse some of the code from play_dead() implementations, but have the separation to allow them to function differently where needed (eg. the new callbacks could halt all threads in MT systems without caring about cpu_online_mask).

This would also give us a natural way to check whether a system actually supports kexec properly, as we could just return with an error from machine_kexec_prepare() if the stop_other_cpus callback isn't implemented for the system (ie. is NULL).

Thanks,
Paul

#12 lore raw

Hi Paul,

On cache issue, what if kexec_this_cpu() does local i$ flush and scache invalidation? This is done only on CPU0, no IPI needed. And only CUP0 will use reboot_code_buffer. In this way, marking other CPUs offline to prevent being sent IPI is not needed.

As to "the associated fragility of needing to avoid anything that might IPI them", I hope that's not a problem as the kexec sequence is determined: The last IPI received is to ask them to stand by on reboot signal, because other than kexec there is no other kernel code beyond our control, which might send IPIs.

Quoted 1 line

...but the new kernel will have no knowledge of whether the old kernel

had CPUs marked online or offline, so I don't follow the argument?

[Dengcheng] I meant the new kernel shouldn't need CPU online info from the old kernel. The new one is doing a "fresh" boot, provided CPUs 1+ have been halted.

Regarding the use of play_dead(), it's simple and generic - MIPS CPUs have implemented their own play_dead(), which we can use to stop execution. Other options may still come down to play_dead() to handle MT/MC details. The concern of running it in parallel is really about the online status, which has been explained above and in my last email.

A 'nicer' implementation stopping non-boot CPUs from one CPU might be available, but handling code duplication of play_dead() and saving CPU states could be a problem.

For now, no __flush_cache_all() in machine_kexec() + no marking offline in kexec_this_cpu() + scache invalidation in kexec_this_cpu() might be a way to go.

Thanks,

Dengcheng

---

From: Paul Burton
Sent: Thursday, September 6, 2018 4:21 PM
To: Dengcheng Zhu
Cc: Paul Burton; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH v4 0/6] MIPS: kexec/kdump: Fix smp reboot and other issues

Hi Dengcheng,

Quoted 3 lines · Dengcheng Zhu

Regarding cache flushing in machine_kexec(), I recommend simply removing __flush_cache_all() as CPU0 will do local icache flush before jumping to reboot_code_buffer, and other CPUs don't jump at all.

Unfortunately that doesn't work - what if CPU 1 runs machine_kexec() & writes to reboot_code_buffer, then CPU 0 fetches stale/garbage code from L2 into its icache?

The icache flush I added in kexec_this_cpu() isn't enough in all systems unless the CPU that wrote the code writes it back far enough for a remote CPU to observe. The existing __flush_cache_all() is one heavy-handed way to achieve that.

Quoted 2 lines

Re: marking CPU offline in kexec_this_cpu(), it's probably good NOT doing it either, as the system is going to reboot from CPU0 soon.

...but the new kernel will have no knowledge of whether the old kernel had CPUs marked online or offline, so I don't follow the argument?

Marking CPUs offline does have the advantage that we won't try to IPI them. This just makes perfect sense to me, and note that both arch/arm & arch/x86 offline CPUs during kexec too (I *really* like the way arch/arm just uses disable_nonboot_cpus() to go through the usual hotplug sequence in the non-crash case).

Quoted 7 lines

HALT is good enough, no need to gate core power.
As to whether it's safe running play_dead() in parallel, it shouldn't be a problem, as the loop is based
on cpu online mask (which we don't mark offline as mentioned above) -- The CPU will simply halt itself. For non-MT CPUs, play_dead() does make sense as well, as it's supposed to stop this CPU's execution, getting ready to reboot from CPU0.

That feels like relying on play_dead() accidentally working rather than doing something it's designed for though, and it would come at the expense of not marking CPUs offline & the associated fragility of needing to avoid anything that might IPI them (like some cache flush functions, as we've seen). I'd much rather we figure out a way to do this without all that.

One option might be to add something like arch/x86's stop_other_cpus & crash_stop_other_cpus callbacks in struct plat_smp_ops, which we can then implement as appropriate. We'd hopefully still reuse some of the code from play_dead() implementations, but have the separation to allow them to function differently where needed (eg. the new callbacks could halt all threads in MT systems without caring about cpu_online_mask).

This would also give us a natural way to check whether a system actually supports kexec properly, as we could just return with an error from machine_kexec_prepare() if the stop_other_cpus callback isn't implemented for the system (ie. is NULL).

Thanks,
    Paul

#13 lore raw

Hi Dengcheng,

Quoted 4 lines · Dengcheng Zhu

On cache issue, what if kexec_this_cpu() does local i$ flush and scache invalidation? This is done only on CPU0, no IPI needed. And only CUP0 will use reboot_code_buffer. In this way, marking other CPUs offline to prevent being sent IPI is not needed.

The problem is that the "only CPU0 will use reboot_code_buffer" part isn't true because CPU0 might not be the one that writes to it. Though perhaps if we moved the memcpy() from machine_kexec() to kexec_this_cpu() that could work.

It still doesn't seem ideal to me that we'd have to even consider IPIs & the fact that we might silently hang if we accidentally send one. I really think it would be cleaner to mark CPUs offline & the problem is solved naturally - we shouldn't IPI these CPUs, and marking them offline means that we won't. Problem solved, and consistently with other architectures.

Quoted 4 lines

As to "the associated fragility of needing to avoid anything that might IPI them", I hope that's not a problem as the kexec sequence is determined: The last IPI received is to ask them to stand by on reboot signal, because other than kexec there is no other kernel code beyond our control, which might send IPIs.

But it does mean we can't call certain things, like __flush_cache_all(), which means that developers need to 1) know that and 2) remember to avoid any functions like this that perform an IPI behind the scenes. It just feels like loading a gun & leaving it pointed at our feet - by itself it's not necessarily going to cause a problem, but it would be oh so easy to trigger problems later.

Quoted 11 lines
Quoted 1 line

...but the new kernel will have no knowledge of whether the old kernel

had CPUs marked online or offline, so I don't follow the argument?

[Dengcheng] I meant the new kernel shouldn't need CPU online info from the old kernel. The new one is doing a "fresh" boot, provided CPUs 1+ have been halted.

Regarding the use of play_dead(), it's simple and generic - MIPS CPUs have implemented their own play_dead(), which we can use to stop execution. Other options may still come down to play_dead() to handle MT/MC details. The concern of running it in parallel is really about the online status, which has been explained above and in my last email.

But play_dead() really doesn't mean what you're suggesting it means.

What we want for kexec is that the CPU is stopped entirely - either halted or powered down. Failing that it needs to be running a loop from somewhere that we know isn't going to be overwritten, and the code it's running needs to be able to hand over to the new kernel later (eg. the existing kexec_smp_wait loop implements this latter approach).

Whilst the smp-cps.c implementation of play_dead() will currently halt or power down the CPU, that's not universally true & doesn't necessarily even need to remain true for smp-cps.c in the future (eg. we could decide the halt/power down step would be cleaner to do in cps_cpu_die).

A quick look at the other play_dead() implementations we have shows that both the cavium-octeon & loongson3 versions leave the CPU running a loop (within the current/old kernel), whilst the bmips version just sits at a wait instruction before jumping back to the fixed location of bmips_secondary_reentry (in the current/old kernel) once interrupted. None of those are suitable for kexec.

In general the CPU hotplug code just needs that when play_dead() runs on the CPU going offline, and the struct plat_smp_ops cpu_die() callback runs on some other CPU, the CPU being offlined will be quiesced to a degree that it won't interfere with Linux. This is different to kexec where we need the CPU to stop in such a way that it's fine that we might overwrite the old kernel, and then be able to start it running the new kernel later. As described before, that means it either needs to halt or power down until the new kernel takes control of it or it needs to run some loop from some location that we know won't be overwritten.

play_dead() is simple & already exists, it just doesn't do quite what we need kexec to do outside of the one system you're looking at.

Quoted 3 lines

A 'nicer' implementation stopping non-boot CPUs from one CPU might be available, but handling code duplication of play_dead() and saving CPU states could be a problem.

Code that can be shared between play_dead() & kexec can still be shared if we have a different callback - the particular implementation of the two would just have both call a common function for the part that's common.

Quoted 2 lines

For now, no __flush_cache_all() in machine_kexec() + no marking offline in kexec_this_cpu() + scache invalidation in kexec_this_cpu() might be a way to go.

For the system you're interested in, sure that works. But as I've pointed out in my last few emails it does not work in general for other systems that we also need to support in the kernel.

Thanks,
Paul

#14 lore raw

Hi Paul,

Quoted 1 line

The problem is that the "only CPU0 will use reboot_code_buffer" part

isn't true because CPU0 might not be the one that writes to it.

[Dengcheng]: I didn't say CPU0 is always the one running machine_kexec(). Actually, in my testing, I intentionally did something like:

taskset -c 3 echo c > /proc/sysrq-trigger

And for "either halted or powered down", do you have any idea/plan for Octeon, Loognson and bmips?

Thanks,

Dengcheng

From: Paul Burton
Sent: Friday, September 7, 2018 2:42 PM
To: Dengcheng Zhu
Cc: Paul Burton; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH v4 0/6] MIPS: kexec/kdump: Fix smp reboot and other issues

Hi Dengcheng,

Quoted 4 lines · Dengcheng Zhu

On cache issue, what if kexec_this_cpu() does local i$ flush and scache invalidation? This is done only on CPU0, no IPI needed. And only CUP0 will use reboot_code_buffer. In this way, marking other CPUs offline to prevent being sent IPI is not needed.

The problem is that the "only CPU0 will use reboot_code_buffer" part isn't true because CPU0 might not be the one that writes to it. Though perhaps if we moved the memcpy() from machine_kexec() to kexec_this_cpu() that could work.

It still doesn't seem ideal to me that we'd have to even consider IPIs & the fact that we might silently hang if we accidentally send one. I really think it would be cleaner to mark CPUs offline & the problem is solved naturally - we shouldn't IPI these CPUs, and marking them offline means that we won't. Problem solved, and consistently with other architectures.

Quoted 4 lines

As to "the associated fragility of needing to avoid anything that might IPI them", I hope that's not a problem as the kexec sequence is determined: The last IPI received is to ask them to stand by on reboot signal, because other than kexec there is no other kernel code beyond our control, which might send IPIs.

But it does mean we can't call certain things, like __flush_cache_all(), which means that developers need to 1) know that and 2) remember to avoid any functions like this that perform an IPI behind the scenes. It just feels like loading a gun & leaving it pointed at our feet - by itself it's not necessarily going to cause a problem, but it would be oh so easy to trigger problems later.

Quoted 11 lines
Quoted 1 line

...but the new kernel will have no knowledge of whether the old kernel

had CPUs marked online or offline, so I don't follow the argument?

[Dengcheng] I meant the new kernel shouldn't need CPU online info from the old kernel. The new one is doing a "fresh" boot, provided CPUs 1+ have been halted.

Regarding the use of play_dead(), it's simple and generic - MIPS CPUs have implemented their own play_dead(), which we can use to stop execution. Other options may still come down to play_dead() to handle MT/MC details. The concern of running it in parallel is really about the online status, which has been explained above and in my last email.

But play_dead() really doesn't mean what you're suggesting it means.

What we want for kexec is that the CPU is stopped entirely - either halted or powered down. Failing that it needs to be running a loop from somewhere that we know isn't going to be overwritten, and the code it's running needs to be able to hand over to the new kernel later (eg. the existing kexec_smp_wait loop implements this latter approach).

Whilst the smp-cps.c implementation of play_dead() will currently halt or power down the CPU, that's not universally true & doesn't necessarily even need to remain true for smp-cps.c in the future (eg. we could decide the halt/power down step would be cleaner to do in cps_cpu_die).

A quick look at the other play_dead() implementations we have shows that both the cavium-octeon & loongson3 versions leave the CPU running a loop (within the current/old kernel), whilst the bmips version just sits at a wait instruction before jumping back to the fixed location of bmips_secondary_reentry (in the current/old kernel) once interrupted. None of those are suitable for kexec.

In general the CPU hotplug code just needs that when play_dead() runs on the CPU going offline, and the struct plat_smp_ops cpu_die() callback runs on some other CPU, the CPU being offlined will be quiesced to a degree that it won't interfere with Linux. This is different to kexec where we need the CPU to stop in such a way that it's fine that we might overwrite the old kernel, and then be able to start it running the new kernel later. As described before, that means it either needs to halt or power down until the new kernel takes control of it or it needs to run some loop from some location that we know won't be overwritten.

play_dead() is simple & already exists, it just doesn't do quite what we need kexec to do outside of the one system you're looking at.

Quoted 3 lines

A 'nicer' implementation stopping non-boot CPUs from one CPU might be available, but handling code duplication of play_dead() and saving CPU states could be a problem.

Code that can be shared between play_dead() & kexec can still be shared if we have a different callback - the particular implementation of the two would just have both call a common function for the part that's common.

Quoted 2 lines

For now, no __flush_cache_all() in machine_kexec() + no marking offline in kexec_this_cpu() + scache invalidation in kexec_this_cpu() might be a way to go.

For the system you're interested in, sure that works. But as I've pointed out in my last few emails it does not work in general for other systems that we also need to support in the kernel.

Thanks,
    Paul

#15 lore raw
Quoted 10 lines · Dengcheng Zhu

Hi Paul,

Quoted 1 line

The problem is that the "only CPU0 will use reboot_code_buffer" part

isn't true because CPU0 might not be the one that writes to it.

[Dengcheng]: I didn't say CPU0 is always the one running machine_kexec(). Actually, in my testing, I intentionally did something like:

taskset -c 3 echo c > /proc/sysrq-trigger

Well, if CPU 0 isn't running machine_kexec() then some other CPU is doing the write to reboot_code_buffer and therefore CPU 0 isn't the only one accessing/using it.

As I walked through in an earlier email, the safe way for this to happen is for the machine_kexec() CPU to write back its dcache as far as needed for a remote CPU's icache to observe the stores. Having CPU 0 be the only one to do any cache maintenance will not achieve that.

As I mentioned before, if you're testing on I6x00 then you probably got away with it because the icache fills from the dcache. You'd probably also get away with having CPU 0 be the only one to do cache maintenance since the CM globalizes hit cache ops. But not all MIPS CPUs would be so lucky.

Quoted 2 lines

And for "either halted or powered down", do you have any idea/plan for Octeon, Loognson and bmips?

Well like I said before if we add a callback to struct plat_smp_ops to stop other CPUs, at least in the crash case, then we can check that & refuse to kexec() on systems that haven't implemented it.

Octeon right now has kexec, so any changes like this should bring Octeon along for the ride by implementing the new callback there. Loongson has a patch pending [1] so that should probably be taken into account too. bmips would just be left alone, and wouldn't support kexec until someone implements the new callback.

[1] https://patchwork.linux-mips.org/patch/20432/

Thanks,
Paul

#16 lore raw
Quoted 1 line

Well, if CPU 0 isn't running machine_kexec() then some other CPU is

doing the write to reboot_code_buffer and therefore CPU 0 isn't the only one accessing/using it.

[Dengcheng]: I meant only CPU0 will jump to reboot_code_buffer. This buffer is surely possibly written by another CPU. Any issue with the machine_kexec() CPU flush "local" dache, and CPU0 invalidate "local" icache and scahe before jumping?

Thanks,

Dengcheng

From: Paul Burton
Sent: Friday, September 7, 2018 4:11 PM
To: Dengcheng Zhu
Cc: Paul Burton; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH v4 0/6] MIPS: kexec/kdump: Fix smp reboot and other issues

Quoted 10 lines · Dengcheng Zhu

Hi Paul,

Quoted 1 line

The problem is that the "only CPU0 will use reboot_code_buffer" part

isn't true because CPU0 might not be the one that writes to it.

[Dengcheng]: I didn't say CPU0 is always the one running machine_kexec(). Actually, in my testing, I intentionally did something like:

taskset -c 3 echo c > /proc/sysrq-trigger

Well, if CPU 0 isn't running machine_kexec() then some other CPU is doing the write to reboot_code_buffer and therefore CPU 0 isn't the only one accessing/using it.

As I walked through in an earlier email, the safe way for this to happen is for the machine_kexec() CPU to write back its dcache as far as needed for a remote CPU's icache to observe the stores. Having CPU 0 be the only one to do any cache maintenance will not achieve that.

As I mentioned before, if you're testing on I6x00 then you probably got away with it because the icache fills from the dcache. You'd probably also get away with having CPU 0 be the only one to do cache maintenance since the CM globalizes hit cache ops. But not all MIPS CPUs would be so lucky.

Quoted 2 lines

And for "either halted or powered down", do you have any idea/plan for Octeon, Loognson and bmips?

Well like I said before if we add a callback to struct plat_smp_ops to stop other CPUs, at least in the crash case, then we can check that & refuse to kexec() on systems that haven't implemented it.

Octeon right now has kexec, so any changes like this should bring Octeon along for the ride by implementing the new callback there. Loongson has a patch pending [1] so that should probably be taken into account too. bmips would just be left alone, and wouldn't support kexec until someone implements the new callback.

[1] https://patchwork.linux-mips.org/patch/20432/

Thanks,
    Paul