MIPS: kexec/kdump: Fix smp reboot and other issues
Subsystems (4)
v4 cover MIPS: kexec/kdump: Fix smp reboot and other issues
The issues are mentioned in patches 1/4/5/6. I will update kdump documentation for MIPS if the series gets accepted. Testing has been done on single core i6500/Boston with IOCU, dual core i6500 without IOCU, and dual core interAptiv without IOCU.
Changes:
v4 - v3:
* In patch #1, idle_task_exit() is moved out from play_dead() to its sole caller arch_cpu_idle_dead(). So no interface change of play_dead().
* In patch #6, the kexec_prepare method for the Generic platform is defined as uhi_machine_kexec_prepare() for all platforms using UHI boot protocol.
v3 - v2:
* Code style changes according to scripts/checkpatch.pl --strict. Patch #6, like before, still has a warning message reminding if MAINTAINERS needs updating. But it does NOT involve a maintainer change.
* Add LIBFDT to CPU_LOONGSON3 for default_machine_kexec_prepare().
v2 - v1:
* Tested on MIPS32R2 platform in addition to MIPS64R6.
* Added patches #5 and #6.
* In patch #2, removed the unnecessary inclusion of asm/mipsmtregs.h
Dengcheng Zhu (6):
MIPS: Make play_dead() work for kexec
MIPS: kexec: Let the new kernel handle all CPUs
MIPS: kexec: Deprecate (relocated_)kexec_smp_wait
MIPS: kexec: Do not flush system wide caches in machine_kexec()
MIPS: kexec: Relax memory restriction
MIPS: kexec: Use prepare method from Generic for UHI platforms
arch/mips/Kconfig | 4 ++ arch/mips/cavium-octeon/setup.c | 2 +- arch/mips/cavium-octeon/smp.c | 34 ++++++----- arch/mips/generic/Makefile | 1 - arch/mips/generic/kexec.c | 44 -------------- arch/mips/include/asm/kexec.h | 11 ++-- arch/mips/include/asm/smp.h | 4 +- arch/mips/kernel/crash.c | 4 +- arch/mips/kernel/machine_kexec.c | 82 +++++++++++++++++++++++---- arch/mips/kernel/process.c | 2 + arch/mips/kernel/relocate_kernel.S | 39 ------------- arch/mips/kernel/smp-bmips.c | 8 ++- arch/mips/kernel/smp-cps.c | 49 +++++++++------- arch/mips/loongson64/loongson-3/smp.c | 82 ++++++++++++++------------- 14 files changed, 183 insertions(+), 183 deletions(-) delete mode 100644 arch/mips/generic/kexec.c
Signature · 1 line
2.17.1
PATCH v4 2/6 MIPS: kexec: Let the new kernel handle all CPUs
Use play_dead() to prepare an environment for the new kernel. Do not rely on non-crashing CPUs jumping to kexec_start_address.
---
Changes:
* Use play_dead() without an extra parameter.
arch/mips/cavium-octeon/setup.c | 2 +- arch/mips/include/asm/kexec.h | 3 ++- arch/mips/kernel/crash.c | 4 +++- arch/mips/kernel/machine_kexec.c | 28 ++++++++++++++++++++++++---- 4 files changed, 30 insertions(+), 7 deletions(-)
arch/mips/cavium-octeon/setup.c +1 -1
@@ -95,7 +95,7 @@ static void octeon_kexec_smp_down(void *ignored)" sync \n" " synci ($0) \n"); - relocated_kexec_smp_wait(NULL); + kexec_smp_reboot(); }#endif
arch/mips/include/asm/kexec.h +2 -1
@@ -39,12 +39,13 @@ extern unsigned long kexec_args[4];extern int (*_machine_kexec_prepare)(struct kimage *);extern void (*_machine_kexec_shutdown)(void );extern void (*_machine_crash_shutdown)(struct pt_regs *regs); -extern void default_machine_crash_shutdown (struct pt_regs *regs); +void default_machine_crash_shutdown (struct pt_regs *regs);#ifdef CONFIG_SMP extern const unsigned char kexec_smp_wait[];extern unsigned long secondary_kexec_args[4 ];extern void (*relocated_kexec_smp_wait) (void *);extern atomic_t kexec_ready_to_reboot; +void kexec_smp_reboot (void );extern void (*_crash_smp_send_stop)(void );#endif #endif
arch/mips/kernel/crash.c +3 -1
@@ -43,7 +43,9 @@ static void crash_shutdown_secondary(void *passed_regs)while (!atomic_read (&kexec_ready_to_reboot))cpu_relax (); -relocated_kexec_smp_wait (NULL ); + +kexec_smp_reboot (); +/* NOTREACHED */ }
arch/mips/kernel/machine_kexec.c +24 -4
@@ -19,6 +19,10 @@ extern const size_t relocate_new_kernel_size;extern unsigned long kexec_start_address;extern unsigned long kexec_indirection_page; +static unsigned long reboot_code_buffer; + +typedef void (*noretfun_t )(void )__noreturn ; +int (*_machine_kexec_prepare)(struct kimage *) =NULL ;void (*_machine_kexec_shutdown)(void ) =NULL ;void (*_machine_crash_shutdown)(struct pt_regs *regs) =NULL ;
@@ -26,6 +30,20 @@ void (*_machine_crash_shutdown)(struct pt_regs *regs) = NULL;void (*relocated_kexec_smp_wait) (void *);atomic_t kexec_ready_to_reboot =ATOMIC_INIT (0 );void (*_crash_smp_send_stop)(void ) =NULL ; + +void kexec_smp_reboot (void ) +{ +if (smp_processor_id () >0 ) { +/* +* Instead of cpu_relax() or wait, this is needed for kexec +* smp reboot. Kdump usually doesn't require an smp new +* kernel, but kexec may do. +*/ +play_dead (); + }else { + ((noretfun_t )reboot_code_buffer)(); + } +}#endif static void kexec_image_info (const struct kimage *kimage)
@@ -79,12 +97,9 @@ machine_crash_shutdown(struct pt_regs *regs)default_machine_crash_shutdown (regs); } -typedef void (*noretfun_t )(void )__noreturn ; -void machine_kexec (struct kimage *image) { -unsigned long reboot_code_buffer;unsigned long entry;unsigned long *ptr;
@@ -132,6 +147,11 @@ machine_kexec(struct kimage *image) (void *)(kexec_smp_wait - relocate_new_kernel);smp_wmb ();atomic_set (&kexec_ready_to_reboot,1 ); -#endif + +kexec_smp_reboot (); + +/* NOT REACHED */ +#else ((noretfun_t ) reboot_code_buffer)(); +#endif }
Signature · 1 line
2.17.1
PATCH v4 3/6 MIPS: kexec: Deprecate (relocated_)kexec_smp_wait
In reply to Dengcheng ZhuSince we've changed the SMP boot mechanism (by having the new kernel handle all CPUs), now remove (relocated_)kexec_smp_wait.
--- arch/mips/include/asm/kexec.h | 2 -- arch/mips/kernel/machine_kexec.c | 5 ---- arch/mips/kernel/relocate_kernel.S | 39 ------------------------------ 3 files changed, 46 deletions(-)
arch/mips/include/asm/kexec.h +0 -2
@@ -41,9 +41,7 @@ extern void (*_machine_kexec_shutdown)(void);extern void (*_machine_crash_shutdown)(struct pt_regs *regs);void default_machine_crash_shutdown (struct pt_regs *regs);#ifdef CONFIG_SMP -extern const unsigned char kexec_smp_wait[];extern unsigned long secondary_kexec_args[4 ]; -extern void (*relocated_kexec_smp_wait) (void *);extern atomic_t kexec_ready_to_reboot;void kexec_smp_reboot (void );extern void (*_crash_smp_send_stop)(void );
arch/mips/kernel/machine_kexec.c +0 -5
@@ -27,7 +27,6 @@ int (*_machine_kexec_prepare)(struct kimage *) = NULL;void (*_machine_kexec_shutdown)(void ) =NULL ;void (*_machine_crash_shutdown)(struct pt_regs *regs) =NULL ;#ifdef CONFIG_SMP -void (*relocated_kexec_smp_wait) (void *);atomic_t kexec_ready_to_reboot =ATOMIC_INIT (0 );void (*_crash_smp_send_stop)(void ) =NULL ;
@@ -142,10 +141,6 @@ machine_kexec(struct kimage *image)printk ("Bye ...\n" );__flush_cache_all ();#ifdef CONFIG_SMP -/* All secondary cpus now may jump to kexec_wait cycle */ - relocated_kexec_smp_wait = reboot_code_buffer + - (void *)(kexec_smp_wait - relocate_new_kernel); -smp_wmb ();atomic_set (&kexec_ready_to_reboot,1 );kexec_smp_reboot ();
arch/mips/kernel/relocate_kernel.S +0 -39
@@ -100,45 +100,6 @@ done: j s1 END(relocate_new_kernel) -#ifdef CONFIG_SMP -/* - * Other CPUs should wait until code is relocated and - * then start at entry (?) point. - */ -LEAF(kexec_smp_wait) - PTR_L a0, s_arg0 - PTR_L a1, s_arg1 - PTR_L a2, s_arg2 - PTR_L a3, s_arg3 - PTR_L s1, kexec_start_address - - /* Non-relocated address works for args and kexec_start_address ( old - * kernel is not overwritten). But we need relocated address of - * kexec_flag. - */ - - bal 1f -1: move t1,ra; - PTR_LA t2,1b - PTR_LA t0,kexec_flag - PTR_SUB t0,t0,t2; - PTR_ADD t0,t1,t0; - -1: LONG_L s0, (t0) - bne s0, zero,1b - -#ifdef CONFIG_CPU_CAVIUM_OCTEON - .set push - .set noreorder - synci 0($0) - .set pop -#else - sync -#endif - j s1 - END(kexec_smp_wait) -#endif - #ifdef __mips64 /* all PTR's must be aligned to 8 byte in 64-bit mode */ .align 3
Signature · 1 line
2.17.1
PATCH v4 1/6 MIPS: Make play_dead() work for kexec
In reply to Dengcheng ZhuExtract play_dead() from CONFIG_HOTPLUG_CPU and share with CONFIG_KEXEC. Meanwhile, move idle_task_exit() out to arch_cpu_idle_dead() because it's not meant for kexec.
This is needed to correctly support SMP new kernel in kexec. Before doing this, all non-crashing CPUs are waiting for the reboot signal (kexec_ready_to_reboot) to jump to kexec_start_address in kexec_smp_wait(), which creates some problems like incorrect CPU topology upon booting from new UP kernel, sluggish performance in MT environment during and after reboot, new SMP kernel not able to bring up secondary CPU etc.
It would make sense to let the new (SMP) kernel manage all CPUs, instead of crashing CPU booting from reboot_code_buffer whereas others jumping to kexec_start_address. To do this, play_dead() is well suitable for preparing a boot environment for the new kernel.
---
Changes:
* idle_task_exit() is moved out from play_dead() to its sole caller arch_cpu_idle_dead(). So no interface change of play_dead().
arch/mips/cavium-octeon/smp.c | 34 ++++++----- arch/mips/include/asm/smp.h | 4 +- arch/mips/kernel/process.c | 2 + arch/mips/kernel/smp-bmips.c | 8 ++- arch/mips/kernel/smp-cps.c | 49 +++++++++------- arch/mips/loongson64/loongson-3/smp.c | 82 ++++++++++++++------------- 6 files changed, 99 insertions(+), 80 deletions(-)
arch/mips/cavium-octeon/smp.c +19 -15
@@ -280,11 +280,29 @@ static void octeon_smp_finish(void)local_irq_enable (); } -#ifdef CONFIG_HOTPLUG_CPU +#if defined(CONFIG_HOTPLUG_CPU ) || defined(CONFIG_KEXEC )/* State of each CPU. */ DEFINE_PER_CPU (int, cpu_state); +void play_dead (void ) +{ +int cpu =cpu_number_map (cvmx_get_core_num ()); + + octeon_processor_boot =0xff ; +per_cpu (cpu_state, cpu) =CPU_DEAD ; + +/* make the change visible before going off */ +mb (); + +while (1 )/* core will be reset here */ + ; +} + +#endif /* CONFIG_HOTPLUG_CPU || CONFIG_KEXEC */ + +#ifdef CONFIG_HOTPLUG_CPU +static int octeon_cpu_disable (void ) {unsigned int cpu =smp_processor_id ();
@@ -343,20 +361,6 @@ static void octeon_cpu_die(unsigned int cpu)cvmx_write_csr (CVMX_CIU_PP_RST ,0 ); } -void play_dead (void ) -{ -int cpu =cpu_number_map (cvmx_get_core_num ()); - -idle_task_exit (); - octeon_processor_boot =0xff ; -per_cpu (cpu_state, cpu) =CPU_DEAD ; - -mb (); - -while (1 )/* core will be reset here */ - ; -} -static void start_after_reset (void ) {kernel_entry (0 ,0 ,0 );/* set a2 = 0 for secondary core */
arch/mips/include/asm/smp.h +3 -1
@@ -77,8 +77,10 @@ static inline void __cpu_die(unsigned int cpu) mp_ops->cpu_die(cpu); } +#endif -extern void play_dead (void ); +#if defined(CONFIG_HOTPLUG_CPU ) || defined(CONFIG_KEXEC ) +void play_dead (void );#endif /*
arch/mips/kernel/process.c +2 -0
@@ -14,6 +14,7 @@#include <linux/sched/debug.h> #include <linux/sched/task.h> #include <linux/sched/task_stack.h> +#include <linux/sched/hotplug.h> #include <linux/tick.h> #include <linux/kernel.h> #include <linux/mm.h>
@@ -53,6 +54,7 @@#ifdef CONFIG_HOTPLUG_CPU void arch_cpu_idle_dead (void ) { +idle_task_exit ();play_dead (); }#endif
arch/mips/kernel/smp-bmips.c +5 -3
@@ -381,10 +381,12 @@ static void bmips_cpu_die(unsigned int cpu) { } +#endif /* CONFIG_HOTPLUG_CPU */ + +#if defined(CONFIG_HOTPLUG_CPU ) || defined(CONFIG_KEXEC ) +void __ref play_dead (void ) { -idle_task_exit (); -/* flush data cache */ _dma_cache_wback_inv (0 , ~0 );
@@ -409,7 +411,7 @@ void __ref play_dead(void) : : : "memory"); } -#endif /* CONFIG_HOTPLUG_CPU */ +#endif /* CONFIG_HOTPLUG_CPU || CONFIG_KEXEC */ const struct plat_smp_ops bmips43xx_smp_ops = { .smp_setup = bmips_smp_setup,
arch/mips/kernel/smp-cps.c +27 -22
67 lines not on this page · raw message
arch/mips/loongson64/loongson-3/smp.c +43 -39
100 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v4 4/6 MIPS: kexec: Do not flush system wide caches in machine_kexec()
In reply to Dengcheng ZhuInstead of __flush_cache_all(), simply flush local icache range. In systems without IOCU, flushing system wide caches require sending IPIs. But other CPUs have disabled local IRQs waiting for the reboot signal. It will then cause system hang.
This patch fixes this problem.
--- arch/mips/kernel/machine_kexec.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
arch/mips/kernel/machine_kexec.c +10 -1
@@ -139,7 +139,16 @@ machine_kexec(struct kimage *image)printk ("Will call new kernel at %08lx\n" , image->start);printk ("Bye ...\n" ); -__flush_cache_all (); +/* +* __flush_cache_all() is expensive but unnecessary. More +* importantly, it could freeze the system as it may need to send +* IPIs, whereas other CPUs have been waiting for the reboot signal +* (kexec_ready_to_reboot) with local irqs disabled, because +* machine_crash_shutdown() has been called prior to entering +* this function - machine_kexec(). +*/ +local_flush_icache_range (reboot_code_buffer, + reboot_code_buffer + relocate_new_kernel_size);#ifdef CONFIG_SMP atomic_set (&kexec_ready_to_reboot,1 );
Signature · 1 line
2.17.1
PATCH v4 5/6 MIPS: kexec: Relax memory restriction
In reply to Dengcheng ZhuWe can rely on the system kernel and the dump capture kernel themselves in memory usage.
Being restrictive with 512MB limit may cause kexec tool failure on some platforms.
--- arch/mips/include/asm/kexec.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/mips/include/asm/kexec.h +3 -3
@@ -12,11 +12,11 @@#include <asm/stacktrace.h> /* Maximum physical address we can use pages from */ -#define KEXEC_SOURCE_MEMORY_LIMIT (0x20000000) +#define KEXEC_SOURCE_MEMORY_LIMIT (-1UL)/* Maximum address we can reach in physical address mode */ -#define KEXEC_DESTINATION_MEMORY_LIMIT (0x20000000) +#define KEXEC_DESTINATION_MEMORY_LIMIT (-1UL)/* Maximum address we can use for the control code buffer */ -#define KEXEC_CONTROL_MEMORY_LIMIT (0x20000000) +#define KEXEC_CONTROL_MEMORY_LIMIT (-1UL)/* Reserve 3*4096 bytes for board-specific info */ #define KEXEC_CONTROL_PAGE_SIZE (4096 + 3*4096)
Signature · 1 line
2.17.1
PATCH v4 6/6 MIPS: kexec: Use prepare method from Generic for UHI platforms
In reply to Dengcheng ZhuThe kexec_prepare method for the Generic platform should be applicable to others using UHI boot protocol. In arch/mips/Kconfig, "select UHI_BOOT" is needed for such platforms.
If the platform has its own method for _machine_kexec_prepare(), that one will be used.
---
Changes:
* The kexec_prepare method for the Generic platform is defined as uhi_machine_kexec_prepare() for all platforms using UHI boot protocol.
arch/mips/Kconfig | 4 +++ arch/mips/generic/Makefile | 1 - arch/mips/generic/kexec.c | 44 -------------------------------- arch/mips/kernel/machine_kexec.c | 38 ++++++++++++++++++++++++++- 4 files changed, 41 insertions(+), 46 deletions(-) delete mode 100644 arch/mips/generic/kexec.c
arch/mips/Kconfig +4 -0
@@ -129,6 +129,7 @@ config MIPS_GENERIC select USB_UHCI_BIG_ENDIAN_DESC if CPU_BIG_ENDIAN select USB_UHCI_BIG_ENDIAN_MMIO if CPU_BIG_ENDIAN select USE_OF + select UHI_BOOT help Select this to build a kernel which aims to support multiple boards, generally using a flattened device tree passed from the bootloader
@@ -2907,6 +2908,9 @@ config USE_OF select OF_EARLY_FLATTREE select IRQ_DOMAIN +config UHI_BOOT + bool + config BUILTIN_DTB bool
arch/mips/generic/Makefile +0 -1
@@ -15,5 +15,4 @@ obj-y += proc.o obj-$(CONFIG_YAMON_DT_SHIM) += yamon-dt.o obj-$(CONFIG_LEGACY_BOARD_SEAD3) += board-sead3.o obj-$(CONFIG_LEGACY_BOARD_OCELOT) += board-ocelot.o -obj-$(CONFIG_KEXEC) += kexec.o obj-$(CONFIG_VIRT_BOARD_RANCHU) += board-ranchu.o
arch/mips/generic/kexec.c deleted +0 -44
44 lines not on this page · raw message
arch/mips/kernel/machine_kexec.c +37 -1
@@ -9,6 +9,7 @@#include <linux/kexec.h> #include <linux/mm.h> #include <linux/delay.h> +#include <linux/libfdt.h> #include <asm/cacheflush.h> #include <asm/page.h>
@@ -23,7 +24,6 @@ static unsigned long reboot_code_buffer;typedef void (*noretfun_t )(void )__noreturn ; -int (*_machine_kexec_prepare)(struct kimage *) =NULL ;void (*_machine_kexec_shutdown)(void ) =NULL ;void (*_machine_crash_shutdown)(struct pt_regs *regs) =NULL ;#ifdef CONFIG_SMP
@@ -65,6 +65,42 @@ static void kexec_image_info(const struct kimage *kimage) } } +#ifdef CONFIG_UHI_BOOT +static int uhi_machine_kexec_prepare (struct kimage *kimage) +{ +int i; + +/* +* In case DTB file is not passed to the new kernel, a flat device +* tree will be created by kexec tool. It holds modified command +* line for the new kernel. +*/ +for (i =0 ; i < kimage->nr_segments; i++) { +struct fdt_header fdt; + +if (kimage->segment[i].memsz <=sizeof (fdt)) +continue ; + +if (copy_from_user (&fdt, kimage->segment[i].buf,sizeof (fdt))) +continue ; + +if (fdt_check_header (&fdt)) +continue ; + + kexec_args[0 ] =-2 ; + kexec_args[1 ] = (unsigned long ) +phys_to_virt ((unsigned long )kimage->segment[i].mem); +break ; + } + +return 0 ; +} + +int (*_machine_kexec_prepare)(struct kimage *) = uhi_machine_kexec_prepare; +#else +int (*_machine_kexec_prepare)(struct kimage *) =NULL ; +#endif /* CONFIG_UHI_BOOT */ +int machine_kexec_prepare (struct kimage *kimage) {
Signature · 1 line
2.17.1
Hi Dengcheng,
Quoted 5 lines · Dengcheng Zhu
The issues are mentioned in patches 1/4/5/6. I will update kdump documentation for MIPS if the series gets accepted. Testing has been done on single core i6500/Boston with IOCU, dual core i6500 without IOCU, and dual core interAptiv without IOCU.
Quoted 7 lines
Changes:
v4 - v3:
* In patch #1,idle_task_exit()is moved out fromplay_dead()to its sole callerarch_cpu_idle_dead(). So no interface change ofplay_dead().
* In patch #6, thekexec_preparemethod for the Generic platform is defined asuhi_machine_kexec_prepare()for all platforms using UHI boot protocol.
Thanks! I've applied patches 1,2,3,5 to a test branch with a few changes:
git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux.git test-kexec
https://git.kernel.org/pub/scm/linux/kernel/git/mips/linux.git/log/?h=test-kexec
I didn't apply patch 4 because I'm not sure it's correct & I believe the changes in the branch above should take care of it - CPUs that reach kexec_this_cpu() are maked offline, so they shouldn't be IPI'd by __flush_cache_all(). The CPU that runs machine_kexec() should still flush its dcache (& the L2), and then CPU 0 invalidates its icache in kexec_this_cpu() prior to jumping into reboot_code_buffer.
I'm also still not sure about patch 6 - since no platforms besides the arch/mips/generic/ make use of the UHI boot code yet I think it'd be best to leave as-is. If we do ever need to use it from another platform then we can deal with the problem then. If an out of tree platform needs to use it then for now it could copy generic_kexec_prepare() and deal with removing the duplication when it heads upstream.
Could you take a look & let me know if you see any problems?
Thanks,
Paul
Hi Paul,
Quoted 18 lines · Paul Burton
Hi Dengcheng,
Quoted 5 lines · Dengcheng Zhu
The issues are mentioned in patches 1/4/5/6. I will update kdump documentation for MIPS if the series gets accepted. Testing has been done on single core i6500/Boston with IOCU, dual core i6500 without IOCU, and dual core interAptiv without IOCU.
Quoted 7 lines
Changes:
v4 - v3:
* In patch #1,idle_task_exit()is moved out fromplay_dead()to its sole callerarch_cpu_idle_dead(). So no interface change ofplay_dead().
* In patch #6, thekexec_preparemethod for the Generic platform is defined asuhi_machine_kexec_prepare()for all platforms using UHI boot protocol.Thanks! I've applied patches 1,2,3,5 to a test branch with a few changes:
Quoted 8 lines
git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux.gittest-kexechttps://git.kernel.org/pub/scm/linux/kernel/git/mips/linux.git/log/?h=test-kexec
I didn't apply patch 4 because I'm not sure it's correct & I believe the changes in the branch above should take care of it - CPUs that reach
kexec_this_cpu()are maked offline, so they shouldn't be IPI'd by__flush_cache_all().
I believe patch 4 is necessary. As mentioned in the code comment and patch comment of that patch, machine_crash_shutdown() is called prior to machine_kexec() in the kdump sequence. So other CPUs have disabled local IRQs waiting for the reboot signal.
In fact, in kexec_this_cpu() [you renamed and modified kexec_smp_reboot()], the added marking CPU offline will cause system hang (tested). This is because it will change how play_dead() will work.
Quoted 10 lines
The CPU that runs
machine_kexec()should still
flush its dcache (& the L2), and then CPU 0 invalidates its icache inkexec_this_cpu()prior to jumping intoreboot_code_buffer.I'm also still not sure about patch 6 - since no platforms besides the
arch/mips/generic/make use of the UHI boot code yet I think it'd be best to leave as-is. If we do ever need to use it from another platform then we can deal with the problem then. If an out of tree platform needs to use it then for now it could copygeneric_kexec_prepare()and deal with removing the duplication when it heads upstream.
Understood. It really depends on how this problem is viewed. If patch #6 is considered creating a framework for future UHI platforms, then it has the following facts:
* It doesn't create code redundancy. I mean, it does not add unnecessary code to the kernel.
* Out of tree platforms will get access to this functionality by a simple "select UHI_BOOT". When the kernel developer of an out-of-tree platform wants to make kexec work, they will naturally look at machine_kexec.c, where they will find this UHI stuff, obviously telling them to do "select UHI_BOOT". Otherwise, unless they google onto this discussion thread, it's harder to know the solution to the kexec_args related problem hides in the code of another platform (Generic).
* It simplifies work if the out of tree platform wants to upstream.
Quoted 5 lines
Could you take a look & let me know if you see any problems?
Thanks,
Paul
Thanks,
Dengcheng
---------------------------------------------------------------------------
*From:* Paul Burton <mailto:[email protected]>
*Sent:* Wednesday, September 05, 2018 3:54PM
*To:* Dengcheng Zhu <mailto:[email protected]>
*Cc:* Pburton <mailto:[email protected]>, Ralf <mailto:[email protected]>, Linux-mips <mailto:[email protected]>, Rachel.mozes <mailto:[email protected]> *Subject:* Re: [PATCH v4 0/6] MIPS: kexec/kdump: Fix smp reboot and other issues
Hi Dengcheng,
Quoted 5 lines · Dengcheng Zhu
The issues are mentioned in patches 1/4/5/6. I will update kdump documentation for MIPS if the series gets accepted. Testing has been done on single core i6500/Boston with IOCU, dual core i6500 without IOCU, and dual core interAptiv without IOCU.
Quoted 7 lines
Changes:
v4 - v3:
* In patch #1,idle_task_exit()is moved out fromplay_dead()to its sole callerarch_cpu_idle_dead(). So no interface change ofplay_dead().
* In patch #6, thekexec_preparemethod for the Generic platform is defined asuhi_machine_kexec_prepare()for all platforms using UHI boot protocol.
Thanks! I've applied patches 1,2,3,5 to a test branch with a few changes:
git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux.git test-kexec
https://git.kernel.org/pub/scm/linux/kernel/git/mips/linux.git/log/?h=test-kexec
I didn't apply patch 4 because I'm not sure it's correct & I believe the changes in the branch above should take care of it - CPUs that reach kexec_this_cpu() are maked offline, so they shouldn't be IPI'd by __flush_cache_all(). The CPU that runs machine_kexec() should still flush its dcache (& the L2), and then CPU 0 invalidates its icache in kexec_this_cpu() prior to jumping into reboot_code_buffer.
I'm also still not sure about patch 6 - since no platforms besides the arch/mips/generic/ make use of the UHI boot code yet I think it'd be best to leave as-is. If we do ever need to use it from another platform then we can deal with the problem then. If an out of tree platform needs to use it then for now it could copy generic_kexec_prepare() and deal with removing the duplication when it heads upstream.
Could you take a look & let me know if you see any problems?
Thanks,
Paul
Hi Dengcheng,
Quoted 5 lines · Dengcheng Zhu
Quoted 4 lines
I didn't apply patch 4 because I'm not sure it's correct & I believe the changes in the branch above should take care of it - CPUs that reach
kexec_this_cpu()are maked offline, so they shouldn't be IPI'd by__flush_cache_all().
Quoted 8 lines
I believe patch 4 is necessary. As mentioned in the code comment and patch comment of that patch,
machine_crash_shutdown()is called prior tomachine_kexec()in the kdump sequence. So other CPUs have disabled local IRQs waiting for the reboot signal.In fact, in
kexec_this_cpu()[you renamed and modifiedkexec_smp_reboot()], the added marking CPU offline will cause system hang (tested). This is because it will change howplay_dead()will work.
So the problem is that patch 4 isn't really right either, and I suspect the hang you mention probably shows a problem with the whole play_dead() approach from patches 1 & 2 too.
On the cache flushing, what we really need is for stores performed by the CPU running machine_kexec() via its dcache to be observed by the icache of the CPU that jumps into reboot_code_buffer, which is always CPU 0 after patch 2. Using local_flush_icache_range() will only ensure that the icache of the CPU running machine_kexec() observes the changes made by that same CPU, and does nothing with CPU 0 unless they happen to be the same CPU or siblings sharing caches.
Consider I6x00 CPUs for example - patch 4 may *almost* work OK because both cpu_has_ic_fills_f_dc & cpu_icache_snoops_remote_store are true, so when the machine_kexec() CPU writes to reboot_code_buffer CPU 0's icache will observe that & will fill from dcache without needing it to be written back to L2. It's still not quite right because if CPU 0's icache already contained any valid lines within reboot_code_buffer (which could happen speculatively) then it'll execute stale/garbage code.
The hang you mention is likely down to the fact that play_dead(), at least the smp-cps.c implementation of it, is written for the CPU hotplug infrastructure which will only operate on one CPU at a time. The loop looking for a sibling CPU just isn't going to be safe to run on multiple CPUs in parallel like patch 2 does. That's true of either your original patch or my modification - it's just that my modified patch will cause siblings to race towards powering off the core, whilst your original will cause them to all halt themselves & never power off the core.
Which inclines me to return to my earlier thought that perhaps we shouldn't use play_dead() for this at all.
Quoted 26 lines
Quoted 10 lines
The CPU that runs
machine_kexec()should still
flush its dcache (& the L2), and then CPU 0 invalidates its icache inkexec_this_cpu()prior to jumping intoreboot_code_buffer.I'm also still not sure about patch 6 - since no platforms besides the
arch/mips/generic/make use of the UHI boot code yet I think it'd be best to leave as-is. If we do ever need to use it from another platform then we can deal with the problem then. If an out of tree platform needs to use it then for now it could copygeneric_kexec_prepare()and deal with removing the duplication when it heads upstream.Understood. It really depends on how this problem is viewed. If patch #6 is considered creating a framework for future UHI platforms, then it has the following facts:
* It doesn't create code redundancy. I mean, it does not add unnecessary code to the kernel.
* Out of tree platforms will get access to this functionality by a simple "select
UHI_BOOT". When the kernel developer of an out-of-tree platform wants to make kexec work, they will naturally look atmachine_kexec.c, where they will find this UHI stuff, obviously telling them to do "selectUHI_BOOT". Otherwise, unless they google onto this discussion thread, it's harder to know the solution to thekexec_argsrelated problem hides in the code of another platform (Generic).
Quoted 1 line
* It simplifies work if the out of tree platform wants to upstream.
Well, ideally future platforms will just use arch/mips/generic rather than adding more platform/board code at all. Right now the only reason not to is if a system has a memory map that doesn't fit nicely, which should be resolved at some point by mapping the kernel which will allow the generic kernel to better support differing physical address maps.
So I hope we don't add more platform code anyway, which would make all this moot. I certainly don't want to encourage adding more by explicitly making it easier to do. And if there does come a point where we need to add more platform code for some good reason then we can deal with this at that point rather than speculating now.
For out of tree platforms, I don't think that copying the generic_kexec_prepare() function is particularly onerous anyway, and should be trivial to remove if such code is submitted upstream. Whilst I wouldn't want to make submitting code upstream more difficult, it's likely to need some amount of rework if submitted upstream anyway so this doesn't seem like a big deal.
I don't think upstream should be particularly concerned with making life easy for out of tree code - if one wants the benefit of their code being taken into account upstream, then one should submit one's code upstream. In the paraphrased words of my pastor at a marriage prep class - no benefits (except those already conferred by free access to open source software) without commitment.
Thanks,
Paul
Hi Paul,
Regarding cache flushing in machine_kexec(), I recommend simply removing __flush_cache_all() as CPU0 will do local icache flush before jumping to reboot_code_buffer, and other CPUs don't jump at all.
Re: marking CPU offline in kexec_this_cpu(), it's probably good NOT doing it either, as the system is going to reboot from CPU0 soon. HALT is good enough, no need to gate core power. As to whether it's safe running play_dead() in parallel, it shouldn't be a problem, as the loop is based on cpu online mask (which we don't mark offline as mentioned above) -- The CPU will simply halt itself. For non-MT CPUs, play_dead() does make sense as well, as it's supposed to stop this CPU's execution, getting ready to reboot from CPU0.
On UHI stuff, if it's true "future platforms will just use arch/mips/generic", then certainly no need of getting code out of Generic. My feeling is the customer is NOT doing so, according to "I fixed it by moving the code to another file". Other MIPS systems may be like this as well. I understand your "upstream-or-copy-code" point, but the only thing, which IMO meaningfully justifies patch 6 *for_such_cases*, is already said:
-----------------
When the kernel developer of an out-of-tree platform
wants to make kexec work, they will naturally look at machine_kexec.c, where they will find this UHI stuff, obviously telling them to do "select UHI_BOOT". Otherwise, unless they google onto this discussion thread, it's harder to know the solution to the kexec_args related problem hides in the code of another platform (Generic).
-----------------
In other words, it may take considerable efforts to realize "copying the generic_kexec_prepare() function" can solve the problem. Copying code itself is certainly not onerous.
Regards,
Dengcheng
Quoted 99 lines · Paul Burton
Hi Dengcheng,
Quoted 4 lines · Dengcheng Zhu
Quoted 4 lines
I didn't apply patch 4 because I'm not sure it's correct & I believe the changes in the branch above should take care of it - CPUs that reach
kexec_this_cpu()are maked offline, so they shouldn't be IPI'd by__flush_cache_all().Quoted 8 lines
I believe patch 4 is necessary. As mentioned in the code comment and patch comment of that patch,
machine_crash_shutdown()is called prior tomachine_kexec()in the kdump sequence. So other CPUs have disabled local IRQs waiting for the reboot signal.In fact, in
kexec_this_cpu()[you renamed and modifiedkexec_smp_reboot()], the added marking CPU offline will cause system hang (tested). This is because it will change howplay_dead()will work.So the problem is that patch 4 isn't really right either, and I suspect the hang you mention probably shows a problem with the whole
play_dead()approach from patches 1 & 2 too.On the cache flushing, what we really need is for stores performed by the CPU running
machine_kexec()via its dcache to be observed by the icache of the CPU that jumps intoreboot_code_buffer, which is always CPU 0 after patch 2. Usinglocal_flush_icache_range()will only ensure that the icache of the CPU runningmachine_kexec()observes the changes made by that same CPU, and does nothing with CPU 0 unless they happen to be the same CPU or siblings sharing caches.Consider I6x00 CPUs for example - patch 4 may *almost* work OK because both
cpu_has_ic_fills_f_dc&cpu_icache_snoops_remote_storeare true, so when themachine_kexec()CPU writes toreboot_code_bufferCPU 0's icache will observe that & will fill from dcache without needing it to be written back to L2. It's still not quite right because if CPU 0's icache already contained any valid lines withinreboot_code_buffer(which could happen speculatively) then it'll execute stale/garbage code.The hang you mention is likely down to the fact that
play_dead(), at least thesmp-cps.cimplementation of it, is written for the CPU hotplug infrastructure which will only operate on one CPU at a time. The loop looking for a sibling CPU just isn't going to be safe to run on multiple CPUs in parallel like patch 2 does. That's true of either your original patch or my modification - it's just that my modified patch will cause siblings to race towards powering off the core, whilst your original will cause them to all halt themselves & never power off the core.Which inclines me to return to my earlier thought that perhaps we shouldn't use
play_dead()for this at all.Quoted 25 lines
Quoted 10 lines
The CPU that runs
machine_kexec()should still
flush its dcache (& the L2), and then CPU 0 invalidates its icache inkexec_this_cpu()prior to jumping intoreboot_code_buffer.I'm also still not sure about patch 6 - since no platforms besides the
arch/mips/generic/make use of the UHI boot code yet I think it'd be best to leave as-is. If we do ever need to use it from another platform then we can deal with the problem then. If an out of tree platform needs to use it then for now it could copygeneric_kexec_prepare()and deal with removing the duplication when it heads upstream.Understood. It really depends on how this problem is viewed. If patch #6 is considered creating a framework for future UHI platforms, then it has the following facts:
* It doesn't create code redundancy. I mean, it does not add unnecessary code to the kernel.
* Out of tree platforms will get access to this functionality by a simple "select
UHI_BOOT". When the kernel developer of an out-of-tree platform wants to make kexec work, they will naturally look atmachine_kexec.c, where they will find this UHI stuff, obviously telling them to do "selectUHI_BOOT". Otherwise, unless they google onto this discussion thread, it's harder to know the solution to thekexec_argsrelated problem hides in the code of another platform (Generic).Quoted 1 line
* It simplifies work if the out of tree platform wants to upstream.
Well, ideally future platforms will just use
arch/mips/genericrather than adding more platform/board code at all. Right now the only reason not to is if a system has a memory map that doesn't fit nicely, which should be resolved at some point by mapping the kernel which will allow the generic kernel to better support differing physical address maps.So I hope we don't add more platform code anyway, which would make all this moot. I certainly don't want to encourage adding more by explicitly making it easier to do. And if there does come a point where we need to add more platform code for some good reason then we can deal with this at that point rather than speculating now.
For out of tree platforms, I don't think that copying the
generic_kexec_prepare()function is particularly onerous anyway, and should be trivial to remove if such code is submitted upstream. Whilst I wouldn't want to make submitting code upstream more difficult, it's likely to need some amount of rework if submitted upstream anyway so this doesn't seem like a big deal.I don't think upstream should be particularly concerned with making life easy for out of tree code - if one wants the benefit of their code being taken into account upstream, then one should submit one's code upstream. In the paraphrased words of my pastor at a marriage prep class - no benefits (except those already conferred by free access to open source software) without commitment.
Quoted 2 lines
Thanks,
Paul
Dengcheng
---------------------------------------------------------------------------
*From:* Paul Burton <mailto:[email protected]>
*Sent:* Thursday, September 06, 2018 1:34PM
*To:* Dengcheng Zhu <mailto:[email protected]>
*Cc:* Pburton <mailto:[email protected]>, Ralf <mailto:[email protected]>, Linux-mips <mailto:[email protected]>, Rachel.mozes <mailto:[email protected]> *Subject:* Re: [PATCH v4 0/6] MIPS: kexec/kdump: Fix smp reboot and other issues
Hi Dengcheng,
Quoted 4 lines · Dengcheng Zhu
Quoted 4 lines
I didn't apply patch 4 because I'm not sure it's correct & I believe the changes in the branch above should take care of it - CPUs that reach
kexec_this_cpu()are maked offline, so they shouldn't be IPI'd by__flush_cache_all().
Quoted 8 lines
I believe patch 4 is necessary. As mentioned in the code comment and patch comment of that patch,
machine_crash_shutdown()is called prior tomachine_kexec()in the kdump sequence. So other CPUs have disabled local IRQs waiting for the reboot signal.In fact, in
kexec_this_cpu()[you renamed and modifiedkexec_smp_reboot()], the added marking CPU offline will cause system hang (tested). This is because it will change howplay_dead()will work.
So the problem is that patch 4 isn't really right either, and I suspect the hang you mention probably shows a problem with the whole play_dead() approach from patches 1 & 2 too.
On the cache flushing, what we really need is for stores performed by the CPU running machine_kexec() via its dcache to be observed by the icache of the CPU that jumps into reboot_code_buffer, which is always CPU 0 after patch 2. Using local_flush_icache_range() will only ensure that the icache of the CPU running machine_kexec() observes the changes made by that same CPU, and does nothing with CPU 0 unless they happen to be the same CPU or siblings sharing caches.
Consider I6x00 CPUs for example - patch 4 may *almost* work OK because both cpu_has_ic_fills_f_dc & cpu_icache_snoops_remote_store are true, so when the machine_kexec() CPU writes to reboot_code_buffer CPU 0's icache will observe that & will fill from dcache without needing it to be written back to L2. It's still not quite right because if CPU 0's icache already contained any valid lines within reboot_code_buffer (which could happen speculatively) then it'll execute stale/garbage code.
The hang you mention is likely down to the fact that play_dead(), at least the smp-cps.c implementation of it, is written for the CPU hotplug infrastructure which will only operate on one CPU at a time. The loop looking for a sibling CPU just isn't going to be safe to run on multiple CPUs in parallel like patch 2 does. That's true of either your original patch or my modification - it's just that my modified patch will cause siblings to race towards powering off the core, whilst your original will cause them to all halt themselves & never power off the core.
Which inclines me to return to my earlier thought that perhaps we shouldn't use play_dead() for this at all.
Quoted 25 lines
Quoted 10 lines
The CPU that runs
machine_kexec()should still
flush its dcache (& the L2), and then CPU 0 invalidates its icache inkexec_this_cpu()prior to jumping intoreboot_code_buffer.I'm also still not sure about patch 6 - since no platforms besides the
arch/mips/generic/make use of the UHI boot code yet I think it'd be best to leave as-is. If we do ever need to use it from another platform then we can deal with the problem then. If an out of tree platform needs to use it then for now it could copygeneric_kexec_prepare()and deal with removing the duplication when it heads upstream.Understood. It really depends on how this problem is viewed. If patch #6 is considered creating a framework for future UHI platforms, then it has the following facts:
* It doesn't create code redundancy. I mean, it does not add unnecessary code to the kernel.
* Out of tree platforms will get access to this functionality by a simple "select
UHI_BOOT". When the kernel developer of an out-of-tree platform wants to make kexec work, they will naturally look atmachine_kexec.c, where they will find this UHI stuff, obviously telling them to do "selectUHI_BOOT". Otherwise, unless they google onto this discussion thread, it's harder to know the solution to thekexec_argsrelated problem hides in the code of another platform (Generic).
Quoted 1 line
* It simplifies work if the out of tree platform wants to upstream.
Well, ideally future platforms will just use arch/mips/generic rather than adding more platform/board code at all. Right now the only reason not to is if a system has a memory map that doesn't fit nicely, which should be resolved at some point by mapping the kernel which will allow the generic kernel to better support differing physical address maps.
So I hope we don't add more platform code anyway, which would make all this moot. I certainly don't want to encourage adding more by explicitly making it easier to do. And if there does come a point where we need to add more platform code for some good reason then we can deal with this at that point rather than speculating now.
For out of tree platforms, I don't think that copying the generic_kexec_prepare() function is particularly onerous anyway, and should be trivial to remove if such code is submitted upstream. Whilst I wouldn't want to make submitting code upstream more difficult, it's likely to need some amount of rework if submitted upstream anyway so this doesn't seem like a big deal.
I don't think upstream should be particularly concerned with making life easy for out of tree code - if one wants the benefit of their code being taken into account upstream, then one should submit one's code upstream. In the paraphrased words of my pastor at a marriage prep class - no benefits (except those already conferred by free access to open source software) without commitment.
Thanks,
Paul
Hi Dengcheng,
Quoted 3 lines · Dengcheng Zhu
Regarding cache flushing in
machine_kexec(), I recommend simply removing__flush_cache_all()as CPU0 will do local icache flush before jumping toreboot_code_buffer, and other CPUs don't jump at all.
Unfortunately that doesn't work - what if CPU 1 runs machine_kexec() & writes to reboot_code_buffer, then CPU 0 fetches stale/garbage code from L2 into its icache?
The icache flush I added in kexec_this_cpu() isn't enough in all systems unless the CPU that wrote the code writes it back far enough for a remote CPU to observe. The existing __flush_cache_all() is one heavy-handed way to achieve that.
Quoted 2 lines
Re: marking CPU offline in
kexec_this_cpu(), it's probably good NOT doing it either, as the system is going to reboot from CPU0 soon.
...but the new kernel will have no knowledge of whether the old kernel had CPUs marked online or offline, so I don't follow the argument?
Marking CPUs offline does have the advantage that we won't try to IPI them. This just makes perfect sense to me, and note that both arch/arm & arch/x86 offline CPUs during kexec too (I *really* like the way arch/arm just uses disable_nonboot_cpus() to go through the usual hotplug sequence in the non-crash case).
Quoted 7 lines
HALT is good enough, no need to gate core power.
As to whether it's safe runningplay_dead()in parallel, it shouldn't be a problem, as the loop is based
on cpu online mask (which we don't mark offline as mentioned above) -- The CPU will simply halt itself. For non-MT CPUs,play_dead()does make sense as well, as it's supposed to stop this CPU's execution, getting ready to reboot from CPU0.
That feels like relying on play_dead() accidentally working rather than doing something it's designed for though, and it would come at the expense of not marking CPUs offline & the associated fragility of needing to avoid anything that might IPI them (like some cache flush functions, as we've seen). I'd much rather we figure out a way to do this without all that.
One option might be to add something like arch/x86's stop_other_cpus & crash_stop_other_cpus callbacks in struct plat_smp_ops, which we can then implement as appropriate. We'd hopefully still reuse some of the code from play_dead() implementations, but have the separation to allow them to function differently where needed (eg. the new callbacks could halt all threads in MT systems without caring about cpu_online_mask).
This would also give us a natural way to check whether a system actually supports kexec properly, as we could just return with an error from machine_kexec_prepare() if the stop_other_cpus callback isn't implemented for the system (ie. is NULL).
Thanks,
Paul
Hi Paul,
On cache issue, what if kexec_this_cpu() does local i$ flush and scache invalidation? This is done only on CPU0, no IPI needed. And only CUP0 will use reboot_code_buffer. In this way, marking other CPUs offline to prevent being sent IPI is not needed.
As to "the associated fragility of needing to avoid anything that might IPI them", I hope that's not a problem as the kexec sequence is determined: The last IPI received is to ask them to stand by on reboot signal, because other than kexec there is no other kernel code beyond our control, which might send IPIs.
Quoted 1 line
...but the new kernel will have no knowledge of whether the old kernel
had CPUs marked online or offline, so I don't follow the argument?
[Dengcheng] I meant the new kernel shouldn't need CPU online info from the old kernel. The new one is doing a "fresh" boot, provided CPUs 1+ have been halted.
Regarding the use of play_dead(), it's simple and generic - MIPS CPUs have implemented their own play_dead(), which we can use to stop execution. Other options may still come down to play_dead() to handle MT/MC details. The concern of running it in parallel is really about the online status, which has been explained above and in my last email.
A 'nicer' implementation stopping non-boot CPUs from one CPU might be available, but handling code duplication of play_dead() and saving CPU states could be a problem.
For now, no __flush_cache_all() in machine_kexec() + no marking offline in kexec_this_cpu() + scache invalidation in kexec_this_cpu() might be a way to go.
Thanks,
Dengcheng
---
From: Paul Burton
Sent: Thursday, September 6, 2018 4:21 PM
To: Dengcheng Zhu
Cc: Paul Burton; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH v4 0/6] MIPS: kexec/kdump: Fix smp reboot and other issues
Hi Dengcheng,
Quoted 3 lines · Dengcheng Zhu
Regarding cache flushing in
machine_kexec(), I recommend simply removing__flush_cache_all()as CPU0 will do local icache flush before jumping toreboot_code_buffer, and other CPUs don't jump at all.
Unfortunately that doesn't work - what if CPU 1 runs machine_kexec() & writes to reboot_code_buffer, then CPU 0 fetches stale/garbage code from L2 into its icache?
The icache flush I added in kexec_this_cpu() isn't enough in all systems unless the CPU that wrote the code writes it back far enough for a remote CPU to observe. The existing __flush_cache_all() is one heavy-handed way to achieve that.
Quoted 2 lines
Re: marking CPU offline in
kexec_this_cpu(), it's probably good NOT doing it either, as the system is going to reboot from CPU0 soon.
...but the new kernel will have no knowledge of whether the old kernel had CPUs marked online or offline, so I don't follow the argument?
Marking CPUs offline does have the advantage that we won't try to IPI them. This just makes perfect sense to me, and note that both arch/arm & arch/x86 offline CPUs during kexec too (I *really* like the way arch/arm just uses disable_nonboot_cpus() to go through the usual hotplug sequence in the non-crash case).
Quoted 7 lines
HALT is good enough, no need to gate core power.
As to whether it's safe runningplay_dead()in parallel, it shouldn't be a problem, as the loop is based
on cpu online mask (which we don't mark offline as mentioned above) -- The CPU will simply halt itself. For non-MT CPUs,play_dead()does make sense as well, as it's supposed to stop this CPU's execution, getting ready to reboot from CPU0.
That feels like relying on play_dead() accidentally working rather than doing something it's designed for though, and it would come at the expense of not marking CPUs offline & the associated fragility of needing to avoid anything that might IPI them (like some cache flush functions, as we've seen). I'd much rather we figure out a way to do this without all that.
One option might be to add something like arch/x86's stop_other_cpus & crash_stop_other_cpus callbacks in struct plat_smp_ops, which we can then implement as appropriate. We'd hopefully still reuse some of the code from play_dead() implementations, but have the separation to allow them to function differently where needed (eg. the new callbacks could halt all threads in MT systems without caring about cpu_online_mask).
This would also give us a natural way to check whether a system actually supports kexec properly, as we could just return with an error from machine_kexec_prepare() if the stop_other_cpus callback isn't implemented for the system (ie. is NULL).
Thanks,
Paul
Hi Dengcheng,
Quoted 4 lines · Dengcheng Zhu
On cache issue, what if
kexec_this_cpu()does local i$ flush and scache invalidation? This is done only on CPU0, no IPI needed. And only CUP0 will usereboot_code_buffer. In this way, marking other CPUs offline to prevent being sent IPI is not needed.
The problem is that the "only CPU0 will use reboot_code_buffer" part isn't true because CPU0 might not be the one that writes to it. Though perhaps if we moved the memcpy() from machine_kexec() to kexec_this_cpu() that could work.
It still doesn't seem ideal to me that we'd have to even consider IPIs & the fact that we might silently hang if we accidentally send one. I really think it would be cleaner to mark CPUs offline & the problem is solved naturally - we shouldn't IPI these CPUs, and marking them offline means that we won't. Problem solved, and consistently with other architectures.
Quoted 4 lines
As to "the associated fragility of needing to avoid anything that might IPI them", I hope that's not a problem as the kexec sequence is determined: The last IPI received is to ask them to stand by on reboot signal, because other than kexec there is no other kernel code beyond our control, which might send IPIs.
But it does mean we can't call certain things, like __flush_cache_all(), which means that developers need to 1) know that and 2) remember to avoid any functions like this that perform an IPI behind the scenes. It just feels like loading a gun & leaving it pointed at our feet - by itself it's not necessarily going to cause a problem, but it would be oh so easy to trigger problems later.
Quoted 11 lines
Quoted 1 line
...but the new kernel will have no knowledge of whether the old kernel
had CPUs marked online or offline, so I don't follow the argument?
[Dengcheng] I meant the new kernel shouldn't need CPU online info from the old kernel. The new one is doing a "fresh" boot, provided CPUs 1+ have been halted.
Regarding the use of
play_dead(), it's simple and generic - MIPS CPUs have implemented their ownplay_dead(), which we can use to stop execution. Other options may still come down toplay_dead()to handle MT/MC details. The concern of running it in parallel is really about the online status, which has been explained above and in my last email.
But play_dead() really doesn't mean what you're suggesting it means.
What we want for kexec is that the CPU is stopped entirely - either halted or powered down. Failing that it needs to be running a loop from somewhere that we know isn't going to be overwritten, and the code it's running needs to be able to hand over to the new kernel later (eg. the existing kexec_smp_wait loop implements this latter approach).
Whilst the smp-cps.c implementation of play_dead() will currently halt or power down the CPU, that's not universally true & doesn't necessarily even need to remain true for smp-cps.c in the future (eg. we could decide the halt/power down step would be cleaner to do in cps_cpu_die).
A quick look at the other play_dead() implementations we have shows that both the cavium-octeon & loongson3 versions leave the CPU running a loop (within the current/old kernel), whilst the bmips version just sits at a wait instruction before jumping back to the fixed location of bmips_secondary_reentry (in the current/old kernel) once interrupted. None of those are suitable for kexec.
In general the CPU hotplug code just needs that when play_dead() runs on the CPU going offline, and the struct plat_smp_ops cpu_die() callback runs on some other CPU, the CPU being offlined will be quiesced to a degree that it won't interfere with Linux. This is different to kexec where we need the CPU to stop in such a way that it's fine that we might overwrite the old kernel, and then be able to start it running the new kernel later. As described before, that means it either needs to halt or power down until the new kernel takes control of it or it needs to run some loop from some location that we know won't be overwritten.
play_dead() is simple & already exists, it just doesn't do quite what we need kexec to do outside of the one system you're looking at.
Quoted 3 lines
A 'nicer' implementation stopping non-boot CPUs from one CPU might be available, but handling code duplication of
play_dead()and saving CPU states could be a problem.
Code that can be shared between play_dead() & kexec can still be shared if we have a different callback - the particular implementation of the two would just have both call a common function for the part that's common.
Quoted 2 lines
For now, no
__flush_cache_all()inmachine_kexec()+ no marking offline inkexec_this_cpu()+ scache invalidation inkexec_this_cpu()might be a way to go.
For the system you're interested in, sure that works. But as I've pointed out in my last few emails it does not work in general for other systems that we also need to support in the kernel.
Thanks,
Paul
Hi Paul,
Quoted 1 line
The problem is that the "only CPU0 will use
reboot_code_buffer" part
isn't true because CPU0 might not be the one that writes to it.
[Dengcheng]: I didn't say CPU0 is always the one running machine_kexec(). Actually, in my testing, I intentionally did something like:
taskset -c 3 echo c > /proc/sysrq-trigger
And for "either halted or powered down", do you have any idea/plan for Octeon, Loognson and bmips?
Thanks,
Dengcheng
From: Paul Burton
Sent: Friday, September 7, 2018 2:42 PM
To: Dengcheng Zhu
Cc: Paul Burton; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH v4 0/6] MIPS: kexec/kdump: Fix smp reboot and other issues
Hi Dengcheng,
Quoted 4 lines · Dengcheng Zhu
On cache issue, what if
kexec_this_cpu()does local i$ flush and scache invalidation? This is done only on CPU0, no IPI needed. And only CUP0 will usereboot_code_buffer. In this way, marking other CPUs offline to prevent being sent IPI is not needed.
The problem is that the "only CPU0 will use reboot_code_buffer" part isn't true because CPU0 might not be the one that writes to it. Though perhaps if we moved the memcpy() from machine_kexec() to kexec_this_cpu() that could work.
It still doesn't seem ideal to me that we'd have to even consider IPIs & the fact that we might silently hang if we accidentally send one. I really think it would be cleaner to mark CPUs offline & the problem is solved naturally - we shouldn't IPI these CPUs, and marking them offline means that we won't. Problem solved, and consistently with other architectures.
Quoted 4 lines
As to "the associated fragility of needing to avoid anything that might IPI them", I hope that's not a problem as the kexec sequence is determined: The last IPI received is to ask them to stand by on reboot signal, because other than kexec there is no other kernel code beyond our control, which might send IPIs.
But it does mean we can't call certain things, like __flush_cache_all(), which means that developers need to 1) know that and 2) remember to avoid any functions like this that perform an IPI behind the scenes. It just feels like loading a gun & leaving it pointed at our feet - by itself it's not necessarily going to cause a problem, but it would be oh so easy to trigger problems later.
Quoted 11 lines
Quoted 1 line
...but the new kernel will have no knowledge of whether the old kernel
had CPUs marked online or offline, so I don't follow the argument?
[Dengcheng] I meant the new kernel shouldn't need CPU online info from the old kernel. The new one is doing a "fresh" boot, provided CPUs 1+ have been halted.
Regarding the use of
play_dead(), it's simple and generic - MIPS CPUs have implemented their ownplay_dead(), which we can use to stop execution. Other options may still come down toplay_dead()to handle MT/MC details. The concern of running it in parallel is really about the online status, which has been explained above and in my last email.
But play_dead() really doesn't mean what you're suggesting it means.
What we want for kexec is that the CPU is stopped entirely - either halted or powered down. Failing that it needs to be running a loop from somewhere that we know isn't going to be overwritten, and the code it's running needs to be able to hand over to the new kernel later (eg. the existing kexec_smp_wait loop implements this latter approach).
Whilst the smp-cps.c implementation of play_dead() will currently halt or power down the CPU, that's not universally true & doesn't necessarily even need to remain true for smp-cps.c in the future (eg. we could decide the halt/power down step would be cleaner to do in cps_cpu_die).
A quick look at the other play_dead() implementations we have shows that both the cavium-octeon & loongson3 versions leave the CPU running a loop (within the current/old kernel), whilst the bmips version just sits at a wait instruction before jumping back to the fixed location of bmips_secondary_reentry (in the current/old kernel) once interrupted. None of those are suitable for kexec.
In general the CPU hotplug code just needs that when play_dead() runs on the CPU going offline, and the struct plat_smp_ops cpu_die() callback runs on some other CPU, the CPU being offlined will be quiesced to a degree that it won't interfere with Linux. This is different to kexec where we need the CPU to stop in such a way that it's fine that we might overwrite the old kernel, and then be able to start it running the new kernel later. As described before, that means it either needs to halt or power down until the new kernel takes control of it or it needs to run some loop from some location that we know won't be overwritten.
play_dead() is simple & already exists, it just doesn't do quite what we need kexec to do outside of the one system you're looking at.
Quoted 3 lines
A 'nicer' implementation stopping non-boot CPUs from one CPU might be available, but handling code duplication of
play_dead()and saving CPU states could be a problem.
Code that can be shared between play_dead() & kexec can still be shared if we have a different callback - the particular implementation of the two would just have both call a common function for the part that's common.
Quoted 2 lines
For now, no
__flush_cache_all()inmachine_kexec()+ no marking offline inkexec_this_cpu()+ scache invalidation inkexec_this_cpu()might be a way to go.
For the system you're interested in, sure that works. But as I've pointed out in my last few emails it does not work in general for other systems that we also need to support in the kernel.
Thanks,
Paul
Quoted 10 lines · Dengcheng Zhu
Hi Paul,
Quoted 1 line
The problem is that the "only CPU0 will use
reboot_code_buffer" partisn't true because CPU0 might not be the one that writes to it.
[Dengcheng]: I didn't say CPU0 is always the one running
machine_kexec(). Actually, in my testing, I intentionally did something like:taskset -c 3 echo c >
/proc/sysrq-trigger
Well, if CPU 0 isn't running machine_kexec() then some other CPU is doing the write to reboot_code_buffer and therefore CPU 0 isn't the only one accessing/using it.
As I walked through in an earlier email, the safe way for this to happen is for the machine_kexec() CPU to write back its dcache as far as needed for a remote CPU's icache to observe the stores. Having CPU 0 be the only one to do any cache maintenance will not achieve that.
As I mentioned before, if you're testing on I6x00 then you probably got away with it because the icache fills from the dcache. You'd probably also get away with having CPU 0 be the only one to do cache maintenance since the CM globalizes hit cache ops. But not all MIPS CPUs would be so lucky.
Quoted 2 lines
And for "either halted or powered down", do you have any idea/plan for Octeon, Loognson and bmips?
Well like I said before if we add a callback to struct plat_smp_ops to stop other CPUs, at least in the crash case, then we can check that & refuse to kexec() on systems that haven't implemented it.
Octeon right now has kexec, so any changes like this should bring Octeon along for the ride by implementing the new callback there. Loongson has a patch pending [1] so that should probably be taken into account too. bmips would just be left alone, and wouldn't support kexec until someone implements the new callback.
[1] https://patchwork.linux-mips.org/patch/20432/
Thanks,
Paul
Quoted 1 line
Well, if CPU 0 isn't running
machine_kexec()then some other CPU is
doing the write to reboot_code_buffer and therefore CPU 0 isn't the only one accessing/using it.
[Dengcheng]: I meant only CPU0 will jump to reboot_code_buffer. This buffer is surely possibly written by another CPU. Any issue with the machine_kexec() CPU flush "local" dache, and CPU0 invalidate "local" icache and scahe before jumping?
Thanks,
Dengcheng
From: Paul Burton
Sent: Friday, September 7, 2018 4:11 PM
To: Dengcheng Zhu
Cc: Paul Burton; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH v4 0/6] MIPS: kexec/kdump: Fix smp reboot and other issues
Quoted 10 lines · Dengcheng Zhu
Hi Paul,
Quoted 1 line
The problem is that the "only CPU0 will use
reboot_code_buffer" partisn't true because CPU0 might not be the one that writes to it.
[Dengcheng]: I didn't say CPU0 is always the one running
machine_kexec(). Actually, in my testing, I intentionally did something like:taskset -c 3 echo c >
/proc/sysrq-trigger
Well, if CPU 0 isn't running machine_kexec() then some other CPU is doing the write to reboot_code_buffer and therefore CPU 0 isn't the only one accessing/using it.
As I walked through in an earlier email, the safe way for this to happen is for the machine_kexec() CPU to write back its dcache as far as needed for a remote CPU's icache to observe the stores. Having CPU 0 be the only one to do any cache maintenance will not achieve that.
As I mentioned before, if you're testing on I6x00 then you probably got away with it because the icache fills from the dcache. You'd probably also get away with having CPU 0 be the only one to do cache maintenance since the CM globalizes hit cache ops. But not all MIPS CPUs would be so lucky.
Quoted 2 lines
And for "either halted or powered down", do you have any idea/plan for Octeon, Loognson and bmips?
Well like I said before if we add a callback to struct plat_smp_ops to stop other CPUs, at least in the crash case, then we can check that & refuse to kexec() on systems that haven't implemented it.
Octeon right now has kexec, so any changes like this should bring Octeon along for the ride by implementing the new callback there. Loongson has a patch pending [1] so that should probably be taken into account too. bmips would just be left alone, and wouldn't support kexec until someone implements the new callback.
[1] https://patchwork.linux-mips.org/patch/20432/
Thanks,
Paul