Cure kexec() vs. mwait_play_dead() troubles
Get series with b4
b4 am [email protected]Versions newest first
- v1 Latest6 patches 21 repliesIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/6 | x86/smp: Remove pointless wmb() from native_stop_other_cpus()arch/ | +0 | 0 replies | |
| 2/6 | x86/smp: Acquire stopping_cpu unconditionallyarch/ | +4 | 0 replies | |
| 3/6 | x86/smp: Use dedicated cache-line for mwait_play_dead()arch/ | +14 | 0 replies | |
| 4/6 | x86/smp: Cure kexec() vs. mwait_play_dead() breakagearch/ arch/ arch/ | +71 | 3 replies | |
| 5/6 | x86/smp: Split sending INIT IPI out into a helper functionarch/ | +22 | 4 replies | |
| 6/6 | x86/smp: Put CPUs into INIT on shutdown if possiblearch/ arch/ arch/ | +50 | 1 reply |
Reviews trailers per patch
2 of 6 patches have a Reviewed-by or Acked-by.
| Patch | Ashok Raj |
|---|---|
| 1/6 | |
| 2/6 | |
| 3/6 | |
| 4/6 | R-b T-b |
| 5/6 | |
| 6/6 | R-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [patch 0/6] Cure kexec() vs. mwait_play_dead() troubles
Hi!
Ashok observed triple faults when executing kexec() on a kernel which has 'nosmt' on the kernel commandline and HT enabled in the BIOS.
'nosmt' brings up the HT siblings to the point where they initiliazed the CPU and then rolls the bringup back which parks them in mwait_play_dead(). The reason is that all CPUs should have CR4.MCE set. Otherwise a broadcast MCE will immediately shut down the machine.
Some detective work revealed that:
1) The kexec kernel can overwrite text, pagetables, stack and data of the previous kernel.
2) If the kexec kernel writes to the memory which is monitored by an "offline" CPU, that CPU resumes execution. That's obviously doomed when the kexec kernel overwrote text, pagetables, data or stack.
While on my test machine the first kexec() after reset always "worked", the second one reliably ended up in a triple fault.
The following series cures this by:
1) Bringing offline CPUs which are stuck in mwait_play_dead() out of mwait by writing to the monitored cacheline
2) Let the woken up CPUs check the written control word and drop into a HLT loop if the control word requests so.
This is only half safe because HLT can resume execution due to NMI, SMI and MCE. Unfortunately there is no real safe mechanism to "park" a CPU reliably, but there is at least one which prevents the NMI and SMI cause: INIT.
3) If the system uses the regular INIT/STARTUP sequence to wake up secondary CPUS, then "park" all CPUs including the "offline" ones by sending them INIT IPIs.
The INIT IPI brings the CPU into a wait for wakeup state which is not affected by NMI and SMI, but INIT also clears CR4.MCE, so the broadcast MCE problem comes back.
But that's not really any different from a CPU sitting in the HLT loop on the previous kernel. If a broadcast MCE arrives, HLT resumes execution and the CPU tries to handle the MCE on overwritten text, pagetables etc.
So parking them via INIT is not completely solving the problem, but it takes at least NMI and SMI out of the picture.
The series is also available from git:
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git x86/kexec
Thanks,
tglx
--- include/asm/smp.h | 4 + kernel/smp.c | 62 +++++++++++++--------- kernel/smpboot.c | 151 ++++++++++++++++++++++++++++++++++++++++-------------- 3 files changed, 156 insertions(+), 61 deletions(-)