Add restrictions for kexec/kdump jumping between 5-level and 4-level kernel
v2 cover Add restrictions for kexec/kdump jumping between 5-level and 4-level kernel
This is v2 post.
The original v1 post can be found here:
http://lkml.kernel.org/r/[email protected]
Later a v1 RESEND version:
http://lkml.kernel.org/r/[email protected]
This patchset is trying to fix several issues for kexec/kdump when dynamic switching of paging mode is enabled in x86_64. The current kernel supports 5-level paging mode, and supports dynamically choosing paging mode during bootup according to kernel image, hardware and kernel parameter setting. This flexibility brings several issues for kexec/kdump:
Issues:
1)
Dynamic switching between paging mode requires code change in target kernel. So we can't kexec jump from 5-level kernel to old 4-level kernel which lacks the code change.
2)
Switching from 5-level paging to 4-level paging kernel would fail, if kexec() put kernel image above 64TiB of memory.
3)
Kdump jumping has similar issue as 2). This require us to only reserve crashkernel below 64TB, otherwise jumping from 5-level to 4-level kernel will fail.
Note:
Since we have two interfaces kexec_load() and kexec_file_load() to load kexec/kdump kernel, handling for them is a little different. For kexec_load(), most of the loading job is done in user space utility kexec_tools. However, for kexec_file_load(), most of the loading codes have moved into kernel because of kernel image verification.
Fixes:
a) For issue 1), we need check if XLF_5LEVEL is set, otherwise error out a message.
-This need be done in both kernel and kexec_tools utility. -Patch 2/3 is the handling of kernel part.
-Will post user space patch to kexec mailing list later.
b) For issue 2), we need check if both XLF_5LEVEL and XLF_5LEVEL_ENABLED are set, otherwise error out a message.
-This only need be done in kexec_tools utility. Because for
kexec_file_load(), the current code searches area to put kernel from bottom to up in system RAM, we usually can always find an area below 4 GB, no need to worry about 5-level kernel jumping to 4-level kernel. While for kexec_load(), it's top down seraching area for kernel loading, and implemented in user space. We need make sure that 5-level kernel find an area under 64 TB for a kexec-ed kernel of 4-level.
-Will post user space patch to kexec mailing list later.
c) For issues 3), just limit kernel to reserve crashkernel below 64 TB. -This only need be done in kernel.
-It doesn't need to check bit XLF_5LEVEL or XLF_5LEVEL_ENABLED, we
just simply limit it below 64 TB which should be enough. Because crashernel is reserved during the 1st kernel's bootup, we don't know what kernel will be loaded for kdump usage.
-Patch 3/3 handles this.
Concerns from reviewing comments:
1)
In v1, hpa raised concern that why the paging mode checking is not done before kexec jumping, the discussion can be found here:
http://lkml.kernel.org/r/[email protected]
As tglx said, it might be not doable for kdump since kdump kernel's reserved crashkernel region only owns a portion of memory, may be above 4G; and might be not safer to do paging mode checking and switching thing after crash.
2)
In v1 RESEND post, tglx asked why only bit XLF_5LEVEL is checked, even though two bits XLF_5LEVEL or XLF_5LEVEL_ENABLED. So add sentences to explain this in *Fixes* b).
v1->v2:
Correct the subject of patch 1 according to tglx's comment; Add more information to cover-letter to address reviewers' concerns;
Baoquan He (3):
x86/boot: Add xloadflags bits for 5-level kernel checking x86/kexec/64: Error out if try to jump to old 4-level kernel from
5-level kernel
x86/kdump/64: Change the upper limit of crashkernel reservation
arch/x86/boot/header.S | 12 +++++++++++- arch/x86/include/uapi/asm/bootparam.h | 2 ++ arch/x86/kernel/kexec-bzimage64.c | 5 +++++ arch/x86/kernel/setup.c | 18 ++++++++++++++---- 4 files changed, 32 insertions(+), 5 deletions(-)
Signature · 1 line
2.17.2
PATCH v2 1/3 x86/boot: Add xloadflags bits for 5-level kernel checking
Add two bits XLF_5LEVEL and XLF_5LEVEL_ENABLED for 5-level kernel. Bit XLF_5LEVEL indicates if 5-level related code is contained in this kernel.
Bit XLF_5LEVEL_ENABLED indicates if CONFIG_X86_5LEVEL=y is set.
They are being used in later patch to check if kexec/kdump kernel is loaded in right place.
--- arch/x86/boot/header.S | 12 +++++++++++- arch/x86/include/uapi/asm/bootparam.h | 2 ++ 2 files changed, 13 insertions(+), 1 deletion(-)
arch/x86/boot/header.S +11 -1
@@ -419,7 +419,17 @@ xloadflags: # define XLF4 0 #endif - .word XLF0 | XLF1 | XLF23 | XLF4 +#ifdef CONFIG_X86_64 +#ifdef CONFIG_X86_5LEVEL +#define XLF56 (XLF_5LEVEL|XLF_5LEVEL_ENABLED) +#else +#define XLF56 XLF_5LEVEL +#endif +#else +#define XLF56 0 +#endif + + .word XLF0 | XLF1 | XLF23 | XLF4 | XLF56 cmdline_size: .long COMMAND_LINE_SIZE-1 #length of the command line, #added with boot protocol
arch/x86/include/uapi/asm/bootparam.h +2 -0
@@ -29,6 +29,8 @@#define XLF_EFI_HANDOVER_32 (1<<2)#define XLF_EFI_HANDOVER_64 (1<<3)#define XLF_EFI_KEXEC (1<<4) +#define XLF_5LEVEL (1<<5) +#define XLF_5LEVEL_ENABLED (1<<6)#ifndef __ASSEMBLY__
Signature · 1 line
2.17.2
PATCH v2 2/3 x86/kexec/64: Error out if try to jump to old 4-level kernel from 5-level kernel
In reply to Baoquan HeIn relocate_kernel() CR4.LA57 flag is set before kexec jumping if the kernel has 5-level paging enabled. Then in boot/compressed/head_64.S, it will check if the booting kernel is in 4-level or 5-level paging mode, and handle accordingly. However, the old kernel which doesn't contain the 5-level codes doesn't know how to cope with it, then #GP triggered.
Instead of triggering #GP during kexec kernel boot, error out during kexec loading if find out we are trying to jump to old 4-level kernel from 5-level kernel.
--- arch/x86/kernel/kexec-bzimage64.c | 5 +++++ 1 file changed, 5 insertions(+)
arch/x86/kernel/kexec-bzimage64.c +5 -0
@@ -321,6 +321,11 @@ static int bzImage64_probe(const char *buf, unsigned long len)return ret; } +if (!(header->xloadflags &XLF_5LEVEL ) &&pgtable_l5_enabled ()) { +pr_err ("Can not jump to old 4-level kernel from 5-level kernel.\n" ); +return ret; + } +/* I've got a bzImage */ pr_debug ("It's a relocatable bzImage64\n" ); ret =0 ;
Signature · 1 line
2.17.2
PATCH v2 3/3 x86/kdump/64: Change the upper limit of crashkernel reservation
In reply to Baoquan HeRestrict kdump to only reserve crashkernel below 64TB. Since the kdump jumping may be from 5-level to 4-level, and the kdump kernel is put above 64TB in 5-level kernel, then the jumping will fail. And the crashkernel reservation is done during the 1st kernel bootup, there's no way to detect the paging mode of kdump kernel at that time.
Hence change the upper limit of crashkernel reservation to 64TB on x86_64.
--- arch/x86/kernel/setup.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
arch/x86/kernel/setup.c +14 -4
@@ -451,16 +451,26 @@ static void __init memblock_x86_reserve_range_setup_data(void)#define CRASH_ALIGN (16 << 20)/* -* Keep the crash kernel below this limit. On 32 bits earlier kernels -* would limit the kernel to the low 512 MiB due to mapping restrictions. -* On 64bit, old kexec-tools need to under 896MiB. +* Keep the crash kernel below this limit. +* +* On 32 bits earlier kernels would limit the kernel to the low +* 512 MiB due to mapping restrictions. +* +* On 64bit, old kexec-tools need to be under 896MiB. The later +* supports to put kernel above 4G, up to system RAM top. Here +* kdump kernel need be restricted to be under 64TB, which is +* the upper limit of system RAM in 4-level paing mode. Since +* the kdump jumping could be from 5-level to 4-level, the jumping +* will fail if kernel is put above 64TB, and there's no way to +* detect the paging mode of the kernel which will be loaded for +* dumping during the 1st kernel bootup. */ #ifdef CONFIG_X86_32 # define CRASH_ADDR_LOW_MAX (512 << 20)# define CRASH_ADDR_HIGH_MAX (512 << 20)#else # define CRASH_ADDR_LOW_MAX (896UL << 20) -# define CRASH_ADDR_HIGH_MAX MAXMEM +# define CRASH_ADDR_HIGH_MAX (1ULL < 46)#endif static int __init reserve_crashkernel_low (void )
Signature · 1 line
2.17.2
Re: [PATCH v2 3/3] x86/kdump/64: Change the upper limit of crashkernel reservation
Quoted 36 lines · Baoquan He
Restrict kdump to only reserve crashkernel below 64TB. Since the kdump jumping may be from 5-level to 4-level, and the kdump kernel is put above 64TB in 5-level kernel, then the jumping will fail. And the crashkernel reservation is done during the 1st kernel bootup, there's no way to detect the paging mode of kdump kernel at that time.
Hence change the upper limit of crashkernel reservation to 64TB on
x86_64.Signed-off-by Baoquan He <[email protected]>--- arch/x86/kernel/setup.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)arch/x86/kernel/setup.c +11
-3@@ -451,16 +451,26 @@ static void __init memblock_x86_reserve_range_setup_data(void)#define CRASH_ALIGN (16 << 20)/* -* Keep the crash kernel below this limit. On 32 bits earlier kernels -* would limit the kernel to the low 512 MiB due to mapping restrictions. -* On 64bit, old kexec-tools need to under 896MiB. +* Keep the crash kernel below this limit. +* +* On 32 bits earlier kernels would limit the kernel to the low +* 512 MiB due to mapping restrictions. +* +* On 64bit, old kexec-tools need to be under 896MiB. The later +* supports to put kernel above 4G, up to system RAM top. Here +* kdump kernel need be restricted to be under 64TB, which is +* the upper limit of system RAM in 4-level paing mode. Since +* the kdump jumping could be from 5-level to 4-level, the jumping +* will fail if kernel is put above 64TB, and there's no way to
arch/x86/kernel/setup.c static void __init memblock_x86_reserve_range_setup_data(void) · line 473
Quoted 10 lines
arch/x86/kernel/setup.c +3
-1+* detect the paging mode of the kernel which will be loaded for +* dumping during the 1st kernel bootup. */ #ifdef CONFIG_X86_32 # define CRASH_ADDR_LOW_MAX (512 << 20)# define CRASH_ADDR_HIGH_MAX (512 << 20)#else # define CRASH_ADDR_LOW_MAX (896UL << 20) -# define CRASH_ADDR_HIGH_MAX MAXMEM +# define CRASH_ADDR_HIGH_MAX (1ULL < 46)
Maybe
# define CRASH_ADDR_HIGH_MAX (64UL << 40)
to match the notation for the rest of defines?
Otherwise the patchset look good to me. You can use my ack for all of them:
Quoted 6 lines
arch/x86/kernel/setup.c +0
-0#endif static int __init reserve_crashkernel_low (void )Signature · 1 line
2.17.2
Signature · 1 line
Kirill A. Shutemov
Re: [PATCH v2 3/3] x86/kdump/64: Change the upper limit of crashkernel reservation
Quoted 10 lines · Kirill A. Shutemov
arch/x86/kernel/setup.c static void __init memblock_x86_reserve_range_setup_data(void) · line 473
Quoted 7 lines
#ifdef
CONFIG_X86_32# define CRASH_ADDR_LOW_MAX (512 << 20) # define CRASH_ADDR_HIGH_MAX (512 << 20) #else # define CRASH_ADDR_LOW_MAX (896UL << 20)arch/x86/kernel/setup.c +1
-1-# define CRASH_ADDR_HIGH_MAX MAXMEM +# define CRASH_ADDR_HIGH_MAX (1ULL < 46)Maybe
Quoted 4 lines
# define CRASH_ADDR_HIGH_MAX (64UL << 40)to match the notation for the rest of defines?
Sure, I will replace it, and repost a whole patchset with your ack added on each patch. Thanks for your careful reviewing and great suggestions.
Quoted 15 lines
Otherwise the patchset look good to me. You can use my ack for all of them:
Quoted 6 lines
#endif
static int
__initreserve_crashkernel_low(void)Signature · 1 line
2.17.2Signature · 1 line
Kirill A. Shutemov