Add restrictions for kexec/kdump jumping between 5-level and 4-level kernel
v3 cover Add restrictions for kexec/kdump jumping between 5-level and 4-level kernel
This is v3 post.
The original v1 post can be found here:
http://lkml.kernel.org/r/[email protected]
Later a v1 RESEND version:
http://lkml.kernel.org/r/[email protected]
V2 post is here:
http://lkml.kernel.org/r/[email protected]
This patchset is trying to fix several issues for kexec/kdump when dynamic switching of paging mode is enabled in x86_64. The current kernel supports 5-level paging mode, and supports dynamically choosing paging mode during bootup according to kernel image, hardware and kernel parameter setting. This flexibility brings several issues for kexec/kdump:
Issues:
1)
Dynamic switching between paging mode requires code change in target kernel. So we can't kexec jump from 5-level kernel to old 4-level kernel which lacks the code change.
2)
Switching from 5-level paging to 4-level paging kernel would fail, if kexec() put kernel image above 64TiB of memory.
3)
Kdump jumping has similar issue as 2). This require us to only reserve crashkernel below 64TB, otherwise jumping from 5-level to 4-level kernel will fail.
Note:
Since we have two interfaces kexec_load() and kexec_file_load() to load kexec/kdump kernel, handling for them is a little different. For kexec_load(), most of the loading job is done in user space utility kexec_tools. However, for kexec_file_load(), most of the loading codes have moved into kernel because of kernel image verification.
Fixes:
a) For issue 1), we need check if XLF_5LEVEL is set, otherwise error out a message.
-This need be done in both kernel and kexec_tools utility. -Patch 2/3 is the handling of kernel part.
-Will post user space patch to kexec mailing list later.
b) For issue 2), we need check if both XLF_5LEVEL and XLF_5LEVEL_ENABLED are set, otherwise error out a message.
-This only need be done in kexec_tools utility. Because for
kexec_file_load(), the current code searches area to put kernel from bottom to up in system RAM, we usually can always find an area below 4 GB, no need to worry about 5-level kernel jumping to 4-level kernel. While for kexec_load(), it's top down seraching area for kernel loading, and implemented in user space. We need make sure that 5-level kernel find an area under 64 TB for a kexec-ed kernel of 4-level.
-Will post user space patch to kexec mailing list later.
c) For issues 3), just limit kernel to reserve crashkernel below 64 TB. -This only need be done in kernel.
-It doesn't need to check bit XLF_5LEVEL or XLF_5LEVEL_ENABLED, we
just simply limit it below 64 TB which should be enough. Because crashernel is reserved during the 1st kernel's bootup, we don't know what kernel will be loaded for kdump usage.
-Patch 3/3 handles this.
Concerns from reviewing comments:
1)
In v1, hpa raised concern that why the paging mode checking is not done before kexec jumping, the discussion can be found here:
http://lkml.kernel.org/r/[email protected]
As tglx said, it might be not doable for kdump since kdump kernel's reserved crashkernel region only owns a portion of memory, may be above 4G; and might be not safer to do paging mode checking and switching thing after crash.
2)
In v1 RESEND post, tglx asked why only bit XLF_5LEVEL is checked, even though two bits XLF_5LEVEL or XLF_5LEVEL_ENABLED added. So add more words to explain it in *Fixes* b).
Changelog:v2->v3:
Change the constant to match the notation for the rest of defines as Kirill suggested;v1->v2:
Correct the subject of patch 1 according to tglx's comment; Add more information to cover-letter to address reviewers' concerns;
Baoquan He (3):
x86/boot: Add xloadflags bits for 5-level kernel checking x86/kexec/64: Error out if try to jump to old 4-level kernel from
5-level kernel
x86/kdump/64: Change the upper limit of crashkernel reservation
arch/x86/boot/header.S | 12 +++++++++++- arch/x86/include/uapi/asm/bootparam.h | 2 ++ arch/x86/kernel/kexec-bzimage64.c | 5 +++++ arch/x86/kernel/setup.c | 18 ++++++++++++++---- 4 files changed, 32 insertions(+), 5 deletions(-)
Signature · 1 line
2.17.2
PATCH v3 1/3 x86/boot: Add xloadflags bits for 5-level kernel checking
Add two bits XLF_5LEVEL and XLF_5LEVEL_ENABLED for 5-level kernel. Bit XLF_5LEVEL indicates if 5-level related code is contained in this kernel.
Bit XLF_5LEVEL_ENABLED indicates if CONFIG_X86_5LEVEL=y is set.
They are being used in later patch to check if kexec/kdump kernel is loaded in right place.
--- arch/x86/boot/header.S | 12 +++++++++++- arch/x86/include/uapi/asm/bootparam.h | 2 ++ 2 files changed, 13 insertions(+), 1 deletion(-)
arch/x86/boot/header.S +11 -1
@@ -419,7 +419,17 @@ xloadflags: # define XLF4 0 #endif - .word XLF0 | XLF1 | XLF23 | XLF4 +#ifdef CONFIG_X86_64 +#ifdef CONFIG_X86_5LEVEL +#define XLF56 (XLF_5LEVEL|XLF_5LEVEL_ENABLED) +#else +#define XLF56 XLF_5LEVEL +#endif +#else +#define XLF56 0 +#endif + + .word XLF0 | XLF1 | XLF23 | XLF4 | XLF56 cmdline_size: .long COMMAND_LINE_SIZE-1 #length of the command line, #added with boot protocol
arch/x86/include/uapi/asm/bootparam.h +2 -0
@@ -29,6 +29,8 @@#define XLF_EFI_HANDOVER_32 (1<<2)#define XLF_EFI_HANDOVER_64 (1<<3)#define XLF_EFI_KEXEC (1<<4) +#define XLF_5LEVEL (1<<5) +#define XLF_5LEVEL_ENABLED (1<<6)#ifndef __ASSEMBLY__
Signature · 1 line
2.17.2
Re: [PATCH v3 1/3] x86/boot: Add xloadflags bits for 5-level kernel checking
Quoted 7 lines · Baoquan He
Add two bits
XLF_5LEVELandXLF_5LEVEL_ENABLEDfor 5-level kernel. BitXLF_5LEVELindicates if 5-level related code is contained in this kernel.
BitXLF_5LEVEL_ENABLEDindicates ifCONFIG_X86_5LEVEL=yis set.They are being used in later patch to check if kexec/kdump kernel is loaded in right place.
Only XLF_5LEVEL is checked. So what's the second flag for?
Thanks,
tglx
Re: [PATCH v3 1/3] x86/boot: Add xloadflags bits for 5-level kernel checking
Quoted 11 lines · Thomas Gleixner
Quoted 7 lines · Baoquan He
Add two bits
XLF_5LEVELandXLF_5LEVEL_ENABLEDfor 5-level kernel. BitXLF_5LEVELindicates if 5-level related code is contained in this kernel.
BitXLF_5LEVEL_ENABLEDindicates ifCONFIG_X86_5LEVEL=yis set.They are being used in later patch to check if kexec/kdump kernel is loaded in right place.
Only
XLF_5LEVELis checked. So what's the second flag for?
I put the explanation why only XLF_5LEVEL is checked in cover letter. Will add them in this log.
PATCH v3 2/3 x86/kexec/64: Error out if try to jump to old 4-level kernel from 5-level kernel
In reply to Baoquan HeIn relocate_kernel() CR4.LA57 flag is set before kexec jumping if the kernel has 5-level paging enabled. Then in boot/compressed/head_64.S, it will check if the booting kernel is in 4-level or 5-level paging mode, and handle accordingly. However, the old kernel which doesn't contain the 5-level codes doesn't know how to cope with it, then #GP triggered.
Instead of triggering #GP during kexec kernel boot, error out during kexec loading if find out we are trying to jump to old 4-level kernel from 5-level kernel.
--- arch/x86/kernel/kexec-bzimage64.c | 5 +++++ 1 file changed, 5 insertions(+)
arch/x86/kernel/kexec-bzimage64.c +5 -0
@@ -321,6 +321,11 @@ static int bzImage64_probe(const char *buf, unsigned long len)return ret; } +if (!(header->xloadflags &XLF_5LEVEL ) &&pgtable_l5_enabled ()) { +pr_err ("Can not jump to old 4-level kernel from 5-level kernel.\n" ); +return ret; + } +/* I've got a bzImage */ pr_debug ("It's a relocatable bzImage64\n" ); ret =0 ;
Signature · 1 line
2.17.2
Re: [PATCH v3 2/3] x86/kexec/64: Error out if try to jump to old 4-level kernel from 5-level kernel
Quoted 6 lines · Baoquan He
In
relocate_kernel()CR4.LA57 flag is set before kexec jumping if the kernel has 5-level paging enabled. Then inboot/compressed/head_64.S, it will check if the booting kernel is in 4-level or 5-level paging mode, and handle accordingly. However, the old kernel which doesn't contain the 5-level codes doesn't know how to cope with it, then #GP triggered.
The above is more than confusing. I assume you want to say:
If the running kernel has 5-level paging activated, the 5-level paging mode is preserved across kexec. If the kexec'ed kernel does not contain support for handling active 5-level paging mode in the decompressor, the decompressor will crash with #GP.
Quoted 3 lines
Instead of triggering #GP during kexec kernel boot, error out during kexec loading if find out we are trying to jump to old 4-level kernel from 5-level kernel.
Prevent this situation at load time. If 5-level paging is active, check the xloadflags whether the kexec kernel can handle 5-level paging at least in the decompressor. If not, reject the load attempt.
Quoted 11 lines
Signed-off-by Baoquan He <[email protected]>--- arch/x86/kernel/kexec-bzimage64.c | 5 +++++ 1 file changed, 5 insertions(+)arch/x86/kernel/kexec-bzimage64.c +0
-0
arch/x86/kernel/kexec-bzimage64.c static int bzImage64_probe(const char *buf, unsigned long len) · line 325
Quoted 5 lines
arch/x86/kernel/kexec-bzimage64.c +2
-0@@ -321,6 +321,11 @@ static int bzImage64_probe(const char *buf, unsigned long len)return ret; } +if (!(header->xloadflags &XLF_5LEVEL ) &&pgtable_l5_enabled ()) { +pr_err ("Can not jump to old 4-level kernel from 5-level kernel.\n" );
This is confusing at best.
"bzImage cannot handle 5-level paging mode\n"
or something like this.
arch/x86/kernel/kexec-bzimage64.c static int bzImage64_probe(const char *buf, unsigned long len) · line 326
Quoted 1 line
arch/x86/kernel/kexec-bzimage64.c +1
-0+return ret;
Thanks,
tglx
Re: [PATCH v3 2/3] x86/kexec/64: Error out if try to jump to old 4-level kernel from 5-level kernel
Quoted 16 lines · Thomas Gleixner
Quoted 6 lines · Baoquan He
In
relocate_kernel()CR4.LA57 flag is set before kexec jumping if the kernel has 5-level paging enabled. Then inboot/compressed/head_64.S, it will check if the booting kernel is in 4-level or 5-level paging mode, and handle accordingly. However, the old kernel which doesn't contain the 5-level codes doesn't know how to cope with it, then #GP triggered.The above is more than confusing. I assume you want to say:
If the running kernel has 5-level paging activated, the 5-level paging mode is preserved across kexec. If the kexec'ed kernel does not contain support for handling active 5-level paging mode in the decompressor, the decompressor will crash with #GP.
Quoted 7 lines
Quoted 3 lines
Instead of triggering #GP during kexec kernel boot, error out during kexec loading if find out we are trying to jump to old 4-level kernel from 5-level kernel.
Prevent this situation at load time. If 5-level paging is active, check the xloadflags whether the kexec kernel can handle 5-level paging at least in the decompressor. If not, reject the load attempt.
Yes, exactly. I will rewrite patch log with simpler sentences as you have demonstrated. Thanks.
Quoted 20 lines
Quoted 12 lines
Signed-off-by Baoquan He <[email protected]>--- arch/x86/kernel/kexec-bzimage64.c | 5 +++++ 1 file changed, 5 insertions(+)arch/x86/kernel/kexec-bzimage64.c +0
-0arch/x86/kernel/kexec-bzimage64.c static int bzImage64_probe(const char *buf, unsigned long len) · line 325
Quoted 5 lines
arch/x86/kernel/kexec-bzimage64.c +2
-0@@ -321,6 +321,11 @@ static int bzImage64_probe(const char *buf, unsigned long len)return ret; } +if (!(header->xloadflags &XLF_5LEVEL ) &&pgtable_l5_enabled ()) { +pr_err ("Can not jump to old 4-level kernel from 5-level kernel.\n" );This is confusing at best.
Quoted 5 lines
"bzImage cannot handle 5-level paging mode\n"
or something like this.
arch/x86/kernel/kexec-bzimage64.c static int bzImage64_probe(const char *buf, unsigned long len) · line 326
Quoted 1 line
arch/x86/kernel/kexec-bzimage64.c +1
-0+return ret;
Will change too as you suggested.
PATCH v3 3/3 x86/kdump/64: Change the upper limit of crashkernel reservation
In reply to Baoquan HeRestrict kdump to only reserve crashkernel below 64TB. Since the kdump jumping may be from 5-level to 4-level, and the kdump kernel is put above 64TB in 5-level kernel, then the jumping will fail. And the crashkernel reservation is done during the 1st kernel bootup, there's no way to detect the paging mode of kdump kernel at that time.
Hence change the upper limit of crashkernel reservation to 64TB on x86_64.
--- arch/x86/kernel/setup.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
arch/x86/kernel/setup.c +14 -4
@@ -451,16 +451,26 @@ static void __init memblock_x86_reserve_range_setup_data(void)#define CRASH_ALIGN (16 << 20)/* -* Keep the crash kernel below this limit. On 32 bits earlier kernels -* would limit the kernel to the low 512 MiB due to mapping restrictions. -* On 64bit, old kexec-tools need to under 896MiB. +* Keep the crash kernel below this limit. +* +* On 32 bits earlier kernels would limit the kernel to the low +* 512 MiB due to mapping restrictions. +* +* On 64bit, old kexec-tools need to be under 896MiB. The later +* supports to put kernel above 4G, up to system RAM top. Here +* kdump kernel need be restricted to be under 64TB, which is +* the upper limit of system RAM in 4-level paing mode. Since +* the kdump jumping could be from 5-level to 4-level, the jumping +* will fail if kernel is put above 64TB, and there's no way to +* detect the paging mode of the kernel which will be loaded for +* dumping during the 1st kernel bootup. */ #ifdef CONFIG_X86_32 # define CRASH_ADDR_LOW_MAX (512 << 20)# define CRASH_ADDR_HIGH_MAX (512 << 20)#else # define CRASH_ADDR_LOW_MAX (896UL << 20) -# define CRASH_ADDR_HIGH_MAX MAXMEM +# define CRASH_ADDR_HIGH_MAX (64UL << 40)#endif static int __init reserve_crashkernel_low (void )
Signature · 1 line
2.17.2
PING
Any comment for this patch, Or consider to merge?
Quoted 107 lines · Baoquan He
This is v3 post.
The original v1 post can be found here:
http://lkml.kernel.org/r/[email protected]Later a v1 RESEND version:
http://lkml.kernel.org/r/[email protected]V2 post is here:
http://lkml.kernel.org/r/[email protected]This patchset is trying to fix several issues for kexec/kdump when dynamic switching of paging mode is enabled in
x86_64. The current kernel supports 5-level paging mode, and supports dynamically choosing paging mode during bootup according to kernel image, hardware and kernel parameter setting. This flexibility brings several issues for kexec/kdump:Issues:
1)
Dynamic switching between paging mode requires code change in target kernel. So we can't kexec jump from 5-level kernel to old 4-level kernel which lacks the code change.2)
Switching from 5-level paging to 4-level paging kernel would fail, ifkexec()put kernel image above 64TiB of memory.3)
Kdump jumping has similar issue as 2). This require us to only reserve crashkernel below 64TB, otherwise jumping from 5-level to 4-level kernel will fail.Note:
Since we have two interfaceskexec_load()andkexec_file_load()to load kexec/kdump kernel, handling for them is a little different. Forkexec_load(), most of the loading job is done in user space utilitykexec_tools. However, forkexec_file_load(), most of the loading codes have moved into kernel because of kernel image verification.Fixes:
a) For issue 1), we need check ifXLF_5LEVELis set, otherwise error out a message.
-This need be done in both kernel andkexec_toolsutility. -Patch 2/3 is the handling of kernel part.
-Will post user space patch to kexec mailing list later.b) For issue 2), we need check if both
XLF_5LEVELandXLF_5LEVEL_ENABLEDare set, otherwise error out a message.
-This only need be done inkexec_toolsutility. Because for
kexec_file_load(), the current code searches area to put kernel from bottom to up in system RAM, we usually can always find an area below 4 GB, no need to worry about 5-level kernel jumping to 4-level kernel. While forkexec_load(), it's top down seraching area for kernel loading, and implemented in user space. We need make sure that 5-level kernel find an area under 64 TB for a kexec-ed kernel of 4-level.
-Will post user space patch to kexec mailing list later.c) For issues 3), just limit kernel to reserve crashkernel below 64 TB. -This only need be done in kernel.
-It doesn't need to check bitXLF_5LEVELorXLF_5LEVEL_ENABLED, we
just simply limit it below 64 TB which should be enough. Because crashernel is reserved during the 1st kernel's bootup, we don't know what kernel will be loaded for kdump usage.
-Patch 3/3 handles this.Concerns from reviewing comments:
1)
In v1, hpa raised concern that why the paging mode checking is not done before kexec jumping, the discussion can be found here:http://lkml.kernel.org/r/[email protected]
As tglx said, it might be not doable for kdump since kdump kernel's reserved crashkernel region only owns a portion of memory, may be above 4G; and might be not safer to do paging mode checking and switching thing after crash.
2)
In v1 RESEND post, tglx asked why only bitXLF_5LEVELis checked, even though two bitsXLF_5LEVELorXLF_5LEVEL_ENABLEDadded. So add more words to explain it in *Fixes* b).Changelog:
v2->v3:
Change the constant to match the notation for the rest of defines as Kirill suggested;v1->v2:
Correct the subject of patch 1 according to tglx's comment; Add more information to cover-letter to address reviewers' concerns;Baoquan He (3):
x86/boot: Add xloadflags bits for 5-level kernel checking x86/kexec/64: Error out if try to jump to old 4-level kernel from
5-level kernel
x86/kdump/64: Change the upper limit of crashkernel reservationarch/x86/boot/header.S | 12 +++++++++++- arch/x86/include/uapi/asm/bootparam.h | 2 ++ arch/x86/kernel/kexec-bzimage64.c | 5 +++++ arch/x86/kernel/setup.c | 18 ++++++++++++++---- 4 files changed, 32 insertions(+), 5 deletions(-)Signature · 1 line
2.17.2
Quoted 47 lines · Baoquan He
This is v3 post.
The original v1 post can be found here:
http://lkml.kernel.org/r/[email protected]Later a v1 RESEND version:
http://lkml.kernel.org/r/[email protected]V2 post is here:
http://lkml.kernel.org/r/[email protected]This patchset is trying to fix several issues for kexec/kdump when dynamic switching of paging mode is enabled in
x86_64. The current kernel supports 5-level paging mode, and supports dynamically choosing paging mode during bootup according to kernel image, hardware and kernel parameter setting. This flexibility brings several issues for kexec/kdump:Issues:
1)
Dynamic switching between paging mode requires code change in target kernel. So we can't kexec jump from 5-level kernel to old 4-level kernel which lacks the code change.2)
Switching from 5-level paging to 4-level paging kernel would fail, ifkexec()put kernel image above 64TiB of memory.3)
Kdump jumping has similar issue as 2). This require us to only reserve crashkernel below 64TB, otherwise jumping from 5-level to 4-level kernel will fail.Note:
Since we have two interfaceskexec_load()andkexec_file_load()to load kexec/kdump kernel, handling for them is a little different. Forkexec_load(), most of the loading job is done in user space utilitykexec_tools. However, forkexec_file_load(), most of the loading codes have moved into kernel because of kernel image verification.Fixes:
a) For issue 1), we need check ifXLF_5LEVELis set, otherwise error out a message.
-This need be done in both kernel andkexec_toolsutility. -Patch 2/3 is the handling of kernel part.
-Will post user space patch to kexec mailing list later.
Quoted 10 lines
b) For issue 2), we need check if both
XLF_5LEVELandXLF_5LEVEL_ENABLEDare set, otherwise error out a message.
-This only need be done inkexec_toolsutility. Because for
kexec_file_load(), the current code searches area to put kernel from bottom to up in system RAM, we usually can always find an area below 4 GB, no need to worry about 5-level kernel jumping to 4-level kernel. While forkexec_load(), it's top down seraching area for kernel loading, and implemented in user space. We need make sure that 5-level kernel find an area under 64 TB for a kexec-ed kernel of 4-level.
This paragraph is used to explain why XLF_5LEVEL_ENABLED is not checked in kernel. I will add these information to log of patch 1/3.
Quoted 50 lines
-Will post user space patch to kexec mailing list later.
c) For issues 3), just limit kernel to reserve crashkernel below 64 TB. -This only need be done in kernel.
-It doesn't need to check bitXLF_5LEVELorXLF_5LEVEL_ENABLED, we
just simply limit it below 64 TB which should be enough. Because crashernel is reserved during the 1st kernel's bootup, we don't know what kernel will be loaded for kdump usage.
-Patch 3/3 handles this.Concerns from reviewing comments:
1)
In v1, hpa raised concern that why the paging mode checking is not done before kexec jumping, the discussion can be found here:http://lkml.kernel.org/r/[email protected]
As tglx said, it might be not doable for kdump since kdump kernel's reserved crashkernel region only owns a portion of memory, may be above 4G; and might be not safer to do paging mode checking and switching thing after crash.
2)
In v1 RESEND post, tglx asked why only bitXLF_5LEVELis checked, even though two bitsXLF_5LEVELorXLF_5LEVEL_ENABLEDadded. So add more words to explain it in *Fixes* b).Changelog:
v2->v3:
Change the constant to match the notation for the rest of defines as Kirill suggested;v1->v2:
Correct the subject of patch 1 according to tglx's comment; Add more information to cover-letter to address reviewers' concerns;Baoquan He (3):
x86/boot: Add xloadflags bits for 5-level kernel checking x86/kexec/64: Error out if try to jump to old 4-level kernel from
5-level kernel
x86/kdump/64: Change the upper limit of crashkernel reservationarch/x86/boot/header.S | 12 +++++++++++- arch/x86/include/uapi/asm/bootparam.h | 2 ++ arch/x86/kernel/kexec-bzimage64.c | 5 +++++ arch/x86/kernel/setup.c | 18 ++++++++++++++---- 4 files changed, 32 insertions(+), 5 deletions(-)Signature · 1 line
2.17.2