Skip to contentSkip to conversation

Add restrictions for kexec/kdump jumping between 5-level and 4-level kernel

PATCH Superseded Baoquan He 2 participants 6 messages Mar 12, 2019 x86

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Baoquan He

author
#0 lore raw

v2 cover Add restrictions for kexec/kdump jumping between 5-level and 4-level kernel

This is v2 post.

The original v1 post can be found here:
http://lkml.kernel.org/r/[email protected]

Later a v1 RESEND version:
http://lkml.kernel.org/r/[email protected]

This patchset is trying to fix several issues for kexec/kdump when dynamic switching of paging mode is enabled in x86_64. The current kernel supports 5-level paging mode, and supports dynamically choosing paging mode during bootup according to kernel image, hardware and kernel parameter setting. This flexibility brings several issues for kexec/kdump:

Issues:
1)
Dynamic switching between paging mode requires code change in target kernel. So we can't kexec jump from 5-level kernel to old 4-level kernel which lacks the code change.

2)
Switching from 5-level paging to 4-level paging kernel would fail, if kexec() put kernel image above 64TiB of memory.

3)
Kdump jumping has similar issue as 2). This require us to only reserve crashkernel below 64TB, otherwise jumping from 5-level to 4-level kernel will fail.

Note:
Since we have two interfaces kexec_load() and kexec_file_load() to load kexec/kdump kernel, handling for them is a little different. For kexec_load(), most of the loading job is done in user space utility kexec_tools. However, for kexec_file_load(), most of the loading codes have moved into kernel because of kernel image verification.

Fixes:
a) For issue 1), we need check if XLF_5LEVEL is set, otherwise error out a message.
-This need be done in both kernel and kexec_tools utility. -Patch 2/3 is the handling of kernel part.
-Will post user space patch to kexec mailing list later.

b) For issue 2), we need check if both XLF_5LEVEL and XLF_5LEVEL_ENABLED are set, otherwise error out a message.
-This only need be done in kexec_tools utility. Because for
kexec_file_load(), the current code searches area to put kernel from bottom to up in system RAM, we usually can always find an area below 4 GB, no need to worry about 5-level kernel jumping to 4-level kernel. While for kexec_load(), it's top down seraching area for kernel loading, and implemented in user space. We need make sure that 5-level kernel find an area under 64 TB for a kexec-ed kernel of 4-level.
-Will post user space patch to kexec mailing list later.

c) For issues 3), just limit kernel to reserve crashkernel below 64 TB. -This only need be done in kernel.
-It doesn't need to check bit XLF_5LEVEL or XLF_5LEVEL_ENABLED, we
just simply limit it below 64 TB which should be enough. Because crashernel is reserved during the 1st kernel's bootup, we don't know what kernel will be loaded for kdump usage.
-Patch 3/3 handles this.

Concerns from reviewing comments:
1)
In v1, hpa raised concern that why the paging mode checking is not done before kexec jumping, the discussion can be found here:

http://lkml.kernel.org/r/[email protected]

As tglx said, it might be not doable for kdump since kdump kernel's reserved crashkernel region only owns a portion of memory, may be above 4G; and might be not safer to do paging mode checking and switching thing after crash.

2)
In v1 RESEND post, tglx asked why only bit XLF_5LEVEL is checked, even though two bits XLF_5LEVEL or XLF_5LEVEL_ENABLED. So add sentences to explain this in *Fixes* b).

v1->v2:
Correct the subject of patch 1 according to tglx's comment; Add more information to cover-letter to address reviewers' concerns;

Baoquan He (3):
x86/boot: Add xloadflags bits for 5-level kernel checking x86/kexec/64: Error out if try to jump to old 4-level kernel from
5-level kernel
x86/kdump/64: Change the upper limit of crashkernel reservation

 arch/x86/boot/header.S                | 12 +++++++++++-
 arch/x86/include/uapi/asm/bootparam.h |  2 ++
 arch/x86/kernel/kexec-bzimage64.c     |  5 +++++
 arch/x86/kernel/setup.c               | 18 ++++++++++++++----
 4 files changed, 32 insertions(+), 5 deletions(-)
Signature · 1 line
2.17.2
author
#1 lore raw

PATCH v2 1/3 x86/boot: Add xloadflags bits for 5-level kernel checking

Add two bits XLF_5LEVEL and XLF_5LEVEL_ENABLED for 5-level kernel. Bit XLF_5LEVEL indicates if 5-level related code is contained in this kernel.
Bit XLF_5LEVEL_ENABLED indicates if CONFIG_X86_5LEVEL=y is set.

They are being used in later patch to check if kexec/kdump kernel is loaded in right place.

---
 arch/x86/boot/header.S                | 12 +++++++++++-
 arch/x86/include/uapi/asm/bootparam.h |  2 ++
 2 files changed, 13 insertions(+), 1 deletion(-)
arch/x86/boot/header.S +11 -1
index 850b8762e889..be19f4199727 100644
@@ -419,7 +419,17 @@ xloadflags:
 # define XLF4 0
 #endif
 
-			.word XLF0 | XLF1 | XLF23 | XLF4
+#ifdef CONFIG_X86_64
+#ifdef CONFIG_X86_5LEVEL
+#define XLF56 (XLF_5LEVEL|XLF_5LEVEL_ENABLED)
+#else
+#define XLF56 XLF_5LEVEL
+#endif
+#else
+#define XLF56 0
+#endif
+
+			.word XLF0 | XLF1 | XLF23 | XLF4 | XLF56
 
 cmdline_size:   .long   COMMAND_LINE_SIZE-1     #length of the command line,
                                                 #added with boot protocol
arch/x86/include/uapi/asm/bootparam.h +2 -0
index 60733f137e9a..c895df5482c5 100644
@@ -29,6 +29,8 @@
 #define XLF_EFI_HANDOVER_32		(1<<2)
 #define XLF_EFI_HANDOVER_64		(1<<3)
 #define XLF_EFI_KEXEC			(1<<4)
+#define XLF_5LEVEL			(1<<5)
+#define XLF_5LEVEL_ENABLED		(1<<6)
 
 #ifndef __ASSEMBLY__
 
Signature · 1 line
2.17.2
author
#2 lore raw

PATCH v2 2/3 x86/kexec/64: Error out if try to jump to old 4-level kernel from 5-level kernel

In reply to Baoquan He

In relocate_kernel() CR4.LA57 flag is set before kexec jumping if the kernel has 5-level paging enabled. Then in boot/compressed/head_64.S, it will check if the booting kernel is in 4-level or 5-level paging mode, and handle accordingly. However, the old kernel which doesn't contain the 5-level codes doesn't know how to cope with it, then #GP triggered.

Instead of triggering #GP during kexec kernel boot, error out during kexec loading if find out we are trying to jump to old 4-level kernel from 5-level kernel.

---
 arch/x86/kernel/kexec-bzimage64.c | 5 +++++
 1 file changed, 5 insertions(+)
arch/x86/kernel/kexec-bzimage64.c +5 -0
index 1f3b77367948..4c9c079b5673 100644
@@ -321,6 +321,11 @@ static int bzImage64_probe(const char *buf, unsigned long len)
 		return ret;
 	}
 
+	if (!(header->xloadflags & XLF_5LEVEL) && pgtable_l5_enabled()) {
+		pr_err("Can not jump to old 4-level kernel from 5-level kernel.\n");
+		return ret;
+	}
+
 	/* I've got a bzImage */
 	pr_debug("It's a relocatable bzImage64\n");
 	ret = 0;
Signature · 1 line
2.17.2
author
#3 lore raw

PATCH v2 3/3 x86/kdump/64: Change the upper limit of crashkernel reservation

In reply to Baoquan He

Restrict kdump to only reserve crashkernel below 64TB. Since the kdump jumping may be from 5-level to 4-level, and the kdump kernel is put above 64TB in 5-level kernel, then the jumping will fail. And the crashkernel reservation is done during the 1st kernel bootup, there's no way to detect the paging mode of kdump kernel at that time.

Hence change the upper limit of crashkernel reservation to 64TB on x86_64.

---
 arch/x86/kernel/setup.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
arch/x86/kernel/setup.c +14 -4
index 3d872a527cd9..d4d1366738a4 100644
@@ -451,16 +451,26 @@ static void __init memblock_x86_reserve_range_setup_data(void)
 #define CRASH_ALIGN		(16 << 20)
 
 /*
- * Keep the crash kernel below this limit.  On 32 bits earlier kernels
- * would limit the kernel to the low 512 MiB due to mapping restrictions.
- * On 64bit, old kexec-tools need to under 896MiB.
+ * Keep the crash kernel below this limit.
+ *
+ * On 32 bits earlier kernels would limit the kernel to the low
+ * 512 MiB due to mapping restrictions.
+ *
+ * On 64bit, old kexec-tools need to be under 896MiB. The later
+ * supports to put kernel above 4G, up to system RAM top. Here
+ * kdump kernel need be restricted to be under 64TB, which is
+ * the upper limit of system RAM in 4-level paing mode. Since
+ * the kdump jumping could be from 5-level to 4-level, the jumping
+ * will fail if kernel is put above 64TB, and there's no way to
+ * detect the paging mode of the kernel which will be loaded for
+ * dumping during the 1st kernel bootup.
  */
 #ifdef CONFIG_X86_32
 # define CRASH_ADDR_LOW_MAX	(512 << 20)
 # define CRASH_ADDR_HIGH_MAX	(512 << 20)
 #else
 # define CRASH_ADDR_LOW_MAX	(896UL << 20)
-# define CRASH_ADDR_HIGH_MAX	MAXMEM
+# define CRASH_ADDR_HIGH_MAX	(1ULL < 46)
 #endif
 
 static int __init reserve_crashkernel_low(void)
Signature · 1 line
2.17.2
Acked-by on v2 3/3
#4 lore raw

Re: [PATCH v2 3/3] x86/kdump/64: Change the upper limit of crashkernel reservation

Quoted 36 lines · Baoquan He

Restrict kdump to only reserve crashkernel below 64TB. Since the kdump jumping may be from 5-level to 4-level, and the kdump kernel is put above 64TB in 5-level kernel, then the jumping will fail. And the crashkernel reservation is done during the 1st kernel bootup, there's no way to detect the paging mode of kdump kernel at that time.

Hence change the upper limit of crashkernel reservation to 64TB on x86_64.

---
 arch/x86/kernel/setup.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
arch/x86/kernel/setup.c +11 -3
index 3d872a527cd9..d4d1366738a4 100644
@@ -451,16 +451,26 @@ static void __init memblock_x86_reserve_range_setup_data(void)
 #define CRASH_ALIGN		(16 << 20)
 
 /*
- * Keep the crash kernel below this limit.  On 32 bits earlier kernels
- * would limit the kernel to the low 512 MiB due to mapping restrictions.
- * On 64bit, old kexec-tools need to under 896MiB.
+ * Keep the crash kernel below this limit.
+ *
+ * On 32 bits earlier kernels would limit the kernel to the low
+ * 512 MiB due to mapping restrictions.
+ *
+ * On 64bit, old kexec-tools need to be under 896MiB. The later
+ * supports to put kernel above 4G, up to system RAM top. Here
+ * kdump kernel need be restricted to be under 64TB, which is
+ * the upper limit of system RAM in 4-level paing mode. Since
+ * the kdump jumping could be from 5-level to 4-level, the jumping
+ * will fail if kernel is put above 64TB, and there's no way to

arch/x86/kernel/setup.c static void __init memblock_x86_reserve_range_setup_data(void) · line 473

Quoted 10 lines
arch/x86/kernel/setup.c +3 -1
+ * detect the paging mode of the kernel which will be loaded for
+ * dumping during the 1st kernel bootup.
  */
 #ifdef CONFIG_X86_32
 # define CRASH_ADDR_LOW_MAX	(512 << 20)
 # define CRASH_ADDR_HIGH_MAX	(512 << 20)
 #else
 # define CRASH_ADDR_LOW_MAX	(896UL << 20)
-# define CRASH_ADDR_HIGH_MAX	MAXMEM
+# define CRASH_ADDR_HIGH_MAX	(1ULL < 46)

Maybe

# define CRASH_ADDR_HIGH_MAX (64UL << 40)

to match the notation for the rest of defines?

Otherwise the patchset look good to me. You can use my ack for all of them:

Quoted 6 lines
arch/x86/kernel/setup.c +0 -0
 #endif
 
 static int __init reserve_crashkernel_low(void)
Signature · 1 line
2.17.2
Signature · 1 line
 Kirill A. Shutemov
author
#5 lore raw

Re: [PATCH v2 3/3] x86/kdump/64: Change the upper limit of crashkernel reservation

Quoted 10 lines · Kirill A. Shutemov

arch/x86/kernel/setup.c static void __init memblock_x86_reserve_range_setup_data(void) · line 473

Quoted 7 lines

#ifdef CONFIG_X86_32

 # define CRASH_ADDR_LOW_MAX	(512 << 20)
 # define CRASH_ADDR_HIGH_MAX	(512 << 20)
 #else
 # define CRASH_ADDR_LOW_MAX	(896UL << 20)
arch/x86/kernel/setup.c +1 -1
-# define CRASH_ADDR_HIGH_MAX	MAXMEM
+# define CRASH_ADDR_HIGH_MAX	(1ULL < 46)

Maybe

Quoted 4 lines
# define CRASH_ADDR_HIGH_MAX (64UL << 40)

to match the notation for the rest of defines?

Sure, I will replace it, and repost a whole patchset with your ack added on each patch. Thanks for your careful reviewing and great suggestions.

Quoted 15 lines

Otherwise the patchset look good to me. You can use my ack for all of them:

Quoted 6 lines

#endif

static int __init reserve_crashkernel_low(void)

Signature · 1 line
2.17.2
Signature · 1 line
 Kirill A. Shutemov