Skip to contentSkip to conversation

Add restrictions for kexec/kdump jumping between 5-level and 4-level kernel

PATCH RESEND Superseded Baoquan He 3 participants 8 messages Jan 25, 2019 – Feb 11, 2019

Subsystems (3)

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Baoquan He

author
#0 lore raw

cover Add restrictions for kexec/kdump jumping between 5-level and 4-level kernel

This is a RESEND post.

The original v1 post can be found here:
http://lkml.kernel.org/r/[email protected]

It's trying to fix several corner case issues for kexec/kdump when dynamic switching of paging mode is enabled in x86_64. Please click above link to check the details.

In v1, hpa raised concern that why the paging mode checking is not done before kexec jumping, the discussion can be found here:

http://lkml.kernel.org/r/[email protected]

As tglx said, it might be not doable for kdump since kdump kernel's reserved crashkernel region only owns a portion of memory, may be above 4G; and might be not safer to do paging mode checking and switching thing after crash.

So resend this patchset.

Baoquan He (3):
x86/boot: Add bit fields into xloadflags for 5-level kernel checking x86/kexec/64: Error out if try to jump to old 4-level kernel from
5-level kernel
x86/kdump/64: Change the upper limit of crashkernel reservation

 arch/x86/boot/header.S                | 12 +++++++++++-
 arch/x86/include/uapi/asm/bootparam.h |  2 ++
 arch/x86/kernel/kexec-bzimage64.c     |  5 +++++
 arch/x86/kernel/setup.c               | 18 ++++++++++++++----
 4 files changed, 32 insertions(+), 5 deletions(-)
Signature · 1 line
2.13.6
author
#1 lore raw

PATCH RESEND 1/3 x86/boot: Add bit fields into xloadflags for 5-level kernel checking

Add two bit fields XLF_5LEVEL and XLF_5LEVEL_ENABLED for 5-level kernel. Bit XLF_5LEVEL indicates if 5-level related code is contained in this kernel.
Bit XLF_5LEVEL_ENABLED indicates if CONFIG_X86_5LEVEL=y is set.

They are being used in later patch to check if kexec/kdump kernel is loaded in right place.

---
 arch/x86/boot/header.S                | 12 +++++++++++-
 arch/x86/include/uapi/asm/bootparam.h |  2 ++
 2 files changed, 13 insertions(+), 1 deletion(-)
arch/x86/boot/header.S +11 -1
index 850b8762e889..be19f4199727 100644
@@ -419,7 +419,17 @@ xloadflags:
 # define XLF4 0
 #endif
 
-			.word XLF0 | XLF1 | XLF23 | XLF4
+#ifdef CONFIG_X86_64
+#ifdef CONFIG_X86_5LEVEL
+#define XLF56 (XLF_5LEVEL|XLF_5LEVEL_ENABLED)
+#else
+#define XLF56 XLF_5LEVEL
+#endif
+#else
+#define XLF56 0
+#endif
+
+			.word XLF0 | XLF1 | XLF23 | XLF4 | XLF56
 
 cmdline_size:   .long   COMMAND_LINE_SIZE-1     #length of the command line,
                                                 #added with boot protocol
arch/x86/include/uapi/asm/bootparam.h +2 -0
index 60733f137e9a..c895df5482c5 100644
@@ -29,6 +29,8 @@
 #define XLF_EFI_HANDOVER_32		(1<<2)
 #define XLF_EFI_HANDOVER_64		(1<<3)
 #define XLF_EFI_KEXEC			(1<<4)
+#define XLF_5LEVEL			(1<<5)
+#define XLF_5LEVEL_ENABLED		(1<<6)
 
 #ifndef __ASSEMBLY__
 
Signature · 7 lines
2.13.6


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#2 lore raw

Re: [PATCH RESEND 1/3] x86/boot: Add bit fields into xloadflags for 5-level kernel checking

Quoted 1 line · Baoquan He

Add two bit fields XLF_5LEVEL and XLF_5LEVEL_ENABLED for 5-level kernel.

These are not bit fields. These are simple bits.

Quoted 3 lines

Bit XLF_5LEVEL indicates if 5-level related code is contained in this kernel.
Bit XLF_5LEVEL_ENABLED indicates if CONFIG_X86_5LEVEL=y is set.

I'm confused.

arch/x86/boot/header.S xloadflags: · line 432

Quoted 12 lines
arch/x86/boot/header.S +11 -1
-			.word XLF0 | XLF1 | XLF23 | XLF4
+#ifdef CONFIG_X86_64
+#ifdef CONFIG_X86_5LEVEL
+#define XLF56 (XLF_5LEVEL|XLF_5LEVEL_ENABLED)
+#else
+#define XLF56 XLF_5LEVEL
+#endif
+#else
+#define XLF56 0
+#endif
+
+			.word XLF0 | XLF1 | XLF23 | XLF4 | XLF56

So this actually stores the bits, but looking at the following patch which fixes the real issue:

Review

Quoted 4 lines
(quoted diff) +4 -0
+	if (!(header->xloadflags & XLF_5LEVEL) && pgtable_l5_enabled()) {
+		pr_err("Can not jump to old 4-level kernel from 5-level kernel.\n");
+		return ret;
+	}

So what is XLF_5LEVEL_ENABLED used for and why does it exist at all?

Thanks,

tglx

author
#3 lore raw

Re: [PATCH RESEND 1/3] x86/boot: Add bit fields into xloadflags for 5-level kernel checking

Thanks for reviewing. I was in vacation, sorry for late reply.

Quoted 5 lines · Thomas Gleixner
Quoted 1 line · Baoquan He

Add two bit fields XLF_5LEVEL and XLF_5LEVEL_ENABLED for 5-level kernel.

These are not bit fields. These are simple bits.

Indeed, they are only xloadflags bits, will change. Thanks.

Quoted 23 lines
Quoted 4 lines

Bit XLF_5LEVEL indicates if 5-level related code is contained in this kernel.
Bit XLF_5LEVEL_ENABLED indicates if CONFIG_X86_5LEVEL=y is set.

I'm confused.

arch/x86/boot/header.S xloadflags: · line 432

Quoted 12 lines
arch/x86/boot/header.S +11 -1
-			.word XLF0 | XLF1 | XLF23 | XLF4
+#ifdef CONFIG_X86_64
+#ifdef CONFIG_X86_5LEVEL
+#define XLF56 (XLF_5LEVEL|XLF_5LEVEL_ENABLED)
+#else
+#define XLF56 XLF_5LEVEL
+#endif
+#else
+#define XLF56 0
+#endif
+
+			.word XLF0 | XLF1 | XLF23 | XLF4 | XLF56

So this actually stores the bits, but looking at the following patch which fixes the real issue:

Quoted 6 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+	if (!(header->xloadflags & XLF_5LEVEL) && pgtable_l5_enabled()) {
+		pr_err("Can not jump to old 4-level kernel from 5-level kernel.\n");
+		return ret;
+	}

So what is XLF_5LEVEL_ENABLED used for and why does it exist at all?

Yes, this is a little bit confusing. I explained it in the v1 cover letter:
http://lists.infradead.org/pipermail/kexec/2018-August/021419.html

As told at above, XLF_5LEVEL marks the new kernel containing 5level code, while XLF_5LEVEL_ENABLED marking the CONFIG_X86_5LEVEL option enabling. Hence if XLF_5LEVEL is set, XLF_5LEVEL_ENABLED not, means it's new kernel but can't be switched into 5-level.

For kexec_load and kexec_file_load, there's difference in loading behaviour. kexec_load will search available area top down to put kernel in system RAM, we need check if the kexec-ed kernel is in leve-5 paging mode, and limit the loading postion below 64 TB if not. But for kexec_file_load, it's searching area bottom up to put kernel, most of time area found below 4G. We don't have worry about the kexec_file_load interface which implements the loading functionality in kernel. That's why the XLF_5LEVEL_ENABLED bit is not used in this kernel patch set, I would like to post patch to kexec-tools for kexec_load after these patches have been accepted.

I ever tried to unify the behavious of these two interfaces on loading kernel, to make both kexec_load and kexec_file_load search and put kernel top to down, but that involves many lines of code change, seems people are worried about it and hesitated to offere ack, I just gave up. Please check below link:

https://lore.kernel.org/lkml/[email protected]/T/#u

Sorry for the inconvenience because of my missing explanation.

Thanks
Baoquan

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#4 lore raw

PATCH v2 1/3 x86/boot: Add xloadflags bits for 5-level kernel checking

In reply to Baoquan He

Add two bits XLF_5LEVEL and XLF_5LEVEL_ENABLED for 5-level kernel. Bit XLF_5LEVEL indicates if 5-level related code is contained in this kernel.
Bit XLF_5LEVEL_ENABLED indicates if CONFIG_X86_5LEVEL=y is set.

They are being used in later patch to check if kexec/kdump kernel is loaded in right place.

---
v1->v2:
Change the wrong term in subject pointed out by tglx.

 arch/x86/boot/header.S                | 12 +++++++++++-
 arch/x86/include/uapi/asm/bootparam.h |  2 ++
 2 files changed, 13 insertions(+), 1 deletion(-)
arch/x86/boot/header.S +11 -1
index 850b8762e889..be19f4199727 100644
@@ -419,7 +419,17 @@ xloadflags:
 # define XLF4 0
 #endif
 
-			.word XLF0 | XLF1 | XLF23 | XLF4
+#ifdef CONFIG_X86_64
+#ifdef CONFIG_X86_5LEVEL
+#define XLF56 (XLF_5LEVEL|XLF_5LEVEL_ENABLED)
+#else
+#define XLF56 XLF_5LEVEL
+#endif
+#else
+#define XLF56 0
+#endif
+
+			.word XLF0 | XLF1 | XLF23 | XLF4 | XLF56
 
 cmdline_size:   .long   COMMAND_LINE_SIZE-1     #length of the command line,
                                                 #added with boot protocol
arch/x86/include/uapi/asm/bootparam.h +2 -0
index 60733f137e9a..c895df5482c5 100644
@@ -29,6 +29,8 @@
 #define XLF_EFI_HANDOVER_32		(1<<2)
 #define XLF_EFI_HANDOVER_64		(1<<3)
 #define XLF_EFI_KEXEC			(1<<4)
+#define XLF_5LEVEL			(1<<5)
+#define XLF_5LEVEL_ENABLED		(1<<6)
 
 #ifndef __ASSEMBLY__
 
Signature · 7 lines
2.17.2


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
author
#5 lore raw

PATCH RESEND 2/3 x86/kexec/64: Error out if try to jump to old 4-level kernel from 5-level kernel

In reply to Baoquan He

In relocate_kernel() CR4.LA57 flag is set before kexec jumping if the kernel has 5-level paging enabled. Then in boot/compressed/head_64.S, it will check if the booting kernel is in 4-level or 5-level paging mode, and handle accordingly. However, the old kernel which doesn't contain the 5-level codes doesn't know how to cope with it, then #GP triggered.

Instead of triggering #GP during kexec kernel boot, error out during kexec loading if find out we are trying to jump to old 4-level kernel from 5-level kernel.

---
 arch/x86/kernel/kexec-bzimage64.c | 5 +++++
 1 file changed, 5 insertions(+)
arch/x86/kernel/kexec-bzimage64.c +5 -0
index 278cd07228dd..5bfb39fc56fe 100644
@@ -316,6 +316,11 @@ static int bzImage64_probe(const char *buf, unsigned long len)
 		return ret;
 	}
 
+	if (!(header->xloadflags & XLF_5LEVEL) && pgtable_l5_enabled()) {
+		pr_err("Can not jump to old 4-level kernel from 5-level kernel.\n");
+		return ret;
+	}
+
 	/* I've got a bzImage */
 	pr_debug("It's a relocatable bzImage64\n");
 	ret = 0;
Signature · 7 lines
2.13.6


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
author
#6 lore raw

PATCH RESEND 3/3 x86/kdump/64: Change the upper limit of crashkernel reservation

In reply to Baoquan He

Restrict kdump to only reserve crashkernel below 64TB. Since the kdump jumping may be from 5-level to 4-level, and the kdump kernel is put above 64TB in 5-level kernel, then the jumping will fail. And the crashkernel reservation is done during the 1st kernel bootup, there's no way to detect the paging mode of kdump kernel at that time.

Hence change the upper limit of crashkernel reservation to 64TB on x86_64.

---
 arch/x86/kernel/setup.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
arch/x86/kernel/setup.c +14 -4
index 3d872a527cd9..d4d1366738a4 100644
@@ -451,16 +451,26 @@ static void __init memblock_x86_reserve_range_setup_data(void)
 #define CRASH_ALIGN		(16 << 20)
 
 /*
- * Keep the crash kernel below this limit.  On 32 bits earlier kernels
- * would limit the kernel to the low 512 MiB due to mapping restrictions.
- * On 64bit, old kexec-tools need to under 896MiB.
+ * Keep the crash kernel below this limit.
+ *
+ * On 32 bits earlier kernels would limit the kernel to the low
+ * 512 MiB due to mapping restrictions.
+ *
+ * On 64bit, old kexec-tools need to be under 896MiB. The later
+ * supports to put kernel above 4G, up to system RAM top. Here
+ * kdump kernel need be restricted to be under 64TB, which is
+ * the upper limit of system RAM in 4-level paing mode. Since
+ * the kdump jumping could be from 5-level to 4-level, the jumping
+ * will fail if kernel is put above 64TB, and there's no way to
+ * detect the paging mode of the kernel which will be loaded for
+ * dumping during the 1st kernel bootup.
  */
 #ifdef CONFIG_X86_32
 # define CRASH_ADDR_LOW_MAX	(512 << 20)
 # define CRASH_ADDR_HIGH_MAX	(512 << 20)
 #else
 # define CRASH_ADDR_LOW_MAX	(896UL << 20)
-# define CRASH_ADDR_HIGH_MAX	MAXMEM
+# define CRASH_ADDR_HIGH_MAX	(1ULL < 46)
 #endif
 
 static int __init reserve_crashkernel_low(void)
Signature · 1 line
2.13.6
Acked-by on the v1 series
#7 lore raw
In reply to Baoquan He
Quoted 12 lines · Baoquan He

This is a RESEND post.

The original v1 post can be found here:
http://lkml.kernel.org/r/[email protected]

It's trying to fix several corner case issues for kexec/kdump when dynamic switching of paging mode is enabled in x86_64. Please click above link to check the details.

In v1, hpa raised concern that why the paging mode checking is not done before kexec jumping, the discussion can be found here:

Quoted 8 lines

http://lkml.kernel.org/r/[email protected]

As tglx said, it might be not doable for kdump since kdump kernel's reserved crashkernel region only owns a portion of memory, may be above 4G; and might be not safer to do paging mode checking and switching thing after crash.

So resend this patchset.

Changes look good to me:

Signature · 6 lines
 Kirill A. Shutemov

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec