Skip to contentSkip to conversation

Add restrictions for kexec/kdump jumping between 5-level and 4-level kernel

PATCH In review Baoquan He 2 participants 8 messages May 24, 2019 – Jun 28, 2019 x86

lore
Share & tools
lore

Baoquan He

author
#0 lore raw

v5 cover Add restrictions for kexec/kdump jumping between 5-level and 4-level kernel

The v4 cover letter tells the background about this adding, paste the link here for reference:
http://lkml.kernel.org/r/[email protected]

Changelog:
v4->v5:
Tune code and log per Dave's comments, no functional change.
- In patch 2, change the printed erorr message;
- In patch 3, add macro SZ_64T and use it in code, and remove the obsolete code comment.
v3->v4:
No functional change.
- Rewrite log of patch 1/3 tell who the newly added bits are gonna be used.
- Rewrite log of patch 2/3 per tglx's words.
- Add Kirill's Acked-by.

v2->v3:
Change the constant to match the notation for the rest of defines as Kirill suggested;
v1->v2:
Correct the subject of patch 1 according to tglx's comment; Add more information to cover-letter to address reviewers' concerns;

Baoquan He (3):
x86/boot: Add xloadflags bits for 5-level kernel checking x86/kexec/64: Error out if try to jump to old 4-level kernel from
5-level kernel
x86/kdump/64: Change the upper limit of crashkernel reservation

 arch/x86/boot/header.S                | 12 +++++++++++-
 arch/x86/include/uapi/asm/bootparam.h |  2 ++
 arch/x86/kernel/kexec-bzimage64.c     |  5 +++++
 arch/x86/kernel/setup.c               | 15 ++++++++++++---
 include/linux/sizes.h                 |  1 +
 5 files changed, 31 insertions(+), 4 deletions(-)
Signature · 1 line
2.17.2
author
#1 lore raw

PATCH v5 1/3 x86/boot: Add xloadflags bits for 5-level kernel checking

The current kernel supports 5-level paging mode, and supports dynamically choosing paging mode during bootup according to kernel image, hardware and kernel parameter setting. This flexibility brings several issues to kexec/kdump:
1)
Dynamic switching between paging modes requires code change in target kernel. So we can't do kexec-jumping from 5-level kernel to old 4-level kernel which lacks the code change.

2)
Kexec jumping from 5-level kernel to 4-level kernel would fail, if kexec loading puts kernel image above 64 TB of memory. Because kexec loading searches area to put kernel from top to down in system RAM, the 2nd kernel will be loaded above 64 TB if the amount of system RAM is bigger than 64 TB. Here no need to worry about kexec_file loading, because it searches area from bottom to up, and can always find area below 4 GB.

Solution:

Add two bits XLF_5LEVEL and XLF_5LEVEL_ENABLED for 5-level kernel.
- Bit XLF_5LEVEL indicates if 5-level related code is contained in this kernel.
- Bit XLF_5LEVEL_ENABLED indicates if CONFIG_X86_5LEVEL=y is set.

a) For issue 1), need check if XLF_5LEVEL is set, otherwise print out error message.
- This checking need be added in kernel, for the code of kexec_file

    loading;

- And also need be added into kexec_tools utility, for kexec loading.

b) For issue 2), need check if both XLF_5LEVEL and XLF_5LEVEL_ENABLED are set, otherwise print out error message.
- This only need be done in kexec_tools utility, because kexec loading does the searching in user space kexec_tools.

So here add XLF_5LEVEL and XLF_5LEVEL_ENABLED into xloadflags. The later code will check XLF_5LEVEL bit in kexec_file implementation of kernel. And the kexec_tools code will check both XLF_5LEVEL and XLF_5LEVEL_ENABLED for kexec loading.

---
 arch/x86/boot/header.S                | 12 +++++++++++-
 arch/x86/include/uapi/asm/bootparam.h |  2 ++
 2 files changed, 13 insertions(+), 1 deletion(-)
arch/x86/boot/header.S +11 -1
index 850b8762e889..be19f4199727 100644
@@ -419,7 +419,17 @@ xloadflags:
 # define XLF4 0
 #endif
 
-			.word XLF0 | XLF1 | XLF23 | XLF4
+#ifdef CONFIG_X86_64
+#ifdef CONFIG_X86_5LEVEL
+#define XLF56 (XLF_5LEVEL|XLF_5LEVEL_ENABLED)
+#else
+#define XLF56 XLF_5LEVEL
+#endif
+#else
+#define XLF56 0
+#endif
+
+			.word XLF0 | XLF1 | XLF23 | XLF4 | XLF56
 
 cmdline_size:   .long   COMMAND_LINE_SIZE-1     #length of the command line,
                                                 #added with boot protocol
arch/x86/include/uapi/asm/bootparam.h +2 -0
index 60733f137e9a..c895df5482c5 100644
@@ -29,6 +29,8 @@
 #define XLF_EFI_HANDOVER_32		(1<<2)
 #define XLF_EFI_HANDOVER_64		(1<<3)
 #define XLF_EFI_KEXEC			(1<<4)
+#define XLF_5LEVEL			(1<<5)
+#define XLF_5LEVEL_ENABLED		(1<<6)
 
 #ifndef __ASSEMBLY__
 
Signature · 1 line
2.17.2
#2 lore raw

[tip:x86/boot] x86/boot: Add xloadflags bits to check for 5-level paging support

x86/boot: Add xloadflags bits to check for 5-level paging support
Commit-ID:  f2d08c5d3bcf3f7ef788af122b57a919efa1e9d0
Gitweb:     https://git.kernel.org/tip/f2d08c5d3bcf3f7ef788af122b57a919efa1e9d0
Author:     Baoquan He <[email protected]>
AuthorDate: Fri, 24 May 2019 15:38:08 +0800
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Fri, 28 Jun 2019 07:14:59 +0200

x86/boot: Add xloadflags bits to check for 5-level paging support

The current kernel supports 5-level paging mode, and supports dynamically choosing the paging mode during bootup depending on the kernel image, hardware and kernel parameter settings. This flexibility brings several issues to kexec/kdump:

1) Dynamic switching between paging modes requires support in the target kernel. This means kexec from a 5-level paging kernel into a kernel which does not support mode switching is not possible. So the loader needs to be able to analyze the supported paging modes of the kexec target kernel.

2) If running on a 5-level paging kernel and the kexec target kernel is a 4-level paging kernel, the target immage cannot be loaded above the 64TB address space limit. But the kexec loader searches for a load area from top to bottom which would eventually put the target kernel above 64TB when the machine has large enough RAM size. So the loader needs to be able to analyze the paging mode of the target kernel to load it at a suitable spot in the address space.

Solution:

Add two bits XLF_5LEVEL and XLF_5LEVEL_ENABLED:

- Bit XLF_5LEVEL indicates whether 5-level paging mode switching support is available. (Issue #1)

- Bit XLF_5LEVEL_ENABLED indicates whether the kernel was compiled with full 5-level paging support (CONFIG_X86_5LEVEL=y). (Issue #2)

The loader will use these bits to verify whether the target kernel is suitable to be kexec'ed to from a 5-level paging kernel and to determine the constraints of the target kernel load address.

The flags will be used by the kernel kexec subsystem and the userspace kexec tools.

[ tglx: Massaged changelog ]

---
 arch/x86/boot/header.S                | 12 +++++++++++-
 arch/x86/include/uapi/asm/bootparam.h |  2 ++
 2 files changed, 13 insertions(+), 1 deletion(-)
arch/x86/boot/header.S +11 -1
index 850b8762e889..be19f4199727 100644
@@ -419,7 +419,17 @@ xloadflags:
 # define XLF4 0
 #endif
 
-			.word XLF0 | XLF1 | XLF23 | XLF4
+#ifdef CONFIG_X86_64
+#ifdef CONFIG_X86_5LEVEL
+#define XLF56 (XLF_5LEVEL|XLF_5LEVEL_ENABLED)
+#else
+#define XLF56 XLF_5LEVEL
+#endif
+#else
+#define XLF56 0
+#endif
+
+			.word XLF0 | XLF1 | XLF23 | XLF4 | XLF56
 
 cmdline_size:   .long   COMMAND_LINE_SIZE-1     #length of the command line,
                                                 #added with boot protocol
arch/x86/include/uapi/asm/bootparam.h +2 -0
index 60733f137e9a..c895df5482c5 100644
@@ -29,6 +29,8 @@
 #define XLF_EFI_HANDOVER_32		(1<<2)
 #define XLF_EFI_HANDOVER_64		(1<<3)
 #define XLF_EFI_KEXEC			(1<<4)
+#define XLF_5LEVEL			(1<<5)
+#define XLF_5LEVEL_ENABLED		(1<<6)
 
 #ifndef __ASSEMBLY__
 
author
#3 lore raw

PATCH v5 2/3 x86/kexec/64: Error out if try to jump to old 4-level kernel from 5-level kernel

In reply to Baoquan He

If the running kernel has 5-level paging activated, the 5-level paging mode is preserved across kexec. If the kexec'ed kernel does not contain support for handling active 5-level paging mode in the decompressor, the decompressor will crash with #GP.

Prevent this situation at load time. If 5-level paging is active, check the xloadflags whether the kexec kernel can handle 5-level paging at least in the decompressor. If not, reject the load attempt and print out error message.

---
 arch/x86/kernel/kexec-bzimage64.c | 5 +++++
 1 file changed, 5 insertions(+)
arch/x86/kernel/kexec-bzimage64.c +5 -0
index 22f60dd26460..7f439739ea3d 100644
@@ -321,6 +321,11 @@ static int bzImage64_probe(const char *buf, unsigned long len)
 		return ret;
 	}
 
+	if (!(header->xloadflags & XLF_5LEVEL) && pgtable_l5_enabled()) {
+		pr_err("bzImage cannot handle 5-level paging mode.\n");
+		return ret;
+	}
+
 	/* I've got a bzImage */
 	pr_debug("It's a relocatable bzImage64\n");
 	ret = 0;
Signature · 1 line
2.17.2
#4 lore raw

[tip:x86/boot] x86/kexec/64: Prevent kexec from 5-level paging to a 4-level only kernel

x86/kexec/64: Prevent kexec from 5-level paging to a 4-level only kernel
Commit-ID:  ee338b9ee2822e65a85750da6129946c14962410
Gitweb:     https://git.kernel.org/tip/ee338b9ee2822e65a85750da6129946c14962410
Author:     Baoquan He <[email protected]>
AuthorDate: Fri, 24 May 2019 15:38:09 +0800
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Fri, 28 Jun 2019 07:14:59 +0200

x86/kexec/64: Prevent kexec from 5-level paging to a 4-level only kernel

If the running kernel has 5-level paging activated, the 5-level paging mode is preserved across kexec. If the kexec'ed kernel does not contain support for handling active 5-level paging mode in the decompressor, the decompressor will crash with #GP.

Prevent this situation at load time. If 5-level paging is active, check the xloadflags whether the kexec kernel can handle 5-level paging at least in the decompressor. If not, reject the load attempt and print out an error message.

---
 arch/x86/kernel/kexec-bzimage64.c | 5 +++++
 1 file changed, 5 insertions(+)
arch/x86/kernel/kexec-bzimage64.c +5 -0
index 22f60dd26460..7f439739ea3d 100644
@@ -321,6 +321,11 @@ static int bzImage64_probe(const char *buf, unsigned long len)
 		return ret;
 	}
 
+	if (!(header->xloadflags & XLF_5LEVEL) && pgtable_l5_enabled()) {
+		pr_err("bzImage cannot handle 5-level paging mode.\n");
+		return ret;
+	}
+
 	/* I've got a bzImage */
 	pr_debug("It's a relocatable bzImage64\n");
 	ret = 0;
author
#5 lore raw

PATCH v5 3/3 x86/kdump/64: Change the upper limit of crashkernel reservation

In reply to Baoquan He

Restrict kdump to only reserve crashkernel below 64TB.

The reaons is that the kdump may jump from 5-level to 4-level, and if the kdump kernel is put above 64TB, then the jumping will fail. While the 1st kernel reserves crashkernel region during bootup, we don't know yet which kind of kernel will be loaded after system bootup, 5-level kernel or 4-level kernel.

---
 arch/x86/kernel/setup.c | 15 ++++++++++++---
 include/linux/sizes.h   |  1 +
 2 files changed, 13 insertions(+), 3 deletions(-)
arch/x86/kernel/setup.c +12 -3
index 08a5f4a131f5..dcbdf54fb5c1 100644
@@ -453,15 +453,24 @@ static void __init memblock_x86_reserve_range_setup_data(void)
 #define CRASH_ALIGN		SZ_16M
 
 /*
- * Keep the crash kernel below this limit.  On 32 bits earlier kernels
- * would limit the kernel to the low 512 MiB due to mapping restrictions.
+ * Keep the crash kernel below this limit.
+ *
+ * On 32 bits earlier kernels would limit the kernel to the low 512 MiB
+ * due to mapping restrictions.
+ *
+ * On 64bit, kdump kernel need be restricted to be under 64TB, which is
+ * the upper limit of system RAM in 4-level paing mode. Since the kdump
+ * jumping could be from 5-level to 4-level, the jumping will fail if
+ * kernel is put above 64TB, and there's no way to detect the paging mode
+ * of the kernel which will be loaded for dumping during the 1st kernel
+ * bootup.
  */
 #ifdef CONFIG_X86_32
 # define CRASH_ADDR_LOW_MAX	SZ_512M
 # define CRASH_ADDR_HIGH_MAX	SZ_512M
 #else
 # define CRASH_ADDR_LOW_MAX	SZ_4G
-# define CRASH_ADDR_HIGH_MAX	MAXMEM
+# define CRASH_ADDR_HIGH_MAX	SZ_64T
 #endif
 
 static int __init reserve_crashkernel_low(void)
include/linux/sizes.h +1 -0
index fbde0bc7e882..8651269cb46c 100644
@@ -47,5 +47,6 @@
 #define SZ_2G				0x80000000
 
 #define SZ_4G				_AC(0x100000000, ULL)
+#define SZ_64T				_AC(0x400000000000, ULL)
 
 #endif /* __LINUX_SIZES_H__ */
Signature · 1 line
2.17.2
#6 lore raw

[tip:x86/boot] x86/kdump/64: Restrict kdump kernel reservation to <64TB

x86/kdump/64: Restrict kdump kernel reservation to <64TB
Commit-ID:  8ff80fbe7e9870078b1cc3c2cdd8f3f223b333a9
Gitweb:     https://git.kernel.org/tip/8ff80fbe7e9870078b1cc3c2cdd8f3f223b333a9
Author:     Baoquan He <[email protected]>
AuthorDate: Fri, 24 May 2019 15:38:10 +0800
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Fri, 28 Jun 2019 07:14:59 +0200

x86/kdump/64: Restrict kdump kernel reservation to <64TB

Restrict kdump to only reserve crashkernel below 64TB.

The reaons is that the kdump may jump from a 5-level paging mode to a 4-level paging mode kernel. If a 4-level paging mode kdump kernel is put above 64TB, then the kdump kernel cannot start.

The 1st kernel reserves the kdump kernel region during bootup. At that point it is not known whether the kdump kernel has 5-level or 4-level paging support.

To support both restrict the kdump kernel reservation to the lower 64TB address space to ensure that a 4-level paging mode kdump kernel can be loaded and successfully started.

[ tglx: Massaged changelog ]

---
 arch/x86/kernel/setup.c | 15 ++++++++++++---
 include/linux/sizes.h   |  1 +
 2 files changed, 13 insertions(+), 3 deletions(-)
arch/x86/kernel/setup.c +12 -3
index 08a5f4a131f5..dcbdf54fb5c1 100644
@@ -453,15 +453,24 @@ static void __init memblock_x86_reserve_range_setup_data(void)
 #define CRASH_ALIGN		SZ_16M
 
 /*
- * Keep the crash kernel below this limit.  On 32 bits earlier kernels
- * would limit the kernel to the low 512 MiB due to mapping restrictions.
+ * Keep the crash kernel below this limit.
+ *
+ * On 32 bits earlier kernels would limit the kernel to the low 512 MiB
+ * due to mapping restrictions.
+ *
+ * On 64bit, kdump kernel need be restricted to be under 64TB, which is
+ * the upper limit of system RAM in 4-level paing mode. Since the kdump
+ * jumping could be from 5-level to 4-level, the jumping will fail if
+ * kernel is put above 64TB, and there's no way to detect the paging mode
+ * of the kernel which will be loaded for dumping during the 1st kernel
+ * bootup.
  */
 #ifdef CONFIG_X86_32
 # define CRASH_ADDR_LOW_MAX	SZ_512M
 # define CRASH_ADDR_HIGH_MAX	SZ_512M
 #else
 # define CRASH_ADDR_LOW_MAX	SZ_4G
-# define CRASH_ADDR_HIGH_MAX	MAXMEM
+# define CRASH_ADDR_HIGH_MAX	SZ_64T
 #endif
 
 static int __init reserve_crashkernel_low(void)
include/linux/sizes.h +1 -0
index fbde0bc7e882..8651269cb46c 100644
@@ -47,5 +47,6 @@
 #define SZ_2G				0x80000000
 
 #define SZ_4G				_AC(0x100000000, ULL)
+#define SZ_64T				_AC(0x400000000000, ULL)
 
 #endif /* __LINUX_SIZES_H__ */
author
#7 lore raw
In reply to Baoquan He

Hi,

Quoted 1 line · Baoquan He

Ping.

Can anyone help do further reviewing on this patchset? Or consider merging since people have ack-ed?

Thanks
Baoquan

Quoted 40 lines

The v4 cover letter tells the background about this adding, paste the link here for reference:
http://lkml.kernel.org/r/[email protected]

Changelog:
v4->v5:
Tune code and log per Dave's comments, no functional change.
- In patch 2, change the printed erorr message;
- In patch 3, add macro SZ_64T and use it in code, and remove the obsolete code comment.
v3->v4:
No functional change.
- Rewrite log of patch 1/3 tell who the newly added bits are gonna be used.
- Rewrite log of patch 2/3 per tglx's words.
- Add Kirill's Acked-by.

v2->v3:
Change the constant to match the notation for the rest of defines as Kirill suggested;
v1->v2:
Correct the subject of patch 1 according to tglx's comment; Add more information to cover-letter to address reviewers' concerns;

Baoquan He (3):
x86/boot: Add xloadflags bits for 5-level kernel checking x86/kexec/64: Error out if try to jump to old 4-level kernel from
5-level kernel
x86/kdump/64: Change the upper limit of crashkernel reservation

 arch/x86/boot/header.S                | 12 +++++++++++-
 arch/x86/include/uapi/asm/bootparam.h |  2 ++
 arch/x86/kernel/kexec-bzimage64.c     |  5 +++++
 arch/x86/kernel/setup.c               | 15 ++++++++++++---
 include/linux/sizes.h                 |  1 +
 5 files changed, 31 insertions(+), 4 deletions(-)
Signature · 1 line
2.17.2