Skip to contentSkip to conversation

kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

DISCUSSION Thomas D. 5 participants 21 messages Aug 17, 2014 – Aug 22, 2014

Subsystems (3)
lore
Share & tools
lore

Thomas D.

author
#0 lore raw

Hi,

looks like kexec doesn't support kernels where

CONFIG_RANDOMIZE_BASE=y

is set.

I already created a bug for Debian in June for this problem [1].

I am able to confirm the same problem with kexec 2.0.7 on Gentoo and when building from source (943ba35f8143408d8ada9a24d0986663cc612df9).

Is this a known problem/limitation?

Is there anything I can help you with to get this fixed?

Thanks.

See also:
=========
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=754288

-Thomas

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#1 lore raw

Hi Thomas,

I think kexec is broken with CONFIG_RANDOMIZE_BASE=y. Chao had raised this issue some time back when this option was introduced. I don't remember the details though that why it is broken.

CCing Kees Cook.

Kees, is there a quick fix to the issue. Sooner or later people will start enabling CONFIG_RANDOMIZE_BASE=y and broken kexec will be a big issue.

Thanks
Vivek

Quoted 26 lines · Thomas D.

Hi,

looks like kexec doesn't support kernels where

CONFIG_RANDOMIZE_BASE=y

is set.

I already created a bug for Debian in June for this problem [1].

I am able to confirm the same problem with kexec 2.0.7 on Gentoo and when building from source (943ba35f8143408d8ada9a24d0986663cc612df9).

Is this a known problem/limitation?

Is there anything I can help you with to get this fixed?

Thanks.

See also:
=========
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=754288

Quoted 6 lines

-Thomas

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#2 lore raw
Quoted 5 lines · Vivek Goyal

Hi Thomas,

I think kexec is broken with CONFIG_RANDOMIZE_BASE=y. Chao had raised this issue some time back when this option was introduced. I don't remember the details though that why it is broken.

The following fix the problem for kdump case:

commit 0d52644

Author: WANG Chao <[email protected]>
Date:   Fri Mar 28 15:05:00 2014 +0800

x86, kaslr: add alternative way to locate kernel text mapping area

For kexec case, it hangs in purgatory:

[  556.859384] kexec: Starting new kernel

I'm in purgatory

Quoted 43 lines

CCing Kees Cook.

Kees, is there a quick fix to the issue. Sooner or later people will start enabling CONFIG_RANDOMIZE_BASE=y and broken kexec will be a big issue.

Thanks
Vivek

Quoted 32 lines · Thomas D.

Hi,

looks like kexec doesn't support kernels where

CONFIG_RANDOMIZE_BASE=y

is set.

I already created a bug for Debian in June for this problem [1].

I am able to confirm the same problem with kexec 2.0.7 on Gentoo and when building from source (943ba35f8143408d8ada9a24d0986663cc612df9).

Is this a known problem/limitation?

Is there anything I can help you with to get this fixed?

Thanks.

See also:
=========
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=754288

-Thomas

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#3 lore raw
Quoted 13 lines · WANG Chao
Quoted 5 lines · Vivek Goyal

Hi Thomas,

I think kexec is broken with CONFIG_RANDOMIZE_BASE=y. Chao had raised this issue some time back when this option was introduced. I don't remember the details though that why it is broken.

The following fix the problem for kdump case:

commit 0d52644

Author: WANG Chao <[email protected]>
Date:   Fri Mar 28 15:05:00 2014 +0800
Quoted 7 lines

x86, kaslr: add alternative way to locate kernel text mapping area

For kexec case, it hangs in purgatory:

[  556.859384] kexec: Starting new kernel

I'm in purgatory

Chao,

Do you know why does it hang in purgatory in case of kexec?

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#4 lore raw
Quoted 7 lines · Vivek Goyal
Quoted 6 lines · WANG Chao
Quoted 5 lines · Vivek Goyal

Hi Thomas,

I think kexec is broken with CONFIG_RANDOMIZE_BASE=y. Chao had raised this issue some time back when this option was introduced. I don't remember the details though that why it is broken.

The "normal" problems with kaslr have to do with areas of memory that shouldn't be stomped on, or if 1-to-1 page tables are not in place. What state are the page tables in when doing the kexec, and how are kernel parameters (including e820) passed?

Quoted 8 lines
Quoted 8 lines

The following fix the problem for kdump case:

commit 0d52644

Author: WANG Chao <[email protected]>
Date:   Fri Mar 28 15:05:00 2014 +0800

x86, kaslr: add alternative way to locate kernel text mapping area

I don't see this in Linus's tree? Where can I find this commit?

Quoted 7 lines
Quoted 6 lines

For kexec case, it hangs in purgatory:

[  556.859384] kexec: Starting new kernel

I'm in purgatory

Quoted 6 lines

Chao,

Do you know why does it hang in purgatory in case of kexec?

Thanks
Vivek

-Kees

Signature · 7 lines
Kees Cook
Chrome OS Security

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#5 lore raw
Quoted 5 lines · Kees Cook
Quoted 4 lines · Vivek Goyal
Quoted 3 lines · WANG Chao
Quoted 2 lines · Vivek Goyal

Hi Thomas,

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I think kexec is broken with CONFIG_RANDOMIZE_BASE=y. Chao had raised this issue some time back when this option was introduced. I don't remember the details though that why it is broken.

The "normal" problems with kaslr have to do with areas of memory that shouldn't be stomped on, or if 1-to-1 page tables are not in place. What state are the page tables in when doing the kexec, and how are kernel parameters (including e820) passed?

Hi Kees,

I suspect that it has something to do with overwriting page tables or some other data by the new kernel.

IIUC, we are preparing identity mapped page tables for "nr_pfn_mapped".

arch/x86/kernel/machine_kexec_64.c

init_pgtable() {
        for (i = 0; i < nr_pfn_mapped; i++) {
                mstart = pfn_mapped[i].start << PAGE_SHIFT;
                mend   = pfn_mapped[i].end << PAGE_SHIFT; 

                result = kernel_ident_mapping_init(&info,
                                                 level4p, mstart, mend);
                if (result)
                        return result;
        }
}

So most likely page tables have been setup right. In fact, if one is trying to load bzimage32, then we drop to 32bit mode and disable paging. This new 64bit must be using the page tables setup by old kernel.

Mememory map (e820) and kernel parameters are passed in bootparams. This is that 4K page setup by old kernel.

So I have question. How does kASLR work. Previously x86_64 relocatable kernel will move itself to proper alignment boundary and then page tables will be updated properly. IIRC, virtual addresses reamined fixed and PAGE_OFFSET was not fixed.

Now with kASLR, are you moving kernel physically significantly or kernel is not moved physically just that its placement in virtual address is not fixed and it is chosen randomly?

If kernel is being moved physically, then we potentially have the issue of it stomping other things. So how do we make sure that it does not overwrite initramfs, or previous kernel's page tables or something else?

If kernel is not moving physically and only its location in virtual address space changes, then it is very puzzling that why it should be a problem.

If kernel always moves itself to higher addresses then one solution could be that load everything else below kernel and load kernel at higher addresses. But old kexec system call will not be able to cope with it as user space determines the load location of kernel and other segments while running kernel decides location of pages for page table and kernel has no idea where user space has loaded new kernel. New system call still might be able to handle it.

Also, I vaguely recall that there was a kernel parameter to disable kASLR. And kexec/kdump initially can use that paramter as a work around. What was that parameter.

Quoted 11 lines
Quoted 9 lines
Quoted 9 lines

The following fix the problem for kdump case:

commit 0d52644

Author: WANG Chao <[email protected]>
Date:   Fri Mar 28 15:05:00 2014 +0800

x86, kaslr: add alternative way to locate kernel text mapping area

I don't see this in Linus's tree? Where can I find this commit?

This is a kexec-tools patch and not kernel patch, that's why you don't see it in linus tree. kexec-tools has to prepare ELF headers for kernel text area. As with kASLR kernel text virtual addresses moved, kexec-tools had to be modified to look at /proc/kallsyms and look for symbol _stext to figure out where kernel text is and prepare ELF header accordingly.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#6 lore raw

On Thu, Aug 21, 2014 at 02:10:00PM -0400, Vivek Goyal wrote:

[..]

Quoted 3 lines

Also, I vaguely recall that there was a kernel parameter to disable kASLR. And kexec/kdump initially can use that paramter as a work around. What was that parameter.

I see that "nokaslr" will disable it at run time. I remember that we had discussions that it can be used as a work around. But I don't see kexec-tools or fedora kdump scripts appending "nokaslr" in kdump case. That's strange.

Chao, would you remember anything about it.

Thomas, can you please test kexec with "nokaslr" specified in command line. This can atleast get you going for now till the problem is actually fixed.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#9 lore raw
In reply to Vivek Goyal

On Thu, Aug 21, 2014 at 02:10:00PM -0400, Vivek Goyal wrote:

[..]

Quoted 7 lines

If kernel always moves itself to higher addresses then one solution could be that load everything else below kernel and load kernel at higher addresses. But old kexec system call will not be able to cope with it as user space determines the load location of kernel and other segments while running kernel decides location of pages for page table and kernel has no idea where user space has loaded new kernel. New system call still might be able to handle it.

So I am reading kaslr code a bit. First of all we are moving kernel in physical address space. That would substantiate the theory that kernel movement stepped onto something.

Basically we seem to be just going through all the e820 entries, putting them one slots[] array and choosing one entry randomly.

That means kernel could move up or down. So the notion of loading everything else below kernel will not work.

Now this makes me wonder that how does this whole thing work with grub. IOW, how would one make sure that kernel does not stomp initramfs.

I guess only protection is BASE_MAX_OFFSET and making sure initramfs is loaded higher than that.

If that's the case, then I think minimum and maximum range of physical memory where kernel can move should be exported through bzImage header and kexec-tools should make sure that anything else is outside of that range so that new kernel will not stomp over it.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#7 lore raw
In reply to Vivek Goyal

Hi,

Quoted 3 lines · Vivek Goyal

Thomas, can you please test kexec with "nokaslr" specified in command line. This can atleast get you going for now till the problem is actually fixed.

I booted linux-3.15.10 with "nokaslr"

Quoted 3 lines
 # dmesg | grep -i command
[    0.000000] Command line: BOOT_IMAGE=/kernel dolvm video=1280x1024 root=UUID=6d42... rootfs=ext4 nokaslr initrd=/initramfs
[    0.000000] Kernel command line: BOOT_IMAGE=/kernel dolvm video=1280x1024 root=UUID=6d42... rootfs=ext4 nokaslr initrd=/initramfs

and run kexec with

Quoted 1 line

--reuse-cmdline -l /boot/kernel root=/dev/dm-3 --initrd=/boot/initramfs

But it still doesn't reboot. I see no difference between using "nokaslr" or not.

-Thomas

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#10 lore raw
In reply to Vivek Goyal
Quoted 4 lines · Vivek Goyal

If kernel is being moved physically, then we potentially have the issue of it stomping other things. So how do we make sure that it does not overwrite initramfs, or previous kernel's page tables or something else?

In case you don't read that part, memory of initrd, cmdline and others won't be overwritten.

This work is done in x86/boot/compressed/aslr.c::mem_avoid_init().

Thanks
WANG Chao

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#11 lore raw
Quoted 10 lines · WANG Chao
Quoted 4 lines · Vivek Goyal

If kernel is being moved physically, then we potentially have the issue of it stomping other things. So how do we make sure that it does not overwrite initramfs, or previous kernel's page tables or something else?

In case you don't read that part, memory of initrd, cmdline and others won't be overwritten.

This work is done in x86/boot/compressed/aslr.c::mem_avoid_init().

Yeah, it's just as Chao said.

And I think it's still about the relocation calculation issue. I tried the patch Lu Yinghai suggested, now kexec can work.

But this patch only make kexec work. The slot choosing still doesn't work per the limitation checking, since only range below CONFIG_RANDOMIZE_BASE_MAX_OFFSET can be chosen and kexec-tools always likes putting kexec kernel from the end of system ram.

So we have 2 choices for kexec/kdump:
1) kexec/kdump kernel need not randomize the kernel starting point. Since kexec/kdump kernel is only for testing or emergencey, its life is not too long.

2) makes slots around the kernel input addr. This is only useful for kexec. I can't imagine why kdump need it.

Hi Thomas,
Could you test below patch?

This is patch is from Lu Yinghai.

---
 arch/x86/boot/compressed/misc.c |   14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)
arch/x86/boot/compressed/misc.c +9 -5
Index: linux-2.6/arch/x86/boot/compressed/misc.c
===================================================================
@@ -235,8 +235,9 @@ static void error(char *x)
 		asm("hlt");
 }
 
-#if CONFIG_X86_NEED_RELOCS
-static void handle_relocations(void *output, unsigned long output_len)
+#ifdef CONFIG_X86_NEED_RELOCS
+static void handle_relocations(void *output_orig, void *output,
+			       unsigned long output_len)
 {
 	int *reloc;
 	unsigned long delta, map, ptr;
@@ -247,7 +248,7 @@ static void handle_relocations(void *out
 	 * Calculate the delta between where vmlinux was linked to load
 	 * and where it was actually loaded.
 	 */
-	delta = min_addr - LOAD_PHYSICAL_ADDR;
+	delta = min_addr - (unsigned long)output_orig;
 	if (!delta) {
 		debug_putstr("No relocation needed... ");
 		return;
@@ -304,7 +305,8 @@ static void handle_relocations(void *out
 #endif
 }
 #else
-static inline void handle_relocations(void *output, unsigned long output_len)
+static inline void handle_relocations(void *output_orig, void *output,
+				      unsigned long output_len)
 { }
 #endif
 
@@ -365,6 +367,8 @@ asmlinkage void *decompress_kernel(void
 				  unsigned char *output,
 				  unsigned long output_len)
 {
+	unsigned char *output_orig = output;
+
 	real_mode = rmode;
 
 	sanitize_boot_params(real_mode);
@@ -417,7 +421,7 @@ asmlinkage void *decompress_kernel(void
 	debug_putstr("... ");
 	decompress(input_data, input_len, NULL, NULL, output, NULL, error);
 	parse_elf(output);
-	handle_relocations(output, output_len);
+	handle_relocations(output_orig, output, output_len);
 	debug_putstr("done.\nBooting the kernel.\n");
 	return output;
 }

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#12 lore raw

Hi,

arch/x86/boot/compressed/misc.c

Quoted 13 lines · Baoquan He

Could you test below patch?

This is patch is from Lu Yinghai.

---
 arch/x86/boot/compressed/misc.c |   14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

Index: linux-2.6/arch/x86/boot/compressed/misc.c
===================================================================

arch/x86/boot/compressed/misc.c +0 -0
[...]

Sorry, I don't know how to apply this patch.

I am building kexec-tools from
git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git master branch and there is no misc.c?!

-Thomas

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#17 lore raw
In reply to WANG Chao
Quoted 10 lines · WANG Chao
Quoted 4 lines · Vivek Goyal

If kernel is being moved physically, then we potentially have the issue of it stomping other things. So how do we make sure that it does not overwrite initramfs, or previous kernel's page tables or something else?

In case you don't read that part, memory of initrd, cmdline and others won't be overwritten.

This work is done in x86/boot/compressed/aslr.c::mem_avoid_init().

Ok, got it. so cmdline and initrd info is present in bootparams and kernel can use that to avoid stomping those.

But that still does not protect ELF header segment (prepared in case of kdump) and page tables setup by previous kernel.

So may be this failure happens because new kernel stomps over page tables of old kernel. But if that theory is right, then we should be able to kexec using 32bit entry point of bzImage.

Thomas, have you had any success with that?

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#13 lore raw
In reply to Thomas D.
Quoted 19 lines · Thomas D.

Hi,

arch/x86/boot/compressed/misc.c

Quoted 13 lines · Baoquan He

Could you test below patch?

This is patch is from Lu Yinghai.

---
 arch/x86/boot/compressed/misc.c |   14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

Index: linux-2.6/arch/x86/boot/compressed/misc.c
===================================================================

arch/x86/boot/compressed/misc.c +0 -0
[...]

Sorry, I don't know how to apply this patch.

Quoted 3 lines

I am building kexec-tools from
git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git master branch and there is no misc.c?!

This is a kernel patch and not kexec-tools patch.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#18 lore raw
In reply to Vivek Goyal
Quoted 5 lines · Vivek Goyal

So may be this failure happens because new kernel stomps over page tables of old kernel. But if that theory is right, then we should be able to kexec using 32bit entry point of bzImage.

Thomas, have you had any success with that?

When I execute kexec with "--entry-32bit" (I understand that you asked me to try that) the new kernel will boot as expected.

I'll now try the kernel patch...

-Thomas

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#19 lore raw
Quoted 9 lines · Thomas D.
Quoted 5 lines · Vivek Goyal

So may be this failure happens because new kernel stomps over page tables of old kernel. But if that theory is right, then we should be able to kexec using 32bit entry point of bzImage.

Thomas, have you had any success with that?

When I execute kexec with "--entry-32bit" (I understand that you asked me to try that) the new kernel will boot as expected.

Good to know that --entry-32bit option makes kexec work with kASLR enabled. So this indeed sounds like the issue of page tables being stomped over by new kernel.

IIUC, there are bootloaders which can use 64bit entry point of kernel. I think they must run into similar issue too.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#15 lore raw
In reply to Baoquan He

On Fri, Aug 22, 2014 at 07:59:02PM +0800, Baoquan He wrote:

[..]

Quoted 37 lines

So we have 2 choices for kexec/kdump:
1) kexec/kdump kernel need not randomize the kernel starting point. Since kexec/kdump kernel is only for testing or emergencey, its life is not too long.

2) makes slots around the kernel input addr. This is only useful for kexec. I can't imagine why kdump need it.

Hi Thomas,
Could you test below patch?

This is patch is from Lu Yinghai.

---
 arch/x86/boot/compressed/misc.c |   14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

Index: linux-2.6/arch/x86/boot/compressed/misc.c
===================================================================

arch/x86/boot/compressed/misc.c +3 -2
@@ -235,8 +235,9 @@ static void error(char *x)
 		asm("hlt");
 }
 
-#if CONFIG_X86_NEED_RELOCS
-static void handle_relocations(void *output, unsigned long output_len)
+#ifdef CONFIG_X86_NEED_RELOCS
+static void handle_relocations(void *output_orig, void *output,
+			       unsigned long output_len)
 {
 	int *reloc;
 	unsigned long delta, map, ptr;

arch/x86/boot/compressed/misc.c static void handle_relocations(void *out · line 251

Quoted 5 lines
arch/x86/boot/compressed/misc.c +1 -1
@@ -247,7 +248,7 @@ static void handle_relocations(void *out
 	 * Calculate the delta between where vmlinux was linked to load
 	 * and where it was actually loaded.
 	 */
-	delta = min_addr - LOAD_PHYSICAL_ADDR;
+	delta = min_addr - (unsigned long)output_orig;

So what does this patch actuall do? If I try to trace back output_orig, it seems to be same as LOAD_PHYSICAL_ADDR. That means there should not be any effect of this change? Or did I not understand it.

Look at head_64.S

movq $LOAD_PHYSICAL_ADDR, %rbp
...
...

movq    %rbp, %r8               /* output target address */
call    decompress_kernel       /* returns kernel location in %rax */

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#14 lore raw
In reply to Vivek Goyal

Hi,

the patch works!

I am able to reboot my system using kexec when the kernel has this patch.

I had to slightly modify the patch, so it will apply against linux-3.15.10:

x86/boot/compressed/misc.c renamed from x86/boot/compressed/misc.c.old +4 -4
@@ -230,8 +230,9 @@
 		asm("hlt");
 }
 
-#if CONFIG_X86_NEED_RELOCS
-static void handle_relocations(void *output, unsigned long output_len)
+#ifdef CONFIG_X86_NEED_RELOCS
+static void handle_relocations(void *output_orig, void *output,
+			       unsigned long output_len)
 {
 	int *reloc;
 	unsigned long delta, map, ptr;
@@ -242,7 +243,7 @@
 	 * Calculate the delta between where vmlinux was linked to load
 	 * and where it was actually loaded.
 	 */
-	delta = min_addr - LOAD_PHYSICAL_ADDR;
+	delta = min_addr - (unsigned long)output_orig;
 	if (!delta) {
 		debug_putstr("No relocation needed... ");
 		return;
@@ -299,7 +300,8 @@
 #endif
 }
 #else
-static inline void handle_relocations(void *output, unsigned long

output_len)
+static inline void handle_relocations(void *output_orig, void *output,
+ unsigned long output_len)
{ }
#endif

@@ -360,6 +362,8 @@
unsigned char *output, unsigned long output_len)
{
+ unsigned char *output_orig = output;
+
real_mode = rmode;

sanitize_boot_params(real_mode);
@@ -402,7 +406,7 @@
debug_putstr("\nDecompressing Linux... "); decompress(input_data, input_len, NULL, NULL, output, NULL, error); parse_elf(output);
- handle_relocations(output, output_len);
+ handle_relocations(output_orig, output, output_len);
debug_putstr("done.\nBooting the kernel.\n");
return output;
}

So now we have two "solutions":

1) Using kexec with "--entry-32bit" parameter

2) Use a patched kernel

-Thomas

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#16 lore raw
In reply to Vivek Goyal
Quoted 41 lines · Vivek Goyal

On Fri, Aug 22, 2014 at 07:59:02PM +0800, Baoquan He wrote:

[..]

Quoted 38 lines

So we have 2 choices for kexec/kdump:
1) kexec/kdump kernel need not randomize the kernel starting point. Since kexec/kdump kernel is only for testing or emergencey, its life is not too long.

2) makes slots around the kernel input addr. This is only useful for kexec. I can't imagine why kdump need it.

Hi Thomas,
Could you test below patch?

This is patch is from Lu Yinghai.

---
 arch/x86/boot/compressed/misc.c |   14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

Index: linux-2.6/arch/x86/boot/compressed/misc.c
===================================================================

arch/x86/boot/compressed/misc.c +3 -2
@@ -235,8 +235,9 @@ static void error(char *x)
 		asm("hlt");
 }
 
-#if CONFIG_X86_NEED_RELOCS
-static void handle_relocations(void *output, unsigned long output_len)
+#ifdef CONFIG_X86_NEED_RELOCS
+static void handle_relocations(void *output_orig, void *output,
+			       unsigned long output_len)
 {
 	int *reloc;
 	unsigned long delta, map, ptr;
@@ -247,7 +248,7 @@ static void handle_relocations(void *out
 	 * Calculate the delta between where vmlinux was linked to load
Quoted 8 lines

arch/x86/boot/compressed/misc.c static void handle_relocations(void *out · line 251

Quoted 4 lines
arch/x86/boot/compressed/misc.c +1 -1
 	 * and where it was actually loaded.
 	 */
-	delta = min_addr - LOAD_PHYSICAL_ADDR;
+	delta = min_addr - (unsigned long)output_orig;

So what does this patch actuall do? If I try to trace back output_orig, it seems to be same as LOAD_PHYSICAL_ADDR. That means there should not be any effect of this change? Or did I not understand it.

No, kexec kernel has to be relocatable kernel. so it shoule be the value calculated by code block surrounded by the MACRO checking.

(startup_32 + %rip + BP_kernel_alignment -1 )& ~(BP_kernel_alignment-1)

#ifdef CONFIG_RELOCATABLE

        leaq    startup_32(%rip) /* - $startup_32 */, %rbp
        movl    BP_kernel_alignment(%rsi), %eax 
        decl    %eax
        addq    %rax, %rbp     
        notq    %rax           
        andq    %rax, %rbp     
        cmpq    $LOAD_PHYSICAL_ADDR, %rbp
        jge     1f

#endif
movq $LOAD_PHYSICAL_ADDR, %rbp
1:

/* Target address to relocate to for decompression */
leaq z_extract_offset(%rbp), %rbx

Quoted 9 lines

Look at head_64.S

movq $LOAD_PHYSICAL_ADDR, %rbp
...
...

movq    %rbp, %r8               /* output target address */
call    decompress_kernel       /* returns kernel location in %rax */
Quoted 7 lines

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#20 lore raw
In reply to Vivek Goyal
Quoted 6 lines · Vivek Goyal
Quoted 5 lines · Thomas D.
Quoted 4 lines · Vivek Goyal

So may be this failure happens because new kernel stomps over page tables of old kernel. But if that theory is right, then we should be able to kexec using 32bit entry point of bzImage.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Thomas, have you had any success with that?

When I execute kexec with "--entry-32bit" (I understand that you asked me to try that) the new kernel will boot as expected.

Good to know that --entry-32bit option makes kexec work with kASLR enabled. So this indeed sounds like the issue of page tables being stomped over by new kernel.

I don't think page tables could be stomped over by new kernel. The reason why --entry-32bit can work is kexec-tools get the memory near the beginning of system RAM. You can check the value passed to argument buf_end. when --entry-32bit it's 0, that means find memoey hole from low to high.

But in 64bit mode, the buf_end is always -1, this cause all memory ready for kernel/initrd is close to th end of system ram. On my PC with 16G memory, all those memory around 16G. I changed buf_end to 1, then use this to load bzImage64, it works too. And addr for kernel/initrd/real_mode are all allocated from low to high. I don't know why we need put kernel/initrd/xxx close to the end of system ram.

kexec/add_buffer.c +1 -1
index 4d4a55f..ce76669 100644
@@ -10,5 +10,5 @@ unsigned long add_buffer(struct kexec_info *info,
                         int buf_end)
 {
        return add_buffer_virt(info, buf, bufsz, memsz, buf_align,
-                              buf_min, buf_max, buf_end);
+                              buf_min, buf_max, 1);
 }
Quoted 11 lines

IIUC, there are bootloaders which can use 64bit entry point of kernel. I think they must run into similar issue too.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#8 lore raw

Kexec failing in handle_relocations() (Was: Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set)

In reply to Thomas D.
Quoted 13 lines · Thomas D.

Hi,

Quoted 3 lines · Vivek Goyal

Thomas, can you please test kexec with "nokaslr" specified in command line. This can atleast get you going for now till the problem is actually fixed.

I booted linux-3.15.10 with "nokaslr"

Quoted 3 lines
 # dmesg | grep -i command
[    0.000000] Command line: BOOT_IMAGE=/kernel dolvm video=1280x1024 root=UUID=6d42... rootfs=ext4 nokaslr initrd=/initramfs
[    0.000000] Kernel command line: BOOT_IMAGE=/kernel dolvm video=1280x1024 root=UUID=6d42... rootfs=ext4 nokaslr initrd=/initramfs
Quoted 6 lines

and run kexec with

Quoted 1 line

--reuse-cmdline -l /boot/kernel root=/dev/dm-3 --initrd=/boot/initramfs

But it still doesn't reboot. I see no difference between using "nokaslr" or not.

[ CC hpa ]

I think handle_relocations() is contributing to the problem here.

I enabled earlyprintk and enabled debug boot messages. I also put some debug messages of my own to print values of some variables and I see following when I try to kexec.

**********************************************************************

[  340.709078] kexec: Starting new kernel

early console in decompress_kernel
KASLR disabled by default...

Decompressing Linux... Parsing ELF...

min_addr=000000042e000000
delta=000000042d000000
Performing relocations...
map=00000004ad000000

32-bit relocation outside of kernel!

-- System halted
*****************************************************************

So min_addr is the location where kernel is actually being loaded and will be run from. delta is the difference between compile time address and run time address.

KASLR is disabled by default. I am wondering that why do we have to go through handle_relocations() if KASLR is disabled. Should we fall back to old logic of manipulating page tables.

Or we need to figure out how to make handle_relocations() work in such a way that kernel can be loaded anywhere in the memory.

I suspect that 32bit relocations might not be able to handle it and we probably might need all 64bit relocations to handle that case. But with this resulting virtual addresses might still be the problem.

I summary, I am wondering what kind of limitation handle_relocations() put on physical location of kernel and how can we remove that restriction to be able to load kernel anywhere in memory.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec