Skip to contentSkip to conversation

Elf loader fixes

PATCH In review Gerd Hoffmann 4 participants 8 messages Feb 22, 2006

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Elf loader fixes

Hi folks,

The Xen ELF kernel loader is quite quirky wrt. physical and virtual addresses, probably for historical reasons, linux got that wrong too until very recently (kexec merge in 2.6.14 or so). The patch below fixes that.

Changes:
* Fix linux kernel ELF entry point (also submitted to lkml)
* Drop LOAD_OFFSET re-#define hack in xen headers.
* Fix both dom0 and libxc elf loaders.
* add quick mode so loading old linux kernels doesn't break.

Linux-wise everything should be OK with that, but it might break other OS'es which also use the ELF loader (in case they create bug-compatible ELF headers with broken paddr entries ...).

please apply,

Gerd

linux-2.6-xen-sparse/arch/i386/kernel/vmlinux.lds.S +1 -1
diff -r 5abf652c4c52 linux-2.6-xen-sparse/arch/i386/kernel/vmlinux.lds.S
@@ -10,7 +10,7 @@
 
 OUTPUT_FORMAT("elf32-i386", "elf32-i386", "elf32-i386")
 OUTPUT_ARCH(i386)
-ENTRY(phys_startup_32)
+ENTRY(startup_32)
 jiffies = jiffies_64;
 SECTIONS
 {
linux-2.6-xen-sparse/include/asm-i386/mach-xen/asm/page.h +0 -4
diff -r 5abf652c4c52 linux-2.6-xen-sparse/include/asm-i386/mach-xen/asm/page.h
@@ -288,10 +288,6 @@
 #endif
 #define __KERNEL_START		(__PAGE_OFFSET + __PHYSICAL_START)
 
-#undef LOAD_OFFSET
-#define LOAD_OFFSET		0
-
-
 #define PAGE_OFFSET		((unsigned long)__PAGE_OFFSET)
 #define VMALLOC_RESERVE		((unsigned long)__VMALLOC_RESERVE)
 #define MAXMEM			(HYPERVISOR_VIRT_START-__PAGE_OFFSET-__VMALLOC_RESERVE)
tools/libxc/xc_load_elf.c +22 -6
diff -r 5abf652c4c52 tools/libxc/xc_load_elf.c
@@ -138,10 +138,10 @@
         phdr = (Elf_Phdr *)(image + ehdr->e_phoff + (h*ehdr->e_phentsize));
         if ( !is_loadable_phdr(phdr) )
             continue;
-        if ( phdr->p_paddr < kernstart )
-            kernstart = phdr->p_paddr;
-        if ( (phdr->p_paddr + phdr->p_memsz) > kernend )
-            kernend = phdr->p_paddr + phdr->p_memsz;
+        if ( phdr->p_vaddr < kernstart )
+            kernstart = phdr->p_vaddr;
+        if ( (phdr->p_vaddr + phdr->p_memsz) > kernend )
+            kernend = phdr->p_vaddr + phdr->p_memsz;
     }
 
     if ( (kernstart > kernend) || 
@@ -189,7 +189,18 @@
         
         for ( done = 0; done < phdr->p_filesz; done += chunksz )
         {
-            pa = (phdr->p_paddr + done) - dsi->v_start;
+            if (phdr->p_paddr == phdr->p_vaddr) {
+                /*
+                 * Bug compatibility alert: In older linux kernels
+                 * p_paddr is broken, it doesn't contain the physical
+                 * address but instead is identical to p_vaddr.  Thus
+                 * we can't use it directly, instead we'll guess it
+                 * using dsi->v_start.
+                 */
+                pa = (phdr->p_vaddr + done) - dsi->v_start;
+            } else {
+                pa = (phdr->p_paddr + done);
+            }
             va = xc_map_foreign_range(
                 xch, dom, PAGE_SIZE, PROT_WRITE, parray[pa>>PAGE_SHIFT]);
             chunksz = phdr->p_filesz - done;
@@ -202,7 +213,12 @@
 
         for ( ; done < phdr->p_memsz; done += chunksz )
         {
-            pa = (phdr->p_paddr + done) - dsi->v_start;
+            if (phdr->p_paddr == phdr->p_vaddr) {
+                /* bug compatibility alert, see above */
+                pa = (phdr->p_vaddr + done) - dsi->v_start;
+            } else {
+                pa = (phdr->p_paddr + done);
+            }
             va = xc_map_foreign_range(
                 xch, dom, PAGE_SIZE, PROT_WRITE, parray[pa>>PAGE_SHIFT]);
             chunksz = phdr->p_memsz - done;
xen/arch/x86/domain.c +1 -1
diff -r 5abf652c4c52 xen/arch/x86/domain.c
@@ -346,7 +346,7 @@
     struct vcpu *v, struct vcpu_guest_context *c)
 {
     struct domain *d = v->domain;
-    unsigned long phys_basetab;
+    unsigned long phys_basetab = 0;
     int i, rc;
 
     /*
xen/common/elf.c +42 -15

90 lines not on this page · raw message

xen/include/xen/sched.h +2 -0
diff -r 5abf652c4c52 xen/include/xen/sched.h
@@ -165,6 +165,8 @@
     unsigned long v_end;
     unsigned long v_kernstart;
     unsigned long v_kernend;
+    unsigned long p_kernstart;
+    unsigned long p_kernend;
     unsigned long v_kernentry;
     /* Initialised by loader: Private. */
     unsigned int  load_symtab;

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#1 lore raw

I'm afraid this won't work for 64-bits, because of the way the vsyscall page gets set up there. If you dump the program headers you'll see that the respective segment is at an address close to the top of the address space, which would make the image appear to need nearly 2 Gb of memory (making the load fail if there's less than that available to the domain).

Also, I'm unclear why you need to make persistent the physical address information for the dom0 case, and hence why the bug compatibility hack is needed there.

Jan

Quoted 1 line
Quoted 1 line
Quoted 1 line

Gerd Hoffmann <[email protected]> 22.02.06 12:37:26 >>>

Hi folks,

The Xen ELF kernel loader is quite quirky wrt. physical and virtual addresses, probably for historical reasons, linux got that wrong too until very recently (kexec merge in 2.6.14 or so). The patch below fixes that.

Changes:
* Fix linux kernel ELF entry point (also submitted to lkml)
* Drop LOAD_OFFSET re-#define hack in xen headers.
* Fix both dom0 and libxc elf loaders.
* add quick mode so loading old linux kernels doesn't break.

Linux-wise everything should be OK with that, but it might break other OS'es which also use the ELF loader (in case they create bug-compatible ELF headers with broken paddr entries ...).

please apply,

Gerd

#2 lore raw
Quoted 4 lines · Jan Beulich

I'm afraid this won't work for 64-bits, because of the way the vsyscall page gets set up there. If you dump the program headers you'll see that the respective segment is at an address close to the top of the address space, which would make the image appear to need nearly 2 Gb of memory (making the load fail if there's less than that available to the domain).

Yep, will not work work as-is, the domain builder needs more fixes for that.

Quoted 2 lines

Also, I'm unclear why you need to make persistent the physical address information for the dom0 case, and hence why the bug compatibility hack is needed there.

Well, the libxc loader needs it to be able to load both old kernels and kernels with correct physical addresses in the phdr.

The dom0 loader doesn't really need it, at least on x32, right now it doesn't look at the physical addresses anyway. Instead it depends on the VIRT_BASE hack to get it right. It's sort of documentation only ;)

For x64 it will certainly be needed to get the memory size calculations right (use physical not virtual addresses) and fix the issue you've outlined above that way.

cheers,

Gerd

Signature · 3 lines
Gerd 'just married' Hoffmann <[email protected]>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg
#3 lore raw
In reply to Jan Beulich
Quoted 4 lines · Jan Beulich

I'm afraid this won't work for 64-bits, because of the way the vsyscall page gets set up there. If you dump the program headers you'll see that the respective segment is at an address close to the top of the address space, which would make the image appear to need nearly 2 Gb of memory (making the load fail if there's less than that available to the domain).

Hmm, who sets up the vsyscall mapping right now? The linux kernel I guess? The elf kernel loader certainly doesn't look like it cares much about virtual address (other than VIRT_BASE) ...

cheers,

Gerd

Signature · 3 lines
Gerd 'just married' Hoffmann <[email protected]>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg
#4 lore raw
Quoted 3 lines
Quoted 1 line
Quoted 1 line

Gerd Hoffmann <[email protected]> 22.02.06 16:12:25 >>>

Quoted 1 line · Jan Beulich

I'm afraid this won't work for 64-bits, because of the way the vsyscall page gets set up there. If you dump the

program

Quoted 1 line
Quoted 1 line

headers you'll see that the respective segment is at an address close to the top of the address space, which would

make

Quoted 5 lines
Quoted 2 lines

the image appear to need nearly 2 Gb of memory (making the load fail if there's less than that available to the domain).

Hmm, who sets up the vsyscall mapping right now? The linux kernel I guess?

Yes.

Quoted 2 lines

The elf kernel loader certainly doesn't look like it cares much about virtual address (other than VIRT_BASE) ...

Correct. And the same applies to boot loaders.

Jan

#5 lore raw
Quoted 4 lines
Quoted 2 lines

The elf kernel loader certainly doesn't look like it cares much about virtual address (other than VIRT_BASE) ...

Correct. And the same applies to boot loaders.

Well, the difference between xen and other boot loaders is that xen boots the kernel with paging already enabled ...

Right now it's completely broken: the loader works only with virtual addresses in the paddr header field. It can't stay that way, especially if we'll want to have kexec working with xenlinux kernels some day ;)

As I see things there are basically two ways to fixup this: Either create a simple linear mapping using paddr + VIRT_BASE, which would be pretty close to the current behaviour (and other boot loaders). Or do a complete virtual memory setup, which probably can't be done without major changes in the domain builder ...

cheers,

Gerd

Signature · 3 lines
Gerd 'just married' Hoffmann <[email protected]>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg
#6 lore raw
Quoted 8 lines · Gerd Hoffmann
Quoted 4 lines
Quoted 2 lines

The elf kernel loader certainly doesn't look like it cares much about virtual address (other than VIRT_BASE) ...

Correct. And the same applies to boot loaders.

Well, the difference between xen and other boot loaders is that xen boots the kernel with paging already enabled ...

actually, there have historically been a ton of boot loaders that boot with paging already enabled: just go back to the early Suns; part of the task of SunOS was to move the firmware's page tables to kernel space and use them until it could build its own.

I hit this when I ported sunos to a machine with firmware that did not set up page tables.

it's been very common, in the non-PC world, to have paging turned on very early.

Just a historical comment of no value to this discussion :-)

ron

#7 lore raw
In reply to Gerd Hoffmann
Quoted 5 lines · Gerd Hoffmann

As I see things there are basically two ways to fixup this: Either create a simple linear mapping using paddr + VIRT_BASE, which would be pretty close to the current behaviour (and other boot loaders). Or do a complete virtual memory setup, which probably can't be done without major changes in the domain builder ...

hi,

as previously mentioned here and at the XenSummit, I have a simple yet functioning domU bootloader which defers ELF-decoding to domU-space. It currently only works when the input is a ramdisk image, created with a small tool called 'pack. It used to work from a TCP-connection (provided by the fabulous UIP which is still in my tree), hence the weird state-machine look of the code.

I suppose taking this approach could solve all our domU-building woes for good. It should also provide a more elegant solution to the attestation problems the Intel guys were talking about and trying to solve with their two-stage domain building proposal.

Clone my tree at http://www.distlab.dk/hg/index.cgi/xen-gfx.hg and have a look at the extras/mstrap subdir. The 'pack' tool is in tools/migrate.

You will need Jam to build, or you have to roll your own Makefile. The tree also contains my very experimental 'Blink' display system as demoed at the Summit in tools/gfx, and my self-migration patch to XenLinux which at the moment is mostly useful for self-checkpointing to disk (see tools/minimig.c). Your mileage may vary.

Jacob