Skip to contentSkip to conversation

kexec for 2.5.44 (Who do I send this to?)

PATCH In review Eric W. Biederman 7 participants 16 messages Oct 19, 2002 – Oct 31, 2002 kexec

lore
Share & tools
lore
#0 lore raw

The kexec code has gone through a fairly decent review, and all known bugs are resolved. There are still BIOS's that don't work after you have run a kernel but that is an entirely different problem.

My real question: With Linus off on vacation my real question is who should I send this to?

kexec is a system call that allows you to load another kernel from the currently executing Linux kernel. The current implementation has only been tested, and had the kinks worked out on x86, but the generic code should work on any architecture.

Some machines have BIOSes that are either extremely slow to reboot, or that cannot reliably perform a reboot. In which case kexec may be the only alternative to reboot in a reliable and timely manner.

The patch is archived at:
http://www.xmission.com/~ebiederm/files/kexec/

A compatible user space is at:
http://www.xmission.com/~ebiederm/files/kexec/kexec-tools-1.?.tar.gz
This code boots either a static ELF executable or a bzImage.

A kernel reformater is makes images that seem to boot more reliably is at:
ftp://ftp.lnxi.com/pub/mkelfImage/mkelfImage-1.17.tar.gz

In bug reports please include the serial console output of kexec kexec_test. kexec_test exercises most of the interesting code paths that are needed to load a kernel with lots of debugging print statements, so hangs can easily be detected.

I have been using this technique for the last several years and what the kernel needs to know is well understood, and currently implemented. The worst of the kinks when dealing with BIOS's have been worked out, but it is still easy to get on the wrong side of a BIOS.

The expected future work is digging through drivers to make certain they place their hardware in a quiesent state, and digging through weird BIOS incompatibilites or developing ways to bypass the BIOS entirely.

The system call signature is fixed.

 MAINTAINERS                        |    7 
 arch/i386/Config.help              |    9 
 arch/i386/config.in                |    3 
 arch/i386/kernel/Makefile          |    1 
 arch/i386/kernel/apic.c            |   51 +++
 arch/i386/kernel/dmi_scan.c        |   27 -
 arch/i386/kernel/entry.S           |    1 
 arch/i386/kernel/i8259.c           |   24 +
 arch/i386/kernel/io_apic.c         |    2 
 arch/i386/kernel/machine_kexec.c   |  144 ++++++++
 arch/i386/kernel/reboot.c          |   44 --
 arch/i386/kernel/relocate_kernel.S |   99 +++++
 arch/i386/kernel/smp.c             |   24 +
 include/asm-i386/apic.h            |    3 
 include/asm-i386/apicdef.h         |    1 
 include/asm-i386/kexec.h           |   25 +
 include/asm-i386/unistd.h          |    2 
 include/linux/kexec.h              |   48 ++
 kernel/Makefile                    |    3 
 kernel/kexec.c                     |  624 +++++++++++++++++++++++++++++++++++++
 kernel/sys.c                       |   61 +++
 21 files changed, 1135 insertions, 68 deletions
MAINTAINERS +7 -0
diff -uNr linux-2.5.44/MAINTAINERS linux-2.5.44.x86kexec/MAINTAINERS
@@ -934,6 +934,13 @@
 W:	http://www.cse.unsw.edu.au/~neilb/patches/linux-devel/
 S:	Maintained
 
+KEXEC
+P:	Eric Biederman
+M:	[email protected]
+M:	[email protected]
+L:	[email protected]
+S:	Maintained
+
 LANMEDIA WAN CARD DRIVER
 P:	Andrew Stanley-Jones
 M:	[email protected]
arch/i386/Config.help +9 -0
diff -uNr linux-2.5.44/arch/i386/Config.help linux-2.5.44.x86kexec/arch/i386/Config.help
@@ -417,6 +417,15 @@
   you have use for it; the module is called binfmt_misc.o. If you
   don't know what to answer at this point, say Y.
 
+CONFIG_KEXEC
+  kexec is a system call that implements kernel level exec support.
+  Or the ability to boot Linux from Linux.  The kexec system call
+  allows you to replace your current kernel with another kernel, 
+  not necessarily Linux.
+
+  It is recommend to also enable device hotplugging support to be
+  certain of properly shutting down all of your hardware.
+  
 CONFIG_M386
   This is the processor type of your CPU. This information is used for
   optimizing purposes. In order to compile a kernel that can run on
arch/i386/config.in +3 -0
diff -uNr linux-2.5.44/arch/i386/config.in linux-2.5.44.x86kexec/arch/i386/config.in
@@ -247,6 +247,9 @@
    fi
 fi
 
+if [ "$CONFIG_EXPERIMENTAL" = "y" ]; then
+   bool 'Kernel execing kernel support' CONFIG_KEXEC
+fi
 endmenu
 
 mainmenu_option next_comment
arch/i386/kernel/Makefile +1 -0
diff -uNr linux-2.5.44/arch/i386/kernel/Makefile linux-2.5.44.x86kexec/arch/i386/kernel/Makefile
@@ -25,6 +25,7 @@
 obj-$(CONFIG_X86_MPPARSE)	+= mpparse.o
 obj-$(CONFIG_X86_LOCAL_APIC)	+= apic.o nmi.o
 obj-$(CONFIG_X86_IO_APIC)	+= io_apic.o
+obj-$(CONFIG_KEXEC)		+= machine_kexec.o relocate_kernel.o
 obj-$(CONFIG_SOFTWARE_SUSPEND)	+= suspend.o
 obj-$(CONFIG_X86_NUMAQ)		+= numaq.o
 obj-$(CONFIG_PROFILING)		+= profile.o
arch/i386/kernel/apic.c +51 -0

69 lines not on this page · raw message

arch/i386/kernel/dmi_scan.c +1 -26
diff -uNr linux-2.5.44/arch/i386/kernel/dmi_scan.c linux-2.5.44.x86kexec/arch/i386/kernel/dmi_scan.c
@@ -214,31 +214,6 @@
 	return 0;
 }
 
-/*
- * Some machines require the "reboot=s"  commandline option, this quirk makes that automatic.
- */
-static __init int set_smp_reboot(struct dmi_blacklist *d)
-{
-#ifdef CONFIG_SMP
-	extern int reboot_smp;
-	if (reboot_smp == 0)
-	{
-		reboot_smp = 1;
-		printk(KERN_INFO "%s series board detected. Selecting SMP-method for reboots.\n", d->ident);
-	}
-#endif
-	return 0;
-}
-
-/*
- * Some machines require the "reboot=b,s"  commandline option, this quirk makes that automatic.
- */
-static __init int set_smp_bios_reboot(struct dmi_blacklist *d)
-{
-	set_smp_reboot(d);
-	set_bios_reboot(d);
-	return 0;
-}
 
 /*
  * Some bioses have a broken protected mode poweroff and need to use realmode
@@ -529,7 +504,7 @@
 			MATCH(DMI_BIOS_VERSION, "4.60 PGMA"),
 			MATCH(DMI_BIOS_DATE, "134526184"), NO_MATCH
 			} },
-	{ set_smp_bios_reboot, "Dell PowerEdge 1300", {	/* Handle problems with rebooting on Dell 1300's */
+	{ set_bios_reboot, "Dell PowerEdge 1300", {	/* Handle problems with rebooting on Dell 1300's */
 			MATCH(DMI_SYS_VENDOR, "Dell Computer Corporation"),
 			MATCH(DMI_PRODUCT_NAME, "PowerEdge 1300/"),
 			NO_MATCH, NO_MATCH
arch/i386/kernel/entry.S +1 -0
diff -uNr linux-2.5.44/arch/i386/kernel/entry.S linux-2.5.44.x86kexec/arch/i386/kernel/entry.S
@@ -737,6 +737,7 @@
 	.long sys_free_hugepages
 	.long sys_exit_group
 	.long sys_lookup_dcookie
+	.long sys_kexec
 
 	.rept NR_syscalls-(.-sys_call_table)/4
 		.long sys_ni_syscall
arch/i386/kernel/i8259.c +24 -0
diff -uNr linux-2.5.44/arch/i386/kernel/i8259.c linux-2.5.44.x86kexec/arch/i386/kernel/i8259.c
@@ -246,10 +246,34 @@
 	return 0;
 }
 
+static void i8259A_remove(struct device *dev)
+{   
+	/* Restore the i8259A to it's legacy dos setup.
+	 * The kernel won't be using it any more, and it
+	 * just might make reboots, and kexec type applications
+	 * more stable.
+	 */
+	outb(0xff, 0x21);	/* mask all of 8259A-1 */
+	outb(0xff, 0xA1);	/* mask all of 8259A-1 */
+
+	outb_p(0x11, 0x20);	/* ICW1: select 8259A-1 init */
+	outb_p(0x08, 0x21);	/* ICW2: 8259A-1 IR0-7 mappend to 0x8-0xf */
+	outb_p(0x01, 0x21);	/* Normal 8086 auto EOI mode */
+
+	outb_p(0x11, 0xA0);	/* ICW1: select 8259A-2 init */
+	outb_p(0x08, 0xA1);	/* ICW2: 8259A-2 IR0-7 mappend to 0x70-0x77 */
+	outb_p(0x01, 0xA1);	/* Normal 8086 auto EOI mode */
+
+	udelay(100);		/* wait for 8259A to initialize */
+
+	/* Should I unmask interrupts here?  */
+}
+
 static struct device_driver i8259A_driver = {
 	.name		= "pic",
 	.bus		= &system_bus_type,
 	.resume		= i8259A_resume,
+	.remove		= i8259A_remove,
 };
 
 static struct sys_device device_i8259A = {
arch/i386/kernel/io_apic.c +0 -2
diff -uNr linux-2.5.44/arch/i386/kernel/io_apic.c linux-2.5.44.x86kexec/arch/i386/kernel/io_apic.c
@@ -1115,8 +1115,6 @@
 	 * Clear the IO-APIC before rebooting:
 	 */
 	clear_IO_APIC();
-
-	disconnect_bsp_APIC();
 }
 
 /*
arch/i386/kernel/machine_kexec.c +144 -0

144 lines not on this page · raw message

arch/i386/kernel/reboot.c +5 -39

78 lines not on this page · raw message

arch/i386/kernel/relocate_kernel.S +99 -0

99 lines not on this page · raw message

arch/i386/kernel/smp.c +24 -0
diff -uNr linux-2.5.44/arch/i386/kernel/smp.c linux-2.5.44.x86kexec/arch/i386/kernel/smp.c
@@ -611,6 +611,30 @@
 
 void smp_send_stop(void)
 {
+ 	extern int reboot_cpu;
+	int reboot_cpu_id;
+
+ 	/* The boot cpu is always logical cpu 0 */
+	reboot_cpu_id = 0;
+
+	/* See if there has been give a command line override .
+	 */
+	if ((reboot_cpu != -1) && !(reboot_cpu >= NR_CPUS) && 
+		test_bit(reboot_cpu, &cpu_online_map)) {
+		reboot_cpu_id = reboot_cpu;
+	}
+	 
+	/* Make certain the the cpu I'm rebooting on is online */
+	if (!test_bit(reboot_cpu_id, &cpu_online_map)) {
+		reboot_cpu_id = smp_processor_id();
+	}
+
+	/* Make certain I only run on the appropriate processor */
+	set_cpus_allowed(current, 1 << reboot_cpu_id);
+
+	/* O.k. Now that I'm on the appropriate processor stop
+	 * all of the others.
+	 */
 	smp_call_function(stop_this_cpu, NULL, 1, 0);
 
 	local_irq_disable();
include/asm-i386/apic.h +3 -0
diff -uNr linux-2.5.44/include/asm-i386/apic.h linux-2.5.44.x86kexec/include/asm-i386/apic.h
@@ -96,6 +96,9 @@
 #define NMI_LOCAL_APIC	2
 #define NMI_INVALID	3
 
+extern void stop_apics(void);
+#else
+static inline void stop_apics(void) { }
 #endif /* CONFIG_X86_LOCAL_APIC */
 
 #endif /* __ASM_APIC_H */
include/asm-i386/apicdef.h +1 -0
diff -uNr linux-2.5.44/include/asm-i386/apicdef.h linux-2.5.44.x86kexec/include/asm-i386/apicdef.h
@@ -88,6 +88,7 @@
 #define			APIC_LVT_REMOTE_IRR		(1<<14)
 #define			APIC_INPUT_POLARITY		(1<<13)
 #define			APIC_SEND_PENDING		(1<<12)
+#define			APIC_MODE_MASK			0x700
 #define			GET_APIC_DELIVERY_MODE(x)	(((x)>>8)&0x7)
 #define			SET_APIC_DELIVERY_MODE(x,y)	(((x)&~0x700)|((y)<<8))
 #define				APIC_MODE_FIXED		0x0
include/asm-i386/kexec.h +25 -0
diff -uNr linux-2.5.44/include/asm-i386/kexec.h linux-2.5.44.x86kexec/include/asm-i386/kexec.h
@@ -0,0 +1,25 @@
+#ifndef _I386_KEXEC_H
+#define _I386_KEXEC_H
+
+#include <asm/fixmap.h>
+
+/*
+ * KEXEC_SOURCE_MEMORY_LIMIT maximum page get_free_page can return.
+ * I.e. Maximum page that is mapped directly into kernel memory,
+ * and kmap is not required.
+ *
+ * Someone correct me if FIXADDR_START - PAGEOFFSET is not the correct
+ * calculation for the amount of memory directly mappable into the
+ * kernel memory space.
+ */
+
+/* Maximum physical address we can use pages from */
+#define KEXEC_SOURCE_MEMORY_LIMIT (FIXADDR_START - PAGE_OFFSET) 
+/* Maximum address we can reach in physical address mode */
+#define KEXEC_DESTINATION_MEMORY_LIMIT (-1UL)
+
+#define KEXEC_REBOOT_CODE_SIZE	4096
+#define KEXEC_REBOOT_CODE_ALIGN 0
+
+
+#endif /* _I386_KEXEC_H */
include/asm-i386/unistd.h +1 -1
diff -uNr linux-2.5.44/include/asm-i386/unistd.h linux-2.5.44.x86kexec/include/asm-i386/unistd.h
@@ -258,7 +258,7 @@
 #define __NR_free_hugepages	251
 #define __NR_exit_group		252
 #define __NR_lookup_dcookie	253
-  
+#define __NR_kexec		254
 
 /* user-visible error numbers are in the range -1 - -124: see <asm-i386/errno.h> */
 
include/linux/kexec.h +48 -0

48 lines not on this page · raw message

kernel/Makefile +3 -0
diff -uNr linux-2.5.44/kernel/Makefile linux-2.5.44.x86kexec/kernel/Makefile
@@ -21,6 +21,7 @@
 obj-$(CONFIG_CPU_FREQ) += cpufreq.o
 obj-$(CONFIG_BSD_PROCESS_ACCT) += acct.o
 obj-$(CONFIG_SOFTWARE_SUSPEND) += suspend.o
+obj-$(CONFIG_KEXEC) += kexec.o
 
 ifneq ($(CONFIG_IA64),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is
@@ -30,5 +31,7 @@
 # to get a correct value for the wait-channel (WCHAN in ps). --davidm
 CFLAGS_sched.o := $(PROFILING) -fno-omit-frame-pointer
 endif
+
+obj-$(CONFIG_KEXEC)             += kexec.o
 
 include $(TOPDIR)/Rules.make
kernel/kexec.c +624 -0

624 lines not on this page · raw message

kernel/sys.c +61 -0

73 lines not on this page · raw message

#1 lore raw
Quoted 7 lines · Eric W. Biederman

The kexec code has gone through a fairly decent review, and all known bugs are resolved. There are still BIOS's that don't work after you have run a kernel but that is an entirely different problem.

My real question: With Linus off on vacation my real question is who should I send this to?

I believe Linus explicitly said he wasn't going to tell anyone, which means we're back to the days of "through it on the list over and over until someone admits to seeing it." Or send it to everyone who might be willing to push it to Linus when he gets back.

By not accepting stuff at this point I would guess that the defacto freeze is here. Hope I'm wrong, there is some good stuff which would be ready by Oct 31.

Signature · 3 lines
bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
#2 lore raw
Quoted 7 lines · Bill Davidsen
Quoted 5 lines · Eric W. Biederman

The kexec code has gone through a fairly decent review, and all known bugs are resolved. There are still BIOS's that don't work after you have run a kernel but that is an entirely different problem.

Quoted 7 lines
Quoted 2 lines

My real question: With Linus off on vacation my real question is who should I send this to?

I believe Linus explicitly said he wasn't going to tell anyone, which means we're back to the days of "through it on the list over and over until someone admits to seeing it." Or send it to everyone who might be willing to push it to Linus when he gets back.

I'm hoping one of the guys who work regularly with Linus will volunteer...

Quoted 3 lines

By not accepting stuff at this point I would guess that the defacto freeze is here. Hope I'm wrong, there is some good stuff which would be ready by Oct 31.

The only strategy that I know works at this point is being tenacious. So I'm not planning to throw in the towel just yet.

Eric

#3 lore raw
In reply to Bill Davidsen
Quoted 8 lines · Bill Davidsen
Quoted 6 lines · Eric W. Biederman

The kexec code has gone through a fairly decent review, and all known bugs are resolved. There are still BIOS's that don't work after you have run a kernel but that is an entirely different problem.

My real question: With Linus off on vacation my real question is who should I send this to?

Quoted 8 lines

I believe Linus explicitly said he wasn't going to tell anyone, which means we're back to the days of "through it on the list over and over until someone admits to seeing it." Or send it to everyone who might be willing to push it to Linus when he gets back.

By not accepting stuff at this point I would guess that the defacto freeze is here. Hope I'm wrong, there is some good stuff which would be ready by Oct 31.

Linus implied there would be one more last minute surge of merges. (Why else have people patch hoovering for anything other than bugs in the meantime?)

Probably the best thing to do is assemble a list of candidates, well-tested patches that have been up on the list and with as many endorsements from testers going "AOL!" as possible. The last go-through when lkinus gets back is probably going to be simple thumbs-up thumbs down on each entry in the pending feature list, and then into the 3.0-pre series...

The current "pending feature list" would probably be here:

http://kernelnewbies.org/status/

Specifically:

o in -ac PCMCIA Zoom video support (Alan Cox)
o in -ac Device mapper for Logical Volume Manager (LVM2) (LVM2 team)
o in -mm VM large page support (Many people)
o in -mm Page table sharing (Daniel Phillips, Dave McCracken)

o Ready Build option for Linux Trace Toolkit (LTT) (Karim Yaghmour)
o Ready Dynamic Probes (dprobes team)
o Ready Zerocopy NFS (Hirokazu Takahashi)
o Ready High resolution timers (George Anzinger, etc.)
o Ready EVMS (Enterprise Volume Management System) (EVMS team)
o Ready Linux Kernel Crash Dumps (Matt Robinson, LKCD team)
o Ready Rewrite of the console layer (James Simmons)

If you're not on this list, contact Guillaume, or start your own list of "final merge candidates"...

Rob

#4 lore raw
In reply to Bill Davidsen

I will start picking stuff up now I'm back and catching up on email (I've been away for a week). I'm not however interested in kexec or ltt or a lot of the other large stuff so don't bother resending me that kind of thing.

I really want to collect up bug fixes/compile fixes/driver updates and small but ready to merge stuff like the console updates if James Simmons is paying attention.

#5 lore raw
Quoted 8 lines · Alan Cox

I will start picking stuff up now I'm back and catching up on email (I've been away for a week). I'm not however interested in kexec or ltt or a lot of the other large stuff so don't bother resending me that kind of thing.

I really want to collect up bug fixes/compile fixes/driver updates and small but ready to merge stuff like the console updates if James Simmons is paying attention.

I am half insulted. kexec is about as ready to merge as it comes... But I do admit the code to safely position the new kernel safely in memory safely is a little large.

And there is not a lot of point in widespread testing of kexec until the drivers get fixed to actually do something when device_shutdown runs. As drivers that don't shutdown properly are a fairly serious issue.

If you are interested I can break out my reboot time hardware bug fixes, and send you those.

Eric

#6 lore raw
In reply to Alan Cox
Quoted 4 lines · Alan Cox

I will start picking stuff up now I'm back and catching up on email (I've been away for a week). I'm not however interested in kexec or ltt or a lot of the other large stuff so don't bother resending me that kind of thing.

I can appreciate that you are more interested in stability than features now, and that's good in general. However, someone should be looking at functional stuff like this, because it's (a) out and working and (b) adds a whole new and important capability to Linux.

When I look at things like zero copy NFS and Reiser-4, my first impression is that they are essentially incremental improvements to existing features. And as hard as I've yelled about NFS4, I have been told that is too. But things like JFS, XFS and kexec generate that "Holy shit, look what they added now!" feeling. And as a whole new capability I really would like to see it go in, even as I appreciate that it's not your job, I hope it becomes someone's job, because I sure boot a lot of slow computers on a regular basis :-(

Quoted 4 lines

I really want to collect up bug fixes/compile fixes/driver updates and small but ready to merge stuff like the console updates if James Simmons is paying attention.

I realize that you're busy, but there seem to be a shitload of drivers which work compiled in and not as modules.

Signature · 3 lines
bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
#7 lore raw
In reply to Eric W. Biederman

Hi!

Quoted 3 lines

The kexec code has gone through a fairly decent review, and all known bugs are resolved. There are still BIOS's that don't work after you have run a kernel but that is an entirely different problem.

Looks good... Few comments follow.

Quoted 16 lines
arch/i386/kernel/apic.c +12 -0
@@ -1128,6 +1159,26 @@
 	printk (KERN_INFO "APIC error on CPU%d: %02lx(%02lx)\n",
 	        smp_processor_id(), v , v1);
 	irq_exit();
+}
+
+void stop_apics(void)
+{
+	/* By resetting the APIC's we disable the nmi watchdog */
+#if CONFIG_SMP
+	/*
+	 * Stop all CPUs and turn off local APICs and the IO-APIC, so
+	 * other OSs see a clean IRQ state.
+	 */
+	smp_send_stop();
+#else

arch/i386/kernel/apic.c line 1183

Quoted 10 lines
arch/i386/kernel/apic.c +8 -0
+	disable_local_APIC();
+#endif
+#if defined(CONFIG_X86_IO_APIC)
+	if (smp_found_config) {
+		disable_IO_APIC();
+	}
+#endif
+	disconnect_bsp_APIC();
 }
 

Perhaps this should be done using driverfs callbacks?

Quoted 26 lines
arch/i386/kernel/i8259.c +20 -0
@@ -246,10 +246,34 @@
 	return 0;
 }
 
+static void i8259A_remove(struct device *dev)
+{   
+	/* Restore the i8259A to it's legacy dos setup.
+	 * The kernel won't be using it any more, and it
+	 * just might make reboots, and kexec type applications
+	 * more stable.
+	 */
+	outb(0xff, 0x21);	/* mask all of 8259A-1 */
+	outb(0xff, 0xA1);	/* mask all of 8259A-1 */
+
+	outb_p(0x11, 0x20);	/* ICW1: select 8259A-1 init */
+	outb_p(0x08, 0x21);	/* ICW2: 8259A-1 IR0-7 mappend to 0x8-0xf */
+	outb_p(0x01, 0x21);	/* Normal 8086 auto EOI mode */
+
+	outb_p(0x11, 0xA0);	/* ICW1: select 8259A-2 init */
+	outb_p(0x08, 0xA1);	/* ICW2: 8259A-2 IR0-7 mappend to 0x70-0x77 */
+	outb_p(0x01, 0xA1);	/* Normal 8086 auto EOI mode */
+
+	udelay(100);		/* wait for 8259A to initialize */
+

arch/i386/kernel/i8259.c line 278

Quoted 10 lines
arch/i386/kernel/i8259.c +4 -0
+	/* Should I unmask interrupts here?  */
+}
+
 static struct device_driver i8259A_driver = {
 	.name		= "pic",
 	.bus		= &system_bus_type,
 	.resume		= i8259A_resume,
+	.remove		= i8259A_remove,
 };
 

Here you did it right.
Pavel

Signature · 2 lines
Worst form of spam? Adding advertisment signatures ala sourceforge.net.
What goes next? Inserting advertisment *into* email?
#8 lore raw
Quoted 9 lines · Pavel Machek

Hi!

Quoted 3 lines

The kexec code has gone through a fairly decent review, and all known bugs are resolved. There are still BIOS's that don't work after you have run a kernel but that is an entirely different problem.

Looks good... Few comments follow.

Perhaps this should be done using driverfs callbacks?

SMP must be stopped and the APIC shutdown as the very last thing to happen. The order dependency is a very real, and ugly things happen when it doesn't happen as the very last thing. Does driverfs have a way to express that?

Eric

#9 lore raw
In reply to Pavel Machek
Quoted 10 lines · Pavel Machek

Hi!

Quoted 3 lines

The kexec code has gone through a fairly decent review, and all known bugs are resolved. There are still BIOS's that don't work after you have run a kernel but that is an entirely different problem.

Looks good... Few comments follow.

Perhaps this should be done using driverfs callbacks?

O.k. For the cpu shutdown case I am not certain if this can be modeled properly but here is a shot at thinking it through.

Drawing the APIC dependencies I get a tree is in parallel and usually flatter than the normal device tree.

boot_strap_processor
apic
|

  +----+--------+---------+--------------+
       |        |         |              |
       +        +         +              +
   IOAPIC   IOAPIC        CPU#2         CPU#3
    |
    +--+--------+---------------+--------- .....
       |        |               |
 Legacy PIC     PCI device 1  PCI device 2

The only sane way I can think to do is to put the apic tree, above the pci bus trees. This should preserve the requirement not to shutdown IOAPICs, before shutting down the pci devices, that send interrupts through them. The hard challenge is that IOAPICs can appear as pci devices, so it is at least possible for devices on one pci bus to depend on the IOAPIC on anther pci bus.

However, if we do not disable the PCI bridges/busses it should not be an issue. We just need to keep from disabling the IOAPICs, and other PICs until after we have shutdown all of the other devices.

To properly shutdown an SMP system so it can be reinitialized the PICs must be placed in legacy mode. Which means all of the IOAPICs must come down.

Does this sound like something that is reasonable to tackle? Getting it right is a challenge but I do have a good test case :)

Eric

#10 lore raw

[snip] discussion on how to model IOAPIC and pic and cpus using the device model.

I now need to go code this thing don't I? Or did I get lucky and get you interested Pavel?

Eric

#11 lore raw

Hi!

Quoted 5 lines

[snip] discussion on how to model IOAPIC and pic and cpus using the device model.

I now need to go code this thing don't I? Or did I get lucky and get you interested Pavel?

I'm not sure if putting APICS above PCI will look right to Patrick... You'd better ask him. (Sorry, I was offline for > week.)

Pavel

Signature · 2 lines
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.
#12 lore raw
Quoted 10 lines · Pavel Machek

Hi!

Quoted 5 lines

[snip] discussion on how to model IOAPIC and pic and cpus using the device model.

I now need to go code this thing don't I? Or did I get lucky and get you interested Pavel?

I'm not sure if putting APICS above PCI will look right to Patrick... You'd better ask him. (Sorry, I was offline for > week.)

Ok I will. I primarily meant to put an APIC bus above pci. And maybe filter the APICS down. The real challenge is that the APIC connection is in parallel with the PCI connection. It would take a DAG to truly model the connections between hardware, and I have been told that people don't want that...

Eric

#13 lore raw

PATCH 1/1 kexec 2.5.44 (minimal)

In reply to Pavel Machek

kexec is a system call that allows you to load another kernel from the currently executing Linux kernel. The current implementation has only been tested, and had the kinks worked out on x86, but the generic code should work on any architecture.

Some machines have BIOSes that are either extremely slow to reboot, or that cannot reliably perform a reboot. In which case kexec may be the only alternative to reboot in a reliable and timely manner.

The patch is archived at:
http://www.xmission.com/~ebiederm/files/kexec/

And is currently kept in two pieces.
The pure system call.
http://www.xmission.com/~ebiederm/files/kexec/linux-2.5.44.x86kexec-2.diff

And the set of hardware fixes known to help kexec.
http://www.xmission.com/~ebiederm/files/kexec/linux-2.5.44.x86kexec-hwfixes.diff

A compatible user space is at:
http://www.xmission.com/~ebiederm/files/kexec/kexec-tools-1.?.tar.gz
This code boots either a static ELF executable or a bzImage.

A kernel reformater that bypasses setup.S in favor of a version that uses fewer BIOS calls, (increasing the reliability) is at:
ftp://ftp.lnxi.com/pub/mkelfImage/mkelfImage-1.17.tar.gz

In bug reports please include the serial console output of kexec kexec_test. kexec_test exercises most of the interesting code paths that are needed to load a kernel (mainly BIOS calls) with lots of debugging print statements, so hangs can easily be detected.

I have been using this technique for the last several years and what the kernel needs to do is well understood, and currently implemented. I have also been working with etherboot which is a very minimal kernel whose sole purpose is to download a kernel over the network and boot it.

Quoted 1 line

From etherboot it is possible to download a DOS image and run it.

I have not yet reached this level of firmware reliably after using the kexec syscall, but the fact that etherboot does it shows it can be done. Having worked with etherboot as a primary bootloader I had forgotten how challenging it is theoretically to shut down a kernel that uses it's own hardware drivers and have the firmware still work reliably.

My expectation is that I can have complete BIOS functionality when I compile a kernel whose sole user space is a ramdisk, the kernel has no hardware drivers, and I use a bootloader that does not mess up the BIOS. And I expect that using kernel drivers will only mess up the firmware drivers where they work with the same hardware.

So far failures have broken into three categories. Broken kernels even without kexec. Kernels that hang when running setup.S during bootup. Kernels that don't quite boot because a the kernel driver cannot reinitialize the hardware from the state that same driver left the hardware in.

For failures during setup.S I have seen three cases. DOS running before loadlin modified the real mode IDT so as to be useless without DOS. Interrupt 0x13 ah=0x15 print dasd type. And the new EDD code. These failures fall into the expected cases. But except for the loadlin case which is hopeless have not been completely tracked.

In the plan forward there are several goals. To find what it takes to have the BIOS work reliably after a kexec system call. To boot bzImages reliably, by bypassing setup.S completely. To fix the linux device drivers that prevent a kernel called with sys_kexec from initializing.

An important point is that none of these future projects require modifying the current kexec system call. The needed work is either in user space or in the device drivers.

Shortly Linus will be getting back and I will be submitting the kexec system call to him. Below is the patch I will be aiming at getting into the kernel. All of the device driver fixes have been removed in the interest of having a clean patch to submit. In particular to use this from an SMP kernel you will also need to apply linux-2.5.44.x86kexec-hwfixes.diff.

Thanks to everyone who has provided me with feedback so far.

Eric

 MAINTAINERS                        |    7 
 arch/i386/Config.help              |   14 
 arch/i386/config.in                |    3 
 arch/i386/kernel/Makefile          |    1 
 arch/i386/kernel/entry.S           |    1 
 arch/i386/kernel/machine_kexec.c   |  142 ++++++++
 arch/i386/kernel/relocate_kernel.S |   99 +++++
 include/asm-i386/kexec.h           |   25 +
 include/asm-i386/unistd.h          |    2 
 include/linux/kexec.h              |   48 ++
 kernel/Makefile                    |    1 
 kernel/kexec.c                     |  624 +++++++++++++++++++++++++++++++++++++
 kernel/sys.c                       |   61 +++
 13 files changed, 1027 insertions, 1 deletion
MAINTAINERS +7 -0
diff -uNr linux-2.5.44/MAINTAINERS linux-2.5.44.x86kexec-2/MAINTAINERS
@@ -934,6 +934,13 @@
 W:	http://www.cse.unsw.edu.au/~neilb/patches/linux-devel/
 S:	Maintained
 
+KEXEC
+P:	Eric Biederman
+M:	[email protected]
+M:	[email protected]
+L:	[email protected]
+S:	Maintained
+
 LANMEDIA WAN CARD DRIVER
 P:	Andrew Stanley-Jones
 M:	[email protected]
arch/i386/Config.help +14 -0
diff -uNr linux-2.5.44/arch/i386/Config.help linux-2.5.44.x86kexec-2/arch/i386/Config.help
@@ -417,6 +417,20 @@
   you have use for it; the module is called binfmt_misc.o. If you
   don't know what to answer at this point, say Y.
 
+CONFIG_KEXEC
+  kexec is a system call that implements the ability to  shutdown your
+  current kernel, and to start another kernel.  It is like a reboot
+  but it is indepedent of the system firmware.   And like a reboot the
+  you can start any kernel with it not just Linux.  
+
+  The name comes from the similiarity to the exec system call. 
+
+  It is on an going process to be certain the hardware in a machine
+  is properly shutdown, so do not be surprised if this code does not
+  initially work for you.  It may help to enable device hotplugging
+  support.  As of this writing the exact hardware interface is
+  strongly in flux, so no good recommendation can be made.
+
 CONFIG_M386
   This is the processor type of your CPU. This information is used for
   optimizing purposes. In order to compile a kernel that can run on
arch/i386/config.in +3 -0
diff -uNr linux-2.5.44/arch/i386/config.in linux-2.5.44.x86kexec-2/arch/i386/config.in
@@ -247,6 +247,9 @@
    fi
 fi
 
+if [ "$CONFIG_EXPERIMENTAL" = "y" ]; then
+   bool 'kexec system call (EXPERIMENTAL)' CONFIG_KEXEC
+fi
 endmenu
 
 mainmenu_option next_comment
arch/i386/kernel/Makefile +1 -0
diff -uNr linux-2.5.44/arch/i386/kernel/Makefile linux-2.5.44.x86kexec-2/arch/i386/kernel/Makefile
@@ -25,6 +25,7 @@
 obj-$(CONFIG_X86_MPPARSE)	+= mpparse.o
 obj-$(CONFIG_X86_LOCAL_APIC)	+= apic.o nmi.o
 obj-$(CONFIG_X86_IO_APIC)	+= io_apic.o
+obj-$(CONFIG_KEXEC)		+= machine_kexec.o relocate_kernel.o
 obj-$(CONFIG_SOFTWARE_SUSPEND)	+= suspend.o
 obj-$(CONFIG_X86_NUMAQ)		+= numaq.o
 obj-$(CONFIG_PROFILING)		+= profile.o
arch/i386/kernel/entry.S +1 -0
diff -uNr linux-2.5.44/arch/i386/kernel/entry.S linux-2.5.44.x86kexec-2/arch/i386/kernel/entry.S
@@ -737,6 +737,7 @@
 	.long sys_free_hugepages
 	.long sys_exit_group
 	.long sys_lookup_dcookie
+	.long sys_kexec
 
 	.rept NR_syscalls-(.-sys_call_table)/4
 		.long sys_ni_syscall
arch/i386/kernel/machine_kexec.c +142 -0

142 lines not on this page · raw message

arch/i386/kernel/relocate_kernel.S +99 -0

99 lines not on this page · raw message

include/asm-i386/kexec.h +25 -0
diff -uNr linux-2.5.44/include/asm-i386/kexec.h linux-2.5.44.x86kexec-2/include/asm-i386/kexec.h
@@ -0,0 +1,25 @@
+#ifndef _I386_KEXEC_H
+#define _I386_KEXEC_H
+
+#include <asm/fixmap.h>
+
+/*
+ * KEXEC_SOURCE_MEMORY_LIMIT maximum page get_free_page can return.
+ * I.e. Maximum page that is mapped directly into kernel memory,
+ * and kmap is not required.
+ *
+ * Someone correct me if FIXADDR_START - PAGEOFFSET is not the correct
+ * calculation for the amount of memory directly mappable into the
+ * kernel memory space.
+ */
+
+/* Maximum physical address we can use pages from */
+#define KEXEC_SOURCE_MEMORY_LIMIT (FIXADDR_START - PAGE_OFFSET) 
+/* Maximum address we can reach in physical address mode */
+#define KEXEC_DESTINATION_MEMORY_LIMIT (-1UL)
+
+#define KEXEC_REBOOT_CODE_SIZE	4096
+#define KEXEC_REBOOT_CODE_ALIGN 0
+
+
+#endif /* _I386_KEXEC_H */
include/asm-i386/unistd.h +1 -1
diff -uNr linux-2.5.44/include/asm-i386/unistd.h linux-2.5.44.x86kexec-2/include/asm-i386/unistd.h
@@ -258,7 +258,7 @@
 #define __NR_free_hugepages	251
 #define __NR_exit_group		252
 #define __NR_lookup_dcookie	253
-  
+#define __NR_kexec		254
 
 /* user-visible error numbers are in the range -1 - -124: see <asm-i386/errno.h> */
 
include/linux/kexec.h +48 -0

48 lines not on this page · raw message

kernel/Makefile +1 -0
diff -uNr linux-2.5.44/kernel/Makefile linux-2.5.44.x86kexec-2/kernel/Makefile
@@ -21,6 +21,7 @@
 obj-$(CONFIG_CPU_FREQ) += cpufreq.o
 obj-$(CONFIG_BSD_PROCESS_ACCT) += acct.o
 obj-$(CONFIG_SOFTWARE_SUSPEND) += suspend.o
+obj-$(CONFIG_KEXEC) += kexec.o
 
 ifneq ($(CONFIG_IA64),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is
kernel/kexec.c +624 -0

624 lines not on this page · raw message

kernel/sys.c +61 -0

73 lines not on this page · raw message

#14 lore raw

Re: [CFT] [PATCH] kexec 2.5.44 (minimal)

Quoted 6 lines · Eric W. Biederman

And is currently kept in two pieces.
The pure system call.
http://www.xmission.com/~ebiederm/files/kexec/linux-2.5.44.x86kexec-2.diff

And the set of hardware fixes known to help kexec.
http://www.xmission.com/~ebiederm/files/kexec/linux-2.5.44.x86kexec-hwfixes.diff

Eric,

Hmmm... I'm having a lot more problems on my troublesome machine with this patchset than I did with the previous iteration. I've triple-checked the application of the patches, but I can't even get kexec_test to start, much less run to completion.

The new behavior is that the system appears to hang immediately after invoking "kexec kexec_test".

What could I have done wrong?

Thanks,
Andy

#15 lore raw

Re: [CFT] [PATCH] kexec 2.5.44 (minimal)

Quoted 17 lines · Andy Pfiffer
Quoted 6 lines · Eric W. Biederman

And is currently kept in two pieces.
The pure system call.
http://www.xmission.com/~ebiederm/files/kexec/linux-2.5.44.x86kexec-2.diff

And the set of hardware fixes known to help kexec.

http://www.xmission.com/~ebiederm/files/kexec/linux-2.5.44.x86kexec-hwfixes.diff

Eric,

Hmmm... I'm having a lot more problems on my troublesome machine with this patchset than I did with the previous iteration. I've triple-checked the application of the patches, but I can't even get kexec_test to start, much less run to completion.

Quoted 4 lines

The new behavior is that the system appears to hang immediately after invoking "kexec kexec_test".

What could I have done wrong?

With out some amount of output I don't have a clue.

With just the first patch it doesn't work on SMP, but I don't try it to see what the failure mode was.

With the second patch all should be the same except the legacy pic actually gets disabled. I wonder if disabling the legacy pic locks your system?

There should be two printks one about shuting down devices, and another about kexecing the image. And knowing when which of those printed could be useful.

In any event. Remove the shutdown routine in arch/i386/kernel/i8259.c That should restore the kernel to it's previous behavior if something really changed...

static struct device_driver i8259A_driver = {
.name = "pic",
.bus = &system_bus_type,
.resume = i8259A_resume,
- .shutdown = i8259A_shutdown,
};

Or you could try it with a uniprocessor kernel and just the first patch.

Eric