Skip to contentSkip to conversation

Xen i386 paravirtualization support

RFC PATCH In review Chris Wright 17 participants 113 messages Mar 22, 2006 – Mar 28, 2006

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

cover Xen i386 paravirtualization support

Unlike full virtualization in which the virtual machine provides the same platform interface as running natively on the hardware, paravirtualization requires modification to the guest operating system to work with the platform interface provided by the hypervisor.

Xen was designed with performance in mind. Calls to the hypervisor are minimized, batched if necessary, and non-critical codepaths are left unmodified in the case where the privileged instruction can be trapped and emulated by the hypervisor. The Xen API is designed to be OS agnostic and has had Linux, NetBSD, FreeBSD, Solaris, Plan9 and Netware ported to it. Xen also provides support for running directly on native hardware.

The following patch series provides the minimal support required to launch Xen paravirtual guests on standard x86 hardware running the Xen hypervisor. These patches effectively port the Linux kernel to run on the platform interface provided by Xen. This port is done as an i386 subarch.

With these patches you will be able to launch an unprivileged guest running the modified Linux kernel and unmodified userspace. This guest is x86, UP only, runs in shadow translated mode, and has no direct access to hardware. This simplifies the patchset to the minimum functionality needed to support a paravirtualized guest. It's worth noting that a fair amount of this patchset deals with paravirtualizing I/O, not just CPU-only. The additional missing functionality is primarily about full SMP support, optimizations such as direct writable page tables, and the management interface. Those refinements will be posted later.

At a high-level, the patches provide the following:

- Kconfig and Makefile changes required to support Xen
- subarch changes to allow more platform functionality to be implemented by an i386 subarch
- Xen subarch implementation
- start of day code for running in the hypervisor provided environment (paging enabled)
- basic Xen drivers to provide a fully functional guest

The Xen platform API encapsulates the following types of requirements:

- idt, gdt, ldt (descriptor table handling)
- cr2, fpu_taskswitch, debug registers (privileged register handling)
- mmu (page table, tlb, and cache handling)
- memory reservations
- time and timer
- vcpu (init, up/down vcpu)
- schedule (processor yield, shutdown, etc)
- event channel (generalized virtual interrupt handling)
- grant table (shared memory interface for high speed interdomain communication)
- block device I/O
- network device I/O
- console device I/O
- Xen feature map
- Xen version info

Thanks to those who provided early feedback to this patchset: Andi Kleen, Gerd Hoffmann, Jan Beulich, Rik van Riel, Stephen Tweedie, and the Xen team. And thanks to the Xen community: AMD, HP, IBM, Intel, Novell, Red Hat, Virtual Iron, XenSource -- see Xen changelog for full details.

Known issues:

CodingStyle conformance is still in progress
/proc interface needs to be replaced with something more appropriate entry.S cleanups are possible

#1 lore raw

RFC PATCH 1/35 Add XEN config options and disable unsupported config options.

The XEN config option is selected from the i386 subarch menu by choosing the X86_XEN "Xen-compatible" subarch.

The XEN_SHADOW_MODE option defines the memory virtualization mode for the kernel -- with it enabled, the kernel expects the hypervisor to perform translation between pseudo-physical and machine addresses on its behalf.

The disabled config options are:
- DOUBLEFAULT: are trapped by Xen and not virtualized
- HZ: defaults to 100 in Xen VMs
- Power management: not supported in unprivileged VMs
- SMP: not supported in this set of patches
- X86_{UP,LOCAL,IO}_APIC: not supported in unprivileged VMs

---
 arch/i386/Kconfig      |   18 ++++++++++++++----
 drivers/xen/Kconfig    |   21 +++++++++++++++++++++
 kernel/Kconfig.hz      |    4 ++--
 kernel/Kconfig.preempt |    1 +
 4 files changed, 38 insertions(+), 6 deletions(-)
arch/i386/Kconfig +14 -4
@@ -58,6 +58,12 @@ config X86_PC
 	help
 	  Choose this option if your computer is a standard PC or compatible.
 
+config X86_XEN
+	bool "Xen-compatible"
+	help
+	  Choose this option if you plan to run this kernel on top of the
+	  Xen Hypervisor.
+
 config X86_ELAN
 	bool "AMD Elan"
 	help
@@ -175,6 +181,7 @@ config HPET_EMULATE_RTC
 
 config SMP
 	bool "Symmetric multi-processing support"
+	depends on !X86_XEN
 	---help---
 	  This enables support for systems with more than one CPU. If you have
 	  a system with only one CPU, like most personal computers, say N. If
@@ -230,7 +237,7 @@ source "kernel/Kconfig.preempt"
 
 config X86_UP_APIC
 	bool "Local APIC support on uniprocessors"
-	depends on !SMP && !(X86_VISWS || X86_VOYAGER)
+	depends on !SMP && !(X86_VISWS || X86_VOYAGER || X86_XEN)
 	help
 	  A local APIC (Advanced Programmable Interrupt Controller) is an
 	  integrated interrupt controller in the CPU. If you have a single-CPU
@@ -255,12 +262,12 @@ config X86_UP_IOAPIC
 
 config X86_LOCAL_APIC
 	bool
-	depends on X86_UP_APIC || ((X86_VISWS || SMP) && !X86_VOYAGER)
+	depends on X86_UP_APIC || ((X86_VISWS || SMP) && !(X86_VOYAGER || X86_XEN))
 	default y
 
 config X86_IO_APIC
 	bool
-	depends on X86_UP_IOAPIC || (SMP && !(X86_VISWS || X86_VOYAGER))
+	depends on X86_UP_IOAPIC || (SMP && !(X86_VISWS || X86_VOYAGER || X86_XEN))
 	default y
 
 config X86_VISWS_APIC
@@ -743,6 +750,7 @@ config HOTPLUG_CPU
 config DOUBLEFAULT
 	default y
 	bool "Enable doublefault exception handler" if EMBEDDED
+	depends on !X86_XEN
 	help
           This option allows trapping of rare doublefault exceptions that
           would otherwise cause a system to silently reboot. Disabling this
@@ -753,7 +761,7 @@ endmenu
 
 
 menu "Power management options (ACPI, APM)"
-	depends on !X86_VOYAGER
+	depends on !(X86_VOYAGER || X86_XEN)
 
 source kernel/power/Kconfig
 
@@ -1075,6 +1083,8 @@ source "security/Kconfig"
 
 source "crypto/Kconfig"
 
+source "drivers/xen/Kconfig"
+
 source "lib/Kconfig"
 
 #
kernel/Kconfig.hz +2 -2
@@ -3,7 +3,7 @@
 #
 
 choice
-	prompt "Timer frequency"
+	prompt "Timer frequency" if !XEN
 	default HZ_250
 	help
 	 Allows the configuration of the timer frequency. It is customary
@@ -40,7 +40,7 @@ endchoice
 
 config HZ
 	int
-	default 100 if HZ_100
+	default 100 if HZ_100 || XEN
 	default 250 if HZ_250
 	default 1000 if HZ_1000
 
kernel/Kconfig.preempt +1 -0
@@ -35,6 +35,7 @@ config PREEMPT_VOLUNTARY
 
 config PREEMPT
 	bool "Preemptible Kernel (Low-Latency Desktop)"
+	depends on !XEN
 	help
 	  This option reduces the latency of the kernel by making
 	  all kernel code (that is not executing in a critical section)
drivers/xen/Kconfig new file +21 -0
@@ -0,0 +1,21 @@
+#
+# This Kconfig describe xen options
+#
+
+mainmenu "Xen Configuration"
+
+config XEN
+	bool
+	default y if X86_XEN
+	help
+	  This is the Linux Xen port.
+
+if XEN
+
+config XEN_SHADOW_MODE
+	bool
+	default y
+	help
+	  Fakes out a shadow mode kernel
+
+endif

--

#2 lore raw

RFC PATCH 2/35 Makefile support to build Xen subarch

In reply to Chris Wright

Use arch/i386/mach-xen when building Xen subarch. The separate subarchitecture allows us to hide details of interfacing with the hypervisor from i386 common code.

---
 arch/i386/Makefile          |    5 +++++
 arch/i386/mach-xen/Makefile |    7 +++++++
 2 files changed, 12 insertions(+)
arch/i386/Makefile +5 -0
@@ -68,6 +68,10 @@ mcore-$(CONFIG_X86_BIGSMP)	:= mach-defau
 mflags-$(CONFIG_X86_SUMMIT) := -Iinclude/asm-i386/mach-summit
 mcore-$(CONFIG_X86_SUMMIT)  := mach-default
 
+# Xen subarch support
+mflags-$(CONFIG_X86_XEN)	:= -Iinclude/asm-i386/mach-xen
+mcore-$(CONFIG_X86_XEN)		:= mach-xen
+
 # generic subarchitecture
 mflags-$(CONFIG_X86_GENERICARCH) := -Iinclude/asm-i386/mach-generic
 mcore-$(CONFIG_X86_GENERICARCH) := mach-default
@@ -96,6 +100,7 @@ drivers-$(CONFIG_PM)			+= arch/i386/powe
 
 CFLAGS += $(mflags-y)
 AFLAGS += $(mflags-y)
+CPPFLAGS += $(mflags-y)
 
 boot := arch/i386/boot
 
arch/i386/mach-xen/Makefile new file +7 -0
@@ -0,0 +1,7 @@
+#
+# Makefile for the linux kernel.
+#
+
+obj-y				:= setup.o
+ 
+setup-y				:= ../mach-default/setup.o

--

#3 lore raw

RFC PATCH 3/35 Add Xen interface header files

In reply to Chris Wright

Add Xen interface header files. These are taken fairly directly from the Xen tree and hence the style is not entirely in accordance with Linux guidelines. There is a tension between fitting with Linux coding rules and ease of maintenance.

---
 include/xen/interface/arch-x86_32.h   |  185 ++++++++++++++
 include/xen/interface/event_channel.h |  205 +++++++++++++++
 include/xen/interface/features.h      |   53 ++++
 include/xen/interface/grant_table.h   |  311 +++++++++++++++++++++++
 include/xen/interface/io/blkif.h      |   85 ++++++
 include/xen/interface/io/console.h    |   33 ++
 include/xen/interface/io/netif.h      |   76 +++++
 include/xen/interface/io/ring.h       |  262 ++++++++++++++++++++
 include/xen/interface/io/xenbus.h     |   42 +++
 include/xen/interface/io/xs_wire.h    |   97 +++++++
 include/xen/interface/memory.h        |  155 +++++++++++
 include/xen/interface/physdev.h       |   71 +++++
 include/xen/interface/sched.h         |   87 ++++++
 include/xen/interface/vcpu.h          |  119 +++++++++
 include/xen/interface/version.h       |   70 +++++
 include/xen/interface/xen.h           |  441 ++++++++++++++++++++++++++++++++++
 16 files changed, 2292 insertions(+)
include/xen/interface/arch-x86_32.h new file +185 -0
@@ -0,0 +1,185 @@
+/******************************************************************************
+ * arch-x86_32.h
+ * 
+ * Guest OS interface to x86 32-bit Xen.
+ * 
+ * Copyright (c) 2004, K A Fraser
+ */
+
+#ifndef __XEN_PUBLIC_ARCH_X86_32_H__
+#define __XEN_PUBLIC_ARCH_X86_32_H__
+
+#ifdef __XEN__
+#define __DEFINE_GUEST_HANDLE(name, type) \
+    typedef struct { type *p; } __guest_handle_ ## name
+#else
+#define __DEFINE_GUEST_HANDLE(name, type) \
+    typedef type * __guest_handle_ ## name
+#endif
+
+#define DEFINE_GUEST_HANDLE_STRUCT(name) \
+	__DEFINE_GUEST_HANDLE(name, struct name)
+#define DEFINE_GUEST_HANDLE(name) __DEFINE_GUEST_HANDLE(name, name)
+#define GUEST_HANDLE(name)        __guest_handle_ ## name
+
+#ifndef __ASSEMBLY__
+/* Guest handles for primitive C types. */
+__DEFINE_GUEST_HANDLE(uchar, unsigned char);
+__DEFINE_GUEST_HANDLE(uint,  unsigned int);
+__DEFINE_GUEST_HANDLE(ulong, unsigned long);
+DEFINE_GUEST_HANDLE(char);
+DEFINE_GUEST_HANDLE(int);
+DEFINE_GUEST_HANDLE(long);
+DEFINE_GUEST_HANDLE(void);
+#endif
+
+/*
+ * SEGMENT DESCRIPTOR TABLES
+ */
+/*
+ * A number of GDT entries are reserved by Xen. These are not situated at the
+ * start of the GDT because some stupid OSes export hard-coded selector values
+ * in their ABI. These hard-coded values are always near the start of the GDT,
+ * so Xen places itself out of the way, at the far end of the GDT.
+ */
+#define FIRST_RESERVED_GDT_PAGE  14
+#define FIRST_RESERVED_GDT_BYTE  (FIRST_RESERVED_GDT_PAGE * 4096)
+#define FIRST_RESERVED_GDT_ENTRY (FIRST_RESERVED_GDT_BYTE / 8)
+
+/*
+ * These flat segments are in the Xen-private section of every GDT. Since these
+ * are also present in the initial GDT, many OSes will be able to avoid
+ * installing their own GDT.
+ */
+#define FLAT_RING1_CS 0xe019    /* GDT index 259 */
+#define FLAT_RING1_DS 0xe021    /* GDT index 260 */
+#define FLAT_RING1_SS 0xe021    /* GDT index 260 */
+#define FLAT_RING3_CS 0xe02b    /* GDT index 261 */
+#define FLAT_RING3_DS 0xe033    /* GDT index 262 */
+#define FLAT_RING3_SS 0xe033    /* GDT index 262 */
+
+#define FLAT_KERNEL_CS FLAT_RING1_CS
+#define FLAT_KERNEL_DS FLAT_RING1_DS
+#define FLAT_KERNEL_SS FLAT_RING1_SS
+#define FLAT_USER_CS    FLAT_RING3_CS
+#define FLAT_USER_DS    FLAT_RING3_DS
+#define FLAT_USER_SS    FLAT_RING3_SS
+
+/* And the trap vector is... */
+#define TRAP_INSTR "int $0x82"
+
+/*
+ * Virtual addresses beyond this are not modifiable by guest OSes. The 
+ * machine->physical mapping table starts at this address, read-only.
+ */
+#ifdef CONFIG_X86_PAE
+#define __HYPERVISOR_VIRT_START 0xF5800000
+#else
+#define __HYPERVISOR_VIRT_START 0xFC000000
+#endif
+
+#ifndef HYPERVISOR_VIRT_START
+#define HYPERVISOR_VIRT_START mk_unsigned_long(__HYPERVISOR_VIRT_START)
+#endif
+
+#ifndef machine_to_phys_mapping
+#define machine_to_phys_mapping ((unsigned long *)HYPERVISOR_VIRT_START)
+#endif
+
+/* Maximum number of virtual CPUs in multi-processor guests. */
+#define MAX_VIRT_CPUS 32
+
+#ifndef __ASSEMBLY__
+
+/*
+ * Send an array of these to HYPERVISOR_set_trap_table()
+ */
+#define TI_GET_DPL(_ti)      ((_ti)->flags & 3)
+#define TI_GET_IF(_ti)       ((_ti)->flags & 4)
+#define TI_SET_DPL(_ti,_dpl) ((_ti)->flags |= (_dpl))
+#define TI_SET_IF(_ti,_if)   ((_ti)->flags |= ((!!(_if))<<2))
+struct trap_info {
+    uint8_t       vector;  /* exception vector                              */
+    uint8_t       flags;   /* 0-3: privilege level; 4: clear event enable?  */
+    uint16_t      cs;      /* code selector                                 */
+    unsigned long address; /* code offset                                   */
+};
+DEFINE_GUEST_HANDLE_STRUCT(trap_info);
+
+struct cpu_user_regs {
+    uint32_t ebx;
+    uint32_t ecx;
+    uint32_t edx;
+    uint32_t esi;
+    uint32_t edi;
+    uint32_t ebp;
+    uint32_t eax;
+    uint16_t error_code;    /* private */
+    uint16_t entry_vector;  /* private */
+    uint32_t eip;
+    uint16_t cs;
+    uint8_t  saved_upcall_mask;
+    uint8_t  _pad0;
+    uint32_t eflags;        /* eflags.IF == !saved_upcall_mask */
+    uint32_t esp;
+    uint16_t ss, _pad1;
+    uint16_t es, _pad2;
+    uint16_t ds, _pad3;
+    uint16_t fs, _pad4;
+    uint16_t gs, _pad5;
+};
+DEFINE_GUEST_HANDLE_STRUCT(cpu_user_regs);
+
+typedef uint64_t tsc_timestamp_t; /* RDTSC timestamp */
+
+/*
+ * The following is all CPU context. Note that the fpu_ctxt block is filled 
+ * in by FXSAVE if the CPU has feature FXSR; otherwise FSAVE is used.
+ */
+struct vcpu_guest_context {
+    /* FPU registers come first so they can be aligned for FXSAVE/FXRSTOR. */
+    struct { char x[512]; } fpu_ctxt;       /* User-level FPU registers     */
+#define VGCF_I387_VALID (1<<0)
+#define VGCF_HVM_GUEST  (1<<1)
+#define VGCF_IN_KERNEL  (1<<2)
+    unsigned long flags;                    /* VGCF_* flags                 */
+    struct cpu_user_regs user_regs;         /* User-level CPU registers     */
+    struct trap_info trap_ctxt[256];        /* Virtual IDT                  */
+    unsigned long ldt_base, ldt_ents;       /* LDT (linear address, # ents) */
+    unsigned long gdt_frames[16], gdt_ents; /* GDT (machine frames, # ents) */
+    unsigned long kernel_ss, kernel_sp;     /* Virtual TSS (only SS1/SP1)   */
+    unsigned long ctrlreg[8];               /* CR0-CR7 (control registers)  */
+    unsigned long debugreg[8];              /* DB0-DB7 (debug registers)    */
+    unsigned long event_callback_cs;        /* CS:EIP of event callback     */
+    unsigned long event_callback_eip;
+    unsigned long failsafe_callback_cs;     /* CS:EIP of failsafe callback  */
+    unsigned long failsafe_callback_eip;
+    unsigned long vm_assist;                /* VMASST_TYPE_* bitmap */
+};
+DEFINE_GUEST_HANDLE_STRUCT(vcpu_guest_context);
+
+struct arch_shared_info {
+    unsigned long max_pfn;                  /* max pfn that appears in table */
+    /* Frame containing list of mfns containing list of mfns containing p2m. */
+    unsigned long pfn_to_mfn_frame_list_list;
+    unsigned long nmi_reason;
+};
+
+struct arch_vcpu_info {
+    unsigned long cr2;
+    unsigned long pad[5]; /* sizeof(struct vcpu_info) == 64 */
+};
+
+#endif /* !__ASSEMBLY__ */
+
+#endif
+
+/*
+ * Local variables:
+ * mode: C
+ * c-set-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
include/xen/interface/event_channel.h new file +205 -0

205 lines not on this page · raw message

include/xen/interface/features.h new file +53 -0

53 lines not on this page · raw message

include/xen/interface/grant_table.h new file +311 -0

311 lines not on this page · raw message

include/xen/interface/io/blkif.h new file +85 -0

85 lines not on this page · raw message

include/xen/interface/io/console.h new file +33 -0
@@ -0,0 +1,33 @@
+/******************************************************************************
+ * console.h
+ * 
+ * Console I/O interface for Xen guest OSes.
+ * 
+ * Copyright (c) 2005, Keir Fraser
+ */
+
+#ifndef __XEN_PUBLIC_IO_CONSOLE_H__
+#define __XEN_PUBLIC_IO_CONSOLE_H__
+
+typedef uint32_t XENCONS_RING_IDX;
+
+#define MASK_XENCONS_IDX(idx, ring) ((idx) & (sizeof(ring)-1))
+
+struct xencons_interface {
+    char in[1024];
+    char out[2048];
+    XENCONS_RING_IDX in_cons, in_prod;
+    XENCONS_RING_IDX out_cons, out_prod;
+};
+
+#endif /* __XEN_PUBLIC_IO_CONSOLE_H__ */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-set-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
include/xen/interface/io/netif.h new file +76 -0

76 lines not on this page · raw message

include/xen/interface/io/ring.h new file +262 -0

262 lines not on this page · raw message

include/xen/interface/io/xenbus.h new file +42 -0

42 lines not on this page · raw message

include/xen/interface/io/xs_wire.h new file +97 -0

97 lines not on this page · raw message

include/xen/interface/memory.h new file +155 -0

155 lines not on this page · raw message

include/xen/interface/physdev.h new file +71 -0

71 lines not on this page · raw message

include/xen/interface/sched.h new file +87 -0

87 lines not on this page · raw message

include/xen/interface/vcpu.h new file +119 -0

119 lines not on this page · raw message

include/xen/interface/version.h new file +70 -0

70 lines not on this page · raw message

include/xen/interface/xen.h new file +441 -0

441 lines not on this page · raw message

--

#4 lore raw

RFC PATCH 4/35 Hypervisor interface header files.

In reply to Chris Wright

Define macros and inline functions for doing hypercalls into the hypervisor.

---
 include/asm-i386/hypercall.h  |  306 ++++++++++++++++++++++++++++++++++++++++++
 include/asm-i386/hypervisor.h |   67 +++++++++
 2 files changed, 373 insertions(+)
include/asm-i386/hypercall.h new file +306 -0
@@ -0,0 +1,306 @@
+/******************************************************************************
+ * hypercall.h
+ *
+ * Linux-specific hypervisor handling.
+ *
+ * Copyright (c) 2002-2004, K A Fraser
+ *
+ * This file may be distributed separately from the Linux kernel, or
+ * incorporated into other software packages, subject to the following license:
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this source file (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use, copy, modify,
+ * merge, publish, distribute, sublicense, and/or sell copies of the Software,
+ * and to permit persons to whom the Software is furnished to do so, subject to
+ * the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ */
+
+#ifndef __HYPERCALL_H__
+#define __HYPERCALL_H__
+
+#include <linux/config.h>
+
+#include <xen/interface/xen.h>
+#include <xen/interface/sched.h>
+
+#define __STR(x) #x
+#define STR(x) __STR(x)
+
+#define _hypercall0(type, name)			\
+({						\
+	long __res;				\
+	asm volatile (				\
+		"call hypercall_page + ("STR(__HYPERVISOR_##name)" * 32)"\
+		: "=a" (__res)			\
+		:				\
+		: "memory" );			\
+	(type)__res;				\
+})
+
+#define _hypercall1(type, name, a1)				\
+({								\
+	long __res, __ign1;					\
+	asm volatile (						\
+		"call hypercall_page + ("STR(__HYPERVISOR_##name)" * 32)"\
+		: "=a" (__res), "=b" (__ign1)			\
+		: "1" ((long)(a1))				\
+		: "memory" );					\
+	(type)__res;						\
+})
+
+#define _hypercall2(type, name, a1, a2)				\
+({								\
+	long __res, __ign1, __ign2;				\
+	asm volatile (						\
+		"call hypercall_page + ("STR(__HYPERVISOR_##name)" * 32)"\
+		: "=a" (__res), "=b" (__ign1), "=c" (__ign2)	\
+		: "1" ((long)(a1)), "2" ((long)(a2))		\
+		: "memory" );					\
+	(type)__res;						\
+})
+
+#define _hypercall3(type, name, a1, a2, a3)			\
+({								\
+	long __res, __ign1, __ign2, __ign3;			\
+	asm volatile (						\
+		"call hypercall_page + ("STR(__HYPERVISOR_##name)" * 32)"\
+		: "=a" (__res), "=b" (__ign1), "=c" (__ign2), 	\
+		"=d" (__ign3)					\
+		: "1" ((long)(a1)), "2" ((long)(a2)),		\
+		"3" ((long)(a3))				\
+		: "memory" );					\
+	(type)__res;						\
+})
+
+#define _hypercall4(type, name, a1, a2, a3, a4)			\
+({								\
+	long __res, __ign1, __ign2, __ign3, __ign4;		\
+	asm volatile (						\
+		"call hypercall_page + ("STR(__HYPERVISOR_##name)" * 32)"\
+		: "=a" (__res), "=b" (__ign1), "=c" (__ign2),	\
+		"=d" (__ign3), "=S" (__ign4)			\
+		: "1" ((long)(a1)), "2" ((long)(a2)),		\
+		"3" ((long)(a3)), "4" ((long)(a4))		\
+		: "memory" );					\
+	(type)__res;						\
+})
+
+#define _hypercall5(type, name, a1, a2, a3, a4, a5)		\
+({								\
+	long __res, __ign1, __ign2, __ign3, __ign4, __ign5;	\
+	asm volatile (						\
+		"call hypercall_page + ("STR(__HYPERVISOR_##name)" * 32)"\
+		: "=a" (__res), "=b" (__ign1), "=c" (__ign2),	\
+		"=d" (__ign3), "=S" (__ign4), "=D" (__ign5)	\
+		: "1" ((long)(a1)), "2" ((long)(a2)),		\
+		"3" ((long)(a3)), "4" ((long)(a4)),		\
+		"5" ((long)(a5))				\
+		: "memory" );					\
+	(type)__res;						\
+})
+
+static inline int
+HYPERVISOR_set_trap_table(
+	struct trap_info *table)
+{
+	return _hypercall1(int, set_trap_table, table);
+}
+
+static inline int
+HYPERVISOR_mmu_update(
+	struct mmu_update *req, int count, int *success_count, domid_t domid)
+{
+	return _hypercall4(int, mmu_update, req, count, success_count, domid);
+}
+
+static inline int
+HYPERVISOR_mmuext_op(
+	struct mmuext_op *op, int count, int *success_count, domid_t domid)
+{
+	return _hypercall4(int, mmuext_op, op, count, success_count, domid);
+}
+
+static inline int
+HYPERVISOR_set_gdt(
+	unsigned long *frame_list, int entries)
+{
+	return _hypercall2(int, set_gdt, frame_list, entries);
+}
+
+static inline int
+HYPERVISOR_stack_switch(
+	unsigned long ss, unsigned long esp)
+{
+	return _hypercall2(int, stack_switch, ss, esp);
+}
+
+static inline int
+HYPERVISOR_set_callbacks(
+	unsigned long event_selector, unsigned long event_address,
+	unsigned long failsafe_selector, unsigned long failsafe_address)
+{
+	return _hypercall4(int, set_callbacks,
+			   event_selector, event_address,
+			   failsafe_selector, failsafe_address);
+}
+
+static inline int
+HYPERVISOR_fpu_taskswitch(
+	int set)
+{
+	return _hypercall1(int, fpu_taskswitch, set);
+}
+
+static inline int
+HYPERVISOR_sched_op(
+	int cmd, unsigned long arg)
+{
+	return _hypercall2(int, sched_op, cmd, arg);
+}
+
+static inline long
+HYPERVISOR_set_timer_op(
+	u64 timeout)
+{
+	unsigned long timeout_hi = (unsigned long)(timeout>>32);
+	unsigned long timeout_lo = (unsigned long)timeout;
+	return _hypercall2(long, set_timer_op, timeout_lo, timeout_hi);
+}
+
+static inline int
+HYPERVISOR_set_debugreg(
+	int reg, unsigned long value)
+{
+	return _hypercall2(int, set_debugreg, reg, value);
+}
+
+static inline unsigned long
+HYPERVISOR_get_debugreg(
+	int reg)
+{
+	return _hypercall1(unsigned long, get_debugreg, reg);
+}
+
+static inline int
+HYPERVISOR_update_descriptor(
+	u64 ma, u64 desc)
+{
+	return _hypercall4(int, update_descriptor, ma, ma>>32, desc, desc>>32);
+}
+
+static inline int
+HYPERVISOR_memory_op(
+	unsigned int cmd, void *arg)
+{
+	return _hypercall2(int, memory_op, cmd, arg);
+}
+
+static inline int
+HYPERVISOR_multicall(
+	void *call_list, int nr_calls)
+{
+	return _hypercall2(int, multicall, call_list, nr_calls);
+}
+
+static inline int
+HYPERVISOR_update_va_mapping(
+	unsigned long va, pte_t new_val, unsigned long flags)
+{
+	unsigned long pte_hi = 0;
+#ifdef CONFIG_X86_PAE
+	pte_hi = new_val.pte_high;
+#endif
+	return _hypercall4(int, update_va_mapping, va,
+			   new_val.pte_low, pte_hi, flags);
+}
+
+static inline int
+HYPERVISOR_event_channel_op(
+	void *op)
+{
+	return _hypercall1(int, event_channel_op, op);
+}
+
+static inline int
+HYPERVISOR_xen_version(
+	int cmd, void *arg)
+{
+	return _hypercall2(int, xen_version, cmd, arg);
+}
+
+static inline int
+HYPERVISOR_console_io(
+	int cmd, int count, char *str)
+{
+	return _hypercall3(int, console_io, cmd, count, str);
+}
+
+static inline int
+HYPERVISOR_physdev_op(
+	void *physdev_op)
+{
+	return _hypercall1(int, physdev_op, physdev_op);
+}
+
+static inline int
+HYPERVISOR_grant_table_op(
+	unsigned int cmd, void *uop, unsigned int count)
+{
+	return _hypercall3(int, grant_table_op, cmd, uop, count);
+}
+
+static inline int
+HYPERVISOR_update_va_mapping_otherdomain(
+	unsigned long va, pte_t new_val, unsigned long flags, domid_t domid)
+{
+	unsigned long pte_hi = 0;
+#ifdef CONFIG_X86_PAE
+	pte_hi = new_val.pte_high;
+#endif
+	return _hypercall5(int, update_va_mapping_otherdomain, va,
+			   new_val.pte_low, pte_hi, flags, domid);
+}
+
+static inline int
+HYPERVISOR_vm_assist(
+	unsigned int cmd, unsigned int type)
+{
+	return _hypercall2(int, vm_assist, cmd, type);
+}
+
+static inline int
+HYPERVISOR_vcpu_op(
+	int cmd, int vcpuid, void *extra_args)
+{
+	return _hypercall3(int, vcpu_op, cmd, vcpuid, extra_args);
+}
+
+static inline int
+HYPERVISOR_suspend(
+	unsigned long srec)
+{
+	return _hypercall3(int, sched_op, SCHEDOP_shutdown,
+			   SHUTDOWN_suspend, srec);
+}
+
+static inline int
+HYPERVISOR_nmi_op(
+	unsigned long op,
+	unsigned long arg)
+{
+	return _hypercall2(int, nmi_op, op, arg);
+}
+
+#endif /* __HYPERCALL_H__ */
include/asm-i386/hypervisor.h new file +67 -0

67 lines not on this page · raw message

--

#5 lore raw

Re: [RFC PATCH 04/35] Hypervisor interface header files.

Quoted 15 lines · Chris Wright

plain text document attachment (03-i386-hypervisor-interface) Define macros and inline functions for doing hypercalls into the hypervisor.

---
 include/asm-i386/hypercall.h  |  306 ++++++++++++++++++++++++++++++++++++++++++
 include/asm-i386/hypervisor.h |   67 +++++++++
 2 files changed, 373 insertions(+)

include/asm-i386/hypercall.h line 10

Quoted 10 lines
include/asm-i386/hypercall.h new file +10 -0
@@ -0,0 +1,306 @@
+/******************************************************************************
+ * hypercall.h
+ *
+ * Linux-specific hypervisor handling.
+ *
+ * Copyright (c) 2002-2004, K A Fraser
+ *
+ * This file may be distributed separately from the Linux kernel, or
+ * incorporated into other software packages, subject to the following license:
+ *

and what, if any, is the license when distributed with the kernel, as you propose? Right now there doesn't seem to be any at all, and thus it would be undistributable.

#6 lore raw

Re: [RFC PATCH 04/35] Hypervisor interface header files.

Quoted 8 lines · Arjan van de Ven
Quoted 4 lines

+ * This file may be distributed separately from the Linux kernel, or
+ * incorporated into other software packages, subject to the following license:
+ *

and what, if any, is the license when distributed with the kernel, as you propose? Right now there doesn't seem to be any at all, and thus it would be undistributable.

I thought GPLv2 would be implicit. I'll add the short GPL stanza to each of the offending source files.

Thanks for your other excellent comments by the way.

-- Keir

#7 lore raw

Re: [RFC PATCH 04/35] Hypervisor interface header files.

Quoted 7 lines · Keir Fraser
Quoted 5 lines · Arjan van de Ven
Quoted 4 lines

+ * This file may be distributed separately from the Linux kernel, or
+ * incorporated into other software packages, subject to the following license:
+ *

Quoted 5 lines
Quoted 3 lines

and what, if any, is the license when distributed with the kernel, as you propose? Right now there doesn't seem to be any at all, and thus it would be undistributable.

I thought GPLv2 would be implicit.

implicit license == bad :)

#8 lore raw

RE: [RFC PATCH 04/35] Hypervisor interface header files.

In reply to Keir Fraser
Quoted 8 lines · Arjan van de Ven
Quoted 5 lines
Quoted 5 lines

+ * This file may be distributed separately from the Linux kernel, or
+ * incorporated into other software packages, subject to the following license:
+ *

Quoted 6 lines
Quoted 3 lines

and what, if any, is the license when distributed with the kernel, as you propose? Right now there doesn't seem to be any at all, and thus it would be undistributable.

I thought GPLv2 would be implicit. I'll add the short GPL stanza to each of the offending source files.

It seems rather illogical to me to add a GPL stanza. The GPL adds new
rights and imposes requirements on you only if you could get those rights no other way. Since there is another way, the alternative license, the GPL requirements would never kick in. Although, as far as I can tell, it doesn't change or harm anything.

To be precise, it's like saying "you may use this any way you please, or at
your option, you may use it subject to certain restrictions".

It would make a difference, however, if your notice was excisable, allowing
someone to change the license to GPL only. However, your notice is not excisable, so the license (for this file and any future modifications to it) will perpetually be less restrictive than the GPL and the GPL requirements can never kick in.

The license is clearly GPL-compatible so I see no need to change it.

The only thing I can think of is that someone is concerned about the
hypothetical case where someone modifies this file, compiles it, distributes the result, and refuses to distribute the source code to this module under the reasoning that he opted for the license that had no such requirement. But that fails the giggle test, since he needs the GPL to distribute the GPL'd portions of the binary and the GPL requires all source to be distributed. (If I could do that, I could *certainly* add my own code to the kernel and refuse to deliver the source to it, since my own code is under any license I want it to be under. And I can't do that.)

DS

#9 lore raw

Re: [RFC PATCH 04/35] Hypervisor interface header files.

Quoted 8 lines · David Schwartz
Quoted 2 lines

I thought GPLv2 would be implicit. I'll add the short GPL stanza to each of the offending source files.

It seems rather illogical to me to add a GPL stanza. The GPL adds new
rights and imposes requirements on you only if you could get those rights no other way. Since there is another way, the alternative license, the GPL requirements would never kick in. Although, as far as I can tell, it doesn't change or harm anything.

Yes, that's the same logic I applied. I think it is redundant, but I think it makes sense to add a sentence to the effect that the file is GPL if you want it to be, just to avoid any fears or complaints, and to show that we really aren't trying to do anything fishy. And IANAL so I'll err on the side of caution and redundancy. :-)

-- Keir

#10 lore raw

RFC PATCH 5/35 Add sync bitops

In reply to Chris Wright

Add "always lock'd" implementations of set_bit, clear_bit and change_bit and the corresponding test_and_ functions. Also add "always lock'd" implementation of cmpxchg. These give guaranteed strong synchronisation and are required for non-SMP kernels running on an SMP hypervisor.

---
 include/asm-i386/bitops-fns.h |   97 +++++++++++++++++
 include/asm-i386/bitops.h     |  230 ++++++++++++++++--------------------------
 include/asm-i386/system.h     |   33 ++++++
 3 files changed, 222 insertions(+), 138 deletions(-)
include/asm-i386/bitops.h +92 -138
@@ -24,6 +24,34 @@
 
 #define ADDR (*(volatile long *) addr)
 
+#define __LOCK_PREFIX LOCK_PREFIX
+#define __NAME(name) name
+#define __BARRIER : "memory"
+#include "bitops-fns.h"
+#undef __LOCK_PREFIX
+#undef __NAME
+#undef __BARRIER
+
+#define __LOCK_PREFIX
+#define __NAME(name) __##name
+#define __BARRIER
+#include "bitops-fns.h"
+#undef __LOCK_PREFIX
+#undef __NAME
+#undef __BARRIER
+
+#define __LOCK_PREFIX "lock ; "
+#define __NAME(name) synch_##name
+#define __BARRIER : "memory"
+#include "bitops-fns.h"
+#undef __LOCK_PREFIX
+#undef __NAME
+#undef __BARRIER
+
+#define smp_mb__before_clear_bit()	barrier()
+#define smp_mb__after_clear_bit()	barrier()
+
+#if 0 /* Fool kernel-doc */
 /**
  * set_bit - Atomically set a bit in memory
  * @nr: the bit to set
@@ -39,30 +67,7 @@
  * Note that @nr may be almost arbitrarily large; this function is not
  * restricted to acting on a single-word quantity.
  */
-static inline void set_bit(int nr, volatile unsigned long * addr)
-{
-	__asm__ __volatile__( LOCK_PREFIX
-		"btsl %1,%0"
-		:"+m" (ADDR)
-		:"Ir" (nr));
-}
-
-/**
- * __set_bit - Set a bit in memory
- * @nr: the bit to set
- * @addr: the address to start counting from
- *
- * Unlike set_bit(), this function is non-atomic and may be reordered.
- * If it's called on the same region of memory simultaneously, the effect
- * may be that only one operation succeeds.
- */
-static inline void __set_bit(int nr, volatile unsigned long * addr)
-{
-	__asm__(
-		"btsl %1,%0"
-		:"+m" (ADDR)
-		:"Ir" (nr));
-}
+static inline void set_bit(int nr, volatile unsigned long *addr);
 
 /**
  * clear_bit - Clears a bit in memory
@@ -74,40 +79,7 @@ static inline void __set_bit(int nr, vol
  * you should call smp_mb__before_clear_bit() and/or smp_mb__after_clear_bit()
  * in order to ensure changes are visible on other processors.
  */
-static inline void clear_bit(int nr, volatile unsigned long * addr)
-{
-	__asm__ __volatile__( LOCK_PREFIX
-		"btrl %1,%0"
-		:"+m" (ADDR)
-		:"Ir" (nr));
-}
-
-static inline void __clear_bit(int nr, volatile unsigned long * addr)
-{
-	__asm__ __volatile__(
-		"btrl %1,%0"
-		:"+m" (ADDR)
-		:"Ir" (nr));
-}
-#define smp_mb__before_clear_bit()	barrier()
-#define smp_mb__after_clear_bit()	barrier()
-
-/**
- * __change_bit - Toggle a bit in memory
- * @nr: the bit to change
- * @addr: the address to start counting from
- *
- * Unlike change_bit(), this function is non-atomic and may be reordered.
- * If it's called on the same region of memory simultaneously, the effect
- * may be that only one operation succeeds.
- */
-static inline void __change_bit(int nr, volatile unsigned long * addr)
-{
-	__asm__ __volatile__(
-		"btcl %1,%0"
-		:"+m" (ADDR)
-		:"Ir" (nr));
-}
+static inline void clear_bit(int nr, volatile unsigned long *addr);
 
 /**
  * change_bit - Toggle a bit in memory
@@ -119,13 +91,7 @@ static inline void __change_bit(int nr, 
  * Note that @nr may be almost arbitrarily large; this function is not
  * restricted to acting on a single-word quantity.
  */
-static inline void change_bit(int nr, volatile unsigned long * addr)
-{
-	__asm__ __volatile__( LOCK_PREFIX
-		"btcl %1,%0"
-		:"+m" (ADDR)
-		:"Ir" (nr));
-}
+static inline void change_bit(int nr, volatile unsigned long *addr);
 
 /**
  * test_and_set_bit - Set a bit and return its old value
@@ -136,56 +102,72 @@ static inline void change_bit(int nr, vo
  * It may be reordered on other architectures than x86.
  * It also implies a memory barrier.
  */
-static inline int test_and_set_bit(int nr, volatile unsigned long * addr)
-{
-	int oldbit;
-
-	__asm__ __volatile__( LOCK_PREFIX
-		"btsl %2,%1\n\tsbbl %0,%0"
-		:"=r" (oldbit),"+m" (ADDR)
-		:"Ir" (nr) : "memory");
-	return oldbit;
-}
+static inline int test_and_set_bit(int nr, volatile unsigned long *addr);
 
 /**
- * __test_and_set_bit - Set a bit and return its old value
- * @nr: Bit to set
+ * test_and_clear_bit - Clear a bit and return its old value
+ * @nr: Bit to clear
  * @addr: Address to count from
  *
- * This operation is non-atomic and can be reordered.  
- * If two examples of this operation race, one can appear to succeed
- * but actually fail.  You must protect multiple accesses with a lock.
+ * This operation is atomic and cannot be reordered.
+ * It can be reordered on other architectures other than x86.
+ * It also implies a memory barrier.
  */
-static inline int __test_and_set_bit(int nr, volatile unsigned long * addr)
-{
-	int oldbit;
-
-	__asm__(
-		"btsl %2,%1\n\tsbbl %0,%0"
-		:"=r" (oldbit),"+m" (ADDR)
-		:"Ir" (nr));
-	return oldbit;
-}
+static inline int test_and_clear_bit(int nr, volatile unsigned long *addr);
 
 /**
- * test_and_clear_bit - Clear a bit and return its old value
- * @nr: Bit to clear
+ * test_and_change_bit - Change a bit and return its old value
+ * @nr: Bit to change
  * @addr: Address to count from
  *
- * This operation is atomic and cannot be reordered.
- * It can be reorderdered on other architectures other than x86.
+ * This operation is atomic and cannot be reordered.  
  * It also implies a memory barrier.
  */
-static inline int test_and_clear_bit(int nr, volatile unsigned long * addr)
-{
-	int oldbit;
+static inline int test_and_change_bit(int nr, volatile unsigned long *addr);
 
-	__asm__ __volatile__( LOCK_PREFIX
-		"btrl %2,%1\n\tsbbl %0,%0"
-		:"=r" (oldbit),"+m" (ADDR)
-		:"Ir" (nr) : "memory");
-	return oldbit;
-}
+/**
+ * __set_bit - Set a bit in memory
+ * @nr: the bit to set
+ * @addr: the address to start counting from
+ *
+ * Unlike set_bit(), this function is non-atomic and may be reordered.
+ * If it's called on the same region of memory simultaneously, the effect
+ * may be that only one operation succeeds.
+ */
+static inline void __set_bit(int nr, volatile unsigned long * addr);
+
+/**
+ * __clear_bit - Clear a bit in memory
+ * @nr: the bit to clear
+ * @addr: the address to start counting from
+ *
+ * Unlike clear_bit(), this function is non-atomic and may be reordered.
+ * If it's called on the same region of memory simultaneously, the effect
+ * may be that only one operation succeeds.
+ */
+static inline void __clear_bit(int nr, volatile unsigned long * addr);
+
+/**
+ * __change_bit - Toggle a bit in memory
+ * @nr: the bit to change
+ * @addr: the address to start counting from
+ *
+ * Unlike change_bit(), this function is non-atomic and may be reordered.
+ * If it's called on the same region of memory simultaneously, the effect
+ * may be that only one operation succeeds.
+ */
+static inline void __change_bit(int nr, volatile unsigned long * addr);
+
+/**
+ * __test_and_set_bit - Set a bit and return its old value
+ * @nr: Bit to set
+ * @addr: Address to count from
+ *
+ * This operation is non-atomic and can be reordered.  
+ * If two examples of this operation race, one can appear to succeed
+ * but actually fail.  You must protect multiple accesses with a lock.
+ */
+static inline int __test_and_set_bit(int nr, volatile unsigned long * addr);
 
 /**
  * __test_and_clear_bit - Clear a bit and return its old value
@@ -196,47 +178,19 @@ static inline int test_and_clear_bit(int
  * If two examples of this operation race, one can appear to succeed
  * but actually fail.  You must protect multiple accesses with a lock.
  */
-static inline int __test_and_clear_bit(int nr, volatile unsigned long *addr)
-{
-	int oldbit;
-
-	__asm__(
-		"btrl %2,%1\n\tsbbl %0,%0"
-		:"=r" (oldbit),"+m" (ADDR)
-		:"Ir" (nr));
-	return oldbit;
-}
-
-/* WARNING: non atomic and it can be reordered! */
-static inline int __test_and_change_bit(int nr, volatile unsigned long *addr)
-{
-	int oldbit;
-
-	__asm__ __volatile__(
-		"btcl %2,%1\n\tsbbl %0,%0"
-		:"=r" (oldbit),"+m" (ADDR)
-		:"Ir" (nr) : "memory");
-	return oldbit;
-}
+static inline int __test_and_clear_bit(int nr, volatile unsigned long *addr);
 
 /**
- * test_and_change_bit - Change a bit and return its old value
+ * __test_and_clear_bit - Change a bit and return its old value
  * @nr: Bit to change
  * @addr: Address to count from
  *
- * This operation is atomic and cannot be reordered.  
- * It also implies a memory barrier.
+ * This operation is non-atomic and can be reordered.  
+ * If two examples of this operation race, one can appear to succeed
+ * but actually fail.  You must protect multiple accesses with a lock.
  */
-static inline int test_and_change_bit(int nr, volatile unsigned long* addr)
-{
-	int oldbit;
-
-	__asm__ __volatile__( LOCK_PREFIX
-		"btcl %2,%1\n\tsbbl %0,%0"
-		:"=r" (oldbit),"+m" (ADDR)
-		:"Ir" (nr) : "memory");
-	return oldbit;
-}
+static inline int __test_and_change_bit(int nr, volatile unsigned long *addr);
+#endif
 
 #if 0 /* Fool kernel-doc since it doesn't do macros yet */
 /**
include/asm-i386/system.h +33 -0
@@ -263,6 +263,9 @@ static inline unsigned long __xchg(unsig
 #define cmpxchg(ptr,o,n)\
 	((__typeof__(*(ptr)))__cmpxchg((ptr),(unsigned long)(o),\
 					(unsigned long)(n),sizeof(*(ptr))))
+#define synch_cmpxchg(ptr,o,n)\
+	((__typeof__(*(ptr)))__synch_cmpxchg((ptr),(unsigned long)(o),\
+					(unsigned long)(n),sizeof(*(ptr))))
 #endif
 
 static inline unsigned long __cmpxchg(volatile void *ptr, unsigned long old,
@@ -292,6 +295,36 @@ static inline unsigned long __cmpxchg(vo
 	return old;
 }
 
+#define __LOCK_PREFIX "lock ; "
+static inline unsigned long __synch_cmpxchg(volatile void *ptr,
+					    unsigned long old,
+					    unsigned long new, int size)
+{
+	unsigned long prev;
+	switch (size) {
+	case 1:
+		__asm__ __volatile__(__LOCK_PREFIX "cmpxchgb %b1,%2"
+				     : "=a"(prev)
+				     : "q"(new), "m"(*__xg(ptr)), "0"(old)
+				     : "memory");
+		return prev;
+	case 2:
+		__asm__ __volatile__(__LOCK_PREFIX "cmpxchgw %w1,%2"
+				     : "=a"(prev)
+				     : "r"(new), "m"(*__xg(ptr)), "0"(old)
+				     : "memory");
+		return prev;
+	case 4:
+		__asm__ __volatile__(__LOCK_PREFIX "cmpxchgl %1,%2"
+				     : "=a"(prev)
+				     : "r"(new), "m"(*__xg(ptr)), "0"(old)
+				     : "memory");
+		return prev;
+	}
+	return old;
+}
+#undef __LOCK_PREFIX
+
 #ifndef CONFIG_X86_CMPXCHG
 /*
  * Building a kernel capable running on 80386. It may be necessary to
include/asm-i386/bitops-fns.h new file +97 -0

97 lines not on this page · raw message

--

#11 lore raw

RFC PATCH 6/35 Add vmlinuz build target.

In reply to Chris Wright

The vmlinuz image is a stripped and compressed kernel image, it is smaller than the vmlinux image and the Xen domain builder supports loading compressed images directly.

---
 arch/i386/Makefile      |    5 +++--
 arch/i386/boot/Makefile |   10 +++++++++-
 2 files changed, 12 insertions(+), 3 deletions(-)
arch/i386/Makefile +3 -2
@@ -105,15 +105,16 @@ CPPFLAGS += $(mflags-y)
 boot := arch/i386/boot
 
 .PHONY: zImage bzImage compressed zlilo bzlilo \
-	zdisk bzdisk fdimage fdimage144 fdimage288 install
+	zdisk bzdisk fdimage fdimage144 fdimage288 install vmlinuz
 
 all: bzImage
 
 # KBUILD_IMAGE specify target image being built
                     KBUILD_IMAGE := $(boot)/bzImage
 zImage zlilo zdisk: KBUILD_IMAGE := arch/i386/boot/zImage
+vmlinuz:            KBUILD_IMAGE := $(boot)/vmlinuz
 
-zImage bzImage: vmlinux
+zImage bzImage vmlinuz: vmlinux
 	$(Q)$(MAKE) $(build)=$(boot) $(KBUILD_IMAGE)
 
 compressed: zImage
arch/i386/boot/Makefile +9 -1
@@ -26,7 +26,7 @@ SVGA_MODE := -DSVGA_MODE=NORMAL_VGA
 #RAMDISK := -DRAMDISK=512
 
 targets		:= vmlinux.bin bootsect bootsect.o \
-		   setup setup.o zImage bzImage
+		   setup setup.o zImage bzImage vmlinuz
 subdir- 	:= compressed
 
 hostprogs-y	:= tools/build
@@ -100,5 +100,13 @@ zlilo: $(BOOTIMAGE)
 	cp System.map $(INSTALL_PATH)/
 	if [ -x /sbin/lilo ]; then /sbin/lilo; else /etc/lilo/install; fi
 
+$(obj)/vmlinuz: $(obj)/vmlinux-stripped FORCE
+	$(call if_changed,gzip)
+	@echo 'Kernel: $@ is ready' ' (#'`cat .version`')'
+
+$(obj)/vmlinux-stripped: OBJCOPYFLAGS := -g --strip-unneeded
+$(obj)/vmlinux-stripped: vmlinux FORCE
+	$(call if_changed,objcopy)
+
 install:
 	sh $(srctree)/$(src)/install.sh $(KERNELRELEASE) $(BOOTIMAGE) System.map "$(INSTALL_PATH)"

--

#12 lore raw

RFC PATCH 7/35 Make LOAD_OFFSET defined by subarch

In reply to Chris Wright

Change LOAD_OFFSET so that the kernel has virtual addresses in the elf header fields.

Unlike bare metal kernels, Xen kernels start with virtual address management turned on and thus the addresses to load to should be virtual addresses.

---
 arch/i386/kernel/vmlinux.lds.S                   |    2 +-
 include/asm-i386/mach-default/mach_vmlinux.lds.h |    6 ++++++
 include/asm-i386/mach-xen/mach_vmlinux.lds.h     |    6 ++++++
 3 files changed, 13 insertions(+), 1 deletion(-)
arch/i386/kernel/vmlinux.lds.S +1 -1
@@ -2,7 +2,7 @@
  * Written by Martin Mares <[email protected]>;
  */
 
-#define LOAD_OFFSET __PAGE_OFFSET
+#include "mach_vmlinux.lds.h"
 
 #include <asm-generic/vmlinux.lds.h>
 #include <asm/thread_info.h>
include/asm-i386/mach-default/mach_vmlinux.lds.h new file +6 -0
@@ -0,0 +1,6 @@
+#ifndef __ASM_MACH_VMLINUX_LDS_H
+#define __ASM_MACH_VMLINUX_LDS_H
+
+#define LOAD_OFFSET __PAGE_OFFSET
+
+#endif /* __ASM_MACH_VMLINUX_LDS_H */
include/asm-i386/mach-xen/mach_vmlinux.lds.h new file +6 -0
@@ -0,0 +1,6 @@
+#ifndef __ASM_MACH_VMLINUX_LDS_H
+#define __ASM_MACH_VMLINUX_LDS_H
+
+#define LOAD_OFFSET 0
+
+#endif /* __ASM_MACH_VMLINUX_LDS_H */

--

#13 lore raw

Re: [RFC PATCH 07/35] Make LOAD_OFFSET defined by subarch

Quoted 6 lines · Chris Wright

Change LOAD_OFFSET so that the kernel has virtual addresses in the elf header fields.

Unlike bare metal kernels, Xen kernels start with virtual address management turned on and thus the addresses to load to should be virtual addresses.

Rather than changing LOAD_OFFSET in Linux, why not leave this alone and change the Xen domain builder to properly interpret the ELF program header fields?

i.e. with this change, we'd have

p_paddr = __PAGE_OFFSET + segment_offset
p_vaddr = __PAGE_OFFSET + segment_offset
VIRT_BASE = __PAGE_OFFSET

where, the VA mapping p_paddr -> (p_paddr-VIRT_BASE) is established by the domain builder.

Instead, why not drop this patch, and the VIRT_BASE portion of the __xen_guest section, and instead change Xen's domain builder to treat p_paddr and p_vaddr in a more standard way? Since Xen starts the domain with virtual address management enabled, it makes sense for it to use p_vaddr to determine the virtual address to load the kernel to. Then, p_paddr could be used to determine which pseudo-physical pages back that virtual address range.

i.e. use, just like vanilla linux:

p_paddr = segment_offset
p_vaddr = __PAGE_OFFSET + segment_offset

so these two fields directly indicate the same mapping as before, but now in terms of p_vaddr -> p_paddr, which makes sense, and no need for the extra VIRT_BASE attribute in the __xen_guest section.

Dan

#14 lore raw

Re: [RFC PATCH 07/35] Make LOAD_OFFSET defined by subarch

Hi,

Quoted 3 lines

Rather than changing LOAD_OFFSET in Linux, why not leave this alone and change the Xen domain builder to properly interpret the ELF program header fields?

I've a patch in the queue which does exactly that ;) Planned to submit just after xen 3.0.2 release ...

cheers,

Gerd

Signature · 3 lines
Gerd 'just married' Hoffmann <[email protected]>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg
#15 lore raw

Re: [RFC PATCH 07/35] Make LOAD_OFFSET defined by subarch

Quoted 7 lines · Gerd Hoffmann
Quoted 4 lines

Rather than changing LOAD_OFFSET in Linux, why not leave this alone and change the Xen domain builder to properly interpret the ELF program header fields?

I've a patch in the queue which does exactly that ;) Planned to submit just after xen 3.0.2 release ...

(CC list reduced)

This makes me a little nervous, since the ELF loading code is working fine for PPC at the moment. What exactly are you planning on changing?

Signature · 2 lines
Hollis Blanchard
IBM Linux Technology Center
#16 lore raw

Re: [RFC PATCH 07/35] Make LOAD_OFFSET defined by subarch

Hi,

Quoted 2 lines

This makes me a little nervous, since the ELF loading code is working fine for PPC at the moment. What exactly are you planning on changing?

Basically what the suggestion outlined: use physical address + VIRT_BASE instead of placing virtual addresses into the physical address fields. Some discussions on that went over the list some weeks ago. Patch attached for reference.

There is another change in the patch: introduce a xc_unmap_foreign_range() as symmetric call for xc_map_foreign_range() and make domain builders use it instead of calling munmap directly. That is one of the changes I need to make the domain builders usable for kexec, more will follow ;)

I'll go split stuff into individual patches for submission (planned for post-3.0.2 development cycle).

cheers,

Gerd

Signature · 3 lines
Gerd 'just married' Hoffmann <[email protected]>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg
linux-2.6-xen-sparse/include/asm-i386/mach-xen/asm/page.h +0 -4
diff -r 25e1c3b1a9f2 linux-2.6-xen-sparse/include/asm-i386/mach-xen/asm/page.h
@@ -289,10 +289,6 @@
 #endif
 #define __KERNEL_START		(__PAGE_OFFSET + __PHYSICAL_START)
 
-#undef LOAD_OFFSET
-#define LOAD_OFFSET		0
-
-
 #define PAGE_OFFSET		((unsigned long)__PAGE_OFFSET)
 #define VMALLOC_RESERVE		((unsigned long)__VMALLOC_RESERVE)
 #define MAXMEM			(__FIXADDR_TOP-__PAGE_OFFSET-__VMALLOC_RESERVE)
linux-2.6-xen-sparse/include/asm-x86_64/mach-xen/asm/page.h +0 -3
diff -r 25e1c3b1a9f2 linux-2.6-xen-sparse/include/asm-x86_64/mach-xen/asm/page.h
@@ -260,9 +260,6 @@
 #define __PAGE_OFFSET           0xffff880000000000
 #endif /* !__ASSEMBLY__ */
 
-#undef LOAD_OFFSET
-#define LOAD_OFFSET		0
-
 /* to align the pointer to the (next) page boundary */
 #define PAGE_ALIGN(addr)	(((addr)+PAGE_SIZE-1)&PAGE_MASK)
 
tools/libxc/xc_core.c +1 -1
diff -r 25e1c3b1a9f2 tools/libxc/xc_core.c
@@ -20,7 +20,7 @@
     if ( vaddr == NULL )
         return -1;
     memcpy(dst_page, vaddr, PAGE_SIZE);
-    munmap(vaddr, PAGE_SIZE);
+    xc_unmap_foreign_range(xc_handle, domid, vaddr, PAGE_SIZE);
     return 0;
 }
 
tools/libxc/xc_linux_build.c +24 -24

118 lines not on this page · raw message

tools/libxc/xc_load_aout9.c +1 -1
diff -r 25e1c3b1a9f2 tools/libxc/xc_load_aout9.c
@@ -126,7 +126,7 @@
         pg = xc_map_foreign_range(xch, dom, PAGE_SIZE, PROT_WRITE, 
                                   parray[off>>PAGE_SHIFT]);
         memcpy(pg + pgoff, buf, chunksz);
-        munmap(pg, PAGE_SIZE);
+        xc_unmap_foreign_range(xch, dom, pg, PAGE_SIZE);
 
         off += chunksz;
         buf += chunksz;
tools/libxc/xc_load_bin.c +2 -2
diff -r 25e1c3b1a9f2 tools/libxc/xc_load_bin.c
@@ -273,7 +273,7 @@
         if ( chunksz > PAGE_SIZE )
             chunksz = PAGE_SIZE;
         memcpy(va, image + done, chunksz);
-        munmap(va, PAGE_SIZE);
+        xc_unmap_foreign_range(xch, dom, va, PAGE_SIZE);
     }
 
     if ( 0 != image_info->bss_end_addr &&
@@ -289,7 +289,7 @@
         if ( chunksz > (PAGE_SIZE - (done & (PAGE_SIZE-1))) )
             chunksz = PAGE_SIZE - (done & (PAGE_SIZE-1));
         memset(va + (done & (PAGE_SIZE-1)), 0, chunksz);
-        munmap(va, PAGE_SIZE);
+        xc_unmap_foreign_range(xch, dom, va, PAGE_SIZE);
     }
 
     return 0;
tools/libxc/xc_load_elf.c +39 -9

88 lines not on this page · raw message

tools/libxc/xc_pagetab.c +4 -4
diff -r 25e1c3b1a9f2 tools/libxc/xc_pagetab.c
@@ -170,13 +170,13 @@
     }
 
  out_unmap_pt:
-    munmap(pt, PAGE_SIZE);
+    xc_unmap_foreign_range(xc_handle, dom, pt, PAGE_SIZE);
  out_unmap_pd:
-    munmap(pd, PAGE_SIZE);
+    xc_unmap_foreign_range(xc_handle, dom, pd, PAGE_SIZE);
  out_unmap_pdp:
-    munmap(pdppage, PAGE_SIZE);
+    xc_unmap_foreign_range(xc_handle, dom, pdppage, PAGE_SIZE);
  out_unmap_pml:
-    munmap(pml, PAGE_SIZE);
+    xc_unmap_foreign_range(xc_handle, dom, pml, PAGE_SIZE);
  out:
     return mfn;
 }
tools/libxc/xc_private.c +9 -3
diff -r 25e1c3b1a9f2 tools/libxc/xc_private.c
@@ -60,6 +60,12 @@
         return NULL;
     }
     return addr;
+}
+
+int xc_unmap_foreign_range(int xc_handle, uint32_t dom,
+                           void *ptr, int size)
+{
+    return munmap(ptr, size);
 }
 
 /*******************/
@@ -371,7 +377,7 @@
     if ( vaddr == NULL )
         return -1;
     memcpy(vaddr, src_page, PAGE_SIZE);
-    munmap(vaddr, PAGE_SIZE);
+    xc_unmap_foreign_range(xc_handle, domid, vaddr, PAGE_SIZE);
     return 0;
 }
 
@@ -384,7 +390,7 @@
     if ( vaddr == NULL )
         return -1;
     memset(vaddr, 0, PAGE_SIZE);
-    munmap(vaddr, PAGE_SIZE);
+    xc_unmap_foreign_range(xc_handle, domid, vaddr, PAGE_SIZE);
     return 0;
 }
 
@@ -426,7 +432,7 @@
         if ( chunksz > (PAGE_SIZE - (pa & (PAGE_SIZE-1))) )
             chunksz = PAGE_SIZE - (pa & (PAGE_SIZE-1));
         memcpy(va + (pa & (PAGE_SIZE-1)), src + done, chunksz);
-        munmap(va, PAGE_SIZE);
+        xc_unmap_foreign_range(xch, dom, va, PAGE_SIZE);
     }
 }
 
tools/libxc/xc_ptrace.c +4 -4
diff -r 25e1c3b1a9f2 tools/libxc/xc_ptrace.c
@@ -188,7 +188,7 @@
 
     p = l1[l1_table_offset_pae(va)] >> PAGE_SHIFT;
     if ( v != NULL )
-        munmap(v, PAGE_SIZE);
+        xc_unmap_foreign_range(xc_handle, current_domid, v, PAGE_SIZE);
     v = xc_map_foreign_range(xc_handle, current_domid, PAGE_SIZE, perm, p);
     if ( v == NULL )
         return NULL;
@@ -255,7 +255,7 @@
     {
         cr3_phys[cpu] = ctxt[cpu].ctrlreg[3];
         if ( cr3_virt[cpu] )
-            munmap(cr3_virt[cpu], PAGE_SIZE);
+            xc_unmap_foreign_range(xc_handle, current_domid, cr3_virt[cpu], PAGE_SIZE);
         cr3_virt[cpu] = xc_map_foreign_range(
             xc_handle, current_domid, PAGE_SIZE, PROT_READ,
             cr3_phys[cpu] >> PAGE_SHIFT);
@@ -270,7 +270,7 @@
     {
         pde_phys[cpu] = pde;
         if ( pde_virt[cpu] )
-            munmap(pde_virt[cpu], PAGE_SIZE);
+            xc_unmap_foreign_range(xc_handle, current_domid, pde_virt[cpu], PAGE_SIZE);
         pde_virt[cpu] = xc_map_foreign_range(
             xc_handle, current_domid, PAGE_SIZE, PROT_READ,
             pde_phys[cpu] >> PAGE_SHIFT);
@@ -285,7 +285,7 @@
     {
         page_phys[cpu] = page;
         if ( page_virt[cpu] )
-            munmap(page_virt[cpu], PAGE_SIZE);
+            xc_unmap_foreign_range(xc_handle, current_domid, page_virt[cpu], PAGE_SIZE);
         page_virt[cpu] = xc_map_foreign_range(
             xc_handle, current_domid, PAGE_SIZE, perm,
             page_phys[cpu] >> PAGE_SHIFT);
tools/libxc/xenctrl.h +4 -2
diff -r 25e1c3b1a9f2 tools/libxc/xenctrl.h
@@ -458,8 +458,10 @@
  * @parm mfn the frame address to map.
  */
 void *xc_map_foreign_range(int xc_handle, uint32_t dom,
-                            int size, int prot,
-                            unsigned long mfn );
+			   int size, int prot,
+			   unsigned long mfn );
+int xc_unmap_foreign_range(int xc_handle, uint32_t dom,
+			   void *ptr, int size);
 
 void *xc_map_foreign_batch(int xc_handle, uint32_t dom, int prot,
                            unsigned long *arr, int num );
tools/libxc/xg_private.h +1 -0
diff -r 25e1c3b1a9f2 tools/libxc/xg_private.h
@@ -135,6 +135,7 @@
     unsigned long v_kernstart;
     unsigned long v_kernend;
     unsigned long v_kernentry;
+    unsigned long virt_base;
 
     unsigned int  load_symtab;
     unsigned int  pae_kernel;
xen/arch/x86/mm.c +1 -0
diff -r 25e1c3b1a9f2 xen/arch/x86/mm.c
@@ -1603,6 +1603,7 @@
         if ( unlikely(!okay) )
         {
             /* Switch to idle pagetable: this VCPU has no active p.t. now. */
+            MEM_LOG("%s: slow path via idle table", __FUNCTION__);
             old_base_mfn = pagetable_get_pfn(v->arch.guest_table);
             v->arch.guest_table = mk_pagetable(0);
             update_pagetables(v);
xen/common/elf.c +44 -7

84 lines not on this page · raw message

xen/include/xen/sched.h +1 -0
diff -r 25e1c3b1a9f2 xen/include/xen/sched.h
@@ -168,6 +168,7 @@
     unsigned long v_kernstart;
     unsigned long v_kernend;
     unsigned long v_kernentry;
+    unsigned long virt_base;
     /* Initialised by loader: Private. */
     unsigned int  load_symtab;
     unsigned long symtab_addr;

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#17 lore raw

Re: [RFC PATCH 07/35] Make LOAD_OFFSET defined by subarch

Quoted 2 lines · Gerd Hoffmann
Quoted 2 lines

This makes me a little nervous, since the ELF loading code is working fine

for

tools/libxc/xc_load_elf.c line 150

Quoted 11 lines
Quoted 1 line

PPC at the moment. What exactly are you planning on changing?

Basically what the suggestion outlined: use physical address + VIRT_BASE instead of placing virtual addresses into the physical address fields. Some discussions on that went over the list some weeks ago. Patch attached for reference.

tools/libxc/xc_load_elf.c +4 -0
+         * bug comparibility alert: old linux kernels used to have
+         * virtual addresses in the paddr headers, whereas newer ones
+         * (since kexec merge, around 2.6.14) correctly use physical
+         * addresses.

("comparibility" -> "compatibility")

I can't speak for IA64, but PPC kernels still have virtual addresses in the paddr fields. Accordingly, I would reword it like this:
Some Linux kernels have virtual addresses in the paddr headers, and some correctly use physical addresses.

Please CC xen-ppc-devel (and probably xen-ia64-devel) when you submit the patch so we can make sure this logic works for PPC. Thanks!

Signature · 2 lines
Hollis Blanchard
IBM Linux Technology Center
#18 lore raw

Re: Re: [RFC PATCH 07/35] Make LOAD_OFFSET defined by subarch

Quoted 5 lines · Hollis Blanchard
Quoted 4 lines · Gerd Hoffmann

Basically what the suggestion outlined: use physical address + VIRT_BASE instead of placing virtual addresses into the physical address fields. Some discussions on that went over the list some weeks ago. Patch attached for reference.

and there was no conclusion.

Quoted 9 lines

tools/libxc/xc_load_elf.c line 150

Quoted 4 lines
tools/libxc/xc_load_elf.c +4 -0
+         * bug comparibility alert: old linux kernels used to have
+         * virtual addresses in the paddr headers, whereas newer ones
+         * (since kexec merge, around 2.6.14) correctly use physical
+         * addresses.

I can't speak for IA64, but PPC kernels still have virtual addresses in the paddr fields. Accordingly, I would reword it like this:
Some Linux kernels have virtual addresses in the paddr headers, and some correctly use physical addresses.

How is that correct? The ELF spec is quite vague on what is supposed to be in the paddr fields...

christian

#19 lore raw

Re: ELF headers

Quoted 8 lines · Christian Limpach
Quoted 7 lines · Hollis Blanchard

tools/libxc/xc_load_elf.c line 150

Quoted 4 lines · Gerd Hoffmann
tools/libxc/xc_load_elf.c +4 -0
+         * bug comparibility alert: old linux kernels used to have
+         * virtual addresses in the paddr headers, whereas newer ones
+         * (since kexec merge, around 2.6.14) correctly use physical
+         * addresses.

I can't speak for IA64, but PPC kernels still have virtual addresses in

the

Quoted 2 lines
Quoted 2 lines

paddr fields. Accordingly, I would reword it like this:
Some Linux kernels have virtual addresses in the paddr headers,

and some

Quoted 4 lines
Quoted 1 line

correctly use physical addresses.

How is that correct? The ELF spec is quite vague on what is supposed to be in the paddr fields...

It seems to me that 0xc000000000000000 is not supposed to be in a field that contains a "physical address". Is that in question?

Signature · 2 lines
Hollis Blanchard
IBM Linux Technology Center
#20 lore raw

RFC PATCH 8/35 Add Xen-specific memory management definitions

In reply to Chris Wright

Add extra memory management definitions used by Xen-specific code. These allow conversion between 'pseudophysical' memory addresses, which provide the illusion of a physically contiguous memory map, and underlying real machine addresses. This conversion is neceesary when interpreting and updating PTEs. Also support write protection of page mappings, which is needed to allow successful validation of page tables.

The current definitions are incomplete and only a stub implementation, allowing us to re-use existing code (drivers) which references these memory management code changes.

---
 include/asm-i386/hypervisor.h             |    3 
 include/asm-i386/mach-default/mach_page.h |    4 +
 include/asm-i386/mach-xen/mach_page.h     |   99 ++++++++++++++++++++++++++++++
 include/asm-i386/page.h                   |    2 
 4 files changed, 108 insertions(+)
include/asm-i386/hypervisor.h +3 -0
@@ -64,4 +64,7 @@ u64 jiffies_to_st(unsigned long jiffies)
 
 #define xen_init()	(0)
 
+#include <xen/interface/version.h>
+#include <xen/features.h>
+
 #endif /* __HYPERVISOR_H__ */
include/asm-i386/page.h +2 -0
@@ -82,6 +82,8 @@ typedef struct { unsigned long pgprot; }
 /* to align the pointer to the (next) page boundary */
 #define PAGE_ALIGN(addr)	(((addr)+PAGE_SIZE-1)&PAGE_MASK)
 
+#include <mach_page.h>
+
 /*
  * This handles the memory map.. We could make this a config
  * option, but too many people screw it up, and too few need
include/asm-i386/mach-default/mach_page.h new file +4 -0
@@ -0,0 +1,4 @@
+#ifndef __ASM_MACH_PAGE_H
+#define __ASM_MACH_PAGE_H
+
+#endif /* __ASM_MACH_PAGE_H */
include/asm-i386/mach-xen/mach_page.h new file +99 -0

99 lines not on this page · raw message

--

#21 lore raw

RFC PATCH 9/35 Change __FIXADDR_TOP to leave room for the hypervisor.

In reply to Chris Wright

Move the definition of __FIXADDR_TOP into a subarch include file so that it can be overridden for subarch xen -- the hypervisor needs about 64MB at the top of the address space.

---
 include/asm-i386/fixmap.h                   |    8 +-------
 include/asm-i386/mach-default/mach_fixmap.h |   11 +++++++++++
 include/asm-i386/mach-xen/mach_fixmap.h     |   11 +++++++++++
 3 files changed, 23 insertions(+), 7 deletions(-)
include/asm-i386/fixmap.h +1 -7
@@ -14,13 +14,7 @@
 #define _ASM_FIXMAP_H
 
 #include <linux/config.h>
-
-/* used by vmalloc.c, vsyscall.lds.S.
- *
- * Leave one empty page between vmalloc'ed areas and
- * the start of the fixmap.
- */
-#define __FIXADDR_TOP	0xfffff000
+#include <mach_fixmap.h>
 
 #ifndef __ASSEMBLY__
 #include <linux/kernel.h>
include/asm-i386/mach-default/mach_fixmap.h new file +11 -0
@@ -0,0 +1,11 @@
+#ifndef __ASM_MACH_FIXMAP_H
+#define __ASM_MACH_FIXMAP_H
+
+/* used by vmalloc.c, vsyscall.lds.S.
+ *
+ * Leave one empty page between vmalloc'ed areas and
+ * the start of the fixmap.
+ */
+#define __FIXADDR_TOP	0xfffff000
+
+#endif /* __ASM_MACH_FIXMAP_H */
include/asm-i386/mach-xen/mach_fixmap.h new file +11 -0
@@ -0,0 +1,11 @@
+#ifndef __ASM_MACH_FIXMAP_H
+#define __ASM_MACH_FIXMAP_H
+
+/* used by vmalloc.c, vsyscall.lds.S.
+ *
+ * Leave one empty page between vmalloc'ed areas and
+ * the start of the fixmap.
+ */
+#define __FIXADDR_TOP	(HYPERVISOR_VIRT_START - 2 * PAGE_SIZE)
+
+#endif /* __ASM_MACH_FIXMAP_H */

--

#22 lore raw

Re: [RFC PATCH 09/35] Change __FIXADDR_TOP to leave room for the hypervisor.

Quoted 4 lines · Chris Wright

Move the definition of __FIXADDR_TOP into a subarch include file so that it can be overridden for subarch xen -- the hypervisor needs about 64MB at the top of the address space.

I think this is more generally useful if it's actually a CONFIG option (as it was in the VMI patches) instead of subarch specific. Qemu has had a "fast" patch for a while that pretty much just increases the size of the memory hole and changes ___PAGE_OFFSET to be lower in memory. There are a number of interesting things one can do once there's an adequately sized hole too (assuming you're doing full-virtualization).

Regards,

Anthony Liguori

Quoted 64 lines
---
 include/asm-i386/fixmap.h                   |    8 +-------
 include/asm-i386/mach-default/mach_fixmap.h |   11 +++++++++++
 include/asm-i386/mach-xen/mach_fixmap.h     |   11 +++++++++++
 3 files changed, 23 insertions(+), 7 deletions(-)
include/asm-i386/fixmap.h +1 -7
@@ -14,13 +14,7 @@
 #define _ASM_FIXMAP_H
 
 #include <linux/config.h>
-
-/* used by vmalloc.c, vsyscall.lds.S.
- *
- * Leave one empty page between vmalloc'ed areas and
- * the start of the fixmap.
- */
-#define __FIXADDR_TOP	0xfffff000
+#include <mach_fixmap.h>
 
 #ifndef __ASSEMBLY__
 #include <linux/kernel.h>
include/asm-i386/mach-default/mach_fixmap.h new file +11 -0
@@ -0,0 +1,11 @@
+#ifndef __ASM_MACH_FIXMAP_H
+#define __ASM_MACH_FIXMAP_H
+
+/* used by vmalloc.c, vsyscall.lds.S.
+ *
+ * Leave one empty page between vmalloc'ed areas and
+ * the start of the fixmap.
+ */
+#define __FIXADDR_TOP	0xfffff000
+
+#endif /* __ASM_MACH_FIXMAP_H */
include/asm-i386/mach-xen/mach_fixmap.h new file +11 -2
@@ -0,0 +1,11 @@
+#ifndef __ASM_MACH_FIXMAP_H
+#define __ASM_MACH_FIXMAP_H
+
+/* used by vmalloc.c, vsyscall.lds.S.
+ *
+ * Leave one empty page between vmalloc'ed areas and
+ * the start of the fixmap.
+ */
+#define __FIXADDR_TOP	(HYPERVISOR_VIRT_START - 2 * PAGE_SIZE)
+
+#endif /* __ASM_MACH_FIXMAP_H */
 
--
 
------------------------------------------------------------------------

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.osdl.org/mailman/listinfo/virtualization

#23 lore raw

Re: [RFC PATCH 09/35] Change __FIXADDR_TOP to leave room for the hypervisor.

Quoted 6 lines · * Anthony Liguori ([email protected]) wrote:

I think this is more generally useful if it's actually a CONFIG option (as it was in the VMI patches) instead of subarch specific. Qemu has had a "fast" patch for a while that pretty much just increases the size of the memory hole and changes ___PAGE_OFFSET to be lower in memory. There are a number of interesting things one can do once there's an adequately sized hole too (assuming you're doing full-virtualization).

Good point, easy to change. Although I think dynamic may be the better approach.

thanks,
-chris

#24 lore raw

RFC PATCH 10/35 Add a new head.S start-of-day file for booting on Xen.

In reply to Chris Wright

When running on Xen, the kernel is started with paging enabled. Also don't check for cpu features which are present on all cpus supported by Xen.

Don't define segments which are not supported when running on Xen.

Define the __xen_guest section which exports information about the kernel to the domain builder.

---
 arch/i386/Makefile          |    7 +
 arch/i386/mach-xen/Makefile |    2 
 arch/i386/mach-xen/head.S   |  184 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 191 insertions(+), 2 deletions(-)
arch/i386/Makefile +5 -2
@@ -45,6 +45,9 @@ CFLAGS				+= $(shell if [ $(call cc-vers
 
 CFLAGS += $(cflags-y)
 
+# Default subarch head files
+head-y := arch/i386/kernel/head.o arch/i386/kernel/init_task.o
+
 # Default subarch .c files
 mcore-y  := mach-default
 
@@ -71,6 +74,8 @@ mcore-$(CONFIG_X86_SUMMIT)  := mach-defa
 # Xen subarch support
 mflags-$(CONFIG_X86_XEN)	:= -Iinclude/asm-i386/mach-xen
 mcore-$(CONFIG_X86_XEN)		:= mach-xen
+head-$(CONFIG_X86_XEN)		:= arch/i386/mach-xen/head.o \
+				   arch/i386/kernel/init_task.o
 
 # generic subarchitecture
 mflags-$(CONFIG_X86_GENERICARCH) := -Iinclude/asm-i386/mach-generic
@@ -85,8 +90,6 @@ core-$(CONFIG_X86_ES7000)	:= arch/i386/m
 # default subarch .h files
 mflags-y += -Iinclude/asm-i386/mach-default
 
-head-y := arch/i386/kernel/head.o arch/i386/kernel/init_task.o
-
 libs-y 					+= arch/i386/lib/
 core-y					+= arch/i386/kernel/ \
 					   arch/i386/mm/ \
arch/i386/mach-xen/Makefile +2 -0
@@ -2,6 +2,8 @@
 # Makefile for the linux kernel.
 #
 
+extra-y				:= head.o
+
 obj-y				:= setup.o
  
 setup-y				:= ../mach-default/setup.o
arch/i386/mach-xen/head.S new file +184 -0

184 lines not on this page · raw message

--

#25 lore raw

Re: [RFC PATCH 10/35] Add a new head.S start-of-day file for booting on Xen.

Quoted 10 lines
arch/i386/mach-xen/head.S new file +9 -0
+	/* get vendor info */
+	xorl %eax,%eax			# call CPUID with 0 -> return vendor ID
+	cpuid
+	movl %eax,X86_CPUID		# save CPUID level
+	movl %ebx,X86_VENDOR_ID		# lo 4 chars
+	movl %edx,X86_VENDOR_ID+4	# next 4 chars
+	movl %ecx,X86_VENDOR_ID+8	# last 4 chars
+
+	movl $1,%eax		# Use the CPUID instruction to get CPU type

arch/i386/mach-xen/head.S line 51

Quoted 10 lines
arch/i386/mach-xen/head.S new file +10 -0
+	cpuid
+	movb %al,%cl		# save reg for future use
+	andb $0x0f,%ah		# mask processor family
+	movb %ah,X86
+	andb $0xf0,%al		# mask model
+	shrb $4,%al
+	movb %al,X86_MODEL
+	andb $0x0f,%cl		# mask mask revision
+	movb %cl,X86_MASK
+	movl %edx,X86_CAPABILITY

Can you make the CPU detection a common subfunction with the normal head.S ?

Quoted 18 lines
arch/i386/mach-xen/head.S new file +18 -0
+/*
+ * BSS section
+ */
+.section ".bss.page_aligned","w"
+ENTRY(swapper_pg_dir)
+	.fill 1024,4,0
+ENTRY(empty_zero_page)
+	.fill 4096,1,0
+
+/*
+ * This starts the data section.
+ */
+.data
+
+	ALIGN
+	.word 0				# 32 bit align gdt_desc.address
+	.globl cpu_gdt_descr
+cpu_gdt_descr:

arch/i386/mach-xen/head.S line 103

Quoted 10 lines
arch/i386/mach-xen/head.S new file +10 -0
+	.word GDT_SIZE
+	.long cpu_gdt_table
+
+	.fill NR_CPUS-1,8,0		# space for the other GDT descriptors
+
+/*
+ * The Global Descriptor Table contains 28 quadwords, per-CPU.
+ */
+	.align PAGE_SIZE_asm
+ENTRY(cpu_gdt_table)

GDT and empty_zero_page should be shared (they're identical right?) Put them into a new separate common file.

arch/i386/mach-xen/head.S line 167

Quoted 12 lines

+ * __xen_guest information
+ */
+.macro utoa value
+ .if (\value) < 0 || (\value) >= 0x10

arch/i386/mach-xen/head.S new file +8 -0
+	utoa (((\value)>>4)&0x0fffffff)
+ .endif
+ .if ((\value) & 0xf) < 10
+  .byte '0' + ((\value) & 0xf)
+ .else
+  .byte 'A' + ((\value) & 0xf) - 10
+ .endif
+.endm

Interesting macro abuse.

-Andi

#26 lore raw

Re: [RFC PATCH 10/35] Add a new head.S start-of-day file for booting on Xen.

Quoted 13 lines · * Andi Kleen ([email protected]) wrote:
Quoted 13 lines
arch/i386/mach-xen/head.S new file +13 -0
+	/* get vendor info */
+	xorl %eax,%eax			# call CPUID with 0 -> return vendor ID
+	cpuid
+	movl %eax,X86_CPUID		# save CPUID level
+	movl %ebx,X86_VENDOR_ID		# lo 4 chars
+	movl %edx,X86_VENDOR_ID+4	# next 4 chars
+	movl %ecx,X86_VENDOR_ID+8	# last 4 chars
+
+	movl $1,%eax		# Use the CPUID instruction to get CPU type
+	cpuid
+	movb %al,%cl		# save reg for future use
+	andb $0x0f,%ah		# mask processor family
+	movb %ah,X86
Quoted 8 lines

arch/i386/mach-xen/head.S line 51

Quoted 6 lines
arch/i386/mach-xen/head.S new file +6 -0
+	andb $0xf0,%al		# mask model
+	shrb $4,%al
+	movb %al,X86_MODEL
+	andb $0x0f,%cl		# mask mask revision
+	movb %cl,X86_MASK
+	movl %edx,X86_CAPABILITY

Can you make the CPU detection a common subfunction with the normal head.S ?

I don't see why not, prefer to share as much as possible.

Quoted 23 lines
Quoted 23 lines
arch/i386/mach-xen/head.S new file +23 -0
+/*
+ * BSS section
+ */
+.section ".bss.page_aligned","w"
+ENTRY(swapper_pg_dir)
+	.fill 1024,4,0
+ENTRY(empty_zero_page)
+	.fill 4096,1,0
+
+/*
+ * This starts the data section.
+ */
+.data
+
+	ALIGN
+	.word 0				# 32 bit align gdt_desc.address
+	.globl cpu_gdt_descr
+cpu_gdt_descr:
+	.word GDT_SIZE
+	.long cpu_gdt_table
+
+	.fill NR_CPUS-1,8,0		# space for the other GDT descriptors
+
Quoted 8 lines

arch/i386/mach-xen/head.S line 103

Quoted 5 lines
arch/i386/mach-xen/head.S new file +5 -0
+/*
+ * The Global Descriptor Table contains 28 quadwords, per-CPU.
+ */
+	.align PAGE_SIZE_asm
+ENTRY(cpu_gdt_table)

GDT and empty_zero_page should be shared (they're identical right?) Put them into a new separate common file.

There's still kernel/user cs/ds in gdt, so it's not all zero.

thanks,
-chris

#27 lore raw

Re: [RFC PATCH 10/35] Add a new head.S start-of-day file for booting on Xen.

Quoted 2 lines · Chris Wright

There's still kernel/user cs/ds in gdt, so it's not all zero.

Make them symbol with a common macro then.

-Andi

#28 lore raw

Re: [RFC PATCH 10/35] Add a new head.S start-of-day file for booting on Xen.

Quoted 4 lines · * Andi Kleen ([email protected]) wrote:
Quoted 1 line · Chris Wright

There's still kernel/user cs/ds in gdt, so it's not all zero.

Make them symbol with a common macro then.

To fill entries as defined by subarch? That should work.

thanks,
-chris

#29 lore raw

RFC PATCH 11/35 Add support for Xen to entry.S.

In reply to Chris Wright

- change cli/sti
- change test for user mode return to work for kernel mode in ring1
- check hypervisor saved event mask on return from exception
- add entry points for the hypervisor upcall handlers
- avoid math emulation check when running on Xen
- add nmi handler for running on Xen

---
 arch/i386/kernel/entry.S |  228 +++++++++++++++++++++++++++++++++++++++++++----
 1 files changed, 211 insertions(+), 17 deletions(-)
arch/i386/kernel/entry.S +211 -17
@@ -75,8 +75,42 @@ DF_MASK		= 0x00000400 
 NT_MASK		= 0x00004000
 VM_MASK		= 0x00020000
 
+#ifndef CONFIG_XEN
+#define DISABLE_INTERRUPTS	cli
+#define ENABLE_INTERRUPTS	sti
+#else
+#include <xen/interface/xen.h>
+
+EVENT_MASK	= 0x2E
+
+/* Pseudo-eflags. */
+NMI_MASK	= 0x80000000
+
+/* Offsets into shared_info_t. */
+#define evtchn_upcall_pending		/* 0 */
+#define evtchn_upcall_mask		1
+
+#define sizeof_vcpu_shift		6
+
+#ifdef CONFIG_SMP
+#define GET_VCPU_INFO		movl TI_cpu(%ebp),%esi			; \
+				shl  $sizeof_vcpu_shift,%esi		; \
+				addl HYPERVISOR_shared_info,%esi
+#else
+#define GET_VCPU_INFO		movl HYPERVISOR_shared_info,%esi
+#endif
+
+#define __DISABLE_INTERRUPTS	movb $1,evtchn_upcall_mask(%esi)
+#define __ENABLE_INTERRUPTS	movb $0,evtchn_upcall_mask(%esi)
+#define DISABLE_INTERRUPTS	GET_VCPU_INFO				; \
+				__DISABLE_INTERRUPTS
+#define ENABLE_INTERRUPTS	GET_VCPU_INFO				; \
+				__ENABLE_INTERRUPTS
+#define __TEST_PENDING		testb $0xFF,evtchn_upcall_pending(%esi)
+#endif
+
 #ifdef CONFIG_PREEMPT
-#define preempt_stop		cli
+#define preempt_stop		DISABLE_INTERRUPTS
 #else
 #define preempt_stop
 #define resume_kernel		restore_nocheck
@@ -145,10 +179,10 @@ ret_from_intr:
 	GET_THREAD_INFO(%ebp)
 	movl EFLAGS(%esp), %eax		# mix EFLAGS and CS
 	movb CS(%esp), %al
-	testl $(VM_MASK | 3), %eax
+	testl $(VM_MASK | USER_MODE_MASK), %eax
 	jz resume_kernel
 ENTRY(resume_userspace)
- 	cli				# make sure we don't miss an interrupt
+	DISABLE_INTERRUPTS		# make sure we don't miss an interrupt
 					# setting need_resched or sigpending
 					# between sampling and the iret
 	movl TI_flags(%ebp), %ecx
@@ -159,7 +193,7 @@ ENTRY(resume_userspace)
 
 #ifdef CONFIG_PREEMPT
 ENTRY(resume_kernel)
-	cli
+	DISABLE_INTERRUPTS
 	cmpl $0,TI_preempt_count(%ebp)	# non-zero preempt_count ?
 	jnz restore_nocheck
 need_resched:
@@ -179,7 +213,7 @@ need_resched:
 ENTRY(sysenter_entry)
 	movl TSS_sysenter_esp0(%esp),%esp
 sysenter_past_esp:
-	sti
+	ENABLE_INTERRUPTS
 	pushl $(__USER_DS)
 	pushl %ebp
 	pushfl
@@ -209,7 +243,7 @@ sysenter_past_esp:
 	jae syscall_badsys
 	call *sys_call_table(,%eax,4)
 	movl %eax,EAX(%esp)
-	cli
+	DISABLE_INTERRUPTS
 	movl TI_flags(%ebp), %ecx
 	testw $_TIF_ALLWORK_MASK, %cx
 	jne syscall_exit_work
@@ -217,7 +251,7 @@ sysenter_past_esp:
 	movl EIP(%esp), %edx
 	movl OLDESP(%esp), %ecx
 	xorl %ebp,%ebp
-	sti
+	ENABLE_INTERRUPTS
 	sysexit
 
 
@@ -236,7 +270,7 @@ syscall_call:
 	call *sys_call_table(,%eax,4)
 	movl %eax,EAX(%esp)		# store the return value
 syscall_exit:
-	cli				# make sure we don't miss an interrupt
+	DISABLE_INTERRUPTS		# make sure we don't miss an interrupt
 					# setting need_resched or sigpending
 					# between sampling and the iret
 	movl TI_flags(%ebp), %ecx
@@ -244,6 +278,7 @@ syscall_exit:
 	jne syscall_exit_work
 
 restore_all:
+#ifndef CONFIG_XEN
 	movl EFLAGS(%esp), %eax		# mix EFLAGS, SS and CS
 	# Warning: OLDSS(%esp) contains the wrong/random values if we
 	# are returning to the kernel.
@@ -254,12 +289,25 @@ restore_all:
 	cmpl $((4 << 8) | 3), %eax
 	je ldt_ss			# returning to user-space with LDT SS
 restore_nocheck:
+#else
+restore_nocheck:
+	testl $(VM_MASK|NMI_MASK), EFLAGS(%esp)
+	jnz hypervisor_iret
+	movb EVENT_MASK(%esp), %al
+	notb %al			# %al == ~saved_mask
+	GET_VCPU_INFO
+	andb evtchn_upcall_mask(%esi),%al
+	andb $1,%al			# %al == mask & ~saved_mask
+	jnz restore_all_enable_events	#     != 0 => reenable event delivery
+#endif
 	RESTORE_REGS
 	addl $4, %esp
 1:	iret
 .section .fixup,"ax"
 iret_exc:
-	sti
+#ifndef CONFIG_XEN
+	ENABLE_INTERRUPTS
+#endif
 	pushl $0			# no error code
 	pushl $do_iret_error
 	jmp error_code
@@ -269,6 +317,7 @@ iret_exc:
 	.long 1b,iret_exc
 .previous
 
+#ifndef CONFIG_XEN
 ldt_ss:
 	larl OLDSS(%esp), %eax
 	jnz restore_nocheck
@@ -281,7 +330,7 @@ ldt_ss:
 	 * CPUs, which we can try to work around to make
 	 * dosemu and wine happy. */
 	subl $8, %esp		# reserve space for switch16 pointer
-	cli
+	DISABLE_INTERRUPTS
 	movl %esp, %eax
 	/* Set up the 16bit stack frame with switch32 pointer on top,
 	 * and a switch16 pointer on top of the current frame. */
@@ -293,6 +342,13 @@ ldt_ss:
 	.align 4
 	.long 1b,iret_exc
 .previous
+#else
+hypervisor_iret:
+	andl $~NMI_MASK, EFLAGS(%esp)
+	RESTORE_REGS
+	addl $4, %esp
+	jmp  hypercall_page + (__HYPERVISOR_iret * 32)
+#endif
 
 	# perform work that needs to be done immediately before resumption
 	ALIGN
@@ -301,7 +357,7 @@ work_pending:
 	jz work_notifysig
 work_resched:
 	call schedule
-	cli				# make sure we don't miss an interrupt
+	DISABLE_INTERRUPTS		# make sure we don't miss an interrupt
 					# setting need_resched or sigpending
 					# between sampling and the iret
 	movl TI_flags(%ebp), %ecx
@@ -353,7 +409,7 @@ syscall_trace_entry:
 syscall_exit_work:
 	testb $(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP), %cl
 	jz work_pending
-	sti				# could let do_syscall_trace() call
+	ENABLE_INTERRUPTS		# could let do_syscall_trace() call
 					# schedule() instead
 	movl %esp, %eax
 	movl $1, %edx
@@ -373,6 +429,7 @@ syscall_badsys:
 	movl $-ENOSYS,EAX(%esp)
 	jmp resume_userspace
 
+#ifndef CONFIG_XEN
 #define FIXUP_ESPFIX_STACK \
 	movl %esp, %eax; \
 	/* switch to 32bit stack using the pointer on top of 16bit stack */ \
@@ -431,6 +488,9 @@ ENTRY(name)				\
 
 /* The include is where all of the SMP etc. interrupts come from */
 #include "entry_arch.h"
+#else
+#define UNWIND_ESPFIX_STACK
+#endif
 
 ENTRY(divide_error)
 	pushl $0			# no error code
@@ -462,6 +522,126 @@ error_code:
 	call *%edi
 	jmp ret_from_exception
 
+#ifdef CONFIG_XEN
+# A note on the "critical region" in our callback handler.
+# We want to avoid stacking callback handlers due to events occurring
+# during handling of the last event. To do this, we keep events disabled
+# until we've done all processing. HOWEVER, we must enable events before
+# popping the stack frame (can't be done atomically) and so it would still
+# be possible to get enough handler activations to overflow the stack.
+# Although unlikely, bugs of that kind are hard to track down, so we'd
+# like to avoid the possibility.
+# So, on entry to the handler we detect whether we interrupted an
+# existing activation in its critical region -- if so, we pop the current
+# activation and restart the handler using the previous one.
+ENTRY(hypervisor_callback)
+	pushl %eax
+	SAVE_ALL
+	movl EIP(%esp),%eax
+	cmpl $scrit,%eax
+	jb   11f
+	cmpl $ecrit,%eax
+	jb   critical_region_fixup
+11:	push %esp
+	call evtchn_do_upcall
+	add  $4,%esp
+	jmp  ret_from_intr
+
+        ALIGN
+restore_all_enable_events:
+	__ENABLE_INTERRUPTS
+scrit:	/**** START OF CRITICAL REGION ****/
+	__TEST_PENDING
+	jnz  14f			# process more events if necessary...
+	RESTORE_REGS
+	addl $4, %esp
+1:	iret
+.section .fixup,"ax"
+2:	pushl $0
+	pushl $do_iret_error
+	jmp error_code
+.previous
+.section __ex_table,"a"
+	.align 4
+	.long 1b,2b
+.previous
+14:	__DISABLE_INTERRUPTS
+	jmp  11b
+ecrit:  /**** END OF CRITICAL REGION ****/
+# [How we do the fixup]. We want to merge the current stack frame with the
+# just-interrupted frame. How we do this depends on where in the critical
+# region the interrupted handler was executing, and so how many saved
+# registers are in each frame. We do this quickly using the lookup table
+# 'critical_fixup_table'. For each byte offset in the critical region, it
+# provides the number of bytes which have already been popped from the
+# interrupted stack frame.
+critical_region_fixup:
+	addl $critical_fixup_table-scrit,%eax
+	movzbl (%eax),%eax		# %eax contains num bytes popped
+	cmpb $0xff,%al                  # 0xff => vcpu_info critical region
+	jne  15f
+	GET_THREAD_INFO(%ebp)
+        xorl %eax,%eax
+15:	mov  %esp,%esi
+	add  %eax,%esi			# %esi points at end of src region
+	mov  %esp,%edi
+	add  $0x34,%edi			# %edi points at end of dst region
+	mov  %eax,%ecx
+	shr  $2,%ecx			# convert words to bytes
+	je   17f			# skip loop if nothing to copy
+16:	subl $4,%esi			# pre-decrementing copy loop
+	subl $4,%edi
+	movl (%esi),%eax
+	movl %eax,(%edi)
+	loop 16b
+17:	movl %edi,%esp			# final %edi is top of merged stack
+	jmp  11b
+
+critical_fixup_table:
+	.byte 0xff,0xff,0xff		# testb $0xff,(%esi) = __TEST_PENDING
+	.byte 0xff,0xff			# jnz  14f
+	.byte 0x00			# pop  %ebx
+	.byte 0x04			# pop  %ecx
+	.byte 0x08			# pop  %edx
+	.byte 0x0c			# pop  %esi
+	.byte 0x10			# pop  %edi
+	.byte 0x14			# pop  %ebp
+	.byte 0x18			# pop  %eax
+	.byte 0x1c			# pop  %ds
+	.byte 0x20			# pop  %es
+	.byte 0x24,0x24,0x24		# add  $4,%esp
+	.byte 0x28			# iret
+	.byte 0xff,0xff,0xff,0xff	# movb $1,1(%esi)
+	.byte 0x00,0x00			# jmp  11b
+
+# Hypervisor uses this for application faults while it executes.
+ENTRY(failsafe_callback)
+1:	popl %ds
+2:	popl %es
+3:	popl %fs
+4:	popl %gs
+	subl $4,%esp
+	SAVE_ALL
+	jmp  ret_from_exception
+.section .fixup,"ax";	\
+6:	movl $0,(%esp);	\
+	jmp 1b;		\
+7:	movl $0,(%esp);	\
+	jmp 2b;		\
+8:	movl $0,(%esp);	\
+	jmp 3b;		\
+9:	movl $0,(%esp);	\
+	jmp 4b;		\
+.previous;		\
+.section __ex_table,"a";\
+	.align 4;	\
+	.long 1b,6b;	\
+	.long 2b,7b;	\
+	.long 3b,8b;	\
+	.long 4b,9b;	\
+.previous
+#endif
+
 ENTRY(coprocessor_error)
 	pushl $0
 	pushl $do_coprocessor_error
@@ -475,17 +655,19 @@ ENTRY(simd_coprocessor_error)
 ENTRY(device_not_available)
 	pushl $-1			# mark this as an int
 	SAVE_ALL
+#ifndef CONFIG_XEN
 	movl %cr0, %eax
 	testl $0x4, %eax		# EM (math emulation bit)
-	jne device_not_available_emulate
-	preempt_stop
-	call math_state_restore
-	jmp ret_from_exception
-device_not_available_emulate:
+	je device_available_emulate
 	pushl $0			# temporary storage for ORIG_EIP
 	call math_emulate
 	addl $4, %esp
 	jmp ret_from_exception
+device_available_emulate:
+#endif
+	preempt_stop
+	call math_state_restore
+	jmp ret_from_exception
 
 /*
  * Debug traps and NMI can happen at the one SYSENTER instruction
@@ -521,6 +703,8 @@ debug_stack_correct:
 	call do_debug
 	jmp ret_from_exception
 	.previous .text
+
+#ifndef CONFIG_XEN
 /*
  * NMI is doubly nasty. It can happen _while_ we're handling
  * a debug fault, and the debug fault hasn't yet been able to
@@ -591,6 +775,16 @@ nmi_16bit_stack:
 	.align 4
 	.long 1b,iret_exc
 .previous
+#else
+ENTRY(nmi)
+	pushl %eax
+	SAVE_ALL
+	xorl %edx,%edx		# zero error code
+	movl %esp,%eax		# pt_regs pointer
+	call do_nmi
+	orl  $NMI_MASK, EFLAGS(%esp)
+	jmp restore_all
+#endif
 
 KPROBE_ENTRY(int3)
 	pushl $-1			# mark this as an int

--

#30 lore raw

Re: [RFC PATCH 11/35] Add support for Xen to entry.S.

arch/i386/kernel/entry.S DF_MASK = 0x00000400 · line 93

Quoted 6 lines · Chris Wright
arch/i386/kernel/entry.S +6 -0
+
+/* Offsets into shared_info_t. */
+#define evtchn_upcall_pending		/* 0 */
+#define evtchn_upcall_mask		1
+
+#define sizeof_vcpu_shift		6

This should be all generated in asm-offsets.c

Quoted 1 line

+#else

This needs a comment explaining why it is different

arch/i386/kernel/entry.S restore_all: · line 300

Quoted 8 lines
arch/i386/kernel/entry.S +8 -0
+restore_nocheck:
+	testl $(VM_MASK|NMI_MASK), EFLAGS(%esp)
+	jnz hypervisor_iret
+	movb EVENT_MASK(%esp), %al
+	notb %al			# %al == ~saved_mask
+	GET_VCPU_INFO
+	andb evtchn_upcall_mask(%esi),%al
+	andb $1,%al			# %al == mask & ~saved_mask

This would be faster if you used 32bit because it wouldn't cause partial register stalls.

Quoted 15 lines
arch/i386/kernel/entry.S +5 -1
+	jnz restore_all_enable_events	#     != 0 => reenable event delivery
+#endif
 	RESTORE_REGS
 	addl $4, %esp
 1:	iret
 .section .fixup,"ax"
 iret_exc:
-	sti
+#ifndef CONFIG_XEN
+	ENABLE_INTERRUPTS
+#endif
 	pushl $0			# no error code
 	pushl $do_iret_error
 	jmp error_code

arch/i386/kernel/entry.S iret_exc: · line 321

Quoted 5 lines
arch/i386/kernel/entry.S +1 -0
@@ -269,6 +317,7 @@ iret_exc:
 	.long 1b,iret_exc
 .previous
 
+#ifndef CONFIG_XEN
 ldt_ss:

So are you sure that problem this ugly piece of code tries to work around isn't in Xen kernels too? Or do you just not care? If yes add a comment.

Quoted 12 lines
arch/i386/kernel/entry.S +1 -1
 	larl OLDSS(%esp), %eax
 	jnz restore_nocheck
@@ -281,7 +330,7 @@ ldt_ss:
 	 * CPUs, which we can try to work around to make
 	 * dosemu and wine happy. */
 	subl $8, %esp		# reserve space for switch16 pointer
-	cli
+	DISABLE_INTERRUPTS
 	movl %esp, %eax
 	/* Set up the 16bit stack frame with switch32 pointer on top,
 	 * and a switch16 pointer on top of the current frame. */

arch/i386/kernel/entry.S ldt_ss: · line 345

Quoted 4 lines
arch/i386/kernel/entry.S +1 -0
@@ -293,6 +342,13 @@ ldt_ss:
 	.align 4
 	.long 1b,iret_exc
 .previous
+#else

Needs comment.

Quoted 12 lines
arch/i386/kernel/entry.S +6 -0
+hypervisor_iret:
+	andl $~NMI_MASK, EFLAGS(%esp)
+	RESTORE_REGS
+	addl $4, %esp
+	jmp  hypercall_page + (__HYPERVISOR_iret * 32)
+#endif
 
 
 

ENTRY(divide_error)

arch/i386/kernel/entry.S +0 -0
 	pushl $0			# no error code

arch/i386/kernel/entry.S error_code: · line 525

Quoted 4 lines
arch/i386/kernel/entry.S +1 -0
@@ -462,6 +522,126 @@ error_code:
 	call *%edi
 	jmp ret_from_exception
 
+#ifdef CONFIG_XEN

It would be nicer to put that one into a separate file

Quoted 18 lines
arch/i386/kernel/entry.S +2 -5
+# A note on the "critical region" in our callback handler.
 
 
 
 
 	pushl $do_coprocessor_error
@@ -475,17 +655,19 @@ ENTRY(simd_coprocessor_error)
 ENTRY(device_not_available)
 	pushl $-1			# mark this as an int
 	SAVE_ALL
+#ifndef CONFIG_XEN
 	movl %cr0, %eax
 	testl $0x4, %eax		# EM (math emulation bit)
-	jne device_not_available_emulate
-	preempt_stop
-	call math_state_restore
-	jmp ret_from_exception
-device_not_available_emulate:

arch/i386/kernel/entry.S ENTRY(simd_coprocessor_error) · line 670

Quoted 10 lines
arch/i386/kernel/entry.S +6 -0
+	je device_available_emulate
 	pushl $0			# temporary storage for ORIG_EIP
 	call math_emulate
 	addl $4, %esp
 	jmp ret_from_exception
+device_available_emulate:
+#endif
+	preempt_stop
+	call math_state_restore
+	jmp ret_from_exception

It's not quite clear why the Xen special case is needed here. Just because of the CR0 access? If yes better put that one into a macro that just writes 0 to the register for the Xen case.

-Andi

#31 lore raw

Re: [Xen-devel] Re: [RFC PATCH 11/35] Add support for Xen to entry.S.

Quoted 19 lines · Andi Kleen
Quoted 19 lines
arch/i386/kernel/entry.S +5 -1
+	jnz restore_all_enable_events	#     != 0 => reenable event delivery
+#endif
 	RESTORE_REGS
 	addl $4, %esp
 1:	iret
 .section .fixup,"ax"
 iret_exc:
-	sti
+#ifndef CONFIG_XEN
+	ENABLE_INTERRUPTS
+#endif
 	pushl $0			# no error code
 	pushl $do_iret_error
 	jmp error_code
@@ -269,6 +317,7 @@ iret_exc:
 	.long 1b,iret_exc
 .previous
 
Quoted 7 lines

arch/i386/kernel/entry.S iret_exc:

Quoted 3 lines
arch/i386/kernel/entry.S +1 -0
+#ifndef CONFIG_XEN
 ldt_ss:
 

So are you sure that problem this ugly piece of code tries to work around isn't in Xen kernels too? Or do you just not care? If yes add a comment.

This code would otherwise be broken. ENABLE_INTERRUPTS in Xen requires access to the data segment, and the data segment is not available at this point. Plus, it corrupts the %esi register. Hint - use %ebp.

The LDT SS code is broken as well because the iret onto a 16-bit stack is a pretty crippling blow to transparency in this code. Then, you don't have data or even stack segments that are reliable for calling out to hypervisor assist code. We never really fixed this code either in our implementation, although we did consider several approaches. Leaving it out does break userspace applications.

Zach

#32 lore raw

RFC PATCH 12/35 Add start-of-day setup hooks to subarch

In reply to Chris Wright

Implement the start-of-day subarchitecture setup hooks for booting on Xen. Add subarch macros for determining loader type and initrd location.

---
 arch/i386/kernel/setup.c                    |    5 ++
 include/asm-i386/hypervisor.h               |    4 +
 include/asm-i386/mach-default/mach_setup.h  |   12 +++++
 include/asm-i386/mach-xen/mach_setup.h      |   13 ++++++
 include/asm-i386/mach-xen/setup_arch_post.h |   60 ++++++++++++++++++++++++++++
 include/asm-i386/mach-xen/setup_arch_pre.h  |   22 ++++++++++
 include/asm-i386/setup.h                    |    8 ++-
 7 files changed, 121 insertions(+), 3 deletions(-)
arch/i386/kernel/setup.c +5 -0
@@ -456,6 +456,7 @@ static void __init print_memory_map(char
 	}
 }
 
+#ifndef HAVE_ARCH_E820_SANITIZE
 /*
  * Sanitize the BIOS e820 map.
  *
@@ -675,6 +676,7 @@ static int __init copy_e820_map(struct e
 	} while (biosmap++,--nr_map);
 	return 0;
 }
+#endif
 
 #if defined(CONFIG_EDD) || defined(CONFIG_EDD_MODULE)
 struct edd edd;
@@ -1632,6 +1634,9 @@ void __init setup_arch(char **cmdline_p)
 	conswitchp = &dummy_con;
 #endif
 #endif
+#ifdef ARCH_FINAL_SETUP
+	ARCH_FINAL_SETUP
+#endif
 }
 
 #include "setup_arch_post.h"
include/asm-i386/hypervisor.h +4 -0
@@ -53,6 +53,10 @@
 extern struct shared_info *HYPERVISOR_shared_info;
 extern struct start_info *xen_start_info;
 
+/* arch/i386/mach-xen/entry.S */
+extern void hypervisor_callback(void);
+extern void failsafe_callback(void);
+
 /* arch/i386/mach-xen/evtchn.c */
 /* Force a proper event-channel callback from Xen. */
 extern void force_evtchn_callback(void);
include/asm-i386/setup.h +5 -3
@@ -51,10 +51,10 @@ extern unsigned char boot_params[PARAM_S
 #define VIDEO_MODE (*(unsigned short *) (PARAM+0x1FA))
 #define ORIG_ROOT_DEV (*(unsigned short *) (PARAM+0x1FC))
 #define AUX_DEVICE_INFO (*(unsigned char *) (PARAM+0x1FF))
-#define LOADER_TYPE (*(unsigned char *) (PARAM+0x210))
+#define LOADER_TYPE MACH_LOADER_TYPE
 #define KERNEL_START (*(unsigned long *) (PARAM+0x214))
-#define INITRD_START (*(unsigned long *) (PARAM+0x218))
-#define INITRD_SIZE (*(unsigned long *) (PARAM+0x21c))
+#define INITRD_START MACH_INITRD_START
+#define INITRD_SIZE MACH_INITRD_SIZE
 #define EDID_INFO   (*(struct edid_info *) (PARAM+0x140))
 #define EDD_NR     (*(unsigned char *) (PARAM+EDDNR))
 #define EDD_MBR_SIG_NR (*(unsigned char *) (PARAM+EDD_MBR_SIG_NR_BUF))
@@ -63,4 +63,6 @@ extern unsigned char boot_params[PARAM_S
 
 #endif /* __ASSEMBLY__ */
 
+#include <mach_setup.h>
+
 #endif /* _i386_SETUP_H */
include/asm-i386/mach-default/mach_setup.h new file +12 -0
@@ -0,0 +1,12 @@
+#ifndef __ASM_MACH_SETUP_H
+#define __ASM_MACH_SETUP_H
+
+#ifndef __ASSEMBLY__
+
+#define MACH_LOADER_TYPE (*(unsigned char *) (PARAM+0x210))
+#define MACH_INITRD_START (*(unsigned long *) (PARAM+0x218))
+#define MACH_INITRD_SIZE (*(unsigned long *) (PARAM+0x21c))
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* __ASM_MACH_SETUP_H */
include/asm-i386/mach-xen/mach_setup.h new file +13 -0
@@ -0,0 +1,13 @@
+#ifndef __ASM_MACH_SETUP_H
+#define __ASM_MACH_SETUP_H
+
+#ifndef __ASSEMBLY__
+
+#define MACH_LOADER_TYPE 0x6e6558	/* "Xen" */
+#define MACH_INITRD_START \
+	(xen_start_info->mod_start ? __pa(xen_start_info->mod_start) : 0)
+#define MACH_INITRD_SIZE (xen_start_info->mod_len)
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* __ASM_MACH_SETUP_H */
include/asm-i386/mach-xen/setup_arch_post.h new file +60 -0

60 lines not on this page · raw message

include/asm-i386/mach-xen/setup_arch_pre.h new file +22 -0
@@ -0,0 +1,22 @@
+
+#include <xen/interface/xen.h>
+#include <asm/hypervisor.h>
+
+struct start_info *xen_start_info;
+EXPORT_SYMBOL(xen_start_info);
+
+/*
+ * Point at the empty zero page to start with. We map the real shared_info
+ * page as soon as fixmap is up and running.
+ */
+struct shared_info *HYPERVISOR_shared_info =
+	(struct shared_info *)empty_zero_page;
+EXPORT_SYMBOL(HYPERVISOR_shared_info);
+
+#define ARCH_SETUP machine_specific_arch_setup();
+#define ARCH_FINAL_SETUP machine_specific_arch_final_setup();
+
+static void __init machine_specific_arch_setup(void);
+static void __init machine_specific_arch_final_setup(void);
+
+#define HAVE_ARCH_E820_SANITIZE

--

#33 lore raw

RFC PATCH 13/35 Support loading an initrd when running on Xen

In reply to Chris Wright

Due to the initial physical memory layout when booting on Xen, the initrd image ends up below min_low_pfn (as registered with the bootstrap memory allocator). Add an i386 build option to allow this scenario by setting the initrd_below_start_ok flag.

---
 arch/i386/kernel/setup.c |    7 ++++++-
 drivers/block/Kconfig    |    4 ++++
 2 files changed, 10 insertions(+), 1 deletion(-)
arch/i386/kernel/setup.c +6 -1
@@ -1237,10 +1237,15 @@ void __init setup_bootmem_allocator(void
 #ifdef CONFIG_BLK_DEV_INITRD
 	if (LOADER_TYPE && INITRD_START) {
 		if (INITRD_START + INITRD_SIZE <= (max_low_pfn << PAGE_SHIFT)) {
-			reserve_bootmem(INITRD_START, INITRD_SIZE);
 			initrd_start =
 				INITRD_START ? INITRD_START + PAGE_OFFSET : 0;
 			initrd_end = initrd_start+INITRD_SIZE;
+#ifdef CONFIG_BLK_DEV_INITRD_BELOW_START_OK
+			if (initrd_start < min_low_pfn << PAGE_SHIFT)
+				initrd_below_start_ok = 1;
+			else
+#endif
+			reserve_bootmem(INITRD_START, INITRD_SIZE);
 		}
 		else {
 			printk(KERN_ERR "initrd extends beyond end of memory "
drivers/block/Kconfig +4 -0
@@ -409,6 +409,10 @@ config BLK_DEV_INITRD
 	  "real" root file system, etc. See <file:Documentation/initrd.txt>
 	  for details.
 
+config BLK_DEV_INITRD_BELOW_START_OK
+	bool
+	depends on XEN
+	default BLK_DEV_INITRD
 
 config CDROM_PKTCDVD
 	tristate "Packet writing on CD/DVD media"

--

#34 lore raw

Re: [RFC PATCH 13/35] Support loading an initrd when running on Xen

Quoted 4 lines · Chris Wright

Due to the initial physical memory layout when booting on Xen, the initrd image ends up below min_low_pfn (as registered with the bootstrap memory allocator). Add an i386 build option to allow this scenario by setting the initrd_below_start_ok flag.

Better would be to just handle that by default without any CONFIGs.

-Andi

#35 lore raw

Re: [RFC PATCH 13/35] Support loading an initrd when running on Xen

Quoted 7 lines · * Andi Kleen ([email protected]) wrote:
Quoted 4 lines · Chris Wright

Due to the initial physical memory layout when booting on Xen, the initrd image ends up below min_low_pfn (as registered with the bootstrap memory allocator). Add an i386 build option to allow this scenario by setting the initrd_below_start_ok flag.

Better would be to just handle that by default without any CONFIGs.

Indeed, thanks.
-chris

#36 lore raw

RFC PATCH 14/35 subarch modify CPU capabilities

In reply to Chris Wright

Allow subarchitectures to modify CPU capabilities during bootstrap CPU identification. Add a subarch implementation for Xen which hides features unsupported by the hypervisor.

---
 arch/i386/kernel/cpu/common.c               |    3 +++
 include/asm-i386/mach-default/mach_cpu.h    |    5 +++++
 include/asm-i386/mach-xen/mach_cpu.h        |    2 ++
 include/asm-i386/mach-xen/setup_arch_post.h |   15 +++++++++++++++
 include/asm-i386/mach-xen/setup_arch_pre.h  |    1 +
 5 files changed, 26 insertions(+)
arch/i386/kernel/cpu/common.c +3 -0
@@ -16,6 +16,7 @@
 #include <asm/apic.h>
 #include <mach_apic.h>
 #endif
+#include <mach_cpu.h>
 
 #include "cpu.h"
 
@@ -420,6 +421,8 @@ void __devinit identify_cpu(struct cpuin
 				c->x86_vendor, c->x86_model);
 	}
 
+	machine_specific_modify_cpu_capabilities(c);
+
 	/* Now the feature flags better reflect actual CPU features! */
 
 	printk(KERN_DEBUG "CPU: After all inits, caps:");
include/asm-i386/mach-xen/setup_arch_post.h +15 -0
@@ -18,6 +18,19 @@ static char * __init machine_specific_me
 	return "Xen";
 }
 
+void __devinit machine_specific_modify_cpu_capabilities(struct cpuinfo_x86 *c)
+{
+	clear_bit(X86_FEATURE_VME, c->x86_capability);
+	clear_bit(X86_FEATURE_DE, c->x86_capability);
+	clear_bit(X86_FEATURE_PSE, c->x86_capability);
+	clear_bit(X86_FEATURE_PGE, c->x86_capability);
+	clear_bit(X86_FEATURE_SEP, c->x86_capability);
+	clear_bit(X86_FEATURE_MWAIT, c->x86_capability);
+	if (!(xen_start_info->flags & SIF_PRIVILEGED))
+		clear_bit(X86_FEATURE_MTRR, c->x86_capability);
+	c->hlt_works_ok = 0;
+}
+
 static void __init machine_specific_arch_setup(void)
 {
 	struct physdev_op op;
@@ -30,6 +43,8 @@ static void __init machine_specific_arch
 	    __KERNEL_CS, (unsigned long)hypervisor_callback,
 	    __KERNEL_CS, (unsigned long)failsafe_callback);
 
+	machine_specific_modify_cpu_capabilities(&boot_cpu_data);
+
 	init_pg_tables_end = __pa(xen_start_info->pt_base) +
 		PFN_PHYS(xen_start_info->nr_pt_frames);
 
include/asm-i386/mach-xen/setup_arch_pre.h +1 -0
@@ -1,6 +1,7 @@
 
 #include <xen/interface/xen.h>
 #include <asm/hypervisor.h>
+#include <mach_cpu.h>
 
 struct start_info *xen_start_info;
 EXPORT_SYMBOL(xen_start_info);
include/asm-i386/mach-default/mach_cpu.h new file +5 -0
@@ -0,0 +1,5 @@
+
+static inline void
+machine_specific_modify_cpu_capabilities(struct cpuinfo_x86 *c)
+{
+}
include/asm-i386/mach-xen/mach_cpu.h new file +2 -0
@@ -0,0 +1,2 @@
+
+extern void machine_specific_modify_cpu_capabilities(struct cpuinfo_x86 *);

--

#37 lore raw

Re: [Xen-devel] [RFC PATCH 14/35] subarch modify CPU capabilities

include/asm-i386/mach-xen/setup_arch_post.h static char * __init machine_specific_me · line 31

Quoted 13 lines · Chris Wright
include/asm-i386/mach-xen/setup_arch_post.h +11 -0
+void __devinit machine_specific_modify_cpu_capabilities(struct cpuinfo_x86 *c)
+{
+	clear_bit(X86_FEATURE_VME, c->x86_capability);
+	clear_bit(X86_FEATURE_DE, c->x86_capability);
+	clear_bit(X86_FEATURE_PSE, c->x86_capability);
+	clear_bit(X86_FEATURE_PGE, c->x86_capability);
+	clear_bit(X86_FEATURE_SEP, c->x86_capability);
+	clear_bit(X86_FEATURE_MWAIT, c->x86_capability);
+	if (!(xen_start_info->flags & SIF_PRIVILEGED))
+		clear_bit(X86_FEATURE_MTRR, c->x86_capability);
+	c->hlt_works_ok = 0;

That's pretty heinous. Suppose Xen decides to start supporting any of these features. Now, you need to change all of the Xen modified kernels to remove pieces of this.

You've effectively introduced completely arbitrary, but most importantly, static constraints into Linux based on what features Xen currently supports. This is not productive on either end.

This is the entire point of the VMI interface. You can arbitrarily enable and disable these features in a hidden layer, call it VMI, call it Xen-bridge, call it hypercall page, whatever. Using this layer, you can automatically abstract away exactly this type of nasty, deliberately deprecating extraneous code. We use this feature exactly to strip away these bits, completely hidden from the guest, by "trapping" the CPUID instruction in the VMI layer. The VMI interface as it stands today is unimportant - it has and will continue to change to accommodate Xen. But the principles of it are important for maintainability, flexibility, and future compatibility. One of these principles is to avoid unbalanced changes like this to the guest kernel.

And we certainly support VME, DE, PSE, PGE, and SEP features, and really would not like them disabled unconditionally in a virtual environment.

Zach

#38 lore raw

Re: [Xen-devel] [RFC PATCH 14/35] subarch modify CPU capabilities

Quoted 1 line · Zachary Amsden

That's pretty heinous. Suppose Xen decides to start supporting any of these features. Now, you need to change all of the Xen modified kernels to remove pieces of this.

Well, old kernels would still work and most of those features aren't all that useful in a virtualised environment anyway. But, point taken, adding a hypervisor trap for CPUID and making use of it would clean up that code.

-- Keir

#39 lore raw

Re: [Xen-devel] [RFC PATCH 14/35] subarch modify CPU capabilities

Quoted 11 lines · * Keir Fraser ([email protected]) wrote:
Quoted 3 lines · Zachary Amsden

That's pretty heinous. Suppose Xen decides to start supporting any of these features. Now, you need to change all of the Xen modified kernels to remove pieces of this.

Well, old kernels would still work and most of those features aren't all that useful in a virtualised environment anyway. But, point taken, adding a hypervisor trap for CPUID and making use of it would clean up that code.

Agreed, added to todo.

thanks,
-chris

#40 lore raw

RFC PATCH 15/35 subarch support for controlling interrupt delivery

In reply to Chris Wright

Abstract the code that controls interrupt delivery, and add a separate subarch implementation for Xen that manipulates a shared-memory event delivery mask.

---
 include/asm-i386/mach-default/mach_system.h |   24 ++++++
 include/asm-i386/mach-xen/mach_system.h     |  103 ++++++++++++++++++++++++++++
 include/asm-i386/system.h                   |   20 -----
 3 files changed, 128 insertions(+), 19 deletions(-)
include/asm-i386/system.h +1 -19
@@ -551,25 +551,7 @@ struct alt_instr { 
 
 #define set_wmb(var, value) do { var = value; wmb(); } while (0)
 
-/* interrupt control.. */
-#define local_save_flags(x)	do { typecheck(unsigned long,x); __asm__ __volatile__("pushfl ; popl %0":"=g" (x): /* no input */); } while (0)
-#define local_irq_restore(x) 	do { typecheck(unsigned long,x); __asm__ __volatile__("pushl %0 ; popfl": /* no output */ :"g" (x):"memory", "cc"); } while (0)
-#define local_irq_disable() 	__asm__ __volatile__("cli": : :"memory")
-#define local_irq_enable()	__asm__ __volatile__("sti": : :"memory")
-/* used in the idle loop; sti takes one instruction cycle to complete */
-#define safe_halt()		__asm__ __volatile__("sti; hlt": : :"memory")
-/* used when interrupts are already enabled or to shutdown the processor */
-#define halt()			__asm__ __volatile__("hlt": : :"memory")
-
-#define irqs_disabled()			\
-({					\
-	unsigned long flags;		\
-	local_save_flags(flags);	\
-	!(flags & (1<<9));		\
-})
-
-/* For spinlocks etc */
-#define local_irq_save(x)	__asm__ __volatile__("pushfl ; popl %0 ; cli":"=g" (x): /* no input */ :"memory")
+#include <mach_system.h>
 
 /*
  * disable hlt during certain critical i/o operations
include/asm-i386/mach-default/mach_system.h new file +24 -0
@@ -0,0 +1,24 @@
+#ifndef __ASM_MACH_SYSTEM_H
+#define __ASM_MACH_SYSTEM_H
+
+/* interrupt control.. */
+#define local_save_flags(x)	do { typecheck(unsigned long,x); __asm__ __volatile__("pushfl ; popl %0":"=g" (x): /* no input */); } while (0)
+#define local_irq_restore(x) 	do { typecheck(unsigned long,x); __asm__ __volatile__("pushl %0 ; popfl": /* no output */ :"g" (x):"memory", "cc"); } while (0)
+#define local_irq_disable() 	__asm__ __volatile__("cli": : :"memory")
+#define local_irq_enable()	__asm__ __volatile__("sti": : :"memory")
+/* used in the idle loop; sti takes one instruction cycle to complete */
+#define safe_halt()		__asm__ __volatile__("sti; hlt": : :"memory")
+/* used when interrupts are already enabled or to shutdown the processor */
+#define halt()			__asm__ __volatile__("hlt": : :"memory")
+
+#define irqs_disabled()			\
+({					\
+	unsigned long flags;		\
+	local_save_flags(flags);	\
+	!(flags & (1<<9));		\
+})
+
+/* For spinlocks etc */
+#define local_irq_save(x)	__asm__ __volatile__("pushfl ; popl %0 ; cli":"=g" (x): /* no input */ :"memory")
+
+#endif /* __ASM_MACH_SYSTEM_H */
include/asm-i386/mach-xen/mach_system.h new file +103 -0

103 lines not on this page · raw message

--

#41 lore raw

RFC PATCH 16/35 subarch support for interrupt and exception gates

In reply to Chris Wright

Abstract the code that sets up interrupt and exception gates, and add a separate subarch implementation for Xen.

---
 arch/i386/kernel/traps.c                   |   49 ---------------------------
 include/asm-i386/mach-default/mach_idt.h   |   52 +++++++++++++++++++++++++++++
 include/asm-i386/mach-xen/mach_idt.h       |   50 +++++++++++++++++++++++++++
 include/asm-i386/mach-xen/setup_arch_pre.h |    2 +
 4 files changed, 105 insertions(+), 48 deletions(-)
arch/i386/kernel/traps.c +1 -48
@@ -1043,54 +1043,7 @@ void __init trap_init_f00f_bug(void)
 }
 #endif
 
-#define _set_gate(gate_addr,type,dpl,addr,seg) \
-do { \
-  int __d0, __d1; \
-  __asm__ __volatile__ ("movw %%dx,%%ax\n\t" \
-	"movw %4,%%dx\n\t" \
-	"movl %%eax,%0\n\t" \
-	"movl %%edx,%1" \
-	:"=m" (*((long *) (gate_addr))), \
-	 "=m" (*(1+(long *) (gate_addr))), "=&a" (__d0), "=&d" (__d1) \
-	:"i" ((short) (0x8000+(dpl<<13)+(type<<8))), \
-	 "3" ((char *) (addr)),"2" ((seg) << 16)); \
-} while (0)
-
-
-/*
- * This needs to use 'idt_table' rather than 'idt', and
- * thus use the _nonmapped_ version of the IDT, as the
- * Pentium F0 0F bugfix can have resulted in the mapped
- * IDT being write-protected.
- */
-void set_intr_gate(unsigned int n, void *addr)
-{
-	_set_gate(idt_table+n,14,0,addr,__KERNEL_CS);
-}
-
-/*
- * This routine sets up an interrupt gate at directory privilege level 3.
- */
-static inline void set_system_intr_gate(unsigned int n, void *addr)
-{
-	_set_gate(idt_table+n, 14, 3, addr, __KERNEL_CS);
-}
-
-static void __init set_trap_gate(unsigned int n, void *addr)
-{
-	_set_gate(idt_table+n,15,0,addr,__KERNEL_CS);
-}
-
-static void __init set_system_gate(unsigned int n, void *addr)
-{
-	_set_gate(idt_table+n,15,3,addr,__KERNEL_CS);
-}
-
-static void __init set_task_gate(unsigned int n, unsigned int gdt_entry)
-{
-	_set_gate(idt_table+n,5,0,0,(gdt_entry<<3));
-}
-
+#include <mach_idt.h>
 
 void __init trap_init(void)
 {
include/asm-i386/mach-xen/setup_arch_pre.h +2 -0
@@ -6,6 +6,8 @@
 struct start_info *xen_start_info;
 EXPORT_SYMBOL(xen_start_info);
 
+struct trap_info xen_trap_table[257];
+
 /*
  * Point at the empty zero page to start with. We map the real shared_info
  * page as soon as fixmap is up and running.
include/asm-i386/mach-default/mach_idt.h new file +52 -0

52 lines not on this page · raw message

include/asm-i386/mach-xen/mach_idt.h new file +50 -0

50 lines not on this page · raw message

--

#42 lore raw

Re: [RFC PATCH 16/35] subarch support for interrupt and exception gates

Quoted 2 lines · Chris Wright

Abstract the code that sets up interrupt and exception gates, and add a separate subarch implementation for Xen.

AFAIK the only difference is that Xen uses a table of them to pass the hypervisor and normal Linux calls the macros directly, right?

I would suggest you just use the table for normal Linux too
and make the function that processes them natively !CONFIG_XEN I guess it will make the code smaller for the normal case and people happy.

That would be much cleaner than just separating it out.

-Andi

#43 lore raw

Re: [RFC PATCH 16/35] subarch support for interrupt and exception gates

Quoted 6 lines · * Andi Kleen ([email protected]) wrote:
Quoted 2 lines · Chris Wright

Abstract the code that sets up interrupt and exception gates, and add a separate subarch implementation for Xen.

AFAIK the only difference is that Xen uses a table of them to pass the hypervisor and normal Linux calls the macros directly, right?

Yes.

Quoted 3 lines

I would suggest you just use the table for normal Linux too
and make the function that processes them natively !CONFIG_XEN I guess it will make the code smaller for the normal case and people happy.

Hadn't considered that type of consolidation. So load_idt() would generally be one step further from lidt insn on native, which shouldn't be a problem. Looks like kexec would need to be taught about building the table.

thanks,
-chris

#44 lore raw

RFC PATCH 17/35 Segment register changes for Xen

In reply to Chris Wright

1. We clear FS/GS before changing TLS entries and switching LDT, as otherwise the hypervisor will fail to restore thread-local values on return to the guest kernel and we take a slow exception path.

2. We allow for the fact that the guest kernel may not run in ring 0. This requires some abstraction in a few places when setting %cs or checking privilege level (user vs kernel).

---
 arch/i386/kernel/process.c                   |    4 +++-
 arch/i386/mm/fault.c                         |    8 +++++---
 include/asm-i386/mach-default/mach_segment.h |    8 ++++++++
 include/asm-i386/mach-default/mach_system.h  |    2 ++
 include/asm-i386/mach-xen/mach_segment.h     |    9 +++++++++
 include/asm-i386/mach-xen/mach_system.h      |    2 ++
 include/asm-i386/mach-xen/setup_arch_post.h  |    2 ++
 include/asm-i386/ptrace.h                    |    6 ++++--
 include/asm-i386/segment.h                   |    2 ++
 9 files changed, 37 insertions(+), 6 deletions(-)
arch/i386/kernel/process.c +3 -1
@@ -348,7 +348,7 @@ int kernel_thread(int (*fn)(void *), voi
 	regs.xes = __USER_DS;
 	regs.orig_eax = -1;
 	regs.eip = (unsigned long) kernel_thread_helper;
-	regs.xcs = __KERNEL_CS;
+	regs.xcs = get_kernel_cs();
 	regs.eflags = X86_EFLAGS_IF | X86_EFLAGS_SF | X86_EFLAGS_PF | 0x2;
 
 	/* Ok, create the new process.. */
@@ -655,6 +655,8 @@ struct task_struct fastcall * __switch_t
 	 */
 	savesegment(fs, prev->fs);
 	savesegment(gs, prev->gs);
+	clearsegment(fs);
+	clearsegment(gs);
 
 	/*
 	 * Load the per-thread Thread-Local Storage descriptor.
arch/i386/mm/fault.c +5 -3
@@ -28,6 +28,8 @@
 #include <asm/desc.h>
 #include <asm/kdebug.h>
 
+#include <mach_segment.h>
+
 extern void die(const char *,struct pt_regs *,long);
 
 /*
@@ -78,14 +80,14 @@ static inline unsigned long get_segment_
 	u32 seg_ar, seg_limit, base, *desc;
 
 	/* The standard kernel/user address space limit. */
-	*eip_limit = (seg & 3) ? USER_DS.seg : KERNEL_DS.seg;
+	*eip_limit = (seg & USER_MODE_MASK) ? USER_DS.seg : KERNEL_DS.seg;
 
 	/* Unlikely, but must come before segment checks. */
 	if (unlikely((regs->eflags & VM_MASK) != 0))
 		return eip + (seg << 4);
 	
 	/* By far the most common cases. */
-	if (likely(seg == __USER_CS || seg == __KERNEL_CS))
+	if (likely(seg == __USER_CS || seg == get_kernel_cs()))
 		return eip;
 
 	/* Check the segment exists, is within the current LDT/GDT size,
@@ -331,7 +333,7 @@ good_area:
 	switch (error_code & 3) {
 		default:	/* 3: write, present */
 #ifdef TEST_VERIFY_AREA
-			if (regs->cs == KERNEL_CS)
+			if (regs->cs == get_kernel_cs())
 				printk("WP fault at %08lx\n", regs->eip);
 #endif
 			/* fall through */
include/asm-i386/mach-default/mach_system.h +2 -0
@@ -1,6 +1,8 @@
 #ifndef __ASM_MACH_SYSTEM_H
 #define __ASM_MACH_SYSTEM_H
 
+#define clearsegment(seg)
+
 /* interrupt control.. */
 #define local_save_flags(x)	do { typecheck(unsigned long,x); __asm__ __volatile__("pushfl ; popl %0":"=g" (x): /* no input */); } while (0)
 #define local_irq_restore(x) 	do { typecheck(unsigned long,x); __asm__ __volatile__("pushl %0 ; popfl": /* no output */ :"g" (x):"memory", "cc"); } while (0)
include/asm-i386/mach-xen/mach_system.h +2 -0
@@ -5,6 +5,8 @@
 
 #include <asm/hypervisor.h>
 
+#define clearsegment(seg) loadsegment(seg, 0)
+
 #ifdef CONFIG_SMP
 #define __vcpu_id smp_processor_id()
 #else
include/asm-i386/mach-xen/setup_arch_post.h +2 -0
@@ -39,6 +39,8 @@ static void __init machine_specific_arch
 		(struct shared_info *)__va(xen_start_info->shared_info);
 	memset(empty_zero_page, 0, sizeof(empty_zero_page));
 
+	HYPERVISOR_vm_assist(VMASST_CMD_enable, VMASST_TYPE_4gb_segments);
+
 	HYPERVISOR_set_callbacks(
 	    __KERNEL_CS, (unsigned long)hypervisor_callback,
 	    __KERNEL_CS, (unsigned long)failsafe_callback);
include/asm-i386/ptrace.h +4 -2
@@ -1,6 +1,8 @@
 #ifndef _I386_PTRACE_H
 #define _I386_PTRACE_H
 
+#include <mach_segment.h>
+
 #define EBX 0
 #define ECX 1
 #define EDX 2
@@ -73,11 +75,11 @@ extern void send_sigtrap(struct task_str
  */
 static inline int user_mode(struct pt_regs *regs)
 {
-	return (regs->xcs & 3) != 0;
+	return (regs->xcs & USER_MODE_MASK) != 0;
 }
 static inline int user_mode_vm(struct pt_regs *regs)
 {
-	return ((regs->xcs & 3) | (regs->eflags & VM_MASK)) != 0;
+	return ((regs->xcs & USER_MODE_MASK) | (regs->eflags & VM_MASK)) != 0;
 }
 #define instruction_pointer(regs) ((regs)->eip)
 #if defined(CONFIG_SMP) && defined(CONFIG_FRAME_POINTER)
include/asm-i386/segment.h +2 -0
@@ -1,6 +1,8 @@
 #ifndef _ASM_SEGMENT_H
 #define _ASM_SEGMENT_H
 
+#include <mach_segment.h>
+
 /*
  * The layout of the per-CPU GDT under Linux:
  *
include/asm-i386/mach-default/mach_segment.h new file +8 -0
@@ -0,0 +1,8 @@
+#ifndef __ASM_MACH_SEGMENT_H
+#define __ASM_MACH_SEGMENT_H
+
+#define USER_MODE_MASK 3
+
+#define get_kernel_cs() __KERNEL_CS
+
+#endif /* __ASM_MACH_SEGMENT_H */
include/asm-i386/mach-xen/mach_segment.h new file +9 -0
@@ -0,0 +1,9 @@
+#ifndef __ASM_MACH_SEGMENT_H
+#define __ASM_MACH_SEGMENT_H
+
+#define USER_MODE_MASK 2
+
+#define get_kernel_cs() \
+	(__KERNEL_CS + (xen_feature(XENFEAT_supervisor_mode_kernel) ? 0 : 1))
+
+#endif /* __ASM_MACH_SEGMENT_H */

--

#45 lore raw

Re: [RFC PATCH 17/35] Segment register changes for Xen

include/asm-i386/mach-xen/mach_segment.h line 7

Quoted 2 lines · Chris Wright
include/asm-i386/mach-xen/mach_segment.h new file +2 -0
+#define get_kernel_cs() \
+	(__KERNEL_CS + (xen_feature(XENFEAT_supervisor_mode_kernel) ? 0 : 1))

Under what circumstances is that feature set?

-Andi

#46 lore raw

Re: [RFC PATCH 17/35] Segment register changes for Xen

Quoted 6 lines · * Andi Kleen ([email protected]) wrote:

include/asm-i386/mach-xen/mach_segment.h line 7

Quoted 2 lines · Chris Wright
include/asm-i386/mach-xen/mach_segment.h new file +2 -0
+#define get_kernel_cs() \
+	(__KERNEL_CS + (xen_feature(XENFEAT_supervisor_mode_kernel) ? 0 : 1))

Under what circumstances is that feature set?

When building hypervisor with that support. It then builds a different hypercall_page for the guest to use, and both co-exist in ring0. It's for running directly on native.

thanks,
-chris

#47 lore raw

Re: [RFC PATCH 17/35] Segment register changes for Xen

In reply to Chris Wright
Quoted 4 lines · Chris Wright

1. We clear FS/GS before changing TLS entries and switching LDT, as otherwise the hypervisor will fail to restore thread-local values on return to the guest kernel and we take a slow exception path.

This should not be needed. You should clear FS/GS that match updated descriptors in your descriptor validation code instead. And you will get better performance for both native and the hypervisor cases.

#48 lore raw

RFC PATCH 18/35 Support gdt/idt/ldt handling on Xen.

In reply to Chris Wright

Move the macros which handle gdt/idt/ldt's into a subarch include file and add implementations for running on Xen.

---
 include/asm-i386/desc.h                   |   65 ++--------------------------
 include/asm-i386/mach-default/mach_desc.h |   67 +++++++++++++++++++++++++++++
 include/asm-i386/mach-xen/mach_desc.h     |   69 ++++++++++++++++++++++++++++++
 3 files changed, 141 insertions(+), 60 deletions(-)
include/asm-i386/desc.h +5 -60
@@ -33,18 +33,7 @@ static inline struct desc_struct *get_cp
 	return (struct desc_struct *)per_cpu(cpu_gdt_descr, cpu).address;
 }
 
-#define load_TR_desc() __asm__ __volatile__("ltr %w0"::"q" (GDT_ENTRY_TSS*8))
-#define load_LDT_desc() __asm__ __volatile__("lldt %w0"::"q" (GDT_ENTRY_LDT*8))
-
-#define load_gdt(dtr) __asm__ __volatile("lgdt %0"::"m" (*dtr))
-#define load_idt(dtr) __asm__ __volatile("lidt %0"::"m" (*dtr))
-#define load_tr(tr) __asm__ __volatile("ltr %0"::"mr" (tr))
-#define load_ldt(ldt) __asm__ __volatile("lldt %0"::"mr" (ldt))
-
-#define store_gdt(dtr) __asm__ ("sgdt %0":"=m" (*dtr))
-#define store_idt(dtr) __asm__ ("sidt %0":"=m" (*dtr))
-#define store_tr(tr) __asm__ ("str %0":"=mr" (tr))
-#define store_ldt(ldt) __asm__ ("sldt %0":"=mr" (ldt))
+#include <mach_desc.h>
 
 /*
  * This is the ldt that every process will get unless we need
@@ -53,30 +42,6 @@ static inline struct desc_struct *get_cp
 extern struct desc_struct default_ldt[];
 extern void set_intr_gate(unsigned int irq, void * addr);
 
-#define _set_tssldt_desc(n,addr,limit,type) \
-__asm__ __volatile__ ("movw %w3,0(%2)\n\t" \
-	"movw %w1,2(%2)\n\t" \
-	"rorl $16,%1\n\t" \
-	"movb %b1,4(%2)\n\t" \
-	"movb %4,5(%2)\n\t" \
-	"movb $0,6(%2)\n\t" \
-	"movb %h1,7(%2)\n\t" \
-	"rorl $16,%1" \
-	: "=m"(*(n)) : "q" (addr), "r"(n), "ir"(limit), "i"(type))
-
-static inline void __set_tss_desc(unsigned int cpu, unsigned int entry, void *addr)
-{
-	_set_tssldt_desc(&get_cpu_gdt_table(cpu)[entry], (int)addr,
-		offsetof(struct tss_struct, __cacheline_filler) - 1, 0x89);
-}
-
-#define set_tss_desc(cpu,addr) __set_tss_desc(cpu, GDT_ENTRY_TSS, addr)
-
-static inline void set_ldt_desc(unsigned int cpu, void *addr, unsigned int size)
-{
-	_set_tssldt_desc(&get_cpu_gdt_table(cpu)[GDT_ENTRY_LDT], (int)addr, ((size << 3)-1), 0x82);
-}
-
 #define LDT_entry_a(info) \
 	((((info)->base_addr & 0x0000ffff) << 16) | ((info)->limit & 0x0ffff))
 
@@ -102,30 +67,11 @@ static inline void set_ldt_desc(unsigned
 	(info)->seg_not_present	== 1	&& \
 	(info)->useable		== 0	)
 
-static inline void write_ldt_entry(void *ldt, int entry, __u32 entry_a, __u32 entry_b)
-{
-	__u32 *lp = (__u32 *)((char *)ldt + entry*8);
-	*lp = entry_a;
-	*(lp+1) = entry_b;
-}
-
-#if TLS_SIZE != 24
-# error update this code.
-#endif
-
-static inline void load_TLS(struct thread_struct *t, unsigned int cpu)
-{
-#define C(i) get_cpu_gdt_table(cpu)[GDT_ENTRY_TLS_MIN + i] = t->tls_array[i]
-	C(0); C(1); C(2);
-#undef C
-}
-
 static inline void clear_LDT(void)
 {
 	int cpu = get_cpu();
 
-	set_ldt_desc(cpu, &default_ldt[0], 5);
-	load_LDT_desc();
+	__set_ldt(cpu, DEFAULT_LDT, DEFAULT_LDT_SIZE);
 	put_cpu();
 }
 
@@ -138,12 +84,11 @@ static inline void load_LDT_nolock(mm_co
 	int count = pc->size;
 
 	if (likely(!count)) {
-		segments = &default_ldt[0];
-		count = 5;
+		segments = DEFAULT_LDT;
+		count = DEFAULT_LDT_SIZE;
 	}
 		
-	set_ldt_desc(cpu, segments, count);
-	load_LDT_desc();
+	__set_ldt(cpu, segments, count);
 }
 
 static inline void load_LDT(mm_context_t *pc)
include/asm-i386/mach-default/mach_desc.h new file +67 -0

67 lines not on this page · raw message

include/asm-i386/mach-xen/mach_desc.h new file +69 -0

69 lines not on this page · raw message

--

#49 lore raw

Re: [RFC PATCH 18/35] Support gdt/idt/ldt handling on Xen.

include/asm-i386/desc.h static inline struct desc_struct *get_cp · line 47

Quoted 13 lines · Chris Wright
include/asm-i386/desc.h +0 -12
-#define load_TR_desc() __asm__ __volatile__("ltr %w0"::"q" (GDT_ENTRY_TSS*8))
-#define load_LDT_desc() __asm__ __volatile__("lldt %w0"::"q" (GDT_ENTRY_LDT*8))
-
-#define load_gdt(dtr) __asm__ __volatile("lgdt %0"::"m" (*dtr))
-#define load_idt(dtr) __asm__ __volatile("lidt %0"::"m" (*dtr))
-#define load_tr(tr) __asm__ __volatile("ltr %0"::"mr" (tr))
-#define load_ldt(ldt) __asm__ __volatile("lldt %0"::"mr" (ldt))
-
-#define store_gdt(dtr) __asm__ ("sgdt %0":"=m" (*dtr))
-#define store_idt(dtr) __asm__ ("sidt %0":"=m" (*dtr))
-#define store_tr(tr) __asm__ ("str %0":"=mr" (tr))
-#define store_ldt(ldt) __asm__ ("sldt %0":"=mr" (ldt))

These are all very infrequent except perhaps LLDT. I suspect trapping would work too. But ok.

Quoted 1 line

-#define _set_tssldt_desc(n,addr,limit,type) \

[...]

Why are you moving these? Xen should just be parsing the same structures as the hardware, shouldn't it?

-Andi

#50 lore raw

Re: [Xen-devel] Re: [RFC PATCH 18/35] Support gdt/idt/ldt handling on Xen.

Quoted 4 lines · Andi Kleen
Quoted 1 line · Chris Wright
Quoted 17 lines

include/asm-i386/desc.h static inline struct desc_struct *get_cp · line 47

Quoted 13 lines
include/asm-i386/desc.h +0 -12
-#define load_TR_desc() __asm__ __volatile__("ltr %w0"::"q" (GDT_ENTRY_TSS*8))
-#define load_LDT_desc() __asm__ __volatile__("lldt %w0"::"q" (GDT_ENTRY_LDT*8))
-
-#define load_gdt(dtr) __asm__ __volatile("lgdt %0"::"m" (*dtr))
-#define load_idt(dtr) __asm__ __volatile("lidt %0"::"m" (*dtr))
-#define load_tr(tr) __asm__ __volatile("ltr %0"::"mr" (tr))
-#define load_ldt(ldt) __asm__ __volatile("lldt %0"::"mr" (ldt))
-
-#define store_gdt(dtr) __asm__ ("sgdt %0":"=m" (*dtr))
-#define store_idt(dtr) __asm__ ("sidt %0":"=m" (*dtr))
-#define store_tr(tr) __asm__ ("str %0":"=mr" (tr))
-#define store_ldt(ldt) __asm__ ("sldt %0":"=mr" (ldt))

These are all very infrequent except perhaps LLDT. I suspect trapping would work too. But ok.

Yes, trapping works fine. Even LLDT is infrequent. But, you do impose a very large amount of complexity on the hypervisor by trapping on any instruction. Suppose you wanted to write a minimal hypervisor, which consisted of pretty much a wrapper based on the Xen or VMI interface that just stole a couple pages of physical memory, and hooked trap handlers by hooking the call out for lidt.

If you instead remove the hypervisor wrapper for lidt, and require the hypervisor to trap and emulate it, you have just imposed an insidious amount of overhead on it. It doesn't seem too much at first - trap and emulate, right?

No. First, you have to create a special #GP handler for the general protection fault. But the fault doesn't tell you anything about why it happened - just that it was a general protection fault, and maybe a segment related to it. To figure out what happened, you have to decode the instruction stream. To decode the instruction stream, you have to take the EIP pointer and read from it, right? Wrong. You have to extract segment information from the code segment, apply segmentation rules to the access, rule out invalid processor modes, deal with wrap around conditions, etc. But lets say you do all that. Now, you have to read the guest memory to decode. Which requires reading guest page tables. The memory in question has to be mapped present and executable. You have to deal conditionally with PAE / non-PAE paging modes. And race conditions where self-modifying code can trick you into decoding something that really didn't happen. Then, finally, you can interpret the instruction, go through the whole process of reading guest memory again (fortunately, this time, without segmentation), read the guest IDT, and hook in your trap handlers where appropriate.

Yeah, it really is that bad, and that is really only a scratch on the surface. Trap and emulate, while possible, is basically about as evil a requirement as you can impose on a hypervisor. Everyone who is serious about the market needs to support it in one form or another eventually, but it raises the bar to such a point as to stop proliferation of minimal hypervisors, which could make extremely useful research tools for the community under an open source license.

Zach

#51 lore raw

Re: [Xen-devel] Re: [RFC PATCH 18/35] Support gdt/idt/ldt handling on Xen.

Quoted 2 lines · Zachary Amsden

Yes, trapping works fine. Even LLDT is infrequent.

Not when you use old style LinuxThreads which use the LDT for TLS.

Quoted 2 lines

No. First, you have to create a special #GP handler for the general protection fault.

[... etc ...]

Sure but Xen already has the infrastructure for all of this and last time I checked it was approaching and exceeding the size of the main core kernel so a bit more of instruction emulation probably wouldn't do too much harm.

In general I think any x86 hypervisor that attempts to work on current platforms needs instruction emulation because it is the only way to virtualize IO devices.

If this was supposed to be a interface for lots of hypervisors then maybe, but so far it seems to only cover Xen and possibly some other bloatware ones.

That said I don't feel very strongly about emulating these instructions or not as long as they can do that without too much code duplication. The current patch are still a bit too excessive on the duplication front.

-Andi

#52 lore raw

RFC PATCH 19/35 subarch support for control register accesses

In reply to Chris Wright

Abstract the code that accesses control register, and add a separate subarch implementation for Xen.

---
 include/asm-i386/mach-default/mach_system.h |   58 ++++++++++++++++++++++++++++
 include/asm-i386/mach-xen/mach_system.h     |   53 +++++++++++++++++++++++++
 include/asm-i386/system.h                   |   58 ----------------------------
 3 files changed, 111 insertions(+), 58 deletions(-)
include/asm-i386/mach-default/mach_system.h +58 -0
@@ -3,6 +3,64 @@
 
 #define clearsegment(seg)
 
+/*
+ * Clear and set 'TS' bit respectively
+ */
+#define clts() __asm__ __volatile__ ("clts")
+#define read_cr0() ({ \
+	unsigned int __dummy; \
+	__asm__ __volatile__( \
+		"movl %%cr0,%0\n\t" \
+		:"=r" (__dummy)); \
+	__dummy; \
+})
+#define write_cr0(x) \
+	__asm__ __volatile__("movl %0,%%cr0": :"r" (x));
+
+#define read_cr2() ({ \
+	unsigned int __dummy; \
+	__asm__ __volatile__( \
+		"movl %%cr2,%0\n\t" \
+		:"=r" (__dummy)); \
+	__dummy; \
+})
+#define write_cr2(x) \
+	__asm__ __volatile__("movl %0,%%cr2": :"r" (x));
+
+#define read_cr3() ({ \
+	unsigned int __dummy; \
+	__asm__ ( \
+		"movl %%cr3,%0\n\t" \
+		:"=r" (__dummy)); \
+	__dummy; \
+})
+#define write_cr3(x) \
+	__asm__ __volatile__("movl %0,%%cr3": :"r" (x));
+
+#define read_cr4() ({ \
+	unsigned int __dummy; \
+	__asm__( \
+		"movl %%cr4,%0\n\t" \
+		:"=r" (__dummy)); \
+	__dummy; \
+})
+
+#define read_cr4_safe() ({			      \
+	unsigned int __dummy;			      \
+	/* This could fault if %cr4 does not exist */ \
+	__asm__("1: movl %%cr4, %0		\n"   \
+		"2:				\n"   \
+		".section __ex_table,\"a\"	\n"   \
+		".long 1b,2b			\n"   \
+		".previous			\n"   \
+		: "=r" (__dummy): "0" (0));	      \
+	__dummy;				      \
+})
+
+#define write_cr4(x) \
+	__asm__ __volatile__("movl %0,%%cr4": :"r" (x));
+#define stts() write_cr0(8 | read_cr0())
+
 /* interrupt control.. */
 #define local_save_flags(x)	do { typecheck(unsigned long,x); __asm__ __volatile__("pushfl ; popl %0":"=g" (x): /* no input */); } while (0)
 #define local_irq_restore(x) 	do { typecheck(unsigned long,x); __asm__ __volatile__("pushl %0 ; popfl": /* no output */ :"g" (x):"memory", "cc"); } while (0)
include/asm-i386/mach-xen/mach_system.h +53 -0

59 lines not on this page · raw message

include/asm-i386/system.h +0 -58

64 lines not on this page · raw message

--

#53 lore raw

Re: [RFC PATCH 19/35] subarch support for control register accesses

include/asm-i386/mach-default/mach_system.h line 58

Quoted 11 lines · Chris Wright
include/asm-i386/mach-default/mach_system.h +11 -0
+#define read_cr4_safe() ({			      \
+	unsigned int __dummy;			      \
+	/* This could fault if %cr4 does not exist */ \
+	__asm__("1: movl %%cr4, %0		\n"   \
+		"2:				\n"   \
+		".section __ex_table,\"a\"	\n"   \
+		".long 1b,2b			\n"   \
+		".previous			\n"   \
+		: "=r" (__dummy): "0" (0));	      \
+	__dummy;				      \
+})

I think you'll find trap and emulate quite sufficient for this one.

Quoted 2 lines

+#define stts() write_cr0(8 | read_cr0())

Nit: You shouldn't need to redefine stts() in the subarch.

#54 lore raw

Re: [RFC PATCH 19/35] subarch support for control register accesses

Quoted 14 lines · * Zachary Amsden ([email protected]) wrote:

include/asm-i386/mach-default/mach_system.h line 58

Quoted 11 lines · Chris Wright
include/asm-i386/mach-default/mach_system.h +11 -0
+#define read_cr4_safe() ({			      \
+	unsigned int __dummy;			      \
+	/* This could fault if %cr4 does not exist */ \
+	__asm__("1: movl %%cr4, %0		\n"   \
+		"2:				\n"   \
+		".section __ex_table,\"a\"	\n"   \
+		".long 1b,2b			\n"   \
+		".previous			\n"   \
+		: "=r" (__dummy): "0" (0));	      \
+	__dummy;				      \
+})

I think you'll find trap and emulate quite sufficient for this one.

Heh ;-)

Quoted 4 lines
Quoted 2 lines

+#define stts() write_cr0(8 | read_cr0())

Nit: You shouldn't need to redefine stts() in the subarch.

Yes, you're right, better to keep things consolidated.

#55 lore raw

RFC PATCH 20/35 subarch stack pointer update

In reply to Chris Wright

Register the new kernel ('ring 0') stack pointer with the hypervisor during context switch.

---
 include/asm-i386/mach-default/mach_processor.h |    8 ++++++++
 include/asm-i386/mach-xen/mach_processor.h     |    9 +++++++++
 include/asm-i386/processor.h                   |    3 +++
 3 files changed, 20 insertions(+)
include/asm-i386/processor.h +3 -0
@@ -472,6 +472,8 @@ struct thread_struct {
 	.io_bitmap_ptr = NULL,						\
 }
 
+#include <mach_processor.h>
+
 /*
  * Note that the .io_bitmap member must be extra-big. This is because
  * the CPU will access an additional byte beyond the end of the IO
@@ -494,6 +496,7 @@ static inline void load_esp0(struct tss_
 		tss->ss1 = thread->sysenter_cs;
 		wrmsr(MSR_IA32_SYSENTER_CS, thread->sysenter_cs, 0);
 	}
+	mach_load_esp0(tss, thread);
 }
 
 #define start_thread(regs, new_eip, new_esp) do {		\
include/asm-i386/mach-xen/mach_processor.h new file +9 -0
@@ -0,0 +1,9 @@
+#ifndef __ASM_MACH_PROCESSOR_H
+#define __ASM_MACH_PROCESSOR_H
+
+static inline void mach_load_esp0(struct tss_struct *tss, struct thread_struct *thread)
+{
+	HYPERVISOR_stack_switch(tss->ss0, tss->esp0);
+}
+
+#endif /* __ASM_MACH_PROCESSOR_H */
include/asm-i386/mach-default/mach_processor.h new file +8 -0
@@ -0,0 +1,8 @@
+#ifndef __ASM_MACH_PROCESSOR_H
+#define __ASM_MACH_PROCESSOR_H
+
+static inline void mach_load_esp0(struct tss_struct *tss, struct thread_struct *thread)
+{
+}
+
+#endif /* __ASM_MACH_PROCESSOR_H */

--

#56 lore raw

RFC PATCH 21/35 subarch TLB support

In reply to Chris Wright

Paravirtualize TLB flushes by using the flush interfaces provided by the hypervisor. These hide the details of cross-CPU shootdowns and allow significant optimisations (for example, by avoiding shooting down on virtual CPUs that are descheduled). This is considerably faster in most cases than performing virtual IPIs in the guest kernel.

---
 include/asm-i386/mach-default/mach_tlbflush.h |   59 ++++++++++++++++++++++++++
 include/asm-i386/mach-xen/mach_tlbflush.h     |   25 +++++++++++
 include/asm-i386/tlbflush.h                   |   55 ------------------------
 3 files changed, 85 insertions(+), 54 deletions(-)
include/asm-i386/tlbflush.h +1 -54
@@ -5,64 +5,11 @@
 #include <linux/mm.h>
 #include <asm/processor.h>
 
-#define __flush_tlb()							\
-	do {								\
-		unsigned int tmpreg;					\
-									\
-		__asm__ __volatile__(					\
-			"movl %%cr3, %0;              \n"		\
-			"movl %0, %%cr3;  # flush TLB \n"		\
-			: "=r" (tmpreg)					\
-			:: "memory");					\
-	} while (0)
-
-/*
- * Global pages have to be flushed a bit differently. Not a real
- * performance problem because this does not happen often.
- */
-#define __flush_tlb_global()						\
-	do {								\
-		unsigned int tmpreg, cr4, cr4_orig;			\
-									\
-		__asm__ __volatile__(					\
-			"movl %%cr4, %2;  # turn off PGE     \n"	\
-			"movl %2, %1;                        \n"	\
-			"andl %3, %1;                        \n"	\
-			"movl %1, %%cr4;                     \n"	\
-			"movl %%cr3, %0;                     \n"	\
-			"movl %0, %%cr3;  # flush TLB        \n"	\
-			"movl %2, %%cr4;  # turn PGE back on \n"	\
-			: "=&r" (tmpreg), "=&r" (cr4), "=&r" (cr4_orig)	\
-			: "i" (~X86_CR4_PGE)				\
-			: "memory");					\
-	} while (0)
-
 extern unsigned long pgkern_mask;
 
-# define __flush_tlb_all()						\
-	do {								\
-		if (cpu_has_pge)					\
-			__flush_tlb_global();				\
-		else							\
-			__flush_tlb();					\
-	} while (0)
-
 #define cpu_has_invlpg	(boot_cpu_data.x86 > 3)
 
-#define __flush_tlb_single(addr) \
-	__asm__ __volatile__("invlpg %0": :"m" (*(char *) addr))
-
-#ifdef CONFIG_X86_INVLPG
-# define __flush_tlb_one(addr) __flush_tlb_single(addr)
-#else
-# define __flush_tlb_one(addr)						\
-	do {								\
-		if (cpu_has_invlpg)					\
-			__flush_tlb_single(addr);			\
-		else							\
-			__flush_tlb();					\
-	} while (0)
-#endif
+#include <mach_tlbflush.h>
 
 /*
  * TLB flushing:
include/asm-i386/mach-default/mach_tlbflush.h new file +59 -0

59 lines not on this page · raw message

include/asm-i386/mach-xen/mach_tlbflush.h new file +25 -0
@@ -0,0 +1,25 @@
+#ifndef __ASM_MACH_TLBFLUSH_H
+#define __ASM_MACH_TLBFLUSH_H
+
+static inline void xen_tlb_flush(void)
+{
+        struct mmuext_op op;
+        op.cmd = MMUEXT_TLB_FLUSH_LOCAL;
+        BUG_ON(HYPERVISOR_mmuext_op(&op, 1, NULL, DOMID_SELF) < 0);
+}
+
+static inline void xen_invlpg(unsigned long ptr)
+{
+        struct mmuext_op op;
+        op.cmd = MMUEXT_INVLPG_LOCAL;
+        op.arg1.linear_addr = ptr & PAGE_MASK;
+        BUG_ON(HYPERVISOR_mmuext_op(&op, 1, NULL, DOMID_SELF) < 0);
+}
+
+#define __flush_tlb() xen_tlb_flush()
+#define __flush_tlb_global() xen_tlb_flush()
+#define __flush_tlb_all() xen_tlb_flush()
+#define __flush_tlb_single(addr) xen_invlpg(addr)
+#define __flush_tlb_one(addr) __flush_tlb_single(addr)
+
+#endif /* __ASM_MACH_TLBFLUSH_H */

--

#57 lore raw

RFC PATCH 22/35 subarch suport for idle loop (NO_IDLE_HZ for Xen)

In reply to Chris Wright

Paravirtualize the idle loop to explicitly trap to the hypervisor when blocking, and to use the NO_IDLE_HZ functionality introduced by s390 to inform the rcu subsystem that the CPU is quiescent.

---
 drivers/xen/Kconfig                         |    8 ++++++++
 include/asm-i386/mach-xen/setup_arch_post.h |   24 ++++++++++++++++++++++++
 2 files changed, 32 insertions(+)
drivers/xen/Kconfig +8 -0
@@ -12,6 +12,14 @@ config XEN
 
 if XEN
 
+config NO_IDLE_HZ
+	bool
+	default y
+	help
+	  Switches the regular HZ timer off when the system is going idle.
+	  This helps Xen to detect that the Linux system is idle, reducing
+	  the overhead of idle systems.
+
 config XEN_SHADOW_MODE
 	bool
 	default y
include/asm-i386/mach-xen/setup_arch_post.h +24 -0
@@ -8,6 +8,11 @@
 
 #include <xen/interface/physdev.h>
 
+extern void stop_hz_timer(void);
+extern void start_hz_timer(void);
+
+void xen_idle(void);
+
 static char * __init machine_specific_memory_setup(void)
 {
 	unsigned long max_pfn = xen_start_info->nr_pages;
@@ -74,4 +79,23 @@ static void __init machine_specific_arch
 		console_use_vt = 0;
 		conswitchp = NULL;
 	}
+
+	pm_idle = xen_idle;
+}
+
+void xen_idle(void)
+{
+	local_irq_disable();
+
+	if (need_resched())
+		local_irq_enable();
+	else {
+		clear_thread_flag(TIF_POLLING_NRFLAG);
+		smp_mb__after_clear_bit();
+		stop_hz_timer();
+		/* Blocking includes an implicit local_irq_enable(). */
+		HYPERVISOR_sched_op(SCHEDOP_block, 0);
+		start_hz_timer();
+		set_thread_flag(TIF_POLLING_NRFLAG);
+	}
 }

--

#58 lore raw

RFC PATCH 23/35 Add support for Xen event channels.

In reply to Chris Wright

Support Xen event channels instead of the i8259 PIC.

Event channels are used to inject events into the kernel, either from the hypervisor or from another VM. The injected events are mapped to interrupts.

If an event needs to be injected, the hypervisor causes an upcall into the kernel. The upcall handler then scans the event pending bitmap and calls do_IRQ for each pending event.

---
 arch/i386/kernel/Makefile               |    6 
 arch/i386/mach-xen/evtchn.c             |  844 ++++++++++++++++++++++++++++++++
 drivers/xen/core/Makefile               |    2 
 include/asm-i386/hw_irq.h               |    4 
 include/asm-i386/mach-xen/irq_vectors.h |  126 ++++
 include/xen/evtchn.h                    |  113 ++++
 6 files changed, 1093 insertions(+), 2 deletions(-)
arch/i386/kernel/Makefile +5 -1
@@ -5,7 +5,7 @@
 extra-y := head.o init_task.o vmlinux.lds
 
 obj-y	:= process.o semaphore.o signal.o entry.o traps.o irq.o \
-		ptrace.o time.o ioport.o ldt.o setup.o i8259.o sys_i386.o \
+		ptrace.o time.o ioport.o ldt.o setup.o hw_irq.o sys_i386.o \
 		pci-dma.o i386_ksyms.o i387.o dmi_scan.o bootflag.o \
 		quirks.o i8237.o topology.o
 
@@ -42,6 +42,10 @@ EXTRA_AFLAGS   := -traditional
 
 obj-$(CONFIG_SCx200)		+= scx200.o
 
+hw_irq-y			:= i8259.o
+
+hw_irq-$(CONFIG_XEN)		:= ../mach-xen/evtchn.o
+
 # vsyscall.o contains the vsyscall DSO images as __initdata.
 # We must build both images before we can assemble it.
 # Note: kbuild does not track this dependency due to usage of .incbin
include/asm-i386/hw_irq.h +3 -1
@@ -68,7 +68,9 @@ extern atomic_t irq_mis_count;
 
 #define IO_APIC_IRQ(x) (((x) >= 16) || ((1<<(x)) & io_apic_irqs))
 
-#if defined(CONFIG_X86_IO_APIC)
+#if defined(CONFIG_X86_XEN)
+extern void hw_resend_irq(struct hw_interrupt_type *h, unsigned int i);
+#elif defined(CONFIG_X86_IO_APIC)
 static inline void hw_resend_irq(struct hw_interrupt_type *h, unsigned int i)
 {
 	if (IO_APIC_IRQ(i))
arch/i386/mach-xen/evtchn.c new file +844 -0

844 lines not on this page · raw message

drivers/xen/core/Makefile new file +2 -0
@@ -0,0 +1,2 @@
+
+obj-y   := evtchn.o
include/asm-i386/mach-xen/irq_vectors.h new file +126 -0

126 lines not on this page · raw message

include/xen/evtchn.h new file +113 -0

113 lines not on this page · raw message

--

#59 lore raw

Re: [RFC PATCH 23/35] Add support for Xen event channels.

Quoted 61 lines · Chris Wright

plain text document attachment (22-evtchn) Support Xen event channels instead of the i8259 PIC.

Event channels are used to inject events into the kernel, either from the hypervisor or from another VM. The injected events are mapped to interrupts.

If an event needs to be injected, the hypervisor causes an upcall into the kernel. The upcall handler then scans the event pending bitmap and calls do_IRQ for each pending event.

---
 arch/i386/kernel/Makefile               |    6 
 arch/i386/mach-xen/evtchn.c             |  844 ++++++++++++++++++++++++++++++++
 drivers/xen/core/Makefile               |    2 
 include/asm-i386/hw_irq.h               |    4 
 include/asm-i386/mach-xen/irq_vectors.h |  126 ++++
 include/xen/evtchn.h                    |  113 ++++
 6 files changed, 1093 insertions(+), 2 deletions(-)
arch/i386/kernel/Makefile +5 -1
@@ -5,7 +5,7 @@
 extra-y := head.o init_task.o vmlinux.lds
 
 obj-y	:= process.o semaphore.o signal.o entry.o traps.o irq.o \
-		ptrace.o time.o ioport.o ldt.o setup.o i8259.o sys_i386.o \
+		ptrace.o time.o ioport.o ldt.o setup.o hw_irq.o sys_i386.o \
 		pci-dma.o i386_ksyms.o i387.o dmi_scan.o bootflag.o \
 		quirks.o i8237.o topology.o
 
@@ -42,6 +42,10 @@ EXTRA_AFLAGS   := -traditional
 
 obj-$(CONFIG_SCx200)		+= scx200.o
 
+hw_irq-y			:= i8259.o
+
+hw_irq-$(CONFIG_XEN)		:= ../mach-xen/evtchn.o
+
 # vsyscall.o contains the vsyscall DSO images as __initdata.
 # We must build both images before we can assemble it.
 # Note: kbuild does not track this dependency due to usage of .incbin
include/asm-i386/hw_irq.h +3 -1
@@ -68,7 +68,9 @@ extern atomic_t irq_mis_count;
 
 #define IO_APIC_IRQ(x) (((x) >= 16) || ((1<<(x)) & io_apic_irqs))
 
-#if defined(CONFIG_X86_IO_APIC)
+#if defined(CONFIG_X86_XEN)
+extern void hw_resend_irq(struct hw_interrupt_type *h, unsigned int i);
+#elif defined(CONFIG_X86_IO_APIC)
 static inline void hw_resend_irq(struct hw_interrupt_type *h, unsigned int i)
 {
 	if (IO_APIC_IRQ(i))

arch/i386/mach-xen/evtchn.c line 10

Quoted 10 lines
arch/i386/mach-xen/evtchn.c new file +10 -0
@@ -0,0 +1,844 @@
+/******************************************************************************
+ * evtchn.c
+ * 
+ * Communication via Xen event channels.
+ * 
+ * Copyright (c) 2002-2005, K A Fraser
+ * 
+ * This file may be distributed separately from the Linux kernel, or
+ * incorporated into other software packages, subject to the following license:
+ * 

again no allowance for use inside the kernel

arch/i386/mach-xen/evtchn.c line 36

Quoted 9 lines
arch/i386/mach-xen/evtchn.c new file +9 -0
+ */
+
+#include <linux/config.h>
+#include <linux/module.h>
+#include <linux/irq.h>
+#include <linux/interrupt.h>
+#include <linux/sched.h>
+#include <linux/kernel_stat.h>
+#include <linux/version.h>

this highly looks that it's not possible to be used outside the linux kernel so the license is odd

arch/i386/mach-xen/evtchn.c line 69

Quoted 6 lines
arch/i386/mach-xen/evtchn.c new file +6 -0
+/* Convenient shorthand for packed representation of an unbound IRQ. */
+#define IRQ_UNBOUND	mk_irq_info(IRQT_UNBOUND, 0, 0)
+/* Accessor macros for packed IRQ information. */
+#define evtchn_from_irq(irq) ((u16)(irq_info[irq]))
+#define index_from_irq(irq)  ((u8)(irq_info[irq] >> 16))
+#define type_from_irq(irq)   ((u8)(irq_info[irq] >> 24))

static inlines please

arch/i386/mach-xen/evtchn.c line 182

Quoted 5 lines
arch/i386/mach-xen/evtchn.c new file +5 -0
+#ifdef CONFIG_XEN_EVTCHN_DEVICE
+				evtchn_device_upcall(port);
+#else
+				mask_evtchn(port);
+#endif

why this ifdef ?

arch/i386/mach-xen/evtchn.c line 197

Quoted 10 lines
arch/i386/mach-xen/evtchn.c new file +10 -0
+static int find_unbound_irq(void)
+{
+	int irq;
+
+	for (irq = 0; irq < NR_IRQS; irq++)
+		if (irq_bindcount[irq] == 0)
+			break;
+
+	if (irq == NR_IRQS) {
+		printk(KERN_ERR "No available IRQ to bind to: increase NR_IRQS!\n");

there is no way to share interrupts? A shame

arch/i386/mach-xen/evtchn.c line 369

Quoted 1 line
arch/i386/mach-xen/evtchn.c new file +1 -0
+EXPORT_SYMBOL(bind_virq_to_irqhandler);

this is highly low level interfact, please make this and the others a _GPL export

#60 lore raw

Re: [RFC PATCH 23/35] Add support for Xen event channels.

Quoted 7 lines · Arjan van de Ven

arch/i386/mach-xen/evtchn.c line 36

Quoted 4 lines
arch/i386/mach-xen/evtchn.c new file +4 -0
+#include <linux/interrupt.h>
+#include <linux/sched.h>
+#include <linux/kernel_stat.h>
+#include <linux/version.h>

this highly looks that it's not possible to be used outside the linux kernel so the license is odd

It can't be used directly, but other OS ports steal fairly directly from this file (and others). For example OpenSolaris and the BSDs.

Quoted 5 lines
Quoted 5 lines
arch/i386/mach-xen/evtchn.c new file +5 -0
+static int find_unbound_irq(void)
+{
+	int irq;
+
+	for (irq = 0; irq < NR_IRQS; irq++)
Quoted 8 lines
Quoted 6 lines
arch/i386/mach-xen/evtchn.c new file +5 -0
+		if (irq_bindcount[irq] == 0)
+			break;
+
+	if (irq == NR_IRQS) {
+		printk(KERN_ERR "No available IRQ to bind to: increase 

NR_IRQS!\n");

there is no way to share interrupts? A shame

256 allocatable IRQs seems plenty to be getting on with. Shared IRQs can certainly be added later -- there are no hidden gotchas I'm pretty sure.

-- Keir

#61 lore raw

Re: [RFC PATCH 23/35] Add support for Xen event channels.

In reply to Chris Wright
Quoted 9 lines · Chris Wright

Support Xen event channels instead of the i8259 PIC.

Event channels are used to inject events into the kernel, either from the hypervisor or from another VM. The injected events are mapped to interrupts.

If an event needs to be injected, the hypervisor causes an upcall into the kernel. The upcall handler then scans the event pending bitmap and calls do_IRQ for each pending event.

This looks like mostly generic code. Shouldn't most of this belong into drivers/xen to be used by other architectures too? Putting so much generic code into i386 looks wrong.

include/asm-i386/hw_irq.h extern atomic_t irq_mis_count; · line 76

Quoted 8 lines
include/asm-i386/hw_irq.h +3 -1
-#if defined(CONFIG_X86_IO_APIC)
+#if defined(CONFIG_X86_XEN)
+extern void hw_resend_irq(struct hw_interrupt_type *h, unsigned int i);
+#elif defined(CONFIG_X86_IO_APIC)
 static inline void hw_resend_irq(struct hw_interrupt_type *h, unsigned int i)
 {
 	if (IO_APIC_IRQ(i))

Better just put this out of line in the native case too.

arch/i386/mach-xen/evtchn.c line 124

Quoted 2 lines
arch/i386/mach-xen/evtchn.c new file +2 -0
+/* Upcall to generic IRQ layer. */
+#ifdef CONFIG_X86

We don't do such things with ifdef trees. Put the necessary code into asm/

include/asm-i386/mach-xen/irq_vectors.h line 3

Quoted 6 lines
include/asm-i386/mach-xen/irq_vectors.h new file +3 -0
@@ -0,0 +1,126 @@
+/*
+ * This file should contain #defines for all of the interrupt vector
+ * numbers used by this architecture.

This is mostly identical with native right? Sharing, sharing, sharing.

-Andi

#62 lore raw

RFC PATCH 24/35 subarch support for mask value for irq nubmers

In reply to Chris Wright

Abstract the mask value for irq numbers into subarch-specific headers. Xen extends the IRQ numbering space to include room for dynamically allocated virtual interrupts (in the range 256-511), which requires a more permissive mask value.

---
 arch/i386/kernel/irq.c                      |    2 +-
 include/asm-i386/mach-default/irq_vectors.h |    2 ++
 include/asm-i386/mach-visws/irq_vectors.h   |    2 ++
 include/asm-i386/mach-voyager/irq_vectors.h |    2 ++
 include/asm-i386/mach-xen/irq_vectors.h     |    8 ++++++--
 5 files changed, 13 insertions(+), 3 deletions(-)
arch/i386/kernel/irq.c +1 -1
@@ -54,7 +54,7 @@ static union irq_ctx *softirq_ctx[NR_CPU
 fastcall unsigned int do_IRQ(struct pt_regs *regs)
 {	
 	/* high bits used in ret_from_ code */
-	int irq = regs->orig_eax & 0xff;
+	int irq = regs->orig_eax & DO_IRQ_MASK;
 #ifdef CONFIG_4KSTACKS
 	union irq_ctx *curctx, *irqctx;
 	u32 *isp;
include/asm-i386/mach-default/irq_vectors.h +2 -0
@@ -86,6 +86,8 @@
 
 #include "irq_vectors_limits.h"
 
+#define DO_IRQ_MASK		255
+
 #define FPU_IRQ			13
 
 #define	FIRST_VM86_IRQ		3
include/asm-i386/mach-visws/irq_vectors.h +2 -0
@@ -53,6 +53,8 @@
 #define NR_IRQS 224
 #define NR_IRQ_VECTORS NR_IRQS
 
+#define DO_IRQ_MASK		255
+
 #define FPU_IRQ			13
 
 #define	FIRST_VM86_IRQ		3
include/asm-i386/mach-voyager/irq_vectors.h +2 -0
@@ -59,6 +59,8 @@
 #define NR_IRQS 224
 #define NR_IRQ_VECTORS NR_IRQS
 
+#define DO_IRQ_MASK 			255
+
 #define FPU_IRQ				13
 
 #define	FIRST_VM86_IRQ		3
include/asm-i386/mach-xen/irq_vectors.h +6 -2
@@ -109,14 +109,18 @@
  */
 
 #define PIRQ_BASE		0
-#define NR_PIRQS		256
+#define PIRQ_BITS		8
+#define NR_PIRQS		(1 << PIRQ_BITS)
 
 #define DYNIRQ_BASE		(PIRQ_BASE + NR_PIRQS)
-#define NR_DYNIRQS		256
+#define DYNIRQ_BITS		8
+#define NR_DYNIRQS		(1 << DYNIRQ_BITS)
 
 #define NR_IRQS			(NR_PIRQS + NR_DYNIRQS)
 #define NR_IRQ_VECTORS		NR_IRQS
 
+#define DO_IRQ_MASK		__IRQ_MASK(PIRQ_BITS + DYNIRQ_BITS)
+
 #define pirq_to_irq(_x)		((_x) + PIRQ_BASE)
 #define irq_to_pirq(_x)		((_x) - PIRQ_BASE)
 

--

#63 lore raw

RFC PATCH 25/35 Add Xen time abstractions

In reply to Chris Wright

Add support for Xen time abstractions. To avoid expensive traps into the hypervisor, the passage of time is extrapolated from the local TSC and a set of timestamps and scaling factors exported to the guest via shared memory. Xen also provides a periodic interrupt facility which is used to drive updates of xtime and jiffies, and perform the usual process accounting and profiling.

---
 arch/i386/kernel/Makefile |    6 
 arch/i386/mach-xen/time.c | 1045 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 1050 insertions(+), 1 deletion(-)
arch/i386/kernel/Makefile +5 -1
@@ -9,8 +9,11 @@ obj-y	:= process.o semaphore.o signal.o 
 		pci-dma.o i386_ksyms.o i387.o dmi_scan.o bootflag.o \
 		quirks.o i8237.o topology.o
 
+timers-y			:= timers/
+timers-$(CONFIG_XEN)		:=
+
 obj-y				+= cpu/
-obj-y				+= timers/
+obj-y				+= $(timers-y)
 obj-y				+= acpi/
 obj-$(CONFIG_X86_BIOS_REBOOT)	+= reboot.o
 obj-$(CONFIG_MCA)		+= mca.o
@@ -45,6 +48,7 @@ obj-$(CONFIG_SCx200)		+= scx200.o
 hw_irq-y			:= i8259.o
 
 hw_irq-$(CONFIG_XEN)		:= ../mach-xen/evtchn.o
+time-$(CONFIG_XEN)		:= ../mach-xen/time.o
 
 # vsyscall.o contains the vsyscall DSO images as __initdata.
 # We must build both images before we can assemble it.
arch/i386/mach-xen/time.c new file +1045 -0

1,045 lines not on this page · raw message

--

#64 lore raw

Re: [RFC PATCH 25/35] Add Xen time abstractions

arch/i386/mach-xen/time.c line 2

Quoted 7 lines

# We must build both images before we can assemble it.

arch/i386/mach-xen/time.c new file +2 -0
@@ -0,0 +1,1045 @@
+/*
+ *  linux/arch/i386/kernel/time.c

no it's not

is there any reason why this is a full copy rather than just a subarching of the modified functions, or some simple hooks into the generic function?

Duplication like this is evil because it means too many places need duplicated bugfixes and rework (and such rework is underway)

#65 lore raw

Re: [RFC PATCH 25/35] Add Xen time abstractions

Quoted 2 lines · * Arjan van de Ven ([email protected]) wrote:

Duplication like this is evil because it means too many places need duplicated bugfixes and rework (and such rework is underway)

Yes, that one is already on the todo list. Will work on consolidation there.

thanks,
-chris

#66 lore raw

Re: [Xen-devel] Re: [RFC PATCH 25/35] Add Xen time abstractions

Quoted 10 lines · Chris Wright
Quoted 2 lines · * Arjan van de Ven ([email protected]) wrote:

Duplication like this is evil because it means too many places need duplicated bugfixes and rework (and such rework is underway)

Yes, that one is already on the todo list. Will work on consolidation there.

thanks,
-chris

These patches from the VMI patch series sent out last week could be easily used to abstract time when running on Xen to accomplish the consolidation and transparency with native goals:

http://lkml.org/lkml/2006/3/13/172
http://lkml.org/lkml/2006/3/13/173

Unlike this Xen specific patch, the VMI time patches already provide transparency with native -- the timer_vmi.c device is used when booting on a hypervisor that provides this functionality, otherwise one of the traditional time sources is fallen back to.

Also, they remove much of the code duplication by taking advantage of the the i386 time interface mechanisms (and can be easily updated for the time rework that is underway).

Finally, the vmi nicely abstracts the details of the formula that is used to compute system_time from the HYPERVISOR_shared_info time parameters by hiding the computation underneath the kernel <-> hypervisor interface. Note that efficiency is not lost since the code to compute system_time, etc would execute in the "ROM" (a.k.a. hypercall page) without needing to call all the way down into the hypervisor. Hiding these details from the Linux code would be nice since it leads to easier to understand linux code (it doesn't have to deal with the details of how time is interpolated from hardware rdtsc), and allows the calculation for "system_time", etc to change without breaking the kernel<->hypervisor interface. It also allows for more freedom as to how the time interface is implemented; hypervisors aren't forced to implement system_time using a particular algorithm. The Xen interface needlessly imposes one particular algorithm to compute system_time from rdtsc.

See this post for a description of how the Xen time interface could be mapped to the timer_vmi.c patches I linked above:

http://lists.xensource.com/archives/html/xen-devel/2006-02/msg00876.html

Dan

#67 lore raw

Re: [Xen-devel] [RFC PATCH 25/35] Add Xen time abstractions

In reply to Chris Wright

arch/i386/kernel/Makefile obj-y := process.o semaphore.o signal.o · line 14

Quoted 9 lines
arch/i386/kernel/Makefile +3 -0
@@ -9,8 +9,11 @@ obj-y	:= process.o semaphore.o signal.o 
 		pci-dma.o i386_ksyms.o i387.o dmi_scan.o bootflag.o \
 		quirks.o i8237.o topology.o
 
+timers-y			:= timers/
+timers-$(CONFIG_XEN)		:=
+

You need to disable CONFIG_HPET_TIMER for CONFIG_XEN, otherwise select_timer is undefined since you don't include timers here.

Eli

#68 lore raw

Re: [Xen-devel] [RFC PATCH 25/35] Add Xen time abstractions

Quoted 6 lines · * Eli Collins ([email protected]) wrote:
Quoted 6 lines
arch/i386/kernel/Makefile +0 -0
@@ -9,8 +9,11 @@ obj-y	:= process.o semaphore.o signal.o 
 		pci-dma.o i386_ksyms.o i387.o dmi_scan.o bootflag.o \
 		quirks.o i8237.o topology.o
 
Quoted 6 lines

arch/i386/kernel/Makefile obj-y := process.o semaphore.o signal.o · line 14

Quoted 3 lines
arch/i386/kernel/Makefile +3 -0
+timers-y			:= timers/
+timers-$(CONFIG_XEN)		:=
+

You need to disable CONFIG_HPET_TIMER for CONFIG_XEN, otherwise select_timer is undefined since you don't include timers here.

Thanks, you're right. It's done the main Xen tree, didn't make it into this patchset, fixed.

thanks,
-chris

#69 lore raw

RFC PATCH 26/35 Add Xen subarch reboot support

In reply to Chris Wright

Add remote reboot capability, so that a virtual machine can be rebooted, halted or 'powered off' by external management tools.

---
 arch/i386/kernel/Makefile   |    1 
 arch/i386/mach-xen/reboot.c |  265 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 266 insertions(+)
arch/i386/kernel/Makefile +1 -0
@@ -49,6 +49,7 @@ hw_irq-y			:= i8259.o
 
 hw_irq-$(CONFIG_XEN)		:= ../mach-xen/evtchn.o
 time-$(CONFIG_XEN)		:= ../mach-xen/time.o
+reboot-$(CONFIG_XEN)		:= ../mach-xen/reboot.o
 
 # vsyscall.o contains the vsyscall DSO images as __initdata.
 # We must build both images before we can assemble it.
arch/i386/mach-xen/reboot.c new file +265 -0

265 lines not on this page · raw message

--

#70 lore raw

Re: [RFC PATCH 26/35] Add Xen subarch reboot support

arch/i386/mach-xen/reboot.c line 81

Quoted 9 lines · Chris Wright
arch/i386/mach-xen/reboot.c new file +9 -0
+
+#ifdef CONFIG_XEN_XENBUS
+/* Ignore multiple shutdown requests. */
+static int shutting_down = SHUTDOWN_INVALID;
+static void __shutdown_handler(void *unused);
+static DECLARE_WORK(shutdown_work, __shutdown_handler, NULL);
+#endif
+
+#ifdef CONFIG_XEN_XENBUS

eh why the re-ifdef

Quoted 27 lines
arch/i386/mach-xen/reboot.c new file +27 -0
+static int shutdown_process(void *__unused)
+{
+	static char *envp[] = { "HOME=/", "TERM=linux",
+				"PATH=/sbin:/usr/sbin:/bin:/usr/bin", NULL };
+	static char *restart_argv[]  = { "/sbin/reboot", NULL };
+	static char *poweroff_argv[] = { "/sbin/poweroff", NULL };
+
+	extern asmlinkage long sys_reboot(int magic1, int magic2,
+					  unsigned int cmd, void *arg);
+
+	daemonize("shutdown");
+
+	switch (shutting_down) {
+	case SHUTDOWN_POWEROFF:
+	case SHUTDOWN_HALT:
+		if (execve("/sbin/poweroff", poweroff_argv, envp) < 0) {
+			sys_reboot(LINUX_REBOOT_MAGIC1,
+				   LINUX_REBOOT_MAGIC2,
+				   LINUX_REBOOT_CMD_POWER_OFF,
+				   NULL);
+		}
+		break;
+
+	case SHUTDOWN_REBOOT:
+		if (execve("/sbin/reboot", restart_argv, envp) < 0) {
+			sys_reboot(LINUX_REBOOT_MAGIC1,
+				   LINUX_REBOOT_MAGIC2,

arch/i386/mach-xen/reboot.c line 118

Quoted 10 lines
arch/i386/mach-xen/reboot.c new file +10 -0
+				   LINUX_REBOOT_CMD_RESTART,
+				   NULL);
+		}
+		break;
+	}
+
+	shutting_down = SHUTDOWN_INVALID; /* could try again */
+
+	return 0;
+}

how is this function different from the generic one? If not, why aren't you using the generic one?

Quoted 1 line

+static struct notifier_block xenstore_notifier;

what is this for? It's not exported and hardly used...

#71 lore raw

Re: [RFC PATCH 26/35] Add Xen subarch reboot support

Quoted 9 lines · Arjan van de Ven

arch/i386/mach-xen/reboot.c line 87

Quoted 6 lines
arch/i386/mach-xen/reboot.c new file +6 -0
+static int shutdown_process(void *__unused)
+{
+	static char *envp[] = { "HOME=/", "TERM=linux",
+				"PATH=/sbin:/usr/sbin:/bin:/usr/bin", NULL };
+	static char *restart_argv[]  = { "/sbin/reboot", NULL };
+	static char *poweroff_argv[] = { "/sbin/poweroff", NULL };

how is this function different from the generic one? If not, why aren't you using the generic one?

The intent is to allow remote management tools to trigger a clean shutdown of the virtual machine. That requires us to notify to userspace, and this function does that by exec'ing one of the standard userspace programs. Given the trigger is received by the kernel in the first instance I don't know a better way of doing this. And if this is the best way, I don't think there is generic code in the kernel which does the same thing.

-- Keir

#72 lore raw

Re: [RFC PATCH 26/35] Add Xen subarch reboot support

Quoted 12 lines · Keir Fraser
Quoted 9 lines · Arjan van de Ven

arch/i386/mach-xen/reboot.c line 87

Quoted 6 lines
arch/i386/mach-xen/reboot.c new file +6 -0
+static int shutdown_process(void *__unused)
+{
+	static char *envp[] = { "HOME=/", "TERM=linux",
+				"PATH=/sbin:/usr/sbin:/bin:/usr/bin", NULL };
+	static char *restart_argv[]  = { "/sbin/reboot", NULL };
+	static char *poweroff_argv[] = { "/sbin/poweroff", NULL };

how is this function different from the generic one? If not, why aren't you using the generic one?

Quoted 7 lines

The intent is to allow remote management tools to trigger a clean shutdown of the virtual machine. That requires us to notify to userspace, and this function does that by exec'ing one of the standard userspace programs. Given the trigger is received by the kernel in the first instance I don't know a better way of doing this. And if this is the best way, I don't think there is generic code in the kernel which does the same thing.

well this isn't really different from the normal ctrl-alt-delete right? I would strongly suggest to follow the normal ctrl-alt-del path.. that follows the normal convention sysadmins are used to.
It's not "/sbin/poweroff" fwiw... at least not hardcoded. Following the normal ctrl-alt-del codepath gets all the policy out of this kind of thing as well..

#73 lore raw

Re: [RFC PATCH 26/35] Add Xen subarch reboot support

Quoted 9 lines · Arjan van de Ven
Quoted 7 lines

The intent is to allow remote management tools to trigger a clean shutdown of the virtual machine. That requires us to notify to userspace, and this function does that by exec'ing one of the standard userspace programs. Given the trigger is received by the kernel in the first instance I don't know a better way of doing this. And if this is the best way, I don't think there is generic code in the kernel which does the same thing.

Quoted 6 lines

well this isn't really different from the normal ctrl-alt-delete right? I would strongly suggest to follow the normal ctrl-alt-del path.. that follows the normal convention sysadmins are used to.
It's not "/sbin/poweroff" fwiw... at least not hardcoded. Following the normal ctrl-alt-del codepath gets all the policy out of this kind of thing as well..

Hmm... that will work okay for reboot, where SIGINT to init is probably a better strategy than what we do now. But we'd still need something special for halt/shutdown. We followed the same principle for this as sparc64/kernel/power.c.

-- Keir

#74 lore raw

Re: [RFC PATCH 26/35] Add Xen subarch reboot support

In reply to Chris Wright

arch/i386/mach-xen/reboot.c line 87

Quoted 4 lines · Chris Wright
arch/i386/mach-xen/reboot.c new file +4 -0
+       static char *envp[] = { "HOME=/", "TERM=linux",
+                               "PATH=/sbin:/usr/sbin:/bin:/usr/bin", NULL };
+       static char *restart_argv[]  = { "/sbin/reboot", NULL };
+       static char *poweroff_argv[] = { "/sbin/poweroff", NULL };

It would be better if that was user configurable.

arch/i386/mach-xen/reboot.c line 90

Quoted 2 lines
arch/i386/mach-xen/reboot.c new file +2 -0
+	extern asmlinkage long sys_reboot(int magic1, int magic2,
+					  unsigned int cmd, void *arg);

This is what linux/syscalls.h is there for.

arch/i386/mach-xen/reboot.c line 92

Quoted 1 line
arch/i386/mach-xen/reboot.c new file +1 -0
+	daemonize("shutdown");

What is that good for?

arch/i386/mach-xen/reboot.c line 97

Quoted 5 lines
arch/i386/mach-xen/reboot.c new file +5 -0
+
+	switch (shutting_down) {
+	case SHUTDOWN_POWEROFF:
+	case SHUTDOWN_HALT:
+		if (execve("/sbin/poweroff", poweroff_argv, envp) < 0) {

You should probably keep track if the execve already happened and if it is called again do the sys_reboot directly.

-Andi

#75 lore raw

Re: [RFC PATCH 26/35] Add Xen subarch reboot support

Quoted 7 lines · Andi Kleen

arch/i386/mach-xen/reboot.c line 87

Quoted 4 lines · Chris Wright
arch/i386/mach-xen/reboot.c new file +4 -0
+       static char *envp[] = { "HOME=/", "TERM=linux",
+                               "PATH=/sbin:/usr/sbin:/bin:/usr/bin", NULL };
+       static char *restart_argv[]  = { "/sbin/reboot", NULL };
+       static char *poweroff_argv[] = { "/sbin/poweroff", NULL };

It would be better if that was user configurable.

acpi also needs to shutdown machine on overheat. It would be nice to consolidate all those places. New signal to init would be best, I'd say.

Signature · 1 line
Thanks, Sharp!
#76 lore raw

RFC PATCH 27/35 Add nosegneg capability to the vsyscall page notes

In reply to Chris Wright

Add the "nosegneg" fake capabilty to the vsyscall page notes. This is used by the runtime linker to select a glibc version which then disables negative-offset accesses to the thread-local segment via %gs. These accesses require emulation in Xen (because segments are truncated to protect the hypervisor address space) and avoiding them provides a measurable performance boost.

---
 arch/i386/kernel/vsyscall-note.S |   29 +++++++++++++++++++++++++++++
 1 files changed, 29 insertions(+)
arch/i386/kernel/vsyscall-note.S +29 -0
@@ -3,6 +3,7 @@
  * Here we can supply some information useful to userland.
  */
 
+#include <linux/config.h>
 #include <linux/uts.h>
 #include <linux/version.h>
 
@@ -23,3 +24,31 @@
 	ASM_ELF_NOTE_BEGIN(".note.kernel-version", "a", UTS_SYSNAME, 0)
 	.long LINUX_VERSION_CODE
 	ASM_ELF_NOTE_END
+
+#ifdef CONFIG_XEN
+/*
+ * Add a special note telling glibc's dynamic linker a fake hardware
+ * flavor that it will use to choose the search path for libraries in the
+ * same way it uses real hardware capabilities like "mmx".
+ * We supply "nosegneg" as the fake capability, to indicate that we
+ * do not like negative offsets in instructions using segment overrides,
+ * since we implement those inefficiently.  This makes it possible to
+ * install libraries optimized to avoid those access patterns in someplace
+ * like /lib/i686/tls/nosegneg.  Note that an /etc/ld.so.conf.d/file
+ * corresponding to the bits here is needed to make ldconfig work right.
+ * It should contain:
+ *	hwcap 0 nosegneg
+ * to match the mapping of bit to name that we give here.
+ */
+#define NOTE_KERNELCAP_BEGIN(ncaps, mask) \
+	ASM_ELF_NOTE_BEGIN(".note.kernelcap", "a", "GNU", 2) \
+	.long ncaps, mask
+#define NOTE_KERNELCAP(bit, name) \
+	.byte bit; .asciz name
+#define NOTE_KERNELCAP_END ASM_ELF_NOTE_END
+
+NOTE_KERNELCAP_BEGIN(1, 1)
+NOTE_KERNELCAP(1, "nosegneg")
+NOTE_KERNELCAP_END
+#endif
+

--

#77 lore raw

RFC PATCH 28/35 add support for Xen feature queries

In reply to Chris Wright

Add support for parsing and interpreting hypervisor feature flags. These allow the kernel to determine what features are provided by the underlying hypervisor. For example, whether page tables need to be write protected explicitly by the kernel, and whether the kernel (appears to) run in ring 0 rather than ring 1. This information allows the kernel to improve performance by avoiding unnecessary actions.

---
 arch/i386/mach-xen/Makefile                 |    2 -
 arch/i386/mach-xen/features.c               |   29 ++++++++++++++++++++++++++++
 include/asm-i386/mach-xen/setup_arch_post.h |    2 +
 include/xen/features.h                      |   20 +++++++++++++++++++
 4 files changed, 52 insertions(+), 1 deletion(-)
arch/i386/mach-xen/Makefile +1 -1
@@ -4,6 +4,6 @@
 
 extra-y				:= head.o
 
-obj-y				:= setup.o
+obj-y				:= setup.o features.o
  
 setup-y				:= ../mach-default/setup.o
include/asm-i386/mach-xen/setup_arch_post.h +2 -0
@@ -40,6 +40,8 @@ static void __init machine_specific_arch
 {
 	struct physdev_op op;
 
+	setup_xen_features();
+
 	HYPERVISOR_shared_info =
 		(struct shared_info *)__va(xen_start_info->shared_info);
 	memset(empty_zero_page, 0, sizeof(empty_zero_page));
arch/i386/mach-xen/features.c new file +29 -0
@@ -0,0 +1,29 @@
+/******************************************************************************
+ * features.c
+ *
+ * Xen feature flags.
+ *
+ * Copyright (c) 2006, Ian Campbell, XenSource Inc.
+ */
+#include <linux/types.h>
+#include <linux/cache.h>
+#include <linux/module.h>
+#include <asm/hypervisor.h>
+#include <xen/features.h>
+
+u8 xen_features[XENFEAT_NR_SUBMAPS * 32] __read_mostly;
+EXPORT_SYMBOL(xen_features);
+
+void setup_xen_features(void)
+{
+	struct xen_feature_info fi;
+	int i, j;
+
+	for (i = 0; i < XENFEAT_NR_SUBMAPS; i++) {
+		fi.submap_idx = i;
+		if (HYPERVISOR_xen_version(XENVER_get_features, &fi) < 0)
+			break;
+		for (j=0; j<32; j++)
+			xen_features[i*32+j] = !!(fi.submap & 1<<j);
+	}
+}
include/xen/features.h new file +20 -0
@@ -0,0 +1,20 @@
+/******************************************************************************
+ * features.h
+ *
+ * Query the features reported by Xen.
+ *
+ * Copyright (c) 2006, Ian Campbell
+ */
+
+#ifndef __ASM_XEN_FEATURES_H__
+#define __ASM_XEN_FEATURES_H__
+
+#include <xen/interface/version.h>
+
+extern void setup_xen_features(void);
+
+extern u8 xen_features[XENFEAT_NR_SUBMAPS * 32];
+
+#define xen_feature(flag)	(xen_features[flag])
+
+#endif /* __ASM_XEN_FEATURES_H__ */

--

#78 lore raw

Re: [RFC PATCH 28/35] add support for Xen feature queries

Quoted 47 lines · Chris Wright

plain text document attachment (27-xen-features)
Add support for parsing and interpreting hypervisor feature flags. These allow the kernel to determine what features are provided by the underlying hypervisor. For example, whether page tables need to be write protected explicitly by the kernel, and whether the kernel (appears to) run in ring 0 rather than ring 1. This information allows the kernel to improve performance by avoiding unnecessary actions.

---
 arch/i386/mach-xen/Makefile                 |    2 -
 arch/i386/mach-xen/features.c               |   29 ++++++++++++++++++++++++++++
 include/asm-i386/mach-xen/setup_arch_post.h |    2 +
 include/xen/features.h                      |   20 +++++++++++++++++++
 4 files changed, 52 insertions(+), 1 deletion(-)
arch/i386/mach-xen/Makefile +1 -1
@@ -4,6 +4,6 @@
 
 extra-y				:= head.o
 
-obj-y				:= setup.o
+obj-y				:= setup.o features.o
 
 setup-y				:= ../mach-default/setup.o
include/asm-i386/mach-xen/setup_arch_post.h +2 -0
@@ -40,6 +40,8 @@ static void __init machine_specific_arch
 {
 	struct physdev_op op;
 
+	setup_xen_features();
+
 	HYPERVISOR_shared_info =
 		(struct shared_info *)__va(xen_start_info->shared_info);
 	memset(empty_zero_page, 0, sizeof(empty_zero_page));
arch/i386/mach-xen/features.c new file +5 -0
@@ -0,0 +1,29 @@
+/******************************************************************************
+ * features.c
+ *
+ * Xen feature flags.
+ *

arch/i386/mach-xen/features.c line 15

Quoted 10 lines
arch/i386/mach-xen/features.c new file +10 -0
+ * Copyright (c) 2006, Ian Campbell, XenSource Inc.
+ */
+#include <linux/types.h>
+#include <linux/cache.h>
+#include <linux/module.h>
+#include <asm/hypervisor.h>
+#include <xen/features.h>
+
+u8 xen_features[XENFEAT_NR_SUBMAPS * 32] __read_mostly;
+EXPORT_SYMBOL(xen_features);

EXPORT_SYMBOL_GPL please; this is very specific, low level functionality

#79 lore raw

RFC PATCH 29/35 Add the Xen virtual console driver.

In reply to Chris Wright

This provides a bootstrap and ongoing emergency console which is intended to be available from very early during boot and at all times thereafter, in contrast with alternatives such as UDP-based syslogd, or logging in via ssh. The protocol is based on a simple shared-memory ring buffer.

---
 drivers/Makefile                   |    1 
 drivers/char/tty_io.c              |    7 
 drivers/xen/Makefile               |    2 
 drivers/xen/console/Makefile       |    2 
 drivers/xen/console/console.c      |  633 +++++++++++++++++++++++++++++++++++++
 drivers/xen/console/xencons_ring.c |  115 ++++++
 include/xen/xencons.h              |   14 
 7 files changed, 773 insertions(+), 1 deletion(-)
drivers/Makefile +1 -0
@@ -34,6 +34,7 @@ obj-y				+= base/ block/ misc/ mfd/ net/
 obj-$(CONFIG_NUBUS)		+= nubus/
 obj-$(CONFIG_ATM)		+= atm/
 obj-$(CONFIG_PPC_PMAC)		+= macintosh/
+obj-$(CONFIG_XEN)		+= xen/
 obj-$(CONFIG_IDE)		+= ide/
 obj-$(CONFIG_FC4)		+= fc4/
 obj-$(CONFIG_SCSI)		+= scsi/
drivers/char/tty_io.c +6 -1
@@ -132,6 +132,8 @@ LIST_HEAD(tty_drivers);			/* linked list
    vt.c for deeply disgusting hack reasons */
 DECLARE_MUTEX(tty_sem);
 
+int console_use_vt = 1;
+
 #ifdef CONFIG_UNIX98_PTYS
 extern struct tty_driver *ptm_driver;	/* Unix98 pty masters; for /dev/ptmx */
 extern int pty_limit;		/* Config limit on Unix98 ptys */
@@ -2054,7 +2056,7 @@ retry_open:
 		goto got_driver;
 	}
 #ifdef CONFIG_VT
-	if (device == MKDEV(TTY_MAJOR,0)) {
+	if (console_use_vt && (device == MKDEV(TTY_MAJOR,0))) {
 		extern struct tty_driver *console_driver;
 		driver = console_driver;
 		index = fg_console;
@@ -3237,6 +3239,8 @@ static int __init tty_init(void)
 #endif
 
 #ifdef CONFIG_VT
+	if (!console_use_vt)
+		goto out_vt;
 	cdev_init(&vc0_cdev, &console_fops);
 	if (cdev_add(&vc0_cdev, MKDEV(TTY_MAJOR, 0), 1) ||
 	    register_chrdev_region(MKDEV(TTY_MAJOR, 0), 1, "/dev/vc/0") < 0)
@@ -3245,6 +3249,7 @@ static int __init tty_init(void)
 	class_device_create(tty_class, NULL, MKDEV(TTY_MAJOR, 0), NULL, "tty0");
 
 	vty_init();
+ out_vt:
 #endif
 	return 0;
 }
drivers/xen/Makefile new file +2 -0
@@ -0,0 +1,2 @@
+
+obj-y	+= console/
drivers/xen/console/Makefile new file +2 -0
@@ -0,0 +1,2 @@
+
+obj-y	:= console.o xencons_ring.o
drivers/xen/console/console.c new file +633 -0

633 lines not on this page · raw message

drivers/xen/console/xencons_ring.c new file +115 -0

115 lines not on this page · raw message

include/xen/xencons.h new file +14 -0
@@ -0,0 +1,14 @@
+#ifndef __ASM_XENCONS_H__
+#define __ASM_XENCONS_H__
+
+void xencons_force_flush(void);
+void xencons_resume(void);
+
+/* Interrupt work hooks. Receive data, or kick data out. */
+void xencons_rx(char *buf, unsigned len, struct pt_regs *regs);
+void xencons_tx(void);
+
+int xencons_ring_init(void);
+int xencons_ring_send(const char *data, unsigned len);
+
+#endif /* __ASM_XENCONS_H__ */

--

#80 lore raw

Re: [RFC PATCH 29/35] Add the Xen virtual console driver.

Quoted 6 lines · Chris Wright

plain text document attachment (28-xen-console)
This provides a bootstrap and ongoing emergency console which is intended to be available from very early during boot and at all times thereafter, in contrast with alternatives such as UDP-based syslogd, or logging in via ssh. The protocol is based on a simple shared-memory ring buffer.

there already exist early consoles. Please just use that infrastructure instead.

#81 lore raw

Re: [RFC PATCH 29/35] Add the Xen virtual console driver.

Quoted 8 lines · Arjan van de Ven
Quoted 5 lines

This provides a bootstrap and ongoing emergency console which is intended to be available from very early during boot and at all times thereafter, in contrast with alternatives such as UDP-based syslogd, or logging in via ssh. The protocol is based on a simple shared-memory ring buffer.

there already exist early consoles. Please just use that infrastructure instead.

By 'early console' I don't mean it's an alternative to the early_printk infrastructure. It's simply an easy way to characters out of the guest when other alternatives (such as using the virtual network device) are unusable.

-- Keir

#82 lore raw

Re: [RFC PATCH 29/35] Add the Xen virtual console driver.

In reply to Chris Wright

On Wednesday 22 March 2006 07:31, Chris Wright wrote:

Basically it's the same thing as the buffer in printk.c except that it does input too.

You could safe a lot of code by just teaching the Hypervisor about that one. Not sure it is fully practical though,

drivers/char/tty_io.c LIST_HEAD(tty_drivers); /* linked list · line 135

Quoted 7 lines
drivers/char/tty_io.c +1 -0
@@ -132,6 +132,8 @@ LIST_HEAD(tty_drivers);			/* linked list
    vt.c for deeply disgusting hack reasons */
 DECLARE_MUTEX(tty_sem);
 
+int console_use_vt = 1;

This doesn't have anything to do with early console.

Why is it needed? Looks like a ugly hack.

drivers/char/tty_io.c LIST_HEAD(tty_drivers); /* linked list · line 136

Quoted 1 line
drivers/char/tty_io.c +1 -0
+#define __RETCODE 0

What is that?

drivers/char/tty_io.c LIST_HEAD(tty_drivers); /* linked list · line 138

Quoted 2 lines
drivers/char/tty_io.c +2 -0
+/*** Useful function for console debugging -- goes straight to Xen. ***/
+asmlinkage int xprintk(const char *fmt, ...)

In normal Linux this is called early_printk()

drivers/char/tty_io.c LIST_HEAD(tty_drivers); /* linked list

Quoted 4 lines
drivers/char/tty_io.c +4 -0
+{
+	va_list args;
+	int printk_len;
+	static char printk_buf[1024];

Who does the locking here?

-Andi

#83 lore raw

Re: [RFC PATCH 29/35] Add the Xen virtual console driver.

In reply to Chris Wright
Quoted 11 lines · Chris Wright

+/*
+ * Modes:

+ *  'xencons=off'  [XC_OFF]:     Console is disabled.
+ *  'xencons=tty'  [XC_TTY]:     Console attached to '/dev/tty[0-9]+'.
+ *  'xencons=ttyS' [XC_SERIAL]:  Console attached to '/dev/ttyS[0-9]+'.

+ * [XC_DEFAULT]: DOM0 -> XC_SERIAL ; all others -> XC_TTY.
+ *
+ * NB. In mode XC_TTY, we create dummy consoles for tty2-63. This suppresses
+ * warnings from standard distro startup scripts.
+ */

This has always seemed a bit wrong to me and makes a number of things kind of awkward (like a virtual video driver).

It would seem better me to treat this driver as what it really is, a virtual serial device. It adds a little bit of additional work to the userspace tools (they just have to make sure to pass console=ttyS0) but it seems worth it.

We could also solve the tty[0-9] problem by implementing a proper console driver that could use multiple virtual serial devices if we wanted to go that route.

Another option would be to just emulate a serial driver. The console driver isn't really performance critical. It seems to me that it's a bit unnecessary to even bother paravirtualizing the console device when it's so easy to emulate.

Regards,

Anthony Liguori

Quoted 711 lines
drivers/xen/console/console.c new file +566 -0
+static enum { XC_OFF, XC_DEFAULT, XC_TTY, XC_SERIAL } xc_mode = XC_DEFAULT;
+static int xc_num = -1;
+
+#ifdef CONFIG_MAGIC_SYSRQ
+static unsigned long sysrq_requested;
+extern int sysrq_enabled;
+#endif
+
+static int __init xencons_setup(char *str)
+{
+	char *q;
+	int n;
+
+	if (!strncmp(str, "ttyS", 4))
+		xc_mode = XC_SERIAL;
+	else if (!strncmp(str, "tty", 3))
+		xc_mode = XC_TTY;
+	else if (!strncmp(str, "off", 3))
+		xc_mode = XC_OFF;
+
+	switch (xc_mode) {
+	case XC_SERIAL:
+		n = simple_strtol(str+4, &q, 10);
+		if (q > (str + 4))
+			xc_num = n;
+		break;
+	case XC_TTY:
+		n = simple_strtol(str+3, &q, 10);
+		if (q > (str + 3))
+			xc_num = n;
+		break;
+	default:
+		break;
+	}
+
+	return 1;
+}
+__setup("xencons=", xencons_setup);
+
+/* The kernel and user-land drivers share a common transmit buffer. */
+static unsigned int wbuf_size = 4096;
+#define WBUF_MASK(_i) ((_i)&(wbuf_size-1))
+static char *wbuf;
+static unsigned int wc, wp; /* write_cons, write_prod */
+
+static int __init xencons_bufsz_setup(char *str)
+{
+	unsigned int goal;
+	goal = simple_strtoul(str, NULL, 0);
+	while (wbuf_size < goal)
+		wbuf_size <<= 1;
+	return 1;
+}
+__setup("xencons_bufsz=", xencons_bufsz_setup);
+
+/* This lock protects accesses to the common transmit buffer. */
+static spinlock_t xencons_lock = SPIN_LOCK_UNLOCKED;
+
+/* Common transmit-kick routine. */
+static void __xencons_tx_flush(void);
+
+static struct tty_driver *xencons_driver;
+
+/******************** Kernel console driver ********************************/
+
+static void kcons_write(
+	struct console *c, const char *s, unsigned int count)
+{
+	int           i = 0;
+	unsigned long flags;
+
+	spin_lock_irqsave(&xencons_lock, flags);
+
+	while (i < count) {
+		for (; i < count; i++) {
+			if ((wp - wc) >= (wbuf_size - 1))
+				break;
+			if ((wbuf[WBUF_MASK(wp++)] = s[i]) == '\n')
+				wbuf[WBUF_MASK(wp++)] = '\r';
+		}
+
+		__xencons_tx_flush();
+	}
+
+	spin_unlock_irqrestore(&xencons_lock, flags);
+}
+
+static void kcons_write_dom0(
+	struct console *c, const char *s, unsigned int count)
+{
+	int rc;
+
+	while ((count > 0) &&
+	       ((rc = HYPERVISOR_console_io(
+			CONSOLEIO_write, count, (char *)s)) > 0)) {
+		count -= rc;
+		s += rc;
+	}
+}
+
+static struct tty_driver *kcons_device(struct console *c, int *index)
+{
+	*index = 0;
+	return xencons_driver;
+}
+
+static struct console kcons_info = {
+	.device	= kcons_device,
+	.flags	= CON_PRINTBUFFER,
+	.index	= -1,
+};
+
+#define __RETCODE 0
+static int __init xen_console_init(void)
+{
+	if (xen_init() < 0)
+		return __RETCODE;
+
+	if (xen_start_info->flags & SIF_INITDOMAIN) {
+		if (xc_mode == XC_DEFAULT)
+			xc_mode = XC_SERIAL;
+		kcons_info.write = kcons_write_dom0;
+		if (xc_mode == XC_SERIAL)
+			kcons_info.flags |= CON_ENABLED;
+	} else {
+		if (xc_mode == XC_DEFAULT)
+			xc_mode = XC_TTY;
+		kcons_info.write = kcons_write;
+	}
+
+	switch (xc_mode) {
+	case XC_SERIAL:
+		strcpy(kcons_info.name, "ttyS");
+		if (xc_num == -1)
+			xc_num = 0;
+		break;
+
+	case XC_TTY:
+		strcpy(kcons_info.name, "tty");
+		if (xc_num == -1)
+			xc_num = 1;
+		break;
+
+	default:
+		return __RETCODE;
+	}
+
+	wbuf = alloc_bootmem(wbuf_size);
+
+	register_console(&kcons_info);
+
+	return __RETCODE;
+}
+console_initcall(xen_console_init);
+
+/*** Useful function for console debugging -- goes straight to Xen. ***/
+asmlinkage int xprintk(const char *fmt, ...)
+{
+	va_list args;
+	int printk_len;
+	static char printk_buf[1024];
+
+	/* Emit the output into the temporary buffer */
+	va_start(args, fmt);
+	printk_len = vsnprintf(printk_buf, sizeof(printk_buf), fmt, args);
+	va_end(args);
+
+	/* Send the processed output directly to Xen. */
+	kcons_write_dom0(NULL, printk_buf, printk_len);
+
+	return 0;
+}
+
+/*** Forcibly flush console data before dying. ***/
+void xencons_force_flush(void)
+{
+	int sz;
+
+	/* Emergency console is synchronous, so there's nothing to flush. */
+	if (xen_start_info->flags & SIF_INITDOMAIN)
+		return;
+
+	/* Spin until console data is flushed through to the daemon. */
+	while (wc != wp) {
+		int sent = 0;
+		if ((sz = wp - wc) == 0)
+			continue;
+		sent = xencons_ring_send(&wbuf[WBUF_MASK(wc)], sz);
+		if (sent > 0)
+			wc += sent;
+	}
+}
+
+
+/******************** User-space console driver (/dev/console) ************/
+
+#define DRV(_d)         (_d)
+#define TTY_INDEX(_tty) ((_tty)->index)
+
+static struct termios *xencons_termios[MAX_NR_CONSOLES];
+static struct termios *xencons_termios_locked[MAX_NR_CONSOLES];
+static struct tty_struct *xencons_tty;
+static int xencons_priv_irq;
+static char x_char;
+
+void xencons_rx(char *buf, unsigned len, struct pt_regs *regs)
+{
+	int           i;
+	unsigned long flags;
+
+	spin_lock_irqsave(&xencons_lock, flags);
+	if (xencons_tty == NULL)
+		goto out;
+
+	for (i = 0; i < len; i++) {
+#ifdef CONFIG_MAGIC_SYSRQ
+		if (sysrq_enabled) {
+			if (buf[i] == '\x0f') { /* ^O */
+				sysrq_requested = jiffies;
+				continue; /* don't print the sysrq key */
+			} else if (sysrq_requested) {
+				unsigned long sysrq_timeout =
+					sysrq_requested + HZ*2;
+				sysrq_requested = 0;
+				if (time_before(jiffies, sysrq_timeout)) {
+					spin_unlock_irqrestore(
+						&xencons_lock, flags);
+					handle_sysrq(
+						buf[i], regs, xencons_tty);
+					spin_lock_irqsave(
+						&xencons_lock, flags);
+					continue;
+				}
+			}
+		}
+#endif
+		tty_insert_flip_char(xencons_tty, buf[i], 0);
+	}
+	tty_flip_buffer_push(xencons_tty);
+
+ out:
+	spin_unlock_irqrestore(&xencons_lock, flags);
+}
+
+static void __xencons_tx_flush(void)
+{
+	int sent, sz, work_done = 0;
+
+	if (x_char) {
+		if (xen_start_info->flags & SIF_INITDOMAIN)
+			kcons_write_dom0(NULL, &x_char, 1);
+		else
+			while (x_char)
+				if (xencons_ring_send(&x_char, 1) == 1)
+					break;
+		x_char = 0;
+		work_done = 1;
+	}
+
+	while (wc != wp) {
+		sz = wp - wc;
+		if (sz > (wbuf_size - WBUF_MASK(wc)))
+			sz = wbuf_size - WBUF_MASK(wc);
+		if (xen_start_info->flags & SIF_INITDOMAIN) {
+			kcons_write_dom0(NULL, &wbuf[WBUF_MASK(wc)], sz);
+			wc += sz;
+		} else {
+			sent = xencons_ring_send(&wbuf[WBUF_MASK(wc)], sz);
+			if (sent == 0)
+				break;
+			wc += sent;
+		}
+		work_done = 1;
+	}
+
+	if (work_done && (xencons_tty != NULL)) {
+		wake_up_interruptible(&xencons_tty->write_wait);
+		if ((xencons_tty->flags & (1 << TTY_DO_WRITE_WAKEUP)) &&
+		    (xencons_tty->ldisc.write_wakeup != NULL))
+			(xencons_tty->ldisc.write_wakeup)(xencons_tty);
+	}
+}
+
+void xencons_tx(void)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&xencons_lock, flags);
+	__xencons_tx_flush();
+	spin_unlock_irqrestore(&xencons_lock, flags);
+}
+
+/* Privileged receive callback and transmit kicker. */
+static irqreturn_t xencons_priv_interrupt(int irq, void *dev_id,
+                                          struct pt_regs *regs)
+{
+	static char rbuf[16];
+	int         l;
+
+	while ((l = HYPERVISOR_console_io(CONSOLEIO_read, 16, rbuf)) > 0)
+		xencons_rx(rbuf, l, regs);
+
+	xencons_tx();
+
+	return IRQ_HANDLED;
+}
+
+static int xencons_write_room(struct tty_struct *tty)
+{
+	return wbuf_size - (wp - wc);
+}
+
+static int xencons_chars_in_buffer(struct tty_struct *tty)
+{
+	return wp - wc;
+}
+
+static void xencons_send_xchar(struct tty_struct *tty, char ch)
+{
+	unsigned long flags;
+
+	if (TTY_INDEX(tty) != 0)
+		return;
+
+	spin_lock_irqsave(&xencons_lock, flags);
+	x_char = ch;
+	__xencons_tx_flush();
+	spin_unlock_irqrestore(&xencons_lock, flags);
+}
+
+static void xencons_throttle(struct tty_struct *tty)
+{
+	if (TTY_INDEX(tty) != 0)
+		return;
+
+	if (I_IXOFF(tty))
+		xencons_send_xchar(tty, STOP_CHAR(tty));
+}
+
+static void xencons_unthrottle(struct tty_struct *tty)
+{
+	if (TTY_INDEX(tty) != 0)
+		return;
+
+	if (I_IXOFF(tty)) {
+		if (x_char != 0)
+			x_char = 0;
+		else
+			xencons_send_xchar(tty, START_CHAR(tty));
+	}
+}
+
+static void xencons_flush_buffer(struct tty_struct *tty)
+{
+	unsigned long flags;
+
+	if (TTY_INDEX(tty) != 0)
+		return;
+
+	spin_lock_irqsave(&xencons_lock, flags);
+	wc = wp = 0;
+	spin_unlock_irqrestore(&xencons_lock, flags);
+}
+
+static inline int __xencons_put_char(int ch)
+{
+	char _ch = (char)ch;
+	if ((wp - wc) == wbuf_size)
+		return 0;
+	wbuf[WBUF_MASK(wp++)] = _ch;
+	return 1;
+}
+
+static int xencons_write(
+	struct tty_struct *tty,
+	const unsigned char *buf,
+	int count)
+{
+	int i;
+	unsigned long flags;
+
+	if (TTY_INDEX(tty) != 0)
+		return count;
+
+	spin_lock_irqsave(&xencons_lock, flags);
+
+	for (i = 0; i < count; i++)
+		if (!__xencons_put_char(buf[i]))
+			break;
+
+	if (i != 0)
+		__xencons_tx_flush();
+
+	spin_unlock_irqrestore(&xencons_lock, flags);
+
+	return i;
+}
+
+static void xencons_put_char(struct tty_struct *tty, u_char ch)
+{
+	unsigned long flags;
+
+	if (TTY_INDEX(tty) != 0)
+		return;
+
+	spin_lock_irqsave(&xencons_lock, flags);
+	(void)__xencons_put_char(ch);
+	spin_unlock_irqrestore(&xencons_lock, flags);
+}
+
+static void xencons_flush_chars(struct tty_struct *tty)
+{
+	unsigned long flags;
+
+	if (TTY_INDEX(tty) != 0)
+		return;
+
+	spin_lock_irqsave(&xencons_lock, flags);
+	__xencons_tx_flush();
+	spin_unlock_irqrestore(&xencons_lock, flags);
+}
+
+static void xencons_wait_until_sent(struct tty_struct *tty, int timeout)
+{
+	unsigned long orig_jiffies = jiffies;
+
+	if (TTY_INDEX(tty) != 0)
+		return;
+
+	while (DRV(tty->driver)->chars_in_buffer(tty)) {
+		set_current_state(TASK_INTERRUPTIBLE);
+		schedule_timeout(1);
+		if (signal_pending(current))
+			break;
+		if (timeout && time_after(jiffies, orig_jiffies + timeout))
+			break;
+	}
+
+	set_current_state(TASK_RUNNING);
+}
+
+static int xencons_open(struct tty_struct *tty, struct file *filp)
+{
+	unsigned long flags;
+
+	if (TTY_INDEX(tty) != 0)
+		return 0;
+
+	spin_lock_irqsave(&xencons_lock, flags);
+	tty->driver_data = NULL;
+	if (xencons_tty == NULL)
+		xencons_tty = tty;
+	__xencons_tx_flush();
+	spin_unlock_irqrestore(&xencons_lock, flags);
+
+	return 0;
+}
+
+static void xencons_close(struct tty_struct *tty, struct file *filp)
+{
+	unsigned long flags;
+
+	if (TTY_INDEX(tty) != 0)
+		return;
+
+	if (tty->count == 1) {
+		tty->closing = 1;
+		tty_wait_until_sent(tty, 0);
+		if (DRV(tty->driver)->flush_buffer != NULL)
+			DRV(tty->driver)->flush_buffer(tty);
+		if (tty->ldisc.flush_buffer != NULL)
+			tty->ldisc.flush_buffer(tty);
+		tty->closing = 0;
+		spin_lock_irqsave(&xencons_lock, flags);
+		xencons_tty = NULL;
+		spin_unlock_irqrestore(&xencons_lock, flags);
+	}
+}
+
+static struct tty_operations xencons_ops = {
+	.open = xencons_open,
+	.close = xencons_close,
+	.write = xencons_write,
+	.write_room = xencons_write_room,
+	.put_char = xencons_put_char,
+	.flush_chars = xencons_flush_chars,
+	.chars_in_buffer = xencons_chars_in_buffer,
+	.send_xchar = xencons_send_xchar,
+	.flush_buffer = xencons_flush_buffer,
+	.throttle = xencons_throttle,
+	.unthrottle = xencons_unthrottle,
+	.wait_until_sent = xencons_wait_until_sent,
+};
+
+static int __init xencons_init(void)
+{
+	int rc;
+
+	if (xen_init() < 0)
+		return -ENODEV;
+
+	if (xc_mode == XC_OFF)
+		return 0;
+
+	xencons_ring_init();
+
+	xencons_driver = alloc_tty_driver((xc_mode == XC_SERIAL) ?
+					  1 : MAX_NR_CONSOLES);
+	if (xencons_driver == NULL)
+		return -ENOMEM;
+
+	DRV(xencons_driver)->name            = "xencons";
+	DRV(xencons_driver)->major           = TTY_MAJOR;
+	DRV(xencons_driver)->type            = TTY_DRIVER_TYPE_SERIAL;
+	DRV(xencons_driver)->subtype         = SERIAL_TYPE_NORMAL;
+	DRV(xencons_driver)->init_termios    = tty_std_termios;
+	DRV(xencons_driver)->flags           =
+		TTY_DRIVER_REAL_RAW |
+		TTY_DRIVER_RESET_TERMIOS |
+		TTY_DRIVER_NO_DEVFS;
+	DRV(xencons_driver)->termios         = xencons_termios;
+	DRV(xencons_driver)->termios_locked  = xencons_termios_locked;
+
+	if (xc_mode == XC_SERIAL) {
+		DRV(xencons_driver)->name        = "ttyS";
+		DRV(xencons_driver)->minor_start = 64 + xc_num;
+		DRV(xencons_driver)->name_base   = 0 + xc_num;
+	} else {
+		DRV(xencons_driver)->name        = "tty";
+		DRV(xencons_driver)->minor_start = xc_num;
+		DRV(xencons_driver)->name_base   = xc_num;
+	}
+
+	tty_set_operations(xencons_driver, &xencons_ops);
+
+	if ((rc = tty_register_driver(DRV(xencons_driver))) != 0) {
+		printk("WARNING: Failed to register Xen virtual "
+		       "console driver as '%s%d'\n",
+		       DRV(xencons_driver)->name,
+		       DRV(xencons_driver)->name_base);
+		put_tty_driver(xencons_driver);
+		xencons_driver = NULL;
+		return rc;
+	}
+
+	tty_register_device(xencons_driver, 0, NULL);
+
+	if (xen_start_info->flags & SIF_INITDOMAIN) {
+		xencons_priv_irq = bind_virq_to_irqhandler(
+			VIRQ_CONSOLE,
+			0,
+			xencons_priv_interrupt,
+			0,
+			"console",
+			NULL);
+		BUG_ON(xencons_priv_irq < 0);
+	}
+
+	printk("Xen virtual console successfully installed as %s%d\n",
+	       DRV(xencons_driver)->name,
+	       DRV(xencons_driver)->name_base );
+
+	return 0;
+}
+
+module_init(xencons_init);
drivers/xen/console/xencons_ring.c new file +115 -0
@@ -0,0 +1,115 @@
+#include <linux/version.h>
+#include <linux/module.h>
+#include <linux/errno.h>
+#include <linux/signal.h>
+#include <linux/sched.h>
+#include <linux/interrupt.h>
+#include <linux/tty.h>
+#include <linux/tty_flip.h>
+#include <linux/serial.h>
+#include <linux/major.h>
+#include <linux/ptrace.h>
+#include <linux/ioport.h>
+#include <linux/mm.h>
+#include <linux/slab.h>
+
+#include <asm/hypervisor.h>
+#include <xen/evtchn.h>
+#include <xen/xencons.h>
+#include <linux/wait.h>
+#include <linux/interrupt.h>
+#include <linux/sched.h>
+#include <linux/err.h>
+#include <xen/interface/io/console.h>
+
+static int xencons_irq;
+
+static inline struct xencons_interface *xencons_interface(void)
+{
+	return mfn_to_virt(xen_start_info->console_mfn);
+}
+
+static inline void notify_daemon(void)
+{
+	/* Use evtchn: this is called early, before irq is set up. */
+	notify_remote_via_evtchn(xen_start_info->console_evtchn);
+}
+
+int xencons_ring_send(const char *data, unsigned len)
+{
+	int sent = 0;
+	struct xencons_interface *intf = xencons_interface();
+	XENCONS_RING_IDX cons, prod;
+
+	cons = intf->out_cons;
+	prod = intf->out_prod;
+	mb();
+	BUG_ON((prod - cons) > sizeof(intf->out));
+
+	while ((sent < len) && ((prod - cons) < sizeof(intf->out)))
+		intf->out[MASK_XENCONS_IDX(prod++, intf->out)] = data[sent++];
+
+	wmb();
+	intf->out_prod = prod;
+
+	notify_daemon();
+
+	return sent;
+}
+
+static irqreturn_t handle_input(int irq, void *unused, struct pt_regs *regs)
+{
+	struct xencons_interface *intf = xencons_interface();
+	XENCONS_RING_IDX cons, prod;
+
+	cons = intf->in_cons;
+	prod = intf->in_prod;
+	mb();
+	BUG_ON((prod - cons) > sizeof(intf->in));
+
+	while (cons != prod) {
+		xencons_rx(intf->in+MASK_XENCONS_IDX(cons,intf->in), 1, regs);
+		cons++;
+	}
+
+	mb();
+	intf->in_cons = cons;
+
+	notify_daemon();
+
+	xencons_tx();
+
+	return IRQ_HANDLED;
+}
+
+int xencons_ring_init(void)
+{
+	int err;
+
+	if (xencons_irq)
+		unbind_from_irqhandler(xencons_irq, NULL);
+	xencons_irq = 0;
+
+	if (!xen_start_info->console_evtchn)
+		return 0;
+
+	err = bind_evtchn_to_irqhandler(
+		xen_start_info->console_evtchn,
+		handle_input, 0, "xencons", NULL);
+	if (err <= 0) {
+		printk(KERN_ERR "XEN console request irq failed %i\n", err);
+		return err;
+	}
+
+	xencons_irq = err;
+
+	/* In case we have in-flight data after save/restore... */
+	notify_daemon();
+
+	return 0;
+}
+
+void xencons_resume(void)
+{
+	(void)xencons_ring_init();
+}
include/xen/xencons.h new file +14 -2
@@ -0,0 +1,14 @@
+#ifndef __ASM_XENCONS_H__
+#define __ASM_XENCONS_H__
+
+void xencons_force_flush(void);
+void xencons_resume(void);
+
+/* Interrupt work hooks. Receive data, or kick data out. */
+void xencons_rx(char *buf, unsigned len, struct pt_regs *regs);
+void xencons_tx(void);
+
+int xencons_ring_init(void);
+int xencons_ring_send(const char *data, unsigned len);
+
+#endif /* __ASM_XENCONS_H__ */
 
--
 
------------------------------------------------------------------------

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.osdl.org/mailman/listinfo/virtualization

#84 lore raw

Re: [RFC PATCH 29/35] Add the Xen virtual console driver.

Quoted 3 lines · Anthony Liguori

This has always seemed a bit wrong to me and makes a number of things kind of awkward (like a virtual video driver).

It would seem better me to treat this driver as what it really is, a virtual serial device. It adds a little bit of additional work to the userspace tools (they just have to make sure to pass console=ttyS0) but it seems worth it.

That already works pretty much (userspace tools have to pass xencons=ttyS as well as console=ttyS0).

Quoted 1 line

We could also solve the tty[0-9] problem by implementing a proper console driver that could use multiple virtual serial devices if we wanted to go that route.

Yes, that could live alongside this driver.

Quoted 1 line

Another option would be to just emulate a serial driver. The console driver isn't really performance critical. It seems to me that it's a bit unnecessary to even bother paravirtualizing the console device when it's so easy to emulate.

Easy except that Xen can't implement the 'console backend', or at least not easily. The console bits need to end up in management virtual machine's user space. We'd have to do something skanky like the current HVM qemu model.

-- Keir

#85 lore raw

Re: [RFC PATCH 29/35] Add the Xen virtual console driver.

Quoted 11 lines · Keir Fraser
Quoted 3 lines · Anthony Liguori

This has always seemed a bit wrong to me and makes a number of things kind of awkward (like a virtual video driver).

It would seem better me to treat this driver as what it really is, a virtual serial device. It adds a little bit of additional work to the userspace tools (they just have to make sure to pass console=ttyS0) but it seems worth it.

That already works pretty much (userspace tools have to pass xencons=ttyS as well as console=ttyS0).

Quoted 1 line

We could also solve the tty[0-9] problem by implementing a proper console driver that could use multiple virtual serial devices if we wanted to go that route.

Yes, that could live alongside this driver.

This is going to come up in the next release if we try to merge the framebuffer stuff.

Quoted 3 lines
Quoted 1 line

Another option would be to just emulate a serial driver. The console driver isn't really performance critical. It seems to me that it's a bit unnecessary to even bother paravirtualizing the console device when it's so easy to emulate.

Easy except that Xen can't implement the 'console backend', or at least not easily. The console bits need to end up in management virtual machine's user space. We'd have to do something skanky like the current HVM qemu model.

We could make an exception for console devices and just have a small buffer in the hypervisor for console input/output (similar to the emergency console). The biggest advantage to doing this IMHO is that it makes guest OS's a bit easier to port. The console already is treated as an exception since it is setup in start_info (instead of through XenBus) so this is not that awful I think.

Regards,

Anthony Liguori

Quoted 2 lines

-- Keir

#86 lore raw

RFC PATCH 30/35 Add generic_page_range() function

In reply to Chris Wright

Add a new mm function generic_page_range() which applies a given function to every pte in a given virtual address range in a given mm structure. This is a generic alternative to cut-and-pasting the Linux idiomatic pagetable walking code in every place that a sequence of PTEs must be accessed.

Although this interface is intended to be useful in a wide range of situations, it is currently used specifically by several Xen subsystems, for example: to ensure that pagetables have been allocated for a virtual address range, and to construct batched special pagetable update requests to map I/O memory (in ioremap()).

---
 include/linux/mm.h |    5 ++
 mm/memory.c        |   90 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 95 insertions(+)
include/linux/mm.h +5 -0
@@ -1013,6 +1013,11 @@ struct page *follow_page(struct vm_area_
 #define FOLL_GET	0x04	/* do get_page on page */
 #define FOLL_ANON	0x08	/* give ZERO_PAGE if no pgtable */
 
+typedef int (*pte_fn_t)(pte_t *pte, struct page *pte_page, unsigned long addr,
+                        void *data);
+extern int generic_page_range(struct mm_struct *mm, unsigned long address,
+                              unsigned long size, pte_fn_t fn, void *data);
+
 #ifdef CONFIG_PROC_FS
 void vm_stat_account(struct mm_struct *, unsigned long, struct file *, long);
 #else
mm/memory.c +90 -0

96 lines not on this page · raw message

--

#87 lore raw

Re: [RFC PATCH 30/35] Add generic_page_range() function

Quoted 6 lines · Chris Wright

Add a new mm function generic_page_range() which applies a given function to every pte in a given virtual address range in a given mm structure. This is a generic alternative to cut-and-pasting the Linux idiomatic pagetable walking code in every place that a sequence of PTEs must be accessed.

Quoted 6 lines

Although this interface is intended to be useful in a wide range of situations, it is currently used specifically by several Xen subsystems, for example: to ensure that pagetables have been allocated for a virtual address range, and to construct batched special pagetable update requests to map I/O memory (in ioremap()).

This interface is great, and highly useful. But it doesn't seem to be able to work on native hardware, as it doesn't support large pages.

Zach

#88 lore raw

Re: [Xen-devel] Re: [RFC PATCH 30/35] Add generic_page_range() function

Quoted 8 lines · Zachary Amsden
Quoted 6 lines

Although this interface is intended to be useful in a wide range of situations, it is currently used specifically by several Xen subsystems, for example: to ensure that pagetables have been allocated for a virtual address range, and to construct batched special pagetable update requests to map I/O memory (in ioremap()).

This interface is great, and highly useful. But it doesn't seem to be able to work on native hardware, as it doesn't support large pages.

Ah, good point. I'll add that.

-- Keir

#89 lore raw

Re: [RFC PATCH 30/35] Add generic_page_range() function

In reply to Chris Wright
Quoted 6 lines · Chris Wright

Add a new mm function generic_page_range() which applies a given function to every pte in a given virtual address range in a given mm structure. This is a generic alternative to cut-and-pasting the Linux idiomatic pagetable walking code in every place that a sequence of PTEs must be accessed.

Quoted 6 lines

Although this interface is intended to be useful in a wide range of situations, it is currently used specifically by several Xen subsystems, for example: to ensure that pagetables have been allocated for a virtual address range, and to construct batched special pagetable update requests to map I/O memory (in ioremap()).

I raised the idea when we were tossing around ideas for the page table walking crapectomy. Of course it was rejected due to use of the indirect function, however I gues it makes sense for code outside mm/

Couple of issues with the current code though:

firstly, the name.

secondly, I think you confuse our (confusing) terminology: the page that holds pte_ts is not the pte_page, the pte_page is the page that a pte points to

lastly, you don't allow any control over the type of pages that are
walked: this could well be unusably slow for some cases. At least you should proably design the interface so we can iterate over present, not present, all, etc so it becomes widely usable. Normally I'd say to wait until users come up but in this case the function isn't a speed demon anyway, and you also don't want to give people any excuses not to use it.

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#90 lore raw

Re: [RFC PATCH 30/35] Add generic_page_range() function

Quoted 3 lines · Nick Piggin

Couple of issues with the current code though:

firstly, the name.

Okay, can you suggest a better one? That's the best I could come up with that wasn't long winded. :-)

Quoted 3 lines

secondly, I think you confuse our (confusing) terminology: the page that holds pte_ts is not the pte_page, the pte_page is the page that a pte points to

What should we call it? Essentially we want to be able to get the physical address of a PTE in some cases, and passing struct page pointer seemed the best way to be able to derive that. I can rename it to something else vaguely plausible if the only problem is the semantic clash with Linux's idiomatic use of pte_page.

Quoted 7 lines

lastly, you don't allow any control over the type of pages that are
walked: this could well be unusably slow for some cases. At least you should proably design the interface so we can iterate over present, not present, all, etc so it becomes widely usable. Normally I'd say to wait until users come up but in this case the function isn't a speed demon anyway, and you also don't want to give people any excuses not to use it.

You mean iterate only over PTEs that are already present, or only those that were *not* previously present, or all (present and non-present)? Is that really useful? If so then yes, it's not hard to add.

Thanks,
Keir

#91 lore raw

Re: [RFC PATCH 30/35] Add generic_page_range() function

Quoted 1 line · Keir Fraser

Okay, can you suggest a better one? That's the best I could come up with that wasn't long winded.

How about apply_to_page_range()?

Quoted 6 lines
Quoted 4 lines

secondly, I think you confuse our (confusing) terminology: the page that holds pte_ts is not the pte_page, the pte_page is the page that a pte points to

What should we call it? Essentially we want to be able to get the physical address of a PTE in some cases, and passing struct page pointer seemed the best way to be able to derive that. I can rename it to something else vaguely plausible if the only problem is the semantic clash with Linux's idiomatic use of pte_page.

Looks like pmd_page is correct?

-- Keir

#92 lore raw

Re: [RFC PATCH 30/35] Add generic_page_range() function

Quoted 8 lines · Keir Fraser
Quoted 1 line · Keir Fraser

Okay, can you suggest a better one? That's the best I could come up with that wasn't long winded.

How about apply_to_page_range()?

That would be better.

Quoted 11 lines
Quoted 7 lines
Quoted 4 lines

secondly, I think you confuse our (confusing) terminology: the page that holds pte_ts is not the pte_page, the pte_page is the page that a pte points to

What should we call it? Essentially we want to be able to get the physical address of a PTE in some cases, and passing struct page pointer seemed the best way to be able to derive that. I can rename it to something else vaguely plausible if the only problem is the semantic clash with Linux's idiomatic use of pte_page.

Looks like pmd_page is correct?

Yes... although maybe you could just pass the 'pmd_t *'? That's what a lot of the mm/memory.c code does.

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#93 lore raw

Re: [RFC PATCH 30/35] Add generic_page_range() function

In reply to Keir Fraser
Quoted 11 lines · Keir Fraser
Quoted 7 lines

lastly, you don't allow any control over the type of pages that are
walked: this could well be unusably slow for some cases. At least you should proably design the interface so we can iterate over present, not present, all, etc so it becomes widely usable. Normally I'd say to wait until users come up but in this case the function isn't a speed demon anyway, and you also don't want to give people any excuses not to use it.

You mean iterate only over PTEs that are already present, or only those that were *not* previously present, or all (present and non-present)? Is that really useful? If so then yes, it's not hard to add.

Yes. If you look at all the code that walks pagetables (including those that just operate on a single pte) in arch/ and other places, there is a fair diversity. A function called generic_page_range() would surely be able to replace all those ;)

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#94 lore raw

RFC PATCH 31/35 Add Xen grant table support

In reply to Chris Wright

Add Xen 'grant table' driver which allows granting of access to selected local memory pages by other virtual machines and, symmetrically, the mapping of remote memory pages which other virtual machines have granted access to.

This driver is a prerequisite for many of the Xen virtual device drivers, which grant the 'device driver domain' restricted and temporary access to only those memory pages that are currently involved in I/O operations.

---
 arch/i386/mach-xen/Makefile |    2 
 arch/i386/mach-xen/gnttab.c |  417 ++++++++++++++++++++++++++++++++++++++++++++
 include/xen/gnttab.h        |  102 ++++++++++
 3 files changed, 520 insertions(+), 1 deletion(-)
arch/i386/mach-xen/Makefile +1 -1
@@ -4,6 +4,6 @@
 
 extra-y				:= head.o
 
-obj-y				:= setup.o features.o
+obj-y				:= setup.o features.o gnttab.o
  
 setup-y				:= ../mach-default/setup.o
arch/i386/mach-xen/gnttab.c new file +417 -0

417 lines not on this page · raw message

include/xen/gnttab.h new file +102 -0

102 lines not on this page · raw message

--

#95 lore raw

Re: [RFC PATCH 31/35] Add Xen grant table support

Quoted 3 lines

+ * This file may be distributed separately from the Linux kernel, or
+ * incorporated into other software packages, subject to the following license:

please again fix the license

Quoted 5 lines
arch/i386/mach-xen/gnttab.c new file +5 -0
+EXPORT_SYMBOL(gnttab_grant_foreign_access);
+EXPORT_SYMBOL(gnttab_end_foreign_access_ref);
+EXPORT_SYMBOL(gnttab_end_foreign_access);
+EXPORT_SYMBOL(gnttab_query_foreign_access);
+EXPORT_SYMBOL(gnttab_grant_foreign_transfer);

arch/i386/mach-xen/gnttab.c line 61

Quoted 10 lines
arch/i386/mach-xen/gnttab.c new file +10 -0
+EXPORT_SYMBOL(gnttab_end_foreign_transfer_ref);
+EXPORT_SYMBOL(gnttab_end_foreign_transfer);
+EXPORT_SYMBOL(gnttab_alloc_grant_references);
+EXPORT_SYMBOL(gnttab_free_grant_references);
+EXPORT_SYMBOL(gnttab_free_grant_reference);
+EXPORT_SYMBOL(gnttab_claim_grant_reference);
+EXPORT_SYMBOL(gnttab_release_grant_reference);
+EXPORT_SYMBOL(gnttab_request_free_callback);
+EXPORT_SYMBOL(gnttab_grant_foreign_access_ref);
+EXPORT_SYMBOL(gnttab_grant_foreign_transfer_ref);

and consider these as _GPL exports

arch/i386/mach-xen/gnttab.c line 343

Quoted 10 lines
arch/i386/mach-xen/gnttab.c new file +10 -0
+#ifndef __ia64__
+static int map_pte_fn(pte_t *pte, struct page *pte_page,
+		      unsigned long addr, void *data)
+{
+	unsigned long **frames = (unsigned long **)data;
+
+	set_pte_at(&init_mm, addr, pte, pfn_pte((*frames)[0], PAGE_KERNEL));
+	(*frames)++;
+	return 0;
+}

looks to me the wrong ifdef for a file in arch/i386... please fix

#96 lore raw

Re: [RFC PATCH 31/35] Add Xen grant table support

Quoted 5 lines · * Arjan van de Ven ([email protected]) wrote:
Quoted 3 lines

+ * This file may be distributed separately from the Linux kernel, or
+ * incorporated into other software packages, subject to the following license:

please again fix the license

*nod*

Quoted 9 lines
Quoted 9 lines
arch/i386/mach-xen/gnttab.c new file +9 -0
+EXPORT_SYMBOL(gnttab_grant_foreign_access);
+EXPORT_SYMBOL(gnttab_end_foreign_access_ref);
+EXPORT_SYMBOL(gnttab_end_foreign_access);
+EXPORT_SYMBOL(gnttab_query_foreign_access);
+EXPORT_SYMBOL(gnttab_grant_foreign_transfer);
+EXPORT_SYMBOL(gnttab_end_foreign_transfer_ref);
+EXPORT_SYMBOL(gnttab_end_foreign_transfer);
+EXPORT_SYMBOL(gnttab_alloc_grant_references);
+EXPORT_SYMBOL(gnttab_free_grant_references);
Quoted 8 lines

arch/i386/mach-xen/gnttab.c line 61

Quoted 6 lines
arch/i386/mach-xen/gnttab.c new file +6 -0
+EXPORT_SYMBOL(gnttab_free_grant_reference);
+EXPORT_SYMBOL(gnttab_claim_grant_reference);
+EXPORT_SYMBOL(gnttab_release_grant_reference);
+EXPORT_SYMBOL(gnttab_request_free_callback);
+EXPORT_SYMBOL(gnttab_grant_foreign_access_ref);
+EXPORT_SYMBOL(gnttab_grant_foreign_transfer_ref);

and consider these as _GPL exports

Yes, I meant to note that as one of the known issues. License clarity and _GPL exports.

Quoted 4 lines
Quoted 4 lines
arch/i386/mach-xen/gnttab.c new file +4 -0
+#ifndef __ia64__
+static int map_pte_fn(pte_t *pte, struct page *pte_page,
+		      unsigned long addr, void *data)
+{
Quoted 8 lines

arch/i386/mach-xen/gnttab.c line 343

Quoted 6 lines
arch/i386/mach-xen/gnttab.c new file +6 -0
+	unsigned long **frames = (unsigned long **)data;
+
+	set_pte_at(&init_mm, addr, pte, pfn_pte((*frames)[0], PAGE_KERNEL));
+	(*frames)++;
+	return 0;
+}

looks to me the wrong ifdef for a file in arch/i386... please fix

Yes, the reuse of grant table amongst each Xen supported architecture suggests it's in the wrong location.

thanks,
-chris

#97 lore raw

RFC PATCH 32/35 Add Xen driver utility functions.

In reply to Chris Wright

Allocate/destroy a 'vmalloc' VM area: alloc_vm_area and free_vm_area The alloc function ensures that page tables are constructed for the region of kernel virtual address space and mapped into init_mm.

Lock an area so that PTEs are accessible in the current address space:
lock_vm_area and unlock_vm_area
The lock function prevents context switches to a lazy mm that doesn't have the area mapped into its page tables. It also ensures that the page tables are mapped into the current mm by causing the page fault handler to copy the page directory pointers from init_mm into the current mm.

---
 drivers/xen/Makefile      |    2 +
 drivers/xen/util.c        |   70 ++++++++++++++++++++++++++++++++++++++++++++++
 include/xen/driver_util.h |   16 ++++++++++
 3 files changed, 88 insertions(+)
drivers/xen/Makefile +2 -0
@@ -1,2 +1,4 @@
 
+obj-y	+= util.o
+
 obj-y	+= console/
drivers/xen/util.c new file +70 -0

70 lines not on this page · raw message

include/xen/driver_util.h new file +16 -0
@@ -0,0 +1,16 @@
+
+#ifndef __ASM_XEN_DRIVER_UTIL_H__
+#define __ASM_XEN_DRIVER_UTIL_H__
+
+#include <linux/config.h>
+#include <linux/vmalloc.h>
+
+/* Allocate/destroy a 'vmalloc' VM area. */
+extern struct vm_struct *alloc_vm_area(unsigned long size);
+extern void free_vm_area(struct vm_struct *area);
+
+/* Lock an area so that PTEs are accessible in the current address space. */
+extern void lock_vm_area(struct vm_struct *area);
+extern void unlock_vm_area(struct vm_struct *area);
+
+#endif /* __ASM_XEN_DRIVER_UTIL_H__ */

--

#98 lore raw

Re: [RFC PATCH 32/35] Add Xen driver utility functions.

drivers/xen/util.c line 62

Quoted 6 lines · Chris Wright
drivers/xen/util.c new file +6 -0
+	/*
+	 * Ensure that the page tables are mapped into the current mm. The
+	 * page-fault path will copy the page directory pointers from init_mm.
+	 */
+	for (i = 0; i < area->size; i += PAGE_SIZE)
+		(void)__get_user(c, (char __user *)area->addr + i);

It would be easier and simpler to just copy the pgd manually.

Iirc Jan B. had a patch for a vmalloc_sync() function for this purpose, maybe that one could be used.

-Andi

#99 lore raw

RFC PATCH 33/35 Add the Xenbus sysfs and virtual device hotplug driver.

In reply to Chris Wright

This communicates with the machine control software via a registry residing in a controlling virtual machine. This allows dynamic creation, destruction and modification of virtual device configurations (network devices, block devices and CPUS, to name some examples).

---
 arch/i386/mach-xen/reboot.c        |    6 
 drivers/xen/Makefile               |    1 
 drivers/xen/xenbus/Makefile        |    7 
 drivers/xen/xenbus/xenbus_client.c |  392 +++++++++++++
 drivers/xen/xenbus/xenbus_comms.c  |  201 ++++++
 drivers/xen/xenbus/xenbus_comms.h  |   40 +
 drivers/xen/xenbus/xenbus_probe.c  | 1076 +++++++++++++++++++++++++++++++++++++
 drivers/xen/xenbus/xenbus_xs.c     |  827 ++++++++++++++++++++++++++++
 include/xen/xenbus.h               |  288 +++++++++
 9 files changed, 2832 insertions(+), 6 deletions(-)
arch/i386/mach-xen/reboot.c +0 -6
@@ -11,9 +11,7 @@
 #include <asm/mmu_context.h>
 #include <xen/evtchn.h>
 #include <asm/hypervisor.h>
-#ifdef CONFIG_XEN_XENBUS
 #include <xen/xenbus.h>
-#endif
 #include <linux/cpu.h>
 #include <linux/kthread.h>
 #include <xen/xencons.h>
@@ -71,14 +69,11 @@ EXPORT_SYMBOL(machine_power_off);
  * Stop/pickle callback handling.
  */
 
-#ifdef CONFIG_XEN_XENBUS
 /* Ignore multiple shutdown requests. */
 static int shutting_down = SHUTDOWN_INVALID;
 static void __shutdown_handler(void *unused);
 static DECLARE_WORK(shutdown_work, __shutdown_handler, NULL);
-#endif
 
-#ifdef CONFIG_XEN_XENBUS
 static int shutdown_process(void *__unused)
 {
 	static char *envp[] = { "HOME=/", "TERM=linux",
@@ -262,4 +257,3 @@ static int __init setup_shutdown_event(v
 }
 
 subsys_initcall(setup_shutdown_event);
-#endif
drivers/xen/Makefile +1 -0
@@ -2,3 +2,4 @@
 obj-y	+= util.o
 
 obj-y	+= console/
+obj-y	+= xenbus/
drivers/xen/xenbus/Makefile new file +7 -0
@@ -0,0 +1,7 @@
+obj-y	+= xenbus.o
+
+xenbus-objs =
+xenbus-objs += xenbus_client.o
+xenbus-objs += xenbus_comms.o
+xenbus-objs += xenbus_xs.o
+xenbus-objs += xenbus_probe.o
drivers/xen/xenbus/xenbus_client.c new file +392 -0

392 lines not on this page · raw message

drivers/xen/xenbus/xenbus_comms.c new file +201 -0

201 lines not on this page · raw message

drivers/xen/xenbus/xenbus_comms.h new file +40 -0
@@ -0,0 +1,40 @@
+/*
+ * Private include for xenbus communications.
+ * 
+ * Copyright (C) 2005 Rusty Russell, IBM Corporation
+ *
+ * This file may be distributed separately from the Linux kernel, or
+ * incorporated into other software packages, subject to the following license:
+ * 
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this source file (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use, copy, modify,
+ * merge, publish, distribute, sublicense, and/or sell copies of the Software,
+ * and to permit persons to whom the Software is furnished to do so, subject to
+ * the following conditions:
+ * 
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ * 
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ */
+
+#ifndef _XENBUS_COMMS_H
+#define _XENBUS_COMMS_H
+
+int xs_init(void);
+int xb_init_comms(void);
+
+/* Low level routines. */
+int xb_write(const void *data, unsigned len);
+int xb_read(void *data, unsigned len);
+int xs_input_avail(void);
+extern wait_queue_head_t xb_waitq;
+
+#endif /* _XENBUS_COMMS_H */
drivers/xen/xenbus/xenbus_probe.c new file +1076 -0

1,076 lines not on this page · raw message

drivers/xen/xenbus/xenbus_xs.c new file +827 -0

827 lines not on this page · raw message

include/xen/xenbus.h new file +288 -0

288 lines not on this page · raw message

--

#100 lore raw

Re: [RFC PATCH 33/35] Add the Xenbus sysfs and virtual device hotplug driver.

drivers/xen/xenbus/xenbus_probe.c line 54

Quoted 2 lines
drivers/xen/xenbus/xenbus_probe.c new file +2 -0
+
+extern struct semaphore xenwatch_mutex;

you know, there is this thing called "struct mutex", please use that instead; don't add new deprecated semaphore's when using them as mutex

Quoted 1 line

+#define streq(a, b) (strcmp((a), (b)) == 0)

ehhh why?

Quoted 1 line

+EXPORT_SYMBOL(xenbus_register_frontend);

please consider _GPL

drivers/xen/xenbus/xenbus_probe.c line 495

Quoted 4 lines
drivers/xen/xenbus/xenbus_probe.c new file +4 -0
+static void xenbus_dev_free(struct xenbus_device *xendev)
+{
+	kfree(xendev);
+}

why this wrapper ?

Quoted 11 lines
drivers/xen/xenbus/xenbus_probe.c new file +11 -0
+
+/* Simplified asprintf. */
+char *kasprintf(const char *fmt, ...)
+{
+	va_list ap;
+	unsigned int len;
+	char *p, dummy[1];
+
+	va_start(ap, fmt);
+	/* FIXME: vsnprintf has a bug, NULL should work */
+	len = vsnprintf(dummy, 0, fmt, ap);

drivers/xen/xenbus/xenbus_probe.c line 522

Quoted 10 lines
drivers/xen/xenbus/xenbus_probe.c new file +10 -0
+	va_end(ap);
+
+	p = kmalloc(len + 1, GFP_KERNEL);
+	if (!p)
+		return NULL;
+	va_start(ap, fmt);
+	vsprintf(p, fmt, ap);
+	va_end(ap);
+	return p;
+}

this should take a gfp mask most likely

drivers/xen/xenbus/xenbus_probe.c line 1030

Quoted 4 lines
drivers/xen/xenbus/xenbus_probe.c new file +4 -0
+
+		/* We don't refcnt properly, so set reserved on page.
+		 * (this allocation is permanent) */
+		SetPageReserved(virt_to_page(page));

you may want to reconsider this given the "recent" vm changes

Review

Quoted 1 line
(quoted diff) +1 -0
+	down(&xs_state.request_mutex);

please use a real mutex instead

#101 lore raw

Re: [RFC PATCH 33/35] Add the Xenbus sysfs and virtual device hotplug driver.

Quoted 4 lines · Arjan van de Ven

drivers/xen/xenbus/xenbus_probe.c line 505

Quoted 2 lines
drivers/xen/xenbus/xenbus_probe.c new file +2 -0
+char *kasprintf(const char *fmt, ...)
+{

this should take a gfp mask most likely

It's internal to xenbus driver, where it is used only in safe kernel thread contexts where GFP_KERNEL is okay. Adding a gfp mask would just clutter the interface I think.

-- Keir

#102 lore raw

RFC PATCH 34/35 Add the Xen virtual network device driver.

In reply to Chris Wright

The network device frontend driver allows the kernel to access network devices exported exported by a virtual machine containing a physical network device driver.

---
 drivers/net/Kconfig             |    2 
 drivers/xen/Kconfig.net         |   14 
 drivers/xen/Makefile            |    3 
 drivers/xen/net_driver_util.c   |   57 +
 drivers/xen/netfront/Makefile   |    4 
 drivers/xen/netfront/netfront.c | 1500 ++++++++++++++++++++++++++++++++++++++++
 include/xen/net_driver_util.h   |   46 +
 7 files changed, 1626 insertions(+)
drivers/net/Kconfig +2 -0
@@ -2326,6 +2326,8 @@ source "drivers/atm/Kconfig"
 
 source "drivers/s390/net/Kconfig"
 
+source "drivers/xen/Kconfig.net"
+
 config ISERIES_VETH
 	tristate "iSeries Virtual Ethernet driver support"
 	depends on PPC_ISERIES
drivers/xen/Makefile +3 -0
@@ -1,5 +1,8 @@
 
+obj-y	+= net_driver_util.o
 obj-y	+= util.o
 
 obj-y	+= console/
 obj-y	+= xenbus/
+
+obj-$(CONFIG_XEN_NETDEV_FRONTEND)	+= netfront/
drivers/xen/Kconfig.net new file +14 -0
@@ -0,0 +1,14 @@
+menu "Xen network device drivers"
+        depends on NETDEVICES && XEN
+
+config XEN_NETDEV_FRONTEND
+	tristate "Network-device frontend driver"
+	depends on XEN
+	default y
+	help
+	  The network-device frontend driver allows the kernel to access
+	  network interfaces within another guest OS. Unless you are building a
+	  dedicated device-driver domain, or your master control domain
+	  (domain 0), then you almost certainly want to say Y here.
+
+endmenu
drivers/xen/net_driver_util.c new file +57 -0

57 lines not on this page · raw message

drivers/xen/netfront/Makefile new file +4 -0
@@ -0,0 +1,4 @@
+
+obj-$(CONFIG_XEN_NETDEV_FRONTEND)	:= xennet.o
+
+xennet-objs := netfront.o
drivers/xen/netfront/netfront.c new file +1500 -0

1,500 lines not on this page · raw message

include/xen/net_driver_util.h new file +46 -0

46 lines not on this page · raw message

--

#103 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

drivers/xen/netfront/netfront.c line 61

Quoted 4 lines
drivers/xen/netfront/netfront.c new file +3 -0
+#ifdef CONFIG_XEN_BALLOON
+#include <xen/balloon.h>
+#endif

such ifdefs are butt-ugly and should be inside the header at least

drivers/xen/netfront/netfront.c line 75

Quoted 3 lines
drivers/xen/netfront/netfront.c new file +3 -0
+#ifndef __GFP_NOWARN
+#define __GFP_NOWARN 0
+#endif

eehhh why?

Quoted 1 line

+#define alloc_xen_skb(_l) __dev_alloc_skb((_l), GFP_ATOMIC|__GFP_NOWARN)

why does xen need it's own alloc_skb variant?

drivers/xen/netfront/netfront.c line 87

Quoted 3 lines
drivers/xen/netfront/netfront.c new file +3 -0
+static unsigned long rx_pfn_array[NET_RX_RING_SIZE];
+static struct multicall_entry rx_mcl[NET_RX_RING_SIZE+1];
+static struct mmu_update rx_mmu[NET_RX_RING_SIZE];

does this mean you can only have 1 network interface? that sounds silly

drivers/xen/netfront/netfront.c line 151

Quoted 9 lines
drivers/xen/netfront/netfront.c new file +9 -0
+
+/* Access macros for acquiring freeing slots in {tx,rx}_skbs[]. */
+#define ADD_ID_TO_FREELIST(_list, _id)			\
+	(_list)[(_id)] = (_list)[0];			\
+	(_list)[0]     = (void *)(unsigned long)(_id);
+#define GET_ID_FROM_FREELIST(_list)				\
+	({ unsigned long _id = (unsigned long)(_list)[0];	\
+	   (_list)[0]  = (_list)[_id];				\
+	   (unsigned short)_id; })

these should be inlines at leas

drivers/xen/netfront/netfront.c line 418

Quoted 3 lines
drivers/xen/netfront/netfront.c new file +3 -0
+
+/** Send a packet on a net device to encourage switches to learn the
+ * MAC. We send a fake ARP request.

why is this inside a driver?????

drivers/xen/netfront/netfront.c line 676

Quoted 11 lines
drivers/xen/netfront/netfront.c new file +11 -0
+	if (unlikely((((unsigned long)skb->data & ~PAGE_MASK) + skb->len) >=
+		     PAGE_SIZE)) {
+		struct sk_buff *nskb;
+		if (unlikely((nskb = alloc_xen_skb(skb->len)) == NULL))
+			goto drop;
+		skb_put(nskb, skb->len);
+		memcpy(nskb->data, skb->data, skb->len);
+		nskb->dev = skb->dev;
+		dev_kfree_skb(skb);
+		skb = nskb;
+	}

hmm this smells fishy

drivers/xen/netfront/netfront.c line 1359

Quoted 2 lines
drivers/xen/netfront/netfront.c new file +2 -0
+
+static int xennet_proc_read(

why is this device driver adding new own /proc crud ?

#104 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

In reply to Chris Wright
Quoted 3 lines · Chris Wright

The network device frontend driver allows the kernel to access network devices exported exported by a virtual machine containing a physical network device driver.

I'd suggest cc'ing network related patches to [email protected].

- James

Signature · 2 lines
James Morris
<[email protected]>
#105 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

In reply to Chris Wright
Quoted 8 lines · On Tue, 21 Mar 2006 22:31:14 -0800 Chris Wright <[email protected]> wrote:

The network device frontend driver allows the kernel to access network devices exported exported by a virtual machine containing a physical network device driver.

---

In addition to earlier comments:

* should handle multiple interfaces, not just one static.

* the whole notifier fake arp call back is fugly

* use sysfs instead of /proc to handle the simple rxbuf_min,max stuff or use ethtool ring parameters

* proc module_owner not set, so ref counting not done. Moot if you switch to sysfs.

* since you add ethtool for checksum,
why not some basic stuff like driver info and carrier state?

* xen_net_read_mac seems like it could just be put inside driver.

* minor
use free_netdev() rather than kfree in error path
don't bother initializing .next/.priority in notifier_block if it is just 0

* what about mtu change?

* what about mac address change?

* Could you support scatter gather?

#106 lore raw

RFC PATCH 35/35 Add Xen virtual block device driver.

In reply to Chris Wright

The block device frontend driver allows the kernel to access block devices exported exported by a virtual machine containing a physical block device driver.

---
 drivers/block/Kconfig           |    2 
 drivers/xen/Kconfig.blk         |   14 
 drivers/xen/Makefile            |    1 
 drivers/xen/blkfront/Makefile   |    5 
 drivers/xen/blkfront/blkfront.c |  812 ++++++++++++++++++++++++++++++++++++++++
 drivers/xen/blkfront/block.h    |  152 +++++++
 drivers/xen/blkfront/vbd.c      |  316 +++++++++++++++
 7 files changed, 1302 insertions(+)
drivers/block/Kconfig +2 -0
@@ -450,6 +450,8 @@ config CDROM_PKTCDVD_WCACHE
 
 source "drivers/s390/block/Kconfig"
 
+source "drivers/xen/Kconfig.blk"
+
 config ATA_OVER_ETH
 	tristate "ATA over Ethernet support"
 	depends on NET
drivers/xen/Makefile +1 -0
@@ -5,4 +5,5 @@ obj-y	+= util.o
 obj-y	+= console/
 obj-y	+= xenbus/
 
+obj-$(CONFIG_XEN_BLKDEV_FRONTEND)	+= blkfront/
 obj-$(CONFIG_XEN_NETDEV_FRONTEND)	+= netfront/
drivers/xen/Kconfig.blk new file +14 -0
@@ -0,0 +1,14 @@
+menu "Xen block device drivers"
+        depends on XEN
+
+config XEN_BLKDEV_FRONTEND
+	tristate "Block device frontend driver"
+	depends on XEN
+	default y
+	help
+	  The block device frontend driver allows the kernel to access block
+	  devices exported from a device driver virtual machine. Unless you
+	  are building a dedicated device driver virtual machine, then you
+	  almost certainly want to say Y here.
+
+endmenu
drivers/xen/blkfront/Makefile new file +5 -0
@@ -0,0 +1,5 @@
+
+obj-$(CONFIG_XEN_BLKDEV_FRONTEND)	:= xenblk.o
+
+xenblk-objs := blkfront.o vbd.o
+
drivers/xen/blkfront/blkfront.c new file +812 -0

812 lines not on this page · raw message

drivers/xen/blkfront/block.h new file +152 -0

152 lines not on this page · raw message

drivers/xen/blkfront/vbd.c new file +316 -0

316 lines not on this page · raw message

--

#107 lore raw

Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

Quoted 17 lines · Chris Wright

The block device frontend driver allows the kernel to access block devices exported exported by a virtual machine containing a physical block device driver.

drivers/xen/blkfront/vbd.c new file +11 -0
+
+static struct xlbd_type_info xlbd_ide_type = {
+	.partn_shift = 6,
+	.disks_per_major = 2,
+	.devname = "ide",
+	.diskname = "hd",
+};
+
+static struct xlbd_type_info xlbd_scsi_type = {
+	.partn_shift = 4,
+	.disks_per_major = 16,

drivers/xen/blkfront/vbd.c line 68

Quoted 10 lines
drivers/xen/blkfront/vbd.c new file +10 -0
+	.devname = "sd",
+	.diskname = "sd",
+};
+
+static struct xlbd_type_info xlbd_vbd_type = {
+	.partn_shift = 4,
+	.disks_per_major = 16,
+	.devname = "xvd",
+	.diskname = "xvd",
+};

This is another thing that has always put me off. The virtual block device driver has the ability to masquerade as other types of block devices. It actually claims to be an IDE or SCSI device allocating the appropriate major/minor numbers.

This seems to be pretty evil and creating interesting failure conditions for users who load IDE or SCSI modules. I've seen it trip up a number of people in the past. I think we should only ever use the major number that was actually allocated to us.

Regards,

Anthony Liguori

#108 lore raw

Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

Quoted 9 lines · Anthony Liguori

This is another thing that has always put me off. The virtual block device driver has the ability to masquerade as other types of block devices. It actually claims to be an IDE or SCSI device allocating the appropriate major/minor numbers.

This seems to be pretty evil and creating interesting failure conditions for users who load IDE or SCSI modules. I've seen it trip up a number of people in the past. I think we should only ever use the major number that was actually allocated to us.

Exactly. We vetoed crap like that in the ibm vio drivers already so it was removed before merging those drivers.

#109 lore raw

Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

In reply to Anthony Liguori

Hi,

Quoted 8 lines

drivers/xen/blkfront/vbd.c line 63

Quoted 3 lines
drivers/xen/blkfront/vbd.c new file +3 -0
+static struct xlbd_type_info xlbd_ide_type = {
+static struct xlbd_type_info xlbd_scsi_type = {
+static struct xlbd_type_info xlbd_vbd_type = {

This is another thing that has always put me off. The virtual block device driver has the ability to masquerade as other types of block devices. It actually claims to be an IDE or SCSI device allocating the appropriate major/minor numbers.

It's useful sometimes. Debian/sarge for example doesn't work with xvd block devices. At least not out-of-the-box, it needs some manual tweaks. Probably it also is handy when moving real machines into an virtual environment. I don't think it should be dropped.

Most modern udev-based distros work just fine with xvd though.

Quoted 4 lines

This seems to be pretty evil and creating interesting failure conditions for users who load IDE or SCSI modules. I've seen it trip up a number of people in the past. I think we should only ever use the major number that was actually allocated to us.

Print a big fat warning? And also change the example config files in the xen source tree to use xvda not hda to advertize them more than we do right now. I think lots of users don't even know about the xvd devices ...

cheers,

Gerd

Signature · 3 lines
Gerd 'just married' Hoffmann <[email protected]>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg
#110 lore raw
In reply to Chris Wright
Quoted 2 lines · Chris Wright

Xen also provides support for running directly on native hardware.

Can someone elaborate on this? Does this mean a Xen guest can run on bare metal?

Is there code available to make this work (it doesn't seem contained in this patchset)? Has any performance analysis been done?

The numbers that have been posted with the VMI patches suggest that some rather tricky stuff is required to achieve native performance when running a guest on bare metal. If this is not the case, it would be very interesting to know because it seems to be the hairiest part of the VMI patches.

Otherwise, if we want to support Xen guests on bare metal, it seems we would have to change things in the subarch code a bit to do something similar to VMI.

Regards,

Anthony Liguori

Quoted 61 lines

The following patch series provides the minimal support required to launch Xen paravirtual guests on standard x86 hardware running the Xen hypervisor. These patches effectively port the Linux kernel to run on the platform interface provided by Xen. This port is done as an i386 subarch.

With these patches you will be able to launch an unprivileged guest running the modified Linux kernel and unmodified userspace. This guest is x86, UP only, runs in shadow translated mode, and has no direct access to hardware. This simplifies the patchset to the minimum functionality needed to support a paravirtualized guest. It's worth noting that a fair amount of this patchset deals with paravirtualizing I/O, not just CPU-only. The additional missing functionality is primarily about full SMP support, optimizations such as direct writable page tables, and the management interface. Those refinements will be posted later.

At a high-level, the patches provide the following:

- Kconfig and Makefile changes required to support Xen
- subarch changes to allow more platform functionality to be implemented by an i386 subarch
- Xen subarch implementation
- start of day code for running in the hypervisor provided environment (paging enabled)
- basic Xen drivers to provide a fully functional guest

The Xen platform API encapsulates the following types of requirements:

- idt, gdt, ldt (descriptor table handling)
- cr2, fpu_taskswitch, debug registers (privileged register handling)
- mmu (page table, tlb, and cache handling)
- memory reservations
- time and timer
- vcpu (init, up/down vcpu)
- schedule (processor yield, shutdown, etc)
- event channel (generalized virtual interrupt handling)
- grant table (shared memory interface for high speed interdomain communication)
- block device I/O
- network device I/O
- console device I/O
- Xen feature map
- Xen version info

Thanks to those who provided early feedback to this patchset: Andi Kleen, Gerd Hoffmann, Jan Beulich, Rik van Riel, Stephen Tweedie, and the Xen team. And thanks to the Xen community: AMD, HP, IBM, Intel, Novell, Red Hat, Virtual Iron, XenSource -- see Xen changelog for full details.

Known issues:

CodingStyle conformance is still in progress
/proc interface needs to be replaced with something more appropriate entry.S cleanups are possible

------------------------------------------------------------------------

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.osdl.org/mailman/listinfo/virtualization

#111 lore raw
Quoted 5 lines · * Anthony Liguori ([email protected]) wrote:
Quoted 1 line · Chris Wright

Xen also provides support for running directly on native hardware.

Can someone elaborate on this? Does this mean a Xen guest can run on bare metal?

Yes. See the Xen code for running the kernel in ring0 with Xen (supervisor_mode_kenel). The hypercall_page is conditionally filled with hypercall traps or direct calls basically.

Quoted 2 lines

Is there code available to make this work (it doesn't seem contained in this patchset)? Has any performance analysis been done?

I don't have any numbers.

Quoted 5 lines

The numbers that have been posted with the VMI patches suggest that some rather tricky stuff is required to achieve native performance when running a guest on bare metal. If this is not the case, it would be very interesting to know because it seems to be the hairiest part of the VMI patches.

It is a hairy part of VMI. They've done a nice job of handling the native case, and have interseting plans for improving the non-native case (inline where possible). One of the differences is things that don't actually require hypercalls are already inline w/ Xen. So it's conceivable that the performance hit is smaller than what VMI found without carefully inlining native code.

Quoted 3 lines

Otherwise, if we want to support Xen guests on bare metal, it seems we would have to change things in the subarch code a bit to do something similar to VMI.

It's a different approach.

thanks,
-chris

#112 lore raw
Quoted 4 lines · Chris Wright
Quoted 2 lines · * Anthony Liguori ([email protected]) wrote:

Chris Wright wrote:

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

Xen also provides support for running directly on native hardware.

Can someone elaborate on this? Does this mean a Xen guest can run on bare metal?

Yes. See the Xen code for running the kernel in ring0 with Xen (supervisor_mode_kenel). The hypercall_page is conditionally filled with hypercall traps or direct calls basically.

Cool! I didn't realize the supervisor_mode_kernel code was in the Xen tree code already.

Regards,

Anthony Liguori

Quoted 25 lines
Quoted 2 lines

Is there code available to make this work (it doesn't seem contained in this patchset)? Has any performance analysis been done?

I don't have any numbers.

Quoted 2 lines

The numbers that have been posted with the VMI patches suggest that some rather tricky stuff is required to achieve native performance when running a guest on bare metal. If this is not the case, it would be very interesting to know because it seems to be the hairiest part of the VMI patches.

It is a hairy part of VMI. They've done a nice job of handling the native case, and have interseting plans for improving the non-native case (inline where possible). One of the differences is things that don't actually require hypercalls are already inline w/ Xen. So it's conceivable that the performance hit is smaller than what VMI found without carefully inlining native code.

Quoted 2 lines

Otherwise, if we want to support Xen guests on bare metal, it seems we would have to change things in the subarch code a bit to do something similar to VMI.

It's a different approach.

thanks,
-chris