Skip to contentSkip to conversation

Xen i386 paravirtualization support

RFC PATCH In review Chris Wright 32 participants 185 messages May 9, 2006 – May 18, 2006

Subsystems (7)
lore
Share & tools
lore
#0 lore raw

cover Xen i386 paravirtualization support

Unlike full virtualization in which the virtual machine provides the same platform interface as running natively on the hardware, paravirtualization requires modification to the guest operating system to work with the platform interface provided by the hypervisor.

Xen was designed with performance in mind. Calls to the hypervisor are minimized, batched if necessary, and non-critical codepaths are left unmodified in the case where the privileged instruction can be trapped and emulated by the hypervisor. The Xen API is designed to be OS agnostic and has had Linux, NetBSD, FreeBSD, Solaris, Plan9 and Netware ported to it. Xen also provides support for running directly on native hardware.

The following patch series provides the minimal support required to launch Xen paravirtual guests on standard x86 hardware running the Xen hypervisor. These patches effectively port the Linux kernel to run on the platform interface provided by Xen. This port is done as an i386 subarch. In the future, we will break this patchset up to place the general infrastrcture and subarch bits that may have common users at the beginning of the series, ripe for picking off and pushing upstream.

With these patches you will be able to launch an unprivileged guest running the modified Linux kernel and unmodified userspace. This guest is x86, UP only, runs in shadow translated mode, and has no direct access to hardware. This simplifies the patchset to the minimum functionality needed to support a paravirtualized guest. It's worth noting that a fair amount of this patchset deals with paravirtualizing I/O, not just CPU-only. The additional missing functionality is primarily about full SMP support, optimizations such as direct writable page tables, and the management interface.

At a high-level, the patches provide the following:

- Kconfig and Makefile changes required to support Xen
- subarch changes to allow more platform functionality to be implemented by an i386 subarch
- Xen subarch implementation
- start of day code for running in the hypervisor provided environment (paging enabled)
- basic Xen drivers to provide a fully functional guest

The Xen platform API encapsulates the following types of requirements:

- idt, gdt, ldt (descriptor table handling)
- cr2, fpu_taskswitch, debug registers (privileged register handling)
- mmu (page table, tlb, and cache handling)
- memory reservations
- time and timer
- vcpu (init, up/down vcpu)
- schedule (processor yield, shutdown, etc)
- event channel (generalized virtual interrupt handling)
- grant table (shared memory interface for high speed interdomain communication)
- block device I/O
- network device I/O
- console device I/O
- Xen feature map
- Xen version info

Thanks to all have reviewed earlier versions of these patches. -chris
--

#1 lore raw

RFC PATCH 1/35 Add XEN config options and disable unsupported config options.

The XEN config option is selected from the i386 subarch menu by choosing the X86_XEN "Xen-compatible" subarch.

The XEN_SHADOW_MODE option defines the memory virtualization mode for the kernel -- with it enabled, the kernel expects the hypervisor to perform translation between pseudo-physical and machine addresses on its behalf.

The disabled config options are:
- DOUBLEFAULT: are trapped by Xen and not virtualized
- HZ: defaults to 100 in Xen VMs
- Power management: not supported in unprivileged VMs
- SMP: not supported in this set of patches
- X86_{UP,LOCAL,IO}_APIC: not supported in unprivileged VMs

---
 arch/i386/Kconfig       |   18 ++++++++++++++----
 arch/i386/Kconfig.debug |    1 +
 drivers/xen/Kconfig     |   21 +++++++++++++++++++++
 kernel/Kconfig.hz       |    4 ++--
 kernel/Kconfig.preempt  |    1 +
 5 files changed, 39 insertions(+), 6 deletions(-)
arch/i386/Kconfig +14 -4
@@ -55,6 +55,7 @@ menu "Processor type and features"
 
 config SMP
 	bool "Symmetric multi-processing support"
+	depends on !X86_XEN
 	---help---
 	  This enables support for systems with more than one CPU. If you have
 	  a system with only one CPU, like most personal computers, say N. If
@@ -91,6 +92,12 @@ config X86_PC
 	help
 	  Choose this option if your computer is a standard PC or compatible.
 
+config X86_XEN
+	bool "Xen-compatible"
+	help
+	  Choose this option if you plan to run this kernel on top of the
+	  Xen Hypervisor.
+
 config X86_ELAN
 	bool "AMD Elan"
 	help
@@ -193,6 +200,7 @@ source "arch/i386/Kconfig.cpu"
 
 config HPET_TIMER
 	bool "HPET Timer Support"
+	depends on !X86_XEN
 	help
 	  This enables the use of the HPET for the kernel's internal timer.
 	  HPET is the next generation timer replacing legacy 8254s.
@@ -244,7 +252,7 @@ source "kernel/Kconfig.preempt"
 
 config X86_UP_APIC
 	bool "Local APIC support on uniprocessors"
-	depends on !SMP && !(X86_VISWS || X86_VOYAGER)
+	depends on !SMP && !(X86_VISWS || X86_VOYAGER || X86_XEN)
 	help
 	  A local APIC (Advanced Programmable Interrupt Controller) is an
 	  integrated interrupt controller in the CPU. If you have a single-CPU
@@ -269,12 +277,12 @@ config X86_UP_IOAPIC
 
 config X86_LOCAL_APIC
 	bool
-	depends on X86_UP_APIC || ((X86_VISWS || SMP) && !X86_VOYAGER)
+	depends on X86_UP_APIC || ((X86_VISWS || SMP) && !(X86_VOYAGER || X86_XEN))
 	default y
 
 config X86_IO_APIC
 	bool
-	depends on X86_UP_IOAPIC || (SMP && !(X86_VISWS || X86_VOYAGER))
+	depends on X86_UP_IOAPIC || (SMP && !(X86_VISWS || X86_VOYAGER || X86_XEN))
 	default y
 
 config X86_VISWS_APIC
@@ -767,7 +775,7 @@ endmenu
 
 
 menu "Power management options (ACPI, APM)"
-	depends on !X86_VOYAGER
+	depends on !(X86_VOYAGER || X86_XEN)
 
 source kernel/power/Kconfig
 
@@ -1089,6 +1097,8 @@ source "security/Kconfig"
 
 source "crypto/Kconfig"
 
+source "drivers/xen/Kconfig"
+
 source "lib/Kconfig"
 
 #
arch/i386/Kconfig.debug +1 -0
@@ -84,6 +84,7 @@ config X86_MPPARSE
 config DOUBLEFAULT
 	default y
 	bool "Enable doublefault exception handler" if EMBEDDED
+	depends on !X86_XEN
 	help
           This option allows trapping of rare doublefault exceptions that
           would otherwise cause a system to silently reboot. Disabling this
kernel/Kconfig.hz +2 -2
@@ -3,7 +3,7 @@
 #
 
 choice
-	prompt "Timer frequency"
+	prompt "Timer frequency" if !XEN
 	default HZ_250
 	help
 	 Allows the configuration of the timer frequency. It is customary
@@ -40,7 +40,7 @@ endchoice
 
 config HZ
 	int
-	default 100 if HZ_100
+	default 100 if HZ_100 || XEN
 	default 250 if HZ_250
 	default 1000 if HZ_1000
 
kernel/Kconfig.preempt +1 -0
@@ -35,6 +35,7 @@ config PREEMPT_VOLUNTARY
 
 config PREEMPT
 	bool "Preemptible Kernel (Low-Latency Desktop)"
+	depends on !XEN
 	help
 	  This option reduces the latency of the kernel by making
 	  all kernel code (that is not executing in a critical section)
drivers/xen/Kconfig new file +21 -0
@@ -0,0 +1,21 @@
+#
+# This Kconfig describe xen options
+#
+
+mainmenu "Xen Configuration"
+
+config XEN
+	bool
+	default y if X86_XEN
+	help
+	  This is the Linux Xen port.
+
+if XEN
+
+config XEN_SHADOW_MODE
+	bool
+	default y
+	help
+	  Fakes out a shadow mode kernel
+
+endif

--

#2 lore raw

Re: [RFC PATCH 01/35] Add XEN config options and disable unsupported config options.

arch/i386/Kconfig config X86_UP_IOAPIC · line 286

Quoted 10 lines · Chris Wright

...

arch/i386/Kconfig +1 -1
...
 config X86_IO_APIC
 	bool
-	depends on X86_UP_IOAPIC || (SMP && !(X86_VISWS || X86_VOYAGER))
+	depends on X86_UP_IOAPIC || (SMP && !(X86_VISWS || X86_VOYAGER || X86_XEN))
 	default y
...

<nitpick>not required</nitpick>

Quoted 12 lines
kernel/Kconfig.hz +1 -1
@@ -3,7 +3,7 @@
 #
 
 choice
-	prompt "Timer frequency"
+	prompt "Timer frequency" if !XEN
 	default HZ_250
 	help
 	 Allows the configuration of the timer frequency. It is customary

kernel/Kconfig.hz endchoice · line 45

Quoted 8 lines
kernel/Kconfig.hz +1 -1
@@ -40,7 +40,7 @@ endchoice
 
 config HZ
 	int
-	default 100 if HZ_100
+	default 100 if HZ_100 || XEN
 	default 250 if HZ_250
 	default 1000 if HZ_1000
...

Why?

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#3 lore raw

Re: [RFC PATCH 01/35] Add XEN config options and disable unsupported config options.

Quoted 27 lines · Adrian Bunk

arch/i386/Kconfig config X86_UP_IOAPIC · line 286

Quoted 10 lines · Chris Wright

...

arch/i386/Kconfig +1 -1
...
 config X86_IO_APIC
 	bool
-	depends on X86_UP_IOAPIC || (SMP && !(X86_VISWS || X86_VOYAGER))
+	depends on X86_UP_IOAPIC || (SMP && !(X86_VISWS || X86_VOYAGER || X86_XEN))
 	default y
...

<nitpick>not required</nitpick>

Quoted 13 lines
kernel/Kconfig.hz +1 -1
@@ -3,7 +3,7 @@
 #
 
 choice
-	prompt "Timer frequency"
+	prompt "Timer frequency" if !XEN
 	default HZ_250
 	help
 	 Allows the configuration of the timer frequency. It is customary
@@ -40,7 +40,7 @@ endchoice
 
Quoted 9 lines

kernel/Kconfig.hz endchoice · line 45

Quoted 7 lines
kernel/Kconfig.hz +1 -1
 config HZ
 	int
-	default 100 if HZ_100
+	default 100 if HZ_100 || XEN
 	default 250 if HZ_250
 	default 1000 if HZ_1000
...

Why?

Guessing, but its probably to limit the number of parahypervisor calls.

Ed Tomlinson

#4 lore raw

Re: [RFC PATCH 01/35] Add XEN config options and disable unsupported config options.

In reply to Adrian Bunk
Quoted 14 lines · Adrian Bunk
Quoted 13 lines · Chris Wright
kernel/Kconfig.hz +1 -1
@@ -3,7 +3,7 @@
 #
 
 choice
-	prompt "Timer frequency"
+	prompt "Timer frequency" if !XEN
 	default HZ_250
 	help
 	 Allows the configuration of the timer frequency. It is customary
@@ -40,7 +40,7 @@ endchoice
 
Quoted 9 lines

kernel/Kconfig.hz endchoice · line 45

Quoted 7 lines
kernel/Kconfig.hz +1 -1
 config HZ
 	int
-	default 100 if HZ_100
+	default 100 if HZ_100 || XEN
 	default 250 if HZ_250
 	default 1000 if HZ_1000
...

Why?

Because the hypervisor sends timer interrupts to the guest at a rate of 100 Hz while the guest is running. We might add support to have an adjustable rate in the future but so far 100 Hz has worked quite well for us.

christian

#5 lore raw

Re: [RFC PATCH 01/35] Add XEN config options and disable unsupported config options.

In reply to Adrian Bunk
Quoted 13 lines · * Adrian Bunk ([email protected]) wrote:

arch/i386/Kconfig config X86_UP_IOAPIC · line 286

Quoted 10 lines · Chris Wright

...

arch/i386/Kconfig +1 -1
...
 config X86_IO_APIC
 	bool
-	depends on X86_UP_IOAPIC || (SMP && !(X86_VISWS || X86_VOYAGER))
+	depends on X86_UP_IOAPIC || (SMP && !(X86_VISWS || X86_VOYAGER || X86_XEN))
 	default y
...

<nitpick>not required</nitpick>

True, although SMP is just disabled in this patchset which is a subset of full Xen support.

thanks,
-chris

#6 lore raw

Re: [RFC PATCH 01/35] Add XEN config options and disable unsupported config options.

In reply to Chris Wright
Quoted 38 lines · Chris Wright

plain text document attachment (config-xen)
The XEN config option is selected from the i386 subarch menu by choosing the X86_XEN "Xen-compatible" subarch.

The XEN_SHADOW_MODE option defines the memory virtualization mode for the kernel -- with it enabled, the kernel expects the hypervisor to perform translation between pseudo-physical and machine addresses on its behalf.

The disabled config options are:
- DOUBLEFAULT: are trapped by Xen and not virtualized
- HZ: defaults to 100 in Xen VMs
- Power management: not supported in unprivileged VMs
- SMP: not supported in this set of patches
- X86_{UP,LOCAL,IO}_APIC: not supported in unprivileged VMs

---
 arch/i386/Kconfig       |   18 ++++++++++++++----
 arch/i386/Kconfig.debug |    1 +
 drivers/xen/Kconfig     |   21 +++++++++++++++++++++
 kernel/Kconfig.hz       |    4 ++--
 kernel/Kconfig.preempt  |    1 +
 5 files changed, 39 insertions(+), 6 deletions(-)
arch/i386/Kconfig +1 -0
@@ -55,6 +55,7 @@ menu "Processor type and features"
 
 config SMP
 	bool "Symmetric multi-processing support"
+	depends on !X86_XEN
 	---help---
 	  This enables support for systems with more than one CPU. If you have
 	  a system with only one CPU, like most personal computers, say N. If

arch/i386/Kconfig config X86_PC · line 100

Quoted 9 lines
arch/i386/Kconfig +6 -0
@@ -91,6 +92,12 @@ config X86_PC
 	help
 	  Choose this option if your computer is a standard PC or compatible.
 
+config X86_XEN
+	bool "Xen-compatible"
+	help
+	  Choose this option if you plan to run this kernel on top of the
+	  Xen Hypervisor.
+

Couldn't you just add "depends on !SMP && .." to the config X86_XEN block ?

Daniel

#7 lore raw

Re: [RFC PATCH 01/35] Add XEN config options and disable unsupported config options.

Quoted 21 lines · Daniel Walker
Quoted 20 lines · Chris Wright

The disabled config options are:
- DOUBLEFAULT: are trapped by Xen and not virtualized
- HZ: defaults to 100 in Xen VMs
- Power management: not supported in unprivileged VMs
- SMP: not supported in this set of patches
- X86_{UP,LOCAL,IO}_APIC: not supported in unprivileged VMs

+++ linus-2.6/arch/i386/Kconfig

arch/i386/Kconfig +1 -0
@@ -55,6 +55,7 @@ menu "Processor type and features"
 
 config SMP
 	bool "Symmetric multi-processing support"
+	depends on !X86_XEN
 	---help---
 	  This enables support for systems with more than one CPU. If you have
 	  a system with only one CPU, like most personal computers, say N. If
@@ -91,6 +92,12 @@ config X86_PC
 	help
 	  Choose this option if your computer is a standard PC or compatible.
 
Quoted 9 lines

arch/i386/Kconfig config X86_PC · line 100

Quoted 6 lines
arch/i386/Kconfig +6 -0
+config X86_XEN
+	bool "Xen-compatible"
+	help
+	  Choose this option if you plan to run this kernel on top of the
+	  Xen Hypervisor.
+

Couldn't you just add "depends on !SMP && .." to the config X86_XEN block ?

I guess you could, but it would make it rather non-obvious and tedious to enable X86_XEN then, wouldn't it?

christian

#8 lore raw

Re: [RFC PATCH 01/35] Add XEN config options and disable unsupported config options.

Quoted 11 lines · Christian Limpach
Quoted 10 lines

arch/i386/Kconfig config X86_PC · line 100

Quoted 7 lines
arch/i386/Kconfig +6 -0
+config X86_XEN
+	bool "Xen-compatible"
+	help
+	  Choose this option if you plan to run this kernel on top of the
+	  Xen Hypervisor.
+

Couldn't you just add "depends on !SMP && .." to the config X86_XEN block ?

Quoted 2 lines

I guess you could, but it would make it rather non-obvious and tedious to enable X86_XEN then, wouldn't it?

I guess that true .. Might be better just to support SMP then ..

Daniel

#9 lore raw

Re: [RFC PATCH 01/35] Add XEN config options and disable unsupported config options.

Quoted 1 line · * Daniel Walker ([email protected]) wrote:

I guess that true .. Might be better just to support SMP then ..

Yes, and of course Xen does. This is just the smallest functional set of patches to get discussion, so the SMP bits were dropped for now.

thanks,
-chris

#10 lore raw

Re: [RFC PATCH 01/35] Add XEN config options and disable unsupported config options.

In reply to Chris Wright
Quoted 2 lines · Chris Wright

The XEN config option is selected from the i386 subarch menu by choosing the X86_XEN "Xen-compatible" subarch.

I really dislike all these negative option checks.

I think it would be better if you defined a positive symbol like BARE_METAL (better name?)and define that only for the non XEN case.

-Andi

#11 lore raw

Re: [Xen-devel] [RFC PATCH 01/35] Add XEN config options and disable unsupported config options.

In reply to Chris Wright
Quoted 3 lines · Chris Wright

plain text document atodiad (config-xen)
The XEN config option is selected from the i386 subarch menu by choosing the X86_XEN "Xen-compatible" subarch.

You need this as well. At least if I read the logic right with regards to Xen and traps it is safe to do the following (although probably not safe to run Xen on such a physical system anyway)

i386/Kconfig.cpu renamed from i386/Kconfig.cpu~ +1 -1
@@ -251,7 +251,7 @@
 
 config X86_F00F_BUG
 	bool
-	depends on M586MMX || M586TSC || M586 || M486 || M386
+	depends on ( M586MMX || M586TSC || M586 || M486 || M386 ) && !XEN
 	default y
 
 config X86_WP_WORKS_OK
#12 lore raw

Re: [Xen-devel] [RFC PATCH 01/35] Add XEN config options and disable unsupported config options.

Quoted 8 lines · Alan Cox
Quoted 3 lines · Chris Wright

plain text document atodiad (config-xen)
The XEN config option is selected from the i386 subarch menu by choosing the X86_XEN "Xen-compatible" subarch.

You need this as well. At least if I read the logic right with regards to Xen and traps it is safe to do the following (although probably not safe to run Xen on such a physical system anyway)

Yes. In our tree, we have a config option which completely removes all the hardware idt table code (X86_NO_IDT) and stores the trap table as a table suitable to pass directly to the hypervisor.

That's not so useful if you want to build a kernel which can run both on a hypervisor and on native. I guess you would need to disable the X86_F00F_BUG code at runtime in such a kernel.

For the non-runtime case, I wonder if it's preferable to disable X86_F00F_BUG like you suggest or if it would be better to disable the cpu types listed?

christian

Quoted 16 lines
i386/Kconfig.cpu renamed from i386/Kconfig.cpu~ +1 -1
@@ -251,7 +251,7 @@
 
 config X86_F00F_BUG
 	bool
-	depends on M586MMX || M586TSC || M586 || M486 || M386
+	depends on ( M586MMX || M586TSC || M586 || M486 || M386 ) && !XEN
 	default y
 
 config X86_WP_WORKS_OK
#13 lore raw

RFC PATCH 2/35 Makefile support to build Xen subarch

In reply to Chris Wright

Use arch/i386/mach-xen when building Xen subarch. The separate subarchitecture allows us to hide details of interfacing with the hypervisor from i386 common code.

---
 arch/i386/Makefile          |    5 +++++
 arch/i386/mach-xen/Makefile |    7 +++++++
 2 files changed, 12 insertions(+)
arch/i386/Makefile +5 -0
@@ -71,6 +71,10 @@ mcore-$(CONFIG_X86_BIGSMP)	:= mach-defau
 mflags-$(CONFIG_X86_SUMMIT) := -Iinclude/asm-i386/mach-summit
 mcore-$(CONFIG_X86_SUMMIT)  := mach-default
 
+# Xen subarch support
+mflags-$(CONFIG_X86_XEN)	:= -Iinclude/asm-i386/mach-xen
+mcore-$(CONFIG_X86_XEN)		:= mach-xen
+
 # generic subarchitecture
 mflags-$(CONFIG_X86_GENERICARCH) := -Iinclude/asm-i386/mach-generic
 mcore-$(CONFIG_X86_GENERICARCH) := mach-default
@@ -99,6 +103,7 @@ drivers-$(CONFIG_PM)			+= arch/i386/powe
 
 CFLAGS += $(mflags-y)
 AFLAGS += $(mflags-y)
+CPPFLAGS += $(mflags-y)
 
 boot := arch/i386/boot
 
arch/i386/mach-xen/Makefile new file +7 -0
@@ -0,0 +1,7 @@
+#
+# Makefile for the linux kernel.
+#
+
+obj-y				:= setup.o
+
+setup-y				:= ../mach-default/setup.o

--

#14 lore raw

RFC PATCH 3/35 Add Xen interface header files

In reply to Chris Wright

Add Xen interface header files. These are taken fairly directly from the Xen tree and hence the style is not entirely in accordance with Linux guidelines. There is a tension between fitting with Linux coding rules and ease of maintenance.

---
 include/xen/interface/arch-x86_32.h   |  197 +++++++++++++++
 include/xen/interface/event_channel.h |  205 +++++++++++++++
 include/xen/interface/features.h      |   53 ++++
 include/xen/interface/grant_table.h   |  311 +++++++++++++++++++++++
 include/xen/interface/io/blkif.h      |   85 ++++++
 include/xen/interface/io/console.h    |   33 ++
 include/xen/interface/io/netif.h      |   84 ++++++
 include/xen/interface/io/ring.h       |  262 ++++++++++++++++++++
 include/xen/interface/io/xenbus.h     |   42 +++
 include/xen/interface/io/xs_wire.h    |   97 +++++++
 include/xen/interface/memory.h        |  155 +++++++++++
 include/xen/interface/physdev.h       |   71 +++++
 include/xen/interface/sched.h         |   87 ++++++
 include/xen/interface/vcpu.h          |  119 +++++++++
 include/xen/interface/version.h       |   70 +++++
 include/xen/interface/xen.h           |  441 ++++++++++++++++++++++++++++++++++
 16 files changed, 2312 insertions(+)
include/xen/interface/arch-x86_32.h new file +197 -0
@@ -0,0 +1,197 @@
+/******************************************************************************
+ * arch-x86_32.h
+ * 
+ * Guest OS interface to x86 32-bit Xen.
+ * 
+ * Copyright (c) 2004, K A Fraser
+ */
+
+#ifndef __XEN_PUBLIC_ARCH_X86_32_H__
+#define __XEN_PUBLIC_ARCH_X86_32_H__
+
+#ifdef __XEN__
+#define __DEFINE_GUEST_HANDLE(name, type) \
+    typedef struct { type *p; } __guest_handle_ ## name
+#else
+#define __DEFINE_GUEST_HANDLE(name, type) \
+    typedef type * __guest_handle_ ## name
+#endif
+
+#define DEFINE_GUEST_HANDLE_STRUCT(name) \
+	__DEFINE_GUEST_HANDLE(name, struct name)
+#define DEFINE_GUEST_HANDLE(name) __DEFINE_GUEST_HANDLE(name, name)
+#define GUEST_HANDLE(name)        __guest_handle_ ## name
+
+#ifndef __ASSEMBLY__
+/* Guest handles for primitive C types. */
+__DEFINE_GUEST_HANDLE(uchar, unsigned char);
+__DEFINE_GUEST_HANDLE(uint,  unsigned int);
+__DEFINE_GUEST_HANDLE(ulong, unsigned long);
+DEFINE_GUEST_HANDLE(char);
+DEFINE_GUEST_HANDLE(int);
+DEFINE_GUEST_HANDLE(long);
+DEFINE_GUEST_HANDLE(void);
+#endif
+
+/*
+ * SEGMENT DESCRIPTOR TABLES
+ */
+/*
+ * A number of GDT entries are reserved by Xen. These are not situated at the
+ * start of the GDT because some stupid OSes export hard-coded selector values
+ * in their ABI. These hard-coded values are always near the start of the GDT,
+ * so Xen places itself out of the way, at the far end of the GDT.
+ */
+#define FIRST_RESERVED_GDT_PAGE  14
+#define FIRST_RESERVED_GDT_BYTE  (FIRST_RESERVED_GDT_PAGE * 4096)
+#define FIRST_RESERVED_GDT_ENTRY (FIRST_RESERVED_GDT_BYTE / 8)
+
+/*
+ * These flat segments are in the Xen-private section of every GDT. Since these
+ * are also present in the initial GDT, many OSes will be able to avoid
+ * installing their own GDT.
+ */
+#define FLAT_RING1_CS 0xe019    /* GDT index 259 */
+#define FLAT_RING1_DS 0xe021    /* GDT index 260 */
+#define FLAT_RING1_SS 0xe021    /* GDT index 260 */
+#define FLAT_RING3_CS 0xe02b    /* GDT index 261 */
+#define FLAT_RING3_DS 0xe033    /* GDT index 262 */
+#define FLAT_RING3_SS 0xe033    /* GDT index 262 */
+
+#define FLAT_KERNEL_CS FLAT_RING1_CS
+#define FLAT_KERNEL_DS FLAT_RING1_DS
+#define FLAT_KERNEL_SS FLAT_RING1_SS
+#define FLAT_USER_CS    FLAT_RING3_CS
+#define FLAT_USER_DS    FLAT_RING3_DS
+#define FLAT_USER_SS    FLAT_RING3_SS
+
+/* And the trap vector is... */
+#define TRAP_INSTR "int $0x82"
+
+/*
+ * Virtual addresses beyond this are not modifiable by guest OSes. The 
+ * machine->physical mapping table starts at this address, read-only.
+ */
+#ifdef CONFIG_X86_PAE
+#define __HYPERVISOR_VIRT_START 0xF5800000
+#else
+#define __HYPERVISOR_VIRT_START 0xFC000000
+#endif
+
+#ifndef HYPERVISOR_VIRT_START
+#define HYPERVISOR_VIRT_START mk_unsigned_long(__HYPERVISOR_VIRT_START)
+#endif
+
+#ifndef machine_to_phys_mapping
+#define machine_to_phys_mapping ((unsigned long *)HYPERVISOR_VIRT_START)
+#endif
+
+/* Maximum number of virtual CPUs in multi-processor guests. */
+#define MAX_VIRT_CPUS 32
+
+#ifndef __ASSEMBLY__
+
+/*
+ * Send an array of these to HYPERVISOR_set_trap_table()
+ */
+#define TI_GET_DPL(_ti)      ((_ti)->flags & 3)
+#define TI_GET_IF(_ti)       ((_ti)->flags & 4)
+#define TI_SET_DPL(_ti,_dpl) ((_ti)->flags |= (_dpl))
+#define TI_SET_IF(_ti,_if)   ((_ti)->flags |= ((!!(_if))<<2))
+struct trap_info {
+    uint8_t       vector;  /* exception vector                              */
+    uint8_t       flags;   /* 0-3: privilege level; 4: clear event enable?  */
+    uint16_t      cs;      /* code selector                                 */
+    unsigned long address; /* code offset                                   */
+};
+DEFINE_GUEST_HANDLE_STRUCT(trap_info);
+
+struct cpu_user_regs {
+    uint32_t ebx;
+    uint32_t ecx;
+    uint32_t edx;
+    uint32_t esi;
+    uint32_t edi;
+    uint32_t ebp;
+    uint32_t eax;
+    uint16_t error_code;    /* private */
+    uint16_t entry_vector;  /* private */
+    uint32_t eip;
+    uint16_t cs;
+    uint8_t  saved_upcall_mask;
+    uint8_t  _pad0;
+    uint32_t eflags;        /* eflags.IF == !saved_upcall_mask */
+    uint32_t esp;
+    uint16_t ss, _pad1;
+    uint16_t es, _pad2;
+    uint16_t ds, _pad3;
+    uint16_t fs, _pad4;
+    uint16_t gs, _pad5;
+};
+DEFINE_GUEST_HANDLE_STRUCT(cpu_user_regs);
+
+typedef uint64_t tsc_timestamp_t; /* RDTSC timestamp */
+
+/*
+ * The following is all CPU context. Note that the fpu_ctxt block is filled 
+ * in by FXSAVE if the CPU has feature FXSR; otherwise FSAVE is used.
+ */
+struct vcpu_guest_context {
+    /* FPU registers come first so they can be aligned for FXSAVE/FXRSTOR. */
+    struct { char x[512]; } fpu_ctxt;       /* User-level FPU registers     */
+#define VGCF_I387_VALID (1<<0)
+#define VGCF_HVM_GUEST  (1<<1)
+#define VGCF_IN_KERNEL  (1<<2)
+    unsigned long flags;                    /* VGCF_* flags                 */
+    struct cpu_user_regs user_regs;         /* User-level CPU registers     */
+    struct trap_info trap_ctxt[256];        /* Virtual IDT                  */
+    unsigned long ldt_base, ldt_ents;       /* LDT (linear address, # ents) */
+    unsigned long gdt_frames[16], gdt_ents; /* GDT (machine frames, # ents) */
+    unsigned long kernel_ss, kernel_sp;     /* Virtual TSS (only SS1/SP1)   */
+    unsigned long ctrlreg[8];               /* CR0-CR7 (control registers)  */
+    unsigned long debugreg[8];              /* DB0-DB7 (debug registers)    */
+    unsigned long event_callback_cs;        /* CS:EIP of event callback     */
+    unsigned long event_callback_eip;
+    unsigned long failsafe_callback_cs;     /* CS:EIP of failsafe callback  */
+    unsigned long failsafe_callback_eip;
+    unsigned long vm_assist;                /* VMASST_TYPE_* bitmap */
+};
+DEFINE_GUEST_HANDLE_STRUCT(vcpu_guest_context);
+
+struct arch_shared_info {
+    unsigned long max_pfn;                  /* max pfn that appears in table */
+    /* Frame containing list of mfns containing list of mfns containing p2m. */
+    unsigned long pfn_to_mfn_frame_list_list;
+    unsigned long nmi_reason;
+};
+
+struct arch_vcpu_info {
+    unsigned long cr2;
+    unsigned long pad[5]; /* sizeof(struct vcpu_info) == 64 */
+};
+
+#endif /* !__ASSEMBLY__ */
+
+/*
+ * Prefix forces emulation of some non-trapping instructions.
+ * Currently only CPUID.
+ */
+#ifdef __ASSEMBLY__
+#define XEN_EMULATE_PREFIX .byte 0x0f,0x0b,0x78,0x65,0x6e ;
+#define XEN_CPUID          XEN_EMULATE_PREFIX cpuid
+#else
+#define XEN_EMULATE_PREFIX ".byte 0x0f,0x0b,0x78,0x65,0x6e ; "
+#define XEN_CPUID          XEN_EMULATE_PREFIX "cpuid"
+#endif
+
+#endif
+
+/*
+ * Local variables:
+ * mode: C
+ * c-set-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
include/xen/interface/event_channel.h new file +205 -0

205 lines not on this page · raw message

include/xen/interface/features.h new file +53 -0

53 lines not on this page · raw message

include/xen/interface/grant_table.h new file +311 -0

311 lines not on this page · raw message

include/xen/interface/io/blkif.h new file +85 -0

85 lines not on this page · raw message

include/xen/interface/io/console.h new file +33 -0
@@ -0,0 +1,33 @@
+/******************************************************************************
+ * console.h
+ * 
+ * Console I/O interface for Xen guest OSes.
+ * 
+ * Copyright (c) 2005, Keir Fraser
+ */
+
+#ifndef __XEN_PUBLIC_IO_CONSOLE_H__
+#define __XEN_PUBLIC_IO_CONSOLE_H__
+
+typedef uint32_t XENCONS_RING_IDX;
+
+#define MASK_XENCONS_IDX(idx, ring) ((idx) & (sizeof(ring)-1))
+
+struct xencons_interface {
+    char in[1024];
+    char out[2048];
+    XENCONS_RING_IDX in_cons, in_prod;
+    XENCONS_RING_IDX out_cons, out_prod;
+};
+
+#endif /* __XEN_PUBLIC_IO_CONSOLE_H__ */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-set-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
include/xen/interface/io/netif.h new file +84 -0

84 lines not on this page · raw message

include/xen/interface/io/ring.h new file +262 -0

262 lines not on this page · raw message

include/xen/interface/io/xenbus.h new file +42 -0

42 lines not on this page · raw message

include/xen/interface/io/xs_wire.h new file +97 -0

97 lines not on this page · raw message

include/xen/interface/memory.h new file +155 -0

155 lines not on this page · raw message

include/xen/interface/physdev.h new file +71 -0

71 lines not on this page · raw message

include/xen/interface/sched.h new file +87 -0

87 lines not on this page · raw message

include/xen/interface/vcpu.h new file +119 -0

119 lines not on this page · raw message

include/xen/interface/version.h new file +70 -0

70 lines not on this page · raw message

include/xen/interface/xen.h new file +441 -0

441 lines not on this page · raw message

--

#15 lore raw

Re: [RFC PATCH 03/35] Add Xen interface header files

This clearly doesn't belong:

Quoted 9 lines

+/*
+ * Local variables:
+ * mode: C
+ * c-set-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */

????

Quoted 9 lines

+/*
+ * Local variables:
+ * mode: C
+ * c-set-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */

???

And the rest of them.

#16 lore raw

Re: [RFC PATCH 03/35] Add Xen interface header files

Quoted 12 lines · Martin J. Bligh

This clearly doesn't belong:

Quoted 9 lines

+/*
+ * Local variables:
+ * mode: C
+ * c-set-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */

Quoted 1 line

????

Removed. Thanks.

christian

#17 lore raw

Re: [RFC PATCH 03/35] Add Xen interface header files

In reply to Chris Wright
Quoted 4 lines
---
 include/xen/interface/arch-x86_32.h   |  197 +++++++++++++++

that kind of stuff needs to go to asm/

Quoted 1 line
 include/xen/interface/event_channel.h |  205 +++++++++++++++

instead of interface please use something shorter, we'll see this all over the includes statements. intf for example.

include/xen/interface/arch-x86_32.h line 18

Quoted 7 lines
include/xen/interface/arch-x86_32.h new file +7 -0
+#ifdef __XEN__
+#define __DEFINE_GUEST_HANDLE(name, type) \
+    typedef struct { type *p; } __guest_handle_ ## name
+#else
+#define __DEFINE_GUEST_HANDLE(name, type) \
+    typedef type * __guest_handle_ ## name
+#endif

please get rid of all these stupid typedefs

include/xen/interface/arch-x86_32.h line 29

Quoted 5 lines
include/xen/interface/arch-x86_32.h new file +5 -0
+#ifndef __ASSEMBLY__
+/* Guest handles for primitive C types. */
+__DEFINE_GUEST_HANDLE(uchar, unsigned char);
+__DEFINE_GUEST_HANDLE(uint,  unsigned int);
+__DEFINE_GUEST_HANDLE(ulong, unsigned long);

don't use uchar/uint/ulong types ever. And in things like hypervisor/kernel interfaces always use __u* types.

#18 lore raw

Re: [RFC PATCH 03/35] Add Xen interface header files

Quoted 10 lines · Christoph Hellwig

include/xen/interface/arch-x86_32.h line 18

Quoted 8 lines
include/xen/interface/arch-x86_32.h new file +7 -0
+#ifdef __XEN__
+#define __DEFINE_GUEST_HANDLE(name, type) \
+    typedef struct { type *p; } __guest_handle_ ## name
+#else
+#define __DEFINE_GUEST_HANDLE(name, type) \
+    typedef type * __guest_handle_ ## name
+#endif

please get rid of all these stupid typedefs

These typedefs are a new hack to work around a basic interface problem:
instead of explicitly-sized types, Xen uses longs and pointers in its interface. On PowerPC in particular, where we need a 32-bit userland communicating with a 64-bit hypervisor, those types don't work.

However, the maintainers are reluctant to switch the interface to use explicitly-sized types because it would break binary compatibility. These ugly "HANDLE" macros allow PowerPC to do what we need without affecting binary compatibility on x86.

Signature · 2 lines
Hollis Blanchard
IBM Linux Technology Center
#19 lore raw

Re: [Xen-devel] Re: [RFC PATCH 03/35] Add Xen interface header files

Quoted 18 lines · Hollis Blanchard
Quoted 10 lines · Christoph Hellwig

include/xen/interface/arch-x86_32.h line 18

Quoted 8 lines
include/xen/interface/arch-x86_32.h new file +7 -0
+#ifdef __XEN__
+#define __DEFINE_GUEST_HANDLE(name, type) \
+    typedef struct { type *p; } __guest_handle_ ## name
+#else
+#define __DEFINE_GUEST_HANDLE(name, type) \
+    typedef type * __guest_handle_ ## name
+#endif

please get rid of all these stupid typedefs

These typedefs are a new hack to work around a basic interface problem:
instead of explicitly-sized types, Xen uses longs and pointers in its interface. On PowerPC in particular, where we need a 32-bit userland communicating with a 64-bit hypervisor, those types don't work.

Quoted 5 lines

However, the maintainers are reluctant to switch the interface to use explicitly-sized types because it would break binary compatibility. These ugly "HANDLE" macros allow PowerPC to do what we need without affecting binary compatibility on x86.

Is this strictly true though? The ABI for Power and x86 are not necessarily dependent on each other. One could just as easily define a typedef like:

#if defined(__ppc__)
typedef uint64_t guest_handle_t;
#elif defined(__x86__)
typedef unsigned long guest_handle_t;
#endif

I thought the use of GUEST_HANDLE was to maintain type safety. It certainly helps the issue you point out but it's not strictly necessary.

IMHO, this trick makes the code pretty ugly. I'd rather see it disappear in favor of something more akin to the above.

Regards,

Anthony Liguori

#20 lore raw

Re: Re: [RFC PATCH 03/35] Add Xen interface header files

Quoted 11 lines · Anthony Liguori

Is this strictly true though? The ABI for Power and x86 are not necessarily dependent on each other. One could just as easily define a typedef like:

#if defined(__ppc__)
typedef uint64_t guest_handle_t;
#elif defined(__x86__)
typedef unsigned long guest_handle_t;
#endif

I thought the use of GUEST_HANDLE was to maintain type safety. It certainly helps the issue you point out but it's not strictly necessary.

IMHO, this trick makes the code pretty ugly. I'd rather see it disappear in favor of something more akin to the above.

Well, I originally thought it was to allow hiding of machine-address scatter-gather lists for PPC (instead of passing virtual addresses). I seem to recall that's what Hollis said at the time. Clearly hiding something like that does need some macro magic. If it's not for that then I'm not at all for keeping it. It is indeed rather fugly.

-- Keir

#21 lore raw

Re: Re: [RFC PATCH 03/35] Add Xen interface header files

Quoted 16 lines · Keir Fraser
Quoted 14 lines · Anthony Liguori

Is this strictly true though? The ABI for Power and x86 are not necessarily dependent on each other. One could just as easily define a typedef like:

#if defined(__ppc__)
typedef uint64_t guest_handle_t;
#elif defined(__x86__)
typedef unsigned long guest_handle_t;
#endif

I thought the use of GUEST_HANDLE was to maintain type safety. It certainly helps the issue you point out but it's not strictly necessary.

Quoted 8 lines
Quoted 2 lines

IMHO, this trick makes the code pretty ugly. I'd rather see it disappear in favor of something more akin to the above.

Well, I originally thought it was to allow hiding of machine-address scatter-gather lists for PPC (instead of passing virtual addresses). I seem to recall that's what Hollis said at the time. Clearly hiding something like that does need some macro magic. If it's not for that then I'm not at all for keeping it. It is indeed rather fugly.

The scatter/gather lists are still needed, but they are hidden inside copy_to/from_guest().

If you change "unsigned long *" to an integer-based guest_handle_t, you will lose type-checking when accessing it, and I believe that's why you created these macros. I'm not particularly worried about type-checking, so I'd be happy to see them go, as long as we still have typedefs.

Signature · 2 lines
Hollis Blanchard
IBM Linux Technology Center
#22 lore raw

Re: Re: [RFC PATCH 03/35] Add Xen interface header files

Quoted 7 lines · Hollis Blanchard

The scatter/gather lists are still needed, but they are hidden inside copy_to/from_guest().

If you change "unsigned long *" to an integer-based guest_handle_t, you will lose type-checking when accessing it, and I believe that's why you created these macros. I'm not particularly worried about type-checking, so I'd be happy to see them go, as long as we still have typedefs

It's tempting. We can still ensure that those fields are only accessed via the correct functions by making the typedef structural. All we lose is the ability to discriminate between different types of handles. Still, it is nice to get some kind of hint from the handle definition the type of the thing that it references. I wouldn't trust a comment beside each guest-handle field to be kept up to date. Having the type passed to the guest_handle() macro does get us some self documentation.

However, macro or no macro: even reducing it to just a typedef won't get past some of the guys on lkml without a fight.

-- Keir

#23 lore raw

Re: [RFC PATCH 03/35] Add Xen interface header files

In reply to Hollis Blanchard
Quoted 9 lines · Hollis Blanchard

These typedefs are a new hack to work around a basic interface problem:
instead of explicitly-sized types, Xen uses longs and pointers in its interface. On PowerPC in particular, where we need a 32-bit userland communicating with a 64-bit hypervisor, those types don't work.

However, the maintainers are reluctant to switch the interface to use explicitly-sized types because it would break binary compatibility. These ugly "HANDLE" macros allow PowerPC to do what we need without affecting binary compatibility on x86.

this stuff needs to be fixed on x86 aswell. if the xen people don't even fix up their code because of silly abi concerns we should better not merge it at all.

#24 lore raw

Re: [RFC PATCH 03/35] Add Xen interface header files

In reply to Christoph Hellwig

Hi Christoph,

Quoted 11 lines · Christoph Hellwig
Quoted 4 lines
---
 include/xen/interface/arch-x86_32.h   |  197 +++++++++++++++

that kind of stuff needs to go to asm/

Quoted 1 line
 include/xen/interface/event_channel.h |  205 +++++++++++++++

instead of interface please use something shorter, we'll see this all over the includes statements. intf for example.

I like them and think they are quite clear.

Documentation/CodingStyle Chapter 4: Naming
seem to apply here.

And since you type the include only ONCE per file, this looks like a good trade, doesn't it?

Regards

Ingo Oeser

#25 lore raw

Re: [RFC PATCH 03/35] Add Xen interface header files

In reply to Chris Wright
Quoted 9 lines · Chris Wright

plain text document attachment (xen-interface-headers)
Add Xen interface header files. These are taken fairly directly from the Xen tree and hence the style is not entirely in accordance with Linux guidelines. There is a tension between fitting with Linux coding rules and ease of maintenance.

Quoted 6 lines
---
 include/xen/interface/arch-x86_32.h   |  197 +++++++++++++++
 include/xen/interface/event_channel.h |  205 +++++++++++++++
 include/xen/interface/features.h      |   53 ++++
 include/xen/interface/grant_table.h   |  311 +++++++++++++++++++++++
 include/xen/interface/io/blkif.h      |   85 ++++++

Shouldn't these be under asm-i386 , or are they used by other architecture ?

Daniel

#26 lore raw

Re: [RFC PATCH 03/35] Add Xen interface header files

Quoted 17 lines · Daniel Walker
Quoted 9 lines · Chris Wright

plain text document attachment (xen-interface-headers)
Add Xen interface header files. These are taken fairly directly from the Xen tree and hence the style is not entirely in accordance with Linux guidelines. There is a tension between fitting with Linux coding rules and ease of maintenance.

Quoted 6 lines
---
 include/xen/interface/arch-x86_32.h   |  197 +++++++++++++++
 include/xen/interface/event_channel.h |  205 +++++++++++++++
 include/xen/interface/features.h      |   53 ++++
 include/xen/interface/grant_table.h   |  311 +++++++++++++++++++++++
 include/xen/interface/io/blkif.h      |   85 ++++++
Quoted 2 lines

Shouldn't these be under asm-i386 , or are they used by other architecture ?

The full set of interface headers supports several architectures.

I think having all the header files in one place is preferable, but will gladly move them wherever is agreed on to be best ;-)

christian

#27 lore raw

Re: [RFC PATCH 03/35] Add Xen interface header files

Quoted 5 lines · Christian Limpach

The full set of interface headers supports several architectures.

I think having all the header files in one place is preferable, but will gladly move them wherever is agreed on to be best ;-)

I'd say include/linux/xen/ would be a better choice, if it's multi architecture ..

Daniel

#28 lore raw

RFC PATCH 4/35 Hypervisor interface header files.

In reply to Chris Wright

Define macros and inline functions for doing hypercalls into the hypervisor.

---
 include/asm-i386/hypercall.h  |  309 ++++++++++++++++++++++++++++++++++++++++++
 include/asm-i386/hypervisor.h |   70 +++++++++
 2 files changed, 379 insertions(+)
include/asm-i386/hypercall.h new file +309 -0
@@ -0,0 +1,309 @@
+/******************************************************************************
+ * hypercall.h
+ *
+ * Linux-specific hypervisor handling.
+ *
+ * Copyright (c) 2002-2004, K A Fraser
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version 2
+ * as published by the Free Software Foundation; or, when distributed
+ * separately from the Linux kernel or incorporated into other
+ * software packages, subject to the following license:
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this source file (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use, copy, modify,
+ * merge, publish, distribute, sublicense, and/or sell copies of the Software,
+ * and to permit persons to whom the Software is furnished to do so, subject to
+ * the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ */
+
+#ifndef __HYPERCALL_H__
+#define __HYPERCALL_H__
+
+#include <linux/config.h>
+
+#include <xen/interface/xen.h>
+#include <xen/interface/sched.h>
+
+#define __STR(x) #x
+#define STR(x) __STR(x)
+
+#define _hypercall0(type, name)			\
+({						\
+	long __res;				\
+	asm volatile (				\
+		"call hypercall_page + ("STR(__HYPERVISOR_##name)" * 32)"\
+		: "=a" (__res)			\
+		:				\
+		: "memory" );			\
+	(type)__res;				\
+})
+
+#define _hypercall1(type, name, a1)				\
+({								\
+	long __res, __ign1;					\
+	asm volatile (						\
+		"call hypercall_page + ("STR(__HYPERVISOR_##name)" * 32)"\
+		: "=a" (__res), "=b" (__ign1)			\
+		: "1" ((long)(a1))				\
+		: "memory" );					\
+	(type)__res;						\
+})
+
+#define _hypercall2(type, name, a1, a2)				\
+({								\
+	long __res, __ign1, __ign2;				\
+	asm volatile (						\
+		"call hypercall_page + ("STR(__HYPERVISOR_##name)" * 32)"\
+		: "=a" (__res), "=b" (__ign1), "=c" (__ign2)	\
+		: "1" ((long)(a1)), "2" ((long)(a2))		\
+		: "memory" );					\
+	(type)__res;						\
+})
+
+#define _hypercall3(type, name, a1, a2, a3)			\
+({								\
+	long __res, __ign1, __ign2, __ign3;			\
+	asm volatile (						\
+		"call hypercall_page + ("STR(__HYPERVISOR_##name)" * 32)"\
+		: "=a" (__res), "=b" (__ign1), "=c" (__ign2), 	\
+		"=d" (__ign3)					\
+		: "1" ((long)(a1)), "2" ((long)(a2)),		\
+		"3" ((long)(a3))				\
+		: "memory" );					\
+	(type)__res;						\
+})
+
+#define _hypercall4(type, name, a1, a2, a3, a4)			\
+({								\
+	long __res, __ign1, __ign2, __ign3, __ign4;		\
+	asm volatile (						\
+		"call hypercall_page + ("STR(__HYPERVISOR_##name)" * 32)"\
+		: "=a" (__res), "=b" (__ign1), "=c" (__ign2),	\
+		"=d" (__ign3), "=S" (__ign4)			\
+		: "1" ((long)(a1)), "2" ((long)(a2)),		\
+		"3" ((long)(a3)), "4" ((long)(a4))		\
+		: "memory" );					\
+	(type)__res;						\
+})
+
+#define _hypercall5(type, name, a1, a2, a3, a4, a5)		\
+({								\
+	long __res, __ign1, __ign2, __ign3, __ign4, __ign5;	\
+	asm volatile (						\
+		"call hypercall_page + ("STR(__HYPERVISOR_##name)" * 32)"\
+		: "=a" (__res), "=b" (__ign1), "=c" (__ign2),	\
+		"=d" (__ign3), "=S" (__ign4), "=D" (__ign5)	\
+		: "1" ((long)(a1)), "2" ((long)(a2)),		\
+		"3" ((long)(a3)), "4" ((long)(a4)),		\
+		"5" ((long)(a5))				\
+		: "memory" );					\
+	(type)__res;						\
+})
+
+static inline int
+HYPERVISOR_set_trap_table(
+	struct trap_info *table)
+{
+	return _hypercall1(int, set_trap_table, table);
+}
+
+static inline int
+HYPERVISOR_mmu_update(
+	struct mmu_update *req, int count, int *success_count, domid_t domid)
+{
+	return _hypercall4(int, mmu_update, req, count, success_count, domid);
+}
+
+static inline int
+HYPERVISOR_mmuext_op(
+	struct mmuext_op *op, int count, int *success_count, domid_t domid)
+{
+	return _hypercall4(int, mmuext_op, op, count, success_count, domid);
+}
+
+static inline int
+HYPERVISOR_set_gdt(
+	unsigned long *frame_list, int entries)
+{
+	return _hypercall2(int, set_gdt, frame_list, entries);
+}
+
+static inline int
+HYPERVISOR_stack_switch(
+	unsigned long ss, unsigned long esp)
+{
+	return _hypercall2(int, stack_switch, ss, esp);
+}
+
+static inline int
+HYPERVISOR_set_callbacks(
+	unsigned long event_selector, unsigned long event_address,
+	unsigned long failsafe_selector, unsigned long failsafe_address)
+{
+	return _hypercall4(int, set_callbacks,
+			   event_selector, event_address,
+			   failsafe_selector, failsafe_address);
+}
+
+static inline int
+HYPERVISOR_fpu_taskswitch(
+	int set)
+{
+	return _hypercall1(int, fpu_taskswitch, set);
+}
+
+static inline int
+HYPERVISOR_sched_op(
+	int cmd, unsigned long arg)
+{
+	return _hypercall2(int, sched_op, cmd, arg);
+}
+
+static inline long
+HYPERVISOR_set_timer_op(
+	u64 timeout)
+{
+	unsigned long timeout_hi = (unsigned long)(timeout>>32);
+	unsigned long timeout_lo = (unsigned long)timeout;
+	return _hypercall2(long, set_timer_op, timeout_lo, timeout_hi);
+}
+
+static inline int
+HYPERVISOR_set_debugreg(
+	int reg, unsigned long value)
+{
+	return _hypercall2(int, set_debugreg, reg, value);
+}
+
+static inline unsigned long
+HYPERVISOR_get_debugreg(
+	int reg)
+{
+	return _hypercall1(unsigned long, get_debugreg, reg);
+}
+
+static inline int
+HYPERVISOR_update_descriptor(
+	u64 ma, u64 desc)
+{
+	return _hypercall4(int, update_descriptor, ma, ma>>32, desc, desc>>32);
+}
+
+static inline int
+HYPERVISOR_memory_op(
+	unsigned int cmd, void *arg)
+{
+	return _hypercall2(int, memory_op, cmd, arg);
+}
+
+static inline int
+HYPERVISOR_multicall(
+	void *call_list, int nr_calls)
+{
+	return _hypercall2(int, multicall, call_list, nr_calls);
+}
+
+static inline int
+HYPERVISOR_update_va_mapping(
+	unsigned long va, pte_t new_val, unsigned long flags)
+{
+	unsigned long pte_hi = 0;
+#ifdef CONFIG_X86_PAE
+	pte_hi = new_val.pte_high;
+#endif
+	return _hypercall4(int, update_va_mapping, va,
+			   new_val.pte_low, pte_hi, flags);
+}
+
+static inline int
+HYPERVISOR_event_channel_op(
+	void *op)
+{
+	return _hypercall1(int, event_channel_op, op);
+}
+
+static inline int
+HYPERVISOR_xen_version(
+	int cmd, void *arg)
+{
+	return _hypercall2(int, xen_version, cmd, arg);
+}
+
+static inline int
+HYPERVISOR_console_io(
+	int cmd, int count, char *str)
+{
+	return _hypercall3(int, console_io, cmd, count, str);
+}
+
+static inline int
+HYPERVISOR_physdev_op(
+	void *physdev_op)
+{
+	return _hypercall1(int, physdev_op, physdev_op);
+}
+
+static inline int
+HYPERVISOR_grant_table_op(
+	unsigned int cmd, void *uop, unsigned int count)
+{
+	return _hypercall3(int, grant_table_op, cmd, uop, count);
+}
+
+static inline int
+HYPERVISOR_update_va_mapping_otherdomain(
+	unsigned long va, pte_t new_val, unsigned long flags, domid_t domid)
+{
+	unsigned long pte_hi = 0;
+#ifdef CONFIG_X86_PAE
+	pte_hi = new_val.pte_high;
+#endif
+	return _hypercall5(int, update_va_mapping_otherdomain, va,
+			   new_val.pte_low, pte_hi, flags, domid);
+}
+
+static inline int
+HYPERVISOR_vm_assist(
+	unsigned int cmd, unsigned int type)
+{
+	return _hypercall2(int, vm_assist, cmd, type);
+}
+
+static inline int
+HYPERVISOR_vcpu_op(
+	int cmd, int vcpuid, void *extra_args)
+{
+	return _hypercall3(int, vcpu_op, cmd, vcpuid, extra_args);
+}
+
+static inline int
+HYPERVISOR_suspend(
+	unsigned long srec)
+{
+	return _hypercall3(int, sched_op, SCHEDOP_shutdown,
+			   SHUTDOWN_suspend, srec);
+}
+
+static inline int
+HYPERVISOR_nmi_op(
+	unsigned long op,
+	unsigned long arg)
+{
+	return _hypercall2(int, nmi_op, op, arg);
+}
+
+#endif /* __HYPERCALL_H__ */
include/asm-i386/hypervisor.h new file +70 -0

70 lines not on this page · raw message

--

#29 lore raw

Re: [RFC PATCH 04/35] Hypervisor interface header files.

Hi Chris,

first of all: Thanks to the Xen-Team for doing this!

include/asm-i386/hypercall.h line 122

Quoted 8 lines · Chris Wright

Define macros and inline functions for doing hypercalls into the hypervisor.

include/asm-i386/hypercall.h new file +6 -0
+static inline int
+HYPERVISOR_set_trap_table(
+	struct trap_info *table)
+{
+	return _hypercall1(int, set_trap_table, table);
+}

This is outright ugly and non-conformant to
Documentation/CodingStyle Chapter 2

Fixing this also saves some code lines.

It also looks like generated code. Maybe you can fix your generator instead?

Regards

Ingo Oeser

#30 lore raw

Re: [RFC PATCH 04/35] Hypervisor interface header files.

Quoted 12 lines · * Ingo Oeser ([email protected]) wrote:

include/asm-i386/hypercall.h line 122

Quoted 8 lines · Chris Wright

Define macros and inline functions for doing hypercalls into the hypervisor.

include/asm-i386/hypercall.h new file +6 -0
+static inline int
+HYPERVISOR_set_trap_table(
+	struct trap_info *table)
+{
+	return _hypercall1(int, set_trap_table, table);
+}

This is outright ugly and non-conformant to
Documentation/CodingStyle Chapter 2

Yes, it's non-conformant, will fix that up.

thanks,
-chris

#31 lore raw

RFC PATCH 5/35 Add sync bitops

In reply to Chris Wright

Add "always lock'd" implementations of set_bit, clear_bit and change_bit and the corresponding test_and_ functions. Also add "always lock'd" implementation of cmpxchg. These give guaranteed strong synchronisation and are required for non-SMP kernels running on an SMP hypervisor.

---
 include/asm-i386/synch_bitops.h |  166 ++++++++++++++++++++++++++++++++++++++++
 include/asm-i386/system.h       |   33 +++++++
 2 files changed, 199 insertions(+)
include/asm-i386/system.h +33 -0
@@ -263,6 +263,9 @@ static inline unsigned long __xchg(unsig
 #define cmpxchg(ptr,o,n)\
 	((__typeof__(*(ptr)))__cmpxchg((ptr),(unsigned long)(o),\
 					(unsigned long)(n),sizeof(*(ptr))))
+#define synch_cmpxchg(ptr,o,n)\
+	((__typeof__(*(ptr)))__synch_cmpxchg((ptr),(unsigned long)(o),\
+					(unsigned long)(n),sizeof(*(ptr))))
 #endif
 
 static inline unsigned long __cmpxchg(volatile void *ptr, unsigned long old,
@@ -292,6 +295,36 @@ static inline unsigned long __cmpxchg(vo
 	return old;
 }
 
+#define __LOCK_PREFIX "lock ; "
+static inline unsigned long __synch_cmpxchg(volatile void *ptr,
+					    unsigned long old,
+					    unsigned long new, int size)
+{
+	unsigned long prev;
+	switch (size) {
+	case 1:
+		__asm__ __volatile__(__LOCK_PREFIX "cmpxchgb %b1,%2"
+				     : "=a"(prev)
+				     : "q"(new), "m"(*__xg(ptr)), "0"(old)
+				     : "memory");
+		return prev;
+	case 2:
+		__asm__ __volatile__(__LOCK_PREFIX "cmpxchgw %w1,%2"
+				     : "=a"(prev)
+				     : "r"(new), "m"(*__xg(ptr)), "0"(old)
+				     : "memory");
+		return prev;
+	case 4:
+		__asm__ __volatile__(__LOCK_PREFIX "cmpxchgl %1,%2"
+				     : "=a"(prev)
+				     : "r"(new), "m"(*__xg(ptr)), "0"(old)
+				     : "memory");
+		return prev;
+	}
+	return old;
+}
+#undef __LOCK_PREFIX
+
 #ifndef CONFIG_X86_CMPXCHG
 /*
  * Building a kernel capable running on 80386. It may be necessary to
include/asm-i386/synch_bitops.h new file +166 -0

166 lines not on this page · raw message

--

#32 lore raw

Re: [RFC PATCH 05/35] Add sync bitops

Quoted 5 lines · Chris Wright

Add "always lock'd" implementations of set_bit, clear_bit and change_bit and the corresponding test_and_ functions. Also add "always lock'd" implementation of cmpxchg. These give guaranteed strong synchronisation and are required for non-SMP kernels running on an SMP hypervisor.

Could you explain why this is done and what is exactly meant with "always looked"? Wh the performance impact?

#33 lore raw

Re: [RFC PATCH 05/35] Add sync bitops

Quoted 10 lines · * Christoph Lameter ([email protected]) wrote:
Quoted 5 lines · Chris Wright

Add "always lock'd" implementations of set_bit, clear_bit and change_bit and the corresponding test_and_ functions. Also add "always lock'd" implementation of cmpxchg. These give guaranteed strong synchronisation and are required for non-SMP kernels running on an SMP hypervisor.

Could you explain why this is done and what is exactly meant with "always looked"? Wh the performance impact?

The standard UP bitops are not atomic. But a UP guest may be on SMP machine, and the bitmaps here are shared between guests. The always locked means the lock prefix is not conditional on either UP build (or smp alternatives patching), and memory barriers are in place. There's no performance penalty unless you use them, as it's a new set (somewhat similar to the bitops changes you were looking into). Although, this is the simplest, with no multiplexing, simply new interface, synch_*. Open to ideas here. Xen is another possible consumer of your bitops changes.

thanks,
-chris

#34 lore raw

Re: [RFC PATCH 05/35] Add sync bitops

In reply to Christoph Lameter
Quoted 10 lines · Christoph Lameter
Quoted 5 lines · Chris Wright

Add "always lock'd" implementations of set_bit, clear_bit and change_bit and the corresponding test_and_ functions. Also add "always lock'd" implementation of cmpxchg. These give guaranteed strong synchronisation and are required for non-SMP kernels running on an SMP hypervisor.

Could you explain why this is done and what is exactly meant with "always looked"? Wh the performance impact?

When UP guest runs on SMP hypervisor they still need the LOCK prefix to talk to the hypervisor through shared memory in a smp safe way.

Normally UP kernels don't use any LOCK prefixes.

I suggested to refactor the bitops this way earlier for this.

-Andi

#35 lore raw

RFC PATCH 6/35 Add vmlinuz build target.

In reply to Chris Wright

The vmlinuz image is a stripped and compressed kernel image, it is smaller than the vmlinux image and the Xen domain builder supports loading compressed images directly.

---
 arch/i386/Makefile      |    5 +++--
 arch/i386/boot/Makefile |   10 +++++++++-
 2 files changed, 12 insertions(+), 3 deletions(-)
arch/i386/Makefile +3 -2
@@ -108,15 +108,16 @@ CPPFLAGS += $(mflags-y)
 boot := arch/i386/boot
 
 PHONY += zImage bzImage compressed zlilo bzlilo \
-         zdisk bzdisk fdimage fdimage144 fdimage288 isoimage install
+         zdisk bzdisk fdimage fdimage144 fdimage288 isoimage install vmlinuz
 
 all: bzImage
 
 # KBUILD_IMAGE specify target image being built
                     KBUILD_IMAGE := $(boot)/bzImage
 zImage zlilo zdisk: KBUILD_IMAGE := arch/i386/boot/zImage
+vmlinuz:            KBUILD_IMAGE := $(boot)/vmlinuz
 
-zImage bzImage: vmlinux
+zImage bzImage vmlinuz: vmlinux
 	$(Q)$(MAKE) $(build)=$(boot) $(KBUILD_IMAGE)
 
 compressed: zImage
arch/i386/boot/Makefile +9 -1
@@ -26,7 +26,7 @@ SVGA_MODE := -DSVGA_MODE=NORMAL_VGA
 #RAMDISK := -DRAMDISK=512
 
 targets		:= vmlinux.bin bootsect bootsect.o \
-		   setup setup.o zImage bzImage
+		   setup setup.o zImage bzImage vmlinuz
 subdir- 	:= compressed
 
 hostprogs-y	:= tools/build
@@ -128,5 +128,13 @@ zlilo: $(BOOTIMAGE)
 	cp System.map $(INSTALL_PATH)/
 	if [ -x /sbin/lilo ]; then /sbin/lilo; else /etc/lilo/install; fi
 
+$(obj)/vmlinuz: $(obj)/vmlinux-stripped FORCE
+	$(call if_changed,gzip)
+	@echo 'Kernel: $@ is ready' ' (#'`cat .version`')'
+
+$(obj)/vmlinux-stripped: OBJCOPYFLAGS := -g --strip-unneeded
+$(obj)/vmlinux-stripped: vmlinux FORCE
+	$(call if_changed,objcopy)
+
 install:
 	sh $(srctree)/$(src)/install.sh $(KERNELRELEASE) $(BOOTIMAGE) System.map "$(INSTALL_PATH)"

--

#36 lore raw

RFC PATCH 7/35 Make LOAD_OFFSET defined by subarch

In reply to Chris Wright

Change LOAD_OFFSET so that the kernel has virtual addresses in the elf header fields.

Unlike bare metal kernels, Xen kernels start with virtual address management turned on and thus the addresses to load to should be virtual addresses.

---
 arch/i386/kernel/vmlinux.lds.S                   |    2 +-
 include/asm-i386/mach-default/mach_vmlinux.lds.h |    6 ++++++
 include/asm-i386/mach-xen/mach_vmlinux.lds.h     |    6 ++++++
 3 files changed, 13 insertions(+), 1 deletion(-)
arch/i386/kernel/vmlinux.lds.S +1 -1
@@ -2,7 +2,7 @@
  * Written by Martin Mares <[email protected]>;
  */
 
-#define LOAD_OFFSET __PAGE_OFFSET
+#include "mach_vmlinux.lds.h"
 
 #include <asm-generic/vmlinux.lds.h>
 #include <asm/thread_info.h>
include/asm-i386/mach-default/mach_vmlinux.lds.h new file +6 -0
@@ -0,0 +1,6 @@
+#ifndef __ASM_MACH_VMLINUX_LDS_H
+#define __ASM_MACH_VMLINUX_LDS_H
+
+#define LOAD_OFFSET __PAGE_OFFSET
+
+#endif /* __ASM_MACH_VMLINUX_LDS_H */
include/asm-i386/mach-xen/mach_vmlinux.lds.h new file +6 -0
@@ -0,0 +1,6 @@
+#ifndef __ASM_MACH_VMLINUX_LDS_H
+#define __ASM_MACH_VMLINUX_LDS_H
+
+#define LOAD_OFFSET 0
+
+#endif /* __ASM_MACH_VMLINUX_LDS_H */

--

#37 lore raw

Re: [RFC PATCH 07/35] Make LOAD_OFFSET defined by subarch

Quoted 5 lines · Chris Wright

Change LOAD_OFFSET so that the kernel has virtual addresses in the elf header fields.

Unlike bare metal kernels, Xen kernels start with virtual address management turned on and thus the addresses to load to should be virtual addresses.

This patch interferes with using a traditional bootloader. The loader for Xen should be smarter - it already has VIRT_BASE from the xen_guest section, and can simply add the relocation to these header fields. This is unnecessary, and one of the many reasons a Xen kernel can't run in a normal environment.

Zach

#38 lore raw

Re: [Xen-devel] Re: [RFC PATCH 07/35] Make LOAD_OFFSET defined by subarch

Quoted 8 lines · Zachary Amsden
Quoted 6 lines · Chris Wright

Change LOAD_OFFSET so that the kernel has virtual addresses in the elf header fields.

Unlike bare metal kernels, Xen kernels start with virtual address management turned on and thus the addresses to load to should be virtual addresses.

Quoted 5 lines

This patch interferes with using a traditional bootloader. The loader for Xen should be smarter - it already has VIRT_BASE from the xen_guest section, and can simply add the relocation to these header fields. This is unnecessary, and one of the many reasons a Xen kernel can't run in a normal environment.

I fully agree. Attached below is a patch (against xen unstable mercurial tree) which does exactly that ;)

cheers,

Gerd

Signature · 4 lines
Gerd Hoffmann <[email protected]>
Erst mal heiraten, ein, zwei Kinder, und wenn alles läuft
geh' ich nach drei Jahren mit der Familie an die Börse.
http://www.suse.de/~kraxel/julika-dora.jpeg
linux-2.6-xen-sparse/include/asm-i386/mach-xen/asm/page.h +0 -4
diff -r 1e3977e029fd linux-2.6-xen-sparse/include/asm-i386/mach-xen/asm/page.h
@@ -289,10 +289,6 @@
 #endif
 #define __KERNEL_START		(__PAGE_OFFSET + __PHYSICAL_START)
 
-#undef LOAD_OFFSET
-#define LOAD_OFFSET		0
-
-
 #define PAGE_OFFSET		((unsigned long)__PAGE_OFFSET)
 #define VMALLOC_RESERVE		((unsigned long)__VMALLOC_RESERVE)
 #define MAXMEM			(__FIXADDR_TOP-__PAGE_OFFSET-__VMALLOC_RESERVE)
linux-2.6-xen-sparse/include/asm-x86_64/mach-xen/asm/page.h +0 -3
diff -r 1e3977e029fd linux-2.6-xen-sparse/include/asm-x86_64/mach-xen/asm/page.h
@@ -260,9 +260,6 @@
 #define __PAGE_OFFSET           0xffff880000000000
 #endif /* !__ASSEMBLY__ */
 
-#undef LOAD_OFFSET
-#define LOAD_OFFSET		0
-
 /* to align the pointer to the (next) page boundary */
 #define PAGE_ALIGN(addr)	(((addr)+PAGE_SIZE-1)&PAGE_MASK)
 
tools/libxc/xc_load_elf.c +37 -7

74 lines not on this page · raw message

tools/libxc/xg_private.h +1 -0
diff -r 1e3977e029fd tools/libxc/xg_private.h
@@ -135,6 +135,7 @@
     unsigned long v_kernstart;
     unsigned long v_kernend;
     unsigned long v_kernentry;
+    unsigned long virt_base;
 
     unsigned int  load_symtab;
     unsigned int  pae_kernel;
xen/common/elf.c +44 -7

84 lines not on this page · raw message

xen/include/xen/sched.h +1 -0
diff -r 1e3977e029fd xen/include/xen/sched.h
@@ -172,6 +172,7 @@
     unsigned long v_kernstart;
     unsigned long v_kernend;
     unsigned long v_kernentry;
+    unsigned long virt_base;
     /* Initialised by loader: Private. */
     unsigned int  load_symtab;
     unsigned long symtab_addr;
#39 lore raw

Re: [Xen-devel] Re: [RFC PATCH 07/35] Make LOAD_OFFSET defined by subarch

Quoted 2 lines · * Gerd Hoffmann ([email protected]) wrote:

I fully agree. Attached below is a patch (against xen unstable mercurial tree) which does exactly that ;)

Thanks Gerd, I thought you had been working on that. Was the concern with vaddr vs. paddr worked out?

thanks,
-chris

#40 lore raw

Re: [Xen-devel] Re: [RFC PATCH 07/35] Make LOAD_OFFSET defined by subarch

Quoted 6 lines · Chris Wright
Quoted 2 lines · * Gerd Hoffmann ([email protected]) wrote:

I fully agree. Attached below is a patch (against xen unstable mercurial tree) which does exactly that ;)

Thanks Gerd, I thought you had been working on that. Was the concern with vaddr vs. paddr worked out?

Not yet, and I didn't feel comfortable pushing it just before the 3.0.2 release, but I think _now_ would be a good time to finally merge it. Having physical addresses in paddr seems to be common practice, and IMO xen should follow that as it makes life easier for everybody.

It's not a big problem that xen guests boot with paging enabled, and as Zachary already pointed out it's trivial to use virt_base from the xen_guest elf section to create correct initial page tables.

Even maintaining backward compatibility with some guesswork is possible as it is _very_ unlikely that the paddr field holds physical addresses larger than virt_base ;)

cheers,

Gerd

Signature · 4 lines
Gerd Hoffmann <[email protected]>
Erst mal heiraten, ein, zwei Kinder, und wenn alles läuft
geh' ich nach drei Jahren mit der Familie an die Börse.
http://www.suse.de/~kraxel/julika-dora.jpeg
#41 lore raw

Re: [RFC PATCH 07/35] Make LOAD_OFFSET defined by subarch

In reply to Zachary Amsden
Quoted 8 lines · Zachary Amsden
Quoted 6 lines · Chris Wright

Change LOAD_OFFSET so that the kernel has virtual addresses in the elf header fields.

Unlike bare metal kernels, Xen kernels start with virtual address management turned on and thus the addresses to load to should be virtual addresses.

Quoted 5 lines

This patch interferes with using a traditional bootloader. The loader for Xen should be smarter - it already has VIRT_BASE from the xen_guest section, and can simply add the relocation to these header fields. This is unnecessary, and one of the many reasons a Xen kernel can't run in a normal environment.

It's certainly not as simple as you make it sound, if you want to support existing kernels without having to guess how the kernel image was built.

I've updated our loader to support this now, so that this patch is no longer necessary. I have at the same time added a new field to xen_guest which allows specifying the entry point, allowing us to have a different entry point when running the kernel image on Xen.

christian

#42 lore raw

[Xen-devel] Re: [RFC PATCH 07/35] Make LOAD_OFFSET defined by subarch

Quoted 4 lines

I've updated our loader to support this now, so that this patch is no longer necessary. I have at the same time added a new field to xen_guest which allows specifying the entry point, allowing us to have a different entry point when running the kernel image on Xen.

Why do you need a separate entry point here? The code should be able to figure out which mode it is run in without problems...

Jan

#43 lore raw

Re: [Xen-devel] Re: [RFC PATCH 07/35] Make LOAD_OFFSET defined by subarch

Quoted 7 lines · Jan Beulich
Quoted 4 lines

I've updated our loader to support this now, so that this patch is no longer necessary. I have at the same time added a new field to xen_guest which allows specifying the entry point, allowing us to have a different entry point when running the kernel image on Xen.

Why do you need a separate entry point here? The code should be able to figure out which mode it is run in without problems...

I think it's the cleanest way to have different startup code for native and non-native in the same kernel. But even if that's not needed (for Linux), then you can have it point at the same address. It is also always pointing to a virtual address, while the elf header one now points to a physical address which doesn't make much sense in the environment we start the kernel.

christian

#44 lore raw

RFC PATCH 8/35 Add Xen-specific memory management definitions

In reply to Chris Wright

Add extra memory management definitions used by Xen-specific code. These allow conversion between 'pseudophysical' memory addresses, which provide the illusion of a physically contiguous memory map, and underlying real machine addresses. This conversion is neceesary when interpreting and updating PTEs. Also support write protection of page mappings, which is needed to allow successful validation of page tables.

The current definitions are incomplete and only a stub implementation, allowing us to re-use existing code (drivers) which references these memory management code changes.

---
 include/asm-i386/hypervisor.h             |    3 
 include/asm-i386/mach-default/mach_page.h |    4 +
 include/asm-i386/mach-xen/mach_page.h     |   99 ++++++++++++++++++++++++++++++
 include/asm-i386/page.h                   |    2 
 4 files changed, 108 insertions(+)
include/asm-i386/hypervisor.h +3 -0
@@ -67,4 +67,7 @@ u64 jiffies_to_st(unsigned long jiffies)
 
 #define xen_init()	(0)
 
+#include <xen/interface/version.h>
+#include <xen/features.h>
+
 #endif /* __HYPERVISOR_H__ */
include/asm-i386/page.h +2 -0
@@ -82,6 +82,8 @@ typedef struct { unsigned long pgprot; }
 /* to align the pointer to the (next) page boundary */
 #define PAGE_ALIGN(addr)	(((addr)+PAGE_SIZE-1)&PAGE_MASK)
 
+#include <mach_page.h>
+
 /*
  * This handles the memory map.. We could make this a config
  * option, but too many people screw it up, and too few need
include/asm-i386/mach-default/mach_page.h new file +4 -0
@@ -0,0 +1,4 @@
+#ifndef __ASM_MACH_PAGE_H
+#define __ASM_MACH_PAGE_H
+
+#endif /* __ASM_MACH_PAGE_H */
include/asm-i386/mach-xen/mach_page.h new file +99 -0

99 lines not on this page · raw message

--

#45 lore raw

Re: [RFC PATCH 08/35] Add Xen-specific memory management definitions

include/asm-i386/mach-xen/mach_page.h line 76

Quoted 8 lines
include/asm-i386/mach-xen/mach_page.h new file +7 -0
+#define virt_to_ptep(__va)						\
+({									\
+	pgd_t *__pgd = pgd_offset_k((unsigned long)(__va));		\
+	pud_t *__pud = pud_offset(__pgd, (unsigned long)(__va));	\
+	pmd_t *__pmd = pmd_offset(__pud, (unsigned long)(__va));	\
+	pte_offset_kernel(__pmd, (unsigned long)(__va));		\
+})

Do we really need yet another function to do this? Especially one in a mult-line #define instead of a real function call, and that doesn't seem to error check at each step?

Quoted 9 lines
include/asm-i386/mach-xen/mach_page.h new file +9 -0
+
+#define arbitrary_virt_to_machine(__va)					\
+({									\
+	maddr_t m = (maddr_t)pte_mfn(*virt_to_ptep(__va)) << PAGE_SHIFT;\
+	m | ((unsigned long)(__va) & (PAGE_SIZE-1));			\
+})
+
+#define make_lowmem_page_readonly(va, feature) do {		\
+	pte_t *pte;						\

include/asm-i386/mach-xen/mach_page.h line 95

Quoted 10 lines
include/asm-i386/mach-xen/mach_page.h new file +10 -0
+	int rc;							\
+								\
+	if (xen_feature(feature))				\
+		return;						\
+								\
+	pte = virt_to_ptep(va);					\
+	rc = HYPERVISOR_update_va_mapping(			\
+		(unsigned long)va, pte_wrprotect(*pte), 0);	\
+	BUG_ON(rc);						\
+} while (0)

Things this long should definitely not be #defines.

#46 lore raw

Re: [RFC PATCH 08/35] Add Xen-specific memory management definitions

Quoted 12 lines · Martin J. Bligh

include/asm-i386/mach-xen/mach_page.h line 76

Quoted 8 lines
include/asm-i386/mach-xen/mach_page.h new file +7 -0
+#define virt_to_ptep(__va)						\
+({									\
+	pgd_t *__pgd = pgd_offset_k((unsigned long)(__va));		\
+	pud_t *__pud = pud_offset(__pgd, (unsigned long)(__va));	\
+	pmd_t *__pmd = pmd_offset(__pud, (unsigned long)(__va));	\
+	pte_offset_kernel(__pmd, (unsigned long)(__va));		\
+})

Do we really need yet another function to do this? Especially one in a mult-line #define instead of a real function call, and that doesn't seem to error check at each step?

Indeed, I'll use lookup_address instead.

Quoted 13 lines
Quoted 13 lines
include/asm-i386/mach-xen/mach_page.h new file +13 -0
+
+#define arbitrary_virt_to_machine(__va)				 \
+({									\
+	maddr_t m = (maddr_t)pte_mfn(*virt_to_ptep(__va)) << PAGE_SHIFT;\
+	m | ((unsigned long)(__va) & (PAGE_SIZE-1));			\
+})
+
+#define make_lowmem_page_readonly(va, feature) do {		\
+	pte_t *pte;						\
+	int rc;							\
+								\
+	if (xen_feature(feature))				\
+		return;						\
Quoted 8 lines

include/asm-i386/mach-xen/mach_page.h line 95

Quoted 6 lines
include/asm-i386/mach-xen/mach_page.h new file +6 -0
+								\
+	pte = virt_to_ptep(va);					\
+	rc = HYPERVISOR_update_va_mapping(			\
+		(unsigned long)va, pte_wrprotect(*pte), 0);	\
+	BUG_ON(rc);						\
+} while (0)

Things this long should definitely not be #defines.

I've changed these to be functions and moved them into a .c file under arch/i386/mach-xen.

christian

#47 lore raw

Re: [RFC PATCH 08/35] Add Xen-specific memory management definitions

In reply to Chris Wright

On Tue, 09 May 2006 00:00:08 -0700, Chris Wright <[email protected]> wrote:

I'm a little concerned with the code below being entirely too smart:

Quoted 17 lines
(quoted diff) +17 -0
+static inline unsigned long mfn_to_pfn(unsigned long mfn)
+{
+#ifndef CONFIG_XEN_SHADOW_MODE
+	unsigned long pfn;
+
+	if (xen_feature(XENFEAT_auto_translated_physmap))
+		return mfn;
+
+	/*
+	 * The array access can fail (e.g., device space beyond end of RAM).
+	 * In such cases it doesn't matter what we return (we return garbage),
+	 * but we must handle the fault without crashing!
+	 */
+	asm (
+		"1:	movl %1,%0\n"
+		"2:\n"
+		".section __ex_table,\"a\"\n"

Review

Quoted 10 lines
(quoted diff) +10 -0
+		"	.align 4\n"
+		"	.long 1b,2b\n"
+		".previous"
+		: "=r" (pfn) : "m" (machine_to_phys_mapping[mfn]) );
+
+	return pfn;
+#else
+	return mfn;
+#endif
+}

I found that if someone tries to use this too early, hypervisor terminates the domain with a message like this:

(XEN) DOM0: (file=mm.c, line=486) Non-privileged attempt to map I/O space 000fec00

Why can't we use something like this, only a proper number of machine pages:

include/asm-x86_64/mach-xen/asm/page.h +3 -18
diff -urp -X dontdiff linux-2.6.15-1.2054_FC5/include/asm-x86_64/mach-xen/asm/page.h linux-2.6.15-1.2054_FC5.z3/include/asm-x86_64/mach-xen/asm/page.h
@@ -101,26 +101,11 @@ static inline int phys_to_machine_mappin
 
 static inline unsigned long mfn_to_pfn(unsigned long mfn)
 {
-	unsigned long pfn;
-
 	if (xen_feature(XENFEAT_auto_translated_physmap))
 		return mfn;
-
-	/*
-	 * The array access can fail (e.g., device space beyond end of RAM).
-	 * In such cases it doesn't matter what we return (we return garbage),
-	 * but we must handle the fault without crashing!
-	 */
-	asm (
-		"1:	movq %1,%0\n"
-		"2:\n"
-		".section __ex_table,\"a\"\n"
-		"	.align 8\n"
-		"	.quad 1b,2b\n"
-		".previous"
-		: "=r" (pfn) : "m" (machine_to_phys_mapping[mfn]) );
-
-	return pfn;
+	if (mfn >= 1048576)	/* 4GB _machine_ RAM. XXX How to find out? */
+		return ~0;
+	return machine_to_phys_mapping[mfn];
 }
 
 /*

I'm sure you considered this, but decided to be tricky. Why? No way to find the safe number of machine pages in a guest?

-- Pete

#48 lore raw

Re: [RFC PATCH 08/35] Add Xen-specific memory management definitions

Quoted 2 lines · Pete Zaitcev

I'm sure you considered this, but decided to be tricky. Why? No way to find the safe number of machine pages in a guest?

We want to allow holes in the table if RAM is sparse. That code shouldn't ever fail after the guest has installed a page-fault handler. If you can make it do so (was it i386 or x86/64?) we're interested in seeing the full crash output.

-- Keir

#49 lore raw

Re: [RFC PATCH 08/35] Add Xen-specific memory management definitions

In reply to Pete Zaitcev
Quoted 2 lines · Pete Zaitcev

I'm sure you considered this, but decided to be tricky. Why? No way to find the safe number of machine pages in a guest?

In addition to wanting to support holes, the number of machine pages will usually change when you move it to another machine.

christian

#50 lore raw

Re: [RFC PATCH 08/35] Add Xen-specific memory management definitions

In reply to Chris Wright

include/asm-i386/mach-xen/mach_page.h line 19

Quoted 11 lines · Chris Wright
include/asm-i386/mach-xen/mach_page.h new file +11 -0
+static inline unsigned long pfn_to_mfn(unsigned long pfn)
+{
+#ifndef CONFIG_XEN_SHADOW_MODE
+	if (xen_feature(XENFEAT_auto_translated_physmap))
+		return pfn;
+	return phys_to_machine_mapping[(unsigned int)(pfn)] &
+		~FOREIGN_FRAME_BIT;
+#else
+	return pfn;
+#endif
+}

Why do we need several modes in Linux guests?

If a significant tradeoff exists (for example, between performance and maximum addressable memory), then we need to think about the real issue instead of throwing config options into the pot.

-- Pete

#51 lore raw

Re: [RFC PATCH 08/35] Add Xen-specific memory management definitions

Quoted 15 lines · * Pete Zaitcev ([email protected]) wrote:

include/asm-i386/mach-xen/mach_page.h line 19

Quoted 11 lines · Chris Wright
include/asm-i386/mach-xen/mach_page.h new file +11 -0
+static inline unsigned long pfn_to_mfn(unsigned long pfn)
+{
+#ifndef CONFIG_XEN_SHADOW_MODE
+	if (xen_feature(XENFEAT_auto_translated_physmap))
+		return pfn;
+	return phys_to_machine_mapping[(unsigned int)(pfn)] &
+		~FOREIGN_FRAME_BIT;
+#else
+	return pfn;
+#endif
+}

Why do we need several modes in Linux guests?

This patchset only supports shadow translated mode, so the extra CONFIG is just an artifact of this simplied patchset. Ultimately, this is not the preferred mode (performance wise), but shadow mode is simpler to port to.

#52 lore raw

RFC PATCH 9/35 Change __FIXADDR_TOP to leave room for the hypervisor.

In reply to Chris Wright

Move the definition of __FIXADDR_TOP into a subarch include file so that it can be overridden for subarch xen -- the hypervisor needs about 64MB at the top of the address space.

---
 include/asm-i386/fixmap.h                   |    8 +-------
 include/asm-i386/mach-default/mach_fixmap.h |   11 +++++++++++
 include/asm-i386/mach-xen/mach_fixmap.h     |   11 +++++++++++
 3 files changed, 23 insertions(+), 7 deletions(-)
include/asm-i386/fixmap.h +1 -7
@@ -14,13 +14,7 @@
 #define _ASM_FIXMAP_H
 
 #include <linux/config.h>
-
-/* used by vmalloc.c, vsyscall.lds.S.
- *
- * Leave one empty page between vmalloc'ed areas and
- * the start of the fixmap.
- */
-#define __FIXADDR_TOP	0xfffff000
+#include <mach_fixmap.h>
 
 #ifndef __ASSEMBLY__
 #include <linux/kernel.h>
include/asm-i386/mach-default/mach_fixmap.h new file +11 -0
@@ -0,0 +1,11 @@
+#ifndef __ASM_MACH_FIXMAP_H
+#define __ASM_MACH_FIXMAP_H
+
+/* used by vmalloc.c, vsyscall.lds.S.
+ *
+ * Leave one empty page between vmalloc'ed areas and
+ * the start of the fixmap.
+ */
+#define __FIXADDR_TOP	0xfffff000
+
+#endif /* __ASM_MACH_FIXMAP_H */
include/asm-i386/mach-xen/mach_fixmap.h new file +11 -0
@@ -0,0 +1,11 @@
+#ifndef __ASM_MACH_FIXMAP_H
+#define __ASM_MACH_FIXMAP_H
+
+/* used by vmalloc.c, vsyscall.lds.S.
+ *
+ * Leave one empty page between vmalloc'ed areas and
+ * the start of the fixmap.
+ */
+#define __FIXADDR_TOP	(HYPERVISOR_VIRT_START - 2 * PAGE_SIZE)
+
+#endif /* __ASM_MACH_FIXMAP_H */

--

#53 lore raw

RFC PATCH 10/35 Add a new head.S start-of-day file for booting on Xen.

In reply to Chris Wright

When running on Xen, the kernel is started with paging enabled. Also don't check for cpu features which are present on all cpus supported by Xen.

Don't define segments which are not supported when running on Xen.

Define the __xen_guest section which exports information about the kernel to the domain builder.

---
 arch/i386/Makefile          |    7 +-
 arch/i386/kernel/head-cpu.S |   88 +++++++++++++++++++++++++++++++
 arch/i386/kernel/head.S     |   69 +-----------------------
 arch/i386/mach-xen/Makefile |    2 
 arch/i386/mach-xen/head.S   |  122 ++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 222 insertions(+), 66 deletions(-)
arch/i386/Makefile +5 -2
@@ -48,6 +48,9 @@ CFLAGS				+= $(shell if [ $(call cc-vers
 
 CFLAGS += $(cflags-y)
 
+# Default subarch head files
+head-y := arch/i386/kernel/head.o arch/i386/kernel/init_task.o
+
 # Default subarch .c files
 mcore-y  := mach-default
 
@@ -74,6 +77,8 @@ mcore-$(CONFIG_X86_SUMMIT)  := mach-defa
 # Xen subarch support
 mflags-$(CONFIG_X86_XEN)	:= -Iinclude/asm-i386/mach-xen
 mcore-$(CONFIG_X86_XEN)		:= mach-xen
+head-$(CONFIG_X86_XEN)		:= arch/i386/mach-xen/head.o \
+				   arch/i386/kernel/init_task.o
 
 # generic subarchitecture
 mflags-$(CONFIG_X86_GENERICARCH) := -Iinclude/asm-i386/mach-generic
@@ -88,8 +93,6 @@ core-$(CONFIG_X86_ES7000)	:= arch/i386/m
 # default subarch .h files
 mflags-y += -Iinclude/asm-i386/mach-default
 
-head-y := arch/i386/kernel/head.o arch/i386/kernel/init_task.o
-
 libs-y 					+= arch/i386/lib/
 core-y					+= arch/i386/kernel/ \
 					   arch/i386/mm/ \
arch/i386/kernel/head.S +5 -64

89 lines not on this page · raw message

arch/i386/mach-xen/Makefile +2 -0
@@ -2,6 +2,8 @@
 # Makefile for the linux kernel.
 #
 
+extra-y				:= head.o
+
 obj-y				:= setup.o
 
 setup-y				:= ../mach-default/setup.o
arch/i386/kernel/head-cpu.S new file +88 -0

88 lines not on this page · raw message

arch/i386/mach-xen/head.S new file +122 -0

122 lines not on this page · raw message

--

#54 lore raw

RFC PATCH 11/35 Add support for Xen to entry.S.

In reply to Chris Wright

- change cli/sti
- change test for user mode return to work for kernel mode in ring1
- check hypervisor saved event mask on return from exception
- add entry points for the hypervisor upcall handlers
- avoid math emulation check when running on Xen
- add nmi handler for running on Xen

---
 arch/i386/kernel/entry.S |  137 +++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 120 insertions(+), 17 deletions(-)
arch/i386/kernel/entry.S +120 -17
@@ -75,8 +75,38 @@ DF_MASK		= 0x00000400 
 NT_MASK		= 0x00004000
 VM_MASK		= 0x00020000
 
+#ifndef CONFIG_XEN
+#define DISABLE_INTERRUPTS	cli
+#define ENABLE_INTERRUPTS	sti
+#else
+#include <xen/interface/xen.h>
+
+EVENT_MASK	= 0x2E
+
+/* Offsets into shared_info_t. */
+#define evtchn_upcall_pending		/* 0 */
+#define evtchn_upcall_mask		1
+
+#define sizeof_vcpu_shift		6
+
+#ifdef CONFIG_SMP
+#define GET_VCPU_INFO		movl TI_cpu(%ebp),%esi			; \
+				shl  $sizeof_vcpu_shift,%esi		; \
+				addl HYPERVISOR_shared_info,%esi
+#else
+#define GET_VCPU_INFO		movl HYPERVISOR_shared_info,%esi
+#endif
+
+#define __DISABLE_INTERRUPTS	movb $1,evtchn_upcall_mask(%esi)
+#define DISABLE_INTERRUPTS	GET_VCPU_INFO				; \
+				__DISABLE_INTERRUPTS
+#define ENABLE_INTERRUPTS	GET_VCPU_INFO				; \
+				movb $0,evtchn_upcall_mask(%esi)
+#define __TEST_PENDING		testb $0xFF,evtchn_upcall_pending(%esi)
+#endif
+
 #ifdef CONFIG_PREEMPT
-#define preempt_stop		cli
+#define preempt_stop		DISABLE_INTERRUPTS
 #else
 #define preempt_stop
 #define resume_kernel		restore_nocheck
@@ -145,10 +175,10 @@ ret_from_intr:
 	GET_THREAD_INFO(%ebp)
 	movl EFLAGS(%esp), %eax		# mix EFLAGS and CS
 	movb CS(%esp), %al
-	testl $(VM_MASK | 3), %eax
+	testl $(VM_MASK | USER_MODE_MASK), %eax
 	jz resume_kernel
 ENTRY(resume_userspace)
- 	cli				# make sure we don't miss an interrupt
+	DISABLE_INTERRUPTS		# make sure we don't miss an interrupt
 					# setting need_resched or sigpending
 					# between sampling and the iret
 	movl TI_flags(%ebp), %ecx
@@ -159,7 +189,7 @@ ENTRY(resume_userspace)
 
 #ifdef CONFIG_PREEMPT
 ENTRY(resume_kernel)
-	cli
+	DISABLE_INTERRUPTS
 	cmpl $0,TI_preempt_count(%ebp)	# non-zero preempt_count ?
 	jnz restore_nocheck
 need_resched:
@@ -179,7 +209,7 @@ need_resched:
 ENTRY(sysenter_entry)
 	movl TSS_sysenter_esp0(%esp),%esp
 sysenter_past_esp:
-	sti
+	ENABLE_INTERRUPTS
 	pushl $(__USER_DS)
 	pushl %ebp
 	pushfl
@@ -209,7 +239,7 @@ sysenter_past_esp:
 	jae syscall_badsys
 	call *sys_call_table(,%eax,4)
 	movl %eax,EAX(%esp)
-	cli
+	DISABLE_INTERRUPTS
 	movl TI_flags(%ebp), %ecx
 	testw $_TIF_ALLWORK_MASK, %cx
 	jne syscall_exit_work
@@ -217,7 +247,7 @@ sysenter_past_esp:
 	movl EIP(%esp), %edx
 	movl OLDESP(%esp), %ecx
 	xorl %ebp,%ebp
-	sti
+	ENABLE_INTERRUPTS
 	sysexit
 
 
@@ -240,7 +270,7 @@ syscall_call:
 	call *sys_call_table(,%eax,4)
 	movl %eax,EAX(%esp)		# store the return value
 syscall_exit:
-	cli				# make sure we don't miss an interrupt
+	DISABLE_INTERRUPTS		# make sure we don't miss an interrupt
 					# setting need_resched or sigpending
 					# between sampling and the iret
 	movl TI_flags(%ebp), %ecx
@@ -248,6 +278,7 @@ syscall_exit:
 	jne syscall_exit_work
 
 restore_all:
+#ifndef CONFIG_XEN
 	movl EFLAGS(%esp), %eax		# mix EFLAGS, SS and CS
 	# Warning: OLDSS(%esp) contains the wrong/random values if we
 	# are returning to the kernel.
@@ -258,12 +289,32 @@ restore_all:
 	cmpl $((4 << 8) | 3), %eax
 	je ldt_ss			# returning to user-space with LDT SS
 restore_nocheck:
+#else
+restore_nocheck:
+	movl EFLAGS(%esp), %eax		# mix EFLAGS and CS
+	movb CS(%esp), %al
+	andl $(VM_MASK | 3), %eax
+	cmpl $3, %eax
+	jne hypervisor_iret
+	ENABLE_INTERRUPTS
+	__TEST_PENDING
+	jz restore_regs_and_iret
+	__DISABLE_INTERRUPTS
+	jmp do_hypervisor_callback
+hypervisor_iret:
+	RESTORE_REGS
+	addl $4, %esp
+	jmp  hypercall_page + (__HYPERVISOR_iret * 32)
+#endif
+restore_regs_and_iret:
 	RESTORE_REGS
 	addl $4, %esp
 1:	iret
 .section .fixup,"ax"
 iret_exc:
-	sti
+#ifndef CONFIG_XEN
+	ENABLE_INTERRUPTS
+#endif
 	pushl $0			# no error code
 	pushl $do_iret_error
 	jmp error_code
@@ -273,6 +324,7 @@ iret_exc:
 	.long 1b,iret_exc
 .previous
 
+#ifndef CONFIG_XEN
 ldt_ss:
 	larl OLDSS(%esp), %eax
 	jnz restore_nocheck
@@ -285,7 +337,7 @@ ldt_ss:
 	 * CPUs, which we can try to work around to make
 	 * dosemu and wine happy. */
 	subl $8, %esp		# reserve space for switch16 pointer
-	cli
+	DISABLE_INTERRUPTS
 	movl %esp, %eax
 	/* Set up the 16bit stack frame with switch32 pointer on top,
 	 * and a switch16 pointer on top of the current frame. */
@@ -297,6 +349,7 @@ ldt_ss:
 	.align 4
 	.long 1b,iret_exc
 .previous
+#endif
 
 	# perform work that needs to be done immediately before resumption
 	ALIGN
@@ -305,7 +358,7 @@ work_pending:
 	jz work_notifysig
 work_resched:
 	call schedule
-	cli				# make sure we don't miss an interrupt
+	DISABLE_INTERRUPTS		# make sure we don't miss an interrupt
 					# setting need_resched or sigpending
 					# between sampling and the iret
 	movl TI_flags(%ebp), %ecx
@@ -357,7 +410,7 @@ syscall_trace_entry:
 syscall_exit_work:
 	testb $(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP), %cl
 	jz work_pending
-	sti				# could let do_syscall_trace() call
+	ENABLE_INTERRUPTS		# could let do_syscall_trace() call
 					# schedule() instead
 	movl %esp, %eax
 	movl $1, %edx
@@ -377,6 +430,7 @@ syscall_badsys:
 	movl $-ENOSYS,EAX(%esp)
 	jmp resume_userspace
 
+#ifndef CONFIG_XEN
 #define FIXUP_ESPFIX_STACK \
 	movl %esp, %eax; \
 	/* switch to 32bit stack using the pointer on top of 16bit stack */ \
@@ -435,6 +489,9 @@ ENTRY(name)				\
 
 /* The include is where all of the SMP etc. interrupts come from */
 #include "entry_arch.h"
+#else
+#define UNWIND_ESPFIX_STACK
+#endif
 
 ENTRY(divide_error)
 	pushl $0			# no error code
@@ -466,6 +523,44 @@ error_code:
 	call *%edi
 	jmp ret_from_exception
 
+#ifdef CONFIG_XEN
+ENTRY(hypervisor_callback)
+	pushl %eax
+	SAVE_ALL
+do_hypervisor_callback:
+	push %esp
+	call evtchn_do_upcall
+	add  $4,%esp
+	jmp  ret_from_intr
+
+# Hypervisor uses this for application faults while it executes.
+ENTRY(failsafe_callback)
+1:	popl %ds
+2:	popl %es
+3:	popl %fs
+4:	popl %gs
+	subl $4,%esp
+	SAVE_ALL
+	jmp  ret_from_exception
+.section .fixup,"ax";	\
+6:	movl $0,(%esp);	\
+	jmp 1b;		\
+7:	movl $0,(%esp);	\
+	jmp 2b;		\
+8:	movl $0,(%esp);	\
+	jmp 3b;		\
+9:	movl $0,(%esp);	\
+	jmp 4b;		\
+.previous;		\
+.section __ex_table,"a";\
+	.align 4;	\
+	.long 1b,6b;	\
+	.long 2b,7b;	\
+	.long 3b,8b;	\
+	.long 4b,9b;	\
+.previous
+#endif
+
 ENTRY(coprocessor_error)
 	pushl $0
 	pushl $do_coprocessor_error
@@ -479,17 +574,19 @@ ENTRY(simd_coprocessor_error)
 ENTRY(device_not_available)
 	pushl $-1			# mark this as an int
 	SAVE_ALL
+#ifndef CONFIG_XEN
 	movl %cr0, %eax
 	testl $0x4, %eax		# EM (math emulation bit)
-	jne device_not_available_emulate
-	preempt_stop
-	call math_state_restore
-	jmp ret_from_exception
-device_not_available_emulate:
+	je device_available_emulate
 	pushl $0			# temporary storage for ORIG_EIP
 	call math_emulate
 	addl $4, %esp
 	jmp ret_from_exception
+device_available_emulate:
+#endif
+	preempt_stop
+	call math_state_restore
+	jmp ret_from_exception
 
 /*
  * Debug traps and NMI can happen at the one SYSENTER instruction
@@ -525,6 +622,8 @@ debug_stack_correct:
 	call do_debug
 	jmp ret_from_exception
 	.previous .text
+
+#ifndef CONFIG_XEN
 /*
  * NMI is doubly nasty. It can happen _while_ we're handling
  * a debug fault, and the debug fault hasn't yet been able to
@@ -595,6 +694,10 @@ nmi_16bit_stack:
 	.align 4
 	.long 1b,iret_exc
 .previous
+#else
+ENTRY(nmi)
+	jmp restore_all
+#endif
 
 KPROBE_ENTRY(int3)
 	pushl $-1			# mark this as an int

--

#55 lore raw

Re: [RFC PATCH 11/35] Add support for Xen to entry.S.

Quoted 1 line · Chris Wright

+#define sizeof_vcpu_shift 6

This should be generated in asm-offsets.c

arch/i386/kernel/entry.S DF_MASK = 0x00000400 · line 95

Quoted 5 lines
arch/i386/kernel/entry.S +5 -0
+
+#ifdef CONFIG_SMP
+#define GET_VCPU_INFO		movl TI_cpu(%ebp),%esi			; \
+				shl  $sizeof_vcpu_shift,%esi		; \
+				addl HYPERVISOR_shared_info,%esi

I think you need some comments on the register usage in the macros. Otherwise people hacking on it later will go crazy.

arch/i386/kernel/entry.S restore_all: · line 292

Quoted 10 lines

restore_all:

arch/i386/kernel/entry.S +2 -0
+#ifndef CONFIG_XEN
 	movl EFLAGS(%esp), %eax		# mix EFLAGS, SS and CS
 	# Warning: OLDSS(%esp) contains the wrong/random values if we
 	# are returning to the kernel.
@@ -258,12 +289,32 @@ restore_all:
 	cmpl $((4 << 8) | 3), %eax
 	je ldt_ss			# returning to user-space with LDT SS
 restore_nocheck:
+#else

Needs comment

arch/i386/kernel/entry.S restore_all:

Quoted 8 lines
arch/i386/kernel/entry.S +7 -0
+restore_nocheck:
+	movl EFLAGS(%esp), %eax		# mix EFLAGS and CS
+	movb CS(%esp), %al
+	andl $(VM_MASK | 3), %eax
+	cmpl $3, %eax
+	jne hypervisor_iret
+	ENABLE_INTERRUPTS
 

-Andi

#56 lore raw

RFC PATCH 12/35 Add start-of-day setup hooks to subarch

In reply to Chris Wright

Implement the start-of-day subarchitecture setup hooks for booting on Xen. Add subarch macros for determining loader type and initrd location.

---
 arch/i386/kernel/setup.c                    |    5 ++
 include/asm-i386/hypervisor.h               |    4 +
 include/asm-i386/mach-default/mach_setup.h  |   12 +++++
 include/asm-i386/mach-xen/mach_setup.h      |   13 ++++++
 include/asm-i386/mach-xen/setup_arch_post.h |   60 ++++++++++++++++++++++++++++
 include/asm-i386/mach-xen/setup_arch_pre.h  |   22 ++++++++++
 include/asm-i386/setup.h                    |    8 ++-
 7 files changed, 121 insertions(+), 3 deletions(-)
arch/i386/kernel/setup.c +5 -0
@@ -458,6 +458,7 @@ static void __init print_memory_map(char
 	}
 }
 
+#ifndef HAVE_ARCH_E820_SANITIZE
 /*
  * Sanitize the BIOS e820 map.
  *
@@ -677,6 +678,7 @@ static int __init copy_e820_map(struct e
 	} while (biosmap++,--nr_map);
 	return 0;
 }
+#endif
 
 #if defined(CONFIG_EDD) || defined(CONFIG_EDD_MODULE)
 struct edd edd;
@@ -1578,6 +1580,9 @@ void __init setup_arch(char **cmdline_p)
 	conswitchp = &dummy_con;
 #endif
 #endif
+#ifdef ARCH_FINAL_SETUP
+	ARCH_FINAL_SETUP
+#endif
 }
 
 static __init int add_pcspkr(void)
include/asm-i386/hypervisor.h +4 -0
@@ -56,6 +56,10 @@
 extern struct shared_info *HYPERVISOR_shared_info;
 extern struct start_info *xen_start_info;
 
+/* arch/i386/mach-xen/entry.S */
+extern void hypervisor_callback(void);
+extern void failsafe_callback(void);
+
 /* arch/i386/mach-xen/evtchn.c */
 /* Force a proper event-channel callback from Xen. */
 extern void force_evtchn_callback(void);
include/asm-i386/setup.h +5 -3
@@ -49,10 +49,10 @@ extern unsigned char boot_params[PARAM_S
 #define VIDEO_MODE (*(unsigned short *) (PARAM+0x1FA))
 #define ORIG_ROOT_DEV (*(unsigned short *) (PARAM+0x1FC))
 #define AUX_DEVICE_INFO (*(unsigned char *) (PARAM+0x1FF))
-#define LOADER_TYPE (*(unsigned char *) (PARAM+0x210))
+#define LOADER_TYPE MACH_LOADER_TYPE
 #define KERNEL_START (*(unsigned long *) (PARAM+0x214))
-#define INITRD_START (*(unsigned long *) (PARAM+0x218))
-#define INITRD_SIZE (*(unsigned long *) (PARAM+0x21c))
+#define INITRD_START MACH_INITRD_START
+#define INITRD_SIZE MACH_INITRD_SIZE
 #define EDID_INFO   (*(struct edid_info *) (PARAM+0x140))
 #define EDD_NR     (*(unsigned char *) (PARAM+EDDNR))
 #define EDD_MBR_SIG_NR (*(unsigned char *) (PARAM+EDD_MBR_SIG_NR_BUF))
@@ -61,4 +61,6 @@ extern unsigned char boot_params[PARAM_S
 
 #endif /* __ASSEMBLY__ */
 
+#include <mach_setup.h>
+
 #endif /* _i386_SETUP_H */
include/asm-i386/mach-default/mach_setup.h new file +12 -0
@@ -0,0 +1,12 @@
+#ifndef __ASM_MACH_SETUP_H
+#define __ASM_MACH_SETUP_H
+
+#ifndef __ASSEMBLY__
+
+#define MACH_LOADER_TYPE (*(unsigned char *) (PARAM+0x210))
+#define MACH_INITRD_START (*(unsigned long *) (PARAM+0x218))
+#define MACH_INITRD_SIZE (*(unsigned long *) (PARAM+0x21c))
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* __ASM_MACH_SETUP_H */
include/asm-i386/mach-xen/mach_setup.h new file +13 -0
@@ -0,0 +1,13 @@
+#ifndef __ASM_MACH_SETUP_H
+#define __ASM_MACH_SETUP_H
+
+#ifndef __ASSEMBLY__
+
+#define MACH_LOADER_TYPE 0x6e6558	/* "Xen" */
+#define MACH_INITRD_START \
+	(xen_start_info->mod_start ? __pa(xen_start_info->mod_start) : 0)
+#define MACH_INITRD_SIZE (xen_start_info->mod_len)
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* __ASM_MACH_SETUP_H */
include/asm-i386/mach-xen/setup_arch_post.h new file +60 -0

60 lines not on this page · raw message

include/asm-i386/mach-xen/setup_arch_pre.h new file +22 -0
@@ -0,0 +1,22 @@
+
+#include <xen/interface/xen.h>
+#include <asm/hypervisor.h>
+
+struct start_info *xen_start_info;
+EXPORT_SYMBOL(xen_start_info);
+
+/*
+ * Point at the empty zero page to start with. We map the real shared_info
+ * page as soon as fixmap is up and running.
+ */
+struct shared_info *HYPERVISOR_shared_info =
+	(struct shared_info *)empty_zero_page;
+EXPORT_SYMBOL(HYPERVISOR_shared_info);
+
+#define ARCH_SETUP machine_specific_arch_setup();
+#define ARCH_FINAL_SETUP machine_specific_arch_final_setup();
+
+static void __init machine_specific_arch_setup(void);
+static void __init machine_specific_arch_final_setup(void);
+
+#define HAVE_ARCH_E820_SANITIZE

--

#57 lore raw

RFC PATCH 13/35 Support loading an initrd when running on Xen

In reply to Chris Wright

Due to the initial physical memory layout when booting on Xen, the initrd image ends up below min_low_pfn (as registered with the bootstrap memory allocator). Add Xen subarch support to enable initrd_below_start_ok flag, and disable initrd_reserve_bootmem.

---
 arch/i386/kernel/setup.c                    |    4 +++-
 include/asm-i386/mach-xen/setup_arch_post.h |    6 ++++++
 2 files changed, 9 insertions(+), 1 deletion(-)
arch/i386/kernel/setup.c +3 -1
@@ -1211,6 +1211,7 @@ void __init zone_sizes_init(void)
 extern unsigned long __init setup_memory(void);
 extern void zone_sizes_init(void);
 #endif /* !CONFIG_NEED_MULTIPLE_NODES */
+int initrd_reserve_bootmem = 1;
 
 void __init setup_bootmem_allocator(void)
 {
@@ -1271,7 +1272,8 @@ void __init setup_bootmem_allocator(void
 #ifdef CONFIG_BLK_DEV_INITRD
 	if (LOADER_TYPE && INITRD_START) {
 		if (INITRD_START + INITRD_SIZE <= (max_low_pfn << PAGE_SHIFT)) {
-			reserve_bootmem(INITRD_START, INITRD_SIZE);
+			if (initrd_reserve_bootmem)
+				reserve_bootmem(INITRD_START, INITRD_SIZE);
 			initrd_start =
 				INITRD_START ? INITRD_START + PAGE_OFFSET : 0;
 			initrd_end = initrd_start+INITRD_SIZE;
include/asm-i386/mach-xen/setup_arch_post.h +6 -0
@@ -48,6 +48,12 @@ static void __init machine_specific_arch
 	memcpy(saved_command_line, xen_start_info->cmd_line,
 	       MAX_GUEST_CMDLINE > COMMAND_LINE_SIZE ?
 	       COMMAND_LINE_SIZE : MAX_GUEST_CMDLINE);
+
+	if (INITRD_START &&
+	    INITRD_START + PAGE_OFFSET < min_low_pfn << PAGE_SHIFT) {
+		initrd_reserve_bootmem = 0;
+		initrd_below_start_ok = 1;
+	}
 }
 
 static void __init machine_specific_arch_final_setup(void)

--

#58 lore raw

RFC PATCH 14/35 Subarch support for CPUID instruction

In reply to Chris Wright

Allow subarchitectures to modify the CPUID instruction. This allows the subarch to provide a limited set of CPUID feature flags during CPU identification. Add a subarch implementation for Xen that traps to the hypervisor where unsupported feature flags can be hidden from guests.

---
 arch/i386/kernel/head-cpu.S                    |    4 ++--
 include/asm-i386/mach-default/mach_processor.h |    7 +++++++
 include/asm-i386/mach-xen/mach_processor.h     |    9 +++++++++
 include/asm-i386/processor.h                   |   13 +++++++------
 4 files changed, 25 insertions(+), 8 deletions(-)
include/asm-i386/processor.h +7 -6
@@ -21,6 +21,7 @@
 #include <linux/threads.h>
 #include <asm/percpu.h>
 #include <linux/cpumask.h>
+#include <mach_processor.h>
 
 /* flag for disabling the tsc */
 extern int tsc_disable;
@@ -148,7 +149,7 @@ static inline void detect_ht(struct cpui
  */
 static inline void cpuid(unsigned int op, unsigned int *eax, unsigned int *ebx, unsigned int *ecx, unsigned int *edx)
 {
-	__asm__("cpuid"
+	__asm__(CPUID_STR
 		: "=a" (*eax),
 		  "=b" (*ebx),
 		  "=c" (*ecx),
@@ -160,7 +161,7 @@ static inline void cpuid(unsigned int op
 static inline void cpuid_count(int op, int count, int *eax, int *ebx, int *ecx,
 	       	int *edx)
 {
-	__asm__("cpuid"
+	__asm__(CPUID_STR
 		: "=a" (*eax),
 		  "=b" (*ebx),
 		  "=c" (*ecx),
@@ -175,7 +176,7 @@ static inline unsigned int cpuid_eax(uns
 {
 	unsigned int eax;
 
-	__asm__("cpuid"
+	__asm__(CPUID_STR
 		: "=a" (eax)
 		: "0" (op)
 		: "bx", "cx", "dx");
@@ -185,7 +186,7 @@ static inline unsigned int cpuid_ebx(uns
 {
 	unsigned int eax, ebx;
 
-	__asm__("cpuid"
+	__asm__(CPUID_STR
 		: "=a" (eax), "=b" (ebx)
 		: "0" (op)
 		: "cx", "dx" );
@@ -195,7 +196,7 @@ static inline unsigned int cpuid_ecx(uns
 {
 	unsigned int eax, ecx;
 
-	__asm__("cpuid"
+	__asm__(CPUID_STR
 		: "=a" (eax), "=c" (ecx)
 		: "0" (op)
 		: "bx", "dx" );
@@ -205,7 +206,7 @@ static inline unsigned int cpuid_edx(uns
 {
 	unsigned int eax, edx;
 
-	__asm__("cpuid"
+	__asm__(CPUID_STR
 		: "=a" (eax), "=d" (edx)
 		: "0" (op)
 		: "bx", "cx");
arch/i386/kernel/head-cpu.S +2 -2
@@ -59,7 +59,7 @@
 .macro CPUID_GET_VENDOR_INFO cpuid_level, x86_vendor_id
 	/* get vendor info */
 	xorl %eax,%eax			# call CPUID with 0 -> return vendor ID
-	cpuid
+	CPUID
 	movl %eax,\cpuid_level		# save CPUID level
 	movl %ebx,\x86_vendor_id	# lo 4 chars
 	movl %edx,\x86_vendor_id+4	# next 4 chars
@@ -75,7 +75,7 @@
  */
 .macro CPUID_GET_CPU_TYPE family, model, mask, capability
 	movl $1,%eax		# Use the CPUID instruction to get CPU type
-	cpuid
+	CPUID
 	movb %al,%cl		# save reg for future use
 	andb $0x0f,%ah		# mask processor family
 	movb %ah,\family
include/asm-i386/mach-default/mach_processor.h new file +7 -0
@@ -0,0 +1,7 @@
+#ifndef __ASM_MACH_PROCESSOR_H
+#define __ASM_MACH_PROCESSOR_H
+
+#define CPUID cpuid
+#define CPUID_STR "cpuid"
+
+#endif /* __ASM_MACH_PROCESSOR_H */
include/asm-i386/mach-xen/mach_processor.h new file +9 -0
@@ -0,0 +1,9 @@
+#ifndef __ASM_MACH_PROCESSOR_H
+#define __ASM_MACH_PROCESSOR_H
+
+#include <xen/interface/arch-x86_32.h>
+
+#define CPUID XEN_CPUID
+#define CPUID_STR XEN_CPUID
+
+#endif /* __ASM_MACH_PROCESSOR_H */

--

#59 lore raw

RFC PATCH 15/35 subarch support for controlling interrupt delivery

In reply to Chris Wright

Abstract the code that controls interrupt delivery, and add a separate subarch implementation for Xen that manipulates a shared-memory event delivery mask.

---
 include/asm-i386/mach-default/mach_system.h |   24 ++++++
 include/asm-i386/mach-xen/mach_system.h     |  103 ++++++++++++++++++++++++++++
 include/asm-i386/system.h                   |   20 -----
 3 files changed, 128 insertions(+), 19 deletions(-)
include/asm-i386/system.h +1 -19
@@ -490,25 +490,7 @@ static inline unsigned long long __cmpxc
 
 #define set_wmb(var, value) do { var = value; wmb(); } while (0)
 
-/* interrupt control.. */
-#define local_save_flags(x)	do { typecheck(unsigned long,x); __asm__ __volatile__("pushfl ; popl %0":"=g" (x): /* no input */); } while (0)
-#define local_irq_restore(x) 	do { typecheck(unsigned long,x); __asm__ __volatile__("pushl %0 ; popfl": /* no output */ :"g" (x):"memory", "cc"); } while (0)
-#define local_irq_disable() 	__asm__ __volatile__("cli": : :"memory")
-#define local_irq_enable()	__asm__ __volatile__("sti": : :"memory")
-/* used in the idle loop; sti takes one instruction cycle to complete */
-#define safe_halt()		__asm__ __volatile__("sti; hlt": : :"memory")
-/* used when interrupts are already enabled or to shutdown the processor */
-#define halt()			__asm__ __volatile__("hlt": : :"memory")
-
-#define irqs_disabled()			\
-({					\
-	unsigned long flags;		\
-	local_save_flags(flags);	\
-	!(flags & (1<<9));		\
-})
-
-/* For spinlocks etc */
-#define local_irq_save(x)	__asm__ __volatile__("pushfl ; popl %0 ; cli":"=g" (x): /* no input */ :"memory")
+#include <mach_system.h>
 
 /*
  * disable hlt during certain critical i/o operations
include/asm-i386/mach-default/mach_system.h new file +24 -0
@@ -0,0 +1,24 @@
+#ifndef __ASM_MACH_SYSTEM_H
+#define __ASM_MACH_SYSTEM_H
+
+/* interrupt control.. */
+#define local_save_flags(x)	do { typecheck(unsigned long,x); __asm__ __volatile__("pushfl ; popl %0":"=g" (x): /* no input */); } while (0)
+#define local_irq_restore(x) 	do { typecheck(unsigned long,x); __asm__ __volatile__("pushl %0 ; popfl": /* no output */ :"g" (x):"memory", "cc"); } while (0)
+#define local_irq_disable() 	__asm__ __volatile__("cli": : :"memory")
+#define local_irq_enable()	__asm__ __volatile__("sti": : :"memory")
+/* used in the idle loop; sti takes one instruction cycle to complete */
+#define safe_halt()		__asm__ __volatile__("sti; hlt": : :"memory")
+/* used when interrupts are already enabled or to shutdown the processor */
+#define halt()			__asm__ __volatile__("hlt": : :"memory")
+
+#define irqs_disabled()			\
+({					\
+	unsigned long flags;		\
+	local_save_flags(flags);	\
+	!(flags & (1<<9));		\
+})
+
+/* For spinlocks etc */
+#define local_irq_save(x)	__asm__ __volatile__("pushfl ; popl %0 ; cli":"=g" (x): /* no input */ :"memory")
+
+#endif /* __ASM_MACH_SYSTEM_H */
include/asm-i386/mach-xen/mach_system.h new file +103 -0

103 lines not on this page · raw message

--

#60 lore raw

Re: [RFC PATCH 15/35] subarch support for controlling interrupt delivery

Quoted 7 lines

+/*
+ * The use of 'barrier' in the following reflects their use as local-lock
+ * operations. Reentrancy must be prevented (e.g., __cli()) /before/ following
+ * critical operations are executed. All critical operations must complete
+ * /before/ reentrancy is permitted (e.g., __sti()). Alpha architecture also
+ * includes these barriers, for example.
+ */

Seems like an odd comment to have in an i386 header file.

include/asm-i386/mach-xen/mach_system.h line 32

Quoted 9 lines
include/asm-i386/mach-xen/mach_system.h new file +9 -0
+#define __cli()								\
+do {									\
+	struct vcpu_info *_vcpu;					\
+	preempt_disable();						\
+	_vcpu = &HYPERVISOR_shared_info->vcpu_info[__vcpu_id];		\
+	_vcpu->evtchn_upcall_mask = 1;					\
+	preempt_enable_no_resched();					\
+	barrier();							\
+} while (0)

Should be a real function

include/asm-i386/mach-xen/mach_system.h line 45

Quoted 12 lines
include/asm-i386/mach-xen/mach_system.h new file +12 -0
+#define __sti()								\
+do {									\
+	struct vcpu_info *_vcpu;					\
+	barrier();							\
+	preempt_disable();						\
+	_vcpu = &HYPERVISOR_shared_info->vcpu_info[__vcpu_id];		\
+	_vcpu->evtchn_upcall_mask = 0;					\
+	barrier(); /* unmask then check (avoid races) */		\
+	if (unlikely(_vcpu->evtchn_upcall_pending))			\
+		force_evtchn_callback();				\
+	preempt_enable();						\
+} while (0)

Should be a real function

include/asm-i386/mach-xen/mach_system.h line 54

Quoted 8 lines
include/asm-i386/mach-xen/mach_system.h new file +8 -0
+#define __save_flags(x)							\
+do {									\
+	struct vcpu_info *_vcpu;					\
+	preempt_disable();						\
+	_vcpu = &HYPERVISOR_shared_info->vcpu_info[__vcpu_id];		\
+	(x) = _vcpu->evtchn_upcall_mask;				\
+	preempt_enable();						\
+} while (0)

Should be a real function

Quoted 4 lines
include/asm-i386/mach-xen/mach_system.h new file +4 -0
+#define __restore_flags(x)						\
+do {									\
+	struct vcpu_info *_vcpu;					\
+	barrier();							\

include/asm-i386/mach-xen/mach_system.h line 69

Quoted 10 lines
include/asm-i386/mach-xen/mach_system.h new file +10 -0
+	preempt_disable();						\
+	_vcpu = &HYPERVISOR_shared_info->vcpu_info[__vcpu_id];		\
+	if ((_vcpu->evtchn_upcall_mask = (x)) == 0) {			\
+		barrier(); /* unmask then check (avoid races) */	\
+		if (unlikely(_vcpu->evtchn_upcall_pending))		\
+			force_evtchn_callback();			\
+		preempt_enable();					\
+	} else								\
+		preempt_enable_no_resched();				\
+} while (0)

Should be a real function

include/asm-i386/mach-xen/mach_system.h line 83

Quoted 13 lines
include/asm-i386/mach-xen/mach_system.h new file +13 -0
+#define safe_halt()		((void)0)
+#define halt()			((void)0)
+
+#define __save_and_cli(x)						\
+do {									\
+	struct vcpu_info *_vcpu;					\
+	preempt_disable();						\
+	_vcpu = &HYPERVISOR_shared_info->vcpu_info[__vcpu_id];		\
+	(x) = _vcpu->evtchn_upcall_mask;				\
+	_vcpu->evtchn_upcall_mask = 1;					\
+	preempt_enable_no_resched();					\
+	barrier();							\
+} while (0)

Should be a real function

Quoted 7 lines
include/asm-i386/mach-xen/mach_system.h new file +7 -0
+#define local_irq_save(x)	__save_and_cli(x)
+#define local_irq_restore(x)	__restore_flags(x)
+#define local_save_flags(x)	__save_flags(x)
+#define local_irq_disable()	__cli()
+#define local_irq_enable()	__sti()
+
+/* Cannot use preempt_enable() here as we would recurse in preempt_sched(). */

include/asm-i386/mach-xen/mach_system.h line 101

Quoted 10 lines
include/asm-i386/mach-xen/mach_system.h new file +10 -0
+#define irqs_disabled()							\
+({	int ___x;							\
+	struct vcpu_info *_vcpu;					\
+	preempt_disable();						\
+	_vcpu = &HYPERVISOR_shared_info->vcpu_info[__vcpu_id];		\
+	___x = (_vcpu->evtchn_upcall_mask != 0);			\
+	preempt_enable_no_resched();					\
+	___x; })
+
+#endif /* __KERNEL__ */

Should be a real function

#61 lore raw

Re: [RFC PATCH 15/35] subarch support for controlling interrupt delivery

Quoted 13 lines · Martin J. Bligh
Quoted 11 lines

+/*
+ * The use of 'barrier' in the following reflects their use as local-lock
+ * operations. Reentrancy must be prevented (e.g., __cli()) /before/ following
+ * critical operations are executed. All critical operations must complete
+ * /before/ reentrancy is permitted (e.g., __sti()). Alpha architecture also
+ * includes these barriers, for example.
+ */

Quoted 1 line

Seems like an odd comment to have in an i386 header file.

Also, it is only talking about compiler barriers, which have nothing to do with the architecture.

And preempt_* macros should contain the correct compiler barriers, so several can be removed.

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#62 lore raw

Re: [RFC PATCH 15/35] subarch support for controlling interrupt delivery

In reply to Martin J. Bligh
Quoted 11 lines · Martin J. Bligh

include/asm-i386/mach-xen/mach_system.h line 32

Quoted 9 lines
include/asm-i386/mach-xen/mach_system.h new file +9 -0
+#define __cli()							 \
+do {									\
+	struct vcpu_info *_vcpu;					\
+	preempt_disable();						\
+	_vcpu = &HYPERVISOR_shared_info->vcpu_info[__vcpu_id];		\
+	_vcpu->evtchn_upcall_mask = 1;					\
+	preempt_enable_no_resched();					\
+	barrier();							\
+} while (0)

Should be a real function

Yes, except it's not trivially done because if __cli was an inline function, you need to have everything that is used in the declaration defined when the function is declared as opposed to when the #define gets used. I'll give it another try, but it very quickly becomes #include hell.

Anybody want to comment on the performance impact of making local_irq_* non-inline functions?

christian

#63 lore raw

Re: [RFC PATCH 15/35] subarch support for controlling interrupt delivery

Quoted 15 lines · Christian Limpach
Quoted 11 lines · Martin J. Bligh

include/asm-i386/mach-xen/mach_system.h line 32

Quoted 9 lines
include/asm-i386/mach-xen/mach_system.h new file +9 -0
+#define __cli()							 \
+do {									\
+	struct vcpu_info *_vcpu;					\
+	preempt_disable();						\
+	_vcpu = &HYPERVISOR_shared_info->vcpu_info[__vcpu_id];		\
+	_vcpu->evtchn_upcall_mask = 1;					\
+	preempt_enable_no_resched();					\
+	barrier();							\
+} while (0)

Should be a real function

Quoted 8 lines

Yes, except it's not trivially done because if __cli was an inline function, you need to have everything that is used in the declaration defined when the function is declared as opposed to when the #define gets used. I'll give it another try, but it very quickly becomes #include hell.

Anybody want to comment on the performance impact of making local_irq_* non-inline functions?

I wasn't concerned with inline vs non-inline - that's your choice. Just the inherent foulness of multi-line macros ;-)

M.

#64 lore raw

Re: [RFC PATCH 15/35] subarch support for controlling interrupt delivery

In reply to Christian Limpach
Quoted 4 lines

Anybody want to comment on the performance impact of making local_irq_* non-inline functions?

I would guess for that much inline code it will be even a win to not inline because it will save icache.

-Andi

#65 lore raw

Re: [RFC PATCH 15/35] subarch support for controlling interrupt delivery

Quoted 7 lines · Andi Kleen
Quoted 4 lines

Anybody want to comment on the performance impact of making local_irq_* non-inline functions?

I would guess for that much inline code it will be even a win to not inline because it will save icache.

Maybe, although some of the macros compile down to only 2-3 instructions.

christian

#66 lore raw

Re: [RFC PATCH 15/35] subarch support for controlling interrupt delivery

Quoted 10 lines · Christian Limpach
Quoted 7 lines · Andi Kleen
Quoted 4 lines

Anybody want to comment on the performance impact of making local_irq_* non-inline functions?

I would guess for that much inline code it will be even a win to not inline because it will save icache.

Maybe, although some of the macros compile down to only 2-3 instructions.

Can you post before/after vmlinux size numbers for inline/out of line?

-Andi

#67 lore raw

Re: [RFC PATCH 15/35] subarch support for controlling interrupt delivery

Quoted 7 lines · Andi Kleen
Quoted 6 lines · Christian Limpach
Quoted 5 lines · Andi Kleen
Quoted 4 lines

Anybody want to comment on the performance impact of making local_irq_* non-inline functions?

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

I would guess for that much inline code it will be even a win to not inline because it will save icache.

Maybe, although some of the macros compile down to only 2-3 instructions.

Can you post before/after vmlinux size numbers for inline/out of line?

Sure, although it is a bit tricky since the #define's pass non-pointer arguments by reference. This would also make it quite ugly to change these.

Everything[1] in line:
-rwxr-xr-x 1 cl349 cl349 2633640 May 9 19:42 vmlinux-inline-stripped Everything out of line:
-rwxr-xr-x 1 cl349 cl349 2621352 May 9 19:45 vmlinux-outline-stripped

Additionally, I changed did a build with only __sti and __restore_flags out of line and the others in line:
-rwxr-xr-x 1 cl349 cl349 2617256 May 9 19:50 vmlinux-hybrid-stripped

__sti and __restore_flags are the ones which generate more code, so it seemed more sensible to make the out of line.

Any conlusions?

christian

[1] __cli, __sti, __save_flags, __restore_flags, __save_and_cli, irqs_disabled

#68 lore raw

Re: [RFC PATCH 15/35] subarch support for controlling interrupt delivery

Quoted 4 lines · * Christian Limpach ([email protected]) wrote:

Everything[1] in line:
-rwxr-xr-x 1 cl349 cl349 2633640 May 9 19:42 vmlinux-inline-stripped Everything out of line:
-rwxr-xr-x 1 cl349 cl349 2621352 May 9 19:45 vmlinux-outline-stripped

Have the output of 'size vmlinux*' handy? Be nice to get the extra details.

thanks,
-chris

#69 lore raw

Re: [RFC PATCH 15/35] subarch support for controlling interrupt delivery

In reply to Christian Limpach
Quoted 6 lines

Everything[1] in line:
-rwxr-xr-x 1 cl349 cl349 2633640 May 9 19:42 vmlinux-inline-stripped Everything out of line:
-rwxr-xr-x 1 cl349 cl349 2621352 May 9 19:45 vmlinux-outline-stripped

Quoted 8 lines

Additionally, I changed did a build with only __sti and __restore_flags out of line and the others in line:
-rwxr-xr-x 1 cl349 cl349 2617256 May 9 19:50 vmlinux-hybrid-stripped

__sti and __restore_flags are the ones which generate more code, so it seemed more sensible to make the out of line.

Any conlusions?

It looks like hybrid is a clear winner at least from the code size, isn't it?

I doubt you will be able to benchmark the difference for anything else anyways so might as well aim for that.

-Andi

#70 lore raw

Re: [RFC PATCH 15/35] subarch support for controlling interrupt delivery

Quoted 10 lines · Andi Kleen
Quoted 10 lines

Everything[1] in line:
-rwxr-xr-x 1 cl349 cl349 2633640 May 9 19:42 vmlinux-inline-stripped Everything out of line:
-rwxr-xr-x 1 cl349 cl349 2621352 May 9 19:45 vmlinux-outline-stripped

Additionally, I changed did a build with only __sti and __restore_flags out of line and the others in line:
-rwxr-xr-x 1 cl349 cl349 2617256 May 9 19:50 vmlinux-hybrid-stripped

Quoted 6 lines
Quoted 4 lines

__sti and __restore_flags are the ones which generate more code, so it seemed more sensible to make the out of line.

Any conlusions?

It looks like hybrid is a clear winner at least from the code size, isn't it?

Yes, which is why I measured that one as well.

Now, the original concern was that we have the five operations implemented as multi-line macros and doing a hybrid solution doesn't really address that.

Also, it's not quite clear to me what's the best way to turn three of the five into functions, whether inline or not.

For measuring the sizes, I did the following:
add void ___restore_flags(unsigned long *x) with the implementation and then:
#define __restore_flags(x) ___restore_flags(&(x))

Alternatively, would it make sense to change __restore_flags to take a pointer to flags instead? That would be quite an invasive change...

Any thoughts?

christian

#71 lore raw

Re: [RFC PATCH 15/35] subarch support for controlling interrupt delivery

es, which is why I measured that one as well.

Quoted 4 lines

Now, the original concern was that we have the five operations implemented as multi-line macros and doing a hybrid solution doesn't really address that.

If it's straight-forward to convert to an inline do it. If not keep it as a macro. After all code style is just a tool, not something self serving.

Quoted 8 lines

Also, it's not quite clear to me what's the best way to turn three of the five into functions, whether inline or not.

For measuring the sizes, I did the following:
add void ___restore_flags(unsigned long *x) with the implementation and then:
#define __restore_flags(x) ___restore_flags(&(x))

Yes that is the standard way to do it

Quoted 2 lines

Alternatively, would it make sense to change __restore_flags to take a pointer to flags instead? That would be quite an invasive change...

No.

-Andi

#72 lore raw

RFC PATCH 16/35 subarch support for interrupt and exception gates

In reply to Chris Wright

Abstract the code that sets up interrupt and exception gates, and add a separate subarch implementation for Xen.

---
 arch/i386/kernel/traps.c                   |   49 ---------------------------
 include/asm-i386/mach-default/mach_idt.h   |   52 +++++++++++++++++++++++++++++
 include/asm-i386/mach-xen/mach_idt.h       |   50 +++++++++++++++++++++++++++
 include/asm-i386/mach-xen/setup_arch_pre.h |    2 +
 4 files changed, 105 insertions(+), 48 deletions(-)
arch/i386/kernel/traps.c +1 -48
@@ -1086,54 +1086,7 @@ void __init trap_init_f00f_bug(void)
 }
 #endif
 
-#define _set_gate(gate_addr,type,dpl,addr,seg) \
-do { \
-  int __d0, __d1; \
-  __asm__ __volatile__ ("movw %%dx,%%ax\n\t" \
-	"movw %4,%%dx\n\t" \
-	"movl %%eax,%0\n\t" \
-	"movl %%edx,%1" \
-	:"=m" (*((long *) (gate_addr))), \
-	 "=m" (*(1+(long *) (gate_addr))), "=&a" (__d0), "=&d" (__d1) \
-	:"i" ((short) (0x8000+(dpl<<13)+(type<<8))), \
-	 "3" ((char *) (addr)),"2" ((seg) << 16)); \
-} while (0)
-
-
-/*
- * This needs to use 'idt_table' rather than 'idt', and
- * thus use the _nonmapped_ version of the IDT, as the
- * Pentium F0 0F bugfix can have resulted in the mapped
- * IDT being write-protected.
- */
-void set_intr_gate(unsigned int n, void *addr)
-{
-	_set_gate(idt_table+n,14,0,addr,__KERNEL_CS);
-}
-
-/*
- * This routine sets up an interrupt gate at directory privilege level 3.
- */
-static inline void set_system_intr_gate(unsigned int n, void *addr)
-{
-	_set_gate(idt_table+n, 14, 3, addr, __KERNEL_CS);
-}
-
-static void __init set_trap_gate(unsigned int n, void *addr)
-{
-	_set_gate(idt_table+n,15,0,addr,__KERNEL_CS);
-}
-
-static void __init set_system_gate(unsigned int n, void *addr)
-{
-	_set_gate(idt_table+n,15,3,addr,__KERNEL_CS);
-}
-
-static void __init set_task_gate(unsigned int n, unsigned int gdt_entry)
-{
-	_set_gate(idt_table+n,5,0,0,(gdt_entry<<3));
-}
-
+#include <mach_idt.h>
 
 void __init trap_init(void)
 {
include/asm-i386/mach-xen/setup_arch_pre.h +2 -0
@@ -5,6 +5,8 @@
 struct start_info *xen_start_info;
 EXPORT_SYMBOL(xen_start_info);
 
+struct trap_info xen_trap_table[257];
+
 /*
  * Point at the empty zero page to start with. We map the real shared_info
  * page as soon as fixmap is up and running.
include/asm-i386/mach-default/mach_idt.h new file +52 -0

52 lines not on this page · raw message

include/asm-i386/mach-xen/mach_idt.h new file +50 -0

50 lines not on this page · raw message

--

#73 lore raw

Re: [RFC PATCH 16/35] subarch support for interrupt and exception gates

include/asm-i386/mach-default/mach_idt.h line 27

Quoted 11 lines
include/asm-i386/mach-default/mach_idt.h new file +10 -0
+/*
+ * This needs to use 'idt_table' rather than 'idt', and
+ * thus use the _nonmapped_ version of the IDT, as the
+ * Pentium F0 0F bugfix can have resulted in the mapped
+ * IDT being write-protected.
+ */
+void set_intr_gate(unsigned int n, void *addr)
+{
+	_set_gate(idt_table+n,14,0,addr,__KERNEL_CS);
+}

No need to duplicate the various set_*_gate functions into the subarchs.

include/asm-i386/mach-xen/mach_idt.h line 48

Quoted 4 lines
include/asm-i386/mach-xen/mach_idt.h new file +4 -0
+static void __init set_task_gate(unsigned int n, unsigned int gdt_entry)
+{
+	/* _set_gate(n, 5, 0, 0, (gdt_entry<<3)); */
+}

Looks weird, but can be handled in the low level function.

-Andi

#74 lore raw

Re: [RFC PATCH 16/35] subarch support for interrupt and exception gates

Quoted 13 lines · Andi Kleen

include/asm-i386/mach-default/mach_idt.h line 27

Quoted 11 lines
include/asm-i386/mach-default/mach_idt.h new file +10 -0
+/*
+ * This needs to use 'idt_table' rather than 'idt', and
+ * thus use the _nonmapped_ version of the IDT, as the
+ * Pentium F0 0F bugfix can have resulted in the mapped
+ * IDT being write-protected.
+ */
+void set_intr_gate(unsigned int n, void *addr)
+{
+	_set_gate(idt_table+n,14,0,addr,__KERNEL_CS);
+}

No need to duplicate the various set_*_gate functions into the subarchs.

Indeed, we'll change that. Thanks.

christian

Quoted 13 lines

include/asm-i386/mach-xen/mach_idt.h line 48

Quoted 4 lines
include/asm-i386/mach-xen/mach_idt.h new file +4 -0
+static void __init set_task_gate(unsigned int n, unsigned int gdt_entry)
+{
+	/* _set_gate(n, 5, 0, 0, (gdt_entry<<3)); */
+}

Looks weird, but can be handled in the low level function.

-Andi

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.osdl.org/mailman/listinfo/virtualization

#75 lore raw

Re: [RFC PATCH 16/35] subarch support for interrupt and exception gates

In reply to Chris Wright

include/asm-i386/mach-xen/setup_arch_pre.h line 12

Quoted 11 lines · On Tue, 09 May 2006 00:00:16 -0700 Chris Wright <[email protected]> wrote:
include/asm-i386/mach-xen/setup_arch_pre.h +2 -0
@@ -5,6 +5,8 @@
 struct start_info *xen_start_info;
 EXPORT_SYMBOL(xen_start_info);
 
+struct trap_info xen_trap_table[257];
+
 /*
  * Point at the empty zero page to start with. We map the real shared_info
  * page as soon as fixmap is up and running.

Is there any particular reason why things-which-should-be-in-a-C-file are present in a .h file?

#76 lore raw

Re: [RFC PATCH 16/35] subarch support for interrupt and exception gates

Quoted 9 lines · * Andrew Morton ([email protected]) wrote:
Quoted 6 lines · On Tue, 09 May 2006 00:00:16 -0700 Chris Wright <[email protected]> wrote:
include/asm-i386/mach-xen/setup_arch_pre.h +0 -0
@@ -5,6 +5,8 @@
 struct start_info *xen_start_info;
 EXPORT_SYMBOL(xen_start_info);
 
Quoted 8 lines

include/asm-i386/mach-xen/setup_arch_pre.h line 12

Quoted 5 lines
include/asm-i386/mach-xen/setup_arch_pre.h +2 -0
+struct trap_info xen_trap_table[257];
+
 /*
  * Point at the empty zero page to start with. We map the real shared_info
  * page as soon as fixmap is up and running.

Is there any particular reason why things-which-should-be-in-a-C-file are present in a .h file?

It's following direction of current subarch interaction with setup.c (namely the setup_arch_post.h). It's definitely not so nice.

#77 lore raw

RFC PATCH 17/35 Segment register changes for Xen

In reply to Chris Wright

1. We clear FS/GS before changing TLS entries and switching LDT, as otherwise the hypervisor will fail to restore thread-local values on return to the guest kernel and we take a slow exception path.

2. We allow for the fact that the guest kernel may not run in ring 0. This requires some abstraction in a few places when setting %cs or checking privilege level (user vs kernel).

---
 arch/i386/kernel/process.c                   |    4 +++-
 arch/i386/mm/fault.c                         |    8 +++++---
 include/asm-i386/mach-default/mach_segment.h |    8 ++++++++
 include/asm-i386/mach-default/mach_system.h  |    2 ++
 include/asm-i386/mach-xen/mach_segment.h     |    9 +++++++++
 include/asm-i386/mach-xen/mach_system.h      |    2 ++
 include/asm-i386/mach-xen/setup_arch_post.h  |    2 ++
 include/asm-i386/ptrace.h                    |    6 ++++--
 include/asm-i386/segment.h                   |    2 ++
 9 files changed, 37 insertions(+), 6 deletions(-)
arch/i386/kernel/process.c +3 -1
@@ -347,7 +347,7 @@ int kernel_thread(int (*fn)(void *), voi
 	regs.xes = __USER_DS;
 	regs.orig_eax = -1;
 	regs.eip = (unsigned long) kernel_thread_helper;
-	regs.xcs = __KERNEL_CS;
+	regs.xcs = get_kernel_cs();
 	regs.eflags = X86_EFLAGS_IF | X86_EFLAGS_SF | X86_EFLAGS_PF | 0x2;
 
 	/* Ok, create the new process.. */
@@ -647,6 +647,8 @@ struct task_struct fastcall * __switch_t
 	 */
 	savesegment(fs, prev->fs);
 	savesegment(gs, prev->gs);
+	clearsegment(fs);
+	clearsegment(gs);
 
 	/*
 	 * Load the per-thread Thread-Local Storage descriptor.
arch/i386/mm/fault.c +5 -3
@@ -28,6 +28,8 @@
 #include <asm/desc.h>
 #include <asm/kdebug.h>
 
+#include <mach_segment.h>
+
 extern void die(const char *,struct pt_regs *,long);
 
 /*
@@ -78,14 +80,14 @@ static inline unsigned long get_segment_
 	u32 seg_ar, seg_limit, base, *desc;
 
 	/* The standard kernel/user address space limit. */
-	*eip_limit = (seg & 3) ? USER_DS.seg : KERNEL_DS.seg;
+	*eip_limit = (seg & USER_MODE_MASK) ? USER_DS.seg : KERNEL_DS.seg;
 
 	/* Unlikely, but must come before segment checks. */
 	if (unlikely((regs->eflags & VM_MASK) != 0))
 		return eip + (seg << 4);
 	
 	/* By far the most common cases. */
-	if (likely(seg == __USER_CS || seg == __KERNEL_CS))
+	if (likely(seg == __USER_CS || seg == get_kernel_cs()))
 		return eip;
 
 	/* Check the segment exists, is within the current LDT/GDT size,
@@ -400,7 +402,7 @@ good_area:
 	switch (error_code & 3) {
 		default:	/* 3: write, present */
 #ifdef TEST_VERIFY_AREA
-			if (regs->cs == KERNEL_CS)
+			if (regs->cs == get_kernel_cs())
 				printk("WP fault at %08lx\n", regs->eip);
 #endif
 			/* fall through */
include/asm-i386/mach-default/mach_system.h +2 -0
@@ -1,6 +1,8 @@
 #ifndef __ASM_MACH_SYSTEM_H
 #define __ASM_MACH_SYSTEM_H
 
+#define clearsegment(seg)
+
 /* interrupt control.. */
 #define local_save_flags(x)	do { typecheck(unsigned long,x); __asm__ __volatile__("pushfl ; popl %0":"=g" (x): /* no input */); } while (0)
 #define local_irq_restore(x) 	do { typecheck(unsigned long,x); __asm__ __volatile__("pushl %0 ; popfl": /* no output */ :"g" (x):"memory", "cc"); } while (0)
include/asm-i386/mach-xen/mach_system.h +2 -0
@@ -5,6 +5,8 @@
 
 #include <asm/hypervisor.h>
 
+#define clearsegment(seg) loadsegment(seg, 0)
+
 #ifdef CONFIG_SMP
 #define __vcpu_id smp_processor_id()
 #else
include/asm-i386/mach-xen/setup_arch_post.h +2 -0
@@ -26,6 +26,8 @@ static void __init machine_specific_arch
 		(struct shared_info *)__va(xen_start_info->shared_info);
 	memset(empty_zero_page, 0, sizeof(empty_zero_page));
 
+	HYPERVISOR_vm_assist(VMASST_CMD_enable, VMASST_TYPE_4gb_segments);
+
 	HYPERVISOR_set_callbacks(
 	    __KERNEL_CS, (unsigned long)hypervisor_callback,
 	    __KERNEL_CS, (unsigned long)failsafe_callback);
include/asm-i386/ptrace.h +4 -2
@@ -1,6 +1,8 @@
 #ifndef _I386_PTRACE_H
 #define _I386_PTRACE_H
 
+#include <mach_segment.h>
+
 #define EBX 0
 #define ECX 1
 #define EDX 2
@@ -73,11 +75,11 @@ extern void send_sigtrap(struct task_str
  */
 static inline int user_mode(struct pt_regs *regs)
 {
-	return (regs->xcs & 3) != 0;
+	return (regs->xcs & USER_MODE_MASK) != 0;
 }
 static inline int user_mode_vm(struct pt_regs *regs)
 {
-	return ((regs->xcs & 3) | (regs->eflags & VM_MASK)) != 0;
+	return ((regs->xcs & USER_MODE_MASK) | (regs->eflags & VM_MASK)) != 0;
 }
 #define instruction_pointer(regs) ((regs)->eip)
 #if defined(CONFIG_SMP) && defined(CONFIG_FRAME_POINTER)
include/asm-i386/segment.h +2 -0
@@ -1,6 +1,8 @@
 #ifndef _ASM_SEGMENT_H
 #define _ASM_SEGMENT_H
 
+#include <mach_segment.h>
+
 /*
  * The layout of the per-CPU GDT under Linux:
  *
include/asm-i386/mach-default/mach_segment.h new file +8 -0
@@ -0,0 +1,8 @@
+#ifndef __ASM_MACH_SEGMENT_H
+#define __ASM_MACH_SEGMENT_H
+
+#define USER_MODE_MASK 3
+
+#define get_kernel_cs() __KERNEL_CS
+
+#endif /* __ASM_MACH_SEGMENT_H */
include/asm-i386/mach-xen/mach_segment.h new file +9 -0
@@ -0,0 +1,9 @@
+#ifndef __ASM_MACH_SEGMENT_H
+#define __ASM_MACH_SEGMENT_H
+
+#define USER_MODE_MASK 2
+
+#define get_kernel_cs() \
+	(__KERNEL_CS + (xen_feature(XENFEAT_supervisor_mode_kernel) ? 0 : 1))
+
+#endif /* __ASM_MACH_SEGMENT_H */

--

#78 lore raw

Re: [RFC PATCH 17/35] Segment register changes for Xen

Hi!

include/asm-i386/mach-default/mach_system.h line 4

Quoted 7 lines
include/asm-i386/mach-default/mach_system.h +1 -0
@@ -1,6 +1,8 @@
 #ifndef __ASM_MACH_SYSTEM_H
 #define __ASM_MACH_SYSTEM_H
 
+#define clearsegment(seg)

do {} while (0), please.

Pavel

Signature · 1 line
Thanks for all the (sleeping) penguins.
#79 lore raw

Re: [RFC PATCH 17/35] Segment register changes for Xen

Quoted 11 lines · Pavel Machek

Hi!

include/asm-i386/mach-default/mach_system.h line 4

Quoted 7 lines
include/asm-i386/mach-default/mach_system.h +1 -0
@@ -1,6 +1,8 @@
 #ifndef __ASM_MACH_SYSTEM_H
 #define __ASM_MACH_SYSTEM_H
 
+#define clearsegment(seg)

do {} while (0), please.

It's not needed. Think about it.

-Andi

#80 lore raw

Re: [RFC PATCH 17/35] Segment register changes for Xen

Quoted 9 lines · Andi Kleen
Quoted 8 lines · Pavel Machek

Hi!

Quoted 6 lines
include/asm-i386/mach-default/mach_system.h +0 -0
@@ -1,6 +1,8 @@
 #ifndef __ASM_MACH_SYSTEM_H
 #define __ASM_MACH_SYSTEM_H
 
Quoted 5 lines
Quoted 3 lines

include/asm-i386/mach-default/mach_system.h line 4

Quoted 1 line
include/asm-i386/mach-default/mach_system.h +1 -0
+#define clearsegment(seg)

do {} while (0), please.

It's not needed. Think about it.

Really? If someone does

	if (something)
		clearsegment(seg)
	somethingelse();

... he'll get very confusing behaviour instead of compile error.

Okay, that's weaker argument than expected...

Also clearsegment(x) clearsegment(y); will compile when it should not.

Also clearsegment(i++) will behave strangely. So perhaps

#define clearsegment(seg) do { seg; } while (0)

is best variant?
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#81 lore raw

Re: [RFC PATCH 17/35] Segment register changes for Xen

Quoted 8 lines · Pavel Machek

Really? If someone does

	if (something)
		clearsegment(seg)
	somethingelse();

... he'll get very confusing behaviour instead of compile error.

Quoted 8 lines

Okay, that's weaker argument than expected...

Also clearsegment(x) clearsegment(y); will compile when it should not.

Also clearsegment(i++) will behave strangely. So perhaps

#define clearsegment(seg) do { seg; } while (0)

static inline void clearsegment(int seg) {}

?

Signature · 1 line
error compiling committee.c: too many arguments to function
#82 lore raw

Re: [RFC PATCH 17/35] Segment register changes for Xen

Quoted 11 lines · Avi Kivity
Quoted 10 lines · Pavel Machek

Really? If someone does

	if (something)
		clearsegment(seg)
	somethingelse();

... he'll get very confusing behaviour instead of compile error.

Okay, that's weaker argument than expected...

Quoted 8 lines
Quoted 6 lines

Also clearsegment(x) clearsegment(y); will compile when it should not.

Also clearsegment(i++) will behave strangely. So perhaps

#define clearsegment(seg) do { seg; } while (0)

static inline void clearsegment(int seg) {}

It's all mood because the complete function is wrongly named and probably should just go.

-Andi

#83 lore raw

Re: [Xen-devel] Re: [RFC PATCH 17/35] Segment register changes for Xen

In reply to Pavel Machek
Quoted 11 lines · Pavel Machek

Hi!

include/asm-i386/mach-default/mach_system.h line 4

Quoted 7 lines
include/asm-i386/mach-default/mach_system.h +1 -0
@@ -1,6 +1,8 @@
 #ifndef __ASM_MACH_SYSTEM_H
 #define __ASM_MACH_SYSTEM_H
 
+#define clearsegment(seg)

do {} while (0), please.

It's off-topic, but: why?

Rusty.

Signature · 1 line
 ccontrol: http://ccontrol.ozlabs.org
#84 lore raw

Re: [RFC PATCH 17/35] Segment register changes for Xen

In reply to Chris Wright

/* fall through */

include/asm-i386/mach-default/mach_system.h line 4

Quoted 7 lines
include/asm-i386/mach-default/mach_system.h +1 -0
@@ -1,6 +1,8 @@
 #ifndef __ASM_MACH_SYSTEM_H
 #define __ASM_MACH_SYSTEM_H
 
+#define clearsegment(seg)

I don't think you can give such a specific hook such a generic name.

I would just add an ifdef around the real user with a comment.

-Andi

#85 lore raw

Re: [RFC PATCH 17/35] Segment register changes for Xen

In reply to Chris Wright

arch/i386/kernel/process.c struct task_struct fastcall * __switch_t · line 652

Quoted 11 lines · Chris Wright

1. We clear FS/GS before changing TLS entries and switching LDT, as otherwise the hypervisor will fail to restore thread-local values on return to the guest kernel and we take a slow exception path.

arch/i386/kernel/process.c +2 -0
@@ -647,6 +647,8 @@ struct task_struct fastcall * __switch_t
 	 */
 	savesegment(fs, prev->fs);
 	savesegment(gs, prev->gs);
+	clearsegment(fs);
+	clearsegment(gs);
 

Really not needed. Think about it. You can even speed up Xen. I'm glad the native operation here is a nop, but it should be hypervisor_clearsegment or xen_clearsegment if you really want to keep it.

Zach

#86 lore raw

Re: [RFC PATCH 17/35] Segment register changes for Xen

Quoted 4 lines · Zachary Amsden
Quoted 3 lines

1. We clear FS/GS before changing TLS entries and switching LDT, as otherwise the hypervisor will fail to restore thread-local values on return to the guest kernel and we take a slow exception path.

Quoted 9 lines

arch/i386/kernel/process.c struct task_struct fastcall * __switch_t · line 652

Quoted 7 lines
arch/i386/kernel/process.c +2 -0
@@ -647,6 +647,8 @@ struct task_struct fastcall * __switch_t
 	 */
 	savesegment(fs, prev->fs);
 	savesegment(gs, prev->gs);
+	clearsegment(fs);
+	clearsegment(gs);
 

Really not needed. Think about it. You can even speed up Xen. I'm glad the native operation here is a nop, but it should be hypervisor_clearsegment or xen_clearsegment if you really want to keep it.

Maybe you could explain in more detail? It's not needed for correctness, but it is faster for us to clear at that point.

-- Keir

#87 lore raw

Re: [RFC PATCH 17/35] Segment register changes for Xen

In reply to Zachary Amsden
Quoted 5 lines · * Zachary Amsden ([email protected]) wrote:
Quoted 3 lines · Chris Wright

1. We clear FS/GS before changing TLS entries and switching LDT, as otherwise the hypervisor will fail to restore thread-local values on return to the guest kernel and we take a slow exception path.

Quoted 9 lines

arch/i386/kernel/process.c struct task_struct fastcall * __switch_t · line 652

Quoted 7 lines
arch/i386/kernel/process.c +2 -0
@@ -647,6 +647,8 @@ struct task_struct fastcall * __switch_t
 	 */
 	savesegment(fs, prev->fs);
 	savesegment(gs, prev->gs);
+	clearsegment(fs);
+	clearsegment(gs);
 

Really not needed. Think about it. You can even speed up Xen.

Please describe how. I'm afraid I'm missing your point, as I don't see the improvement.

Quoted 2 lines

I'm glad the native operation here is a nop, but it should be hypervisor_clearsegment or xen_clearsegment if you really want to keep it.

Yeah, Andi had similar naming concern.

#88 lore raw

RFC PATCH 18/35 Support gdt/idt/ldt handling on Xen.

In reply to Chris Wright

Move the macros which handle gdt/idt/ldt's into a subarch include file and add implementations for running on Xen.

---
 include/asm-i386/desc.h                   |   65 ++--------------------------
 include/asm-i386/mach-default/mach_desc.h |   67 +++++++++++++++++++++++++++++
 include/asm-i386/mach-xen/mach_desc.h     |   69 ++++++++++++++++++++++++++++++
 3 files changed, 141 insertions(+), 60 deletions(-)
include/asm-i386/desc.h +5 -60
@@ -33,18 +33,7 @@ static inline struct desc_struct *get_cp
 	return (struct desc_struct *)per_cpu(cpu_gdt_descr, cpu).address;
 }
 
-#define load_TR_desc() __asm__ __volatile__("ltr %w0"::"q" (GDT_ENTRY_TSS*8))
-#define load_LDT_desc() __asm__ __volatile__("lldt %w0"::"q" (GDT_ENTRY_LDT*8))
-
-#define load_gdt(dtr) __asm__ __volatile("lgdt %0"::"m" (*dtr))
-#define load_idt(dtr) __asm__ __volatile("lidt %0"::"m" (*dtr))
-#define load_tr(tr) __asm__ __volatile("ltr %0"::"mr" (tr))
-#define load_ldt(ldt) __asm__ __volatile("lldt %0"::"mr" (ldt))
-
-#define store_gdt(dtr) __asm__ ("sgdt %0":"=m" (*dtr))
-#define store_idt(dtr) __asm__ ("sidt %0":"=m" (*dtr))
-#define store_tr(tr) __asm__ ("str %0":"=mr" (tr))
-#define store_ldt(ldt) __asm__ ("sldt %0":"=mr" (ldt))
+#include <mach_desc.h>
 
 /*
  * This is the ldt that every process will get unless we need
@@ -53,30 +42,6 @@ static inline struct desc_struct *get_cp
 extern struct desc_struct default_ldt[];
 extern void set_intr_gate(unsigned int irq, void * addr);
 
-#define _set_tssldt_desc(n,addr,limit,type) \
-__asm__ __volatile__ ("movw %w3,0(%2)\n\t" \
-	"movw %w1,2(%2)\n\t" \
-	"rorl $16,%1\n\t" \
-	"movb %b1,4(%2)\n\t" \
-	"movb %4,5(%2)\n\t" \
-	"movb $0,6(%2)\n\t" \
-	"movb %h1,7(%2)\n\t" \
-	"rorl $16,%1" \
-	: "=m"(*(n)) : "q" (addr), "r"(n), "ir"(limit), "i"(type))
-
-static inline void __set_tss_desc(unsigned int cpu, unsigned int entry, void *addr)
-{
-	_set_tssldt_desc(&get_cpu_gdt_table(cpu)[entry], (int)addr,
-		offsetof(struct tss_struct, __cacheline_filler) - 1, 0x89);
-}
-
-#define set_tss_desc(cpu,addr) __set_tss_desc(cpu, GDT_ENTRY_TSS, addr)
-
-static inline void set_ldt_desc(unsigned int cpu, void *addr, unsigned int size)
-{
-	_set_tssldt_desc(&get_cpu_gdt_table(cpu)[GDT_ENTRY_LDT], (int)addr, ((size << 3)-1), 0x82);
-}
-
 #define LDT_entry_a(info) \
 	((((info)->base_addr & 0x0000ffff) << 16) | ((info)->limit & 0x0ffff))
 
@@ -102,30 +67,11 @@ static inline void set_ldt_desc(unsigned
 	(info)->seg_not_present	== 1	&& \
 	(info)->useable		== 0	)
 
-static inline void write_ldt_entry(void *ldt, int entry, __u32 entry_a, __u32 entry_b)
-{
-	__u32 *lp = (__u32 *)((char *)ldt + entry*8);
-	*lp = entry_a;
-	*(lp+1) = entry_b;
-}
-
-#if TLS_SIZE != 24
-# error update this code.
-#endif
-
-static inline void load_TLS(struct thread_struct *t, unsigned int cpu)
-{
-#define C(i) get_cpu_gdt_table(cpu)[GDT_ENTRY_TLS_MIN + i] = t->tls_array[i]
-	C(0); C(1); C(2);
-#undef C
-}
-
 static inline void clear_LDT(void)
 {
 	int cpu = get_cpu();
 
-	set_ldt_desc(cpu, &default_ldt[0], 5);
-	load_LDT_desc();
+	__set_ldt(cpu, DEFAULT_LDT, DEFAULT_LDT_SIZE);
 	put_cpu();
 }
 
@@ -138,12 +84,11 @@ static inline void load_LDT_nolock(mm_co
 	int count = pc->size;
 
 	if (likely(!count)) {
-		segments = &default_ldt[0];
-		count = 5;
+		segments = DEFAULT_LDT;
+		count = DEFAULT_LDT_SIZE;
 	}
 		
-	set_ldt_desc(cpu, segments, count);
-	load_LDT_desc();
+	__set_ldt(cpu, segments, count);
 }
 
 static inline void load_LDT(mm_context_t *pc)
include/asm-i386/mach-default/mach_desc.h new file +67 -0

67 lines not on this page · raw message

include/asm-i386/mach-xen/mach_desc.h new file +69 -0

69 lines not on this page · raw message

--

#89 lore raw

Re: [RFC PATCH 18/35] Support gdt/idt/ldt handling on Xen.

Hi!

include/asm-i386/mach-default/mach_desc.h line 65

Quoted 6 lines
include/asm-i386/mach-default/mach_desc.h new file +6 -0
+static inline void load_TLS(struct thread_struct *t, unsigned int cpu)
+{
+#define C(i) get_cpu_gdt_table(cpu)[GDT_ENTRY_TLS_MIN + i] = t->tls_array[i]
+	C(0); C(1); C(2);
+#undef C
+}

Why not use for loop here? gcc should be able to optimize it...

Quoted 1 line

+#define load_TR_desc()

do {} while (0)...

Quoted 1 line

+#define load_gdt(dtr) do { \

...

Quoted 1 line

+} while (0)

So you know the trick :-)

include/asm-i386/mach-xen/mach_desc.h line 27

Quoted 3 lines
include/asm-i386/mach-xen/mach_desc.h new file +3 -0
+#define load_idt(dtr) HYPERVISOR_set_trap_table(xen_trap_table)
+#define load_tr(tr) __asm__ __volatile("ltr %0"::"mr" (tr))
+#define load_ldt(ldt) __asm__ __volatile("lldt %0"::"mr" (ldt))

__volatile (not __volatile__?). could you just use 'asm volatile' without __s instead?

Pavel

Signature · 1 line
Thanks for all the (sleeping) penguins.
#90 lore raw

Re: [RFC PATCH 18/35] Support gdt/idt/ldt handling on Xen.

Quoted 10 lines · Pavel Machek

Hi!

include/asm-i386/mach-default/mach_desc.h line 65

Quoted 6 lines
include/asm-i386/mach-default/mach_desc.h new file +6 -0
+static inline void load_TLS(struct thread_struct *t, unsigned int cpu)
+{
+#define C(i) get_cpu_gdt_table(cpu)[GDT_ENTRY_TLS_MIN + i] = t->tls_array[i]
+	C(0); C(1); C(2);
+#undef C
+}

Why not use for loop here? gcc should be able to optimize it...

I don't think you can really blame the Xen people for that code - it's just the old code moved.

-Andi

#91 lore raw

Re: [RFC PATCH 18/35] Support gdt/idt/ldt handling on Xen.

In reply to Chris Wright

include/asm-i386/mach-xen/mach_desc.h line 67

Quoted 6 lines
include/asm-i386/mach-xen/mach_desc.h new file +6 -0
+static inline void load_TLS(struct thread_struct *t, unsigned int cpu)
+{
+#define C(i) HYPERVISOR_update_descriptor(virt_to_machine(&get_cpu_gdt_table(cpu)[GDT_ENTRY_TLS_MIN + i]), *(u64 *)&t->tls_array[i])
+	C(0); C(1); C(2);
+#undef C
+}

Please just expand this or make it a real function call (static inline), not a temporary macro ..

#92 lore raw

Re: [RFC PATCH 18/35] Support gdt/idt/ldt handling on Xen.

Quoted 10 lines · Martin J. Bligh

include/asm-i386/mach-default/mach_desc.h line 65

Quoted 7 lines
include/asm-i386/mach-default/mach_desc.h new file +3 -0
+static inline void load_TLS(struct thread_struct *t, unsigned int cpu)
+{
+#define C(i) 

HYPERVISOR_update_descriptor(virt_to_machine(&get_cpu_gdt_table(cpu)[GDT_ENTRY_TLS_MIN + i]), *(u64 *)&t->tls_array[i])

include/asm-i386/mach-default/mach_desc.h new file +3 -0
+	C(0); C(1); C(2);
+#undef C
+}

Please just expand this or make it a real function call (static inline), not a temporary macro ..

Yes, I've added an inline function to do a single descriptor.

Should I change the non-xen case as well? It was the inspiration for this code ;-)

christian

#93 lore raw

Re: [RFC PATCH 18/35] Support gdt/idt/ldt handling on Xen.

Quoted 14 lines · Christian Limpach
Quoted 10 lines · Martin J. Bligh

include/asm-i386/mach-default/mach_desc.h line 65

Quoted 7 lines
include/asm-i386/mach-default/mach_desc.h new file +3 -0
+static inline void load_TLS(struct thread_struct *t, unsigned int cpu)
+{
+#define C(i) 

HYPERVISOR_update_descriptor(virt_to_machine(&get_cpu_gdt_table(cpu)[GDT_ENTRY_TLS_MIN + i]), *(u64 *)&t->tls_array[i])

include/asm-i386/mach-default/mach_desc.h new file +3 -0
+	C(0); C(1); C(2);
+#undef C
+}

Please just expand this or make it a real function call (static inline), not a temporary macro ..

Quoted 4 lines

Yes, I've added an inline function to do a single descriptor.

Should I change the non-xen case as well? It was the inspiration for this code ;-)

If it looks anything like that, then I'd vote yes ;-)

M.

#94 lore raw

RFC PATCH 19/35 subarch support for control register accesses

In reply to Chris Wright

Abstract the code that accesses control register, and add a separate subarch implementation for Xen.

---
TODO:
- better sharing to avoid unneeded (i.e. read_cr4_safe, stts)

 include/asm-i386/mach-default/mach_system.h |   58 ++++++++++++++++++++++++++++
 include/asm-i386/mach-xen/mach_system.h     |   53 +++++++++++++++++++++++++
 include/asm-i386/system.h                   |   58 ----------------------------
 3 files changed, 111 insertions(+), 58 deletions(-)
include/asm-i386/mach-default/mach_system.h +58 -0
@@ -3,6 +3,64 @@
 
 #define clearsegment(seg)
 
+/*
+ * Clear and set 'TS' bit respectively
+ */
+#define clts() __asm__ __volatile__ ("clts")
+#define read_cr0() ({ \
+	unsigned int __dummy; \
+	__asm__ __volatile__( \
+		"movl %%cr0,%0\n\t" \
+		:"=r" (__dummy)); \
+	__dummy; \
+})
+#define write_cr0(x) \
+	__asm__ __volatile__("movl %0,%%cr0": :"r" (x));
+
+#define read_cr2() ({ \
+	unsigned int __dummy; \
+	__asm__ __volatile__( \
+		"movl %%cr2,%0\n\t" \
+		:"=r" (__dummy)); \
+	__dummy; \
+})
+#define write_cr2(x) \
+	__asm__ __volatile__("movl %0,%%cr2": :"r" (x));
+
+#define read_cr3() ({ \
+	unsigned int __dummy; \
+	__asm__ ( \
+		"movl %%cr3,%0\n\t" \
+		:"=r" (__dummy)); \
+	__dummy; \
+})
+#define write_cr3(x) \
+	__asm__ __volatile__("movl %0,%%cr3": :"r" (x));
+
+#define read_cr4() ({ \
+	unsigned int __dummy; \
+	__asm__( \
+		"movl %%cr4,%0\n\t" \
+		:"=r" (__dummy)); \
+	__dummy; \
+})
+
+#define read_cr4_safe() ({			      \
+	unsigned int __dummy;			      \
+	/* This could fault if %cr4 does not exist */ \
+	__asm__("1: movl %%cr4, %0		\n"   \
+		"2:				\n"   \
+		".section __ex_table,\"a\"	\n"   \
+		".long 1b,2b			\n"   \
+		".previous			\n"   \
+		: "=r" (__dummy): "0" (0));	      \
+	__dummy;				      \
+})
+
+#define write_cr4(x) \
+	__asm__ __volatile__("movl %0,%%cr4": :"r" (x));
+#define stts() write_cr0(8 | read_cr0())
+
 /* interrupt control.. */
 #define local_save_flags(x)	do { typecheck(unsigned long,x); __asm__ __volatile__("pushfl ; popl %0":"=g" (x): /* no input */); } while (0)
 #define local_irq_restore(x) 	do { typecheck(unsigned long,x); __asm__ __volatile__("pushl %0 ; popfl": /* no output */ :"g" (x):"memory", "cc"); } while (0)
include/asm-i386/mach-xen/mach_system.h +53 -0

59 lines not on this page · raw message

include/asm-i386/system.h +0 -58

64 lines not on this page · raw message

--

#95 lore raw

RFC PATCH 20/35 subarch stack pointer update

In reply to Chris Wright

Register the new kernel ('ring 0') stack pointer with the hypervisor during context switch.

---
 include/asm-i386/mach-default/mach_processor.h |    7 +++++++
 include/asm-i386/mach-xen/mach_processor.h     |    8 ++++++++
 include/asm-i386/processor.h                   |    1 +
 3 files changed, 16 insertions(+)
include/asm-i386/mach-default/mach_processor.h +7 -0
@@ -4,4 +4,11 @@
 #define CPUID cpuid
 #define CPUID_STR "cpuid"
 
+#ifndef __ASSEMBLY__
+static inline void mach_update_kernel_stack(unsigned long esp0,
+					    unsigned short ss0)
+{
+}
+#endif
+
 #endif /* __ASM_MACH_PROCESSOR_H */
include/asm-i386/mach-xen/mach_processor.h +8 -0
@@ -6,4 +6,12 @@
 #define CPUID XEN_CPUID
 #define CPUID_STR XEN_CPUID
 
+#ifndef __ASSEMBLY__
+static inline void mach_update_kernel_stack(unsigned long esp0,
+					    unsigned short ss0)
+{
+	HYPERVISOR_stack_switch(ss0, esp0);
+}
+#endif
+
 #endif /* __ASM_MACH_PROCESSOR_H */
include/asm-i386/processor.h +1 -0
@@ -500,6 +500,7 @@ static inline void load_esp0(struct tss_
 		tss->ss1 = thread->sysenter_cs;
 		wrmsr(MSR_IA32_SYSENTER_CS, thread->sysenter_cs, 0);
 	}
+	mach_update_kernel_stack(tss->esp0, tss->ss0);
 }
 
 #define start_thread(regs, new_eip, new_esp) do {		\

--

#96 lore raw

RFC PATCH 21/35 subarch TLB support

In reply to Chris Wright

Paravirtualize TLB flushes by using the flush interfaces provided by the hypervisor. These hide the details of cross-CPU shootdowns and allow significant optimisations (for example, by avoiding shooting down on virtual CPUs that are descheduled). This is considerably faster in most cases than performing virtual IPIs in the guest kernel.

---
 include/asm-i386/mach-default/mach_tlbflush.h |   59 ++++++++++++++++++++++++++
 include/asm-i386/mach-xen/mach_tlbflush.h     |   25 +++++++++++
 include/asm-i386/tlbflush.h                   |   55 ------------------------
 3 files changed, 85 insertions(+), 54 deletions(-)
include/asm-i386/tlbflush.h +1 -54
@@ -5,64 +5,11 @@
 #include <linux/mm.h>
 #include <asm/processor.h>
 
-#define __flush_tlb()							\
-	do {								\
-		unsigned int tmpreg;					\
-									\
-		__asm__ __volatile__(					\
-			"movl %%cr3, %0;              \n"		\
-			"movl %0, %%cr3;  # flush TLB \n"		\
-			: "=r" (tmpreg)					\
-			:: "memory");					\
-	} while (0)
-
-/*
- * Global pages have to be flushed a bit differently. Not a real
- * performance problem because this does not happen often.
- */
-#define __flush_tlb_global()						\
-	do {								\
-		unsigned int tmpreg, cr4, cr4_orig;			\
-									\
-		__asm__ __volatile__(					\
-			"movl %%cr4, %2;  # turn off PGE     \n"	\
-			"movl %2, %1;                        \n"	\
-			"andl %3, %1;                        \n"	\
-			"movl %1, %%cr4;                     \n"	\
-			"movl %%cr3, %0;                     \n"	\
-			"movl %0, %%cr3;  # flush TLB        \n"	\
-			"movl %2, %%cr4;  # turn PGE back on \n"	\
-			: "=&r" (tmpreg), "=&r" (cr4), "=&r" (cr4_orig)	\
-			: "i" (~X86_CR4_PGE)				\
-			: "memory");					\
-	} while (0)
-
 extern unsigned long pgkern_mask;
 
-# define __flush_tlb_all()						\
-	do {								\
-		if (cpu_has_pge)					\
-			__flush_tlb_global();				\
-		else							\
-			__flush_tlb();					\
-	} while (0)
-
 #define cpu_has_invlpg	(boot_cpu_data.x86 > 3)
 
-#define __flush_tlb_single(addr) \
-	__asm__ __volatile__("invlpg %0": :"m" (*(char *) addr))
-
-#ifdef CONFIG_X86_INVLPG
-# define __flush_tlb_one(addr) __flush_tlb_single(addr)
-#else
-# define __flush_tlb_one(addr)						\
-	do {								\
-		if (cpu_has_invlpg)					\
-			__flush_tlb_single(addr);			\
-		else							\
-			__flush_tlb();					\
-	} while (0)
-#endif
+#include <mach_tlbflush.h>
 
 /*
  * TLB flushing:
include/asm-i386/mach-default/mach_tlbflush.h new file +59 -0

59 lines not on this page · raw message

include/asm-i386/mach-xen/mach_tlbflush.h new file +25 -0
@@ -0,0 +1,25 @@
+#ifndef __ASM_MACH_TLBFLUSH_H
+#define __ASM_MACH_TLBFLUSH_H
+
+static inline void xen_tlb_flush(void)
+{
+        struct mmuext_op op;
+        op.cmd = MMUEXT_TLB_FLUSH_LOCAL;
+        BUG_ON(HYPERVISOR_mmuext_op(&op, 1, NULL, DOMID_SELF) < 0);
+}
+
+static inline void xen_invlpg(unsigned long ptr)
+{
+        struct mmuext_op op;
+        op.cmd = MMUEXT_INVLPG_LOCAL;
+        op.arg1.linear_addr = ptr & PAGE_MASK;
+        BUG_ON(HYPERVISOR_mmuext_op(&op, 1, NULL, DOMID_SELF) < 0);
+}
+
+#define __flush_tlb() xen_tlb_flush()
+#define __flush_tlb_global() xen_tlb_flush()
+#define __flush_tlb_all() xen_tlb_flush()
+#define __flush_tlb_single(addr) xen_invlpg(addr)
+#define __flush_tlb_one(addr) __flush_tlb_single(addr)
+
+#endif /* __ASM_MACH_TLBFLUSH_H */

--

#97 lore raw

RFC PATCH 22/35 subarch suport for idle loop (NO_IDLE_HZ for Xen)

In reply to Chris Wright

Paravirtualize the idle loop to explicitly trap to the hypervisor when blocking, and to use the NO_IDLE_HZ functionality introduced by s390 to inform the rcu subsystem that the CPU is quiescent.

---
 drivers/xen/Kconfig                         |    8 ++++++++
 include/asm-i386/mach-xen/setup_arch_post.h |   24 ++++++++++++++++++++++++
 2 files changed, 32 insertions(+)
drivers/xen/Kconfig +8 -0
@@ -12,6 +12,14 @@ config XEN
 
 if XEN
 
+config NO_IDLE_HZ
+	bool
+	default y
+	help
+	  Switches the regular HZ timer off when the system is going idle.
+	  This helps Xen to detect that the Linux system is idle, reducing
+	  the overhead of idle systems.
+
 config XEN_SHADOW_MODE
 	bool
 	default y
include/asm-i386/mach-xen/setup_arch_post.h +24 -0
@@ -8,6 +8,11 @@
 
 #include <xen/interface/physdev.h>
 
+extern void stop_hz_timer(void);
+extern void start_hz_timer(void);
+
+void xen_idle(void);
+
 static char * __init machine_specific_memory_setup(void)
 {
 	unsigned long max_pfn = xen_start_info->nr_pages;
@@ -65,4 +70,23 @@ static void __init machine_specific_arch
 		console_use_vt = 0;
 		conswitchp = NULL;
 	}
+
+	pm_idle = xen_idle;
+}
+
+void xen_idle(void)
+{
+	local_irq_disable();
+
+	if (need_resched())
+		local_irq_enable();
+	else {
+		clear_thread_flag(TIF_POLLING_NRFLAG);
+		smp_mb__after_clear_bit();
+		stop_hz_timer();
+		/* Blocking includes an implicit local_irq_enable(). */
+		HYPERVISOR_sched_op(SCHEDOP_block, 0);
+		start_hz_timer();
+		set_thread_flag(TIF_POLLING_NRFLAG);
+	}
 }

--

#98 lore raw

Re: [RFC PATCH 22/35] subarch suport for idle loop (NO_IDLE_HZ for Xen)

Quoted 10 lines
include/asm-i386/mach-xen/setup_arch_post.h +5 -0
+extern void stop_hz_timer(void);
+extern void start_hz_timer(void);
+
+void xen_idle(void);
+
 static char * __init machine_specific_memory_setup(void)
 {
 	unsigned long max_pfn = xen_start_info->nr_pages;

include/asm-i386/mach-xen/setup_arch_post.h static void __init machine_specific_arch · line 77

Quoted 8 lines
include/asm-i386/mach-xen/setup_arch_post.h +5 -0
@@ -65,4 +70,23 @@ static void __init machine_specific_arch
 		console_use_vt = 0;
 		conswitchp = NULL;
 	}
+
+	pm_idle = xen_idle;
+}
+
+void xen_idle(void)

I think that should be in some .c file, not a .h

Probably applies to more of your functions too.

-Andi

#99 lore raw

Re: [RFC PATCH 22/35] subarch suport for idle loop (NO_IDLE_HZ for Xen)

Quoted 21 lines · Andi Kleen
Quoted 10 lines
include/asm-i386/mach-xen/setup_arch_post.h +5 -0
+extern void stop_hz_timer(void);
+extern void start_hz_timer(void);
+
+void xen_idle(void);
+
 static char * __init machine_specific_memory_setup(void)
 {
 	unsigned long max_pfn = xen_start_info->nr_pages;

include/asm-i386/mach-xen/setup_arch_post.h static void __init machine_specific_arch · line 77

Quoted 8 lines
include/asm-i386/mach-xen/setup_arch_post.h +5 -0
@@ -65,4 +70,23 @@ static void __init machine_specific_arch
 		console_use_vt = 0;
 		conswitchp = NULL;
 	}
+
+	pm_idle = xen_idle;
+}
+
+void xen_idle(void)

I think that should be in some .c file, not a .h

Quoted 1 line

Probably applies to more of your functions too.

I guess I agree, although I was mostly following the examples in other mach setup_arch_*.h files.

christian

#100 lore raw

RFC PATCH 23/35 Increase x86 interrupt vector range

In reply to Chris Wright

Remove the limit of 256 interrupt vectors by changing the value stored in orig_{e,r}ax to be the negated interrupt vector. The orig_{e,r}ax needs to be < 0 to allow the signal code to distinguish between return from interrupt and return from syscall. With this change applied, NR_IRQS can be > 256.

Xen extends the IRQ numbering space to include room for dynamically allocated virtual interrupts (in the range 256-511), which requires a more permissive interface to do_IRQ.

---
 arch/i386/kernel/entry.S    |    4 ++--
 arch/i386/kernel/irq.c      |    4 ++--
 arch/x86_64/kernel/entry.S  |    2 +-
 arch/x86_64/kernel/irq.c    |    4 ++--
 arch/x86_64/kernel/smp.c    |    4 ++--
 include/asm-x86_64/hw_irq.h |    2 +-
 6 files changed, 10 insertions(+), 10 deletions(-)
arch/i386/kernel/entry.S +2 -2
@@ -464,7 +464,7 @@ vector=0
 ENTRY(irq_entries_start)
 .rept NR_IRQS
 	ALIGN
-1:	pushl $vector-256
+1:	pushl $~(vector)
 	jmp common_interrupt
 .data
 	.long 1b
@@ -481,7 +481,7 @@ common_interrupt:
 
 #define BUILD_INTERRUPT(name, nr)	\
 ENTRY(name)				\
-	pushl $nr-256;			\
+	pushl $~(nr);			\
 	SAVE_ALL			\
 	movl %esp,%eax;			\
 	call smp_/**/name;		\
arch/i386/kernel/irq.c +2 -2
@@ -53,8 +53,8 @@ static union irq_ctx *softirq_ctx[NR_CPU
  */
 fastcall unsigned int do_IRQ(struct pt_regs *regs)
 {	
-	/* high bits used in ret_from_ code */
-	int irq = regs->orig_eax & 0xff;
+	/* high bit used in ret_from_ code */
+	int irq = ~regs->orig_eax;
 #ifdef CONFIG_4KSTACKS
 	union irq_ctx *curctx, *irqctx;
 	u32 *isp;
arch/x86_64/kernel/entry.S +1 -1
@@ -601,7 +601,7 @@ retint_kernel:	
  */		
 	.macro apicinterrupt num,func
 	INTR_FRAME
-	pushq $\num-256
+	pushq $~(\num)
 	CFI_ADJUST_CFA_OFFSET 8
 	interrupt \func
 	jmp ret_from_intr
arch/x86_64/kernel/irq.c +2 -2
@@ -91,8 +91,8 @@ skip:
  */
 asmlinkage unsigned int do_IRQ(struct pt_regs *regs)
 {	
-	/* high bits used in ret_from_ code  */
-	unsigned irq = regs->orig_rax & 0xff;
+	/* high bit used in ret_from_ code  */
+	unsigned irq = ~regs->orig_rax;
 
 	exit_idle();
 	irq_enter();
arch/x86_64/kernel/smp.c +2 -2
@@ -135,10 +135,10 @@ asmlinkage void smp_invalidate_interrupt
 
 	cpu = smp_processor_id();
 	/*
-	 * orig_rax contains the interrupt vector - 256.
+	 * orig_rax contains the negated interrupt vector.
 	 * Use that to determine where the sender put the data.
 	 */
-	sender = regs->orig_rax + 256 - INVALIDATE_TLB_VECTOR_START;
+	sender = ~regs->orig_rax - INVALIDATE_TLB_VECTOR_START;
 	f = &per_cpu(flush_state, sender);
 
 	if (!cpu_isset(cpu, f->flush_cpumask))
include/asm-x86_64/hw_irq.h +1 -1
@@ -127,7 +127,7 @@ asmlinkage void IRQ_NAME(nr); \
 __asm__( \
 "\n.p2align\n" \
 "IRQ" #nr "_interrupt:\n\t" \
-	"push $" #nr "-256 ; " \
+	"push $~(" #nr ") ; " \
 	"jmp common_interrupt");
 
 #if defined(CONFIG_X86_IO_APIC)

--

#101 lore raw

RFC PATCH 24/35 Add support for Xen event channels.

In reply to Chris Wright

Support Xen event channels instead of the i8259 PIC.

Event channels are used to inject events into the kernel, either from the hypervisor or from another VM. The injected events are mapped to interrupts.

If an event needs to be injected, the hypervisor causes an upcall into the kernel. The upcall handler then scans the event pending bitmap and calls do_IRQ for each pending event.

---
 arch/i386/kernel/Makefile               |    6 
 drivers/xen/core/evtchn.c               |  887 ++++++++++++++++++++++++++++++++
 include/asm-i386/hw_irq.h               |    4 
 include/asm-i386/mach-xen/irq_vectors.h |  109 +++
 include/xen/evtchn.h                    |  116 ++++
 5 files changed, 1120 insertions(+), 2 deletions(-)
arch/i386/kernel/Makefile +5 -1
@@ -5,7 +5,7 @@
 extra-y := head.o init_task.o vmlinux.lds
 
 obj-y	:= process.o semaphore.o signal.o entry.o traps.o irq.o \
-		ptrace.o time.o ioport.o ldt.o setup.o i8259.o sys_i386.o \
+		ptrace.o time.o ioport.o ldt.o setup.o hw_irq.o sys_i386.o \
 		pci-dma.o i386_ksyms.o i387.o bootflag.o \
 		quirks.o i8237.o topology.o alternative.o
 
@@ -42,6 +42,10 @@ EXTRA_AFLAGS   := -traditional
 
 obj-$(CONFIG_SCx200)		+= scx200.o
 
+hw_irq-y			:= i8259.o
+
+hw_irq-$(CONFIG_XEN)		:= ../../../drivers/xen/core/evtchn.o
+
 # vsyscall.o contains the vsyscall DSO images as __initdata.
 # We must build both images before we can assemble it.
 # Note: kbuild does not track this dependency due to usage of .incbin
include/asm-i386/hw_irq.h +3 -1
@@ -68,7 +68,9 @@ extern atomic_t irq_mis_count;
 
 #define IO_APIC_IRQ(x) (((x) >= 16) || ((1<<(x)) & io_apic_irqs))
 
-#if defined(CONFIG_X86_IO_APIC)
+#if defined(CONFIG_X86_XEN)
+extern void hw_resend_irq(struct hw_interrupt_type *h, unsigned int i);
+#elif defined(CONFIG_X86_IO_APIC)
 static inline void hw_resend_irq(struct hw_interrupt_type *h, unsigned int i)
 {
 	if (IO_APIC_IRQ(i))
drivers/xen/core/evtchn.c new file +887 -0

887 lines not on this page · raw message

include/asm-i386/mach-xen/irq_vectors.h new file +109 -0

109 lines not on this page · raw message

include/xen/evtchn.h new file +116 -0

116 lines not on this page · raw message

--

#102 lore raw

Re: [RFC PATCH 24/35] Add support for Xen event channels.

Hi!

Review

Quoted 2 lines

+++ linus-2.6/drivers/xen/core/evtchn.c

(quoted diff) +0 -0
@@ -0,0 +1,887 @@

msssng wwls?

drivers/xen/core/evtchn.c line 200

Quoted 5 lines
drivers/xen/core/evtchn.c new file +5 -0
+/* NB. Interrupts are disabled on entry. */
+asmlinkage void evtchn_do_upcall(struct pt_regs *regs)
+{
+	unsigned long  l1, l2;
+	unsigned int   l1i, l2i, port;

Better variable names would be nice.

Pavel

Signature · 1 line
Thanks for all the (sleeping) penguins.
#103 lore raw

Re: [RFC PATCH 24/35] Add support for Xen event channels.

In reply to Chris Wright

drivers/xen/core/evtchn.c line 839

Quoted 8 lines · On Tue, 09 May 2006 00:00:24 -0700 Chris Wright <[email protected]> wrote:
drivers/xen/core/evtchn.c new file +8 -0
+void __init init_IRQ(void)
+{
+	int i;
+	int cpu;
+
+	irq_ctx_init(0);
+
+	spin_lock_init(&irq_mapping_update_lock);

May as well initialise this at compile time.

drivers/xen/core/evtchn.c line 844

Quoted 4 lines
drivers/xen/core/evtchn.c new file +4 -0
+	init_evtchn_cpu_bindings();
+
+	/* No VIRQ or IPI bindings. */
+	for (cpu = 0; cpu < NR_CPUS; cpu++) {

Using NR_CPUS is a little... old-fashioned. I'd suggest a sweep through all the Xen code, look for places where it should be using for_each_foo_cpu().

#104 lore raw

Re: [RFC PATCH 24/35] Add support for Xen event channels.

Quoted 9 lines · Andrew Morton

drivers/xen/core/evtchn.c line 844

Quoted 5 lines
drivers/xen/core/evtchn.c new file +4 -0
+	init_evtchn_cpu_bindings();
+
+	/* No VIRQ or IPI bindings. */
+	for (cpu = 0; cpu < NR_CPUS; cpu++) {

Using NR_CPUS is a little... old-fashioned. I'd suggest a sweep through all the Xen code, look for places where it should be using for_each_foo_cpu().

Actually that's a particularly good catch in this case, since we use per_cpu() inside the loop and that's only well defined for cpu_possible_map. Oops.

The elusive users of ring.h are our split device drivers. It hides a bunch of details about muxing requests and responses on the same ring, and notification thresholds. There are a few other places we have ring buffers but they are sufficiently simple that implementing in place is clearer.

Thanks,
Keir

#105 lore raw

RFC PATCH 25/35 Add Xen time abstractions

In reply to Chris Wright

Add support for Xen time abstractions. To avoid expensive traps into the hypervisor, the passage of time is extrapolated from the local TSC and a set of timestamps and scaling factors exported to the guest via shared memory. Xen also provides a periodic interrupt facility which is used to drive updates of xtime and jiffies, and perform the usual process accounting and profiling.

---
 arch/i386/kernel/Makefile |    6 
 drivers/xen/core/time.c   | 1045 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 1050 insertions(+), 1 deletion(-)
arch/i386/kernel/Makefile +5 -1
@@ -9,8 +9,11 @@ obj-y	:= process.o semaphore.o signal.o 
 		pci-dma.o i386_ksyms.o i387.o bootflag.o \
 		quirks.o i8237.o topology.o alternative.o
 
+timers-y			:= timers/
+timers-$(CONFIG_XEN)		:=
+
 obj-y				+= cpu/
-obj-y				+= timers/
+obj-y				+= $(timers-y)
 obj-y				+= acpi/
 obj-$(CONFIG_X86_BIOS_REBOOT)	+= reboot.o
 obj-$(CONFIG_MCA)		+= mca.o
@@ -45,6 +48,7 @@ obj-$(CONFIG_SCx200)		+= scx200.o
 hw_irq-y			:= i8259.o
 
 hw_irq-$(CONFIG_XEN)		:= ../../../drivers/xen/core/evtchn.o
+time-$(CONFIG_XEN)		:= ../../../drivers/xen/core/time.o
 
 # vsyscall.o contains the vsyscall DSO images as __initdata.
 # We must build both images before we can assemble it.
drivers/xen/core/time.c new file +1045 -0

1,045 lines not on this page · raw message

--

#106 lore raw

Re: [RFC PATCH 25/35] Add Xen time abstractions

arch/i386/kernel/Makefile obj-y := process.o semaphore.o signal.o · line 13

Quoted 2 lines · Chris Wright
arch/i386/kernel/Makefile +2 -0
+timers-y			:= timers/
+timers-$(CONFIG_XEN)		:=

Is this line suppose to be empty ?

Daniel

#107 lore raw

Re: [RFC PATCH 25/35] Add Xen time abstractions

Quoted 7 lines · Daniel Walker

arch/i386/kernel/Makefile obj-y := process.o semaphore.o signal.o · line 13

Quoted 2 lines · Chris Wright
arch/i386/kernel/Makefile +2 -0
+timers-y			:= timers/
+timers-$(CONFIG_XEN)		:=

Is this line suppose to be empty ?

Yes. We have our own version of time.c which doesn't use any of the timer code in timers but works for both i386 and x86_64 instead.

christian

#108 lore raw

Re: [RFC PATCH 25/35] Add Xen time abstractions

Quoted 11 lines · Christian Limpach
Quoted 7 lines · Daniel Walker

arch/i386/kernel/Makefile obj-y := process.o semaphore.o signal.o · line 13

Quoted 2 lines · Chris Wright
arch/i386/kernel/Makefile +2 -0
+timers-y			:= timers/
+timers-$(CONFIG_XEN)		:=

Is this line suppose to be empty ?

Yes. We have our own version of time.c which doesn't use any of the timer code in timers but works for both i386 and x86_64 instead.

Please add a comment, at first sight it's not even obvious that this line does anything (+= and := are easy to confuse).

Quoted 1 line

christian

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#109 lore raw

Re: [RFC PATCH 25/35] Add Xen time abstractions

In reply to Chris Wright
Quoted 6 lines · Chris Wright

Add support for Xen time abstractions. To avoid expensive traps into the hypervisor, the passage of time is extrapolated from the local TSC and a set of timestamps and scaling factors exported to the guest via shared memory. Xen also provides a periodic interrupt facility which is used to drive updates of xtime and jiffies, and perform the usual process accounting and profiling.

There is far too much code duplication in there. I think you need to refactor the main time.c a bit first and strip that down.

Also you can drop all the __x86_64__ support for now.

-Andi

Quoted 1 line
#110 lore raw

Re: [RFC PATCH 25/35] Add Xen time abstractions

Hi Andi,

Quoted 8 lines · Andi Kleen
Quoted 6 lines · Chris Wright

Add support for Xen time abstractions. To avoid expensive traps into the hypervisor, the passage of time is extrapolated from the local TSC and a set of timestamps and scaling factors exported to the guest via shared memory. Xen also provides a periodic interrupt facility which is used to drive updates of xtime and jiffies, and perform the usual process accounting and profiling.

Quoted 4 lines

There is far too much code duplication in there. I think you need to refactor the main time.c a bit first and strip that down.

Also you can drop all the __x86_64__ support for now.

Isn't time and timer handling a moving target anyway?
The refactoring will be done by the timer people in a completly different manner anyway.

Are you sure, you want to disturb these efforts by requiring another refactoring here?

Regards

Ingo Oeser

#111 lore raw

Re: [RFC PATCH 25/35] Add Xen time abstractions

Quoted 7 lines

Isn't time and timer handling a moving target anyway?
The refactoring will be done by the timer people in a completly different manner anyway.

Are you sure, you want to disturb these efforts by requiring another refactoring here?

Yes I am.

-Andi

#112 lore raw

Re: [RFC PATCH 25/35] Add Xen time abstractions

In reply to Ingo Oeser
Quoted 12 lines · * Ingo Oeser ([email protected]) wrote:
Quoted 11 lines · Andi Kleen
Quoted 6 lines · Chris Wright

Add support for Xen time abstractions. To avoid expensive traps into the hypervisor, the passage of time is extrapolated from the local TSC and a set of timestamps and scaling factors exported to the guest via shared memory. Xen also provides a periodic interrupt facility which is used to drive updates of xtime and jiffies, and perform the usual process accounting and profiling.

There is far too much code duplication in there. I think you need to refactor the main time.c a bit first and strip that down.

Quoted 8 lines
Quoted 1 line

Also you can drop all the __x86_64__ support for now.

Isn't time and timer handling a moving target anyway?
The refactoring will be done by the timer people in a completly different manner anyway.

Are you sure, you want to disturb these efforts by requiring another refactoring here?

Yes. Otherwise we end up with either duplicated code if the moving target winds up not moving, or outdated code if it does. I agree with Andi. It's on the todo list to refactor, but I wanted to get the patches out even though it's a work in progress.

thanks,
-chris

#113 lore raw

Re: [RFC PATCH 25/35] Add Xen time abstractions

In reply to Chris Wright

Hi!

drivers/xen/core/time.c line 4

Quoted 7 lines
drivers/xen/core/time.c new file +4 -0
@@ -0,0 +1,1045 @@
+/*
+ *  time.c
+ *
+ *  Copyright (C) 1991, 1992, 1995  Linus Torvalds

Really?

drivers/xen/core/time.c line 235

Quoted 5 lines
drivers/xen/core/time.c new file +5 -0
+void init_cpu_khz(void)
+{
+	u64 __cpu_khz = 1000000ULL << 32;
+	struct vcpu_time_info *info;
+	info = &HYPERVISOR_shared_info->vcpu_info[0].time;

No, I do not think linus wrote that. You probably want to add your copyright there, and remove obsolete changelog.

Signature · 1 line
Thanks for all the (sleeping) penguins.
#114 lore raw

RFC PATCH 26/35 Add Xen subarch reboot support

In reply to Chris Wright

Add remote reboot capability, so that a virtual machine can be rebooted, halted or 'powered off' by external management tools.

---
TODO:
- move poweroff and halt to generic similar to c_a_d

 arch/i386/kernel/Makefile |    1

drivers/xen/core/reboot.c | 232 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 233 insertions(+)

 arch/i386/kernel/Makefile |    1 
 drivers/xen/core/reboot.c |  232 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 233 insertions(+)
arch/i386/kernel/Makefile +1 -0
@@ -48,6 +48,7 @@ obj-$(CONFIG_SCx200)		+= scx200.o
 hw_irq-y			:= i8259.o
 
 hw_irq-$(CONFIG_XEN)		:= ../../../drivers/xen/core/evtchn.o
+reboot-$(CONFIG_XEN)		:= ../../../drivers/xen/core/reboot.o
 time-$(CONFIG_XEN)		:= ../../../drivers/xen/core/time.o
 
 # vsyscall.o contains the vsyscall DSO images as __initdata.
drivers/xen/core/reboot.c new file +232 -0

232 lines not on this page · raw message

--

#115 lore raw

Re: [RFC PATCH 26/35] Add Xen subarch reboot support

Quoted 5 lines

+++ linus-2.6/drivers/xen/core/reboot.c

drivers/xen/core/reboot.c new file +2 -0
@@ -0,0 +1,232 @@
+#define __KERNEL_SYSCALLS__
+#include <linux/version.h>

drivers/xen/core/reboot.c line 12

Quoted 10 lines
drivers/xen/core/reboot.c new file +10 -0
+#include <linux/kernel.h>
+#include <linux/mm.h>
+#include <linux/unistd.h>
+#include <linux/module.h>
+#include <linux/reboot.h>
+#include <linux/sysrq.h>
+#include <linux/stringify.h>
+#include <linux/syscalls.h>
+#include <linux/cpu.h>
+#include <linux/kthread.h>

Do you really need all these includes?

drivers/xen/core/reboot.c line 22

Quoted 1 line
drivers/xen/core/reboot.c new file +1 -0
+#if defined(__i386__) || defined(__x86_64__)

aka CONFIG_X86

drivers/xen/core/reboot.c line 30

Quoted 8 lines
drivers/xen/core/reboot.c new file +8 -0
+/*
+ * Power off function, if any
+ */
+void (*pm_power_off)(void);
+EXPORT_SYMBOL(pm_power_off);
+#endif
+
+extern void ctrl_alt_del(void);

That should be in some header

drivers/xen/core/reboot.c line 85

Quoted 11 lines
drivers/xen/core/reboot.c new file +11 -0
+
+/* Ignore multiple shutdown requests. */
+static int shutting_down = SHUTDOWN_INVALID;
+static void __shutdown_handler(void *unused);
+static DECLARE_WORK(shutdown_work, __shutdown_handler, NULL);
+
+static int shutdown_process(void *__unused)
+{
+	static char *envp[] = { "HOME=/", "TERM=linux",
+				"PATH=/sbin:/usr/sbin:/bin:/usr/bin", NULL };
+	static char *poweroff_argv[] = { "/sbin/poweroff", NULL };

This should be configurable, probably in a sysctl

Quoted 11 lines
drivers/xen/core/reboot.c new file +11 -0
+
+	if ((shutting_down == SHUTDOWN_POWEROFF) ||
+	    (shutting_down == SHUTDOWN_HALT)) {
+		if (execve(poweroff_argv[0], poweroff_argv, envp) < 0) {
+			sys_reboot(LINUX_REBOOT_MAGIC1,
+				   LINUX_REBOOT_MAGIC2,
+				   LINUX_REBOOT_CMD_POWER_OFF,
+				   NULL);
+		}
+	}
+

drivers/xen/core/reboot.c line 106

Quoted 10 lines
drivers/xen/core/reboot.c new file +10 -0
+	shutting_down = SHUTDOWN_INVALID; /* could try again */
+
+	return 0;
+}
+
+static void __shutdown_handler(void *unused)
+{
+	int err = 0;
+
+	if (shutting_down != SHUTDOWN_SUSPEND)

The whole shutting_down handling looks racy. Probably needs some locking?

-Andi

#116 lore raw

Re: [RFC PATCH 26/35] Add Xen subarch reboot support

Hi!

Quoted 5 lines
Quoted 5 lines
drivers/xen/core/reboot.c new file +5 -0
+
+/* Ignore multiple shutdown requests. */
+static int shutting_down = SHUTDOWN_INVALID;
+static void __shutdown_handler(void *unused);
+static DECLARE_WORK(shutdown_work, __shutdown_handler, NULL);
Quoted 8 lines

drivers/xen/core/reboot.c line 85

Quoted 6 lines
drivers/xen/core/reboot.c new file +6 -0
+
+static int shutdown_process(void *__unused)
+{
+	static char *envp[] = { "HOME=/", "TERM=linux",
+				"PATH=/sbin:/usr/sbin:/bin:/usr/bin", NULL };
+	static char *poweroff_argv[] = { "/sbin/poweroff", NULL };

This should be configurable, probably in a sysctl

Actually we have similar code in sparc and acpi parts, IIRC. We probably want to have one, common, shut-me-off routine.

Signature · 1 line
Thanks for all the (sleeping) penguins.
#117 lore raw

Re: [RFC PATCH 26/35] Add Xen subarch reboot support

Quoted 16 lines · * Pavel Machek ([email protected]) wrote:

Hi!

Quoted 5 lines
Quoted 5 lines
drivers/xen/core/reboot.c new file +5 -0
+
+/* Ignore multiple shutdown requests. */
+static int shutting_down = SHUTDOWN_INVALID;
+static void __shutdown_handler(void *unused);
+static DECLARE_WORK(shutdown_work, __shutdown_handler, NULL);
Quoted 8 lines

drivers/xen/core/reboot.c line 85

Quoted 6 lines
drivers/xen/core/reboot.c new file +6 -0
+
+static int shutdown_process(void *__unused)
+{
+	static char *envp[] = { "HOME=/", "TERM=linux",
+				"PATH=/sbin:/usr/sbin:/bin:/usr/bin", NULL };
+	static char *poweroff_argv[] = { "/sbin/poweroff", NULL };

This should be configurable, probably in a sysctl

Quoted 2 lines

Actually we have similar code in sparc and acpi parts, IIRC. We probably want to have one, common, shut-me-off routine.

Yep, I had that cleanup in mind, the patch said:

TODO:
- move poweroff and halt to generic similar to c_a_d

thanks,
-chris

#118 lore raw

RFC PATCH 27/35 Add nosegneg capability to the vsyscall page notes

In reply to Chris Wright

Add the "nosegneg" fake capabilty to the vsyscall page notes. This is used by the runtime linker to select a glibc version which then disables negative-offset accesses to the thread-local segment via %gs. These accesses require emulation in Xen (because segments are truncated to protect the hypervisor address space) and avoiding them provides a measurable performance boost.

---
 arch/i386/kernel/vsyscall-note.S |   29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)
arch/i386/kernel/vsyscall-note.S +29 -0
@@ -3,6 +3,7 @@
  * Here we can supply some information useful to userland.
  */
 
+#include <linux/config.h>
 #include <linux/uts.h>
 #include <linux/version.h>
 
@@ -23,3 +24,31 @@
 	ASM_ELF_NOTE_BEGIN(".note.kernel-version", "a", UTS_SYSNAME, 0)
 	.long LINUX_VERSION_CODE
 	ASM_ELF_NOTE_END
+
+#ifdef CONFIG_XEN
+/*
+ * Add a special note telling glibc's dynamic linker a fake hardware
+ * flavor that it will use to choose the search path for libraries in the
+ * same way it uses real hardware capabilities like "mmx".
+ * We supply "nosegneg" as the fake capability, to indicate that we
+ * do not like negative offsets in instructions using segment overrides,
+ * since we implement those inefficiently.  This makes it possible to
+ * install libraries optimized to avoid those access patterns in someplace
+ * like /lib/i686/tls/nosegneg.  Note that an /etc/ld.so.conf.d/file
+ * corresponding to the bits here is needed to make ldconfig work right.
+ * It should contain:
+ *	hwcap 0 nosegneg
+ * to match the mapping of bit to name that we give here.
+ */
+#define NOTE_KERNELCAP_BEGIN(ncaps, mask) \
+	ASM_ELF_NOTE_BEGIN(".note.kernelcap", "a", "GNU", 2) \
+	.long ncaps, mask
+#define NOTE_KERNELCAP(bit, name) \
+	.byte bit; .asciz name
+#define NOTE_KERNELCAP_END ASM_ELF_NOTE_END
+
+NOTE_KERNELCAP_BEGIN(1, 1)
+NOTE_KERNELCAP(1, "nosegneg")
+NOTE_KERNELCAP_END
+#endif
+

--

#119 lore raw

RFC PATCH 28/35 add support for Xen feature queries

In reply to Chris Wright

Add support for parsing and interpreting hypervisor feature flags. These allow the kernel to determine what features are provided by the underlying hypervisor. For example, whether page tables need to be write protected explicitly by the kernel, and whether the kernel (appears to) run in ring 0 rather than ring 1. This information allows the kernel to improve performance by avoiding unnecessary actions.

---
 drivers/Makefile                            |    1 
 drivers/xen/Makefile                        |    3 ++
 drivers/xen/core/Makefile                   |    3 ++
 drivers/xen/core/features.c                 |   29 ++++++++++++++++++++++++++++
 include/asm-i386/mach-xen/setup_arch_post.h |    2 +
 include/xen/features.h                      |   20 +++++++++++++++++++
 6 files changed, 58 insertions(+)
include/asm-i386/mach-xen/setup_arch_post.h +2 -0
@@ -27,6 +27,8 @@ static void __init machine_specific_arch
 {
 	struct physdev_op op;
 
+	setup_xen_features();
+
 	HYPERVISOR_shared_info =
 		(struct shared_info *)__va(xen_start_info->shared_info);
 	memset(empty_zero_page, 0, sizeof(empty_zero_page));
drivers/Makefile +1 -0
@@ -31,6 +31,7 @@ obj-y				+= base/ block/ misc/ mfd/ net/
 obj-$(CONFIG_NUBUS)		+= nubus/
 obj-$(CONFIG_ATM)		+= atm/
 obj-$(CONFIG_PPC_PMAC)		+= macintosh/
+obj-$(CONFIG_XEN)		+= xen/
 obj-$(CONFIG_IDE)		+= ide/
 obj-$(CONFIG_FC4)		+= fc4/
 obj-$(CONFIG_SCSI)		+= scsi/
include/xen/features.h new file +20 -0
@@ -0,0 +1,20 @@
+/******************************************************************************
+ * features.h
+ *
+ * Query the features reported by Xen.
+ *
+ * Copyright (c) 2006, Ian Campbell
+ */
+
+#ifndef __ASM_XEN_FEATURES_H__
+#define __ASM_XEN_FEATURES_H__
+
+#include <xen/interface/version.h>
+
+extern void setup_xen_features(void);
+
+extern u8 xen_features[XENFEAT_NR_SUBMAPS * 32];
+
+#define xen_feature(flag)	(xen_features[flag])
+
+#endif /* __ASM_XEN_FEATURES_H__ */
drivers/xen/Makefile new file +3 -0
@@ -0,0 +1,3 @@
+
+obj-y	+= core/
+
drivers/xen/core/Makefile new file +3 -0
@@ -0,0 +1,3 @@
+
+obj-y	:= features.o
+
drivers/xen/core/features.c new file +29 -0
@@ -0,0 +1,29 @@
+/******************************************************************************
+ * features.c
+ *
+ * Xen feature flags.
+ *
+ * Copyright (c) 2006, Ian Campbell, XenSource Inc.
+ */
+#include <linux/types.h>
+#include <linux/cache.h>
+#include <linux/module.h>
+#include <asm/hypervisor.h>
+#include <xen/features.h>
+
+u8 xen_features[XENFEAT_NR_SUBMAPS * 32] __read_mostly;
+EXPORT_SYMBOL_GPL(xen_features);
+
+void setup_xen_features(void)
+{
+	struct xen_feature_info fi;
+	int i, j;
+
+	for (i = 0; i < XENFEAT_NR_SUBMAPS; i++) {
+		fi.submap_idx = i;
+		if (HYPERVISOR_xen_version(XENVER_get_features, &fi) < 0)
+			break;
+		for (j=0; j<32; j++)
+			xen_features[i*32+j] = !!(fi.submap & 1<<j);
+	}
+}

--

#120 lore raw

Re: [RFC PATCH 28/35] add support for Xen feature queries

Hi!

Quoted 17 lines

Add support for parsing and interpreting hypervisor feature flags. These allow the kernel to determine what features are provided by the underlying hypervisor. For example, whether page tables need to be write protected explicitly by the kernel, and whether the kernel (appears to) run in ring 0 rather than ring 1. This information allows the kernel to improve performance by avoiding unnecessary actions.

include/xen/features.h new file +6 -0
@@ -0,0 +1,20 @@
+/******************************************************************************
+ * features.h
+ *
+ * Query the features reported by Xen.
+ *
+ * Copyright (c) 2006, Ian Campbell

include/xen/features.h line 16

Quoted 10 lines
include/xen/features.h new file +10 -0
+ */
+
+#ifndef __ASM_XEN_FEATURES_H__
+#define __ASM_XEN_FEATURES_H__
+
+#include <xen/interface/version.h>
+
+extern void setup_xen_features(void);
+
+extern u8 xen_features[XENFEAT_NR_SUBMAPS * 32];

32 bytes per submap? Why not use __test_bit & friends and make the bitmap compact?

include/xen/features.h line 18

Quoted 1 line
include/xen/features.h new file +1 -0
+#define xen_feature(flag)	(xen_features[flag])

Perhaps this kind of indirection is not neccessary?

drivers/xen/core/features.c line 6

Quoted 9 lines
drivers/xen/core/features.c new file +6 -0
@@ -0,0 +1,29 @@
+/******************************************************************************
+ * features.c
+ *
+ * Xen feature flags.
+ *
+ * Copyright (c) 2006, Ian Campbell, XenSource Inc.

GPL?
Pavel

Signature · 1 line
Thanks for all the (sleeping) penguins.
#121 lore raw

RFC PATCH 29/35 Add the Xen virtual console driver.

In reply to Chris Wright

This provides a bootstrap and ongoing emergency console which is intended to be available from very early during boot and at all times thereafter, in contrast with alternatives such as UDP-based syslogd, or logging in via ssh. The protocol is based on a simple shared-memory ring buffer.

---
 drivers/char/tty_io.c              |    7 
 drivers/xen/Makefile               |    1 
 drivers/xen/console/Makefile       |    2 
 drivers/xen/console/console.c      |  637 +++++++++++++++++++++++++++++++++++++
 drivers/xen/console/xencons_ring.c |  141 ++++++++
 include/xen/xencons.h              |   14 
 6 files changed, 801 insertions(+), 1 deletion(-)
drivers/char/tty_io.c +6 -1
@@ -132,6 +132,8 @@ LIST_HEAD(tty_drivers);			/* linked list
    vt.c for deeply disgusting hack reasons */
 DEFINE_MUTEX(tty_mutex);
 
+int console_use_vt = 1;
+
 #ifdef CONFIG_UNIX98_PTYS
 extern struct tty_driver *ptm_driver;	/* Unix98 pty masters; for /dev/ptmx */
 extern int pty_limit;		/* Config limit on Unix98 ptys */
@@ -2060,7 +2062,7 @@ retry_open:
 		goto got_driver;
 	}
 #ifdef CONFIG_VT
-	if (device == MKDEV(TTY_MAJOR,0)) {
+	if (console_use_vt && (device == MKDEV(TTY_MAJOR,0))) {
 		extern struct tty_driver *console_driver;
 		driver = console_driver;
 		index = fg_console;
@@ -3258,6 +3260,8 @@ static int __init tty_init(void)
 #endif
 
 #ifdef CONFIG_VT
+	if (!console_use_vt)
+		goto out_vt;
 	cdev_init(&vc0_cdev, &console_fops);
 	if (cdev_add(&vc0_cdev, MKDEV(TTY_MAJOR, 0), 1) ||
 	    register_chrdev_region(MKDEV(TTY_MAJOR, 0), 1, "/dev/vc/0") < 0)
@@ -3266,6 +3270,7 @@ static int __init tty_init(void)
 	class_device_create(tty_class, NULL, MKDEV(TTY_MAJOR, 0), NULL, "tty0");
 
 	vty_init();
+ out_vt:
 #endif
 	return 0;
 }
drivers/xen/Makefile +1 -0
@@ -1,3 +1,4 @@
 
 obj-y	+= core/
+obj-y	+= console/
 
drivers/xen/console/Makefile new file +2 -0
@@ -0,0 +1,2 @@
+
+obj-y	:= console.o xencons_ring.o
drivers/xen/console/console.c new file +637 -0

637 lines not on this page · raw message

drivers/xen/console/xencons_ring.c new file +141 -0

141 lines not on this page · raw message

include/xen/xencons.h new file +14 -0
@@ -0,0 +1,14 @@
+#ifndef __ASM_XENCONS_H__
+#define __ASM_XENCONS_H__
+
+void xencons_force_flush(void);
+void xencons_resume(void);
+
+/* Interrupt work hooks. Receive data, or kick data out. */
+void xencons_rx(char *buf, unsigned len, struct pt_regs *regs);
+void xencons_tx(void);
+
+int xencons_ring_init(void);
+int xencons_ring_send(const char *data, unsigned len);
+
+#endif /* __ASM_XENCONS_H__ */

--

#122 lore raw

Re: [RFC PATCH 29/35] Add the Xen virtual console driver.

drivers/char/tty_io.c LIST_HEAD(tty_drivers); /* linked list · line 136

Quoted 9 lines
drivers/char/tty_io.c +2 -0
@@ -132,6 +132,8 @@ LIST_HEAD(tty_drivers);			/* linked list
    vt.c for deeply disgusting hack reasons */
 DEFINE_MUTEX(tty_mutex);
 
+int console_use_vt = 1;
+

Can you explain why this variable is needed? It shouldn't. If you only register your console as the primary console nothing else should be printed.

drivers/char/tty_io.c LIST_HEAD(tty_drivers); /* linked list · line 138

Quoted 2 lines
drivers/char/tty_io.c +2 -0
+/*** Useful function for console debugging -- goes straight to Xen. ***/
+asmlinkage int xprintk(const char *fmt, ...)

This is called early_printk in the rest of i386. Please use that name too.

-Andi

#123 lore raw

Re: [RFC PATCH 29/35] Add the Xen virtual console driver.

Quoted 7 lines · Andi Kleen
Quoted 7 lines
drivers/char/tty_io.c +0 -0
@@ -132,6 +132,8 @@ LIST_HEAD(tty_drivers);			/* linked list
    vt.c for deeply disgusting hack reasons */
 DEFINE_MUTEX(tty_mutex);
 
Quoted 6 lines

drivers/char/tty_io.c LIST_HEAD(tty_drivers); /* linked list · line 136

Quoted 2 lines
drivers/char/tty_io.c +2 -0
+int console_use_vt = 1;
+

Can you explain why this variable is needed? It shouldn't. If you only register your console as the primary console nothing else should be printed.

It is needed for having a kernel which can run both as a dom0 and domU kernel. For dom0, you want to build with CONFIG_VT enabled while for domU, the CONFIG_VT code doesn't let the xen console be the primary console unless you specify it on the command line.

I've removed this from the patchqueue since a kernel built from the patchqueue won't run as a dom0 kernel anyway.

Quoted 4 lines

drivers/char/tty_io.c LIST_HEAD(tty_drivers); /* linked list · line 138

Quoted 2 lines
drivers/char/tty_io.c +2 -0
+/*** Useful function for console debugging -- goes straight to Xen. ***/
+asmlinkage int xprintk(const char *fmt, ...)

This is called early_printk in the rest of i386. Please use that name too.

We have proper early_printk support but it's not part of the patchqueue. This is more of a debug function where you would rename the kernel/printk to something else and rename this to printk. I've removed it.

christian

#124 lore raw

Re: [RFC PATCH 29/35] Add the Xen virtual console driver.

In reply to Chris Wright
Quoted 10 lines · On Tue, 09 May 2006 00:00:29 -0700 Chris Wright <[email protected]> wrote:

This provides a bootstrap and ongoing emergency console which is intended to be available from very early during boot and at all times thereafter, in contrast with alternatives such as UDP-based syslogd, or logging in via ssh. The protocol is based on a simple shared-memory ring buffer.

...

drivers/xen/console/console.c new file +2 -0
+/* The kernel and user-land drivers share a common transmit buffer. */
+static unsigned int wbuf_size = 4096;

drivers/xen/console/console.c line 123

Quoted 10 lines
drivers/xen/console/console.c new file +10 -0
+#define WBUF_MASK(_i) ((_i)&(wbuf_size-1))
+static char *wbuf;
+static unsigned int wc, wp; /* write_cons, write_prod */
+
+static int __init xencons_bufsz_setup(char *str)
+{
+	unsigned int goal;
+	goal = simple_strtoul(str, NULL, 0);
+	while (wbuf_size < goal)
+		wbuf_size <<= 1;

roundup_pow_of_two()

drivers/xen/console/console.c line 129

Quoted 2 lines
drivers/xen/console/console.c new file +2 -0
+/* This lock protects accesses to the common transmit buffer. */
+static spinlock_t xencons_lock = SPIN_LOCK_UNLOCKED;

DEFINE_SPINLOCK() (entire patchset)

Quoted 12 lines
drivers/xen/console/console.c new file +12 -0
+
+static void kcons_write(
+	struct console *c, const char *s, unsigned int count)
+{
+	int           i = 0;
+	unsigned long flags;
+
+	spin_lock_irqsave(&xencons_lock, flags);
+
+	while (i < count) {
+		for (; i < count; i++) {
+			if ((wp - wc) >= (wbuf_size - 1))

drivers/xen/console/console.c line 158

Quoted 10 lines
drivers/xen/console/console.c new file +10 -0
+				break;
+			if ((wbuf[WBUF_MASK(wp++)] = s[i]) == '\n')
+				wbuf[WBUF_MASK(wp++)] = '\r';
+		}
+
+		__xencons_tx_flush();
+	}
+
+	spin_unlock_irqrestore(&xencons_lock, flags);
+}

hm. You have all that elaborate generate-ringbuffer-code-with-C-macros stuff in the header file patch, yet this code (blessedly) doesn't use it.

drivers/xen/console/console.c line 171

Quoted 12 lines
drivers/xen/console/console.c new file +12 -0
+static void kcons_write_dom0(
+	struct console *c, const char *s, unsigned int count)
+{
+	int rc;
+
+	while ((count > 0) &&
+	       ((rc = HYPERVISOR_console_io(
+			CONSOLEIO_write, count, (char *)s)) > 0)) {
+		count -= rc;
+		s += rc;
+	}
+}

must.. not.. mention.. coding.. style..

#125 lore raw

Re: [RFC PATCH 29/35] Add the Xen virtual console driver.

Quoted 7 lines · Andrew Morton
Quoted 7 lines
drivers/xen/console/console.c new file +7 -0
+static void kcons_write_dom0(
+	struct console *c, const char *s, unsigned int count)
+{
+	int rc;
+
+	while ((count > 0) &&
+	       ((rc = HYPERVISOR_console_io(
Quoted 8 lines

drivers/xen/console/console.c line 171

Quoted 5 lines
drivers/xen/console/console.c new file +5 -0
+			CONSOLEIO_write, count, (char *)s)) > 0)) {
+		count -= rc;
+		s += rc;
+	}
+}

must.. not.. mention.. coding.. style..

Someone should write you a script to go through a patch and flag the most common style mistakes. Have the output formatted to look like you're replying to the mail, and wire it up to your inbox ;)

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#126 lore raw

Re: [RFC PATCH 29/35] Add the Xen virtual console driver.

Quoted 19 lines · Nick Piggin
Quoted 7 lines · Andrew Morton
Quoted 7 lines
drivers/xen/console/console.c new file +7 -0
+static void kcons_write_dom0(
+	struct console *c, const char *s, unsigned int count)
+{
+	int rc;
+
+	while ((count > 0) &&
+	       ((rc = HYPERVISOR_console_io(
Quoted 8 lines

drivers/xen/console/console.c line 171

Quoted 5 lines
drivers/xen/console/console.c new file +5 -0
+			CONSOLEIO_write, count, (char *)s)) > 0)) {
+		count -= rc;
+		s += rc;
+	}
+}

must.. not.. mention.. coding.. style..

Quoted 4 lines

Someone should write you a script to go through a patch and flag the most common style mistakes. Have the output formatted to look like you're replying to the mail, and wire it up to your inbox ;)

Even better, someone should write a coding style document, so people get it right from the outset.

Clever, aren't I?

#127 lore raw

Re: [RFC PATCH 29/35] Add the Xen virtual console driver.

Quoted 11 lines · Andrew Morton
Quoted 4 lines · Nick Piggin

Someone should write you a script to go through a patch and flag the most common style mistakes. Have the output formatted to look like you're replying to the mail, and wire it up to your inbox ;)

Even better, someone should write a coding style document, so people get it right from the outset.

I thought that was tried several years back -- I noticed you still do it manually, so I just assumed that the style document scheme hadn't worked.

Quoted 3 lines

Clever, aren't I?

Yes... but I don't think it's your cleverness that's the problem ;)

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#128 lore raw

RFC PATCH 30/35 Add apply_to_page_range() function

In reply to Chris Wright

Add a new mm function apply_to_page_range() which applies a given function to every pte in a given virtual address range in a given mm structure. This is a generic alternative to cut-and-pasting the Linux idiomatic pagetable walking code in every place that a sequence of PTEs must be accessed.

Although this interface is intended to be useful in a wide range of situations, it is currently used specifically by several Xen subsystems, for example: to ensure that pagetables have been allocated for a virtual address range, and to construct batched special pagetable update requests to map I/O memory (in ioremap()).

---
 include/linux/mm.h |    5 ++
 mm/memory.c        |   94 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 99 insertions(+)
include/linux/mm.h +5 -0
@@ -1014,6 +1014,11 @@ struct page *follow_page(struct vm_area_
 #define FOLL_GET	0x04	/* do get_page on page */
 #define FOLL_ANON	0x08	/* give ZERO_PAGE if no pgtable */
 
+typedef int (*pte_fn_t)(pte_t *pte, struct page *pmd_page, unsigned long addr,
+			void *data);
+extern int apply_to_page_range(struct mm_struct *mm, unsigned long address,
+			       unsigned long size, pte_fn_t fn, void *data);
+
 #ifdef CONFIG_PROC_FS
 void vm_stat_account(struct mm_struct *, unsigned long, struct file *, long);
 #else
mm/memory.c +94 -0

100 lines not on this page · raw message

--

#129 lore raw

RFC PATCH 31/35 Add Xen grant table support

In reply to Chris Wright

Add Xen 'grant table' driver which allows granting of access to selected local memory pages by other virtual machines and, symmetrically, the mapping of remote memory pages which other virtual machines have granted access to.

This driver is a prerequisite for many of the Xen virtual device drivers, which grant the 'device driver domain' restricted and temporary access to only those memory pages that are currently involved in I/O operations.

---
 drivers/xen/core/Makefile |    2 
 drivers/xen/core/gnttab.c |  422 ++++++++++++++++++++++++++++++++++++++++++++++
 include/xen/gnttab.h      |  105 +++++++++++
 3 files changed, 528 insertions(+), 1 deletion(-)
drivers/xen/core/Makefile +1 -1
@@ -1,3 +1,3 @@
 
-obj-y	:= features.o
+obj-y	:= features.o gnttab.o
 
include/xen/gnttab.h new file +105 -0

105 lines not on this page · raw message

drivers/xen/core/gnttab.c new file +422 -0

422 lines not on this page · raw message

--

#130 lore raw

RFC PATCH 32/35 Add Xen driver utility functions.

In reply to Chris Wright

Allocate/destroy a 'vmalloc' VM area: alloc_vm_area and free_vm_area The alloc function ensures that page tables are constructed for the region of kernel virtual address space and mapped into init_mm.

Lock an area so that PTEs are accessible in the current address space:
lock_vm_area and unlock_vm_area
The lock function prevents context switches to a lazy mm that doesn't have the area mapped into its page tables. It also ensures that the page tables are mapped into the current mm by causing the page fault handler to copy the page directory pointers from init_mm into the current mm.

---
TODO:
- possible vmalloc_sync use instead

 drivers/xen/Makefile      |    2 +
 drivers/xen/util.c        |   70 ++++++++++++++++++++++++++++++++++++++++++++++
 include/xen/driver_util.h |   16 ++++++++++
 3 files changed, 88 insertions(+)
drivers/xen/Makefile +2 -0
@@ -1,4 +1,6 @@
 
+obj-y	+= util.o
+
 obj-y	+= core/
 obj-y	+= console/
 
drivers/xen/util.c new file +70 -0

70 lines not on this page · raw message

include/xen/driver_util.h new file +16 -0
@@ -0,0 +1,16 @@
+
+#ifndef __ASM_XEN_DRIVER_UTIL_H__
+#define __ASM_XEN_DRIVER_UTIL_H__
+
+#include <linux/config.h>
+#include <linux/vmalloc.h>
+
+/* Allocate/destroy a 'vmalloc' VM area. */
+extern struct vm_struct *alloc_vm_area(unsigned long size);
+extern void free_vm_area(struct vm_struct *area);
+
+/* Lock an area so that PTEs are accessible in the current address space. */
+extern void lock_vm_area(struct vm_struct *area);
+extern void unlock_vm_area(struct vm_struct *area);
+
+#endif /* __ASM_XEN_DRIVER_UTIL_H__ */

--

#131 lore raw

Re: [RFC PATCH 32/35] Add Xen driver utility functions.

drivers/xen/util.c line 70

Quoted 7 lines · Chris Wright
drivers/xen/util.c new file +4 -0
+EXPORT_SYMBOL_GPL(alloc_vm_area);
 
+EXPORT_SYMBOL_GPL(free_vm_area);
 
+EXPORT_SYMBOL_GPL(lock_vm_area);
 
+EXPORT_SYMBOL_GPL(unlock_vm_area);

These are all pretty generic function names. Perhaps they belong in the core kernel code instead of a Xen specific file?

thanks,

greg k-h

#132 lore raw

Re: [RFC PATCH 32/35] Add Xen driver utility functions.

In reply to Chris Wright
Quoted 11 lines · Chris Wright

Allocate/destroy a 'vmalloc' VM area: alloc_vm_area and free_vm_area The alloc function ensures that page tables are constructed for the region of kernel virtual address space and mapped into init_mm.

Lock an area so that PTEs are accessible in the current address space:
lock_vm_area and unlock_vm_area
The lock function prevents context switches to a lazy mm that doesn't have the area mapped into its page tables. It also ensures that the page tables are mapped into the current mm by causing the page fault handler to copy the page directory pointers from init_mm into the current mm.

Having that in drivers/xen looks wrong. It should be probably somewhere generic.

-Andi

#133 lore raw

RFC PATCH 33/35 Add the Xenbus sysfs and virtual device hotplug driver.

In reply to Chris Wright

This communicates with the machine control software via a registry residing in a controlling virtual machine. This allows dynamic creation, destruction and modification of virtual device configurations (network devices, block devices and CPUS, to name some examples).

---
 drivers/xen/Makefile               |    1 
 drivers/xen/xenbus/Makefile        |    7 
 drivers/xen/xenbus/xenbus_client.c |  395 +++++++++++++
 drivers/xen/xenbus/xenbus_comms.c  |  208 +++++++
 drivers/xen/xenbus/xenbus_comms.h  |   43 +
 drivers/xen/xenbus/xenbus_probe.c  | 1065 +++++++++++++++++++++++++++++++++++++
 drivers/xen/xenbus/xenbus_xs.c     |  829 ++++++++++++++++++++++++++++
 include/xen/xenbus.h               |  291 ++++++++++
 8 files changed, 2839 insertions(+)
drivers/xen/Makefile +1 -0
@@ -3,4 +3,5 @@ obj-y	+= util.o
 
 obj-y	+= core/
 obj-y	+= console/
+obj-y	+= xenbus/
 
drivers/xen/xenbus/Makefile new file +7 -0
@@ -0,0 +1,7 @@
+obj-y	+= xenbus.o
+
+xenbus-objs =
+xenbus-objs += xenbus_client.o
+xenbus-objs += xenbus_comms.o
+xenbus-objs += xenbus_xs.o
+xenbus-objs += xenbus_probe.o
drivers/xen/xenbus/xenbus_client.c new file +395 -0

395 lines not on this page · raw message

drivers/xen/xenbus/xenbus_comms.c new file +208 -0

208 lines not on this page · raw message

drivers/xen/xenbus/xenbus_comms.h new file +43 -0

43 lines not on this page · raw message

drivers/xen/xenbus/xenbus_probe.c new file +1065 -0

1,065 lines not on this page · raw message

drivers/xen/xenbus/xenbus_xs.c new file +829 -0

829 lines not on this page · raw message

include/xen/xenbus.h new file +291 -0

291 lines not on this page · raw message

--

#134 lore raw

Re: [RFC PATCH 33/35] Add the Xenbus sysfs and virtual device hotplug driver.

Quoted 10 lines
drivers/xen/xenbus/xenbus_probe.c new file +10 -0
+/* Simplified asprintf. */
+char *kasprintf(const char *fmt, ...)
+{
+	va_list ap;
+	unsigned int len;
+	char *p, dummy[1];
+
+	va_start(ap, fmt);
+	/* FIXME: vsnprintf has a bug, NULL should work */
+	len = vsnprintf(dummy, 0, fmt, ap);

drivers/xen/xenbus/xenbus_probe.c line 517

Quoted 10 lines
drivers/xen/xenbus/xenbus_probe.c new file +10 -0
+	va_end(ap);
+
+	p = kmalloc(len + 1, GFP_KERNEL);
+	if (!p)
+		return NULL;
+	va_start(ap, fmt);
+	vsprintf(p, fmt, ap);
+	va_end(ap);
+	return p;
+}

This should go to lib/

#135 lore raw

Re: [RFC PATCH 33/35] Add the Xenbus sysfs and virtual device hotplug driver.

Quoted 14 lines · Alexey Dobriyan
Quoted 14 lines
drivers/xen/xenbus/xenbus_probe.c new file +14 -0
+/* Simplified asprintf. */
+char *kasprintf(const char *fmt, ...)
+{
+	va_list ap;
+	unsigned int len;
+	char *p, dummy[1];
+
+	va_start(ap, fmt);
+	/* FIXME: vsnprintf has a bug, NULL should work */
+	len = vsnprintf(dummy, 0, fmt, ap);
+	va_end(ap);
+
+	p = kmalloc(len + 1, GFP_KERNEL);
+	if (!p)
Quoted 8 lines

drivers/xen/xenbus/xenbus_probe.c line 517

Quoted 6 lines
drivers/xen/xenbus/xenbus_probe.c new file +6 -0
+		return NULL;
+	va_start(ap, fmt);
+	vsprintf(p, fmt, ap);
+	va_end(ap);
+	return p;
+}

This should go to lib/

First for kernel usage I think it should have a maximum length parameter to avoid dumb code from being easily exploited.

And the bug should be fixed in vsnprintf instead of being worked around.

-Andi

#136 lore raw

Re: [RFC PATCH 33/35] Add the Xenbus sysfs and virtual device hotplug driver.

In reply to Chris Wright

Review

Quoted 2 lines · Chris Wright
(quoted diff) +2 -0
+/* xenbus_probe.c */
+extern char *kasprintf(const char *fmt, ...);

Belongs in a .h file.

drivers/xen/xenbus/xenbus_client.c line 42

Quoted 2 lines
drivers/xen/xenbus/xenbus_client.c new file +2 -0
+#define DPRINTK(fmt, args...) \
+    pr_debug("xenbus_client (%s:%d) " fmt ".\n", __FUNCTION__, __LINE__, ##args)

Please use the dev_dbg() function instead of DPRINTK() or pr_debug(). It's much better and uniquely identifies the driver and device that you are referring to.

Also, all of the printk() calls in these files should be switched to dev_err() or dev_warn() for the same reason.

Quoted 36 lines
drivers/xen/xenbus/xenbus_client.c new file +36 -0
+int xenbus_watch_path(struct xenbus_device *dev, const char *path,
+		      struct xenbus_watch *watch,
+		      void (*callback)(struct xenbus_watch *,
+				       const char **, unsigned int))
+{
+	int err;
+
+	watch->node = path;
+	watch->callback = callback;
+
+	err = register_xenbus_watch(watch);
+
+	if (err) {
+		watch->node = NULL;
+		watch->callback = NULL;
+		xenbus_dev_fatal(dev, err, "adding watch on %s", path);
+	}
+
+	return err;
+}
+EXPORT_SYMBOL_GPL(xenbus_watch_path);
+
+
+int xenbus_watch_path2(struct xenbus_device *dev, const char *path,
+		       const char *path2, struct xenbus_watch *watch,
+		       void (*callback)(struct xenbus_watch *,
+					const char **, unsigned int))
+{
+	int err;
+	char *state = kasprintf("%s/%s", path, path2);
+	if (!state) {
+		xenbus_dev_fatal(dev, -ENOMEM, "allocating path for watch");
+		return -ENOMEM;
+	}
+	err = xenbus_watch_path(dev, state, watch, callback);
+

drivers/xen/xenbus/xenbus_client.c line 89

Quoted 10 lines
drivers/xen/xenbus/xenbus_client.c new file +10 -0
+	if (err)
+		kfree(state);
+	return err;
+}
+EXPORT_SYMBOL_GPL(xenbus_watch_path2);
+
+
+int xenbus_switch_state(struct xenbus_device *dev,
+			xenbus_transaction_t xbt,
+			XenbusState state)

I'm guessing that XenbusState is a typedef? Please fix the naming to be Linux kernel compatible.

Quoted 36 lines
drivers/xen/xenbus/xenbus_client.c new file +36 -0
+{
+	/* We check whether the state is currently set to the given value, and
+	   if not, then the state is set.  We don't want to unconditionally
+	   write the given state, because we don't want to fire watches
+	   unnecessarily.  Furthermore, if the node has gone, we don't write
+	   to it, as the device will be tearing down, and we don't want to
+	   resurrect that directory.
+	 */
+
+	int current_state;
+	int err;
+
+	if (state == dev->state)
+		return 0;
+
+	err = xenbus_scanf(xbt, dev->nodename, "state", "%d",
+			       &current_state);
+	if (err != 1)
+		return 0;
+
+	err = xenbus_printf(xbt, dev->nodename, "state", "%d", state);
+	if (err) {
+		if (state != XenbusStateClosing) /* Avoid looping */
+			xenbus_dev_fatal(dev, err, "writing new state");
+		return err;
+	}
+
+	dev->state = state;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(xenbus_switch_state);
+
+
+/**
+ * Return the path to the error node for the given device, or NULL on failure.

drivers/xen/xenbus/xenbus_client.c line 135

Quoted 10 lines
drivers/xen/xenbus/xenbus_client.c new file +10 -0
+ * If the value returned is non-NULL, then it is the caller's to kfree.
+ */
+static char *error_path(struct xenbus_device *dev)
+{
+	return kasprintf("error/%s", dev->nodename);
+}
+
+
+void _dev_error(struct xenbus_device *dev, int err, const char *fmt,
+		va_list ap)

Global function? With no description of what it does? (hint, describing it in the .h file, in pseudo-kerneldoc form doesn't really count, it only makes the tools break...)

Review

Quoted 2 lines
(quoted diff) +2 -0
+void xenbus_dev_error(struct xenbus_device *dev, int err, const char *fmt,
+		      ...)

No kerneldoc for all of the global functions?

drivers/xen/xenbus/xenbus_comms.c line 45

Quoted 2 lines
drivers/xen/xenbus/xenbus_comms.c new file +2 -0
+extern void xenbus_probe(void *);
+extern int xenstored_ready;

Should be in a .h file.

drivers/xen/xenbus/xenbus_probe.c line 52

Quoted 8 lines
drivers/xen/xenbus/xenbus_probe.c new file +8 -0
+#include <asm/io.h>
+#include <asm/page.h>
+#include <asm/pgtable.h>
+#include <asm/hypervisor.h>
+#include <xen/xenbus.h>
+#ifdef XEN_XENBUS_PROC_INTERFACE
+#include <xen/xen_proc.h>
+#endif

#ifdef is not needed. Put it in the .h file.

drivers/xen/xenbus/xenbus_probe.c line 57

Quoted 5 lines
drivers/xen/xenbus/xenbus_probe.c new file +5 -0
+#include <xen/evtchn.h>
+
+#include "xenbus_comms.h"
+
+extern struct mutex xenwatch_mutex;

Should be in a .h file.

drivers/xen/xenbus/xenbus_probe.c line 90

Quoted 9 lines
drivers/xen/xenbus/xenbus_probe.c new file +9 -0
+struct xen_bus_type
+{
+	char *root;
+	unsigned int levels;
+	int (*get_bus_id)(char bus_id[BUS_ID_SIZE], const char *nodename);
+	int (*probe)(const char *type, const char *dir);
+	struct bus_type bus;
+	struct device dev;
+};

Why have you embedded both a struct bus_type and a struct device into this structure? How is the lifecycle handled due to 2 different reference counted structures?

Also, I note that you statically create this, which isn't the nicest, why not dynamically create it?

And why a different probe function per bus types?

And the bus id is part of the struct bus_type, why a separate function to retrieve it?

And what are you doing with the different "levels"? Is there some description of how you are using sysfs for this? Busses should not be "nested", devices should. How does sysfs look with this code in it? What is the /sys/bus/ structure? What is the /sys/devices/ structure?

Quoted 7 lines
drivers/xen/xenbus/xenbus_probe.c new file +7 -0
+/* device/<type>/<id> => <type>-<id> */
+static int frontend_bus_id(char bus_id[BUS_ID_SIZE], const char *nodename)
+{
+	nodename = strchr(nodename, '/');
+	if (!nodename || strlen(nodename + 1) >= BUS_ID_SIZE) {
+		printk(KERN_WARNING "XENBUS: bad frontend %s\n", nodename);
+		return -EINVAL;

drivers/xen/xenbus/xenbus_probe.c line 109

Quoted 10 lines
drivers/xen/xenbus/xenbus_probe.c new file +10 -0
+	}
+
+	strlcpy(bus_id, nodename + 1, BUS_ID_SIZE);
+	if (!strchr(bus_id, '/')) {
+		printk(KERN_WARNING "XENBUS: bus_id %s no slash\n", bus_id);
+		return -EINVAL;
+	}
+	*strchr(bus_id, '/') = '-';
+	return 0;
+}

And why all the string logic for device ids and names? Is that the only unique way to identify the different devices on your bus? Why not just give them a numerical id then? It would save you a lot of computation...

Quoted 61 lines
drivers/xen/xenbus/xenbus_probe.c new file +61 -0
+/* Bus type for frontend drivers. */
+static int xenbus_probe_frontend(const char *type, const char *name);
+static struct xen_bus_type xenbus_frontend = {
+	.root = "device",
+	.levels = 2, 		/* device/type/<id> */
+	.get_bus_id = frontend_bus_id,
+	.probe = xenbus_probe_frontend,
+	.bus = {
+		.name  = "xen",
+		.match = xenbus_match,
+	},
+	.dev = {
+		.bus_id = "xen",
+	},
+};
+
+/* backend/<type>/<fe-uuid>/<id> => <type>-<fe-domid>-<id> */
+static int backend_bus_id(char bus_id[BUS_ID_SIZE], const char *nodename)
+{
+	int domid, err;
+	const char *devid, *type, *frontend;
+	unsigned int typelen;
+
+	type = strchr(nodename, '/');
+	if (!type)
+		return -EINVAL;
+	type++;
+	typelen = strcspn(type, "/");
+	if (!typelen || type[typelen] != '/')
+		return -EINVAL;
+
+	devid = strrchr(nodename, '/') + 1;
+
+	err = xenbus_gather(XBT_NULL, nodename, "frontend-id", "%i", &domid,
+			    "frontend", NULL, &frontend,
+			    NULL);
+	if (err)
+		return err;
+	if (strlen(frontend) == 0)
+		err = -ERANGE;
+	if (!err && !xenbus_exists(XBT_NULL, frontend, ""))
+		err = -ENOENT;
+
+	kfree(frontend);
+
+	if (err)
+		return err;
+
+	if (snprintf(bus_id, BUS_ID_SIZE,
+		     "%.*s-%i-%s", typelen, type, domid, devid) >= BUS_ID_SIZE)
+		return -ENOSPC;
+	return 0;
+}
+
+static int xenbus_uevent_backend(struct device *dev, char **envp,
+				 int num_envp, char *buffer, int buffer_size);
+static int xenbus_probe_backend(const char *type, const char *domid);
+static struct xen_bus_type xenbus_backend = {
+	.root = "backend",
+	.levels = 3, 		/* backend/type/<frontend>/<id> */
+	.get_bus_id = backend_bus_id,

drivers/xen/xenbus/xenbus_probe.c line 236

Quoted 10 lines
drivers/xen/xenbus/xenbus_probe.c new file +10 -0
+	.probe = xenbus_probe_backend,
+	.bus = {
+		.name  = "xen-backend",
+		.match = xenbus_match,
+		.uevent = xenbus_uevent_backend,
+	},
+	.dev = {
+		.bus_id = "xen-backend",
+	},
+};

What is the "frontend/backend" relationship here?

Quoted 16 lines
drivers/xen/xenbus/xenbus_probe.c new file +16 -0
+static int xenbus_uevent_backend(struct device *dev, char **envp,
+				 int num_envp, char *buffer, int buffer_size)
+{
+	struct xenbus_device *xdev;
+	struct xenbus_driver *drv;
+	int i = 0;
+	int length = 0;
+
+	DPRINTK("");
+
+	if (dev == NULL)
+		return -ENODEV;
+
+	xdev = to_xenbus_device(dev);
+	if (xdev == NULL)
+		return -ENODEV;

drivers/xen/xenbus/xenbus_probe.c line 263

Quoted 10 lines
drivers/xen/xenbus/xenbus_probe.c new file +10 -0
+
+	/* stuff we want to pass to /sbin/hotplug */
+	add_uevent_var(envp, num_envp, &i, buffer, buffer_size, &length,
+		       "XENBUS_TYPE=%s", xdev->devicetype);
+
+	add_uevent_var(envp, num_envp, &i, buffer, buffer_size, &length,
+		       "XENBUS_PATH=%s", xdev->nodename);
+
+	add_uevent_var(envp, num_envp, &i, buffer, buffer_size, &length,
+		       "XENBUS_BASE_PATH=%s", xenbus_backend.root);

Why not use the standard "TYPE" "PATH" and no "XENBUS" stuff?

drivers/xen/xenbus/xenbus_probe.c line 397

Quoted 8 lines
drivers/xen/xenbus/xenbus_probe.c new file +8 -0
+static int xenbus_register_driver_common(struct xenbus_driver *drv,
+					 struct xen_bus_type *bus)
+{
+	int ret;
+
+	drv->driver.name = drv->name;
+	drv->driver.bus = &bus->bus;
+	drv->driver.owner = drv->owner;

Hint, put the owner in the function, which will force the caller to pass it in. It's forgotten to be set in the structure a lot. Look at USB and PCI register functions now for an example of this.

drivers/xen/xenbus/xenbus_probe.c line 403

Quoted 6 lines
drivers/xen/xenbus/xenbus_probe.c new file +6 -0
+	drv->driver.probe = xenbus_dev_probe;
+	drv->driver.remove = xenbus_dev_remove;
+
+	mutex_lock(&xenwatch_mutex);
+	ret = driver_register(&drv->driver);
+	mutex_unlock(&xenwatch_mutex);

What's with the lock? What is wrong with the driver core lock that is taken? What are you trying to protect?

drivers/xen/xenbus/xenbus_probe.c line 845

Quoted 9 lines
drivers/xen/xenbus/xenbus_probe.c new file +9 -0
+void xenbus_suspend(void)
+{
+	DPRINTK("");
+
+	bus_for_each_dev(&xenbus_frontend.bus, NULL, NULL, suspend_dev);
+	bus_for_each_dev(&xenbus_backend.bus, NULL, NULL, suspend_dev);
+	xs_suspend();
+}
+EXPORT_SYMBOL_GPL(xenbus_suspend);

I think the driver core will handle walking your devices and suspending them. You don't have to do it by hand like this.

drivers/xen/xenbus/xenbus_probe.c line 854

Quoted 8 lines
drivers/xen/xenbus/xenbus_probe.c new file +8 -0
+void xenbus_resume(void)
+{
+	xb_init_comms();
+	xs_resume();
+	bus_for_each_dev(&xenbus_frontend.bus, NULL, NULL, resume_dev);
+	bus_for_each_dev(&xenbus_backend.bus, NULL, NULL, resume_dev);
+}
+EXPORT_SYMBOL_GPL(xenbus_resume);

Same thing for resume.

drivers/xen/xenbus/xenbus_probe.c line 948

Quoted 4 lines
drivers/xen/xenbus/xenbus_probe.c new file +4 -0
+#ifdef XEN_XENBUS_PROC_INTERFACE
+static struct file_operations xsd_kva_fops;
+static struct proc_dir_entry *xsd_kva_intf;
+static struct proc_dir_entry *xsd_port_intf;

#ifdef not needed if your .h file is written correctly.

Quoted 6 lines
drivers/xen/xenbus/xenbus_probe.c new file +6 -0
+static int __init xenbus_probe_init(void)
+{
+	int err = 0, dom0;
+
+	DPRINTK("");
+

drivers/xen/xenbus/xenbus_probe.c line 1002

Quoted 10 lines
drivers/xen/xenbus/xenbus_probe.c new file +10 -0
+	if (xen_init() < 0) {
+		DPRINTK("failed");
+		return -ENODEV;
+	}
+
+	/* Register ourselves with the kernel bus & device subsystems */
+	bus_register(&xenbus_frontend.bus);
+	bus_register(&xenbus_backend.bus);
+	device_register(&xenbus_frontend.dev);
+	device_register(&xenbus_backend.dev);

No error handling?

drivers/xen/xenbus/xenbus_probe.c line 1009

Quoted 7 lines
drivers/xen/xenbus/xenbus_probe.c new file +7 -0
+
+	/*
+	 * Domain0 doesn't have a store_evtchn or store_mfn yet.
+	 */
+	dom0 = (xen_start_info->store_evtchn == 0);
+
+#ifdef XEN_XENBUS_PROC_INTERFACE

No #ifdef needed if your .h file is written correctly.

Review

Quoted 2 lines
(quoted diff) +2 -0
+/* xenbus_probe.c */
+extern char *kasprintf(const char *fmt, ...);

Should be in a .h file

drivers/xen/xenbus/xenbus_xs.c line 101

Quoted 8 lines
drivers/xen/xenbus/xenbus_xs.c new file +8 -0
+/*
+ * Details of the xenwatch callback kernel thread. The thread waits on the
+ * watch_events_waitq for work to do (queued on watch_events list). When it
+ * wakes up it acquires the xenwatch_mutex before reading the list and
+ * carrying out work.
+ */
+static pid_t xenwatch_pid;
+/* static */ DEFINE_MUTEX(xenwatch_mutex);

Drop the static comment?

Quoted 6 lines
drivers/xen/xenbus/xenbus_xs.c new file +6 -0
+/* Create a new directory. */
+int xenbus_mkdir(xenbus_transaction_t t,
+		 const char *dir, const char *node)
+{
+	char *path;
+	int ret;

drivers/xen/xenbus/xenbus_xs.c line 415

Quoted 10 lines
drivers/xen/xenbus/xenbus_xs.c new file +10 -0
+
+	path = join(dir, node);
+	if (IS_ERR(path))
+		return PTR_ERR(path);
+
+	ret = xs_error(xs_single(t, XS_MKDIR, path, NULL));
+	kfree(path);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(xenbus_mkdir);

Create a new directory in what? sysfs?

Quoted 5 lines
drivers/xen/xenbus/xenbus_xs.c new file +5 -0
+/* Destroy a file or directory (directories must be empty). */
+int xenbus_rm(xenbus_transaction_t t, const char *dir, const char *node)
+{
+	char *path;
+	int ret;

drivers/xen/xenbus/xenbus_xs.c line 431

Quoted 10 lines
drivers/xen/xenbus/xenbus_xs.c new file +10 -0
+
+	path = join(dir, node);
+	if (IS_ERR(path))
+		return PTR_ERR(path);
+
+	ret = xs_error(xs_single(t, XS_RM, path, NULL));
+	kfree(path);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(xenbus_rm);

Remove a file or directory in what? sysfs?

Quoted 1 line

+#define XBT_NULL 0

What is this for? What's wrong with just "NULL"?

include/xen/xenbus.h line 70

Quoted 10 lines
include/xen/xenbus.h new file +10 -0
+/* A xenbus device. */
+struct xenbus_device {
+	const char *devicetype;
+	const char *nodename;
+	const char *otherend;
+	int otherend_id;
+	struct xenbus_watch otherend_watch;
+	struct device dev;
+	XenbusState state;
+	void *data;

The data field can be dropped. Use the space in the struct device for this.

Quoted 1 line

+typedef u32 xenbus_transaction_t;

Why the typedef? Please don't.

thanks,

greg k-h

#137 lore raw

Re: [RFC PATCH 33/35] Add the Xenbus sysfs and virtual device hotplug driver.

Quoted 5 lines · * Greg KH ([email protected]) wrote:

Review

Quoted 2 lines · Chris Wright
(quoted diff) +2 -0
+/* xenbus_probe.c */
+extern char *kasprintf(const char *fmt, ...);

Belongs in a .h file.

*nod*

Quoted 9 lines

drivers/xen/xenbus/xenbus_client.c line 42

Quoted 2 lines
drivers/xen/xenbus/xenbus_client.c new file +2 -0
+#define DPRINTK(fmt, args...) \
+    pr_debug("xenbus_client (%s:%d) " fmt ".\n", __FUNCTION__, __LINE__, ##args)

Please use the dev_dbg() function instead of DPRINTK() or pr_debug(). It's much better and uniquely identifies the driver and device that you are referring to.

Also, all of the printk() calls in these files should be switched to dev_err() or dev_warn() for the same reason.

will do, thanks.

Quoted 41 lines
Quoted 41 lines
drivers/xen/xenbus/xenbus_client.c new file +41 -0
+int xenbus_watch_path(struct xenbus_device *dev, const char *path,
+		      struct xenbus_watch *watch,
+		      void (*callback)(struct xenbus_watch *,
+				       const char **, unsigned int))
+{
+	int err;
+
+	watch->node = path;
+	watch->callback = callback;
+
+	err = register_xenbus_watch(watch);
+
+	if (err) {
+		watch->node = NULL;
+		watch->callback = NULL;
+		xenbus_dev_fatal(dev, err, "adding watch on %s", path);
+	}
+
+	return err;
+}
+EXPORT_SYMBOL_GPL(xenbus_watch_path);
+
+
+int xenbus_watch_path2(struct xenbus_device *dev, const char *path,
+		       const char *path2, struct xenbus_watch *watch,
+		       void (*callback)(struct xenbus_watch *,
+					const char **, unsigned int))
+{
+	int err;
+	char *state = kasprintf("%s/%s", path, path2);
+	if (!state) {
+		xenbus_dev_fatal(dev, -ENOMEM, "allocating path for watch");
+		return -ENOMEM;
+	}
+	err = xenbus_watch_path(dev, state, watch, callback);
+
+	if (err)
+		kfree(state);
+	return err;
+}
+EXPORT_SYMBOL_GPL(xenbus_watch_path2);
Quoted 8 lines

drivers/xen/xenbus/xenbus_client.c line 89

Quoted 5 lines
drivers/xen/xenbus/xenbus_client.c new file +5 -0
+
+
+int xenbus_switch_state(struct xenbus_device *dev,
+			xenbus_transaction_t xbt,
+			XenbusState state)

I'm guessing that XenbusState is a typedef? Please fix the naming to be Linux kernel compatible.

good point, these are slowly being eliminated, noted.

Quoted 42 lines
Quoted 42 lines
drivers/xen/xenbus/xenbus_client.c new file +42 -0
+{
+	/* We check whether the state is currently set to the given value, and
+	   if not, then the state is set.  We don't want to unconditionally
+	   write the given state, because we don't want to fire watches
+	   unnecessarily.  Furthermore, if the node has gone, we don't write
+	   to it, as the device will be tearing down, and we don't want to
+	   resurrect that directory.
+	 */
+
+	int current_state;
+	int err;
+
+	if (state == dev->state)
+		return 0;
+
+	err = xenbus_scanf(xbt, dev->nodename, "state", "%d",
+			       &current_state);
+	if (err != 1)
+		return 0;
+
+	err = xenbus_printf(xbt, dev->nodename, "state", "%d", state);
+	if (err) {
+		if (state != XenbusStateClosing) /* Avoid looping */
+			xenbus_dev_fatal(dev, err, "writing new state");
+		return err;
+	}
+
+	dev->state = state;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(xenbus_switch_state);
+
+
+/**
+ * Return the path to the error node for the given device, or NULL on failure.
+ * If the value returned is non-NULL, then it is the caller's to kfree.
+ */
+static char *error_path(struct xenbus_device *dev)
+{
+	return kasprintf("error/%s", dev->nodename);
+}
Quoted 8 lines

drivers/xen/xenbus/xenbus_client.c line 135

Quoted 4 lines
drivers/xen/xenbus/xenbus_client.c new file +4 -0
+
+
+void _dev_error(struct xenbus_device *dev, int err, const char *fmt,
+		va_list ap)

Global function? With no description of what it does? (hint, describing it in the .h file, in pseudo-kerneldoc form doesn't really count, it only makes the tools break...)

yeah, all of _dev_error cleanup is already on the todo list. for starters, good names and better docs will help.

Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+void xenbus_dev_error(struct xenbus_device *dev, int err, const char *fmt,
+		      ...)

No kerneldoc for all of the global functions?

will update with kerneldoc.

Quoted 4 lines

drivers/xen/xenbus/xenbus_comms.c line 45

Quoted 2 lines
drivers/xen/xenbus/xenbus_comms.c new file +2 -0
+extern void xenbus_probe(void *);
+extern int xenstored_ready;

Should be in a .h file.

yes

Quoted 10 lines

drivers/xen/xenbus/xenbus_probe.c line 52

Quoted 8 lines
drivers/xen/xenbus/xenbus_probe.c new file +8 -0
+#include <asm/io.h>
+#include <asm/page.h>
+#include <asm/pgtable.h>
+#include <asm/hypervisor.h>
+#include <xen/xenbus.h>
+#ifdef XEN_XENBUS_PROC_INTERFACE
+#include <xen/xen_proc.h>
+#endif

#ifdef is not needed. Put it in the .h file.

better if we can eliminate, but .h for now.

Quoted 7 lines

drivers/xen/xenbus/xenbus_probe.c line 57

Quoted 5 lines
drivers/xen/xenbus/xenbus_probe.c new file +5 -0
+#include <xen/evtchn.h>
+
+#include "xenbus_comms.h"
+
+extern struct mutex xenwatch_mutex;

Should be in a .h file.

yes

Quoted 14 lines

drivers/xen/xenbus/xenbus_probe.c line 90

Quoted 9 lines
drivers/xen/xenbus/xenbus_probe.c new file +9 -0
+struct xen_bus_type
+{
+	char *root;
+	unsigned int levels;
+	int (*get_bus_id)(char bus_id[BUS_ID_SIZE], const char *nodename);
+	int (*probe)(const char *type, const char *dir);
+	struct bus_type bus;
+	struct device dev;
+};

Why have you embedded both a struct bus_type and a struct device into this structure? How is the lifecycle handled due to 2 different reference counted structures?

Quoted 7 lines

Also, I note that you statically create this, which isn't the nicest, why not dynamically create it?

And why a different probe function per bus types?

And the bus id is part of the struct bus_type, why a separate function to retrieve it?

Good questions, I need to dig a little deeper on that. Xenbus is an interesting beast on its own.

Quoted 4 lines

And what are you doing with the different "levels"? Is there some description of how you are using sysfs for this? Busses should not be "nested", devices should. How does sysfs look with this code in it? What is the /sys/bus/ structure? What is the /sys/devices/ structure?

e.g.

/sys/bus/xen

|-- devices
|   |-- vbd-51713 -> ../../../devices/xen/vbd-51713
|   `-- vif-0 -> ../../../devices/xen/vif-0
`-- drivers
    `-- vbd
        |-- bind
        |-- unbind
        `-- vbd-51713 -> ../../../../devices/xen/vbd-51713

/sys/devices/xen

|-- uevent
|-- vbd-51713
|   |-- block:xvda1 -> ../../../block/xvda1
|   |-- bus -> ../../../bus/xen
|   |-- devtype
|   |-- driver -> ../../../bus/xen/drivers/vbd
|   |-- nodename
|   `-- uevent
`-- vif-0
    |-- bus -> ../../../bus/xen
    |-- devtype
    |-- nodename
    `-- uevent
Quoted 88 lines
Quoted 7 lines
drivers/xen/xenbus/xenbus_probe.c new file +7 -0
+/* device/<type>/<id> => <type>-<id> */
+static int frontend_bus_id(char bus_id[BUS_ID_SIZE], const char *nodename)
+{
+	nodename = strchr(nodename, '/');
+	if (!nodename || strlen(nodename + 1) >= BUS_ID_SIZE) {
+		printk(KERN_WARNING "XENBUS: bad frontend %s\n", nodename);
+		return -EINVAL;

drivers/xen/xenbus/xenbus_probe.c line 109

Quoted 10 lines
drivers/xen/xenbus/xenbus_probe.c new file +10 -0
+	}
+
+	strlcpy(bus_id, nodename + 1, BUS_ID_SIZE);
+	if (!strchr(bus_id, '/')) {
+		printk(KERN_WARNING "XENBUS: bus_id %s no slash\n", bus_id);
+		return -EINVAL;
+	}
+	*strchr(bus_id, '/') = '-';
+	return 0;
+}

And why all the string logic for device ids and names? Is that the only unique way to identify the different devices on your bus? Why not just give them a numerical id then? It would save you a lot of computation...

Quoted 65 lines
drivers/xen/xenbus/xenbus_probe.c new file +65 -0
+/* Bus type for frontend drivers. */
+static int xenbus_probe_frontend(const char *type, const char *name);
+static struct xen_bus_type xenbus_frontend = {
+	.root = "device",
+	.levels = 2, 		/* device/type/<id> */
+	.get_bus_id = frontend_bus_id,
+	.probe = xenbus_probe_frontend,
+	.bus = {
+		.name  = "xen",
+		.match = xenbus_match,
+	},
+	.dev = {
+		.bus_id = "xen",
+	},
+};
+
+/* backend/<type>/<fe-uuid>/<id> => <type>-<fe-domid>-<id> */
+static int backend_bus_id(char bus_id[BUS_ID_SIZE], const char *nodename)
+{
+	int domid, err;
+	const char *devid, *type, *frontend;
+	unsigned int typelen;
+
+	type = strchr(nodename, '/');
+	if (!type)
+		return -EINVAL;
+	type++;
+	typelen = strcspn(type, "/");
+	if (!typelen || type[typelen] != '/')
+		return -EINVAL;
+
+	devid = strrchr(nodename, '/') + 1;
+
+	err = xenbus_gather(XBT_NULL, nodename, "frontend-id", "%i", &domid,
+			    "frontend", NULL, &frontend,
+			    NULL);
+	if (err)
+		return err;
+	if (strlen(frontend) == 0)
+		err = -ERANGE;
+	if (!err && !xenbus_exists(XBT_NULL, frontend, ""))
+		err = -ENOENT;
+
+	kfree(frontend);
+
+	if (err)
+		return err;
+
+	if (snprintf(bus_id, BUS_ID_SIZE,
+		     "%.*s-%i-%s", typelen, type, domid, devid) >= BUS_ID_SIZE)
+		return -ENOSPC;
+	return 0;
+}
+
+static int xenbus_uevent_backend(struct device *dev, char **envp,
+				 int num_envp, char *buffer, int buffer_size);
+static int xenbus_probe_backend(const char *type, const char *domid);
+static struct xen_bus_type xenbus_backend = {
+	.root = "backend",
+	.levels = 3, 		/* backend/type/<frontend>/<id> */
+	.get_bus_id = backend_bus_id,
+	.probe = xenbus_probe_backend,
+	.bus = {
+		.name  = "xen-backend",
+		.match = xenbus_match,
Quoted 8 lines

drivers/xen/xenbus/xenbus_probe.c line 236

Quoted 6 lines
drivers/xen/xenbus/xenbus_probe.c new file +6 -0
+		.uevent = xenbus_uevent_backend,
+	},
+	.dev = {
+		.bus_id = "xen-backend",
+	},
+};

What is the "frontend/backend" relationship here?

do you mean in sysfs? or more in general?

Quoted 29 lines
Quoted 16 lines
drivers/xen/xenbus/xenbus_probe.c new file +16 -0
+static int xenbus_uevent_backend(struct device *dev, char **envp,
+				 int num_envp, char *buffer, int buffer_size)
+{
+	struct xenbus_device *xdev;
+	struct xenbus_driver *drv;
+	int i = 0;
+	int length = 0;
+
+	DPRINTK("");
+
+	if (dev == NULL)
+		return -ENODEV;
+
+	xdev = to_xenbus_device(dev);
+	if (xdev == NULL)
+		return -ENODEV;

drivers/xen/xenbus/xenbus_probe.c line 263

Quoted 10 lines
drivers/xen/xenbus/xenbus_probe.c new file +10 -0
+
+	/* stuff we want to pass to /sbin/hotplug */
+	add_uevent_var(envp, num_envp, &i, buffer, buffer_size, &length,
+		       "XENBUS_TYPE=%s", xdev->devicetype);
+
+	add_uevent_var(envp, num_envp, &i, buffer, buffer_size, &length,
+		       "XENBUS_PATH=%s", xdev->nodename);
+
+	add_uevent_var(envp, num_envp, &i, buffer, buffer_size, &length,
+		       "XENBUS_BASE_PATH=%s", xenbus_backend.root);

Why not use the standard "TYPE" "PATH" and no "XENBUS" stuff?

Quoted 12 lines

drivers/xen/xenbus/xenbus_probe.c line 397

Quoted 8 lines
drivers/xen/xenbus/xenbus_probe.c new file +8 -0
+static int xenbus_register_driver_common(struct xenbus_driver *drv,
+					 struct xen_bus_type *bus)
+{
+	int ret;
+
+	drv->driver.name = drv->name;
+	drv->driver.bus = &bus->bus;
+	drv->driver.owner = drv->owner;

Hint, put the owner in the function, which will force the caller to pass it in. It's forgotten to be set in the structure a lot. Look at USB and PCI register functions now for an example of this.

OK, thanks.

Quoted 9 lines

drivers/xen/xenbus/xenbus_probe.c line 403

Quoted 6 lines
drivers/xen/xenbus/xenbus_probe.c new file +6 -0
+	drv->driver.probe = xenbus_dev_probe;
+	drv->driver.remove = xenbus_dev_remove;
+
+	mutex_lock(&xenwatch_mutex);
+	ret = driver_register(&drv->driver);
+	mutex_unlock(&xenwatch_mutex);

What's with the lock? What is wrong with the driver core lock that is taken? What are you trying to protect?

It's serializing with the xenwatch thread, which is a outside the driver core. I think it may be over eager, and can rely on driver core for normal driver registration, and push it deeper to truly protect the watch lists and registration. I'll add this to the things to investigate more deeply.

Quoted 4 lines
Quoted 4 lines
drivers/xen/xenbus/xenbus_probe.c new file +4 -0
+void xenbus_suspend(void)
+{
+	DPRINTK("");
+
Quoted 8 lines

drivers/xen/xenbus/xenbus_probe.c line 845

Quoted 5 lines
drivers/xen/xenbus/xenbus_probe.c new file +5 -0
+	bus_for_each_dev(&xenbus_frontend.bus, NULL, NULL, suspend_dev);
+	bus_for_each_dev(&xenbus_backend.bus, NULL, NULL, suspend_dev);
+	xs_suspend();
+}
+EXPORT_SYMBOL_GPL(xenbus_suspend);

I think the driver core will handle walking your devices and suspending them. You don't have to do it by hand like this.

Hmm, that would be nice, I'll look into that.

Quoted 11 lines

drivers/xen/xenbus/xenbus_probe.c line 854

Quoted 8 lines
drivers/xen/xenbus/xenbus_probe.c new file +8 -0
+void xenbus_resume(void)
+{
+	xb_init_comms();
+	xs_resume();
+	bus_for_each_dev(&xenbus_frontend.bus, NULL, NULL, resume_dev);
+	bus_for_each_dev(&xenbus_backend.bus, NULL, NULL, resume_dev);
+}
+EXPORT_SYMBOL_GPL(xenbus_resume);

Same thing for resume.

Quoted 6 lines

drivers/xen/xenbus/xenbus_probe.c line 948

Quoted 4 lines
drivers/xen/xenbus/xenbus_probe.c new file +4 -0
+#ifdef XEN_XENBUS_PROC_INTERFACE
+static struct file_operations xsd_kva_fops;
+static struct proc_dir_entry *xsd_kva_intf;
+static struct proc_dir_entry *xsd_port_intf;

#ifdef not needed if your .h file is written correctly.

*nod*

Quoted 10 lines
Quoted 10 lines
drivers/xen/xenbus/xenbus_probe.c new file +10 -0
+static int __init xenbus_probe_init(void)
+{
+	int err = 0, dom0;
+
+	DPRINTK("");
+
+	if (xen_init() < 0) {
+		DPRINTK("failed");
+		return -ENODEV;
+	}
Quoted 8 lines

drivers/xen/xenbus/xenbus_probe.c line 1002

Quoted 6 lines
drivers/xen/xenbus/xenbus_probe.c new file +6 -0
+
+	/* Register ourselves with the kernel bus & device subsystems */
+	bus_register(&xenbus_frontend.bus);
+	bus_register(&xenbus_backend.bus);
+	device_register(&xenbus_frontend.dev);
+	device_register(&xenbus_backend.dev);

No error handling?

will fix

Quoted 9 lines

drivers/xen/xenbus/xenbus_probe.c line 1009

Quoted 7 lines
drivers/xen/xenbus/xenbus_probe.c new file +7 -0
+
+	/*
+	 * Domain0 doesn't have a store_evtchn or store_mfn yet.
+	 */
+	dom0 = (xen_start_info->store_evtchn == 0);
+
+#ifdef XEN_XENBUS_PROC_INTERFACE

No #ifdef needed if your .h file is written correctly.

*nod*

Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+/* xenbus_probe.c */
+extern char *kasprintf(const char *fmt, ...);

Should be in a .h file

yup, same as above.

Quoted 10 lines

drivers/xen/xenbus/xenbus_xs.c line 101

Quoted 8 lines
drivers/xen/xenbus/xenbus_xs.c new file +8 -0
+/*
+ * Details of the xenwatch callback kernel thread. The thread waits on the
+ * watch_events_waitq for work to do (queued on watch_events list). When it
+ * wakes up it acquires the xenwatch_mutex before reading the list and
+ * carrying out work.
+ */
+static pid_t xenwatch_pid;
+/* static */ DEFINE_MUTEX(xenwatch_mutex);

Drop the static comment?

yes

Quoted 28 lines
Quoted 6 lines
drivers/xen/xenbus/xenbus_xs.c new file +6 -0
+/* Create a new directory. */
+int xenbus_mkdir(xenbus_transaction_t t,
+		 const char *dir, const char *node)
+{
+	char *path;
+	int ret;

drivers/xen/xenbus/xenbus_xs.c line 415

Quoted 10 lines
drivers/xen/xenbus/xenbus_xs.c new file +10 -0
+
+	path = join(dir, node);
+	if (IS_ERR(path))
+		return PTR_ERR(path);
+
+	ret = xs_error(xs_single(t, XS_MKDIR, path, NULL));
+	kfree(path);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(xenbus_mkdir);

Create a new directory in what? sysfs?

Quoted 9 lines
drivers/xen/xenbus/xenbus_xs.c new file +9 -0
+/* Destroy a file or directory (directories must be empty). */
+int xenbus_rm(xenbus_transaction_t t, const char *dir, const char *node)
+{
+	char *path;
+	int ret;
+
+	path = join(dir, node);
+	if (IS_ERR(path))
+		return PTR_ERR(path);
Quoted 8 lines

drivers/xen/xenbus/xenbus_xs.c line 431

Quoted 6 lines
drivers/xen/xenbus/xenbus_xs.c new file +6 -0
+
+	ret = xs_error(xs_single(t, XS_RM, path, NULL));
+	kfree(path);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(xenbus_rm);

Remove a file or directory in what? sysfs?

These are in xenstore. It's pretty much internal to xen. Deserves a clearer (kerneldoc) comment.

Quoted 3 lines
Quoted 1 line

+#define XBT_NULL 0

What is this for? What's wrong with just "NULL"?

It's special transaction ID. Obfuscated by both the name, and the typedef.

Quoted 5 lines
Quoted 5 lines
include/xen/xenbus.h new file +5 -0
+/* A xenbus device. */
+struct xenbus_device {
+	const char *devicetype;
+	const char *nodename;
+	const char *otherend;
Quoted 8 lines

include/xen/xenbus.h line 70

Quoted 5 lines
include/xen/xenbus.h new file +5 -0
+	int otherend_id;
+	struct xenbus_watch otherend_watch;
+	struct device dev;
+	XenbusState state;
+	void *data;

The data field can be dropped. Use the space in the struct device for this.

At quick glance I agree. Will give that a try.

Quoted 3 lines
Quoted 1 line

+typedef u32 xenbus_transaction_t;

Why the typedef? Please don't.

To see if you were reading? ;-) The typedef elimination is ongoing, will add that as well.

thanks,
-chris

#138 lore raw

Re: [RFC PATCH 33/35] Add the Xenbus sysfs and virtual device hotplug driver.

Quoted 32 lines · Chris Wright
Quoted 4 lines · * Greg KH ([email protected]) wrote:

And what are you doing with the different "levels"? Is there some description of how you are using sysfs for this? Busses should not be "nested", devices should. How does sysfs look with this code in it? What is the /sys/bus/ structure? What is the /sys/devices/ structure?

e.g.

/sys/bus/xen

|-- devices
|   |-- vbd-51713 -> ../../../devices/xen/vbd-51713
|   `-- vif-0 -> ../../../devices/xen/vif-0
`-- drivers
    `-- vbd
        |-- bind
        |-- unbind
        `-- vbd-51713 -> ../../../../devices/xen/vbd-51713

/sys/devices/xen

|-- uevent
|-- vbd-51713
|   |-- block:xvda1 -> ../../../block/xvda1
|   |-- bus -> ../../../bus/xen
|   |-- devtype
|   |-- driver -> ../../../bus/xen/drivers/vbd
|   |-- nodename
|   `-- uevent
`-- vif-0
    |-- bus -> ../../../bus/xen
    |-- devtype
    |-- nodename
    `-- uevent

<snip>

Quoted 3 lines
Quoted 1 line

What is the "frontend/backend" relationship here?

do you mean in sysfs? or more in general?

Either. You seem to mention a lot of nested depths in sysfs or "files", yet your above tree doesn't show that. And I don't understand what you mean by frontend/backend here either? Is it a sysfs thing? Or a Xen thing?

thanks,

greg k-h

#139 lore raw

Re: [RFC PATCH 33/35] Add the Xenbus sysfs and virtual device hotplug driver.

Quoted 4 lines · * Greg KH ([email protected]) wrote:

Either. You seem to mention a lot of nested depths in sysfs or "files", yet your above tree doesn't show that. And I don't understand what you mean by frontend/backend here either? Is it a sysfs thing? Or a Xen thing?

The files are xenstore, it's part of the communication between frontend and backend. The frontend is the device driver in the guest domain which is just an I/O channel to the backend driver. The backend is in the driver domain where the physical hardware can be driven.

thanks,
-chris

#140 lore raw

Re: [RFC PATCH 33/35] Add the Xenbus sysfs and virtual device hotplug driver.

In reply to Greg KH
Quoted 4 lines · Greg KH
Quoted 2 lines · Chris Wright
Quoted 2 lines

What is the "frontend/backend" relationship here?

Quoted 8 lines
Quoted 2 lines

do you mean in sysfs? or more in general?

Either. You seem to mention a lot of nested depths in sysfs or "files", yet your above tree doesn't show that. And I don't understand what you mean by frontend/backend here either? Is it a sysfs thing? Or a Xen thing?

Hi Greg,

XenStore is a shared namespace (similar to sysfs or open firmware) between domains. The interdomain communication primitives exposed by Xen are very lowlevel (virtual IRQ and shared memory). XenStore is implemented on top of these primitives and provides some higher level operations (read a key, write a key, enumerate a directory, notify when a key changes value).

We use XenStore to implement our virtual drivers (this infrastructure is called XenBus). The drivers are split between a backend and frontend. The frontend is the portion of the driver that runs in the guest and the backend is the portion of the driver that runs in the host (and actually virtualizes the underlying device).

The xenbus_mkdir, etc. functions you see operate on XenStore.

Regards,

Anthony Liguori

Quoted 11 lines

thanks,

greg k-h

------------------------------------------------------------------------

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.osdl.org/mailman/listinfo/virtualization

#141 lore raw

Re: [RFC PATCH 33/35] Add the Xenbus sysfs and virtual device hotplug driver.

In reply to Chris Wright

drivers/xen/xenbus/xenbus_probe.c line 52

Quoted 3 lines · Chris Wright
drivers/xen/xenbus/xenbus_probe.c new file +3 -0
+#ifdef XEN_XENBUS_PROC_INTERFACE
+#include <xen/xen_proc.h>
+#endif

Oh, you all never define this anywhere in the series, so anything protected by it should be removed.

And I sure hope you don't have a xen_proc.h file anywhere, we do not need any new non-process files going into /proc...

thanks,

greg k-h

#142 lore raw

Re: [RFC PATCH 33/35] Add the Xenbus sysfs and virtual device hotplug driver.

Quoted 2 lines · * Greg KH ([email protected]) wrote:

And I sure hope you don't have a xen_proc.h file anywhere, we do not need any new non-process files going into /proc...

I'll be happy once we've got all the /proc abuse eliminated, sorry some of this snuck through.

thanks,
-chris

#143 lore raw

Re: [RFC PATCH 33/35] Add the Xenbus sysfs and virtual device hotplug driver.

In reply to Chris Wright

drivers/xen/xenbus/xenbus_client.c line 135

Quoted 11 lines · On Tue, 09 May 2006 00:00:33 -0700 Chris Wright <[email protected]> wrote:

This communicates with the machine control software via a registry residing in a controlling virtual machine. This allows dynamic creation, destruction and modification of virtual device configurations (network devices, block devices and CPUS, to name some examples).

...

drivers/xen/xenbus/xenbus_client.c new file +2 -0
+void _dev_error(struct xenbus_device *dev, int err, const char *fmt,
+		va_list ap)

I don't think this needs global scope? (hopefully not, with that name..)

drivers/xen/xenbus/xenbus_client.c line 139

Quoted 3 lines
drivers/xen/xenbus/xenbus_client.c new file +3 -0
+	int ret;
+	unsigned int len;
+	char *printf_buffer = NULL, *path_buffer = NULL;
	char *print_buffer;
	char *path_buffer = NULL;

Review

Quoted 2 lines
(quoted diff) +2 -0
+#define PRINTF_BUFFER_SIZE 4096
+	printf_buffer = kmalloc(PRINTF_BUFFER_SIZE, GFP_KERNEL);

Assuming that GFP_KERNEL is legal in this context seems like a bad idea.

<looks>

hm, it does that all over the place, so I guess it works.

drivers/xen/xenbus/xenbus_client.c line 330

Quoted 10 lines
drivers/xen/xenbus/xenbus_client.c new file +10 -0
+/* Based on Rusty Russell's skeleton driver's unmap_page */
+int xenbus_unmap_ring_vfree(struct xenbus_device *dev, void *vaddr)
+{
+	struct vm_struct *area;
+	struct gnttab_unmap_grant_ref op = {
+		.host_addr = (unsigned long)vaddr,
+	};
+
+	/* It'd be nice if linux/vmalloc.h provided a find_vm_area(void *addr)
+	 * method so that we don't have to muck with vmalloc internals here.

We take patches ;)

But then, perhaps the requirement doesn't make a lot of sense in a multithreaded environment. We don't refcount the entries on vmlist, so there's no point in being able to look them up. Instead, the calling code is supposed to be able to keep track of its own state.

Which begs the question: why isn't this code able to do that thing?

Quoted 6 lines
drivers/xen/xenbus/xenbus_client.c new file +6 -0
+	 * We could force the user to hang on to their struct vm_struct from
+	 * xenbus_map_ring_valloc, but these 6 lines considerably simplify
+	 * this API.
+	 */
+	read_lock(&vmlist_lock);
+	for (area = vmlist; area != NULL; area = area->next) {

drivers/xen/xenbus/xenbus_client.c line 346

Quoted 10 lines
drivers/xen/xenbus/xenbus_client.c new file +10 -0
+		if (area->addr == vaddr)
+			break;
+	}
+	read_unlock(&vmlist_lock);
+
+	if (!area) {
+		xenbus_dev_error(dev, -ENOENT,
+				 "can't find mapped virtual address %p", vaddr);
+		return GNTST_bad_virt_addr;
+	}

One assumes there's some locking hereabouts which ensures that `area' is still on that list after vmlist_lock got dropped?

drivers/xen/xenbus/xenbus_comms.c line 79

Quoted 10 lines
drivers/xen/xenbus/xenbus_comms.c new file +10 -0
+
+static void *get_output_chunk(XENSTORE_RING_IDX cons,
+			      XENSTORE_RING_IDX prod,
+			      char *buf, uint32_t *len)
+{
+	*len = XENSTORE_RING_SIZE - MASK_XENSTORE_IDX(prod);
+	if ((XENSTORE_RING_SIZE - (prod - cons)) < *len)
+		*len = XENSTORE_RING_SIZE - (prod - cons);
+	return buf + MASK_XENSTORE_IDX(prod);
+}

Another open-coded ringbuffer? Am still seeking the user of the interesting ring.h.

#144 lore raw

RFC PATCH 34/35 Add the Xen virtual network device driver.

In reply to Chris Wright

The network device frontend driver allows the kernel to access network devices exported exported by a virtual machine containing a physical network device driver.

---
TODO:
- drop proc
- more ethtool ops
- s/g support

 drivers/net/Kconfig             |    2 
 drivers/xen/Kconfig.net         |   14 
 drivers/xen/Makefile            |    3 
 drivers/xen/net_driver_util.c   |   58 +
 drivers/xen/netfront/Makefile   |    4 
 drivers/xen/netfront/netfront.c | 1510 ++++++++++++++++++++++++++++++++++++++++
 include/xen/net_driver_util.h   |   48 +
 7 files changed, 1639 insertions(+)
drivers/net/Kconfig +2 -0
@@ -2325,6 +2325,8 @@ source "drivers/atm/Kconfig"
 
 source "drivers/s390/net/Kconfig"
 
+source "drivers/xen/Kconfig.net"
+
 config ISERIES_VETH
 	tristate "iSeries Virtual Ethernet driver support"
 	depends on PPC_ISERIES
drivers/xen/Makefile +3 -0
@@ -1,7 +1,10 @@
 
+obj-y	+= net_driver_util.o
 obj-y	+= util.o
 
 obj-y	+= core/
 obj-y	+= console/
 obj-y	+= xenbus/
 
+obj-$(CONFIG_XEN_NETDEV_FRONTEND)	+= netfront/
+
drivers/xen/Kconfig.net new file +14 -0
@@ -0,0 +1,14 @@
+menu "Xen network device drivers"
+        depends on NETDEVICES && XEN
+
+config XEN_NETDEV_FRONTEND
+	tristate "Network-device frontend driver"
+	depends on XEN
+	default y
+	help
+	  The network-device frontend driver allows the kernel to access
+	  network interfaces within another guest OS. Unless you are building a
+	  dedicated device-driver domain, or your master control domain
+	  (domain 0), then you almost certainly want to say Y here.
+
+endmenu
drivers/xen/net_driver_util.c new file +58 -0

58 lines not on this page · raw message

drivers/xen/netfront/Makefile new file +4 -0
@@ -0,0 +1,4 @@
+
+obj-$(CONFIG_XEN_NETDEV_FRONTEND)	:= xennet.o
+
+xennet-objs := netfront.o
drivers/xen/netfront/netfront.c new file +1510 -0

1,510 lines not on this page · raw message

include/xen/net_driver_util.h new file +48 -0

48 lines not on this page · raw message

--

#145 lore raw

Re: [Xen-devel] [RFC PATCH 34/35] Add the Xen virtual network device driver.

Hi Chris:

drivers/xen/netfront/netfront.c line 427

Quoted 8 lines · Chris Wright
drivers/xen/netfront/netfront.c new file +7 -0
+/** Send a packet on a net device to encourage switches to learn the
+ * MAC. We send a fake ARP request.
+ *
+ * @param dev device
+ * @return 0 on success, error code otherwise
+ */
+static int send_fake_arp(struct net_device *dev)

I think we talked about this before. I don't see why Xen is so special that it needs its own gratuitous arp routine embedded in the driver. If this is needed at all (presumably for migration) then it should be performed by the management scripts which can send grat ARP packets just as easily.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#146 lore raw

Re: [Xen-devel] [RFC PATCH 34/35] Add the Xen virtual network device driver.

Quoted 4 lines · Herbert Xu

Hi Chris:

Quoted 1 line · Chris Wright
Quoted 13 lines

drivers/xen/netfront/netfront.c line 427

Quoted 7 lines
drivers/xen/netfront/netfront.c new file +7 -0
+/** Send a packet on a net device to encourage switches to learn the
+ * MAC. We send a fake ARP request.
+ *
+ * @param dev device
+ * @return 0 on success, error code otherwise
+ */
+static int send_fake_arp(struct net_device *dev)

I think we talked about this before. I don't see why Xen is so special that it needs its own gratuitous arp routine embedded in the driver. If this is needed at all (presumably for migration) then it should be performed by the management scripts which can send grat ARP packets just as easily.

There's at least two reasons why having it in the driver is preferable:
- synchronizing sending the fake ARP request with when the device is operational -- you really want to make this well synchronized to keep unreachability as short as possible, especially when doing live migration
- anybody but the guest might not know (all) the MAC addresses for which to send a fake ARP request

christian

#147 lore raw

Re: [Xen-devel] [RFC PATCH 34/35] Add the Xen virtual network device driver.

Quoted 8 lines · Christian Limpach

There's at least two reasons why having it in the driver is preferable:
- synchronizing sending the fake ARP request with when the device is operational -- you really want to make this well synchronized to keep unreachability as short as possible, especially when doing live migration
- anybody but the guest might not know (all) the MAC addresses for which to send a fake ARP request

Sure. However, what's there to stop you from doing this in user-space inside the guest?

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#148 lore raw

Re: [Xen-devel] [RFC PATCH 34/35] Add the Xen virtual network device driver.

Quoted 10 lines · Herbert Xu
Quoted 8 lines · Christian Limpach

There's at least two reasons why having it in the driver is preferable:
- synchronizing sending the fake ARP request with when the device is operational -- you really want to make this well synchronized to keep unreachability as short as possible, especially when doing live migration
- anybody but the guest might not know (all) the MAC addresses for which to send a fake ARP request

Quoted 2 lines

Sure. However, what's there to stop you from doing this in user-space inside the guest?

I guess they don't trust user space. But the standard ipup script from iproute2 does this already so at least for modern distributions it should just work.

-Andi

#149 lore raw

Re: [Xen-devel] [RFC PATCH 34/35] Add the Xen virtual network device driver.

In reply to Herbert Xu
Quoted 10 lines · Herbert Xu
Quoted 8 lines · Christian Limpach

There's at least two reasons why having it in the driver is preferable:
- synchronizing sending the fake ARP request with when the device is operational -- you really want to make this well synchronized to keep unreachability as short as possible, especially when doing live migration
- anybody but the guest might not know (all) the MAC addresses for which to send a fake ARP request

Quoted 2 lines

Sure. However, what's there to stop you from doing this in user-space inside the guest?

Possibly having to page in the process and switching to it would add to the live migration time. More importantly, having to install an additional program in the guest is certainly not very convenient.

christian

#150 lore raw

Re: [Xen-devel] [RFC PATCH 34/35] Add the Xen virtual network device driver.

Quoted 4 lines · Christian Limpach

Possibly having to page in the process and switching to it would add to the live migration time. More importantly, having to install an additional program in the guest is certainly not very convenient.

Sorry I'm still not convinced. What's there to stop me from suspending my laptop to disk, moving it from port A to port B and resuming it?

Wouldn't I be in exactly the same situation? By the same reasoning we'd be adding a gratuitous ARP routine to every single laptop network driver.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#151 lore raw

Re: [Xen-devel] [RFC PATCH 34/35] Add the Xen virtual network device driver.

Quoted 11 lines · Herbert Xu
Quoted 4 lines · Christian Limpach

Possibly having to page in the process and switching to it would add to the live migration time. More importantly, having to install an additional program in the guest is certainly not very convenient.

Sorry I'm still not convinced. What's there to stop me from suspending my laptop to disk, moving it from port A to port B and resuming it?

Wouldn't I be in exactly the same situation? By the same reasoning we'd be adding a gratuitous ARP routine to every single laptop network driver.

It is the same situation except that in the laptop case you don't care that reconfiguring your network will take a second or a few. For live migration we're looking at network downtime from as low as 60ms to something like 210ms on a busy virtual machine. I'm not saying that a userspace solution wouldn't work but it would probably add a measurable delay to the network downtime during live migration.

You might also find the following paper an interesting read:
http://www.cl.cam.ac.uk/Research/SRG/netos/papers/2005-migration-nsdi-pre.pdf

christian

#152 lore raw

Re: [Xen-devel] [RFC PATCH 34/35] Add the Xen virtual network device driver.

[email protected] wrote on 05/09/2006 09:00:27 AM:

Quoted 8 lines · Herbert Xu
Quoted 7 lines
Quoted 4 lines · Christian Limpach

Possibly having to page in the process and switching to it would add to the live migration time. More importantly, having to install an additional program in the guest is certainly not very convenient.

Sorry I'm still not convinced. What's there to stop me from

suspending

Quoted 3 lines
Quoted 3 lines

my laptop to disk, moving it from port A to port B and resuming it?

Wouldn't I be in exactly the same situation? By the same reasoning

we'd

Quoted 1 line
Quoted 1 line

be adding a gratuitous ARP routine to every single laptop network

driver.

Quoted 6 lines

It is the same situation except that in the laptop case you don't care that reconfiguring your network will take a second or a few. For live migration we're looking at network downtime from as low as 60ms to something like 210ms on a busy virtual machine. I'm not saying that a userspace solution wouldn't work but it would probably add a

measurable

Quoted 1 line

delay to the network downtime during live migration.

Then make a generic solution. VMWare supports migration, the Power virtualization will get around to it eventually. All will need something similar. So either make a common user-land tool, or (if you insist on incorrectly driving this into the kernel) add some kind of common hook to the TCP/IP stack.

Dave Boutcher
IBM Linux Technology Center

#153 lore raw

Re: [Xen-devel] [RFC PATCH 34/35] Add the Xen virtual network device driver.

Quoted 5 lines · * David Boutcher ([email protected]) wrote:

Then make a generic solution. VMWare supports migration, the Power virtualization will get around to it eventually. All will need something similar. So either make a common user-land tool, or (if you insist on incorrectly driving this into the kernel) add some kind of common hook to the TCP/IP stack.

I'm not that fond of the in-kernel solution either. HA failover does this stuff in userspace, and has the same gratuitous arp requirements. Perhaps we should see some numbers showing the migration latency introduced. At the very least, it's easy to factor out as suggested.

thanks,
-chris

#154 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

In reply to Chris Wright
Quoted 3 lines · Chris Wright

The network device frontend driver allows the kernel to access network devices exported exported by a virtual machine containing a physical network device driver.

Please don't add procfs code to new network drivers. Especially if it's oopsable like the code in this driver by simple device renaming.

#155 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

Quoted 7 lines · * Christoph Hellwig ([email protected]) wrote:
Quoted 3 lines · Chris Wright

The network device frontend driver allows the kernel to access network devices exported exported by a virtual machine containing a physical network device driver.

Please don't add procfs code to new network drivers. Especially if it's oopsable like the code in this driver by simple device renaming.

Agreed, no reason to keep the cruft around. I thought I had a comment of the sort in there.

thanks,
-chris

#156 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

In reply to Chris Wright

The stuff in /proc could easily just be added attributes to the class_device kobject of the net device (and then show up in sysfs).

drivers/xen/netfront/netfront.c line 79

Quoted 13 lines
drivers/xen/netfront/netfront.c new file +13 -0
+
+#define GRANT_INVALID_REF	0
+
+#define NET_TX_RING_SIZE __RING_SIZE((struct netif_tx_sring *)0, PAGE_SIZE)
+#define NET_RX_RING_SIZE __RING_SIZE((struct netif_rx_sring *)0, PAGE_SIZE)
+
+static inline void init_skb_shinfo(struct sk_buff *skb)
+{
+	atomic_set(&(skb_shinfo(skb)->dataref), 1);
+	skb_shinfo(skb)->nr_frags = 0;
+	skb_shinfo(skb)->frag_list = NULL;
+}
+

Could you use existing sk_buff_head instead of inventing your own skb queue?

Quoted 43 lines
drivers/xen/netfront/netfront.c new file +43 -0
+struct netfront_info
+{
+	struct list_head list;
+	struct net_device *netdev;
+
+	struct net_device_stats stats;
+	unsigned int tx_full;
+
+	struct netif_tx_front_ring tx;
+	struct netif_rx_front_ring rx;
+
+	spinlock_t   tx_lock;
+	spinlock_t   rx_lock;
+
+	unsigned int handle;
+	unsigned int evtchn, irq;
+
+	/* What is the status of our connection to the remote backend? */
+#define BEST_CLOSED       0
+#define BEST_DISCONNECTED 1
+#define BEST_CONNECTED    2
+	unsigned int backend_state;
+
+	/* Is this interface open or closed (down or up)? */
+#define UST_CLOSED        0
+#define UST_OPEN          1
+	unsigned int user_state;
+
+	/* Receive-ring batched refills. */
+#define RX_MIN_TARGET 8
+#define RX_DFL_MIN_TARGET 64
+#define RX_MAX_TARGET NET_RX_RING_SIZE
+	int rx_min_target, rx_max_target, rx_target;
+	struct sk_buff_head rx_batch;
+
+	struct timer_list rx_refill_timer;
+
+	/*
+	 * {tx,rx}_skbs store outstanding skbuffs. The first entry in each
+	 * array is an index into a chain of free entries.
+	 */
+	struct sk_buff *tx_skbs[NET_TX_RING_SIZE+1];
+	struct sk_buff *rx_skbs[NET_RX_RING_SIZE+1];

drivers/xen/netfront/netfront.c line 132

Quoted 10 lines
drivers/xen/netfront/netfront.c new file +10 -0
+
+	grant_ref_t gref_tx_head;
+	grant_ref_t grant_tx_ref[NET_TX_RING_SIZE + 1];
+	grant_ref_t gref_rx_head;
+	grant_ref_t grant_rx_ref[NET_TX_RING_SIZE + 1];
+
+	struct xenbus_device *xbdev;
+	int tx_ring_ref;
+	int rx_ring_ref;
+	u8 mac[ETH_ALEN];

Isn't mac address already stored in dev->dev_addr and/or dev->perm_addr?

#157 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

Quoted 2 lines · * Stephen Hemminger ([email protected]) wrote:

The stuff in /proc could easily just be added attributes to the class_device kobject of the net device (and then show up in sysfs).

Agreed, it's on the todo list to drop proc support there. Thought that was marked in the patch.

Quoted 6 lines
Quoted 6 lines
drivers/xen/netfront/netfront.c new file +6 -0
+#define GRANT_INVALID_REF	0
+
+#define NET_TX_RING_SIZE __RING_SIZE((struct netif_tx_sring *)0, PAGE_SIZE)
+#define NET_RX_RING_SIZE __RING_SIZE((struct netif_rx_sring *)0, PAGE_SIZE)
+
+static inline void init_skb_shinfo(struct sk_buff *skb)
Quoted 8 lines

drivers/xen/netfront/netfront.c line 78

Quoted 5 lines
drivers/xen/netfront/netfront.c new file +5 -0
+{
+	atomic_set(&(skb_shinfo(skb)->dataref), 1);
+	skb_shinfo(skb)->nr_frags = 0;
+	skb_shinfo(skb)->frag_list = NULL;
+}

Could you use existing sk_buff_head instead of inventing your own skb queue?

Hmm, there is some standard skb_queue_tail happening. I don't have a clear idea what you mean.

Quoted 3 lines

drivers/xen/netfront/netfront.c line 132

Quoted 1 line
drivers/xen/netfront/netfront.c new file +1 -0
+	u8 mac[ETH_ALEN];

Isn't mac address already stored in dev->dev_addr and/or dev->perm_addr?

Yes, I don't see the reason to keep in twice. It's basically a temp buffer, but it certainly appears we can eliminate it.

thanks,
-chris

#158 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

In reply to Chris Wright
Quoted 54 lines

+static int setup_device(struct xenbus_device *dev, struct netfront_info *info) +{

drivers/xen/netfront/netfront.c new file +15 -0
+	struct netif_tx_sring *txs;
+	struct netif_rx_sring *rxs;
+	int err;
+	struct net_device *netdev = info->netdev;
+
+	info->tx_ring_ref = GRANT_INVALID_REF;
+	info->rx_ring_ref = GRANT_INVALID_REF;
+	info->rx.sring = NULL;
+	info->tx.sring = NULL;
+	info->irq = 0;
+
+	txs = (struct netif_tx_sring *)get_zeroed_page(GFP_KERNEL);
+	if (!txs) {
+		err = -ENOMEM;
+		xenbus_dev_fatal(dev, err, "allocating tx ring

page");

drivers/xen/netfront/netfront.c new file +6 -0
+		goto fail;
+	}
+	rxs = (struct netif_rx_sring *)get_zeroed_page(GFP_KERNEL);
+	if (!rxs) {
+		err = -ENOMEM;
+		xenbus_dev_fatal(dev, err, "allocating rx ring

page");

drivers/xen/netfront/netfront.c new file +29 -0
+		free_page((unsigned long)txs);
+		goto fail;
+	}
+	info->backend_state = BEST_DISCONNECTED;
+
+	SHARED_RING_INIT(txs);
+	FRONT_RING_INIT(&info->tx, txs, PAGE_SIZE);
+
+	SHARED_RING_INIT(rxs);
+	FRONT_RING_INIT(&info->rx, rxs, PAGE_SIZE);
+
+	err = xenbus_grant_ring(dev, virt_to_mfn(txs));
+	if (err < 0)
+		goto fail;
+	info->tx_ring_ref = err;
+
+	err = xenbus_grant_ring(dev, virt_to_mfn(rxs));
+	if (err < 0)
+		goto fail;
+	info->rx_ring_ref = err;
+
+	err = xenbus_alloc_evtchn(dev, &info->evtchn);
+	if (err)
+		goto fail;
+
+	memcpy(netdev->dev_addr, info->mac, ETH_ALEN);
+	network_connect(netdev);
+	info->irq = bind_evtchn_to_irqhandler(
+		info->evtchn, netif_int, SA_SAMPLE_RANDOM,
Quoted 2 lines

netdev->name,

This doesn't look like a real random entropy source. packets arriving from another domain are easily timed.

#159 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

Quoted 9 lines · Stephen Hemminger
Quoted 6 lines
drivers/xen/netfront/netfront.c new file +4 -0
+	memcpy(netdev->dev_addr, info->mac, ETH_ALEN);
+	network_connect(netdev);
+	info->irq = bind_evtchn_to_irqhandler(
+		info->evtchn, netif_int, SA_SAMPLE_RANDOM,

netdev->name,

This doesn't look like a real random entropy source. packets arriving from another domain are easily timed.

Where should we get our entropy from in a VM environment? Leaving the pool empty can cause processes to hang.

-- Keir

#160 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

Quoted 13 lines · On Tue, 9 May 2006 21:26:11 +0100 Keir Fraser <[email protected]> wrote:
Quoted 9 lines · Stephen Hemminger
Quoted 6 lines
drivers/xen/netfront/netfront.c new file +4 -0
+	memcpy(netdev->dev_addr, info->mac, ETH_ALEN);
+	network_connect(netdev);
+	info->irq = bind_evtchn_to_irqhandler(
+		info->evtchn, netif_int, SA_SAMPLE_RANDOM,

netdev->name,

This doesn't look like a real random entropy source. packets arriving from another domain are easily timed.

Quoted 3 lines

Where should we get our entropy from in a VM environment? Leaving the pool empty can cause processes to hang.

You probably need to get entropy from dom0 and real hardware sources. Could you piggyback on some other perodic polling/message passing to push some entropy out?

#161 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

In reply to Keir Fraser

Keir> Where should we get our entropy from in a VM environment? Keir> Leaving the pool empty can cause processes to hang.

You could have something like a virtual HW RNG driver (with a frontend and backend), which steals from the dom0 /dev/random pool.

- R.

#162 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

Quoted 5 lines · Roland Dreier

Keir> Where should we get our entropy from in a VM environment? Keir> Leaving the pool empty can cause processes to hang.

You could have something like a virtual HW RNG driver (with a frontend and backend), which steals from the dom0 /dev/random pool.

They already have a vTPM - iirc TPMs support random numbers so that could be used. But it's probably complicated to use.

But if sampling virtual events for randomness is really unsafe (is it really?) then native guests in Xen would also get bad random numbers and this would need to be somehow addressed.

I haven't seen real evidence yet why the virtual events should provide less randomness than the hardware.

-And

#163 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

Quoted 4 lines · Andi Kleen

But if sampling virtual events for randomness is really unsafe (is it really?) then native guests in Xen would also get bad random numbers and this would need to be somehow addressed.

Good point. I wonder what VMWare does in this situation.

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#164 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

Quoted 5 lines · Herbert Xu
Quoted 3 lines

But if sampling virtual events for randomness is really unsafe (is it really?) then native guests in Xen would also get bad random numbers and this would need to be somehow addressed.

Good point. I wonder what VMWare does in this situation.

Well, there's not much they can do except maybe jitter interrupt delivery. I doubt they do that though.

The original complaint in our case was that we take entropy from interrupts caused by other local VMs, as well as external sources. There was a feeling that the former was more predictable and could form the basis of an attack. I have to say I'm unconvinced: I don't really see that it's significantly easier to inject precisely-timed interrupts into a local VM. Certainly not to better than +/- a few microseconds. As long as you add cycle-counter info to the entropy pool, the least significant bits of that will always be noise.

The alternatives are unattractive:
1. We have no good way to distinguish interrupts caused by packets
from local VMs versus packets from remote hosts. Both get muxed on the same virtual interface.
2. An entropy front/back is tricky -- how do we decide how much
entropy to pull from domain0? How much should domain0 be prepared to give other domains? How easy is it to DoS domain0 by draining its entropy pool? Yuk.

-- Keir

#165 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

Quoted 9 lines · Keir Fraser

The alternatives are unattractive:
1. We have no good way to distinguish interrupts caused by packets
from local VMs versus packets from remote hosts. Both get muxed on the same virtual interface.
2. An entropy front/back is tricky -- how do we decide how much
entropy to pull from domain0? How much should domain0 be prepared to give other domains? How easy is it to DoS domain0 by draining its entropy pool? Yuk.

IMHO there just isn't enough real entropy to go around in one physical machine without a proper HRNG. So either use urandom in all the guests or for those that really have to use /dev/random, install a hardware RNG (or wait for it :).

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#166 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

In reply to Keir Fraser
Quoted 10 lines · Keir Fraser
Quoted 5 lines · Herbert Xu
Quoted 3 lines

But if sampling virtual events for randomness is really unsafe (is it really?) then native guests in Xen would also get bad random numbers and this would need to be somehow addressed.

Good point. I wonder what VMWare does in this situation.

Well, there's not much they can do except maybe jitter interrupt delivery. I doubt they do that though.

Quoted 8 lines

The original complaint in our case was that we take entropy from interrupts caused by other local VMs, as well as external sources. There was a feeling that the former was more predictable and could form the basis of an attack. I have to say I'm unconvinced: I don't really see that it's significantly easier to inject precisely-timed interrupts into a local VM. Certainly not to better than +/- a few microseconds. As long as you add cycle-counter info to the entropy pool, the least significant bits of that will always be noise.

I think I agree - e.g. i would expect the virtual interrupts to have enough jitter too. Maybe it would be good if someone could run a few statistics on the resulting numbers?

Ok the randomness added doesn't consist only of the least significant bits. Currently it adds jiffies+full 32bit cycle count. I guess if it was a real problem the code could be changed to leave out the jiffies and only add maybe a 8 bit word from the low bits. But that would only help for the para case because the algorithm for native guests cannot be changed.

Quoted 4 lines

2. An entropy front/back is tricky -- how do we decide how much
entropy to pull from domain0? How much should domain0 be prepared to give other domains? How easy is it to DoS domain0 by draining its entropy pool? Yuk.

I claim (without having read any code) that in theory you need to have solved that problem already in the vTPM @)

-Andi

#167 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

Quoted 31 lines · On Thu, 11 May 2006 11:47:52 +0200 Andi Kleen <[email protected]> wrote:
Quoted 10 lines · Keir Fraser
Quoted 5 lines · Herbert Xu
Quoted 3 lines

But if sampling virtual events for randomness is really unsafe (is it really?) then native guests in Xen would also get bad random numbers and this would need to be somehow addressed.

Good point. I wonder what VMWare does in this situation.

Well, there's not much they can do except maybe jitter interrupt delivery. I doubt they do that though.

Quoted 8 lines

The original complaint in our case was that we take entropy from interrupts caused by other local VMs, as well as external sources. There was a feeling that the former was more predictable and could form the basis of an attack. I have to say I'm unconvinced: I don't really see that it's significantly easier to inject precisely-timed interrupts into a local VM. Certainly not to better than +/- a few microseconds. As long as you add cycle-counter info to the entropy pool, the least significant bits of that will always be noise.

I think I agree - e.g. i would expect the virtual interrupts to have enough jitter too. Maybe it would be good if someone could run a few statistics on the resulting numbers?

Ok the randomness added doesn't consist only of the least significant bits. Currently it adds jiffies+full 32bit cycle count. I guess if it was a real problem the code could be changed to leave out the jiffies and only add maybe a 8 bit word from the low bits. But that would only help for the para case because the algorithm for native guests cannot be changed.

Quoted 8 lines
Quoted 4 lines

2. An entropy front/back is tricky -- how do we decide how much
entropy to pull from domain0? How much should domain0 be prepared to give other domains? How easy is it to DoS domain0 by draining its entropy pool? Yuk.

I claim (without having read any code) that in theory you need to have solved that problem already in the vTPM @)

The base question under all this is "how good does an entropy source have to be?" and then "what guarantees do we make about the entropy inputs used by /dev/random?". If we can resolve those, then the virtual environment answer should fall out.

This is a area where the security tin-foil hat types take over, and it gets real hard to make "good enough" argument. People have built an expectation that /dev/random has really strong entropy, good enough to generate long term keys etc.

#168 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

In reply to Andi Kleen

From the peanut gallery...

Can remote TCP ISN's be considered a source of entropy these days? How about checksums?

rick

#169 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

Quoted 6 lines · On Thu, 11 May 2006 09:48:11 -0700 Rick Jones <[email protected]> wrote:

From the peanut gallery...

Can remote TCP ISN's be considered a source of entropy these days? How about checksums?

rick

No, they are spoofable.

#170 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

In reply to Rick Jones
Quoted 4 lines · Rick Jones

From the peanut gallery...

Can remote TCP ISN's be considered a source of entropy these days? How about checksums?

Indirectly - we measure how long it takes to compute them.

-Andi

#171 lore raw

Re: [RFC PATCH 34/35] Add the Xen virtual network device driver.

In reply to Stephen Hemminger
Quoted 6 lines · * Stephen Hemminger ([email protected]) wrote:
Quoted 3 lines
(quoted diff) +2 -0
+	info->irq = bind_evtchn_to_irqhandler(
+		info->evtchn, netif_int, SA_SAMPLE_RANDOM,

netdev->name,

This doesn't look like a real random entropy source. packets arriving from another domain are easily timed.

Heh, given the path they take, that sadly may not be the case ;-) But point well-taken, that's easy to drop.

thanks,
-chris

#172 lore raw

Re: [Xen-devel] [RFC PATCH 34/35] Add the Xen virtual network device driver.

In reply to Chris Wright
Quoted 2 lines · Chris Wright

+ netdev->features = NETIF_F_IP_CSUM;

Any reason why IP_CSUM was chosen instead of HW_CSUM? Doing the latter would seem to be in fact easier for a virtual driver, no?

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#173 lore raw

Re: [Xen-devel] [RFC PATCH 34/35] Add the Xen virtual network device driver.

Quoted 6 lines · * Herbert Xu ([email protected]) wrote:
Quoted 2 lines · Chris Wright

+ netdev->features = NETIF_F_IP_CSUM;

Any reason why IP_CSUM was chosen instead of HW_CSUM? Doing the latter would seem to be in fact easier for a virtual driver, no?

That, I really don't know.

thanks,
-chris

#174 lore raw

Re: [Xen-devel] [RFC PATCH 34/35] Add the Xen virtual network device driver.

Quoted 9 lines · Chris Wright
Quoted 6 lines · * Herbert Xu ([email protected]) wrote:
Quoted 2 lines · Chris Wright

+ netdev->features = NETIF_F_IP_CSUM;

Any reason why IP_CSUM was chosen instead of HW_CSUM? Doing the latter would seem to be in fact easier for a virtual driver, no?

That, I really don't know.

Checksum offload was added late to the virtual transport and currently not enough info is carried to identify protocol checksum fields in arbitrary locations. When we rev the virtual interface, and include a proper checksum-offset field, we'll be able to switch to NETIF_F_HW_CSUM.

-- Keir

#175 lore raw

RFC PATCH 35/35 Add Xen virtual block device driver.

In reply to Chris Wright

The block device frontend driver allows the kernel to access block devices exported exported by a virtual machine containing a physical block device driver.

---
 drivers/block/Kconfig           |    2 
 drivers/xen/Kconfig.blk         |   14 
 drivers/xen/Makefile            |    1 
 drivers/xen/blkfront/Makefile   |    5 
 drivers/xen/blkfront/blkfront.c |  813 ++++++++++++++++++++++++++++++++++++++++
 drivers/xen/blkfront/block.h    |  156 +++++++
 drivers/xen/blkfront/vbd.c      |  216 ++++++++++
 7 files changed, 1207 insertions(+)
drivers/block/Kconfig +2 -0
@@ -449,6 +449,8 @@ config CDROM_PKTCDVD_WCACHE
 
 source "drivers/s390/block/Kconfig"
 
+source "drivers/xen/Kconfig.blk"
+
 config ATA_OVER_ETH
 	tristate "ATA over Ethernet support"
 	depends on NET
drivers/xen/Makefile +1 -0
@@ -6,5 +6,6 @@ obj-y	+= core/
 obj-y	+= console/
 obj-y	+= xenbus/
 
+obj-$(CONFIG_XEN_BLKDEV_FRONTEND)	+= blkfront/
 obj-$(CONFIG_XEN_NETDEV_FRONTEND)	+= netfront/
 
drivers/xen/Kconfig.blk new file +14 -0
@@ -0,0 +1,14 @@
+menu "Xen block device drivers"
+        depends on XEN
+
+config XEN_BLKDEV_FRONTEND
+	tristate "Block device frontend driver"
+	depends on XEN
+	default y
+	help
+	  The block device frontend driver allows the kernel to access block
+	  devices exported from a device driver virtual machine. Unless you
+	  are building a dedicated device driver virtual machine, then you
+	  almost certainly want to say Y here.
+
+endmenu
drivers/xen/blkfront/Makefile new file +5 -0
@@ -0,0 +1,5 @@
+
+obj-$(CONFIG_XEN_BLKDEV_FRONTEND)	:= xenblk.o
+
+xenblk-objs := blkfront.o vbd.o
+
drivers/xen/blkfront/blkfront.c new file +813 -0

813 lines not on this page · raw message

drivers/xen/blkfront/block.h new file +156 -0

156 lines not on this page · raw message

drivers/xen/blkfront/vbd.c new file +216 -0

216 lines not on this page · raw message

--

#176 lore raw

Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

Quoted 3 lines · Chris Wright

The block device frontend driver allows the kernel to access block devices exported exported by a virtual machine containing a physical block device driver.

Any reason you're using the old crappy xen I/O code instead of Rusty's alternative version?

Also please stop this stupid front/back naming. In Linux terminology the frontend is the client if there's a need for a postfix at all, and the backend the server. Compare that to e.g. ibm vio.

#177 lore raw
In reply to Chris Wright

Congrats on getting this thrashed out. A few comments, most of which are boring style nits, but nonetheless ... will try to take a proper look later.

General comment:

Why is this style used:

HYPERVISOR_foo_bar ?

ie the capitalization of HYPERVISOR. Doesn't seem to fit with the rest of the kernel style.

#178 lore raw
Quoted 4 lines · Martin J. Bligh

Congrats on getting this thrashed out. A few comments, most of which are boring style nits, but nonetheless ... will try to take a proper look later.

Quoted 8 lines

General comment:

Why is this style used:

HYPERVISOR_foo_bar ?

ie the capitalization of HYPERVISOR. Doesn't seem to fit with the rest of the kernel style.

It's also wrong. There's more than one hypervisor and Xen shouldn't just grab this namespace. make it xen_ or xenhv_.

#179 lore raw
Quoted 10 lines · Christoph Hellwig
Quoted 8 lines · Martin J. Bligh

Congrats on getting this thrashed out. A few comments, most of which are boring style nits, but nonetheless ... will try to take a proper look later.

General comment:

Why is this style used:

Quoted 8 lines
Quoted 4 lines

HYPERVISOR_foo_bar ?

ie the capitalization of HYPERVISOR. Doesn't seem to fit with the rest of the kernel style.

It's also wrong. There's more than one hypervisor and Xen shouldn't just grab this namespace. make it xen_ or xenhv_.

I think the intent was to create something generic enough for others to use. There were other projects that already intended to use the same model ... and please, lets not have one stack of this stuff for each hypervisor. So in general, I think the approach is correct, I was just whining about style stuff ;-)

M.

#180 lore raw
In reply to Christoph Hellwig
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Martin J. Bligh

Congrats on getting this thrashed out. A few comments, most of which are boring style nits, but nonetheless ... will try to take a proper look later.

General comment:

Why is this style used:

Quoted 7 lines
Quoted 4 lines

HYPERVISOR_foo_bar ?

ie the capitalization of HYPERVISOR. Doesn't seem to fit with the rest of the kernel style.

It's also wrong. There's more than one hypervisor and Xen shouldn't just grab this namespace. make it xen_ or xenhv_.

You should reject the recent "hypervisor file system" with the same argument then.

-Andi

#181 lore raw
Quoted 5 lines · Andi Kleen
Quoted 2 lines

It's also wrong. There's more than one hypervisor and Xen shouldn't just grab this namespace. make it xen_ or xenhv_.

You should reject the recent "hypervisor file system" with the same argument then.

I prefer it would become lparfs or something like that indeed.

#182 lore raw
Quoted 5 lines · Andi Kleen
Quoted 5 lines
Quoted 2 lines

It's also wrong. There's more than one hypervisor and Xen shouldn't just grab this namespace. make it xen_ or xenhv_.

You should reject the recent "hypervisor file system" with the same argument then.

Quoted 1 line · Christoph Hellwig

I prefer it would become lparfs or something like that indeed.

AFAIK it's called s390-hypfs now.

Pekka

#183 lore raw
In reply to Christoph Hellwig
Quoted 9 lines · Christoph Hellwig
Quoted 5 lines · Andi Kleen
Quoted 2 lines

It's also wrong. There's more than one hypervisor and Xen shouldn't just grab this namespace. make it xen_ or xenhv_.

You should reject the recent "hypervisor file system" with the same argument then.

I prefer it would become lparfs or something like that indeed.

Yes, it did get renamed to something s390-specific.

Also, note

http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/driver-core-add-sys-hypervisor-when-needed.patch

which creates the /sys/hypervisor directory. With the expectation that _all_ hypervisorish subsystems will base their sysfs trees in there.

#184 lore raw
Quoted 6 lines · * Andrew Morton ([email protected]) wrote:

Also, note

http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/driver-core-add-sys-hypervisor-when-needed.patch

which creates the /sys/hypervisor directory. With the expectation that _all_ hypervisorish subsystems will base their sysfs trees in there.

That's fine, Xen is currently using the same dir, so we just need to adjust and select Kconfig. There's some fallout as a a result (using attrs directly vs spinning your own fs, for example), but it makes sense to have common anchor in sysfs.

thanks,
-chris