Subsystems (8)
Hi, Jeremy & Keir
Here is the second version of patchset to enable Xen Hybrid extension support in Linux kernel.
The Hybrid Extension is started from real mode like HVM guest, but also with a a range of PV features(e.g. PV halt, PV timer, event channel, as well as PV drivers). So guest with Hybrid extension feature can takes the advantages of both H/W virtualization and Para-Virtualization.
The first two of the patchset imported several header file from Jeremy's tree and Xen tree, respect to Jeremy and Keir's works.
The whole patchset based on Linux upstream.
Current the patchset support x86_64 only.
The major change from v1:
1. SMP support.
2. Modify the entrance point to avoid most of genernic kernel modification.
3. Binding PV timer with event channel mechanism.
You need a line like:
cpuid = [ '0x40000002:edx=0x3' ]
in HVM configuration file to expose hybrid feature to guest, and
CONFIG_XEN
in the guest kernel configuration file to enable the hybrid support.
And the compiled image can be used as native/pv domU/hvm guest/hybrid kernel.
Comments are welcome!
BTW: For the MSI/MSI-X support, pv_ops dom0 can share the solution with hybrid. We would try to figure out a elegant way to deal with it later.
--
regards
Yang, Sheng
--
arch/x86/include/asm/xen/cpuid.h | 73 +++++++++++++ arch/x86/include/asm/xen/hypercall.h | 6 + arch/x86/kernel/setup.c | 8 ++ arch/x86/xen/enlighten.c | 192 ++++++++++++++++++++++++++++++++++ arch/x86/xen/irq.c | 54 ++++++++++ arch/x86/xen/smp.c | 144 +++++++++++++++++++++++++- arch/x86/xen/xen-head.S | 6 + arch/x86/xen/xen-ops.h | 4 + drivers/block/xen-blkfront.c | 3 + drivers/input/xen-kbdfront.c | 4 + drivers/net/xen-netfront.c | 3 + drivers/video/xen-fbfront.c | 4 + drivers/xen/events.c | 66 +++++++++++- drivers/xen/grant-table.c | 67 ++++++++++++- drivers/xen/xenbus/xenbus_probe.c | 23 ++++- include/xen/events.h | 1 + include/xen/hvm.h | 28 +++++ include/xen/interface/hvm/hvm_op.h | 79 ++++++++++++++ include/xen/interface/hvm/params.h | 111 ++++++++++++++++++++ include/xen/interface/xen.h | 6 +- include/xen/xen.h | 12 ++ 21 files changed, 883 insertions(+), 11 deletions(-)
PATCH 1/6 xen/hybrid: add support for hvm_op
From: Jeremy Fitzhardinge <[email protected]>
Add support for hvm_op hypercall.
--- arch/x86/include/asm/xen/hypercall.h | 6 ++ include/xen/hvm.h | 23 +++++++ include/xen/interface/hvm/hvm_op.h | 72 ++++++++++++++++++++++ include/xen/interface/hvm/params.h | 111 ++++++++++++++++++++++++++++++++++ 4 files changed, 212 insertions(+), 0 deletions(-) create mode 100644 include/xen/hvm.h create mode 100644 include/xen/interface/hvm/hvm_op.h create mode 100644 include/xen/interface/hvm/params.h
arch/x86/include/asm/xen/hypercall.h +6 -0
@@ -417,6 +417,12 @@ HYPERVISOR_nmi_op(unsigned long op, unsigned long arg)return _hypercall2 (int, nmi_op, op, arg); } +static inline unsigned long __must_check +HYPERVISOR_hvm_op (int op,void *arg) +{ + return _hypercall2(unsigned long, hvm_op, op, arg); +} +static inline void MULTI_fpu_taskswitch (struct multicall_entry *mcl,int set) {
include/xen/hvm.h new file +23 -0
@@ -0,0 +1,23 @@ +/* Simple wrappers around HVM functions */ +#ifndef XEN_HVM_H__ +#define XEN_HVM_H__ + +#include <xen/interface/hvm/params.h> + +static inline unsigned long hvm_get_parameter (int idx) +{ +struct xen_hvm_param xhv; +int r; + + xhv.domid =DOMID_SELF ; + xhv.index = idx; + r =HYPERVISOR_hvm_op (HVMOP_get_param, &xhv); +if (r <0 ) { +printk (KERN_ERR "cannot get hvm parameter %d: %d.\n" , + idx, r); +return 0 ; + } +return xhv.value; +} + +#endif /* XEN_HVM_H__ */
include/xen/interface/hvm/hvm_op.h new file +72 -0
72 lines not on this page · raw message
include/xen/interface/hvm/params.h new file +111 -0
111 lines not on this page · raw message
Signature · 1 line
1.5.4.5
PATCH 2/6 xen/hybrid: Import cpuid.h from Xen
In reply to Sheng YangFrom: Keir Fraser <[email protected]>
Which would be used by CPUID detection later
--- arch/x86/include/asm/xen/cpuid.h | 68 ++++++++++++++++++++++++++++++++++++++ 1 files changed, 68 insertions(+), 0 deletions(-) create mode 100644 arch/x86/include/asm/xen/cpuid.h
arch/x86/include/asm/xen/cpuid.h new file +68 -0
@@ -0,0 +1,68 @@ +/****************************************************************************** +* arch/include/asm/xen/cpuid.h +* +* CPUID interface to Xen. +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to +* deal in the Software without restriction, including without limitation the +* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +* sell copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +* DEALINGS IN THE SOFTWARE. +* +* Copyright (c) 2007 Citrix Systems, Inc. +* +* Authors: +* Keir Fraser <[email protected]> +*/ + +#ifndef __ASM_X86_XEN_CPUID_H__ +#define __ASM_X86_XEN_CPUID_H__ + +/* Xen identification leaves start at 0x40000000. */ +#define XEN_CPUID_FIRST_LEAF 0x40000000 +#define XEN_CPUID_LEAF (i) (XEN_CPUID_FIRST_LEAF + (i)) + +/* +* Leaf 1 (0x40000000) +* EAX: Largest Xen-information leaf. All leaves up to an including @EAX +* are supported by the Xen host. +* EBX-EDX: "XenVMMXenVMM" signature, allowing positive identification +* of a Xen host. +*/ +#define XEN_CPUID_SIGNATURE_EBX 0x566e6558/* "XenV" */ +#define XEN_CPUID_SIGNATURE_ECX 0x65584d4d/* "MMXe" */ +#define XEN_CPUID_SIGNATURE_EDX 0x4d4d566e/* "nVMM" */ + +/* +* Leaf 2 (0x40000001) +* EAX[31:16]: Xen major version. +* EAX[15: 0]: Xen minor version. +* EBX-EDX: Reserved (currently all zeroes). +*/ + +/* +* Leaf 3 (0x40000002) +* EAX: Number of hypercall transfer pages. This register is always guaranteed +* to specify one hypercall page. +* EBX: Base address of Xen-specific MSRs. +* ECX: Features 1. Unused bits are set to zero. +* EDX: Features 2. Unused bits are set to zero. +*/ + +/* Does the host support MMU_PT_UPDATE_PRESERVE_AD for this guest? */ +#define _XEN_CPUID_FEAT1_MMU_PT_UPDATE_PRESERVE_AD 0 +#define XEN_CPUID_FEAT1_MMU_PT_UPDATE_PRESERVE_AD (1u<<0) + +#endif /* __XEN_PUBLIC_ARCH_X86_CPUID_H__ */
Signature · 1 line
1.5.4.5
PATCH 3/6 xen/hybrid: Xen Hybrid Extension initialization
In reply to Sheng YangThe Hybrid Extension is started from real mode like HVM guest, but also with a component based PV feature selection(e.g. PV halt, PV timer, event channel, then PV drivers). So guest with Hybrid extension feature can takes the advantages of both H/W virtualization and Para-Virtualization.
This patch introduced the Hybrid Extension guest initialization.
Guest would detect Hybrid capability using CPUID 0x40000002.edx, then call HVMOP_enable_hybrid hypercall to enable hybrid support in hypervisor.
--- arch/x86/include/asm/xen/cpuid.h | 5 ++ arch/x86/xen/enlighten.c | 118 ++++++++++++++++++++++++++++++++++++ arch/x86/xen/irq.c | 21 ++++++ arch/x86/xen/xen-head.S | 6 ++ arch/x86/xen/xen-ops.h | 1 + include/xen/interface/hvm/hvm_op.h | 7 ++ include/xen/xen.h | 9 +++ 7 files changed, 167 insertions(+), 0 deletions(-)
arch/x86/include/asm/xen/cpuid.h +5 -0
@@ -65,4 +65,9 @@#define _XEN_CPUID_FEAT1_MMU_PT_UPDATE_PRESERVE_AD 0#define XEN_CPUID_FEAT1_MMU_PT_UPDATE_PRESERVE_AD (1u<<0) +#define _XEN_CPUID_FEAT2_HYBRID 0 +#define XEN_CPUID_FEAT2_HYBRID (1u<<0) +#define _XEN_CPUID_FEAT2_HYBRID_EVTCHN 1 +#define XEN_CPUID_FEAT2_HYBRID_EVTCHN (1u<<1) +#endif /* __XEN_PUBLIC_ARCH_X86_CPUID_H__ */
arch/x86/xen/enlighten.c +118 -0
133 lines not on this page · raw message
arch/x86/xen/irq.c +21 -0
@@ -131,3 +131,24 @@ void __init xen_init_irq_ops() pv_irq_ops = xen_irq_ops; x86_init.irqs.intr_init = xen_init_IRQ; } + +static void xen_hybrid_safe_halt (void ) +{ +/* Do local_irq_enable() explicitly in hybrid guest */ +local_irq_enable (); +xen_safe_halt (); +} + +static void xen_hybrid_halt (void ) +{ +if (irqs_disabled ()) +HYPERVISOR_vcpu_op (VCPUOP_down,smp_processor_id (),NULL ); +else +xen_hybrid_safe_halt (); +} + +void __init xen_hybrid_init_irq_ops (void ) +{ + pv_irq_ops.safe_halt = xen_hybrid_safe_halt; + pv_irq_ops.halt = xen_hybrid_halt; +}
arch/x86/xen/xen-head.S +6 -0
@@ -33,6 +33,12 @@ ENTRY(hypercall_page) .skip PAGE_SIZE_asm .popsection +.pushsection .data + .align PAGE_SIZE_asm +ENTRY(shared_info_page) + .skip PAGE_SIZE_asm +.popsection + ELFNOTE(Xen, XEN_ELFNOTE_GUEST_OS, .asciz "linux") ELFNOTE(Xen, XEN_ELFNOTE_GUEST_VERSION, .asciz "2.6") ELFNOTE(Xen, XEN_ELFNOTE_XEN_VERSION, .asciz "xen-3.0")
arch/x86/xen/xen-ops.h +1 -0
@@ -41,6 +41,7 @@ void xen_vcpu_restore(void);void __init xen_build_dynamic_phys_to_machine (void );void xen_init_irq_ops (void ); +void xen_hybrid_init_irq_ops (void );void xen_setup_timer (int cpu);void xen_setup_runstate_info (int cpu);void xen_teardown_timer (int cpu);
include/xen/interface/hvm/hvm_op.h +7 -0
@@ -69,4 +69,11 @@ DEFINE_GUEST_HANDLE_STRUCT(xen_hvm_set_pci_link_route);/* Flushes all VCPU TLBs: @arg must be NULL. */ #define HVMOP_flush_tlbs 5 +#define HVMOP_enable_hybrid 9 +struct xen_hvm_hybrid_type { +domid_t domid; +uint32_t flags; +#define HVM_HYBRID_EVTCHN (1ull<<1) +}; +#endif /* __XEN_PUBLIC_HVM_HVM_OP_H__ */
include/xen/xen.h +9 -0
@@ -9,6 +9,7 @@ enum xen_domain_type {#ifdef CONFIG_XEN extern enum xen_domain_type xen_domain_type; +extern void xen_hybrid_init (void );#else #define xen_domain_type XEN_NATIVE#endif
@@ -19,6 +20,14 @@ extern enum xen_domain_type xen_domain_type;#define xen_hvm_domain () (xen_domain() && \ xen_domain_type == XEN_HVM_DOMAIN) +#define XEN_HYBRID_ENABLED (1u << 0) +#define XEN_HYBRID_EVTCHN_ENABLED (1u << 1) +extern u32 xen_hybrid_status; + +#define xen_hybrid_enabled () (xen_hybrid_status & XEN_HYBRID_ENABLED) +#define xen_hybrid_evtchn_enabled () \ + (xen_hybrid_status & XEN_HYBRID_EVTCHN_ENABLED) +#ifdef CONFIG_XEN_DOM0 #include <xen/interface/xen.h> #include <asm/xen/hypervisor.h>
Signature · 1 line
1.5.4.5
PATCH 4/6 xen/hybrid: The entrance for Hybrid
In reply to Sheng Yangxen_hybrid_init() would setup Hybrid environment.
--- arch/x86/kernel/setup.c | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-)
arch/x86/kernel/setup.c +8 -0
@@ -113,6 +113,10 @@#endif #include <asm/mce.h> +#ifdef CONFIG_XEN +#include <xen/xen.h> +#endif +/* * end_pfn only includes RAM, while max_pfn_mapped includes all e820 entries. * The direct mapping extends to max_pfn_mapped, so that we can directly access
@@ -732,6 +736,10 @@ void __init setup_arch(char **cmdline_p) x86_init.oem.arch_setup (); +#ifdef CONFIG_XEN +xen_hybrid_init (); +#endif +setup_memory_map ();parse_setup_data ();/* update the e820_saved too */
Signature · 1 line
1.5.4.5
PATCH 5/6 xen/hybrid: Make event channel work with QEmu emulated devices
In reply to Sheng YangWe mapped each IOAPIC pin to a VIRQ, so that we can deliver interrupt through these VIRQs.
We used X86_PLATFORM_IPI_VECTOR as the noficiation vector for hypervisor to notify guest about the event.
The Xen PV timer is used to provide guest a reliable timer.
The patch also enabled SMP support, then we can support IPI through evtchn as well.
Then we don't need IOAPIC/LAPIC...
--- arch/x86/xen/enlighten.c | 73 ++++++++++++++++++++++ arch/x86/xen/irq.c | 37 ++++++++++- arch/x86/xen/smp.c | 144 ++++++++++++++++++++++++++++++++++++++++++- arch/x86/xen/xen-ops.h | 3 + drivers/xen/events.c | 66 ++++++++++++++++++- include/xen/events.h | 1 + include/xen/hvm.h | 5 ++ include/xen/interface/xen.h | 6 ++- 8 files changed, 327 insertions(+), 8 deletions(-)
arch/x86/xen/enlighten.c +73 -0
@@ -58,6 +58,9 @@#include <asm/reboot.h> #include <asm/stackprotector.h> +#include <xen/hvm.h> +#include <xen/events.h> +#include "xen-ops.h" #include "mmu.h" #include "multicalls.h"
@@ -1207,6 +1210,8 @@ static void __init xen_hybrid_banner(void)printk (KERN_INFO "Booting hybrid kernel on %s\n" , pv_info.name);printk (KERN_INFO "Xen version: %d.%d%s\n" , version >>16 , version &0xffff , extra.extraversion); +if (xen_hybrid_evtchn_enabled ()) +printk (KERN_INFO "Hybrid feature: Event channel enabled\n" ); }static int xen_para_available (void )
@@ -1252,6 +1257,11 @@ static int init_hybrid_info(void) xen_hybrid_status =XEN_HYBRID_ENABLED ; +if (edx &XEN_CPUID_FEAT2_HYBRID_EVTCHN ) { + xen_hybrid_status |=XEN_HYBRID_EVTCHN_ENABLED ; + flags |=HVM_HYBRID_EVTCHN ; + } +/* We only support 1 page of hypercall for now */ if (pages !=1 )return -ENOMEM ;
@@ -1291,12 +1301,42 @@ static void __init init_shared_info(void)per_cpu (xen_vcpu,0 ) = &HYPERVISOR_shared_info->vcpu_info[0 ]; } +static int set_callback_via (uint64_t via) +{ +struct xen_hvm_param a; + + a.domid =DOMID_SELF ; + a.index =HVM_PARAM_CALLBACK_IRQ ; + a.value = via; +return HYPERVISOR_hvm_op (HVMOP_set_param, &a); +} + +void do_hybrid_intr (void ) +{ +#ifdef CONFIG_X86_64 +per_cpu (irq_count,smp_processor_id ())++; +#endif +xen_evtchn_do_upcall (get_irq_regs ()); +#ifdef CONFIG_X86_64 +per_cpu (irq_count,smp_processor_id ())--; +#endif +} + +#ifdef CONFIG_X86_LOCAL_APIC +static void xen_hybrid_apic_write (u32 reg,u32 val) +{ +/* The only one reached here should be EOI */ +WARN_ON (reg !=APIC_EOI ); +} +#endif +void __init xen_hybrid_init (void ) {#ifdef CONFIG_X86_32 return ;#else int r; +uint64_t callback_via;/* Ensure the we won't confused with PV */ if (xen_domain_type ==XEN_PV_DOMAIN )
@@ -1309,6 +1349,39 @@ void __init xen_hybrid_init(void)init_shared_info ();xen_hybrid_init_irq_ops (); + +init_shared_info (); + +if (xen_hybrid_evtchn_enabled ()) { + pv_time_ops = xen_time_ops; + + x86_init.timers.timer_init = xen_time_init; + x86_init.timers.setup_percpu_clockev = x86_init_noop; + x86_cpuinit.setup_percpu_clockev = x86_init_noop; + + x86_platform.calibrate_tsc = xen_tsc_khz; + x86_platform.get_wallclock = xen_get_wallclock; + x86_platform.set_wallclock = xen_set_wallclock; + + pv_apic_ops = xen_apic_ops; +#ifdef CONFIG_X86_LOCAL_APIC +/* +* set up the basic apic ops. +*/ +set_xen_basic_apic_ops (); + apic->write = xen_hybrid_apic_write; +#endif + + callback_via =HVM_CALLBACK_VECTOR (X86_PLATFORM_IPI_VECTOR ); +set_callback_via (callback_via); + + x86_platform_ipi_callback = do_hybrid_intr; + +disable_acpi (); + +xen_hybrid_smp_init (); + machine_ops = xen_machine_ops; + }#endif }
arch/x86/xen/irq.c +35 -2
@@ -5,6 +5,7 @@#include <xen/interface/xen.h> #include <xen/interface/sched.h> #include <xen/interface/vcpu.h> +#include <xen/xen.h> #include <asm/xen/hypercall.h> #include <asm/xen/hypervisor.h>
@@ -132,6 +133,20 @@ void __init xen_init_irq_ops() x86_init.irqs.intr_init = xen_init_IRQ; } +static void xen_hybrid_irq_disable (void ) +{ +native_irq_disable (); +xen_irq_disable (); +} +PV_CALLEE_SAVE_REGS_THUNK (xen_hybrid_irq_disable); + +static void xen_hybrid_irq_enable (void ) +{ +native_irq_enable (); +xen_irq_enable (); +} +PV_CALLEE_SAVE_REGS_THUNK (xen_hybrid_irq_enable); +static void xen_hybrid_safe_halt (void ) {/* Do local_irq_enable() explicitly in hybrid guest */
@@ -147,8 +162,26 @@ static void xen_hybrid_halt(void)xen_hybrid_safe_halt (); } +static const struct pv_irq_ops xen_hybrid_irq_ops__initdata = { + .save_fl =__PV_IS_CALLEE_SAVE (native_save_fl), + .restore_fl =__PV_IS_CALLEE_SAVE (native_restore_fl), + .irq_disable =PV_CALLEE_SAVE (xen_hybrid_irq_disable), + .irq_enable =PV_CALLEE_SAVE (xen_hybrid_irq_enable), + + .safe_halt = xen_hybrid_safe_halt, + .halt = xen_hybrid_halt, +#ifdef CONFIG_X86_64 + .adjust_exception_frame = paravirt_nop, +#endif +}; +void __init xen_hybrid_init_irq_ops (void ) { - pv_irq_ops.safe_halt = xen_hybrid_safe_halt; - pv_irq_ops.halt = xen_hybrid_halt; +if (xen_hybrid_evtchn_enabled ()) { + pv_irq_ops = xen_hybrid_irq_ops; + x86_init.irqs.intr_init = xen_hybrid_init_IRQ; + }else { + pv_irq_ops.safe_halt = xen_hybrid_safe_halt; + pv_irq_ops.halt = xen_hybrid_halt; + } }
arch/x86/xen/smp.c +143 -1
173 lines not on this page · raw message
arch/x86/xen/xen-ops.h +3 -0
@@ -34,6 +34,7 @@ void xen_reserve_top(void);char *__init xen_memory_setup (void );void __init xen_arch_setup (void );void __init xen_init_IRQ (void ); +void __init xen_hybrid_init_IRQ (void );void xen_enable_sysenter (void );void xen_enable_syscall (void );void xen_vcpu_restore (void );
@@ -61,10 +62,12 @@ void xen_setup_vcpu_info_placement(void);#ifdef CONFIG_SMP void xen_smp_init (void ); +void xen_hybrid_smp_init (void );extern cpumask_var_t xen_cpu_initialized_map;#else static inline void xen_smp_init (void ) {} +static inline void xen_hybrid_smp_init (void ) {}#endif #ifdef CONFIG_PARAVIRT_SPINLOCKS
drivers/xen/events.c +62 -4
90 lines not on this page · raw message
include/xen/events.h +1 -0
@@ -56,4 +56,5 @@ void xen_poll_irq(int irq);/* Determine the IRQ which is bound to an event channel */ unsigned irq_from_evtchn (unsigned int evtchn); +void xen_evtchn_do_upcall (struct pt_regs *regs);#endif /* _XEN_EVENTS_H */
include/xen/hvm.h +5 -0
@@ -20,4 +20,9 @@ static inline unsigned long hvm_get_parameter(int idx) return xhv.value; } +#define HVM_CALLBACK_VIA_TYPE_VECTOR 0x2 +#define HVM_CALLBACK_VIA_TYPE_SHIFT 56 +#define HVM_CALLBACK_VECTOR (x) (((uint64_t)HVM_CALLBACK_VIA_TYPE_VECTOR)<<\ + HVM_CALLBACK_VIA_TYPE_SHIFT | (x)) +#endif /* XEN_HVM_H__ */
include/xen/interface/xen.h +5 -1
@@ -90,7 +90,11 @@#define VIRQ_ARCH_6 22#define VIRQ_ARCH_7 23 -#define NR_VIRQS 24 +#define VIRQ_EMUL_PIN_START 24 +#define VIRQ_EMUL_PIN_NUM 16 + +#define NR_VIRQS 40 +/* * MMU-UPDATE REQUESTS *
Signature · 1 line
1.5.4.5
PATCH 6/6 xen/hybrid: Enable grant table and xenbus
In reply to Sheng YangNow vnif and vbd drivers can work.
Notice one memory region(0xfbfe0000ul - 0xfc000000ul) would be reserved in the bios E820 table. This memory region would be used as grant table.
--- arch/x86/xen/enlighten.c | 1 + drivers/block/xen-blkfront.c | 3 ++ drivers/input/xen-kbdfront.c | 4 ++ drivers/net/xen-netfront.c | 3 ++ drivers/video/xen-fbfront.c | 4 ++ drivers/xen/grant-table.c | 67 +++++++++++++++++++++++++++++++++++- drivers/xen/xenbus/xenbus_probe.c | 23 +++++++++++-- include/xen/xen.h | 3 ++ 8 files changed, 103 insertions(+), 5 deletions(-)
arch/x86/xen/enlighten.c +1 -0
@@ -1278,6 +1278,7 @@ static int init_hybrid_info(void) pv_info = xen_info; pv_info.kernel_rpl =0 ; + xen_domain_type =XEN_HYBRID_DOMAIN ;return 0 ; }
drivers/block/xen-blkfront.c +3 -0
@@ -1071,6 +1071,9 @@ static int __init xlblk_init(void)if (!xen_domain ())return -ENODEV ; +if (xen_hybrid_domain () && !xen_hybrid_evtchn_enabled ()) +return -ENODEV ; +if (register_blkdev (XENVBD_MAJOR ,DEV_NAME )) {printk (KERN_WARNING "xen_blk: can't get major %d with name %s\n" ,XENVBD_MAJOR ,DEV_NAME );
drivers/input/xen-kbdfront.c +4 -0
@@ -341,6 +341,10 @@ static int __init xenkbd_init(void)if (!xen_domain ())return -ENODEV ; +/* Xen Hybrid domain don't need vkbd */ +if (xen_hybrid_domain ()) +return -ENODEV ; +/* Nothing to do if running in dom0. */ if (xen_initial_domain ())return -ENODEV ;
drivers/net/xen-netfront.c +3 -0
@@ -1807,6 +1807,9 @@ static int __init netif_init(void)if (!xen_domain ())return -ENODEV ; +if (xen_hybrid_domain () && !xen_hybrid_evtchn_enabled ()) +return -ENODEV ; +if (xen_initial_domain ())return 0 ;
drivers/video/xen-fbfront.c +4 -0
@@ -686,6 +686,10 @@ static int __init xenfb_init(void)if (!xen_domain ())return -ENODEV ; +/* Don't enable vfb in Xen hybrid domain */ +if (xen_hybrid_domain ()) +return -ENODEV ; +/* Nothing to do if running in dom0. */ if (xen_initial_domain ())return -ENODEV ;
drivers/xen/grant-table.c +65 -2
100 lines not on this page · raw message
drivers/xen/xenbus/xenbus_probe.c +20 -3
@@ -55,6 +55,8 @@#include <xen/events.h> #include <xen/page.h> +#include <xen/hvm.h> +#include "xenbus_comms.h" #include "xenbus_probe.h"
@@ -789,6 +791,9 @@ static int __init xenbus_probe_init(void)if (!xen_domain ()) goto out_error; +if (xen_hybrid_domain () && !xen_hybrid_evtchn_enabled ()) + goto out_error; +/* Register ourselves with the kernel bus subsystem */ err =bus_register (&xenbus_frontend.bus);if (err)
@@ -805,10 +810,19 @@ static int __init xenbus_probe_init(void)/* dom0 not yet supported */ }else { xenstored_ready =1 ; - xen_store_evtchn = xen_start_info->store_evtchn; - xen_store_mfn = xen_start_info->store_mfn; +if (xen_hybrid_domain ()) { + xen_store_evtchn = +hvm_get_parameter (HVM_PARAM_STORE_EVTCHN ); + xen_store_mfn = +hvm_get_parameter (HVM_PARAM_STORE_PFN ); + xen_store_interface = +ioremap (xen_store_mfn <<PAGE_SHIFT ,PAGE_SIZE ); + }else { + xen_store_evtchn = xen_start_info->store_evtchn; + xen_store_mfn = xen_start_info->store_mfn; + xen_store_interface =mfn_to_virt (xen_store_mfn); + } } - xen_store_interface =mfn_to_virt (xen_store_mfn);/* Initialize the interface to xenstore. */ err =xs_init ();
@@ -925,6 +939,9 @@ static void wait_for_devices(struct xenbus_driver *xendrv)if (!ready_to_wait_for_devices || !xen_domain ())return ; +if (xen_hybrid_domain () && !xen_hybrid_evtchn_enabled ()) +return ; +while (exists_connecting_device (drv)) {if (time_after (jiffies, start + (seconds_waited+5 )*HZ )) {if (!seconds_waited)
include/xen/xen.h +3 -0
@@ -5,6 +5,7 @@ enum xen_domain_type {XEN_NATIVE ,/* running on bare hardware */ XEN_PV_DOMAIN ,/* running in a PV domain */ XEN_HVM_DOMAIN ,/* running in a Xen hvm domain */ + XEN_HYBRID_DOMAIN, /* running in a Xen hybrid hvm domain */ };#ifdef CONFIG_XEN
@@ -19,6 +20,8 @@ extern void xen_hybrid_init(void); xen_domain_type == XEN_PV_DOMAIN)#define xen_hvm_domain () (xen_domain() && \ xen_domain_type == XEN_HVM_DOMAIN) +#define xen_hybrid_domain () (xen_domain() && \ + xen_domain_type == XEN_HYBRID_DOMAIN)#define XEN_HYBRID_ENABLED (1u << 0)#define XEN_HYBRID_EVTCHN_ENABLED (1u << 1)
Signature · 1 line
1.5.4.5
Re: [Xen-devel] [PATCH 6/6] xen/hybrid: Enable grant table and xenbus
Quoted 2 lines · Sheng Yang
Notice one memory region(0xfbfe0000ul -
0xfc000000ul) would be reserved in the bios E820 table. This memory region would be used as grant table.
I queried this requirement in a reply to the hypervisor patch.
drivers/block/xen-blkfront.c static int __init xlblk_init(void) · line 1076
Quoted 11 lines
drivers/block/xen-blkfront.c +3
-0@@ -1071,6 +1071,9 @@ static int __init xlblk_init(void)if (!xen_domain ())return -ENODEV ; +if (xen_hybrid_domain () && !xen_hybrid_evtchn_enabled ()) +return -ENODEV ; +
This seems ugly, at the very least the check should be something like xen_evtchn_enabled() but preferable would be to hook up evtchn's by demuxing the PCI device IRQ (the exiting PVonHVM drivers mechanism) in the case where hybrid evtchn's are not available and encapsulating the differences inside the evtchn code, there should be no need to scatter these sorts of checks throughout every driver.
If you don't want to demux the PCI device IRQ for the non-hybrid case another option might be simply return failure from the evtchn operations if hybrid evtchns are not available and to ensure that the drivers handle that sort of error gracefully (which they should in any case). At least the difference in mode would be encapsulated that way.
(same for netfront).
Quoted 8 lines
drivers/block/xen-blkfront.c +0
-0if (register_blkdev (XENVBD_MAJOR ,DEV_NAME )) {printk (KERN_WARNING "xen_blk: can't get major %d with name %s\n" ,XENVBD_MAJOR ,DEV_NAME );drivers/input/xen-kbdfront.c +0
-0
drivers/input/xen-kbdfront.c static int __init xenkbd_init(void) · line 347
Quoted 7 lines
drivers/input/xen-kbdfront.c +4
-0@@ -341,6 +341,10 @@ static int __init xenkbd_init(void)if (!xen_domain ())return -ENODEV ; +/* Xen Hybrid domain don't need vkbd */ +if (xen_hybrid_domain ()) +return -ENODEV ; +
Why disallow it if the platform has specified it (same for fbfront)?
include/xen/xen.h enum xen_domain_type { · line 9
Quoted 10 lines
include/xen/xen.h +1
-0@@ -5,6 +5,7 @@ enum xen_domain_type {XEN_NATIVE ,/* running on bare hardware */ XEN_PV_DOMAIN ,/* running in a PV domain */ XEN_HVM_DOMAIN ,/* running in a Xen hvm domain */ + XEN_HYBRID_DOMAIN, /* running in a Xen hybrid hvm domain */ };
I don't think you should need to distinguish HYBRID from HVM mode...
Ian.
Re: [Xen-devel] [PATCH 6/6] xen/hybrid: Enable grant table and xenbus
Quoted 14 lines · On Tuesday 02 February 2010 19:33:10 Ian Campbell wrote:
Quoted 2 lines · Sheng Yang
Notice one memory region(0xfbfe0000ul -
0xfc000000ul) would be reserved in the bios E820 table. This memory region would be used as grant table.I queried this requirement in a reply to the hypervisor patch.
Quoted 8 lines
drivers/block/xen-blkfront.c +0
-0@@ -1071,6 +1071,9 @@ static int __init xlblk_init(void)if (!xen_domain ())return -ENODEV ;
Quoted 6 lines
drivers/block/xen-blkfront.c static int __init xlblk_init(void) · line 1076
Quoted 3 lines
drivers/block/xen-blkfront.c +3
-0+if (xen_hybrid_domain () && !xen_hybrid_evtchn_enabled ()) +return -ENODEV ; +This seems ugly, at the very least the check should be something like
xen_evtchn_enabled()
Yeah, seems indeed ugly...
Quoted 6 lines
but preferable would be to hook up evtchn's by
demuxing the PCI device IRQ (the exiting PVonHVM drivers mechanism) in the case where hybrid evtchn's are not available and encapsulating the differences inside the evtchn code, there should be no need to scatter these sorts of checks throughout every driver.
Quoted 7 lines
If you don't want to demux the PCI device IRQ for the non-hybrid case another option might be simply return failure from the evtchn operations if hybrid evtchns are not available and to ensure that the drivers handle that sort of error gracefully (which they should in any case). At least the difference in mode would be encapsulated that way.
(same for netfront).
I am not sure if I understand you right, but I think the issue is, there is no PVonHVM drivers in Linux upstream. The drivers are currently maintained by OSVs, and the one in Xen upstream code only support 2.6.18. So I didn't take them into consideration at the time.
I think the "xen_evtchn_enable()" looks much better. Would replace these ugly lines in the next version.
Quoted 12 lines
Quoted 12 lines
drivers/block/xen-blkfront.c +0
-0if (register_blkdev (XENVBD_MAJOR ,DEV_NAME )) {printk (KERN_WARNING "xen_blk: can't get major %d with name %s\n" ,XENVBD_MAJOR ,DEV_NAME );drivers/input/xen-kbdfront.c +0
-0@@ -341,6 +341,10 @@ static int __init xenkbd_init(void)if (!xen_domain ())return -ENODEV ;
Quoted 6 lines
drivers/input/xen-kbdfront.c static int __init xenkbd_init(void) · line 347
Quoted 4 lines
drivers/input/xen-kbdfront.c +4
-0+/* Xen Hybrid domain don't need vkbd */ +if (xen_hybrid_domain ()) +return -ENODEV ; +Why disallow it if the platform has specified it (same for fbfront)?
Well.. The direct reason is I didn't test them and don't know what would happen... I would give it a try later.
Quoted 13 lines
include/xen/xen.h enum xen_domain_type { · line 9
Quoted 11 lines
include/xen/xen.h +1
-0@@ -5,6 +5,7 @@ enum xen_domain_type {XEN_NATIVE ,/* running on bare hardware */ XEN_PV_DOMAIN ,/* running in a PV domain */ XEN_HVM_DOMAIN ,/* running in a Xen hvm domain */ + XEN_HYBRID_DOMAIN, /* running in a Xen hybrid hvm domain */ };I don't think you should need to distinguish HYBRID from HVM mode...
The only purpose is for event channel of hybrid... Yes, I think I can find other way to indicate the availability of event channel. :)
Signature · 2 lines
regards Yang, Sheng
Re: [Xen-devel] [PATCH 6/6] xen/hybrid: Enable grant table and xenbus
Quoted 5 lines · Sheng Yang
I am not sure if I understand you right, but I think the issue is, there is no PVonHVM drivers in Linux upstream. The drivers are currently maintained by OSVs, and the one in Xen upstream code only support 2.6.18. So I didn't take them into consideration at the time.
True, but this is something which should be taken care of by the core Xen-aware code not something which should be pushed down into each driver.
Someone who wants to add PVonHVM functionality shouldn't have to go and remove a bunch of conditionals from each driver (or worse add alternative clauses to each check!).
Quoted 2 lines
I think the "
xen_evtchn_enable()" looks much better. Would replace these ugly lines in the next version.
I think it would be cleaner to encapsulate this in the evtchn code rather than leaking platform knowledge into each driver. IOW the evtchn functions should return failure if event channels are not enabled and the driver should cope with this gracefully.
Or perhaps at the xenbus driver level we should be deciding whether or not we have enough paravirtualisation to be worth probing the drivers at all?
Ian.
Re: [Xen-devel] [PATCH 6/6] xen/hybrid: Enable grant table and xenbus
Quoted 14 lines · On Wednesday 03 February 2010 00:24:04 Ian Campbell wrote:
Quoted 4 lines · Sheng Yang
I am not sure if I understand you right, but I think the issue is, there is no PVonHVM drivers in Linux upstream. The drivers are currently maintained by OSVs, and the one in Xen upstream code only support 2.6.18. So I didn't take them into consideration at the time.
True, but this is something which should be taken care of by the core Xen-aware code not something which should be pushed down into each driver.
Someone who wants to add PVonHVM functionality shouldn't have to go and remove a bunch of conditionals from each driver (or worse add alternative clauses to each check!).
Quoted 7 lines
Quoted 2 lines
I think the "
xen_evtchn_enable()" looks much better. Would replace these ugly lines in the next version.I think it would be cleaner to encapsulate this in the evtchn code rather than leaking platform knowledge into each driver. IOW the evtchn functions should return failure if event channels are not enabled and the driver should cope with this gracefully.
Agree. That what I suppose to do. What the drivers should only know is, if event channel is enabled.
Quoted 3 lines
Or perhaps at the xenbus driver level we should be deciding whether or not we have enough paravirtualisation to be worth probing the drivers at all?
I think current scheme is direct enough for now. We can improve it later.
Signature · 2 lines
regards Yang, Sheng
Re: [Xen-devel] [PATCH 6/6] xen/hybrid: Enable grant table and xenbus
Quoted 23 lines · Sheng Yang
Quoted 14 lines · On Wednesday 03 February 2010 00:24:04 Ian Campbell wrote:
Quoted 4 lines · Sheng Yang
I am not sure if I understand you right, but I think the issue is, there is no PVonHVM drivers in Linux upstream. The drivers are currently maintained by OSVs, and the one in Xen upstream code only support 2.6.18. So I didn't take them into consideration at the time.
True, but this is something which should be taken care of by the core Xen-aware code not something which should be pushed down into each driver.
Someone who wants to add PVonHVM functionality shouldn't have to go and remove a bunch of conditionals from each driver (or worse add alternative clauses to each check!).
Quoted 7 lines
Quoted 2 lines
I think the "
xen_evtchn_enable()" looks much better. Would replace these ugly lines in the next version.I think it would be cleaner to encapsulate this in the evtchn code rather than leaking platform knowledge into each driver. IOW the evtchn functions should return failure if event channels are not enabled and the driver should cope with this gracefully.
Quoted 8 lines
Agree. That what I suppose to do. What the drivers should only know is, if event channel is enabled.
Quoted 3 lines
Or perhaps at the xenbus driver level we should be deciding whether or not we have enough paravirtualisation to be worth probing the drivers at all?
I think current scheme is direct enough for now. We can improve it later.
Taking a step back I don't think any of these checks are necessary at all -- in order to get as far as actually probing the devices xenbus needs to be up and running, which implies event channels, as well as everything else required for PV drivers to function, are working.
Drivers for PCI devices don't all start with "if (pci_bus_available())", they just register the driver and let the kernel's driver core take care of things. If someone is worried about the overhead of an extra driver being registered, well that is what modular kernels are for.
I think that even the existing "if (xen_domain())" check is unnecessary, at least in the frontend drivers.
Ian.
Re: [Xen-devel] [PATCH 6/6] xen/hybrid: Enable grant table and xenbus
Quoted 38 lines · On Wednesday 03 February 2010 02:09:03 Ian Campbell wrote:
Quoted 23 lines · Sheng Yang
Quoted 14 lines · On Wednesday 03 February 2010 00:24:04 Ian Campbell wrote:
Quoted 4 lines · Sheng Yang
I am not sure if I understand you right, but I think the issue is, there is no PVonHVM drivers in Linux upstream. The drivers are currently maintained by OSVs, and the one in Xen upstream code only support 2.6.18. So I didn't take them into consideration at the time.
True, but this is something which should be taken care of by the core Xen-aware code not something which should be pushed down into each driver.
Someone who wants to add PVonHVM functionality shouldn't have to go and remove a bunch of conditionals from each driver (or worse add alternative clauses to each check!).
Quoted 7 lines
Quoted 2 lines
I think the "
xen_evtchn_enable()" looks much better. Would replace these ugly lines in the next version.I think it would be cleaner to encapsulate this in the evtchn code rather than leaking platform knowledge into each driver. IOW the evtchn functions should return failure if event channels are not enabled and the driver should cope with this gracefully.
Quoted 8 lines
Agree. That what I suppose to do. What the drivers should only know is, if event channel is enabled.
Quoted 3 lines
Or perhaps at the xenbus driver level we should be deciding whether or not we have enough paravirtualisation to be worth probing the drivers at all?
I think current scheme is direct enough for now. We can improve it later.
Taking a step back I don't think any of these checks are necessary at all -- in order to get as far as actually probing the devices xenbus needs to be up and running, which implies event channels, as well as everything else required for PV drivers to function, are working.
Quoted 7 lines
Drivers for PCI devices don't all start with "if (
pci_bus_available())", they just register the driver and let the kernel's driver core take care of things. If someone is worried about the overhead of an extra driver being registered, well that is what modular kernels are for.I think that even the existing "if (
xen_domain())" check is unnecessary, at least in the frontend drivers.
Um, very reasonable. I would provide another patch for this.
Signature · 2 lines
regards Yang, Sheng
Re: [Xen-devel] [PATCH 6/6] xen/hybrid: Enable grant table and xenbus
In reply to Sheng Yangdrivers/xen/grant-table.c static int gnttab_map(unsigned int start_idx, unsigned int end_idx) · line 501
Quoted 2 lines
drivers/xen/grant-table.c +2
-0+#define GNTTAB_START 0xfbfe0000ul +#define GNTTAB_SIZE 0x20000ul
Is it possible that there would be a PCI device that would be passed in the guest that would conflict with the above mentioned E820 region?
Re: [Xen-devel] [PATCH 6/6] xen/hybrid: Enable grant table and xenbus
Quoted 7 lines · On Wednesday 03 February 2010 01:03:06 Konrad Rzeszutek Wilk wrote:
drivers/xen/grant-table.c static int gnttab_map(unsigned int start_idx, unsigned int end_idx) · line 501
Quoted 2 lines
drivers/xen/grant-table.c +2
-0+#define GNTTAB_START 0xfbfe0000ul +#define GNTTAB_SIZE 0x20000ulIs it possible that there would be a PCI device that would be passed in the guest that would conflict with the above mentioned E820 region?
I would change them to a dedicated PCI MMIO address in the next version. Thanks.
Signature · 2 lines
regards Yang, Sheng
Re: [PATCH 0/6][v2] Hybrid extension for Xen guest
In reply to Sheng YangOops... Miss the subject field when git-send-email...
Signature · 2 lines
regards Yang, Sheng
Quoted 75 lines · On Tuesday 02 February 2010 16:19:01 Sheng Yang wrote:
Hi, Jeremy & Keir
Here is the second version of patchset to enable Xen Hybrid extension
support in Linux kernel.The Hybrid Extension is started from real mode like HVM guest, but also
with a a range of PV features(e.g. PV halt, PV timer, event channel, as well as PV drivers). So guest with Hybrid extension feature can takes the advantages of both H/W virtualization and Para-Virtualization.The first two of the patchset imported several header file from Jeremy's
tree and Xen tree, respect to Jeremy and Keir's works.The whole patchset based on Linux upstream.
Current the patchset support
x86_64only.The major change from v1:
1. SMP support.
2. Modify the entrance point to avoid most of genernic kernel modification.
3. Binding PV timer with event channel mechanism.You need a line like:
cpuid = [ '0x40000002:edx=0x3' ]
in HVM configuration file to expose hybrid feature to guest, and
CONFIG_XENin the guest kernel configuration file to enable the hybrid support.
And the compiled image can be used as native/pv domU/hvm guest/hybrid
kernel.Comments are welcome!
BTW: For the MSI/MSI-X support,
pv_opsdom0 can share the solution with
hybrid. We would try to figure out a elegant way to deal with it later.--
regards
Yang, Sheng--
arch/x86/include/asm/xen/cpuid.h | 73 +++++++++++++ arch/x86/include/asm/xen/hypercall.h | 6 + arch/x86/kernel/setup.c | 8 ++ arch/x86/xen/enlighten.c | 192 ++++++++++++++++++++++++++++++++++ arch/x86/xen/irq.c | 54 ++++++++++ arch/x86/xen/smp.c | 144 +++++++++++++++++++++++++- arch/x86/xen/xen-head.S | 6 + arch/x86/xen/xen-ops.h | 4 + drivers/block/xen-blkfront.c | 3 + drivers/input/xen-kbdfront.c | 4 + drivers/net/xen-netfront.c | 3 + drivers/video/xen-fbfront.c | 4 + drivers/xen/events.c | 66 +++++++++++- drivers/xen/grant-table.c | 67 ++++++++++++- drivers/xen/xenbus/xenbus_probe.c | 23 ++++- include/xen/events.h | 1 + include/xen/hvm.h | 28 +++++ include/xen/interface/hvm/hvm_op.h | 79 ++++++++++++++ include/xen/interface/hvm/params.h | 111 ++++++++++++++++++++ include/xen/interface/xen.h | 6 +- include/xen/xen.h | 12 ++ 21 files changed, 883 insertions(+), 11 deletions(-)_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel