i386 Vmi code patching
RFC PATCH 5/24 i386 Vmi code patching
The VMI ROM detection and code patching mechanism is illustrated in setup.c. There ROM is a binary block published by the hypervisor, and and there are certainly implications of this. ROMs certainly have a history of being proprietary, very differently licensed pieces of software, and mostly under non-free licenses. Before jumping to the conclusion that this is a bad thing, let us consider more carefully why hiding the interface layer to the hypervisor is actually a good thing.
The fact that this code is in a ROM is a design choice we made based on the ease of the delivery vehicle for our implementation, but there is nothing fundamentally different between this and a hyper-support (or vsyscall) page based approach. They are both merely mechanisms to inject hypervisor code into the guest that allows transparent virtualization of the native architecture, with design costs chosen by the hypervisor. In many cases, the calls into this support layer need not have a one to one mapping to the vendor defined hypercall interface; in fact, it is beneficial if they do not. Many of the calls can be optimized into local function calls which do not require an expensive privilege transition into the hypervisor. Many of the calls are obsolete when running under a VT/Pacifica based hypervisor. This is by design, and in cases when more virtualization capabilities are available in hardware, they can be taken advantage of instantly by simply doing nothing - there is no need to patch the guest code for classes of instructions which are efficiently and properly supported. But the fact remains that for in-use hardware today, these encapsulations are necessary, and in fact some will continue to allow for more efficient optimizations even with hardware virtualization.
Currently, the system is designed to boot under a fully virtualized (or native) environment, then switch into paravirtual mode. It does this by detecting the presence of a ROM code page, invoking the hypervisor initialization function, followed by switching the code annotations to the versions which call the VMI ROM code. There are some remnant stub implementations here from when the original approach was to make native functional equivalents for each of the ROM calls. This approach was unwieldy, and had a significant performance impact on critical paths in the system, so the stubs are being deprecated in favor of native inline code that allows for more native optimization potential.
Currently, the kernel does all the fancy code patching, but this step will eventually be done inside of the initialization routine of the hypervisor. Not only does this free the guest from this tedious responsibility (note the tricky code which disassembles the VMI call regions), but it allows the hypervisor to make further choices about inlining the code, eliminating the call/return overhead altogether. This overhead is non-trivial in the native case, which is why the native code must be preferentially inlined for a kernel which gets measurably near-identical native performance in microbenchmarks.
The fact that the ROM hides this interface code is deliberate, and is intended to allow the hypervisor vendor flexibility to change the underlying interface without the kernel needing to change. If the raw hypercall interface were presented to the kernel, it could look quite ugly and inflexible. It does not allow the system to evolve over time. On the other hand, if the layer is hidden, it gives the benefit of allowing alternative implementations to develop. Different versions of the interface can be supported, perhaps tailored with optimizations for different kernels, or adding statistics gathering and debugging code into the layer. The very value of the interface is not in it being visible, but by making it hidden.
The question of licensing of such ROM code is a completely separate issue. We are not trying to hide some proprietary code by putting it inside of a ROM to keep it hidden. In fact, you can disassemble the ROM code and see it quite readily - and you know all of the entry points. Whether we can distribute our ROM code under a GPL compatible license is not something I know at this time. Just as you can't compile a binary using Linux kernel headers and claim that your binary is not subject to the GPL, our ROM code includes headers from other parts of our system that are specifically not under the GPL. How this affects the final license under which the ROM is distributed is not something I think we know at this time. But there is no reason we can't rewrite those headers and make the ROM a separate and freely distributable entity. Even so, the usefulness of it is extremely limited; the interface to the hypervisor is proprietary, and subject to change, so the code merely serves as a sample implementation of one possible way to interface the layer to a hypervisor.
arch/i386/Makefile +5 -0
@@ -78,6 +78,10 @@ mflags-$(CONFIG_X86_ES7000) := -Iinclude mcore-$(CONFIG_X86_ES7000) := mach-default core-$(CONFIG_X86_ES7000) := arch/i386/mach-es7000/ +# VMI subarch support +mflags-$(CONFIG_X86_VMI) := -Iinclude/asm-i386/mach-vmi +mcore-$(CONFIG_X86_VMI) := mach-vmi + # default subarch .h files mflags-y += -Iinclude/asm-i386/mach-default
@@ -95,6 +99,7 @@ drivers-$(CONFIG_OPROFILE) += arch/i386 drivers-$(CONFIG_PM) += arch/i386/power/ CFLAGS += $(mflags-y) +CPPFLAGS += $(mflags-y) AFLAGS += $(mflags-y) boot := arch/i386/boot
arch/i386/mach-vmi/setup.c +309 -0
309 lines not on this page · raw message
arch/i386/mach-vmi/stubs.c +102 -0
102 lines not on this page · raw message
arch/i386/mach-vmi/stubs-asm.S +34 -0
@@ -0,0 +1,34 @@ +#include <linux/linkage.h> + +.section .text.VMI_SetPxELong,"ax" +ENTRY(VMI_SetPxELong) + mov %edx, 4(%ecx) + /* sfence */ + mov %eax, (%ecx) + ret + +.section .text.VMI_DeactivatePxELongAtomic,"ax" +ENTRY(VMI_DeactivatePxELongAtomic) + push %ecx + mov %eax, %ecx + xor %eax, %eax + xchg %eax, (%ecx) + mov 4(%ecx), %edx + movl $0, 4(%ecx) + pop %ecx + ret + +.section .text.VMI_SwapPxELongAtomic,"ax" +ENTRY(VMI_SwapPxELongAtomic) + push %edi + mov %ecx, %edi + push %ebx + mov %eax, %ebx + mov %edx, %ecx + mov (%edi), %eax + mov 4(%edi), %edx +1: lock; cmpxchg8b (%edi) + jne 1b + pop %ebx + pop %edi + ret
arch/i386/mach-vmi/Makefile +9 -0
@@ -0,0 +1,9 @@ +# +# Makefile for the linux kernel. +# + +EXTRA_CFLAGS += -I../kernel + +export CFLAGS_stubs.o += -ffunction-sections + +obj-y := setup.o stubs.o stubs-asm.o
include/asm-i386/bugs.h +5 -0
@@ -175,6 +175,11 @@ static void __init check_config(void) && (boot_cpu_data.x86_mask < 6 || boot_cpu_data.x86_mask == 11))panic ("Kernel compiled for PMMX+, assumes a local APIC without the read-before-write bug!" );#endif + +#ifdef CONFIG_VMI_REQUIRE_HYPERVISOR +if (!vmi_hypervisor_found ()) +panic ("No hypervisor found, aborting!\n" ); +#endif }extern void alternative_instructions (void );
include/asm-i386/mach-vmi/setup_arch_pre.h +6 -0
@@ -0,0 +1,6 @@ +/* Hook to call BIOS initialisation function */ + +/* no action for generic */ + +#define ARCH_SETUP +extern void vmi_init (void );
Quoted 39 lines · * Zachary Amsden ([email protected]) wrote:
arch/i386/mach-vmi/setup.c +39
-0+static void fixup_translation (struct vmi_annotation *a) +{ +unsigned char *c, *start, *end; +int left; + +memcpy (a->nativeEIP, a->translationEIP, a->translation_size); + start = a->nativeEIP; + end = a->nativeEIP + a->translation_size; + +for (c = start; c < end;) { +switch (*c) { +case MNEM_CALL_NEAR : +patch_call_site (a, c); + c+=5 ; +break ; + +case MNEM_PUSH_I : + c+=5 ; +break ; + +case MNEM_PUSH_IB : + c+=2 ; +break ; + +case MNEM_PUSH_EAX : +case MNEM_PUSH_ECX : +case MNEM_PUSH_EDX : +case MNEM_PUSH_EBX : +case MNEM_PUSH_EBP : +case MNEM_PUSH_ESI : +case MNEM_PUSH_EDI : + c+=1 ; +break ; + +case MNEM_LEA : +BUG_ON (*(c+1 ) !=0x64 );/* [--][--]+disp8, %esp */ +BUG_ON (*(c+2 ) !=0x24 );/* none + %esp */ + c+=4 ; +break ;
arch/i386/mach-vmi/setup.c line 127
Quoted 10 lines
arch/i386/mach-vmi/setup.c +10
-0+ +default : +/* +* Don't printk - it may acquire spinlocks with +* partially completed VMI translations, causing +* nuclear meltdown of the core. +*/ +BUG (); +return ; + }
Why these restrictions? How do you do int $0x82, for example?
thanks,
-chris
Quoted 18 lines · Chris Wright
Quoted 4 lines · * Zachary Amsden ([email protected]) wrote:
arch/i386/mach-vmi/setup.c +2
-0+static void fixup_translation (struct vmi_annotation *a) +{arch/i386/mach-vmi/setup.c line 90
Quoted 10 lines
arch/i386/mach-vmi/setup.c +10
-0+unsigned char *c, *start, *end; +int left; + +memcpy (a->nativeEIP, a->translationEIP, a->translation_size); + start = a->nativeEIP; + end = a->nativeEIP + a->translation_size; + +for (c = start; c < end;) { +switch (*c) { +case MNEM_CALL_NEAR :Why these restrictions? How do you do int $0x82, for example?
We don't. This is the minimal set of instructions that are emitted during the annotation. The instruction sequence is only sufficient to call out to the hypervisor ROM.
What we want to do next is to allow the hypervisor itself to do this code fixup - in effect, relinking in the local translations, which in many cases would then convert to int $0x82 for inline Xen calls.
Zach
Quoted 17 lines · Zachary Amsden
arch/i386/mach-vmi/setup.c +16
-0+void __init vmi_init (void ) +{ +int romsize; + +/* +* Setup optional callback functions if we found the VMI ROM +*/ +if (hypervisor_found) { + romsize = vmi_rom->romLength *512 ; +if (VROMFunc (vmi_rom, VMI_CALL_Init)) { +printk (KERN_WARNING "VMI ROM failed to initialize\n" ); + hypervisor_found =0 ; + }else { +memcpy (&__VMI_START, (char *)vmi_rom, romsize); +scan_builtin_annotations (); + }
arch/i386/mach-vmi/setup.c line 270
Quoted 10 lines
arch/i386/mach-vmi/setup.c +10
-0+ } +if (!vmi_rom) +printk (KERN_WARNING "VMI ROM not found" +" - falling back to native mode\n" ); +else if (!hypervisor_found) +printk (KERN_WARNING "VMI ROM version mismatch " +"(kernel requires version >= %d.%d) " +" - falling back to native mode\n" , +VMI_API_REV_MAJOR ,MIN_VMI_API_REV_MINOR ); +}
Minor nitpick.
The error logic here is somewhat confusing. If a VMI_CALL_Init results in a failure, you end up with:
VMI ROM failed to initialize
VMI ROM version mismatch (kernel requires version >= 13.0) - falling
back to native mode
The later error is misleading as the version may actually match. The nesting here probably could be simplified to.
Regards,
Anthony Liguori
Hi!
Quoted 3 lines
Currently, the system is designed to boot under a fully virtualized (or native) environment, then switch into paravirtual mode. It does this by detecting the presence of a ROM code page, invoking the
This is not going to work with xen, is it?
Quoted 5 lines
The question of licensing of such ROM code is a completely separate issue. We are not trying to hide some proprietary code by putting it inside of a ROM to keep it hidden. In fact, you can disassemble the ROM code and see it quite readily - and you know all of the entry points.
Could you disassemble one entry point for us and describe how it works?
Quoted 5 lines
Whether we can distribute our ROM code under a GPL compatible license is not something I know at this time. Just as you can't compile a binary using Linux kernel headers and claim that your binary is not subject to the GPL, our ROM code includes headers from other parts of our system that are specifically not under the GPL. How this
VMware is probably sole copyright holder (no?), you are perfectly free to relicence resulting ROM code under whatever licence you want.
Signature · 1 line
31: }
Quoted 9 lines · Pavel Machek
Quoted 6 lines
The question of licensing of such ROM code is a completely separate issue. We are not trying to hide some proprietary code by putting it inside of a ROM to keep it hidden. In fact, you can disassemble the ROM code and see it quite readily - and you know all of the entry points.
Could you disassemble one entry point for us and describe how it works?
000005e0 <stub_VMI_SwapPxE>:
5e0: e9 db 06 00 00 jmp cc0 <VMI_SwapPxE>
5e5: 0f 0b ud2a (The SwapPxE call does not fit in the 32-byte region).
00000cc0 <VMI_SwapPxE>:
cc0: 83 ec 1c sub $0x1c,%esp
cc3: 89 c1 mov %eax,%ecx
cc5: 89 74 24 10 mov %esi,0x10(%esp,1)
cc9: 89 d6 mov %edx,%esi
ccb: 89 7c 24 14 mov %edi,0x14(%esp,1)
ccf: 89 6c 24 18 mov %ebp,0x18(%esp,1)
cd3: 89 54 24 0c mov %edx,0xc(%esp,1)(Save registers and call values on stack)
cd7: 87 30 xchg %esi,(%eax)
(Atomically swap the PTE in the page)
cd9: 0f b6 44 24 0c movzbl 0xc(%esp,1),%eax
(Extract low byte of the PTE write just done)
cde: 31 d2 xor %edx,%edx
ce0: 09 f0 or %esi,%eax
ce2: a8 01 test $0x1,%al(Were either the new or old PTE's present?)
ce4: 74 11 je cf7 <VMI_SwapPxE+0x37>
(No? jump forward to branch combination)
ce6: 0f b6 44 24 0c movzbl 0xc(%esp,1),%eax
(Extract low byte of the PTE write again)
ceb: 31 f0 xor %esi,%eax
(Compute changed bits in PTE)
ced: a8 c7 test $0xc7,%al
(Have U/S, R/W, P, PS, D bits changed?)
cef: b8 01 00 00 00 mov $0x1,%eax
cf4: 0f 45 d0 cmovne %eax,%edx
cf7: 84 d2 test %dl,%dl(Combine above tests into one branch)
cf9: 74 32 je d2d <VMI_SwapPxE+0x6d>
(If nothing interesting changed, return)
cfb: b8 54 00 00 fc mov $0xfc000054,%eax
(Load start of linear address translation lookaside)
d00: 31 d2 xor %edx,%edx
d02: 89 cf mov %ecx,%edi
d04: 39 08 cmp %ecx,(%eax)(Compare stored lookaside translation address with address of PTE write)
d06: 77 37 ja d3f <VMI_SwapPxE+0x7f>
(Greater? - next)
d08: 39 48 04 cmp %ecx,0x4(%eax)
(Compare stored lookaside translate address end with address of PTE write)
d0b: 76 32 jbe d3f <VMI_SwapPxE+0x7f>
(Below/Equal? - next)
d0d: 03 78 08 add 0x8(%eax),%edi
(Add translation offset to PTE address (perform VA->PA conversion))
d10: c7 04 24 00 00 00 00 movl $0x0,(%esp,1)
d17: 8b 54 24 0c mov 0xc(%esp,1),%edx
d1b: bd 89 00 00 00 mov $0x89,%ebp
d20: 89 6c 24 04 mov %ebp,0x4(%esp,1)
d24: 31 c9 xor %ecx,%ecx
d26: 89 f8 mov %edi,%eax
d28: e8 73 fd ff ff call aa0 <VMIEnqueue>(Put arguments into order, and queue the PTE update)
d2d: 89 f0 mov %esi,%eax
d2f: 8b 74 24 10 mov 0x10(%esp,1),%esi
d33: 8b 7c 24 14 mov 0x14(%esp,1),%edi
d37: 8b 6c 24 18 mov 0x18(%esp,1),%ebp
d3b: 83 c4 1c add $0x1c,%esp
d3e: c3 ret(Restore registers and return)
d3f: 42 inc %edx
d40: 83 c0 10 add $0x10,%eax(Move to next translation index)
d43: 83 fa 03 cmp $0x3,%edx
d46: 7e bc jle d04 <VMI_SwapPxE+0x44>(Retry)
d48: 89 cf mov %ecx,%edi
d4a: eb c4 jmp d10 <VMI_SwapPxE+0x50>(Default case - VA == PA, direct mapping assumed)
And this version may be easier for some of you to read:
/*
* To convert virtual address to physical address, we must
* keep a cache of currently mapped page tables. This allows
* us to perform the VA->PA conversion with guest help, which
* is much faster than having to do a page table walk. For the
* Linux guest, when page tables are not in high memory, the
* first mapping slot is always an identity map of physical
* memory mapped at PAGE_OFFSET.
*/
static INLINE PA
VMITranslate(const LA la)
{
HyperAddressMap *m;
int i;
m = &vmiHypervisorShared->pageTranslation[0];
for (i = 0; i < VMI_LINEAR_MAP_SLOTS; i++, m++) {
if (LIKELY(la >= m->startVA && la < m->endVA)) {
return (m->physOffset + la);
} }
/* Assume identity mapped */
return la;
}
/*
* Note we read and write guest memory directly. This
* is safe for us because we use shadow page tables. The
* Xen ROM would perform similar conditional tests here to
* figure out which hypercall it wants to make or decide
* to write the PTE directly and trigger shadow */
VMICALL(SwapPxE, VM_PTE *const ptep, VM_PTE const pteval) {
VMI_ENTER;
VM_PTE pteold;
pteold = Atomic_ReadWrite((Atomic_uint32 *)ptep, pteval);
if (VMIIsPTEInteresting(pteval, pteold)) {
PA pa = VMITranslate((LA) ptep);
VMIEnqueue(pa, pteval, 0, 0, HC_SetPxEAsync); }
VMI_RETURN(pteold);
}
Herein lies one of our problems. Atomic_ReadWrite comes from a private header file. We need to copy all of these defines and inline functions into a distributable header file rather than a private one - separating public and private headers can be sort of like trying to straighten the vines of a giant thornbush. Every time you straighten one, another wacks you in the head. We have many, many build clients that depend on those same headers, and moving large amounts of things around is a tricky and painstaking process. Not that it can't be done, just that it is not a simple copy and paste job.
Zach
Quoted 7 lines · Pavel Machek
Quoted 5 lines
The question of licensing of such ROM code is a completely separate issue. We are not trying to hide some proprietary code by putting it inside of a ROM to keep it hidden. In fact, you can disassemble the ROM code and see it quite readily - and you know all of the entry points.
Could you disassemble one entry point for us and describe how it works?
Here is how I construct my ROM. My apologies if my email client wraps any of the lines.
I have a set of boundary functions between assembler and C code, identified by a suffix on the function names: _ext (stands for "extended" in response to the evolution of my ROM). They accept a parameter called burn_clobbers_frame_t:
struct burn_clobbers_frame_t {
word_t burn_ret_address; /* Return address to the ROM */
word_t frame_pointer; /* This parameter. */
word_t edx;
word_t ecx;
word_t eax;
word_t guest_ret_address;
word_t params[0]; /* Anything the guest pushed on the stack */
};I use this structure for the non-performance critical functions.
Here are some of the assembler macros for constructing the ROM:
vmi_stub_simple WRMSR, afterburn_cpu_write_msr_ext vmi_stub_simple RDMSR, afterburn_cpu_read_msr_ext
vmi_stub_simple SetGDT, afterburn_cpu_write_gdt32_ext vmi_stub_simple SetLDT, afterburn_cpu_lldt_ext vmi_stub_simple SetIDT, afterburn_cpu_write_idt32_ext vmi_stub_simple SetTR, afterburn_cpu_ltr_ext
macro vmi_stub_simple name func
/* Invoke a front-end C function that expects a burn_clobbers_frame_t as
* the parameter.
*/
vmi_stub_begin \name
__afterburn_save_clobbers
push %esp
subl $8, 0(%esp)
vmi_call_relocatable \func
__afterburn_restore_clobbers 4
ret
vmi_stub_end.endm
.macro vmi_stub_begin name
/* Define the prolog of a VMI stub. */
vmi_area_begin \name
burn_counter VMI_\name
burn_counter_inc VMI_\name
.endm
.macro vmi_stub_end
/* Define the epilog of a VMI stub. */
.previous
.endm
.macro vmi_area_begin name
/* Define the start of a VMI area. */
.section .vmi_rom, "ax"
.org vmi_rom_offset_\name
.globl VMI_\name
.type VMI_\name,@functionVMI_\name:
.endm
.macro vmi_call_relocatable func
9191: call \func
.pushsection .vmi_patchups, "aw"
.balign 4
.long 9191b
.popsection.endm
.macro vmi_jmp_relocatable target
9191: jmp \target
.pushsection .vmi_patchups, "aw"
.balign 4
.long 9191b
.popsection.endm
extern "C" voidafterburn_cpu_write_gdt32_ext( burn_clobbers_frame_t *frame )
{
get_cpu()->gdtr = *(dtr_t *)frame->eax;
}The ROM entry points are only half the solution. The other half involves Xen callbacks and traps. The system call trap is constructed to directly activate Linux's system call trap. Everything else jumps directly into the ROM for filtering reasons. The page-fault handler stays in assembler (because page faults are a performance issue; on a linux kernel build, they occur almost as frequently as system calls). The remaining traps enter C code, and look like this:
trap_wrapper id=8, use_error_code=1 /* Double fault exception */ trap_wrapper id=9, use_error_code=0 /* Coprocessor segment overrun */ trap_wrapper id=10, use_error_code=1 /* Invalid TSS exception */ trap_wrapper id=11, use_error_code=1 /* Segment not present */ trap_wrapper id=12, use_error_code=1 /* Stack fault exception */ trap_wrapper id=13, use_error_code=1 /* general protection fault */
.macro trap_wrapper id, use_error_code
entry trap_wrapper_\id
.if \use_error_code
subl $4, %esp /* Fault addr. */
.else
subl $8, %esp /* Error code, fault addr. */
.endif
pushl $(\id | (\use_error_code << 31)) /* Frame ID. */
cpu_save_all
movl %esp, %eax /* A pointer to the CPU save frame. */
call trap
jmp afterburn_exit
.endm
entry afterburn_exit
cpu_restore_all 0, 12 /* Error code, fault addr, frame id. */
iret
extern "C" void __attribute__(( regparm(1) ))trap( xen_frame_t *frame )
{
if( EXPECT_FALSE(frame->iret.ip >= CONFIG_WEDGE_VIRT) ) {
con << "Unexpected fault in the ROM, ip " << (void *)frame-
Quoted 1 line
iret.ip
<< '\n';
DEBUGGER_ENTER(frame);
panic();
}
u8_t *opstream = (u8_t *)frame->iret.ip;
if( cpu_t::get_segment_privilege(frame->iret.cs) == 3 )
{
// A user-level fault.
... check for TLS issues ...
}
// Update virtual CPU state, and deliver the trap to the guest
kernel.
xen_deliver_async_vector( frame->get_id(), frame,
frame->uses_error_code());
}Quoted 5 lines · * Joshua LeVasseur ([email protected]) wrote:
extern "C" void
afterburn_cpu_write_gdt32_ext( burn_clobbers_frame_t *frame ){ get_cpu()->gdtr = *(dtr_t *)frame->eax; }
What is this get_cpu()? Accessing data structure that's avail. in ROM and shared with hypervisor...could you elaborate a bit here?
thanks,
-chris
Quoted 7 lines · Chris Wright
Quoted 5 lines · * Joshua LeVasseur ([email protected]) wrote:
extern "C" void
afterburn_cpu_write_gdt32_ext( burn_clobbers_frame_t *frame ){ get_cpu()->gdtr = *(dtr_t *)frame->eax; }
Quoted 5 lines
What is this
get_cpu()? Accessing data structure that's avail. in ROM and shared with hypervisor...could you elaborate a bit here?thanks,
-chris
VMI is a very versatile interface due to the ROM; within the ROM you can translate the instruction set architecture and device register activity (as represented by the VMI interface) to a variety of hypervisor interfaces. I use a virtual CPU to help perform the translation. The performance of virtualization depends on the extent to which you can minimize interaction with the hypervisor via hypercalls. Many of the operations needn't be exposed to the hypervisor, and only operate on the virtual CPU, and thus remain completely within the ROM. The goal is to minimize interaction with the hypervisor.
I don't share the virtual CPU with the hypervisor. There probably are performance benefits for codesign between the hypervisor and ROM, but I haven't had that luxury; I take the hypervisors as given and none of them are fundamentally designed to use a ROM. On the other hand, it makes sense to concentrate virtualization within the ROM, rather than the hypervisor, for the same arguments you can make for implementing functionality in an application rather than the kernel.
I've implemented ROMs for two (open source) hypervisors so far, and try to share as much code between them as possible. The get_cpu() is an abstraction to help hide the hypervisor specifics for locating the virtual CPU (and it handles multiprocessor issues).
To help illustrate the role of the ROM, consider using Linux as a hypervisor, i.e., Linux-on-Linux but with the guest kernel using the VMI interface [1]. The ROM would translate the low-level operations of the guest kernel into the system calls of the host Linux, and it would be important to minimize the amount of interaction with the host Linux. Consider interrupt delivery, which would probably be mapped to POSIX signals. VMI offers VMI_EnableInterrupts(), VMI_DisableInterrupts(), VMI_GetInterruptMask(), and VMI_SetInterruptMask(). All of these operations are executed frequently by Linux, and it would be critical to limit their side effects to within the ROM; for performance reasons, they mustn't map to POSIX signal mask/unmask operations. The solution is to update only the EFLAGS in the virtual CPU when the guest kernel invokes VMI_EnableInterrupts, DisableInterrupts, etc.. Then the ROM must always accept asynchronous POSIX signal delivery, and must only forward asynchronous events to the guest kernel if interrupts are enabled in the virtual CPU. If the virtual CPU's interrupts are disabled, then the event is only recorded in the virtual PIC, and delivered at the next VMI_EnableInterrupts() or VMI_SetInterruptMask().
[1] Linux-on-Linux would probably limp with the current VMI. A couple changes would be necessary, such as permitting the Linux kernel to run at ring 3, and offering put_user() and get_user() hooks, since the guest applications and guest kernel must use different host address spaces. Unfortunately, put_user() and get_user() hooks are higher-level interfaces that don't fit well within VMI. For other CPU architectures with only two privilege levels, put_user() and get_user() hooks may be necessary too.
Joshua
Quoted 12 lines
The question of licensing of such ROM code is a completely separate issue. We are not trying to hide some proprietary code by putting it inside of a ROM to keep it hidden. In fact, you can disassemble the ROM code and see it quite readily - and you know all of the entry points. Whether we can distribute our ROM code under a GPL compatible license is not something I know at this time. Just as you can't compile a binary using Linux kernel headers and claim that your binary is not subject to the GPL, our ROM code includes headers from other parts of our system that are specifically not under the GPL. How this affects the final license under which the ROM is distributed is not something I think we know at this time.
The code _you_ wrote can be put under any license(s) you want, so in the worst case you do not need to rewrite your .c files.
Jan Engelhardt
Quoted 2 lines · Jan Engelhardt
The code _you_ wrote can be put under any license(s) you want, so in the worst case you do not need to rewrite your .c files.
The license might affect which other OSes could link in the ROM though...
Signature · 1 line
All Rights Reversed
Quoted 10 lines · Rik van Riel
Quoted 3 lines · Jan Engelhardt
The code _you_ wrote can be put under any license(s) you want, so in the worst case you do not need to rewrite your .c files.
The license might affect which other OSes could link in the ROM though...
That's actually a hideously good point. I think that means, in fact, that the ROM code specifically cannot be GPL'd - otherwise BSD might be unable to use it. The LGPL might be more appropriate.
Zach
Quoted 8 lines · Zachary Amsden
The VMI ROM detection and code patching mechanism is illustrated in
setup.c. There ROM is a binary block published by the hypervisor, and and there are certainly implications of this. ROMs certainly have a history of being proprietary, very differently licensed pieces of software, and mostly under non-free licenses. Before jumping to the conclusion that this is a bad thing, let us consider more carefully why hiding the interface layer to the hypervisor is actually a good thing.
How about you fix all these issues you describe here first and then submit it again?
The disassembly stuff indeed doesn't look like something that belongs in the kernel.
-Andi
Quoted 2 lines · * Andi Kleen ([email protected]) wrote:
The disassembly stuff indeed doesn't look like something that belongs in the kernel.
Strongly agreed. The strict ABI requirements put forth here are not in-line with Linux, IMO. I think source compatibility is the limit of reasonable, and any ROM code be in-tree if something like this were to be viable upstream.
thanks,
-chris
Quoted 5 lines · Chris Wright
Quoted 3 lines · * Andi Kleen ([email protected]) wrote:
The disassembly stuff indeed doesn't look like something that belongs in the kernel.
Agree that. It should be done prior to kernel booting, invisible to the kernel itself. I'm working on it, but there is still a lot to do.
Quoted 6 lines
Strongly agreed. The strict ABI requirements put forth here are not in-line with Linux, IMO. I think source compatibility is the limit of reasonable, and any ROM code be in-tree if something like this were to be viable upstream.
Strongly disagree. Without an ABI, you don't have binary compatibility. Without binary compatibility, you have no way to inline any hypervisor code into the kernel. And this is key for performance. The ROM code is being phased out.
Is it the strictness of the ABI that is the problem? I don't like constraining the native register values any much either, but it was the expedient thing to do. The ABI can be relaxed quite a bit, but it has to be there.
The idea of in-tree ROM code doesn't make sense. The entire point of this layer of code is that it is modular, and specific to the hypervisor, not the kernel. Once you lift the shroud and combine the two layers, you have lost all of the benefit that it was supposed to provide.
Zach
Quoted 8 lines · Zachary Amsden
Quoted 4 lines · Chris Wright
Strongly agreed. The strict ABI requirements put forth here are not in-line with Linux, IMO. I think source compatibility is the limit of reasonable, and any ROM code be in-tree if something like this were to be viable upstream.
The idea of in-tree ROM code doesn't make sense. The entire point of this layer of code is that it is modular, and specific to the hypervisor, not the kernel. Once you lift the shroud and combine the two layers, you have lost all of the benefit that it was supposed to provide.
To elaborate a bit more, the "ROM" layer is "published" by the hypervisor. This layer of abstraction will let you take a VMI-compiled kernel and run it efficiently on any hypervisor that exports a VMI interface - even one that you didn't know about (or didn't exist) when you compiled your kernel.
If the ROM part is compiled into the code, then you have to compile in support for the specific hypervisor(s) you want to run on. It might be reasonable for this code to be in a lodable kernel module, rather than a device ROM per se, but you still want that kernel module to be provided by the hypervisor.
Suppose someone implements a ROM layer for UML, or QEMU, or even for Microsoft's hypervisor. Having the ROM published by the hypervisor now lets you run your kernel on that new hypervisor without recompiling. While this might not be much of a benefit for an individual developer who downloads and compiles his own kernel, this is a huge win for people who distribute binary kernels, or large IT organizations that may have large heterogenous virtual machine farms to maintain.
Going forward, having the ROM layer published by the hypervisor gives the hypervisor more flexibility than having the code statically compiled into the kernel. Consider when hardware virtualization becomes more prevalent. Perhaps there are places where today hypercalls make sense, but with hardware virtualization, you'd rather have the hardware just take care of it. CPUID is the only example I can come up with at the moment, but there are certainly others. VMI lets the hypervisor decide that it doesn't actually need to replace the CPUID instruction with a hypercall. The important factor here is that only the hypervisor, not the kernel, knows about these performance tradeoffs. Or maybe in the next version of Xen, it's possible to use sysenter rather than an interrupt instruction to do hypercalls. If the hypervisor publishes this code, even older kernels can transparently take advantage of faster ways of doing certain things.
Dan.
Quoted 10 lines · * Daniel Arai ([email protected]) wrote:
To elaborate a bit more, the "ROM" layer is "published" by the hypervisor. This layer of abstraction will let you take a VMI-compiled kernel and run it efficiently on any hypervisor that exports a VMI interface - even one that you didn't know about (or didn't exist) when you compiled your kernel.
If the ROM part is compiled into the code, then you have to compile in support for the specific hypervisor(s) you want to run on. It might be reasonable for this code to be in a lodable kernel module, rather than a device ROM per se, but you still want that kernel module to be provided by the hypervisor.
I don't agree. That module may know how to get interface info from a vdso analog (just like a driver knows the hardware details of the device it's interacting with, but the core kernel api is unaware), but placing the binary compatibility on the kernel proper is wrong IMO.
thanks,
-chris
Quoted 10 lines · * Zachary Amsden ([email protected]) wrote:
Quoted 4 lines · Chris Wright
Strongly agreed. The strict ABI requirements put forth here are not in-line with Linux, IMO. I think source compatibility is the limit of reasonable, and any ROM code be in-tree if something like this were to be viable upstream.
Strongly disagree. Without an ABI, you don't have binary compatibility. Without binary compatibility, you have no way to inline any hypervisor code into the kernel. And this is key for performance. The ROM code is being phased out.
With source compatibility you get the ABI at compile time. This is how Linux handles internal interfaces. This is about an internal interface between kernel and a platform layer. The raw hypervisor interface is hidden (translations done in the platform layer), and the hypervisor needs to provide stable ABI.
Quoted 4 lines
Is it the strictness of the ABI that is the problem? I don't like constraining the native register values any much either, but it was the expedient thing to do. The ABI can be relaxed quite a bit, but it has to be there.
It's the very notion of creating such a large internal binary compatible interface.
Quoted 5 lines
The idea of in-tree ROM code doesn't make sense. The entire point of this layer of code is that it is modular, and specific to the hypervisor, not the kernel. Once you lift the shroud and combine the two layers, you have lost all of the benefit that it was supposed to provide.
You could compile all platform layers you want to support with the kernel.
thanks,
-chris
Quoted 2 lines · Chris Wright
You could compile all platform layers you want to support with the kernel.
But the entire point is that you don't know what platform layers you want to support. The platform layers can change. Xen has changed the platform layer and re-optimized kernel / hypervisor transitions how many times? The platform layer provides exactly the flexibility to do that, so that a kernel you compile today against a generic platform can work with the platform layer provided by Xen 4.0 tomorrow.
Compiling the platform layer with the kernel for "source compatibility" is exactly what prevents you from doing this. And you get stuck having the same stable and inflexible ABI to the hypervisor, rather than a carefully architected ABI just before it. The most important design decision in creating the VMI layer was disallowing data dependence between the compiled kernel and the hypervisor ABI.
I have seen, and will continue to see, every single shared data block layout change to meet the demands of new features. Eventually, you get to a point where it is growing antlers and having new hooves grafted onto it, yet still requires all of the original cruft you used to do. It is either a maintenance nightmare, or a compatibility nightmare. If you want compatibility, you really can't break that interface all the time, and the real world demands of customers using virtualization solutions really do want that compatibility. You simply can't certify a complex platform if you have to recompile your kernel for every new release of your chosen hypervisor. Bugs do get introduced this way, the older kernels fall out of maintenance, and eventually you are forcing them to upgrade to the latest kernels, which even worse, may have changed the userspace interface, dropped legacy feature support, and broken your key application that was the entire point of running in a VM to begin with. People throw things in VMs and then expect those VMs to keep running for years, and you really can't break that.
So instead, you impose a giant maintenance burden on the hypervisor, forcing them to go to all efforts to avoid breaking this hypervisor ABI. That leads directly to crufty, unstable, and poor performing code.
So the VMI layer is all about defining an ABI at a slightly higher level. A level which has many benefits you simply can't get from source compatibility. It is about the future, about preparing for the unknown, about giving a powerful abstraction to the platform layer to do whatever it chooses to do.
If Intel announces a new chip tomorrow, with a feature bit that allows selective privileged instructions to operate in non-zero supervisor CPLs, you're really going to regret the fact that you can't issue page invalidations and TLB flushes directly in the kernel because you unwisely decided to compile these in as direct int $0x81 hypercalls. You can change the platform layer and let new versions pick that up, and try to encourage people to move to newer kernels. And you have to make this change for every single operating system you support, leading to greater risk for introducing bugs in addition to any unwanted side effects of a kernel upgrade. Even worse, you may find a bug that _requires_ changing the platform layer. It might be a wide, gaping security hole. We had a few in the course of development (kernel CS entry value stored in shared area..). Now you have to break the hypervisor ABI, all your customers think you suck, and they have to upgrade all their systems. Perhaps they have been happily running the 2.4.26 kernel up to now. What do they do?
Why do you want to bind yourself to source compatibility, when it does not bring you features at all, it only hurts you in terms of deployment flexibility?
Zach
Quoted 9 lines · * Zachary Amsden ([email protected]) wrote:
Quoted 1 line · Chris Wright
You could compile all platform layers you want to support with the kernel.
But the entire point is that you don't know what platform layers you want to support. The platform layers can change. Xen has changed the platform layer and re-optimized kernel / hypervisor transitions how many times? The platform layer provides exactly the flexibility to do that, so that a kernel you compile today against a generic platform can work with the platform layer provided by Xen 4.0 tomorrow.
This only works if you have all possible dreamed of interface bits in the ABI. In Linux, we often don't know what we'll need to support in the future, but we don't write binary compatible interfaces just in case we need to update. Preferring instead, API's that are justifiable right now. This is the issue I have with the ABI proposal. It doesn't fit well with Linux developement.
thanks,
-chris
Quoted 9 lines · Chris Wright
Quoted 6 lines · * Zachary Amsden ([email protected]) wrote:
Quoted 2 lines · Chris Wright
You could compile all platform layers you want to support with the kernel.
But the entire point is that you don't know what platform layers you want to support. The platform layers can change. Xen has changed the platform layer and re-optimized kernel / hypervisor transitions how many times? The platform layer provides exactly the flexibility to do that, so that a kernel you compile today against a generic platform can work with the platform layer provided by Xen 4.0 tomorrow.
Quoted 7 lines
This only works if you have all possible dreamed of interface bits in the ABI. In Linux, we often don't know what we'll need to support in the future, but we don't write binary compatible interfaces just in case we need to update. Preferring instead, API's that are justifiable right now. This is the issue I have with the ABI proposal. It doesn't fit well with Linux developement.
No, you don't need to dream up all the possible interface bits ahead of time. With a la carte interfaces, you can take what you need now, and add features later. You don't need an ABI for features. You need it for compatibility. You will need to update the hypervisor ABI. And you can't force people to upgrade their kernels.
And much of this is so low level, that a C API for it just doesn't make sense. This code is completely hidden from Linux development to begin with, tucked away in the low level sub-arch layer.
Zach
Quoted 5 lines · * Zachary Amsden ([email protected]) wrote:
No, you don't need to dream up all the possible interface bits ahead of time. With a la carte interfaces, you can take what you need now, and add features later. You don't need an ABI for features. You need it for compatibility. You will need to update the hypervisor ABI. And you can't force people to upgrade their kernels.
How do you support an interface that's not already a part of the ABI w/out changing the kernel?
Quoted 8 lines · Chris Wright
Quoted 2 lines · * Zachary Amsden ([email protected]) wrote:
No, you don't need to dream up all the possible interface bits ahead of time. With a la carte interfaces, you can take what you need now, and add features later. You don't need an ABI for features. You need it for compatibility. You will need to update the hypervisor ABI. And you can't force people to upgrade their kernels.
How do you support an interface that's not already a part of the ABI w/out changing the kernel?
You have to change the kernel for VMI interface upgrades - if you want to use the upgrades. You don't need to change the kernel for hypervisor ABI changes, nor does upgrading the interface require a kernel change. Interface upgrades are pretty easy to compartmentalize - you add block device support, you add a block device driver. Hypervisor ABI changes are not so easy, because of the data dependencies and potential for breaking compatibility. The massive security hole scenario is a good example of why you would need to break compatibility, but any number of things might make you want to change the hypervisor ABI.
The point of the VMI is to isolate the kernel from those changes, allowing kernel development to proceed unhindered, and allowing hypervisor innovation to thrive simultaneously.
Zach
Quoted 9 lines · Chris Wright
Quoted 5 lines · * Zachary Amsden ([email protected]) wrote:
No, you don't need to dream up all the possible interface bits ahead of time. With a la carte interfaces, you can take what you need now, and add features later. You don't need an ABI for features. You need it for compatibility. You will need to update the hypervisor ABI. And you can't force people to upgrade their kernels.
How do you support an interface that's not already a part of the ABI w/out changing the kernel?
Since the base ABI primarily consists of the x86's privileged instruction set (actually, the virtualization-sensitive instructions, and padded with NOP instructions), any ROM can work from there, and you don't have to worry about updating Linux to use a new ABI. If you use a new ROM+ABI with an old kernel+ABI, they can fall back to the base ABI. Note that this base ABI isn't arbitrary; it wasn't pulled out of thin air; it is mostly the x86 system ISA.
If an updated hypervisor offers new features that didn't exist when a particular Linux kernel was written and compiled, a new ROM has a very good chance of activating those new features, even if only using the base ABI of the older Linux kernel. The ROM is very versatile because it maps the low-level instructions to high-level hypervisor concepts. And it is very successful: I have built a Linux 2.6.9 binary and executed it on Xen 2.0.2, Xen 2.0.7, and Xen 3.0.1; I have also built a Linux 2.6.12.6 binary and executed it on Xen 2.0.2, Xen 2.0.7, and Xen 3.0.1. This is significant because XenoLinux 2.6.9 shipped with Xen 2.0.2 and it doesn't work on Xen 3.0.1 due to many interface updates; likewise XenoLinux 2.6.12.6 shipped with Xen 3.0.1 and it doesn't work on the older Xen 2 hypervisors; but the ROM hid the interface updates from Xen 2 series to the Xen 3 series, and it takes advantage of the new Xen 3 interfaces (it must since Xen 3 doesn't have a Xen 2 compatibility layer that I'm aware of).
The ROM's interface mapping solves two problems: it converts the x86 low-level instructions into high-performance hypervisor operations, and it maps the low-level instructions into the hypervisor's evolving interface. The ROM gives great independence for hypervisor developers, or in other words, permits proliferation of hypervisors, and freedom to experiment with interfaces (e.g., real time, or formal verification).
Joshua
Quoted 11 lines · Chris Wright
Quoted 3 lines · * Andi Kleen ([email protected]) wrote:
The disassembly stuff indeed doesn't look like something that belongs in the kernel.
Strongly agreed. The strict ABI requirements put forth here are not in-line with Linux, IMO. I think source compatibility is the limit of reasonable, and any ROM code be in-tree if something like this were to be viable upstream.
Hi Chris,
Would you have less trouble if the "ROM" were actually more like a module? Specifically, if it had a proper elf header and symbol table, used symbols as entry points, and was a GPL interface (so that ROM's had to be GPL)? Then it's just a kernel module that's hidden in the option ROM space and has a C interface.
I know you end up losing the ability to do crazy inlining of the ROM code but I think it becomes a much less hairy interface that way.
Regards,
Anthony Liguori
Quoted 10 lines
thanks,
-chris------------------------------------------------------------------------
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.osdl.org/mailman/listinfo/virtualization
Quoted 5 lines · * Anthony Liguori ([email protected]) wrote:
Would you have less trouble if the "ROM" were actually more like a module? Specifically, if it had a proper elf header and symbol table, used symbols as entry points, and was a GPL interface (so that ROM's had to be GPL)? Then it's just a kernel module that's hidden in the option ROM space and has a C interface.
Yeah, point is the interface is normal C API, and has the similar free form that normal kernel API's have.
thanks,
-chris
Quoted 8 lines · Chris Wright
Quoted 5 lines
Would you have less trouble if the "ROM" were actually more like a module? Specifically, if it had a proper elf header and symbol table, used symbols as entry points, and was a GPL interface (so that ROM's had to be GPL)? Then it's just a kernel module that's hidden in the option ROM space and has a C interface.
Yeah, point is the interface is normal C API, and has the similar free form that normal kernel API's have.
i think this sounds very sane, and an OS-specific interface shim gets around problems such as finding CPU-specific state -- we can get at smp_processor_id() just the same as the rest of the kernel, for example. We could extend the concept of the interface shim we already have -- a set of OS-specific high performance shims, plus a fallback OS-agnostic shim.
-- Keir
Quoted 5 lines · Keir Fraser
Quoted 3 lines
Yeah, point is the interface is normal C API, and has the similar free form that normal kernel API's have.
i think this sounds very sane, and an OS-specific interface shim gets around problems such as finding CPU-specific state -- we can get at
smp_processor_id()just the same as the rest of the kernel, for example. We could extend the concept of the interface shim we already have -- a set of OS-specific high performance shims, plus a fallback OS-agnostic shim.
Getting at smp_processor_id() is exactly the type of thing you _don't_ want to do. You really can't have callbacks into the guest in the hypervisor platform layer. It really is not efficient, and you cause yourself more trouble than it is worth.
And where exactly is smp_processor_id() exported to modules? It's not. You've just locked your module into the current kernel's idea of how to get at smp_processor_id(). It changes based on compilation options of the kernel - for example, it is different with 4K stacks. It has changed from a number of other different options in the past.
The fact that XenoLinux needs smp_processor_id() at all is quite ludicrous. To disable interrupts, which is used fairly commonly to disable pre-emption as well, what does XenoLinux have to do?
It has to disable pre-emption to call smp_processor_id() so that it can disable interrupts, the re-enable preemption so that it can disable pre-emption.
That is truly convoluted, and is exactly why you should never get into these types of situations to begin with.
Zach
Quoted 1 line · Keir Fraser
We could extend the concept of the interface shim we already have -- a set of OS-specific high performance shims, plus a fallback OS-agnostic shim.
Currently the lack of a shim is the key difference between the VMI and Xen approaches. Forgive me for summarizing, but I'm not sure it's been made clear. The VMI is the interface between the OS and a shim layer--it is not a hypervisor interface. The kernel makes VMI calls to the shim and the shim makes hypercalls, if needed, to the hypervisor.
VMI VMI native Xen/Xen native
OS OS OS
-------------- VMI -------------- VMI
Shim (ROM)
-------------- HV API -------------- HV API
Hypervisor Native HW Hypervisor
The VMI isolates the kernel from the hypervisor so that the kernel and the hypervisor can evolve w/o hindering each other's development. The Xen approach still tightly couples the hypervisor with the kernel. Coupling the kernel and hypervisor together restricts their evolution and people who want to run different operating systems (or different versions of the same OS) on the same hypervisor. As Josh pointed out, you can run a single VMI Linux kernel on more versions of the Xen hypervisor than you can using a single XenLinux kernel because the VMI does not require a tight coupling.
Tight coupling also means you end up using a hypervisor when running a kernel natively (e.g. "supervisor mode kernel" in the unstable Xen repository). So for the native case you get a level of indirection (the hypervisor) that costs you performance, and for the virtual case you do not get a level of indirection (a shim) that buys you compatibility and diversity. For VMI, it's the reverse, you get the level of indirection in the virtual case and no indirection in the native case. You could have separate kernels, and all the associated costs, for these two cases.
There are many places where the VMI and Xen patches overlap; the key difference is that the VMI makes a distinction between the kernel and the hypervisor interfaces. As others have pointed out this distinction buys you a lot in terms of compatibility, ease of maintenance, and the ability to execute the same kernel in native and virtual environments with high performance.
Which particular bits get in is less important than the decision of whether or not the Linux community wants the kernel tightly coupled to the hypervisor. Extending the hypercall page you already have to decouple the hypervisor and kernel interfaces would be excellent.
Eli
[email protected] wrote on 03/23/2006 06:45:54 PM:
Quoted 14 lines
Quoted 5 lines · Keir Fraser
We could extend the concept of the interface shim we already have -- a
set of OS-specific high performance shims, plus a fallback OS-agnostic
shim.
Currently the lack of a shim is the key difference between the VMI and Xen approaches. Forgive me for summarizing, but I'm not sure it's been made clear. The VMI is the interface between the OS and a shim layer--it
is not a hypervisor interface. The kernel makes VMI calls to the shim and the shim makes hypercalls, if needed, to the hypervisor.
Quoted 9 lines
VMI VMI native Xen/Xen native
OS OS OS
-------------- VMI -------------- VMI
Shim (ROM)
-------------- HV API -------------- HV API
Hypervisor Native HW Hypervisor
Although this is about VMI, one should not loose sight of this one here with an unmodified OS on virtualizeable hardware:
Native
OS
----------------
Hypervisor
I think building this could actually be the driving force that can dictate *a lot* of what happens in 'Hypervisor' and what by that implementation alone is offered to the OS by 'Hypervisor' in terms of components. It would necessarily have to end up being a 'PC emulator'. Parts of that Hypervisor architecture can then also be made available to interface with the ROM shown above - not that this has not been done for Xen, but the Qemu models seem to be very far away in domain 0. Xen in contrast has been optimized for speed. I wonder whether the two goals of performanc and PC emulaton are in fact mutually exclusive.
Stefan
Quoted 5 lines
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 13 lines · Anthony Liguori
Quoted 11 lines · Chris Wright
Quoted 3 lines · * Andi Kleen ([email protected]) wrote:
The disassembly stuff indeed doesn't look like something that belongs in the kernel.
Strongly agreed. The strict ABI requirements put forth here are not in-line with Linux, IMO. I think source compatibility is the limit of reasonable, and any ROM code be in-tree if something like this were to be viable upstream.
Quoted 5 lines
Hi Chris,
Would you have less trouble if the "ROM" were actually more like a module? Specifically, if it had a proper elf header and symbol table, used symbols as entry points, and was a GPL interface (so that ROM's had to be GPL)? Then it's just a kernel module that's hidden in the option ROM space and has a C interface.
I know you end up losing the ability to do crazy inlining of the ROM code but I think it becomes a much less hairy interface that way.
Actually, I think you still can get the ability to do crazy inlining of the ROM code. You have three exports from the ELF module:
vmi_init - enter paravirtual modevmi_annotate - apply inline transformations based on inlining vmi_exit - exit paravirtual mode (required for module unloading).
But you can't require the ROM to be GPL'd. It has to be multi-licensed for compatibility with other open source or, even proprietary operating systems. If the ROM is licensed for use only under the GPL, then by including it in your kernel and allowing it to patch your kernel code, you leave your non-GPL kernel in a questionable license state. If the ROM is licensed under an open license, with a clause allowing its inclusion into GPL'd software, then I don't think you have a problem. Course I could be wrong. This is sort of a unique situation, and finding an identical comparison is tricky.
Zach
Quoted 11 lines · Zachary Amsden
Quoted 5 lines
Hi Chris,
Would you have less trouble if the "ROM" were actually more like a module? Specifically, if it had a proper elf header and symbol table, used symbols as entry points, and was a GPL interface (so that ROM's had to be GPL)? Then it's just a kernel module that's hidden in the option ROM space and has a C interface.
I know you end up losing the ability to do crazy inlining of the ROM code but I think it becomes a much less hairy interface that way.
Actually, I think you still can get the ability to do crazy inlining of the ROM code. You have three exports from the ELF module:
vmi_init- enter paravirtual modevmi_annotate- apply inline transformations based on inliningvmi_exit- exit paravirtual mode (required for module unloading).
Hrm, I was actually thinking that each of the VMI calls would be an export (vmi_init, vmi_set_pxe, etc.). I know that you want the hypervisor to drive the inlining but I that's sufficiently hairy (not to mention, there's not AFAIK performance data yet to justify it) that I think it ought to be left for VMI 2.0.
Quoted 1 line
But you can't require the ROM to be GPL'd. It has to be multi-licensed for compatibility with other open source or, even proprietary operating systems. If the ROM is licensed for use only under the GPL, then by including it in your kernel and allowing it to patch your kernel code, you leave your non-GPL kernel in a questionable license state. If the ROM is licensed under an open license, with a clause allowing its inclusion into GPL'd software, then I don't think you have a problem. Course I could be wrong. This is sort of a unique situation, and finding an identical comparison is tricky.
Multi-licensing is fine as long as one is GPL :-)
Regards,
Anthony Liguori
Quoted 1 line
Zach
Quoted 2 lines · Anthony Liguori
Hrm, I was actually thinking that each of the VMI calls would be an export (
vmi_init,vmi_set_pxe, etc.). I know that you want the hypervisor to drive the inlining but I that's sufficiently hairy (not to mention, there's not AFAIK performance data yet to justify it) that I think it ought to be left for VMI 2.0.
That seems quite ok to me. It is a little weird to have the VMI calls be an export when some of them really can never be properly callable C functions, and you have to overwrite the native code, so the linking step is .. well this magic disassembly glue again. But it could be made to work, and we have discussed it before.
Quoted 1 line
Multi-licensing is fine as long as one is GPL :-)
I agree. But it sort of defeats the point of the GPL if you can optionally redistribute the code under the BSD license as well.