Skip to contentSkip to conversation

API test framework

PATCH Applied Avi Kivity 2 participants 16 messages Dec 15, 2010 – Dec 22, 2010

Subsystems (2)
lore
Share & tools
lore

Avi Kivity

author
#0 lore raw

v2 cover API test framework

This patchset adds an API test framework. Rather than driving kvm from qemu, we now have a way of calling the kvm API directly and observing the results. We can switch to guest mode and back at will and see any micro effects such as the result of executing particular instructions.

A first test is added, for commit edde99ce0 ("KVM: Write protect memory after slot swap"). This would be pretty hard to test for using the current qemu-driven tests, but detects 5 million failures in about two minutes on my machine.

As an experiment, the framework is coded in C++.

Avi Kivity (14):
Makefile: add support for C++
Improve autodepend includes

    v2: new
  Add exception class for kernel errors (errno)
    v2: new
  Add try_main() for running a program under an exception handler
    v2: new
  Introduce a C++ wrapper for the kvm APIs
    v2: throw std::exception exceptions, not ints, for libc failures
        better msr list management
  Add support for calling a function in guest mode
    v2: use tr1::function instead of boost::function
  Add sample test using the api test harness
  api: add support for KVM_SET_USER_MEMORY_REGION flags field
    v2: new
  api: support KVM_GET_DIRTY_LOG ioctl
    v2: new
  api: add memory map management
    v2: new
  Build tests with debug information
    v2: new
  api: Add support for creating an identity map with a hole
    v2: new
  Introduce libapi.a to avoid long Makefile recipes
    v2: new
  Add dirty log test
    v2: new
 Makefile              |    8 +-
 api/api-sample.cc     |   30 ++++++++
 api/dirty-log.cc      |   78 ++++++++++++++++++++
 api/exception.cc      |   33 ++++++++
 api/exception.hh      |   19 +++++
 api/identity.cc       |   95 ++++++++++++++++++++++++
 api/identity.hh       |   43 +++++++++++
 api/kvmxx.cc          |  194 +++++++++++++++++++++++++++++++++++++++++++++++++
 api/kvmxx.hh          |   85 +++++++++++++++++++++
 api/memmap.cc         |   76 +++++++++++++++++++
 api/memmap.hh         |   43 +++++++++++
 config-x86-common.mak |   15 ++++-
 12 files changed, 715 insertions(+), 4 deletions(-)
 create mode 100644 api/api-sample.cc
 create mode 100644 api/dirty-log.cc
 create mode 100644 api/exception.cc
 create mode 100644 api/exception.hh
 create mode 100644 api/identity.cc
 create mode 100644 api/identity.hh
 create mode 100644 api/kvmxx.cc
 create mode 100644 api/kvmxx.hh
 create mode 100644 api/memmap.cc
 create mode 100644 api/memmap.hh
author
#1 lore raw

PATCH v2 1/14 Makefile: add support for C++

---
 Makefile |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
Makefile +3 -1
index d25e6f2..9b0256d 100644
@@ -30,11 +30,13 @@ CFLAGS += -O1
 CFLAGS += $(autodepend-flags) -g -fomit-frame-pointer -Wall
 CFLAGS += $(call cc-option, -fno-stack-protector, "")
 CFLAGS += $(call cc-option, -fno-stack-protector-all, "")
+CFLAGS += -I.
 
-CXXFLAGS = $(autodepend-flags)
+CXXFLAGS += $(CFLAGS)
 
 autodepend-flags = -MMD -MF $(dir $*).$(notdir $*).d
 
+LDFLAGS += $(CFLAGS)
 LDFLAGS += -pthread -lrt
 
 kvmtrace_objs= kvmtrace.o
Signature · 1 line
1.7.1
author
#2 lore raw

PATCH v2 2/14 Improve autodepend includes

In reply to Avi Kivity

Instead of listing all directories explicitly, include all autodepend files.

---
 Makefile              |    2 +-
 config-x86-common.mak |    1 -
 2 files changed, 1 insertions(+), 2 deletions(-)
Makefile +1 -1
index 9b0256d..85ebd37 100644
@@ -50,7 +50,7 @@ $(libcflat): $(cflatobjs)
 %.o: %.S
 	$(CC) $(CFLAGS) -c -nostdlib -o $@ $<
 
--include .*.d
+-include .*.d */.*.d */*/.*.d
 
 install:
 	mkdir -p $(DESTDIR)
config-x86-common.mak +0 -1
index c5508b3..d22df17 100644
@@ -81,4 +81,3 @@ arch_clean:
 	$(RM) $(TEST_DIR)/*.o $(TEST_DIR)/*.flat $(TEST_DIR)/*.elf \
 	$(TEST_DIR)/.*.d $(TEST_DIR)/lib/.*.d $(TEST_DIR)/lib/*.o
 
--include $(TEST_DIR)/.*.d lib/.*.d lib/x86/.*.d
Signature · 1 line
1.7.1
author
#3 lore raw

PATCH v2 3/14 Add exception class for kernel errors (errno)

In reply to Avi Kivity
---
 api/exception.cc |   20 ++++++++++++++++++++
 api/exception.hh |   16 ++++++++++++++++
 2 files changed, 36 insertions(+), 0 deletions(-)
 create mode 100644 api/exception.cc
 create mode 100644 api/exception.hh
api/exception.cc new file +20 -0
new file mode 100644
index 0000000..500569a
@@ -0,0 +1,20 @@
+#include "exception.hh"
+#include <cstdio>
+#include <cstring>
+
+errno_exception::errno_exception(int errno)
+    : _errno(errno)
+{
+}
+
+int errno_exception::errno() const
+{
+    return _errno;
+}
+
+const char *errno_exception::what()
+{
+    std::snprintf(_buf, sizeof _buf, "error: %s (%d)",
+		  std::strerror(_errno), _errno);
+    return _buf;
+}
api/exception.hh new file +16 -0
new file mode 100644
index 0000000..4672760
@@ -0,0 +1,16 @@
+#ifndef EXCEPTION_HH
+#define EXCEPTION_HH
+
+#include <exception>
+
+class errno_exception : public std::exception {
+public:
+    explicit errno_exception(int err_no);
+    int errno() const;
+    virtual const char *what();
+private:
+    int _errno;
+    char _buf[1000];
+};
+
+#endif
Signature · 1 line
1.7.1
author
#4 lore raw

PATCH v2 4/14 Add try_main() for running a program under an exception handler

In reply to Avi Kivity
---
 api/exception.cc |   13 +++++++++++++
 api/exception.hh |    3 +++
 2 files changed, 16 insertions(+), 0 deletions(-)
api/exception.cc +13 -0
index 500569a..910bdff 100644
@@ -18,3 +18,16 @@ const char *errno_exception::what()
 		  std::strerror(_errno), _errno);
     return _buf;
 }
+
+int try_main(int (*main)(int argc, char** argv), int argc, char** argv,
+	     int ret_on_exception)
+{
+    try {
+        return main(argc, argv);
+    } catch (std::exception& e) {
+        std::fprintf(stderr, "exception: %s\n", e.what());
+    } catch (...) {
+        std::fprintf(stderr, "unknown exception\n");
+    }
+    return ret_on_exception;
+}
api/exception.hh +3 -0
index 4672760..f78d9a1 100644
@@ -13,4 +13,7 @@ private:
     char _buf[1000];
 };
 
+int try_main(int (*main)(int argc, char** argv), int argc, char** argv,
+	     int ret_on_exception = 127);
+
 #endif
Signature · 1 line
1.7.1
author
#5 lore raw

PATCH v2 5/14 Introduce a C++ wrapper for the kvm APIs

In reply to Avi Kivity

Introduce exception-safe objects for calling system, vm, and vcpu ioctls.

---
 api/kvmxx.cc |  185 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 api/kvmxx.hh |   83 ++++++++++++++++++++++++++
 2 files changed, 268 insertions(+), 0 deletions(-)
 create mode 100644 api/kvmxx.cc
 create mode 100644 api/kvmxx.hh
api/kvmxx.cc new file +185 -0
new file mode 100644
index 0000000..ad27907
@@ -0,0 +1,185 @@
+#include "kvmxx.hh"
+#include "exception.hh"
+#include <fcntl.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <stdlib.h>
+#include <memory>
+#include <algorithm>
+
+namespace kvm {
+
+static long check_error(long r)
+{
+    if (r == -1) {
+	throw errno_exception(errno);
+    }
+    return r;
+}
+
+fd::fd(int fd)
+    : _fd(fd)
+{
+}
+
+fd::fd(const fd& other)
+    : _fd(::dup(other._fd))
+{
+    check_error(_fd);
+}
+
+fd::fd(std::string device_node, int flags)
+    : _fd(::open(device_node.c_str(), flags))
+{
+    check_error(_fd);
+}
+
+long fd::ioctl(unsigned nr, long arg)
+{
+    return check_error(::ioctl(_fd, nr, arg));
+}
+
+vcpu::vcpu(vm& vm, int id)
+    : _vm(vm), _fd(vm._fd.ioctl(KVM_CREATE_VCPU, id)), _shared(NULL)
+    , _mmap_size(_vm._system._fd.ioctl(KVM_GET_VCPU_MMAP_SIZE, 0))
+
+{
+    kvm_run *shared = static_cast<kvm_run*>(::mmap(NULL, _mmap_size,
+						   PROT_READ | PROT_WRITE,
+						   MAP_SHARED,
+						   _fd.get(), 0));
+    if (shared == MAP_FAILED) {
+	throw errno_exception(errno);
+    }
+    _shared = shared;
+}
+
+vcpu::~vcpu()
+{
+    munmap(_shared, _mmap_size);
+}
+
+void vcpu::run()
+{
+    _fd.ioctl(KVM_RUN, 0);
+}
+
+kvm_regs vcpu::regs()
+{
+    kvm_regs regs;
+    _fd.ioctlp(KVM_GET_REGS, &regs);
+    return regs;
+}
+
+void vcpu::set_regs(const kvm_regs& regs)
+{
+    _fd.ioctlp(KVM_SET_REGS, const_cast<kvm_regs*>(&regs));
+}
+
+kvm_sregs vcpu::sregs()
+{
+    kvm_sregs sregs;
+    _fd.ioctlp(KVM_GET_SREGS, &sregs);
+    return sregs;
+}
+
+void vcpu::set_sregs(const kvm_sregs& sregs)
+{
+    _fd.ioctlp(KVM_SET_SREGS, const_cast<kvm_sregs*>(&sregs));
+}
+
+class vcpu::kvm_msrs_ptr {
+public:
+    explicit kvm_msrs_ptr(size_t nmsrs);
+    ~kvm_msrs_ptr() { ::free(_kvm_msrs); }
+    kvm_msrs* operator->() { return _kvm_msrs; }
+    kvm_msrs* get() { return _kvm_msrs; }
+private:
+    kvm_msrs* _kvm_msrs;
+};
+
+vcpu::kvm_msrs_ptr::kvm_msrs_ptr(size_t nmsrs)
+    : _kvm_msrs(0)
+{
+    size_t size = sizeof(kvm_msrs) + sizeof(kvm_msr_entry) * nmsrs;
+    _kvm_msrs = static_cast<kvm_msrs*>(::malloc(size));
+    if (!_kvm_msrs) {
+	throw std::bad_alloc();
+    }
+}
+
+std::vector<kvm_msr_entry> vcpu::msrs(std::vector<uint32_t> indices)
+{
+    kvm_msrs_ptr msrs(indices.size());
+    msrs->nmsrs = indices.size();
+    for (unsigned i = 0; i < msrs->nmsrs; ++i) {
+	msrs->entries[i].index = indices[i];
+    }
+    _fd.ioctlp(KVM_GET_MSRS, msrs.get());
+    return std::vector<kvm_msr_entry>(msrs->entries,
+				      msrs->entries + msrs->nmsrs);
+}
+
+void vcpu::set_msrs(const std::vector<kvm_msr_entry>& msrs)
+{
+    kvm_msrs_ptr _msrs(msrs.size());
+    _msrs->nmsrs = msrs.size();
+    std::copy(msrs.begin(), msrs.end(), _msrs->entries);
+    _fd.ioctlp(KVM_SET_MSRS, _msrs.get());
+}
+
+void vcpu::set_debug(uint64_t dr[8], bool enabled, bool singlestep)
+{
+    kvm_guest_debug gd;
+
+    gd.control = 0;
+    if (enabled) {
+	gd.control |= KVM_GUESTDBG_ENABLE;
+    }
+    if (singlestep) {
+	gd.control |= KVM_GUESTDBG_SINGLESTEP;
+    }
+    for (int i = 0; i < 8; ++i) {
+	gd.arch.debugreg[i] = dr[i];
+    }
+    _fd.ioctlp(KVM_SET_GUEST_DEBUG, &gd);
+}
+
+vm::vm(system& system)
+    : _system(system), _fd(system._fd.ioctl(KVM_CREATE_VM, 0))
+{
+}
+
+void vm::set_memory_region(int slot, void *addr, uint64_t gpa, size_t len)
+{
+    struct kvm_userspace_memory_region umr;
+
+    umr.slot = slot;
+    umr.flags = 0;
+    umr.guest_phys_addr = gpa;
+    umr.memory_size = len;
+    umr.userspace_addr = reinterpret_cast<uint64_t>(addr);
+    _fd.ioctlp(KVM_SET_USER_MEMORY_REGION, &umr);
+}
+
+void vm::set_tss_addr(uint32_t addr)
+{
+    _fd.ioctl(KVM_SET_TSS_ADDR, addr);
+}
+
+system::system(std::string device_node)
+    : _fd(device_node, O_RDWR)
+{
+}
+
+bool system::check_extension(int extension)
+{
+    return _fd.ioctl(KVM_CHECK_EXTENSION, extension);
+}
+
+int system::get_extension_int(int extension)
+{
+    return _fd.ioctl(KVM_CHECK_EXTENSION, extension);
+}
+
+};
api/kvmxx.hh new file +83 -0

83 lines not on this page · raw message

Signature · 1 line
1.7.1
author
#6 lore raw

PATCH v2 6/14 Add support for calling a function in guest mode

In reply to Avi Kivity

This patch provides a way to establish an "identity" guest which has a 1:1 gva->hva translation. This allows the host to switch to guest mode, call a function in the same address space, and return.

Because long mode virtual addresses are 47 bits long, and some hosts have smaller physical addresses, we target 32-bit mode only. On x86_64 the code needs to be run with 'setarch i386 -3' to limit the address space to 3GB, so the address space occupied by the local APIC is left unused.

---
 api/identity.cc       |   76 +++++++++++++++++++++++++++++++++++++++++++++++++
 api/identity.hh       |   28 ++++++++++++++++++
 config-x86-common.mak |    1 +
 3 files changed, 105 insertions(+), 0 deletions(-)
 create mode 100644 api/identity.cc
 create mode 100644 api/identity.hh
api/identity.cc new file +76 -0
new file mode 100644
index 0000000..de52f68
@@ -0,0 +1,76 @@
+
+#include "identity.hh"
+#include <stdio.h>
+
+namespace identity {
+
+typedef unsigned long ulong;
+
+void setup_vm(kvm::vm& vm)
+{
+    vm.set_memory_region(0, NULL, 0, 3UL << 30);
+    vm.set_tss_addr(3UL << 30);
+}
+
+void vcpu::setup_sregs()
+{
+    kvm_sregs sregs = { };
+    kvm_segment dseg = { };
+    dseg.base = 0; dseg.limit = -1U; dseg.type = 3; dseg.present = 1;
+    dseg.dpl = 3; dseg.db = 1; dseg.s = 1; dseg.l = 0; dseg.g = 1;
+    kvm_segment cseg = dseg;
+    cseg.type = 11;
+
+    sregs.cs = cseg; asm ("mov %%cs, %0" : "=rm"(sregs.cs.selector));
+    sregs.ds = dseg; asm ("mov %%ds, %0" : "=rm"(sregs.ds.selector));
+    sregs.es = dseg; asm ("mov %%es, %0" : "=rm"(sregs.es.selector));
+    sregs.fs = dseg; asm ("mov %%fs, %0" : "=rm"(sregs.fs.selector));
+    sregs.gs = dseg; asm ("mov %%gs, %0" : "=rm"(sregs.gs.selector));
+    sregs.ss = dseg; asm ("mov %%ss, %0" : "=rm"(sregs.ss.selector));
+
+    uint32_t gsbase;
+    asm ("mov %%gs:0, %0" : "=r"(gsbase));
+    sregs.gs.base = gsbase;
+
+    sregs.tr.base = reinterpret_cast<ulong>(&*_stack.begin());
+    sregs.tr.type = 11;
+    sregs.tr.s = 0;
+    sregs.tr.present = 1;
+
+    sregs.cr0 = 0x11; /* PE, ET, !PG */
+    sregs.cr4 = 0;
+    sregs.efer = 0;
+    sregs.apic_base = 0xfee00000;
+    _vcpu.set_sregs(sregs);
+}
+
+void vcpu::thunk(vcpu* zis)
+{
+    zis->_guest_func();
+    asm volatile("outb %%al, %%dx" : : "a"(0), "d"(0));
+}
+
+void vcpu::setup_regs()
+{
+    kvm_regs regs = {};
+    regs.rflags = 0x3202;
+    regs.rsp = reinterpret_cast<ulong>(&*_stack.end());
+    regs.rsp &= ~15UL;
+    ulong* sp = reinterpret_cast<ulong *>(regs.rsp);
+    *--sp = reinterpret_cast<ulong>((char*)this);
+    *--sp = 0;
+    regs.rsp = reinterpret_cast<ulong>(sp);
+    regs.rip = reinterpret_cast<ulong>(&vcpu::thunk);
+    printf("rip %llx\n", regs.rip);
+    _vcpu.set_regs(regs);
+}
+
+vcpu::vcpu(kvm::vcpu& vcpu, std::tr1::function<void ()> guest_func,
+           unsigned long stack_size)
+    : _vcpu(vcpu), _guest_func(guest_func), _stack(stack_size)
+{
+    setup_sregs();
+    setup_regs();
+}
+
+}
api/identity.hh new file +28 -0
new file mode 100644
index 0000000..7401826
@@ -0,0 +1,28 @@
+#ifndef API_IDENTITY_HH
+#define API_IDENTITY_HH
+
+#include "kvmxx.hh"
+#include <tr1/functional>
+#include <vector>
+
+namespace identity {
+
+void setup_vm(kvm::vm& vm);
+
+class vcpu {
+public:
+    vcpu(kvm::vcpu& vcpu, std::tr1::function<void ()> guest_func,
+	 unsigned long stack_size = 256 * 1024);
+private:
+    static void thunk(vcpu* vcpu);
+    void setup_regs();
+    void setup_sregs();
+private:
+    kvm::vcpu& _vcpu;
+    std::tr1::function<void ()> _guest_func;
+    std::vector<char> _stack;
+};
+
+}
+
+#endif
config-x86-common.mak +1 -0
index d22df17..dde4f67 100644
@@ -81,3 +81,4 @@ arch_clean:
 	$(RM) $(TEST_DIR)/*.o $(TEST_DIR)/*.flat $(TEST_DIR)/*.elf \
 	$(TEST_DIR)/.*.d $(TEST_DIR)/lib/.*.d $(TEST_DIR)/lib/*.o
 
+api/%.o: CFLAGS += -m32
Signature · 1 line
1.7.1
author
#7 lore raw

PATCH v2 7/14 Add sample test using the api test harness

In reply to Avi Kivity

Call a function setting a global variable.

---
 api/api-sample.cc     |   29 +++++++++++++++++++++++++++++
 config-x86-common.mak |    7 +++++++
 2 files changed, 36 insertions(+), 0 deletions(-)
 create mode 100644 api/api-sample.cc
api/api-sample.cc new file +29 -0
new file mode 100644
index 0000000..8d57c09
@@ -0,0 +1,29 @@
+
+#include "api/kvmxx.hh"
+#include "api/identity.hh"
+#include "api/exception.hh"
+#include "stdio.h"
+
+static int global = 0;
+
+static void set_global()
+{
+    global = 1;
+}
+
+int test_main(int ac, char** av)
+{
+    kvm::system system;
+    kvm::vm vm(system);
+    identity::setup_vm(vm);
+    kvm::vcpu vcpu(vm, 0);
+    identity::vcpu thread(vcpu, set_global);
+    vcpu.run();
+    printf("global %d\n", global);
+    return global == 1 ? 0 : 1;
+}
+
+int main(int ac, char** av)
+{
+    return try_main(test_main, ac, av);
+}
config-x86-common.mak +7 -0
index dde4f67..3e8e641 100644
@@ -32,6 +32,8 @@ tests-common = $(TEST_DIR)/vmexit.flat $(TEST_DIR)/tsc.flat \
                $(TEST_DIR)/hypercall.flat $(TEST_DIR)/sieve.flat \
                $(TEST_DIR)/kvmclock_test.flat
 
+tests-common += api/api-sample
+
 tests_and_config = $(TEST_DIR)/*.flat $(TEST_DIR)/unittests.cfg
 
 test_cases: $(tests-common) $(tests)
@@ -82,3 +84,8 @@ arch_clean:
 	$(TEST_DIR)/.*.d $(TEST_DIR)/lib/.*.d $(TEST_DIR)/lib/*.o
 
 api/%.o: CFLAGS += -m32
+
+api/api-sample: LDLIBS += -lstdc++
+api/api-sample: LDFLAGS += -m32
+
+api/api-sample: api/api-sample.o api/kvmxx.o api/identity.o api/exception.o
Signature · 1 line
1.7.1
author
#8 lore raw

PATCH v2 8/14 api: add support for KVM_SET_USER_MEMORY_REGION flags field

In reply to Avi Kivity
---
 api/kvmxx.cc |    5 +++--
 api/kvmxx.hh |    3 ++-
 2 files changed, 5 insertions(+), 3 deletions(-)
api/kvmxx.cc +3 -2
index ad27907..42e8781 100644
@@ -150,12 +150,13 @@ vm::vm(system& system)
 {
 }
 
-void vm::set_memory_region(int slot, void *addr, uint64_t gpa, size_t len)
+void vm::set_memory_region(int slot, void *addr, uint64_t gpa, size_t len,
+                           uint32_t flags)
 {
     struct kvm_userspace_memory_region umr;
 
     umr.slot = slot;
-    umr.flags = 0;
+    umr.flags = flags;
     umr.guest_phys_addr = gpa;
     umr.memory_size = len;
     umr.userspace_addr = reinterpret_cast<uint64_t>(addr);
api/kvmxx.hh +2 -1
index 51dbe7a..958d36f 100644
@@ -57,7 +57,8 @@ private:
 class vm {
 public:
     explicit vm(system& system);
-    void set_memory_region(int slot, void *addr, uint64_t gpa, size_t len);
+    void set_memory_region(int slot, void *addr, uint64_t gpa, size_t len,
+                           uint32_t flags = 0);
     void set_tss_addr(uint32_t addr);
     system& sys() { return _system; }
 private:
Signature · 1 line
1.7.1
author
#9 lore raw

PATCH v2 9/14 api: support KVM_GET_DIRTY_LOG ioctl

In reply to Avi Kivity
---
 api/kvmxx.cc |    8 ++++++++
 api/kvmxx.hh |    1 +
 2 files changed, 9 insertions(+), 0 deletions(-)
api/kvmxx.cc +8 -0
index 42e8781..7ebebb5 100644
@@ -163,6 +163,14 @@ void vm::set_memory_region(int slot, void *addr, uint64_t gpa, size_t len,
     _fd.ioctlp(KVM_SET_USER_MEMORY_REGION, &umr);
 }
 
+void vm::get_dirty_log(int slot, void *log)
+{
+    struct kvm_dirty_log kdl;
+    kdl.slot = slot;
+    kdl.dirty_bitmap = log;
+    _fd.ioctlp(KVM_GET_DIRTY_LOG, &kdl);
+}
+
 void vm::set_tss_addr(uint32_t addr)
 {
     _fd.ioctl(KVM_SET_TSS_ADDR, addr);
api/kvmxx.hh +1 -0
index 958d36f..1dcb41d 100644
@@ -59,6 +59,7 @@ public:
     explicit vm(system& system);
     void set_memory_region(int slot, void *addr, uint64_t gpa, size_t len,
                            uint32_t flags = 0);
+    void get_dirty_log(int slot, void *log);
     void set_tss_addr(uint32_t addr);
     system& sys() { return _system; }
 private:
Signature · 1 line
1.7.1
author
#10 lore raw

PATCH v2 10/14 api: add memory map management

In reply to Avi Kivity

Add a class to manage the memory map and a class to represent a memory slot.

---
 api/memmap.cc |   76 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 api/memmap.hh |   43 ++++++++++++++++++++++++++++++++
 2 files changed, 119 insertions(+), 0 deletions(-)
 create mode 100644 api/memmap.cc
 create mode 100644 api/memmap.hh
api/memmap.cc new file +76 -0
new file mode 100644
index 0000000..c625852
@@ -0,0 +1,76 @@
+
+#include "memmap.hh"
+
+mem_slot::mem_slot(mem_map& map, uint64_t gpa, uint64_t size, void* hva)
+    : _map(map)
+    , _slot(map._free_slots.top())
+    , _gpa(gpa)
+    , _size(size)
+    , _hva(hva)
+    , _dirty_log_enabled(false)
+    , _log()
+{
+    map._free_slots.pop();
+    update();
+}
+
+mem_slot::~mem_slot()
+{
+    _size = 0;
+    try {
+        update();
+        _map._free_slots.push(_slot);
+    } catch (...) {
+        // can't do much if we can't undo slot registration - leak the slot
+    }
+}
+
+void mem_slot::set_dirty_logging(bool enabled)
+{
+    if (_dirty_log_enabled != enabled) {
+        _dirty_log_enabled = enabled;
+        if (enabled) {
+            int logsize = ((_size >> 12) + bits_per_word - 1) / bits_per_word;
+            _log.resize(logsize);
+        } else {
+            _log.resize(0);
+        }
+        update();
+    }
+}
+
+void mem_slot::update()
+{
+    uint32_t flags = 0;
+    if (_dirty_log_enabled) {
+        flags |= KVM_MEM_LOG_DIRTY_PAGES;
+    }
+    _map._vm.set_memory_region(_slot, _hva, _gpa, _size, flags);
+}
+
+bool mem_slot::dirty_logging() const
+{
+    return _dirty_log_enabled;
+}
+
+void mem_slot::update_dirty_log()
+{
+    _map._vm.get_dirty_log(_slot, &_log[0]);
+}
+
+bool mem_slot::is_dirty(uint64_t gpa) const
+{
+    uint64_t pagenr = (gpa - _gpa) >> 12;
+    ulong wordnr = pagenr / bits_per_word;
+    ulong bit = 1ULL << (pagenr % bits_per_word);
+    return _log[wordnr] & bit;
+}
+
+mem_map::mem_map(kvm::vm& vm)
+    : _vm(vm)
+{
+    int nr_slots = vm.sys().get_extension_int(KVM_CAP_NR_MEMSLOTS);
+    for (int i = 0; i < nr_slots; ++i) {
+        _free_slots.push(i);
+    }
+}
api/memmap.hh new file +43 -0

43 lines not on this page · raw message

Signature · 1 line
1.7.1
author
#11 lore raw

PATCH v2 11/14 Build tests with debug information

In reply to Avi Kivity
---
 Makefile |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
Makefile +1 -1
index 85ebd37..b6e8759 100644
@@ -6,7 +6,7 @@ DESTDIR := $(PREFIX)/share/qemu/tests
 .PHONY: arch_clean clean
 
 #make sure env CFLAGS variable is not used
-CFLAGS =
+CFLAGS = -g
 
 libgcc := $(shell $(CC) --print-libgcc-file-name)
 
Signature · 1 line
1.7.1
author
#12 lore raw

PATCH v2 12/14 api: Add support for creating an identity map with a hole

In reply to Avi Kivity

The hole may be used for mmio or dirty logging.

---
 api/api-sample.cc     |    3 ++-
 api/identity.cc       |   23 +++++++++++++++++++++--
 api/identity.hh       |   17 ++++++++++++++++-
 config-x86-common.mak |    1 +
 4 files changed, 40 insertions(+), 4 deletions(-)
api/api-sample.cc +2 -1
index 8d57c09..524ad7b 100644
@@ -15,7 +15,8 @@ int test_main(int ac, char** av)
 {
     kvm::system system;
     kvm::vm vm(system);
-    identity::setup_vm(vm);
+    mem_map memmap(vm);
+    identity::vm ident_vm(vm, memmap);
     kvm::vcpu vcpu(vm, 0);
     identity::vcpu thread(vcpu, set_global);
     vcpu.run();
api/identity.cc +21 -2
index de52f68..e04231b 100644
@@ -6,9 +6,28 @@ namespace identity {
 
 typedef unsigned long ulong;
 
-void setup_vm(kvm::vm& vm)
+hole::hole()
+    : address(), size()
 {
-    vm.set_memory_region(0, NULL, 0, 3UL << 30);
+}
+
+hole::hole(void* address, size_t size)
+    : address(address), size(size)
+{
+}
+
+vm::vm(kvm::vm& vm, mem_map& mmap, hole h)
+{
+    uint64_t hole_gpa = reinterpret_cast<uint64_t>(h.address);
+    char* hole_hva = static_cast<char*>(h.address);
+    if (h.address) {
+        _slots.push_back(mem_slot_ptr(new mem_slot(mmap, 0, hole_gpa, NULL)));
+    }
+    uint64_t hole_end = hole_gpa + h.size;
+    uint64_t end = 3U << 30;
+    _slots.push_back(mem_slot_ptr(new mem_slot(mmap, hole_end,
+                                               end - hole_end,
+                                               hole_hva + h.size)));
     vm.set_tss_addr(3UL << 30);
 }
 
api/identity.hh +16 -1
index 7401826..4491043 100644
@@ -2,12 +2,27 @@
 #define API_IDENTITY_HH
 
 #include "kvmxx.hh"
+#include "memmap.hh"
 #include <tr1/functional>
+#include <tr1/memory>
 #include <vector>
 
 namespace identity {
 
-void setup_vm(kvm::vm& vm);
+struct hole {
+    hole();
+    hole(void* address, size_t size);
+    void* address;
+    size_t size;
+};
+
+class vm {
+public:
+    vm(kvm::vm& vm, mem_map& mmap, hole address_space_hole = hole());
+private:
+    typedef std::tr1::shared_ptr<mem_slot> mem_slot_ptr;
+    std::vector<mem_slot_ptr> _slots;
+};
 
 class vcpu {
 public:
config-x86-common.mak +1 -0
index 3e8e641..436f4bd 100644
@@ -89,3 +89,4 @@ api/api-sample: LDLIBS += -lstdc++
 api/api-sample: LDFLAGS += -m32
 
 api/api-sample: api/api-sample.o api/kvmxx.o api/identity.o api/exception.o
+api/api-sample: api/memmap.o
\ No newline at end of file
Signature · 1 line
1.7.1
author
#13 lore raw

PATCH v2 13/14 Introduce libapi.a to avoid long Makefile recipes

In reply to Avi Kivity
---
 config-x86-common.mak |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)
config-x86-common.mak +6 -4
index 436f4bd..ce36cde 100644
@@ -85,8 +85,10 @@ arch_clean:
 
 api/%.o: CFLAGS += -m32
 
-api/api-sample: LDLIBS += -lstdc++
-api/api-sample: LDFLAGS += -m32
+api/%: LDLIBS += -lstdc++
+api/%: LDFLAGS += -m32
 
-api/api-sample: api/api-sample.o api/kvmxx.o api/identity.o api/exception.o
-api/api-sample: api/memmap.o
\ No newline at end of file
+api/libapi.a: api/kvmxx.o api/identity.o api/exception.o api/memmap.o
+	$(AR) rcs $@ $^
+
+api/api-sample: api/api-sample.o api/libapi.a
\ No newline at end of file
Signature · 1 line
1.7.1
author
#14 lore raw

PATCH v2 14/14 Add dirty log test

In reply to Avi Kivity

This checks the failure that was fixed by kernel commit edde99ce0529 ("KVM: Write protect memory after slot swap"). Two threads are used; a guest thread continuously updates a shared variable, which is also sampled by a host thread that also checks if dirty logging marked it as dirty.

It detects about 5 million failures with the fix reverted, and 0 failures with the fix applied.

---
 api/dirty-log.cc      |   78 +++++++++++++++++++++++++++++++++++++++++++++++++
 config-x86-common.mak |    7 +++-
 2 files changed, 83 insertions(+), 2 deletions(-)
 create mode 100644 api/dirty-log.cc
api/dirty-log.cc new file +78 -0
new file mode 100644
index 0000000..1e4ef9e
@@ -0,0 +1,78 @@
+#include "kvmxx.hh"
+#include "memmap.hh"
+#include "identity.hh"
+#include <boost/thread/thread.hpp>
+#include <stdlib.h>
+#include <stdio.h>
+
+namespace {
+
+void delay_loop(unsigned n)
+{
+    for (unsigned i = 0; i < n; ++i) {
+        asm volatile("pause");
+    }
+ }
+
+void write_mem(volatile bool& running, volatile int* shared_var)
+{
+    while (running) {
+        ++*shared_var;
+        delay_loop(1000);
+    }
+}
+
+void check_dirty_log(mem_slot& slot,
+                     volatile bool& running,
+                     volatile int* shared_var,
+                     int& nr_fail)
+{
+    uint64_t shared_var_gpa = reinterpret_cast<uint64_t>(shared_var);
+    slot.set_dirty_logging(true);
+    slot.update_dirty_log();
+    for (int i = 0; i < 10000000; ++i) {
+        int sample1 = *shared_var;
+        delay_loop(600);
+        int sample2 = *shared_var;
+        slot.update_dirty_log();
+        if (!slot.is_dirty(shared_var_gpa) && sample1 != sample2) {
+            ++nr_fail;
+        }
+    }
+    running = false;
+    slot.set_dirty_logging(false);
+}
+
+}
+
+using boost::ref;
+using std::tr1::bind;
+
+int main(int ac, char **av)
+{
+    kvm::system sys;
+    kvm::vm vm(sys);
+    mem_map memmap(vm);
+    void* logged_slot_virt;
+    posix_memalign(&logged_slot_virt, 4096, 4096);
+    int* shared_var = static_cast<int*>(logged_slot_virt);
+    identity::hole hole(logged_slot_virt, 4096);
+    identity::vm ident_vm(vm, memmap, hole);
+    kvm::vcpu vcpu(vm, 0);
+    bool running = true;
+    int nr_fail = 0;
+    mem_slot logged_slot(memmap,
+                         reinterpret_cast<uint64_t>(logged_slot_virt),
+                         4096, logged_slot_virt);
+    boost::thread host_poll_thread(check_dirty_log, ref(logged_slot),
+                                   ref(running),
+                                   ref(shared_var), ref(nr_fail));
+    identity::vcpu guest_write_thread(vcpu,
+                                      bind(write_mem,
+                                           ref(running),
+                                           ref(shared_var)));
+    vcpu.run();
+    host_poll_thread.join();
+    printf("Dirty bitmap failures: %d\n", nr_fail);
+    return nr_fail == 0 ? 0 : 1;
+}
config-x86-common.mak +5 -2
index ce36cde..b5c49f4 100644
@@ -33,6 +33,7 @@ tests-common = $(TEST_DIR)/vmexit.flat $(TEST_DIR)/tsc.flat \
                $(TEST_DIR)/kvmclock_test.flat
 
 tests-common += api/api-sample
+tests-common += api/dirty-log
 
 tests_and_config = $(TEST_DIR)/*.flat $(TEST_DIR)/unittests.cfg
 
@@ -85,10 +86,12 @@ arch_clean:
 
 api/%.o: CFLAGS += -m32
 
-api/%: LDLIBS += -lstdc++
+api/%: LDLIBS += -lstdc++ -lboost_thread-mt -lpthread
 api/%: LDFLAGS += -m32
 
 api/libapi.a: api/kvmxx.o api/identity.o api/exception.o api/memmap.o
 	$(AR) rcs $@ $^
 
-api/api-sample: api/api-sample.o api/libapi.a
\ No newline at end of file
+api/api-sample: api/api-sample.o api/libapi.a
+
+api/dirty-log: api/dirty-log.o api/libapi.a
Signature · 1 line
1.7.1
#15 lore raw
In reply to Avi Kivity
Quoted 56 lines · Avi Kivity

This patchset adds an API test framework. Rather than driving kvm from qemu, we now have a way of calling the kvm API directly and observing the results. We can switch to guest mode and back at will and see any micro effects such as the result of executing particular instructions.

A first test is added, for commit edde99ce0 ("KVM: Write protect memory after slot swap"). This would be pretty hard to test for using the current qemu-driven tests, but detects 5 million failures in about two minutes on my machine.

As an experiment, the framework is coded in C++.

Avi Kivity (14):
Makefile: add support for C++
Improve autodepend includes

    v2: new
  Add exception class for kernel errors (errno)
    v2: new
  Add try_main() for running a program under an exception handler
    v2: new
  Introduce a C++ wrapper for the kvm APIs
    v2: throw std::exception exceptions, not ints, for libc failures
        better msr list management
  Add support for calling a function in guest mode
    v2: use tr1::function instead of boost::function
  Add sample test using the api test harness
  api: add support for KVM_SET_USER_MEMORY_REGION flags field
    v2: new
  api: support KVM_GET_DIRTY_LOG ioctl
    v2: new
  api: add memory map management
    v2: new
  Build tests with debug information
    v2: new
  api: Add support for creating an identity map with a hole
    v2: new
  Introduce libapi.a to avoid long Makefile recipes
    v2: new
  Add dirty log test
    v2: new
 Makefile              |    8 +-
 api/api-sample.cc     |   30 ++++++++
 api/dirty-log.cc      |   78 ++++++++++++++++++++
 api/exception.cc      |   33 ++++++++
 api/exception.hh      |   19 +++++
 api/identity.cc       |   95 ++++++++++++++++++++++++
 api/identity.hh       |   43 +++++++++++
 api/kvmxx.cc          |  194 +++++++++++++++++++++++++++++++++++++++++++++++++
 api/kvmxx.hh          |   85 +++++++++++++++++++++
 api/memmap.cc         |   76 +++++++++++++++++++
 api/memmap.hh         |   43 +++++++++++
 config-x86-common.mak |   15 ++++-
 12 files changed, 715 insertions(+), 4 deletions(-)
 create mode 100644 api/api-sample.cc
 create mode 100644 api/dirty-log.cc
Quoted 8 lines
 create mode 100644 api/exception.cc
 create mode 100644 api/exception.hh
 create mode 100644 api/identity.cc
 create mode 100644 api/identity.hh
 create mode 100644 api/kvmxx.cc
 create mode 100644 api/kvmxx.hh
 create mode 100644 api/memmap.cc
 create mode 100644 api/memmap.hh

Applied, thanks.