Add HAX support

I took a stab at trying to rebase/upstream the support for Intel HAXM. (Hardware Accelerated Execution Manager).
Intel HAX is kernel-based hardware acceleration module for Windows and MacOSX.

I have based my work on the last version of the source code I found:
the emu-2.2-release branch in the external/qemu-android repository as used by the Android emulator.
In patch 2/4, I have forward-ported the core HAX code from there. It has been modified to build and run along with the current code base. It has been simplifying by removing non-UG hardware support / Darwin support / Android-specific leftovers.

Intel nicely fixed the 2 remaining issues on the kernel side:
- the spurious request to emulate MMIO access in un-paged mode is no longer happening (as seen in iPXE).
- the kernel API now provides a way to remove a memory mapping, so we can do a proper MemoryListener implementation.
They will publish soon a new version 6.1.0 of the HAX kernel module including the fixes once their QA cycle is completed.
Thanks Yu Ning for making this happen.

In patch 3/4, I have put the plumbing into the QEMU code base, I did some clean up there and it is reasonably intrusive: i.e.

 Makefile.target           |  1 +
 configure                 | 18 ++++++++++
 cpus.c                    | 87 ++++++++++++++++++++++++++++++++++++++++++++++-
 exec.c                    | 16 +++++++++
 hw/intc/apic_common.c     |  3 +-
 include/qom/cpu.h         |  5 +++
 include/sysemu/hw_accel.h |  9 +++++
 qemu-options.hx           | 11 ++++++
 target-i386/Makefile.objs |  4 +++
 vl.c                      | 15 ++++++--
 10 files changed, 164 insertions(+), 5 deletions(-)

The patch 1/4 just extracts from KVM specific header the cpu_synchronize_ functions that HAX is also using.

The patch 4/4 is the Darwin support. This part is only lightly tested for now, so it can be considered as 'experimental'.

I have tested the end result on a Windows 10 Pro machine (with UG support) with the Intel HAXM module dev version and a large ChromiumOS x86_64 image to exercise various code paths. It looks stable.
I also did a quick regression testing of the integration by running a Linux build with KVM enabled.

Changes from v4 to v5:
- update HAX fastmmio API with the new MMIO to MMIO transfer.

Changes from v3 to v4:
- add RAM unmapping in the MemoryListener thanks to new API in HAX module 6.1.0 and re-wrote the memory mappings management to deal with this.
- marked no longer used MMIO emulation as unsupported.
- clean-up a few left-overs from removed code.
- re-add an experimental version of the Darwin support.

Changes from v2 to v3:
- fix saving/restoring FPU registers as suggested by Paolo.
- fix Windows build on all targets as contributed by Stefan Weil.
- clean-up IO / MMIO emulation.
- more clean-up of emulation leftovers.

Changes from v1 to v2:
- fix all style issues in the original code to get it through checkpatch.pl.
- remove Darwin support, it was barely tested and not fully functional.
- remove the support for CPU without UG mode.
- fix most review comments

Vincent Palatin (4):
kvm: move cpu synchronization code
target-i386: Add Intel HAX files
Plumb the HAXM-based hardware acceleration support
hax: add Darwin support

 Makefile.target             |    1 +
 configure                   |   18 +
 cpus.c                      |   93 +++-
 exec.c                      |   16 +
 gdbstub.c                   |    1 +
 hax-stub.c                  |   39 ++
 hw/i386/kvm/apic.c          |    1 +
 hw/i386/kvmvapic.c          |    1 +
 hw/intc/apic_common.c       |    3 +-
 hw/misc/vmport.c            |    2 +-
 hw/ppc/pnv_xscom.c          |    2 +-
 hw/ppc/ppce500_spin.c       |    4 +-
 hw/ppc/spapr.c              |    2 +-
 hw/ppc/spapr_hcall.c        |    2 +-
 hw/s390x/s390-pci-inst.c    |    1 +
 include/qom/cpu.h           |    5 +
 include/sysemu/hax.h        |   56 +++
 include/sysemu/hw_accel.h   |   48 ++
 include/sysemu/kvm.h        |   23 -
 monitor.c                   |    2 +-
 qemu-options.hx             |   11 +
 qom/cpu.c                   |    2 +-
 target-arm/cpu.c            |    2 +-
 target-i386/Makefile.objs   |    7 +
 target-i386/hax-all.c       | 1155 +++++++++++++++++++++++++++++++++++++++++++
 target-i386/hax-darwin.c    |  316 ++++++++++++
 target-i386/hax-darwin.h    |   63 +++
 target-i386/hax-i386.h      |   94 ++++
 target-i386/hax-interface.h |  361 ++++++++++++++
 target-i386/hax-mem.c       |  271 ++++++++++
 target-i386/hax-windows.c   |  479 ++++++++++++++++++
 target-i386/hax-windows.h   |   89 ++++
 target-i386/helper.c        |    1 +
 target-i386/kvm.c           |    1 +
 target-ppc/mmu-hash64.c     |    2 +-
 target-ppc/translate_init.c |    2 +-
 target-s390x/gdbstub.c      |    1 +
 vl.c                        |   15 +-
 38 files changed, 3153 insertions(+), 39 deletions(-)
 create mode 100644 hax-stub.c
 create mode 100644 include/sysemu/hax.h
 create mode 100644 include/sysemu/hw_accel.h
 create mode 100644 target-i386/hax-all.c
 create mode 100644 target-i386/hax-darwin.c
 create mode 100644 target-i386/hax-darwin.h
 create mode 100644 target-i386/hax-i386.h
 create mode 100644 target-i386/hax-interface.h
 create mode 100644 target-i386/hax-mem.c
 create mode 100644 target-i386/hax-windows.c
 create mode 100644 target-i386/hax-windows.h

2.8.0.rc3.226.g39d4020

Move the generic cpu_synchronize_ functions to the common hw_accel.h header, in order to prepare for the addition of a second hardware accelerator.

---
 cpus.c                      |  1 +
 gdbstub.c                   |  1 +
 hw/i386/kvm/apic.c          |  1 +
 hw/i386/kvmvapic.c          |  1 +
 hw/misc/vmport.c            |  2 +-
 hw/ppc/pnv_xscom.c          |  2 +-
 hw/ppc/ppce500_spin.c       |  4 ++--
 hw/ppc/spapr.c              |  2 +-
 hw/ppc/spapr_hcall.c        |  2 +-
 hw/s390x/s390-pci-inst.c    |  1 +
 include/sysemu/hw_accel.h   | 39 +++++++++++++++++++++++++++++++++++++++
 include/sysemu/kvm.h        | 23 -----------------------
 monitor.c                   |  2 +-
 qom/cpu.c                   |  2 +-
 target-arm/cpu.c            |  2 +-
 target-i386/helper.c        |  1 +
 target-i386/kvm.c           |  1 +
 target-ppc/mmu-hash64.c     |  2 +-
 target-ppc/translate_init.c |  2 +-
 target-s390x/gdbstub.c      |  1 +
 20 files changed, 58 insertions(+), 34 deletions(-)
 create mode 100644 include/sysemu/hw_accel.h

index 5213351..fc78502 100644
@@ -33,6 +33,7 @@
 #include "sysemu/block-backend.h"
 #include "exec/gdbstub.h"
 #include "sysemu/dma.h"
+#include "sysemu/hw_accel.h"
 #include "sysemu/kvm.h"
 #include "qmp-commands.h"
 #include "exec/exec-all.h"

index de62d26..de9b62b 100644
@@ -32,6 +32,7 @@
 #define MAX_PACKET_LENGTH 4096
 
 #include "qemu/sockets.h"
+#include "sysemu/hw_accel.h"
 #include "sysemu/kvm.h"
 #include "exec/semihost.h"
 #include "exec/exec-all.h"

index 01cbaa8..328f80c 100644
@@ -14,6 +14,7 @@
 #include "cpu.h"
 #include "hw/i386/apic_internal.h"
 #include "hw/pci/msi.h"
+#include "sysemu/hw_accel.h"
 #include "sysemu/kvm.h"
 #include "target-i386/kvm_i386.h"
 

index b30d1b9..2f767b6 100644
@@ -14,6 +14,7 @@
 #include "exec/exec-all.h"
 #include "sysemu/sysemu.h"
 #include "sysemu/cpus.h"
+#include "sysemu/hw_accel.h"
 #include "sysemu/kvm.h"
 #include "hw/i386/apic_internal.h"
 #include "hw/sysbus.h"

index c763811..be40930 100644
@@ -25,7 +25,7 @@
 #include "hw/hw.h"
 #include "hw/isa/isa.h"
 #include "hw/i386/pc.h"
-#include "sysemu/kvm.h"
+#include "sysemu/hw_accel.h"
 #include "hw/qdev.h"
 
 //#define VMPORT_DEBUG

index 8da2718..cd5c2b8 100644
@@ -20,7 +20,7 @@
 #include "qapi/error.h"
 #include "hw/hw.h"
 #include "qemu/log.h"
-#include "sysemu/kvm.h"
+#include "sysemu/hw_accel.h"
 #include "target-ppc/cpu.h"
 #include "hw/sysbus.h"
 

index cf958a9..eb219ab 100644
@@ -29,9 +29,9 @@
 
 #include "qemu/osdep.h"
 #include "hw/hw.h"
-#include "sysemu/sysemu.h"
 #include "hw/sysbus.h"
-#include "sysemu/kvm.h"
+#include "sysemu/hw_accel.h"
+#include "sysemu/sysemu.h"
 #include "e500.h"
 
 #define MAX_CPUS 32

index 208ef7b..a642e66 100644
@@ -36,7 +36,7 @@
 #include "sysemu/device_tree.h"
 #include "sysemu/block-backend.h"
 #include "sysemu/cpus.h"
-#include "sysemu/kvm.h"
+#include "sysemu/hw_accel.h"
 #include "kvm_ppc.h"
 #include "migration/migration.h"
 #include "mmu-hash64.h"

index 9a9bedf..b2a8e48 100644
@@ -1,5 +1,6 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
+#include "sysemu/hw_accel.h"
 #include "sysemu/sysemu.h"
 #include "qemu/log.h"
 #include "cpu.h"

@@ -9,7 +10,6 @@
 #include "mmu-hash64.h"
 #include "cpu-models.h"
 #include "trace.h"
-#include "sysemu/kvm.h"
 #include "kvm_ppc.h"
 #include "hw/ppc/spapr_ovec.h"
 

index 0864d9b..4d0775c 100644
@@ -18,6 +18,7 @@
 #include "s390-pci-bus.h"
 #include "exec/memory-internal.h"
 #include "qemu/error-report.h"
+#include "sysemu/hw_accel.h"
 
 /* #define DEBUG_S390PCI_INST */
 #ifdef DEBUG_S390PCI_INST

new file mode 100644
index 0000000..03812cf
@@ -0,0 +1,39 @@
+/*
+ * QEMU Hardware accelertors support
+ *
+ * Copyright 2016 Google, Inc.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef QEMU_HW_ACCEL_H
+#define QEMU_HW_ACCEL_H
+
+#include "qom/cpu.h"
+#include "sysemu/hax.h"
+#include "sysemu/kvm.h"
+
+static inline void cpu_synchronize_state(CPUState *cpu)
+{
+    if (kvm_enabled()) {
+        kvm_cpu_synchronize_state(cpu);
+    }
+}
+
+static inline void cpu_synchronize_post_reset(CPUState *cpu)
+{
+    if (kvm_enabled()) {
+        kvm_cpu_synchronize_post_reset(cpu);
+    }
+}
+
+static inline void cpu_synchronize_post_init(CPUState *cpu)
+{
+    if (kvm_enabled()) {
+        kvm_cpu_synchronize_post_init(cpu);
+    }
+}
+
+#endif /* QEMU_HW_ACCEL_H */

index df67cc0..3045ee7 100644
@@ -461,29 +461,6 @@ void kvm_cpu_synchronize_state(CPUState *cpu);
 void kvm_cpu_synchronize_post_reset(CPUState *cpu);
 void kvm_cpu_synchronize_post_init(CPUState *cpu);
 
-/* generic hooks - to be moved/refactored once there are more users */
-
-static inline void cpu_synchronize_state(CPUState *cpu)
-{
-    if (kvm_enabled()) {
-        kvm_cpu_synchronize_state(cpu);
-    }
-}
-
-static inline void cpu_synchronize_post_reset(CPUState *cpu)
-{
-    if (kvm_enabled()) {
-        kvm_cpu_synchronize_post_reset(cpu);
-    }
-}
-
-static inline void cpu_synchronize_post_init(CPUState *cpu)
-{
-    if (kvm_enabled()) {
-        kvm_cpu_synchronize_post_init(cpu);
-    }
-}
-
 /**
  * kvm_irqchip_add_msi_route - Add MSI route for specific vector
  * @s:      KVM state

index 0841d43..d38956f 100644
@@ -50,7 +50,7 @@
 #include "sysemu/balloon.h"
 #include "qemu/timer.h"
 #include "migration/migration.h"
-#include "sysemu/kvm.h"
+#include "sysemu/hw_accel.h"
 #include "qemu/acl.h"
 #include "sysemu/tpm.h"
 #include "qapi/qmp/qerror.h"

index 03d9190..2c5274f 100644
@@ -22,7 +22,7 @@
 #include "qapi/error.h"
 #include "qemu-common.h"
 #include "qom/cpu.h"
-#include "sysemu/kvm.h"
+#include "sysemu/hw_accel.h"
 #include "qemu/notify.h"
 #include "qemu/log.h"
 #include "exec/log.h"

index 99f0dbe..744cc11 100644
@@ -31,7 +31,7 @@
 #endif
 #include "hw/arm/arm.h"
 #include "sysemu/sysemu.h"
-#include "sysemu/kvm.h"
+#include "sysemu/hw_accel.h"
 #include "kvm_arm.h"
 
 static void arm_cpu_set_pc(CPUState *cs, vaddr value)

index 4ecc091..7530280 100644
@@ -24,6 +24,7 @@
 #include "kvm_i386.h"
 #ifndef CONFIG_USER_ONLY
 #include "sysemu/sysemu.h"
+#include "sysemu/hw_accel.h"
 #include "monitor/monitor.h"
 #include "hw/i386/apic_internal.h"
 #endif

index f62264a..4ed90f6 100644
@@ -23,6 +23,7 @@
 #include "qemu-common.h"
 #include "cpu.h"
 #include "sysemu/sysemu.h"
+#include "sysemu/hw_accel.h"
 #include "sysemu/kvm_int.h"
 #include "kvm_i386.h"
 #include "hyperv.h"

index fdb7a78..0efc8c6 100644
@@ -23,7 +23,7 @@
 #include "exec/exec-all.h"
 #include "exec/helper-proto.h"
 #include "qemu/error-report.h"
-#include "sysemu/kvm.h"
+#include "sysemu/hw_accel.h"
 #include "kvm_ppc.h"
 #include "mmu-hash64.h"
 #include "exec/log.h"

index 626e031..98b8760 100644
@@ -21,10 +21,10 @@
 #include "qemu/osdep.h"
 #include "disas/bfd.h"
 #include "exec/gdbstub.h"
-#include "sysemu/kvm.h"
 #include "kvm_ppc.h"
 #include "sysemu/arch_init.h"
 #include "sysemu/cpus.h"
+#include "sysemu/hw_accel.h"
 #include "cpu-models.h"
 #include "mmu-hash32.h"
 #include "mmu-hash64.h"

index 3d223de..3c652fb 100644
@@ -23,6 +23,7 @@
 #include "exec/exec-all.h"
 #include "exec/gdbstub.h"
 #include "qemu/bitops.h"
+#include "sysemu/hw_accel.h"
 
 int s390_cpu_gdb_read_register(CPUState *cs, uint8_t *mem_buf, int n)
 {

2.8.0.rc3.226.g39d4020

That's a forward port of the core HAX interface code from the emu-2.2-release branch in the external/qemu-android repository as used by the Android emulator.

The original commit was "target-i386: Add Intel HAX to android emulator" saying:
"""
Backport of 2b3098ff27bab079caab9b46b58546b5036f5c0c
from studio-1.4-dev into emu-master-dev

Intel HAX (harware acceleration) will enhance android emulator performance in Windows and Mac OS X in the systems powered by Intel processors with "Intel Hardware Accelerated Execution Manager" package installed when user runs android emulator with Intel target.

"""

It has been modified to build and run along with the current code base. The formatting has been fixed to go through scripts/checkpatch.pl, and the DPRINTF macros have been updated to get the instanciations checked by the compiler.

The FPU registers saving/restoring has been updated to match the current QEMU registers layout.

The implementation has been simplified by doing the following modifications:
- removing the code for supporting the hardware without Unrestricted Guest (UG) mode (including all the code to fallback on TCG emulation).
- not including the Darwin support (which is not yet debugged/tested).
- simplifying the initialization by removing the leftovers from the Android specific code, then trimming down the remaining logic.
- removing the unused MemoryListener callbacks.

---
 hax-stub.c                  |   39 ++
 include/sysemu/hax.h        |   56 +++
 target-i386/hax-all.c       | 1155 +++++++++++++++++++++++++++++++++++++++++++
 target-i386/hax-i386.h      |   86 ++++
 target-i386/hax-interface.h |  361 ++++++++++++++
 target-i386/hax-mem.c       |  271 ++++++++++
 target-i386/hax-windows.c   |  479 ++++++++++++++++++
 target-i386/hax-windows.h   |   89 ++++
 8 files changed, 2536 insertions(+)
 create mode 100644 hax-stub.c
 create mode 100644 include/sysemu/hax.h
 create mode 100644 target-i386/hax-all.c
 create mode 100644 target-i386/hax-i386.h
 create mode 100644 target-i386/hax-interface.h
 create mode 100644 target-i386/hax-mem.c
 create mode 100644 target-i386/hax-windows.c
 create mode 100644 target-i386/hax-windows.h

new file mode 100644
index 0000000..a532dba
@@ -0,0 +1,39 @@
+/*
+ * QEMU HAXM support
+ *
+ * Copyright (c) 2015, Intel Corporation
+ *
+ * Copyright 2016 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "cpu.h"
+#include "sysemu/hax.h"
+
+int hax_sync_vcpus(void)
+{
+    return 0;
+}
+
+int hax_populate_ram(uint64_t va, uint32_t size)
+{
+    return -ENOSYS;
+}
+
+int hax_init_vcpu(CPUState *cpu)
+{
+    return -ENOSYS;
+}
+
+int hax_smp_cpu_exec(CPUState *cpu)
+{
+    return -ENOSYS;
+}

2.8.0.rc3.226.g39d4020

Use the Intel HAX is kernel-based hardware acceleration module for Windows (similar to KVM on Linux).

Based on the "target-i386: Add Intel HAX to android emulator" patch from David Chou <[email protected]>

---
 Makefile.target           |  1 +
 configure                 | 18 ++++++++++
 cpus.c                    | 87 ++++++++++++++++++++++++++++++++++++++++++++++-
 exec.c                    | 16 +++++++++
 hw/intc/apic_common.c     |  3 +-
 include/qom/cpu.h         |  5 +++
 include/sysemu/hw_accel.h |  9 +++++
 qemu-options.hx           | 11 ++++++
 target-i386/Makefile.objs |  4 +++
 vl.c                      | 15 ++++++--
 10 files changed, 164 insertions(+), 5 deletions(-)

index 7a5080e..dab81e7 100644
@@ -96,6 +96,7 @@ obj-y += target-$(TARGET_BASE_ARCH)/
 obj-y += disas.o
 obj-y += tcg-runtime.o
 obj-$(call notempty,$(TARGET_XML_FILES)) += gdbstub-xml.o
+obj-$(call lnot,$(CONFIG_HAX)) += hax-stub.o
 obj-$(call lnot,$(CONFIG_KVM)) += kvm-stub.o
 
 obj-$(CONFIG_LIBDECNUMBER) += libdecnumber/decContext.o

index 3770d7c..ba32bea 100755
@@ -230,6 +230,7 @@ vhost_net="no"
 vhost_scsi="no"
 vhost_vsock="no"
 kvm="no"
+hax="no"
 colo="yes"
 rdma=""
 gprof="no"

@@ -563,6 +564,7 @@ CYGWIN*)
 ;;
 MINGW32*)
   mingw32="yes"
+  hax="yes"
   audio_possible_drivers="dsound sdl"
   if check_include dsound.h; then
     audio_drv_list="dsound"

@@ -612,6 +614,7 @@ OpenBSD)
 Darwin)
   bsd="yes"
   darwin="yes"
+  hax="yes"
   LDFLAGS_SHARED="-bundle -undefined dynamic_lookup"
   if [ "$cpu" = "x86_64" ] ; then
     QEMU_CFLAGS="-arch x86_64 $QEMU_CFLAGS"

@@ -921,6 +924,10 @@ for opt do
   ;;
   --enable-kvm) kvm="yes"
   ;;
+  --disable-hax) hax="no"
+  ;;
+  --enable-hax) hax="yes"
+  ;;
   --disable-colo) colo="no"
   ;;
   --enable-colo) colo="yes"

@@ -1373,6 +1380,7 @@ disabled with --disable-FEATURE, default is enabled if available:
   fdt             fdt device tree
   bluez           bluez stack connectivity
   kvm             KVM acceleration support
+  hax             HAX acceleration support
   colo            COarse-grain LOck-stepping VM for Non-stop Service
   rdma            RDMA-based migration support
   vde             support for vde network

@@ -5051,6 +5059,7 @@ echo "ATTR/XATTR support $attr"
 echo "Install blobs     $blobs"
 echo "KVM support       $kvm"
 echo "COLO support      $colo"
+echo "HAX support       $hax"
 echo "RDMA support      $rdma"
 echo "TCG interpreter   $tcg_interpreter"
 echo "fdt support       $fdt"

@@ -6035,6 +6044,15 @@ case "$target_name" in
       fi
     fi
 esac
+if test "$hax" = "yes" ; then
+  if test "$target_softmmu" = "yes" ; then
+    case "$target_name" in
+    i386|x86_64)
+      echo "CONFIG_HAX=y" >> $config_target_mak
+    ;;
+    esac
+  fi
+fi
 if test "$target_bigendian" = "yes" ; then
   echo "TARGET_WORDS_BIGENDIAN=y" >> $config_target_mak
 fi

index 08c558e..25e393d 100644
@@ -31,6 +31,7 @@
 #include "hw/xen/xen.h"
 #endif
 #include "sysemu/kvm.h"
+#include "sysemu/hax.h"
 #include "sysemu/sysemu.h"
 #include "qemu/timer.h"
 #include "qemu/config-file.h"

@@ -1611,6 +1612,21 @@ static void ram_block_add(RAMBlock *new_block, Error **errp)
                 qemu_mutex_unlock_ramlist();
                 return;
             }
+            /*
+             * In Hax, the qemu allocate the virtual address, and HAX kernel
+             * populate the memory with physical memory. Currently we have no
+             * paging, so user should make sure enough free memory in advance
+             */
+            if (hax_enabled()) {
+                int ret;
+                ret = hax_populate_ram((uint64_t)(uintptr_t)new_block->host,
+                                       new_block->max_length);
+                if (ret < 0) {
+                    error_setg(errp, "Hax failed to populate ram");
+                    return;
+                }
+            }
+
             memory_try_enable_merging(new_block->host, new_block->max_length);
         }
     }

index d78c885..3945dfd 100644
@@ -26,6 +26,7 @@
 #include "hw/i386/apic.h"
 #include "hw/i386/apic_internal.h"
 #include "trace.h"
+#include "sysemu/hax.h"
 #include "sysemu/kvm.h"
 #include "hw/qdev.h"
 #include "hw/sysbus.h"

@@ -316,7 +317,7 @@ static void apic_common_realize(DeviceState *dev, Error **errp)
 
     /* Note: We need at least 1M to map the VAPIC option ROM */
     if (!vapic && s->vapic_control & VAPIC_ENABLE_MASK &&
-        ram_size >= 1024 * 1024) {
+        !hax_enabled() && ram_size >= 1024 * 1024) {
         vapic = sysbus_create_simple("kvmvapic", -1, NULL);
     }
     s->vapic = vapic;

index 3f79a8e..ca4d0fb 100644
@@ -227,6 +227,8 @@ struct CPUWatchpoint {
 struct KVMState;
 struct kvm_run;
 
+struct hax_vcpu_state;
+
 #define TB_JMP_CACHE_BITS 12
 #define TB_JMP_CACHE_SIZE (1 << TB_JMP_CACHE_BITS)
 

@@ -392,6 +394,9 @@ struct CPUState {
        (absolute value) offset as small as possible.  This reduces code
        size, especially for hosts without large memory offsets.  */
     uint32_t tcg_exit_req;
+
+    bool hax_vcpu_dirty;
+    struct hax_vcpu_state *hax_vcpu;
 };
 
 QTAILQ_HEAD(CPUTailQ, CPUState);

index 03812cf..c9b3105 100644
@@ -20,6 +20,9 @@ static inline void cpu_synchronize_state(CPUState *cpu)
     if (kvm_enabled()) {
         kvm_cpu_synchronize_state(cpu);
     }
+    if (hax_enabled()) {
+        hax_cpu_synchronize_state(cpu);
+    }
 }
 
 static inline void cpu_synchronize_post_reset(CPUState *cpu)

@@ -27,6 +30,9 @@ static inline void cpu_synchronize_post_reset(CPUState *cpu)
     if (kvm_enabled()) {
         kvm_cpu_synchronize_post_reset(cpu);
     }
+    if (hax_enabled()) {
+        hax_cpu_synchronize_post_reset(cpu);
+    }
 }
 
 static inline void cpu_synchronize_post_init(CPUState *cpu)

@@ -34,6 +40,9 @@ static inline void cpu_synchronize_post_init(CPUState *cpu)
     if (kvm_enabled()) {
         kvm_cpu_synchronize_post_init(cpu);
     }
+    if (hax_enabled()) {
+        hax_cpu_synchronize_post_init(cpu);
+    }
 }
 
 #endif /* QEMU_HW_ACCEL_H */

index c534a2f..7fc2807 100644
@@ -3280,6 +3280,17 @@ Enable KVM full virtualization support. This option is only available
 if KVM support is enabled when compiling.
 ETEXI
 
+DEF("enable-hax", 0, QEMU_OPTION_enable_hax, \
+    "-enable-hax     enable HAX virtualization support\n", QEMU_ARCH_I386)
+STEXI
+@item -enable-hax
+@findex -enable-hax
+Enable HAX (Hardware-based Acceleration eXecution) support. This option
+is only available if HAX support is enabled when compiling. HAX is only
+applicable to MAC and Windows platform, and thus does not conflict with
+KVM.
+ETEXI
+
 DEF("xen-domid", HAS_ARG, QEMU_OPTION_xen_domid,
     "-xen-domid id   specify xen guest domain id\n", QEMU_ARCH_ALL)
 DEF("xen-create", 0, QEMU_OPTION_xen_create,

index b223d79..acbe7b0 100644
@@ -5,3 +5,7 @@ obj-y += gdbstub.o
 obj-$(CONFIG_SOFTMMU) += machine.o arch_memory_mapping.o arch_dump.o monitor.o
 obj-$(CONFIG_KVM) += kvm.o hyperv.o
 obj-$(call lnot,$(CONFIG_KVM)) += kvm-stub.o
+# HAX support
+ifdef CONFIG_WIN32
+obj-$(CONFIG_HAX) += hax-all.o hax-mem.o hax-windows.o
+endif

index d77dd86..47bbfe7 100644
@@ -92,6 +92,7 @@ int main(int argc, char **argv)
 #include "sysemu/cpus.h"
 #include "migration/colo.h"
 #include "sysemu/kvm.h"
+#include "sysemu/hax.h"
 #include "qapi/qmp/qjson.h"
 #include "qemu/option.h"
 #include "qemu/config-file.h"

@@ -1959,7 +1960,7 @@ static void main_loop(void)
     int64_t ti;
 #endif
     do {
-        nonblocking = !kvm_enabled() && !xen_enabled() && last_io > 0;
+        nonblocking = tcg_enabled() && last_io > 0;
 #ifdef CONFIG_PROFILER
         ti = profile_getclock();
 #endif

@@ -3724,6 +3725,10 @@ int main(int argc, char **argv, char **envp)
                 olist = qemu_find_opts("machine");
                 qemu_opts_parse_noisily(olist, "accel=kvm", false);
                 break;
+            case QEMU_OPTION_enable_hax:
+                olist = qemu_find_opts("machine");
+                qemu_opts_parse_noisily(olist, "accel=hax", false);
+                break;
             case QEMU_OPTION_M:
             case QEMU_OPTION_machine:
                 olist = qemu_find_opts("machine");

@@ -4418,8 +4423,8 @@ int main(int argc, char **argv, char **envp)
 
     cpu_ticks_init();
     if (icount_opts) {
-        if (kvm_enabled() || xen_enabled()) {
-            error_report("-icount is not allowed with kvm or xen");
+        if (!tcg_enabled()) {
+            error_report("-icount is not allowed with hardware virtualization");
             exit(1);
         }
         configure_icount(icount_opts, &error_abort);

@@ -4555,6 +4560,10 @@ int main(int argc, char **argv, char **envp)
 
     numa_post_machine_init();
 
+    if (hax_enabled()) {
+        hax_sync_vcpus();
+    }
+
     if (qemu_opts_foreach(qemu_find_opts("fw_cfg"),
                           parse_fw_cfg, fw_cfg_find(), NULL) != 0) {
         exit(1);

2.8.0.rc3.226.g39d4020

[...]

Sorry I missed it.
I move it to qemu_cpu_kick() as asked in the Darwin patch.

Actually I don't know a good test case to verify such a change, any advice ?

Seems to work, updated for v6.

Vincent

Try computing the latency between qemu_cpu_kick and the exit from HAXM. That is, something like

    int64_t kick_time = -1;
    ...

    atomic_set(&kick_time, qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL));

before SuspendThread or respectively QueueUserAPC, and

    extern volatile int64_t kick_time;
    ...

    int64_t kicked = atomic_xchg(&kick_time, -1);
    if (kicked != -1) {
        printf("%lld\n",
               qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) - kicked);
    }

right after hax_vcpu_run. As long as the printed values remain
smallish, it's okay.

If you don't get any output, passing "info cpus" repeatedly to the QEMU monitor will force a kick.

Paolo

In fact there is a race anyway:

        if (cpu->exit_request) {
            ret = 1;
            break;
        }
					cpu->exit_request
					SuspendThread
					ResumeThread
        hax_vcpu_interrupt(env);
        qemu_mutex_unlock_iothread();
        hax_ret = hax_vcpu_run(vcpu);

and the same race is true for QueueUserAPC. It's rare enough that I guess we can accept the patches with just a FIXME comment, but... Yu Ning, can you tell us what user_event_pending is for? :) My hunch is that we should call hax_raise_event after setting cpu->exit_request, like

	hax_raise_event();
	/* write user_event_pending before exit_request */
	smp_wmb();
	cpu->exit_request = 1;
	SuspendThread/ResumeThread
		(or QueueUserAPC)

and in the hax thread:

        if (cpu->exit_request) {
	    cpu->hax_vcpu->tunnel->user_event_pending = 0;
            ret = 1;
            break;
        }

        hax_vcpu_interrupt(env);
        qemu_mutex_unlock_iothread();

	/* read exit_request before user_event_pending */
	smp_rmb();
        hax_ret = hax_vcpu_run(vcpu);

but I would like some more official documentation than my own reverse engineering of the brain of whoever wrote the interface (I have not looked at the HAXM driver binary).

Paolo

Thanks for the detailed examples and thoughts.
The timing/benchmarking code might actually need some kind of per-vcpu time storage, but that's a detail.
I have experimented with it and so far, I have mainly generated random numbers ...
I have yet to find a use-case where the current code (with SuspendThread/ResumeThread) yields a better latency than just nothing instead :(

:) Does QueueUserAPC generate better latency?

Windows delivers the scheduler tick to the first physical CPU. Try pinning QEMU away from the first CPU.

Paolo

The same kind of random numbers so far.

By the way I have added the THREAD_SET_CONTEXT flag to the OpenThread call in qemu_thread_get_handle() function, as I was getting ACCESS_DENIED on the QueueUserApc call. Probably not terribly harmful. I will publish the v6 series with this and continue my benchmarking quest.

Ok interesting, I will give it a try.

Vincent

Unfortunately, user_event_pending was introduced in 2011 without proper documentation, so I guess even the original author might not have an answer now (even if I could find him). I need more time to analyze the HAXM driver code to understand what it's about. Please feel free to add a FIXME for now.

But one thing I've noticed is that the Darwin driver does not test or reset this flag properly - it will remain 1 after the first hax_raise_event() call. So removing user_event_pending from the QEMU side should not affect Mac.

Re-add the MacOSX/Darwin support:
Use the Intel HAX is kernel-based hardware acceleration module (similar to KVM on Linux).

Based on the original "target-i386: Add Intel HAX to android emulator" patch from David Chou <[email protected]> from emu-2.2-release branch in the external/qemu-android repository.

---
 cpus.c                    |   5 +
 target-i386/Makefile.objs |   3 +
 target-i386/hax-darwin.c  | 316 ++++++++++++++++++++++++++++++++++++++++++++++
 target-i386/hax-darwin.h  |  63 +++++++++
 target-i386/hax-i386.h    |   8 ++
 5 files changed, 395 insertions(+)
 create mode 100644 target-i386/hax-darwin.c
 create mode 100644 target-i386/hax-darwin.h

index 0e01791..b8db313 100644
@@ -1264,6 +1264,11 @@ static void qemu_cpu_kick_thread(CPUState *cpu)
         return;
     }
     cpu->thread_kicked = true;
+#ifdef CONFIG_DARWIN
+    if (hax_enabled()) {
+        cpu->exit_request = 1;
+    }
+#endif
     err = pthread_kill(cpu->thread->thread, SIG_IPI);
     if (err) {
         fprintf(stderr, "qemu:%s: %s", __func__, strerror(err));

index acbe7b0..4fcb7f3 100644
@@ -9,3 +9,6 @@ obj-$(call lnot,$(CONFIG_KVM)) += kvm-stub.o
 ifdef CONFIG_WIN32
 obj-$(CONFIG_HAX) += hax-all.o hax-mem.o hax-windows.o
 endif
+ifdef CONFIG_DARWIN
+obj-$(CONFIG_HAX) += hax-all.o hax-mem.o hax-darwin.o
+endif

index 3f50bd9..b6becbd 100644
@@ -16,6 +16,10 @@
 #include "cpu.h"
 #include "sysemu/hax.h"
 
+#ifdef CONFIG_DARWIN
+typedef int hax_fd;
+#endif
+
 #ifdef CONFIG_WIN32
 typedef HANDLE hax_fd;
 #endif

@@ -77,6 +81,10 @@ hax_fd hax_mod_open(void);
 void hax_memory_region_init(void);
 
 
+#ifdef CONFIG_DARWIN
+#include "target-i386/hax-darwin.h"
+#endif
+
 #ifdef CONFIG_WIN32
 #include "target-i386/hax-windows.h"
 #endif

2.8.0.rc3.226.g39d4020

Done in v6