Skip to contentSkip to conversation

Series short description

RFC PATCH Superseded Pavel Dovgalyuk 5 participants 83 messages Jul 17, 2014 – Jul 31, 2014 3 messages not in the archive

Subsystems (4)

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v2 cover Series short description

This set of patches is related to the reverse execution and deterministic replay of qemu execution Our implementation of deterministic replay can be used for deterministic and reverse debugging of guest code through gdb remote interface.

Execution recording writes non-deterministic events log, which can be later used for replaying the execution anywhere and for unlimited number of times. It also supports checkpointing for faster rewinding during reverse debugging. Execution replaying reads the log and replays all non-deterministic events including external input, hardware clocks, and interrupts.

Reverse execution has the following features:
* Deterministically replays whole system execution and all contents of the memory, state of the hadrware devices, clocks, and screen of the VM.
* Writes execution log into the file for latter replaying for multiple times on different machines.
* Supports i386, x86_64, and ARM hardware platforms.
* Performs deterministic replay of all operations with keyboard, mouse, network adapters, audio devices, serial interfaces, and physical USB devices connected to the emulator.
* Provides support for gdb reverse debugging commands like reverse-step and reverse-continue.
* Supports auto-checkpointing for convenient reverse debugging.

Usage of the record/replay:
* First, record the execution, by adding '-record fname=replay.bin' to the command line.
* Then you can replay it for the multiple times by using another command line option: '-replay fname=replay.bin'
* Virtual machine should have at least one virtual disk, which is used to store checkpoints. If you want to enable automatic checkpointing, simply add ',period=XX' to record options, where XX is the checkpointing period in seconds.
* Using of the network adapters in record/replay mode is possible with the following command-line options:
- '-net user' (or another host adapter) in record mode
- '-net replay' in replay mode. Every host network adapter should be replaced by 'replay' when replaying the execution.
* Reverse debugging can be used through gdb remote interface. reverse-stepi and reverse-continue commands are supported. Other reverse commands should also work, because they reuse these ones.
* Monitor is extended by the following commands:
- replay_info - prints information about replay mode and current step (number of instructions executed)
- replay_break - sets "breakpoint" at the specified instructions count.
- replay_seek - rewinds (using the checkpoints, if possible) to the specified step of replay log.

Paper with short description of deterministic replay implementation:
http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html

Modifications of qemu include:
* adding missed fields of the virtual devices' states to the vmstate structures to allow deterministic saving and restoring the VM state
* adding virtual clock-based timers to vmstate structures, because virtual clock is the part of the virtual machine state
* modification of block layer to support automatic creation of the overlay files to store the changes and snapshots while recording
* disabling of system reset while loading VM state to avoid generating of interrupts by reset handlers
* adding warpers for clock and time functions to save their return values in the log
* saving different asynchronous events (e.g. system shutdown) into the log
* synchronization of the bottom halves execution
* synchronization of the threads from thread pool
* recording/replaying user input (mouse and keyboard), input from virtual serial ports, incoming network packets, input from connected USB devices
* adding HMP/QMP commands to monitor for controlling replay execution

v2 changes:
* Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
* Added QMP versions of replay commands (as suggested by Eric Blake)
* Removed some optional features of replay to make patches cleaner
* Minor changes and code cleanup were made

---

Pavel Dovgalyuk (49):
acpi: accurate overflow check
integratorcp: adding vmstate for save/restore
pcspk: adding vmstate for save/restore
fdc: adding vmstate for save/restore
parallel: adding vmstate for save/restore
serial: fixing vmstate for save/restore
kvmapic: fixing loading vmstate
hpet: fixing saving and loading process
pckbd: adding new fields to vmstate
rtl8139: adding new fields to vmstate
piix: do not raise irq while loading vmstate
mc146818rtc: add missed field to vmstate
pl031: add missed field to vmstate
ide pci: reset status field before loading the vmstate
softmmu: fixing usage of cpu_st/ld* from helpers
target: save cpu state fields
target-i386: update fp status fix
migration: add vmstate for int8 and char arrays
replay: global variables and function stubs
block: add suffix parameter to bdrv_open functions
sysemu: system functions for replay
replay: internal functions for replay log
cpu: invent instruction count for accurate replay
target-arm: instructions counting code for replay
target-i386: instructions counting code for replay
replay: interrupts and exceptions
vga: do not use virtual clock for blinking cursor
replay: asynchronous events infrastructure
replay: recording and replaying clock ticks
replay: recording and replaying different timers
replay: shutdown event
replay: checkpoints
replay: bottom halves
replay: replay aio requests
replay: thread pool
pl031: vmstate in replay mode
replay: initialization and deinitialization
replay: command line options
replay: snapshotting the virtual machine
replay: recording of the user input
tap-win32: destroy the thread at exit
replay: network packets record/replay
replay: audio data record/replay
replay: serial port
replay: USB passthrough
replay: replay_info command
replay: replay_break command
replay: replay_seek_step command
gdbstub: reverse debugging

 Makefile.target                  |    1 
 arch_init.c                      |    8 
 async.c                          |   45 ++
 audio/audio.c                    |   14 +
 audio/audio_win_int.h            |    3 
 audio/winwaveaudio.c             |  167 +++++++--
 block.c                          |  134 ++++++-
 block/blkdebug.c                 |    2 
 block/blkverify.c                |    4 
 block/cow.c                      |    2 
 block/qcow.c                     |    2 
 block/qcow2.c                    |   10 -
 block/qed.c                      |    2 
 block/raw-posix.c                |    6 
 block/raw-win32.c                |    4 
 block/sheepdog.c                 |    4 
 block/vmdk.c                     |    8 
 block/vvfat.c                    |    2 
 blockdev.c                       |   11 -
 cpu-exec.c                       |   34 +-
 cpus.c                           |   81 ++++
 dma-helpers.c                    |   10 -
 exec.c                           |   12 +
 gdbstub.c                        |   79 +++-
 hmp-commands.hx                  |   41 ++
 hw/acpi/core.c                   |    7 
 hw/arm/integratorcp.c            |   38 ++
 hw/audio/pcspk.c                 |   19 +
 hw/block/fdc.c                   |   11 -
 hw/block/virtio-blk.c            |   10 -
 hw/char/parallel.c               |   22 +
 hw/char/serial.c                 |  115 ++++--
 hw/display/vga.c                 |    4 
 hw/i386/kvmvapic.c               |   22 +
 hw/ide/ahci.c                    |    4 
 hw/ide/atapi.c                   |    9 
 hw/ide/core.c                    |   18 +
 hw/ide/pci.c                     |   10 +
 hw/input/pckbd.c                 |    3 
 hw/intc/apic_common.c            |    5 
 hw/net/rtl8139.c                 |    5 
 hw/pci-host/piix.c               |   22 +
 hw/timer/arm_timer.c             |    2 
 hw/timer/hpet.c                  |   13 -
 hw/timer/mc146818rtc.c           |    5 
 hw/timer/pl031.c                 |   35 +-
 hw/usb/hcd-uhci.c                |    2 
 hw/usb/host-libusb.c             |  525 ++++++++++++++++++---------
 include/block/aio.h              |   17 +
 include/block/block.h            |   21 +
 include/block/thread-pool.h      |    4 
 include/exec/cpu-defs.h          |    1 
 include/exec/cpu_ldst_template.h |   28 +
 include/exec/exec-all.h          |   31 ++
 include/hw/host-libusb.h         |  105 +++++
 include/migration/vmstate.h      |   13 +
 include/qemu-common.h            |    3 
 include/qemu/main-loop.h         |    1 
 include/qemu/timer.h             |   48 ++
 include/qom/cpu.h                |    4 
 include/sysemu/char.h            |   25 +
 include/sysemu/cpus.h            |    1 
 include/sysemu/sysemu.h          |    2 
 include/ui/input.h               |    2 
 main-loop.c                      |    5 
 monitor.c                        |   49 +++
 net/Makefile.objs                |    1 
 net/clients.h                    |    3 
 net/dump.c                       |    6 
 net/hub.c                        |    1 
 net/net-replay.c                 |   66 +++
 net/net.c                        |    7 
 net/slirp.c                      |   14 +
 net/socket.c                     |   35 ++
 net/tap-win32.c                  |   25 +
 net/tap.c                        |   23 +
 net/vde.c                        |   14 +
 qapi-schema.json                 |   62 +++
 qemu-char.c                      |   55 +++
 qemu-img.c                       |    6 
 qemu-io-cmds.c                   |    2 
 qemu-io.c                        |    4 
 qemu-nbd.c                       |    2 
 qemu-options.hx                  |   28 +
 qemu-timer.c                     |   45 ++
 qmp-commands.hx                  |   60 +++
 replay/Makefile.objs             |   11 +
 replay/replay-audio.c            |  228 ++++++++++++
 replay/replay-char.c             |   99 +++++
 replay/replay-debug.c            |  148 ++++++++
 replay/replay-events.c           |  381 ++++++++++++++++++++
 replay/replay-input.c            |  107 ++++++
 replay/replay-internal.c         |  159 ++++++++
 replay/replay-internal.h         |  224 ++++++++++++
 replay/replay-net.c              |  190 ++++++++++
 replay/replay-qmp.c              |   57 +++
 replay/replay-time.c             |  181 +++++++++
 replay/replay-usb.c              |  188 ++++++++++
 replay/replay.c                  |  735 ++++++++++++++++++++++++++++++++++++++
 replay/replay.h                  |  218 +++++++++++
 savevm.c                         |   32 +-
 slirp/slirp.c                    |    9 
 softmmu_template.h               |   18 +
 stubs/Makefile.objs              |    1 
 stubs/replay.c                   |   42 ++
 target-arm/Makefile.objs         |    1 
 target-arm/helper.h              |    3 
 target-arm/machine.c             |    5 
 target-arm/replay_helper.c       |   38 ++
 target-arm/translate.c           |   62 +++
 target-i386/Makefile.objs        |    1 
 target-i386/cpu.c                |    1 
 target-i386/cpu.h                |    8 
 target-i386/fpu_helper.c         |    5 
 target-i386/helper.h             |    3 
 target-i386/machine.c            |    7 
 target-i386/replay_helper.c      |   38 ++
 target-i386/translate.c          |   91 ++++-
 tests/test-thread-pool.c         |    7 
 thread-pool.c                    |   53 ++-
 trace-events                     |    2 
 translate-all.c                  |   15 +
 ui/input.c                       |   79 +++-
 util/iov.c                       |    4 
 vl.c                             |  108 +++++-
 vmstate.c                        |    6 
 126 files changed, 5433 insertions(+), 543 deletions(-)
 create mode 100755 include/hw/host-libusb.h
 create mode 100755 net/net-replay.c
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay-audio.c
 create mode 100755 replay/replay-char.c
 create mode 100755 replay/replay-debug.c
 create mode 100755 replay/replay-events.c
 create mode 100755 replay/replay-input.c
 create mode 100755 replay/replay-internal.c
 create mode 100755 replay/replay-internal.h
 create mode 100755 replay/replay-net.c
 create mode 100755 replay/replay-qmp.c
 create mode 100755 replay/replay-time.c
 create mode 100755 replay/replay-usb.c
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c
 create mode 100755 target-arm/replay_helper.c
 create mode 100755 target-i386/replay_helper.c
Signature · 1 line
Pavel Dovgalyuk
#1 lore raw

RFC PATCH v2 1/49 acpi: accurate overflow check

Compare clock in ns, because acpi_pm_tmr_update uses rounded to ns value instead of ticks.

---
 hw/acpi/core.c |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)
hw/acpi/core.c +5 -2
index a7368fb..f7272b4 100644
@@ -376,8 +376,11 @@ static void acpi_notify_wakeup(Notifier *notifier, void *data)
 /* ACPI PM1a EVT */
 uint16_t acpi_pm1_evt_get_sts(ACPIREGS *ar)
 {
-    int64_t d = acpi_pm_tmr_get_clock();
-    if (d >= ar->tmr.overflow_time) {
+    /* Compare ns-clock, not PM timer ticks, because 
+       acpi_pm_tmr_update function uses ns for setting the timer. */
+    int64_t d = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
+    if (d >= muldiv64(ar->tmr.overflow_time, 
+                      get_ticks_per_sec(), PM_TIMER_FREQUENCY)) {
         ar->pm1.evt.sts |= ACPI_BITMASK_TIMER_STATUS;
     }
     return ar->pm1.evt.sts;
#2 lore raw

RFC PATCH v2 2/49 integratorcp: adding vmstate for save/restore

In reply to Pavel Dovgalyuk

VMState added by this patch preserves correct
loading of the integratorcp device state.

---
 hw/arm/integratorcp.c |   38 +++++++++++++++++++++++++++++++++++++-
 1 files changed, 37 insertions(+), 1 deletions(-)
hw/arm/integratorcp.c +37 -1
index 0e476c3..496f84e 100644
@@ -42,6 +42,27 @@ typedef struct IntegratorCMState {
     uint32_t fiq_enabled;
 } IntegratorCMState;
 
+static const VMStateDescription vmstate_integratorcm = {
+    .name = "integratorcm",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .minimum_version_id_old = 1,
+    .fields      = (VMStateField []) {
+        VMSTATE_UINT32(cm_osc, IntegratorCMState),
+        VMSTATE_UINT32(cm_ctrl, IntegratorCMState),
+        VMSTATE_UINT32(cm_lock, IntegratorCMState),
+        VMSTATE_UINT32(cm_auxosc, IntegratorCMState),
+        VMSTATE_UINT32(cm_sdram, IntegratorCMState),
+        VMSTATE_UINT32(cm_init, IntegratorCMState),
+        VMSTATE_UINT32(cm_flags, IntegratorCMState),
+        VMSTATE_UINT32(cm_nvflags, IntegratorCMState),
+        VMSTATE_UINT32(int_level, IntegratorCMState),
+        VMSTATE_UINT32(irq_enabled, IntegratorCMState),
+        VMSTATE_UINT32(fiq_enabled, IntegratorCMState),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
 static uint8_t integrator_spd[128] = {
    128, 8, 4, 11, 9, 1, 64, 0,  2, 0xa0, 0xa0, 0, 0, 8, 0, 1,
    0xe, 4, 0x1c, 1, 2, 0x20, 0xc0, 0, 0, 0, 0, 0x30, 0x28, 0x30, 0x28, 0x40
@@ -272,7 +293,7 @@ static int integratorcm_init(SysBusDevice *dev)
     sysbus_init_mmio(dev, &s->iomem);
 
     integratorcm_do_remap(s);
-    /* ??? Save/restore.  */
+    vmstate_register(NULL, -1, &vmstate_integratorcm, s);
     return 0;
 }
 
@@ -296,6 +317,20 @@ typedef struct icp_pic_state {
     qemu_irq parent_fiq;
 } icp_pic_state;
 
+
+static const VMStateDescription vmstate_icp_pic = {
+    .name = "icp_pic_state",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .minimum_version_id_old = 1,
+    .fields      = (VMStateField []) {
+        VMSTATE_UINT32(level, icp_pic_state),
+        VMSTATE_UINT32(irq_enabled, icp_pic_state),
+        VMSTATE_UINT32(fiq_enabled, icp_pic_state),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
 static void icp_pic_update(icp_pic_state *s)
 {
     uint32_t flags;
@@ -399,6 +434,7 @@ static int icp_pic_init(SysBusDevice *sbd)
     memory_region_init_io(&s->iomem, OBJECT(s), &icp_pic_ops, s,
                           "icp-pic", 0x00800000);
     sysbus_init_mmio(sbd, &s->iomem);
+    vmstate_register(NULL, -1, &vmstate_icp_pic, s);
     return 0;
 }
 
#3 lore raw

RFC PATCH v2 3/49 pcspk: adding vmstate for save/restore

In reply to Pavel Dovgalyuk

VMState added by this patch preserves correct
loading of the PC speaker device state.

---
 hw/audio/pcspk.c |   19 +++++++++++++++++--
 1 files changed, 17 insertions(+), 2 deletions(-)
hw/audio/pcspk.c +17 -2
index 1d81bbe..2afcffb 100644
@@ -50,8 +50,8 @@ typedef struct {
     unsigned int pit_count;
     unsigned int samples;
     unsigned int play_pos;
-    int data_on;
-    int dummy_refresh_clock;
+    uint8_t data_on;
+    uint8_t dummy_refresh_clock;
 } PCSpkState;
 
 static const char *s_spk = "pcspk";
@@ -163,6 +163,19 @@ static const MemoryRegionOps pcspk_io_ops = {
     },
 };
 
+static const VMStateDescription vmstate_spk = {
+    .name = "pcspk",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .minimum_version_id_old = 1,
+    .fields      = (VMStateField []) {
+        //VMSTATE_UINT32(play_pos, PCSpkState),
+        VMSTATE_UINT8(data_on, PCSpkState),
+        VMSTATE_UINT8(dummy_refresh_clock, PCSpkState),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
 static void pcspk_initfn(Object *obj)
 {
     PCSpkState *s = PC_SPEAKER(obj);
@@ -175,6 +188,8 @@ static void pcspk_realizefn(DeviceState *dev, Error **errp)
     ISADevice *isadev = ISA_DEVICE(dev);
     PCSpkState *s = PC_SPEAKER(dev);
 
+    vmstate_register(NULL, 0, &vmstate_spk, s);
+
     isa_register_ioport(isadev, &s->ioport, s->iobase);
 
     pcspk_state = s;
#4 lore raw

RFC PATCH v2 4/49 fdc: adding vmstate for save/restore

In reply to Pavel Dovgalyuk

VMState added by this patch preserves correct
loading of the FDC device state.

---
 hw/block/fdc.c |   11 +++++++++--
 1 files changed, 9 insertions(+), 2 deletions(-)
hw/block/fdc.c +9 -2
index 490d127..132310a 100644
@@ -697,12 +697,17 @@ static const VMStateDescription vmstate_fdrive_media_rate = {
 
 static const VMStateDescription vmstate_fdrive = {
     .name = "fdrive",
-    .version_id = 1,
+    .version_id = 2,
     .minimum_version_id = 1,
     .fields = (VMStateField[]) {
         VMSTATE_UINT8(head, FDrive),
         VMSTATE_UINT8(track, FDrive),
         VMSTATE_UINT8(sect, FDrive),
+        VMSTATE_UINT8_V(last_sect, FDrive, 2),
+        VMSTATE_UINT8_V(max_track, FDrive, 2),
+        VMSTATE_UINT16_V(bps, FDrive, 2),
+        VMSTATE_UINT8_V(ro, FDrive, 2),
+        VMSTATE_UINT8_V(perpendicular, FDrive, 2),
         VMSTATE_END_OF_LIST()
     },
     .subsections = (VMStateSubsection[]) {
@@ -736,7 +741,7 @@ static int fdc_post_load(void *opaque, int version_id)
 
 static const VMStateDescription vmstate_fdc = {
     .name = "fdc",
-    .version_id = 2,
+    .version_id = 3,
     .minimum_version_id = 2,
     .pre_save = fdc_pre_save,
     .post_load = fdc_post_load,
@@ -769,6 +774,8 @@ static const VMStateDescription vmstate_fdc = {
         VMSTATE_UINT8_EQUAL(num_floppies, FDCtrl),
         VMSTATE_STRUCT_ARRAY(drives, FDCtrl, MAX_FD, 1,
                              vmstate_fdrive, FDrive),
+        VMSTATE_INT32_V(reset_sensei, FDCtrl, 3),
+        VMSTATE_TIMER_V(result_timer, FDCtrl, 3),
         VMSTATE_END_OF_LIST()
     }
 };
#5 lore raw

Re: [Qemu-devel] [RFC PATCH v2 04/49] fdc: adding vmstate for save/restore

Il 17/07/2014 13:02, Pavel Dovgalyuk ha scritto:

Quoted 18 lines

VMState added by this patch preserves correct
loading of the FDC device state.

---
 hw/block/fdc.c |   11 +++++++++--
 1 files changed, 9 insertions(+), 2 deletions(-)
hw/block/fdc.c +1 -1
index 490d127..132310a 100644
@@ -697,12 +697,17 @@ static const VMStateDescription vmstate_fdrive_media_rate = {
 
 static const VMStateDescription vmstate_fdrive = {
     .name = "fdrive",
-    .version_id = 1,
+    .version_id = 2,

hw/block/fdc.c static const VMStateDescription vmstate_fdrive_media_rate = { · line 710

Quoted 10 lines
hw/block/fdc.c +5 -0
     .minimum_version_id = 1,
     .fields = (VMStateField[]) {
         VMSTATE_UINT8(head, FDrive),
         VMSTATE_UINT8(track, FDrive),
         VMSTATE_UINT8(sect, FDrive),
+        VMSTATE_UINT8_V(last_sect, FDrive, 2),
+        VMSTATE_UINT8_V(max_track, FDrive, 2),
+        VMSTATE_UINT16_V(bps, FDrive, 2),
+        VMSTATE_UINT8_V(ro, FDrive, 2),
+        VMSTATE_UINT8_V(perpendicular, FDrive, 2),

Perpendicular can be added to a subsection, migrated only if nonzero. The others can be reconstructed by calling fd_revalidate in vmstate_fdrive's post_load callback.

Quoted 13 lines
hw/block/fdc.c +1 -1
         VMSTATE_END_OF_LIST()
     },
     .subsections = (VMStateSubsection[]) {
@@ -736,7 +741,7 @@ static int fdc_post_load(void *opaque, int version_id)
 
 static const VMStateDescription vmstate_fdc = {
     .name = "fdc",
-    .version_id = 2,
+    .version_id = 3,
     .minimum_version_id = 2,
     .pre_save = fdc_pre_save,
     .post_load = fdc_post_load,

hw/block/fdc.c static const VMStateDescription vmstate_fdc = { · line 777

Quoted 4 lines
hw/block/fdc.c +1 -0
@@ -769,6 +774,8 @@ static const VMStateDescription vmstate_fdc = {
         VMSTATE_UINT8_EQUAL(num_floppies, FDCtrl),
         VMSTATE_STRUCT_ARRAY(drives, FDCtrl, MAX_FD, 1,
                              vmstate_fdrive, FDrive),
+        VMSTATE_INT32_V(reset_sensei, FDCtrl, 3),

Subsection, only migrated if nonzero.

hw/block/fdc.c static const VMStateDescription vmstate_fdc = { · line 778

Quoted 1 line
hw/block/fdc.c +1 -0
+        VMSTATE_TIMER_V(result_timer, FDCtrl, 3),

Subsection, only migrated if pending.

Paolo

Quoted 6 lines
hw/block/fdc.c +0 -0
         VMSTATE_END_OF_LIST()
     }
 };
#6 lore raw

RFC PATCH v2 5/49 parallel: adding vmstate for save/restore

In reply to Pavel Dovgalyuk

VMState added by this patch preserves correct
loading of the parallel port controller state.

---
 hw/char/parallel.c |   22 ++++++++++++++++++++++
 1 files changed, 22 insertions(+), 0 deletions(-)
hw/char/parallel.c +22 -0
index 7ac90a5..bc34e55 100644
@@ -477,6 +477,26 @@ static const MemoryRegionPortio isa_parallel_portio_sw_list[] = {
     PORTIO_END_OF_LIST(),
 };
 
+
+static const VMStateDescription vmstate_parallel_isa = {
+    .name = "parallel_isa",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .minimum_version_id_old = 1,
+    .fields      = (VMStateField []) {
+        VMSTATE_UINT8(state.dataw, ISAParallelState),
+        VMSTATE_UINT8(state.datar, ISAParallelState),
+        VMSTATE_UINT8(state.status, ISAParallelState),
+        VMSTATE_UINT8(state.control, ISAParallelState),
+        VMSTATE_INT32(state.irq_pending, ISAParallelState),
+        VMSTATE_INT32(state.hw_driver, ISAParallelState),
+        VMSTATE_INT32(state.epp_timeout, ISAParallelState),
+        VMSTATE_INT32(state.it_shift, ISAParallelState),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
+
 static void parallel_isa_realizefn(DeviceState *dev, Error **errp)
 {
     static int index;
@@ -518,6 +538,8 @@ static void parallel_isa_realizefn(DeviceState *dev, Error **errp)
                               ? &isa_parallel_portio_hw_list[0]
                               : &isa_parallel_portio_sw_list[0]),
                              s, "parallel");
+
+    vmstate_register(NULL, -1, &vmstate_parallel_isa, isa);
 }
 
 /* Memory mapped interface */
#7 lore raw

Re: [Qemu-devel] [RFC PATCH v2 05/49] parallel: adding vmstate for save/restore

Il 17/07/2014 13:02, Pavel Dovgalyuk ha scritto:

hw/char/parallel.c static const MemoryRegionPortio isa_parallel_portio_sw_list[] = { · line 492

Quoted 7 lines
hw/char/parallel.c +7 -0
+    .fields      = (VMStateField []) {
+        VMSTATE_UINT8(state.dataw, ISAParallelState),
+        VMSTATE_UINT8(state.datar, ISAParallelState),
+        VMSTATE_UINT8(state.status, ISAParallelState),
+        VMSTATE_UINT8(state.control, ISAParallelState),
+        VMSTATE_INT32(state.irq_pending, ISAParallelState),
+        VMSTATE_INT32(state.hw_driver, ISAParallelState),

Static, doesn't need migration.

hw/char/parallel.c static const MemoryRegionPortio isa_parallel_portio_sw_list[] = { · line 494

Quoted 2 lines
hw/char/parallel.c +2 -0
+        VMSTATE_INT32(state.epp_timeout, ISAParallelState),
+        VMSTATE_INT32(state.it_shift, ISAParallelState),

Static, doesn't need migration.

Quoted 1 line

+ VMSTATE_END_OF_LIST()

#8 lore raw

RFC PATCH v2 6/49 serial: fixing vmstate for save/restore

In reply to Pavel Dovgalyuk

Some fields were added to VMState by this patch to preserve correct loading of the serial port controller state.
Updating FCR value while loading was also modified to disable generating an interrupt by loadvm.

---
 hw/char/serial.c |  115 ++++++++++++++++++++++++++++++++----------------------
 1 files changed, 69 insertions(+), 46 deletions(-)
hw/char/serial.c +69 -46
index 54180a9..1969723 100644
@@ -267,6 +267,61 @@ static gboolean serial_xmit(GIOChannel *chan, GIOCondition cond, void *opaque)
 }
 
 
+/* Setter for FCR.
+   is_load flag means, that value is set while loading VM state
+   and interrupt should not be invoked */
+static void serial_write_fcr(void *opaque, uint32_t val, int is_load)
+{
+    SerialState *s = opaque;
+    val = val & 0xFF;
+
+    if (s->fcr == val)
+        return;
+
+    /* Did the enable/disable flag change? If so, make sure FIFOs get flushed */
+    if ((val ^ s->fcr) & UART_FCR_FE)
+        val |= UART_FCR_XFR | UART_FCR_RFR;
+
+    /* FIFO clear */
+
+    if (val & UART_FCR_RFR) {
+        timer_del(s->fifo_timeout_timer);
+        s->timeout_ipending=0;
+        fifo8_reset(&s->recv_fifo);
+    }
+
+    if (val & UART_FCR_XFR) {
+        fifo8_reset(&s->xmit_fifo);
+    }
+
+    if (val & UART_FCR_FE) {
+        s->iir |= UART_IIR_FE;
+        /* Set recv_fifo trigger Level */
+        switch (val & 0xC0) {
+        case UART_FCR_ITL_1:
+            s->recv_fifo_itl = 1;
+            break;
+        case UART_FCR_ITL_2:
+            s->recv_fifo_itl = 4;
+            break;
+        case UART_FCR_ITL_3:
+            s->recv_fifo_itl = 8;
+            break;
+        case UART_FCR_ITL_4:
+            s->recv_fifo_itl = 14;
+            break;
+        }
+    } else
+        s->iir &= ~UART_IIR_FE;
+
+    /* Set fcr - or at least the bits in it that are supposed to "stick" */
+    s->fcr = val & 0xC9;
+
+    if (!is_load) {
+        serial_update_irq(s);
+    }
+}
+
 static void serial_ioport_write(void *opaque, hwaddr addr, uint64_t val,
                                 unsigned size)
 {
@@ -320,50 +375,7 @@ static void serial_ioport_write(void *opaque, hwaddr addr, uint64_t val,
         }
         break;
     case 2:
-        val = val & 0xFF;
-
-        if (s->fcr == val)
-            break;
-
-        /* Did the enable/disable flag change? If so, make sure FIFOs get flushed */
-        if ((val ^ s->fcr) & UART_FCR_FE)
-            val |= UART_FCR_XFR | UART_FCR_RFR;
-
-        /* FIFO clear */
-
-        if (val & UART_FCR_RFR) {
-            timer_del(s->fifo_timeout_timer);
-            s->timeout_ipending=0;
-            fifo8_reset(&s->recv_fifo);
-        }
-
-        if (val & UART_FCR_XFR) {
-            fifo8_reset(&s->xmit_fifo);
-        }
-
-        if (val & UART_FCR_FE) {
-            s->iir |= UART_IIR_FE;
-            /* Set recv_fifo trigger Level */
-            switch (val & 0xC0) {
-            case UART_FCR_ITL_1:
-                s->recv_fifo_itl = 1;
-                break;
-            case UART_FCR_ITL_2:
-                s->recv_fifo_itl = 4;
-                break;
-            case UART_FCR_ITL_3:
-                s->recv_fifo_itl = 8;
-                break;
-            case UART_FCR_ITL_4:
-                s->recv_fifo_itl = 14;
-                break;
-            }
-        } else
-            s->iir &= ~UART_IIR_FE;
-
-        /* Set fcr - or at least the bits in it that are supposed to "stick" */
-        s->fcr = val & 0xC9;
-        serial_update_irq(s);
+        serial_write_fcr(s, val, 0);
         break;
     case 3:
         {
@@ -591,20 +603,22 @@ static int serial_post_load(void *opaque, int version_id)
         s->fcr_vmstate = 0;
     }
     /* Initialize fcr via setter to perform essential side-effects */
-    serial_ioport_write(s, 0x02, s->fcr_vmstate, 1);
+    serial_write_fcr(s, s->fcr_vmstate, 1);
     serial_update_parameters(s);
     return 0;
 }
 
 const VMStateDescription vmstate_serial = {
     .name = "serial",
-    .version_id = 3,
+    .version_id = 4,
     .minimum_version_id = 2,
     .pre_save = serial_pre_save,
     .post_load = serial_post_load,
     .fields = (VMStateField[]) {
         VMSTATE_UINT16_V(divider, SerialState, 2),
         VMSTATE_UINT8(rbr, SerialState),
+        VMSTATE_UINT8_V(thr, SerialState, 4),
+        VMSTATE_UINT8_V(tsr, SerialState, 4),
         VMSTATE_UINT8(ier, SerialState),
         VMSTATE_UINT8(iir, SerialState),
         VMSTATE_UINT8(lcr, SerialState),
@@ -613,6 +627,15 @@ const VMStateDescription vmstate_serial = {
         VMSTATE_UINT8(msr, SerialState),
         VMSTATE_UINT8(scr, SerialState),
         VMSTATE_UINT8_V(fcr_vmstate, SerialState, 3),
+        VMSTATE_INT32_V(thr_ipending, SerialState, 4),
+        VMSTATE_INT32_V(last_break_enable, SerialState, 4),
+        VMSTATE_INT32_V(tsr_retry, SerialState, 4),
+        VMSTATE_STRUCT(recv_fifo, SerialState, 4, vmstate_fifo8, Fifo8),
+        VMSTATE_STRUCT(xmit_fifo, SerialState, 4, vmstate_fifo8, Fifo8),
+        VMSTATE_TIMER_V(fifo_timeout_timer, SerialState, 4),
+        VMSTATE_INT32_V(timeout_ipending, SerialState, 4),
+        VMSTATE_INT32_V(poll_msl, SerialState, 4),
+        VMSTATE_TIMER_V(modem_status_poll, SerialState, 4),
         VMSTATE_END_OF_LIST()
     }
 };
#9 lore raw

Re: [Qemu-devel] [RFC PATCH v2 06/49] serial: fixing vmstate for save/restore

Il 17/07/2014 13:02, Pavel Dovgalyuk ha scritto:

Quoted 14 lines
hw/char/serial.c +3 -1
-    .version_id = 3,
+    .version_id = 4,
     .minimum_version_id = 2,
     .pre_save = serial_pre_save,
     .post_load = serial_post_load,
     .fields = (VMStateField[]) {
         VMSTATE_UINT16_V(divider, SerialState, 2),
         VMSTATE_UINT8(rbr, SerialState),
+        VMSTATE_UINT8_V(thr, SerialState, 4),
+        VMSTATE_UINT8_V(tsr, SerialState, 4),
         VMSTATE_UINT8(ier, SerialState),
         VMSTATE_UINT8(iir, SerialState),
         VMSTATE_UINT8(lcr, SerialState),

hw/char/serial.c const VMStateDescription vmstate_serial = { · line 630

Quoted 4 lines
hw/char/serial.c +1 -0
@@ -613,6 +627,15 @@ const VMStateDescription vmstate_serial = {
         VMSTATE_UINT8(msr, SerialState),
         VMSTATE_UINT8(scr, SerialState),
         VMSTATE_UINT8_V(fcr_vmstate, SerialState, 3),
+        VMSTATE_INT32_V(thr_ipending, SerialState, 4),

Subsection, only migrated if it doesn't match "(s->iir & UART_IIR_ID) == UART_IIR_THRI".

hw/char/serial.c const VMStateDescription vmstate_serial = { · line 631

Quoted 1 line
hw/char/serial.c +1 -0
+        VMSTATE_INT32_V(last_break_enable, SerialState, 4),

Can be reconstructed in the post_load callback from s->lcr.

hw/char/serial.c const VMStateDescription vmstate_serial = { · line 632

Quoted 1 line
hw/char/serial.c +1 -0
+        VMSTATE_INT32_V(tsr_retry, SerialState, 4),

Subsection, only migrated if nonzero. thr/tsr can be in this subsection as well.

hw/char/serial.c const VMStateDescription vmstate_serial = { · line 634

Quoted 2 lines
hw/char/serial.c +2 -0
+        VMSTATE_STRUCT(recv_fifo, SerialState, 4, vmstate_fifo8, Fifo8),
+        VMSTATE_STRUCT(xmit_fifo, SerialState, 4, vmstate_fifo8, Fifo8),

Two subsections, only transmitted if nonempty.

hw/char/serial.c const VMStateDescription vmstate_serial = { · line 635

Quoted 1 line
hw/char/serial.c +1 -0
+        VMSTATE_TIMER_V(fifo_timeout_timer, SerialState, 4),

Subsection, only transmitted if pending.

hw/char/serial.c const VMStateDescription vmstate_serial = { · line 636

Quoted 1 line
hw/char/serial.c +1 -0
+        VMSTATE_INT32_V(timeout_ipending, SerialState, 4),

Subsection, transmitted only if nonzero.

hw/char/serial.c const VMStateDescription vmstate_serial = { · line 638

Quoted 2 lines
hw/char/serial.c +2 -0
+        VMSTATE_INT32_V(poll_msl, SerialState, 4),
+        VMSTATE_TIMER_V(modem_status_poll, SerialState, 4),

Both in a subsection, only migrated if poll_msl is not -1.

Paolo

Quoted 1 line
hw/char/serial.c +0 -0
         VMSTATE_END_OF_LIST()
#10 lore raw

Re: [Qemu-devel] [RFC PATCH v2 06/49] serial: fixing vmstate for save/restore

Quoted 23 lines

-----Original Message-----
From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini
Sent: Monday, July 28, 2014 1:59 PM
To: Pavel Dovgalyuk; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH v2 06/49] serial: fixing vmstate for save/restore

Il 17/07/2014 13:02, Pavel Dovgalyuk ha scritto:

Quoted 13 lines
hw/char/serial.c +3 -1
-    .version_id = 3,
+    .version_id = 4,
     .minimum_version_id = 2,
     .pre_save = serial_pre_save,
     .post_load = serial_post_load,
     .fields = (VMStateField[]) {
         VMSTATE_UINT16_V(divider, SerialState, 2),
         VMSTATE_UINT8(rbr, SerialState),
+        VMSTATE_UINT8_V(thr, SerialState, 4),
+        VMSTATE_UINT8_V(tsr, SerialState, 4),
         VMSTATE_UINT8(ier, SerialState),
         VMSTATE_UINT8(iir, SerialState),
         VMSTATE_UINT8(lcr, SerialState),
Quoted 8 lines

hw/char/serial.c const VMStateDescription vmstate_serial = { · line 630

Quoted 5 lines
hw/char/serial.c +1 -0
@@ -613,6 +627,15 @@ const VMStateDescription vmstate_serial = {
         VMSTATE_UINT8(msr, SerialState),
         VMSTATE_UINT8(scr, SerialState),
         VMSTATE_UINT8_V(fcr_vmstate, SerialState, 3),
+        VMSTATE_INT32_V(thr_ipending, SerialState, 4),

Subsection, only migrated if it doesn't match "(s->iir & UART_IIR_ID) == UART_IIR_THRI".

thr_ipending is set here:

    if (s->lsr & UART_LSR_THRE) {
        s->lsr |= UART_LSR_TEMT;
        s->thr_ipending = 1;
        serial_update_irq(s);
    }

serial_update_irq has several if-else branches. One of them sets tmp_iir = UART_IIR_THRI, as you said.
Couldn't be possible, that this branch will not be reached, because one of the previous ones will be executed?

Pavel Dovgalyuk

A message that is not in the archive (sent off-list or not yet received): [email protected]

#12 lore raw

Re: [Qemu-devel] [RFC PATCH v2 06/49] serial: fixing vmstate for save/restore

Il 30/07/2014 09:01, Pavel Dovgaluk ha scritto:

Quoted 33 lines
Quoted 23 lines

-----Original Message-----
From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini
Sent: Monday, July 28, 2014 1:59 PM
To: Pavel Dovgalyuk; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH v2 06/49] serial: fixing vmstate for save/restore

Il 17/07/2014 13:02, Pavel Dovgalyuk ha scritto:

Quoted 13 lines
hw/char/serial.c +3 -1
-    .version_id = 3,
+    .version_id = 4,
     .minimum_version_id = 2,
     .pre_save = serial_pre_save,
     .post_load = serial_post_load,
     .fields = (VMStateField[]) {
         VMSTATE_UINT16_V(divider, SerialState, 2),
         VMSTATE_UINT8(rbr, SerialState),
+        VMSTATE_UINT8_V(thr, SerialState, 4),
+        VMSTATE_UINT8_V(tsr, SerialState, 4),
         VMSTATE_UINT8(ier, SerialState),
         VMSTATE_UINT8(iir, SerialState),
         VMSTATE_UINT8(lcr, SerialState),
Quoted 8 lines

hw/char/serial.c const VMStateDescription vmstate_serial = { · line 630

Quoted 5 lines
hw/char/serial.c +1 -0
@@ -613,6 +627,15 @@ const VMStateDescription vmstate_serial = {
         VMSTATE_UINT8(msr, SerialState),
         VMSTATE_UINT8(scr, SerialState),
         VMSTATE_UINT8_V(fcr_vmstate, SerialState, 3),
+        VMSTATE_INT32_V(thr_ipending, SerialState, 4),

Subsection, only migrated if it doesn't match "(s->iir & UART_IIR_ID) == UART_IIR_THRI".

thr_ipending is set here:

Quoted 8 lines
    if (s->lsr & UART_LSR_THRE) {
        s->lsr |= UART_LSR_TEMT;
        s->thr_ipending = 1;
        serial_update_irq(s);
    }

serial_update_irq has several if-else branches. One of them sets tmp_iir = UART_IIR_THRI, as you said.
Couldn't be possible, that this branch will not be reached, because one of the previous ones will be executed?

Yes, what I wrote is equivalent to the following:

int serial_pre_load(void *opaque)

{
    SerialState *s = opaque;
    s->thr_ipending = -1;
    return 0;
}

int serial_post_load(void *opaque)

{
    SerialState *s = opaque;
    ...
    if (s->thr_ipending == -1) {
        s->thr_ipending = ((s->iir & UART_IIR_ID) == UART_IIR_THRI);
    }
    ...
}

bool thr_ipending_needed(void *opaque)

{
    SerialState *s = opaque;
    bool expected_value = ((s->iir & UART_IIR_ID) == UART_IIR_THRI);
    return s->thr_pending != expected_value;
}

The case you mention is exactly the one that will cause thr_ipending to be migrated.

Paolo

#13 lore raw

RFC PATCH v2 7/49 kvmapic: fixing loading vmstate

In reply to Pavel Dovgalyuk

vapic state should not be synchronized with APIC while loading, because APIC state could be not loaded yet at that moment. We just save vapic_paddr in APIC VMState instead of synchronization.

---
 hw/i386/kvmvapic.c    |   22 +++++++++++++++++++++-
 hw/intc/apic_common.c |    5 ++++-
 2 files changed, 25 insertions(+), 2 deletions(-)
hw/i386/kvmvapic.c +21 -1
index cb855c7..417ab6a 100644
@@ -351,6 +351,24 @@ static int get_kpcr_number(X86CPU *cpu)
     return kpcr.number;
 }
 
+static int vapic_enable_post_load(VAPICROMState *s, X86CPU *cpu)
+{
+    int cpu_number = get_kpcr_number(cpu);
+    hwaddr vapic_paddr;
+    static const uint8_t enabled = 1;
+
+    if (cpu_number < 0) {
+        return -1;
+    }
+    vapic_paddr = s->vapic_paddr +
+        (((hwaddr)cpu_number) << VAPIC_CPU_SHIFT);
+    cpu_physical_memory_rw(vapic_paddr + offsetof(VAPICState, enabled),
+                           (void *)&enabled, sizeof(enabled), 1);
+    s->state = VAPIC_ACTIVE;
+
+    return 0;
+}
+
 static int vapic_enable(VAPICROMState *s, X86CPU *cpu)
 {
     int cpu_number = get_kpcr_number(cpu);
@@ -731,7 +749,9 @@ static void do_vapic_enable(void *data)
     VAPICROMState *s = data;
     X86CPU *cpu = X86_CPU(first_cpu);
 
-    vapic_enable(s, cpu);
+    /* Do not synchronize with APIC, because it was not loaded yet.
+       Just call the enable function which does not have synchronization. */
+    vapic_enable_post_load(s, cpu);
 }
 
 static int vapic_post_load(void *opaque, int version_id)
hw/intc/apic_common.c +4 -1
index ce3d903..9d75ee0 100644
@@ -347,7 +347,7 @@ static int apic_dispatch_post_load(void *opaque, int version_id)
 
 static const VMStateDescription vmstate_apic_common = {
     .name = "apic",
-    .version_id = 3,
+    .version_id = 4,
     .minimum_version_id = 3,
     .minimum_version_id_old = 1,
     .load_state_old = apic_load_old,
@@ -374,6 +374,9 @@ static const VMStateDescription vmstate_apic_common = {
         VMSTATE_INT64(next_time, APICCommonState),
         VMSTATE_INT64(timer_expiry,
                       APICCommonState), /* open-coded timer state */
+        VMSTATE_INT32_V(sipi_vector, APICCommonState, 4),
+        VMSTATE_INT32_V(wait_for_sipi, APICCommonState, 4),
+        VMSTATE_UINT64_V(vapic_paddr, APICCommonState, 4),
         VMSTATE_END_OF_LIST()
     }
 };
#14 lore raw

Re: [Qemu-devel] [RFC PATCH v2 07/49] kvmapic: fixing loading vmstate

Il 17/07/2014 13:02, Pavel Dovgalyuk ha scritto:

Quoted 3 lines

vapic state should not be synchronized with APIC while loading, because APIC state could be not loaded yet at that moment. We just save vapic_paddr in APIC VMState instead of synchronization.

This comment is now obsolete:

include/hw/i386/apic_internal.h: hwaddr vapic_paddr; /* note: persistence via kvmvapic */

Quoted 60 lines
---
 hw/i386/kvmvapic.c    |   22 +++++++++++++++++++++-
 hw/intc/apic_common.c |    5 ++++-
 2 files changed, 25 insertions(+), 2 deletions(-)
hw/i386/kvmvapic.c +21 -1
index cb855c7..417ab6a 100644
@@ -351,6 +351,24 @@ static int get_kpcr_number(X86CPU *cpu)
     return kpcr.number;
 }
 
+static int vapic_enable_post_load(VAPICROMState *s, X86CPU *cpu)
+{
+    int cpu_number = get_kpcr_number(cpu);
+    hwaddr vapic_paddr;
+    static const uint8_t enabled = 1;
+
+    if (cpu_number < 0) {
+        return -1;
+    }
+    vapic_paddr = s->vapic_paddr +
+        (((hwaddr)cpu_number) << VAPIC_CPU_SHIFT);
+    cpu_physical_memory_rw(vapic_paddr + offsetof(VAPICState, enabled),
+                           (void *)&enabled, sizeof(enabled), 1);
+    s->state = VAPIC_ACTIVE;
+
+    return 0;
+}
+
 static int vapic_enable(VAPICROMState *s, X86CPU *cpu)
 {
     int cpu_number = get_kpcr_number(cpu);
@@ -731,7 +749,9 @@ static void do_vapic_enable(void *data)
     VAPICROMState *s = data;
     X86CPU *cpu = X86_CPU(first_cpu);
 
-    vapic_enable(s, cpu);
+    /* Do not synchronize with APIC, because it was not loaded yet.
+       Just call the enable function which does not have synchronization. */
+    vapic_enable_post_load(s, cpu);
 }
 
 static int vapic_post_load(void *opaque, int version_id)
hw/intc/apic_common.c +1 -1
index ce3d903..9d75ee0 100644
@@ -347,7 +347,7 @@ static int apic_dispatch_post_load(void *opaque, int version_id)
 
 static const VMStateDescription vmstate_apic_common = {
     .name = "apic",
-    .version_id = 3,
+    .version_id = 4,
     .minimum_version_id = 3,
     .minimum_version_id_old = 1,
     .load_state_old = apic_load_old,

hw/intc/apic_common.c static const VMStateDescription vmstate_apic_common = { · line 378

Quoted 5 lines
hw/intc/apic_common.c +2 -0
@@ -374,6 +374,9 @@ static const VMStateDescription vmstate_apic_common = {
         VMSTATE_INT64(next_time, APICCommonState),
         VMSTATE_INT64(timer_expiry,
                       APICCommonState), /* open-coded timer state */
+        VMSTATE_INT32_V(sipi_vector, APICCommonState, 4),
+        VMSTATE_INT32_V(wait_for_sipi, APICCommonState, 4),

This could be a subsection. sipi_vector is only used (needed) if wait_for_sipi != 0.

hw/intc/apic_common.c static const VMStateDescription vmstate_apic_common = { · line 379

Quoted 1 line
hw/intc/apic_common.c +1 -0
+        VMSTATE_UINT64_V(vapic_paddr, APICCommonState, 4),

Here you could also use a subsection, where the "needed" function returns false if vapic_paddr == 0.

Paolo

Quoted 6 lines
hw/intc/apic_common.c +0 -0
         VMSTATE_END_OF_LIST()
     }
 };
#15 lore raw

Re: [Qemu-devel] [RFC PATCH v2 07/49] kvmapic: fixing loading vmstate

Quoted 8 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini Il 17/07/2014 13:02, Pavel Dovgalyuk ha scritto:

Quoted 6 lines
hw/intc/apic_common.c +0 -0
index ce3d903..9d75ee0 100644
@@ -347,7 +347,7 @@ static int apic_dispatch_post_load(void *opaque, int version_id)
 
Quoted 15 lines

hw/intc/apic_common.c static const VMStateDescription vmstate_apic_common = { · line 378

Quoted 13 lines
hw/intc/apic_common.c +3 -1
 static const VMStateDescription vmstate_apic_common = {
     .name = "apic",
-    .version_id = 3,
+    .version_id = 4,
     .minimum_version_id = 3,
     .minimum_version_id_old = 1,
     .load_state_old = apic_load_old,
@@ -374,6 +374,9 @@ static const VMStateDescription vmstate_apic_common = {
         VMSTATE_INT64(next_time, APICCommonState),
         VMSTATE_INT64(timer_expiry,
                       APICCommonState), /* open-coded timer state */
+        VMSTATE_INT32_V(sipi_vector, APICCommonState, 4),
+        VMSTATE_INT32_V(wait_for_sipi, APICCommonState, 4),

This could be a subsection. sipi_vector is only used (needed) if wait_for_sipi != 0.

Right, sipi_vector is used when wait_for_sipi != 0. But we can set sipi_vector to non-zero,
save the snapshot, and then set wait_for_sipi. If that snapshot will be loaded, sipi_vector become incorrect. Isn't this scenario possible?

Pavel Dovgalyuk

#16 lore raw

Re: [Qemu-devel] [RFC PATCH v2 07/49] kvmapic: fixing loading vmstate

Il 29/07/2014 14:03, Pavel Dovgaluk ha scritto:

Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

This could be a subsection. sipi_vector is only used (needed) if wait_for_sipi != 0.

Right, sipi_vector is used when wait_for_sipi != 0. But we can set sipi_vector to non-zero,
save the snapshot, and then set wait_for_sipi. If that snapshot will be loaded, sipi_vector become incorrect. Isn't this scenario possible?

sipi_vector will not be used until CPU_INTERRUPT_SIPI is set, and then sipi_vector will have been overwritten with a new value. The architecture guarantees that.

Paolo

#17 lore raw

RFC PATCH v2 8/49 hpet: fixing saving and loading process

In reply to Pavel Dovgalyuk

VM clock does not run while saving, so there is no need for saving the ticks in HPET. Also added saving of hpet_offset field.

---
 hw/timer/hpet.c |   13 +------------
 1 files changed, 1 insertions(+), 12 deletions(-)
hw/timer/hpet.c +1 -12
index e160e8f..73401b9 100644
@@ -222,14 +222,6 @@ static void update_irq(struct HPETTimer *timer, int set)
     }
 }
 
-static void hpet_pre_save(void *opaque)
-{
-    HPETState *s = opaque;
-
-    /* save current counter value */
-    s->hpet_counter = hpet_get_ticks(s);
-}
-
 static int hpet_pre_load(void *opaque)
 {
     HPETState *s = opaque;
@@ -255,9 +247,6 @@ static int hpet_post_load(void *opaque, int version_id)
 {
     HPETState *s = opaque;
 
-    /* Recalculate the offset between the main counter and guest time */
-    s->hpet_offset = ticks_to_ns(s->hpet_counter) - qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
-
     /* Push number of timers into capability returned via HPET_ID */
     s->capability &= ~HPET_ID_NUM_TIM_MASK;
     s->capability |= (s->num_timers - 1) << HPET_ID_NUM_TIM_SHIFT;
@@ -308,13 +297,13 @@ static const VMStateDescription vmstate_hpet = {
     .name = "hpet",
     .version_id = 2,
     .minimum_version_id = 1,
-    .pre_save = hpet_pre_save,
     .pre_load = hpet_pre_load,
     .post_load = hpet_post_load,
     .fields = (VMStateField[]) {
         VMSTATE_UINT64(config, HPETState),
         VMSTATE_UINT64(isr, HPETState),
         VMSTATE_UINT64(hpet_counter, HPETState),
+        VMSTATE_UINT64(hpet_offset, HPETState),
         VMSTATE_UINT8_V(num_timers, HPETState, 2),
         VMSTATE_VALIDATE("num_timers in range", hpet_validate_num_timers),
         VMSTATE_STRUCT_VARRAY_UINT8(timer, HPETState, num_timers, 0,
#18 lore raw

Re: [Qemu-devel] [RFC PATCH v2 08/49] hpet: fixing saving and loading process

Il 17/07/2014 13:02, Pavel Dovgalyuk ha scritto:

Quoted 39 lines

VM clock does not run while saving, so there is no need for saving the ticks in HPET. Also added saving of hpet_offset field.

---
 hw/timer/hpet.c |   13 +------------
 1 files changed, 1 insertions(+), 12 deletions(-)
hw/timer/hpet.c +0 -11
index e160e8f..73401b9 100644
@@ -222,14 +222,6 @@ static void update_irq(struct HPETTimer *timer, int set)
     }
 }
 
-static void hpet_pre_save(void *opaque)
-{
-    HPETState *s = opaque;
-
-    /* save current counter value */
-    s->hpet_counter = hpet_get_ticks(s);
-}
-
 static int hpet_pre_load(void *opaque)
 {
     HPETState *s = opaque;
@@ -255,9 +247,6 @@ static int hpet_post_load(void *opaque, int version_id)
 {
     HPETState *s = opaque;
 
-    /* Recalculate the offset between the main counter and guest time */
-    s->hpet_offset = ticks_to_ns(s->hpet_counter) - qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
-
     /* Push number of timers into capability returned via HPET_ID */
     s->capability &= ~HPET_ID_NUM_TIM_MASK;
     s->capability |= (s->num_timers - 1) << HPET_ID_NUM_TIM_SHIFT;
@@ -308,13 +297,13 @@ static const VMStateDescription vmstate_hpet = {
     .name = "hpet",

hw/timer/hpet.c static const VMStateDescription vmstate_hpet = { · line 306

Quoted 10 lines
hw/timer/hpet.c +1 -1
     .version_id = 2,
     .minimum_version_id = 1,
-    .pre_save = hpet_pre_save,
     .pre_load = hpet_pre_load,
     .post_load = hpet_post_load,
     .fields = (VMStateField[]) {
         VMSTATE_UINT64(config, HPETState),
         VMSTATE_UINT64(isr, HPETState),
         VMSTATE_UINT64(hpet_counter, HPETState),
+        VMSTATE_UINT64(hpet_offset, HPETState),

This needs a version bump.

Paolo

Quoted 6 lines
hw/timer/hpet.c +0 -0
         VMSTATE_UINT8_V(num_timers, HPETState, 2),
         VMSTATE_VALIDATE("num_timers in range", hpet_validate_num_timers),
         VMSTATE_STRUCT_VARRAY_UINT8(timer, HPETState, num_timers, 0,
#19 lore raw

RFC PATCH v2 9/49 pckbd: adding new fields to vmstate

In reply to Pavel Dovgalyuk

This patch adds outport to VMState to allow correct saving and restoring the state of PC keyboard controller.

---
 hw/input/pckbd.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
hw/input/pckbd.c +2 -1
index ca1cffc..19f6658 100644
@@ -371,13 +371,14 @@ static void kbd_reset(void *opaque)
 
 static const VMStateDescription vmstate_kbd = {
     .name = "pckbd",
-    .version_id = 3,
+    .version_id = 4,
     .minimum_version_id = 3,
     .fields = (VMStateField[]) {
         VMSTATE_UINT8(write_cmd, KBDState),
         VMSTATE_UINT8(status, KBDState),
         VMSTATE_UINT8(mode, KBDState),
         VMSTATE_UINT8(pending, KBDState),
+        VMSTATE_UINT8_V(outport, KBDState, 4),
         VMSTATE_END_OF_LIST()
     }
 };
#20 lore raw

Re: [Qemu-devel] [RFC PATCH v2 09/49] pckbd: adding new fields to vmstate

Il 17/07/2014 13:02, Pavel Dovgalyuk ha scritto:

Quoted 21 lines

This patch adds outport to VMState to allow correct saving and restoring the state of PC keyboard controller.

---
 hw/input/pckbd.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
hw/input/pckbd.c +1 -1
index ca1cffc..19f6658 100644
@@ -371,13 +371,14 @@ static void kbd_reset(void *opaque)
 
 static const VMStateDescription vmstate_kbd = {
     .name = "pckbd",
-    .version_id = 3,
+    .version_id = 4,

hw/input/pckbd.c static void kbd_reset(void *opaque) · line 384

Quoted 10 lines
hw/input/pckbd.c +1 -0
     .minimum_version_id = 3,
     .fields = (VMStateField[]) {
         VMSTATE_UINT8(write_cmd, KBDState),
         VMSTATE_UINT8(status, KBDState),
         VMSTATE_UINT8(mode, KBDState),
         VMSTATE_UINT8(pending, KBDState),
+        VMSTATE_UINT8_V(outport, KBDState, 4),
         VMSTATE_END_OF_LIST()
     }
 };

Again it would be nice to use a subsection. You can use as the "default" value

    KBD_OUT_RESET | KBD_OUT_A20
     | (kbd->status & KBD_STAT_OBF ? KBD_OUT_OBF : 0)
     | (kbd->status & KBD_STAT_MOUSE_OBF ? KBD_OUT_MOUSE_OBF : 0)

If the value of outport matches this, you need not write it. It's not trivial, but you could do it like this:

- needed: return false if outport doesn't have the value above

- subsection post_load: set kbd->outport_present = 1

- device post_load: reconstruct outport if kbd->outport_present == 0

Paolo

#21 lore raw

RFC PATCH v2 10/49 rtl8139: adding new fields to vmstate

In reply to Pavel Dovgalyuk

This patch adds virtual clock-dependent timers to VMState to allow correct saving and restoring the state of RTL8139 network controller.

---
 hw/net/rtl8139.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)
hw/net/rtl8139.c +4 -1
index 90bc5ec..992caf0 100644
@@ -3289,7 +3289,7 @@ static void rtl8139_pre_save(void *opaque)
 
 static const VMStateDescription vmstate_rtl8139 = {
     .name = "rtl8139",
-    .version_id = 4,
+    .version_id = 5,
     .minimum_version_id = 3,
     .post_load = rtl8139_post_load,
     .pre_save  = rtl8139_pre_save,
@@ -3363,6 +3363,9 @@ static const VMStateDescription vmstate_rtl8139 = {
         VMSTATE_STRUCT(tally_counters, RTL8139State, 0,
                        vmstate_tally_counters, RTL8139TallyCounters),
 
+        VMSTATE_TIMER_V(timer, RTL8139State, 5),
+        VMSTATE_INT64_V(TimerExpire, RTL8139State, 5),
+
         VMSTATE_UINT32_V(cplus_enabled, RTL8139State, 4),
         VMSTATE_END_OF_LIST()
     },
#22 lore raw

Re: [Qemu-devel] [RFC PATCH v2 10/49] rtl8139: adding new fields to vmstate

Il 17/07/2014 13:02, Pavel Dovgalyuk ha scritto:

Quoted 22 lines

This patch adds virtual clock-dependent timers to VMState to allow correct saving and restoring the state of RTL8139 network controller.

---
 hw/net/rtl8139.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)
hw/net/rtl8139.c +1 -1
index 90bc5ec..992caf0 100644
@@ -3289,7 +3289,7 @@ static void rtl8139_pre_save(void *opaque)
 
 static const VMStateDescription vmstate_rtl8139 = {
     .name = "rtl8139",
-    .version_id = 4,
+    .version_id = 5,
     .minimum_version_id = 3,
     .post_load = rtl8139_post_load,
     .pre_save  = rtl8139_pre_save,

hw/net/rtl8139.c static const VMStateDescription vmstate_rtl8139 = { · line 3366

Quoted 4 lines
hw/net/rtl8139.c +1 -0
@@ -3363,6 +3363,9 @@ static const VMStateDescription vmstate_rtl8139 = {
         VMSTATE_STRUCT(tally_counters, RTL8139State, 0,
                        vmstate_tally_counters, RTL8139TallyCounters),
 
+        VMSTATE_TIMER_V(timer, RTL8139State, 5),

timer need not be migrated, because it is reinstated by rtl8139_post_load.

hw/net/rtl8139.c static const VMStateDescription vmstate_rtl8139 = { · line 3367

Quoted 1 line
hw/net/rtl8139.c +1 -0
+        VMSTATE_INT64_V(TimerExpire, RTL8139State, 5),

This can be in a subsection, migrated only if non-zero.

Paolo

Quoted 6 lines
hw/net/rtl8139.c +0 -0
         VMSTATE_UINT32_V(cplus_enabled, RTL8139State, 4),
         VMSTATE_END_OF_LIST()
     },
#23 lore raw

Re: [Qemu-devel] [RFC PATCH v2 10/49] rtl8139: adding new fields to vmstate

Quoted 27 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini Il 17/07/2014 13:02, Pavel Dovgalyuk ha scritto:

Quoted 25 lines

This patch adds virtual clock-dependent timers to VMState to allow correct saving and restoring the state of RTL8139 network controller.

---
 hw/net/rtl8139.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)
hw/net/rtl8139.c +1 -1
index 90bc5ec..992caf0 100644
@@ -3289,7 +3289,7 @@ static void rtl8139_pre_save(void *opaque)
 
 static const VMStateDescription vmstate_rtl8139 = {
     .name = "rtl8139",
-    .version_id = 4,
+    .version_id = 5,
     .minimum_version_id = 3,
     .post_load = rtl8139_post_load,
     .pre_save  = rtl8139_pre_save,
@@ -3363,6 +3363,9 @@ static const VMStateDescription vmstate_rtl8139 = {
         VMSTATE_STRUCT(tally_counters, RTL8139State, 0,
                        vmstate_tally_counters, RTL8139TallyCounters),
 
Quoted 4 lines

hw/net/rtl8139.c static const VMStateDescription vmstate_rtl8139 = { · line 3366

Quoted 1 line
hw/net/rtl8139.c +1 -0
+        VMSTATE_TIMER_V(timer, RTL8139State, 5),

timer need not be migrated, because it is reinstated by rtl8139_post_load.

That's true for normal execution.
In replay execution mode post_load can be called before cached virtual clock
values are loaded. This may cause invalid setting of the timer and raising an IRQ which didn't happen in record mode.
I will update this patch to fix post_load function and avoid this
non-deterministic behavior.

Pavel Dovgalyuk

A message that is not in the archive (sent off-list or not yet received): [email protected]

#25 lore raw

Re: [Qemu-devel] [RFC PATCH v2 10/49] rtl8139: adding new fields to vmstate

Il 28/07/2014 11:54, Pavel Dovgaluk ha scritto:

Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

+ VMSTATE_TIMER_V(timer, RTL8139State, 5),

timer need not be migrated, because it is reinstated by rtl8139_post_load.

That's true for normal execution.
In replay execution mode post_load can be called before cached virtual clock
values are loaded. This may cause invalid setting of the timer and raising an IRQ which didn't happen in record mode.
I will update this patch to fix post_load function and avoid this
non-deterministic behavior.

This is what worries me of this series. These invariants are not documented anywhere, and people will break them unless you add assertions that also hold in normal mode.

Paolo

#26 lore raw

Re: [Qemu-devel] [RFC PATCH v2 10/49] rtl8139: adding new fields to vmstate

Quoted 13 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini Il 28/07/2014 11:54, Pavel Dovgaluk ha scritto:

Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

+ VMSTATE_TIMER_V(timer, RTL8139State, 5),

timer need not be migrated, because it is reinstated by rtl8139_post_load.

That's true for normal execution.
In replay execution mode post_load can be called before cached virtual clock
values are loaded. This may cause invalid setting of the timer and raising an IRQ which didn't happen in record mode.
I will update this patch to fix post_load function and avoid this
non-deterministic behavior.

Quoted 3 lines

This is what worries me of this series. These invariants are not documented anywhere, and people will break them unless you add assertions that also hold in normal mode.

Assertions is a good idea, we added such warning message to qemu_get_timedate function to be sure, that it is used correctly with replay.

Another thing, that could help for making snapshots - find a way to load replay structures before all other ones. Are there any priorities in migration states list? Priorities could also solve some other issues, because sometimes post_load function of one device uses other devices' functions. And the second ones could be not loaded yet.

Pavel Dovgalyuk

#27 lore raw

Re: [Qemu-devel] [RFC PATCH v2 10/49] rtl8139: adding new fields to vmstate

Il 30/07/2014 10:24, Pavel Dovgaluk ha scritto:

Quoted 5 lines

Assertions is a good idea, we added such warning message to qemu_get_timedate function to be sure, that it is used correctly with replay.

Another thing, that could help for making snapshots - find a way to load replay structures before all other ones. Are there any priorities in migration states list?

No, it is just about vmstate registration order. In general bus parents come before bus children for obvious reasons.

Paolo

Quoted 2 lines

Priorities could also solve some other issues, because sometimes post_load function of one device uses other devices' functions. And the second ones could be not loaded yet.

#28 lore raw

RFC PATCH v2 11/49 piix: do not raise irq while loading vmstate

In reply to Pavel Dovgalyuk

This patch disables raising an irq while loading the state of PCI bridge.

---
 hw/pci-host/piix.c |   22 ++++++++++++++++++++--
 1 files changed, 20 insertions(+), 2 deletions(-)
hw/pci-host/piix.c +20 -2
index e0e0946..86d6d20 100644
@@ -409,7 +409,7 @@ static void piix3_set_irq_pic(PIIX3State *piix3, int pic_irq)
                      (pic_irq * PIIX_NUM_PIRQS))));
 }
 
-static void piix3_set_irq_level(PIIX3State *piix3, int pirq, int level)
+static void piix3_set_irq_level_internal(PIIX3State *piix3, int pirq, int level)
 {
     int pic_irq;
     uint64_t mask;
@@ -422,6 +422,18 @@ static void piix3_set_irq_level(PIIX3State *piix3, int pirq, int level)
     mask = 1ULL << ((pic_irq * PIIX_NUM_PIRQS) + pirq);
     piix3->pic_levels &= ~mask;
     piix3->pic_levels |= mask * !!level;
+}
+
+static void piix3_set_irq_level(PIIX3State *piix3, int pirq, int level)
+{
+    int pic_irq;
+
+    pic_irq = piix3->dev.config[PIIX_PIRQC + pirq];
+    if (pic_irq >= PIIX_NUM_PIC_IRQS) {
+        return;
+    }
+
+    piix3_set_irq_level_internal(piix3, pirq, level);
 
     piix3_set_irq_pic(piix3, pic_irq);
 }
@@ -527,7 +539,13 @@ static void piix3_reset(void *opaque)
 static int piix3_post_load(void *opaque, int version_id)
 {
     PIIX3State *piix3 = opaque;
-    piix3_update_irq_levels(piix3);
+    int pirq;
+
+    piix3->pic_levels = 0;
+    for (pirq = 0; pirq < PIIX_NUM_PIRQS; pirq++) {
+        piix3_set_irq_level_internal(piix3, pirq,
+                            pci_bus_get_irq_level(piix3->dev.bus, pirq));
+    }
     return 0;
 }
 
#29 lore raw

RFC PATCH v2 12/49 mc146818rtc: add missed field to vmstate

In reply to Pavel Dovgalyuk

This patch adds irq_reinject_on_ack_count field to VMState to allow correct saving/loading the state of MC146818 RTC.

---
 hw/timer/mc146818rtc.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
hw/timer/mc146818rtc.c +2 -1
index 9d817ca..c204abb 100644
@@ -735,7 +735,7 @@ static int rtc_post_load(void *opaque, int version_id)
 
 static const VMStateDescription vmstate_rtc = {
     .name = "mc146818rtc",
-    .version_id = 3,
+    .version_id = 4,
     .minimum_version_id = 1,
     .post_load = rtc_post_load,
     .fields = (VMStateField[]) {
@@ -752,6 +752,7 @@ static const VMStateDescription vmstate_rtc = {
         VMSTATE_INT64_V(offset, RTCState, 3),
         VMSTATE_TIMER_V(update_timer, RTCState, 3),
         VMSTATE_UINT64_V(next_alarm_time, RTCState, 3),
+        VMSTATE_UINT16_V(irq_reinject_on_ack_count, RTCState, 4),
         VMSTATE_END_OF_LIST()
     }
 };
#30 lore raw

Re: [Qemu-devel] [RFC PATCH v2 12/49] mc146818rtc: add missed field to vmstate

Il 17/07/2014 13:03, Pavel Dovgalyuk ha scritto:

Quoted 22 lines

This patch adds irq_reinject_on_ack_count field to VMState to allow correct saving/loading the state of MC146818 RTC.

---
 hw/timer/mc146818rtc.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
hw/timer/mc146818rtc.c +1 -1
index 9d817ca..c204abb 100644
@@ -735,7 +735,7 @@ static int rtc_post_load(void *opaque, int version_id)
 
 static const VMStateDescription vmstate_rtc = {
     .name = "mc146818rtc",
-    .version_id = 3,
+    .version_id = 4,
     .minimum_version_id = 1,
     .post_load = rtc_post_load,
     .fields = (VMStateField[]) {

hw/timer/mc146818rtc.c static const VMStateDescription vmstate_rtc = { · line 755

Quoted 4 lines
hw/timer/mc146818rtc.c +1 -0
@@ -752,6 +752,7 @@ static const VMStateDescription vmstate_rtc = {
         VMSTATE_INT64_V(offset, RTCState, 3),
         VMSTATE_TIMER_V(update_timer, RTCState, 3),
         VMSTATE_UINT64_V(next_alarm_time, RTCState, 3),
+        VMSTATE_UINT16_V(irq_reinject_on_ack_count, RTCState, 4),

Also can be a subsection, migrated only if nonzero.

Paolo

Quoted 6 lines
hw/timer/mc146818rtc.c +0 -0
         VMSTATE_END_OF_LIST()
     }
 };
#31 lore raw

RFC PATCH v2 13/49 pl031: add missed field to vmstate

In reply to Pavel Dovgalyuk

This patch adds timer which uses virtual clock to the VMState. Such timers are required for saving because virtual clock is the part of the virtual machine state.

---
 hw/timer/pl031.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
hw/timer/pl031.c +2 -1
index 34d9b44..f8e5abc 100644
@@ -230,12 +230,13 @@ static int pl031_post_load(void *opaque, int version_id)
 
 static const VMStateDescription vmstate_pl031 = {
     .name = "pl031",
-    .version_id = 1,
+    .version_id = 2,
     .minimum_version_id = 1,
     .pre_save = pl031_pre_save,
     .post_load = pl031_post_load,
     .fields = (VMStateField[]) {
         VMSTATE_UINT32(tick_offset_vmstate, PL031State),
+        VMSTATE_TIMER_V(timer, PL031State, 2),
         VMSTATE_UINT32(mr, PL031State),
         VMSTATE_UINT32(lr, PL031State),
         VMSTATE_UINT32(cr, PL031State),
#32 lore raw

RFC PATCH v2 14/49 ide pci: reset status field before loading the vmstate

In reply to Pavel Dovgalyuk

This patch resets status field before loading the state of IDE BMDMA device. Resetting status is needed for replay, because it does not reset whole virtual machine before loading state of the VM.

---
 hw/ide/pci.c |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)
hw/ide/pci.c +10 -0
index 6257a21..47ab24d 100644
@@ -382,6 +382,15 @@ static bool ide_bmdma_status_needed(void *opaque)
     return ((bm->status & abused_bits) != 0);
 }
 
+static int ide_bmdma_pre_load(void *opaque)
+{
+    BMDMAState *bm = opaque;
+    /* Reset is not performed in replay mode,
+       so reset status manually to allow ide_bmdma_post_load function initialize it. */
+    bm->status = 0;
+    return 0;
+}
+
 static void ide_bmdma_pre_save(void *opaque)
 {
     BMDMAState *bm = opaque;
@@ -434,6 +443,7 @@ static const VMStateDescription vmstate_bmdma = {
     .name = "ide bmdma",
     .version_id = 3,
     .minimum_version_id = 0,
+    .pre_load  = ide_bmdma_pre_load,
     .pre_save  = ide_bmdma_pre_save,
     .fields = (VMStateField[]) {
         VMSTATE_UINT8(cmd, BMDMAState),
#33 lore raw

RFC PATCH v2 15/49 softmmu: fixing usage of cpu_st/ld* from helpers

In reply to Pavel Dovgalyuk

MMU helper functions are called from generated code and other helper functions. In both cases they try to get function's return address for using it while restoring virtual CPU state.

When MMU helper is called from some other helper function (like helper_maskmov_xmm) through cpu_st* function, the return address will point to that helper. That is why CPU state cannot be restored in the case of MMU fault.

This patch introduces several inline helpers to load return address which points to the right place.

---
 include/exec/cpu_ldst_template.h |   28 ++++++++++++++++++++++++----
 include/exec/exec-all.h          |   27 +++++++++++++++++++++++++++
 softmmu_template.h               |   18 ++++++++++++++++++
 3 files changed, 69 insertions(+), 4 deletions(-)
include/exec/cpu_ldst_template.h +24 -4
index 006093a..10d9f8f 100644
@@ -61,6 +61,16 @@
 #define MMUSUFFIX _mmu
 #endif
 
+/* inline helper ld function */
+
+static inline DATA_TYPE
+glue(glue(helper_inline_ld, SUFFIX), MEMSUFFIX)(CPUArchState *env, target_ulong addr,
+                                         int mmu_idx)
+{
+    return glue(glue(helper_call_ld, SUFFIX), MMUSUFFIX)(env, addr, mmu_idx,
+                                                         GETRA());
+}
+
 /* generic load/store macros */
 
 static inline RES_TYPE
@@ -76,7 +86,7 @@ glue(glue(cpu_ld, USUFFIX), MEMSUFFIX)(CPUArchState *env, target_ulong ptr)
     mmu_idx = CPU_MMU_INDEX;
     if (unlikely(env->tlb_table[mmu_idx][page_index].ADDR_READ !=
                  (addr & (TARGET_PAGE_MASK | (DATA_SIZE - 1))))) {
-        res = glue(glue(helper_ld, SUFFIX), MMUSUFFIX)(env, addr, mmu_idx);
+        res = glue(glue(helper_inline_ld, SUFFIX), MEMSUFFIX)(env, addr, mmu_idx);
     } else {
         uintptr_t hostaddr = addr + env->tlb_table[mmu_idx][page_index].addend;
         res = glue(glue(ld, USUFFIX), _raw)(hostaddr);
@@ -97,8 +107,8 @@ glue(glue(cpu_lds, SUFFIX), MEMSUFFIX)(CPUArchState *env, target_ulong ptr)
     mmu_idx = CPU_MMU_INDEX;
     if (unlikely(env->tlb_table[mmu_idx][page_index].ADDR_READ !=
                  (addr & (TARGET_PAGE_MASK | (DATA_SIZE - 1))))) {
-        res = (DATA_STYPE)glue(glue(helper_ld, SUFFIX),
-                               MMUSUFFIX)(env, addr, mmu_idx);
+        res = (DATA_STYPE)glue(glue(helper_inline_ld, SUFFIX),
+                               MEMSUFFIX)(env, addr, mmu_idx);
     } else {
         uintptr_t hostaddr = addr + env->tlb_table[mmu_idx][page_index].addend;
         res = glue(glue(lds, SUFFIX), _raw)(hostaddr);
@@ -109,6 +119,16 @@ glue(glue(cpu_lds, SUFFIX), MEMSUFFIX)(CPUArchState *env, target_ulong ptr)
 
 #ifndef SOFTMMU_CODE_ACCESS
 
+/* inline helper st function */
+
+static inline void
+glue(glue(helper_inline_st, SUFFIX), MEMSUFFIX)(CPUArchState *env, target_ulong addr,
+                                         DATA_TYPE val, int mmu_idx)
+{
+    glue(glue(helper_call_st, SUFFIX), MMUSUFFIX)(env, addr, val, mmu_idx,
+                                                  GETRA());
+}
+
 /* generic store macro */
 
 static inline void
@@ -124,7 +144,7 @@ glue(glue(cpu_st, SUFFIX), MEMSUFFIX)(CPUArchState *env, target_ulong ptr,
     mmu_idx = CPU_MMU_INDEX;
     if (unlikely(env->tlb_table[mmu_idx][page_index].addr_write !=
                  (addr & (TARGET_PAGE_MASK | (DATA_SIZE - 1))))) {
-        glue(glue(helper_st, SUFFIX), MMUSUFFIX)(env, addr, v, mmu_idx);
+        glue(glue(helper_inline_st, SUFFIX), MEMSUFFIX)(env, addr, v, mmu_idx);
     } else {
         uintptr_t hostaddr = addr + env->tlb_table[mmu_idx][page_index].addend;
         glue(glue(st, SUFFIX), _raw)(hostaddr, v);
include/exec/exec-all.h +27 -0
index 5e5d86e..528928f 100644
@@ -344,6 +344,33 @@ bool io_mem_write(struct MemoryRegion *mr, hwaddr addr,
 void tlb_fill(CPUState *cpu, target_ulong addr, int is_write, int mmu_idx,
               uintptr_t retaddr);
 
+uint8_t helper_call_ldb_cmmu(CPUArchState *env, target_ulong addr,
+                             int mmu_idx, uintptr_t retaddr);
+uint16_t helper_call_ldw_cmmu(CPUArchState *env, target_ulong addr,
+                              int mmu_idx, uintptr_t retaddr);
+uint32_t helper_call_ldl_cmmu(CPUArchState *env, target_ulong addr,
+                              int mmu_idx, uintptr_t retaddr);
+uint64_t helper_call_ldq_cmmu(CPUArchState *env, target_ulong addr,
+                              int mmu_idx, uintptr_t retaddr);
+
+uint8_t helper_call_ldb_mmu(CPUArchState *env, target_ulong addr,
+                            int mmu_idx, uintptr_t retaddr);
+uint16_t helper_call_ldw_mmu(CPUArchState *env, target_ulong addr,
+                             int mmu_idx, uintptr_t retaddr);
+uint32_t helper_call_ldl_mmu(CPUArchState *env, target_ulong addr,
+                             int mmu_idx, uintptr_t retaddr);
+uint64_t helper_call_ldq_mmu(CPUArchState *env, target_ulong addr,
+                             int mmu_idx, uintptr_t retaddr);
+
+void helper_call_stb_mmu(CPUArchState *env, target_ulong addr,
+                         uint8_t val, int mmu_idx, uintptr_t retaddr);
+void helper_call_stw_mmu(CPUArchState *env, target_ulong addr,
+                         uint16_t val, int mmu_idx, uintptr_t retaddr);
+void helper_call_stl_mmu(CPUArchState *env, target_ulong addr,
+                         uint32_t val, int mmu_idx, uintptr_t retaddr);
+void helper_call_stq_mmu(CPUArchState *env, target_ulong addr,
+                         uint64_t val, int mmu_idx, uintptr_t retaddr);
+
 #endif
 
 #if defined(CONFIG_USER_ONLY)
softmmu_template.h +18 -0
index 5a07f99..1053cf3 100644
@@ -311,6 +311,15 @@ glue(glue(helper_ld, SUFFIX), MMUSUFFIX)(CPUArchState *env, target_ulong addr,
     return helper_te_ld_name (env, addr, mmu_idx, GETRA());
 }
 
+DATA_TYPE
+glue(glue(helper_call_ld, SUFFIX), MMUSUFFIX)(CPUArchState *env,
+                                              target_ulong addr,
+                                              int mmu_idx,
+                                              uintptr_t retaddr)
+{
+    return helper_te_ld_name(env, addr, mmu_idx, retaddr);
+}
+
 #ifndef SOFTMMU_CODE_ACCESS
 
 /* Provide signed versions of the load routines as well.  We can of course
@@ -505,6 +514,15 @@ glue(glue(helper_st, SUFFIX), MMUSUFFIX)(CPUArchState *env, target_ulong addr,
     helper_te_st_name(env, addr, val, mmu_idx, GETRA());
 }
 
+void
+glue(glue(helper_call_st, SUFFIX), MMUSUFFIX)(CPUArchState *env,
+                                              target_ulong addr,
+                                              DATA_TYPE val, int mmu_idx,
+                                              uintptr_t retaddr)
+{
+    helper_te_st_name(env, addr, val, mmu_idx, retaddr);
+}
+
 #endif /* !defined(SOFTMMU_CODE_ACCESS) */
 
 #undef READ_ACCESS_TYPE
#34 lore raw

RFC PATCH v2 16/49 target: save cpu state fields

In reply to Pavel Dovgalyuk

This patch adds interrupt fields to VMState for correct saving the CPU state.

---
 target-arm/machine.c  |    5 ++++-
 target-i386/machine.c |    5 ++++-
 2 files changed, 8 insertions(+), 2 deletions(-)
target-arm/machine.c +4 -1
index 3bcc7cc..29bfc10 100644
@@ -218,7 +218,7 @@ static int cpu_post_load(void *opaque, int version_id)
 
 const VMStateDescription vmstate_arm_cpu = {
     .name = "cpu",
-    .version_id = 20,
+    .version_id = 21,
     .minimum_version_id = 20,
     .pre_save = cpu_pre_save,
     .post_load = cpu_post_load,
@@ -259,6 +259,9 @@ const VMStateDescription vmstate_arm_cpu = {
         VMSTATE_UINT64(env.exception.vaddress, ARMCPU),
         VMSTATE_TIMER(gt_timer[GTIMER_PHYS], ARMCPU),
         VMSTATE_TIMER(gt_timer[GTIMER_VIRT], ARMCPU),
+        /* Fields required by replay */
+        VMSTATE_UINT32_V(parent_obj.interrupt_request, ARMCPU, 21),
+        VMSTATE_INT32_V(parent_obj.exception_index, ARMCPU, 21),
         VMSTATE_END_OF_LIST()
     },
     .subsections = (VMStateSubsection[]) {
target-i386/machine.c +4 -1
index 16d2f6a..9dfac33 100644
@@ -605,7 +605,7 @@ static const VMStateDescription vmstate_msr_hyperv_time = {
 
 VMStateDescription vmstate_x86_cpu = {
     .name = "cpu",
-    .version_id = 12,
+    .version_id = 13,
     .minimum_version_id = 3,
     .pre_save = cpu_pre_save,
     .post_load = cpu_post_load,
@@ -702,6 +702,9 @@ VMStateDescription vmstate_x86_cpu = {
         VMSTATE_UINT64_V(env.xcr0, X86CPU, 12),
         VMSTATE_UINT64_V(env.xstate_bv, X86CPU, 12),
         VMSTATE_YMMH_REGS_VARS(env.ymmh_regs, X86CPU, CPU_NB_REGS, 12),
+        /* Fields required by replay */
+        VMSTATE_UINT32_V(parent_obj.interrupt_request, X86CPU, 13),
+        VMSTATE_INT32_V(parent_obj.exception_index, X86CPU, 13),
         VMSTATE_END_OF_LIST()
         /* The above list is not sorted /wrt version numbers, watch out! */
     },
#35 lore raw

Re: [Qemu-devel] [RFC PATCH v2 16/49] target: save cpu state fields

Am 17.07.2014 13:03, schrieb Pavel Dovgalyuk:

Quoted 45 lines

This patch adds interrupt fields to VMState for correct saving the CPU state.

---
 target-arm/machine.c  |    5 ++++-
 target-i386/machine.c |    5 ++++-
 2 files changed, 8 insertions(+), 2 deletions(-)
target-arm/machine.c +4 -1
index 3bcc7cc..29bfc10 100644
@@ -218,7 +218,7 @@ static int cpu_post_load(void *opaque, int version_id)
 
 const VMStateDescription vmstate_arm_cpu = {
     .name = "cpu",
-    .version_id = 20,
+    .version_id = 21,
     .minimum_version_id = 20,
     .pre_save = cpu_pre_save,
     .post_load = cpu_post_load,
@@ -259,6 +259,9 @@ const VMStateDescription vmstate_arm_cpu = {
         VMSTATE_UINT64(env.exception.vaddress, ARMCPU),
         VMSTATE_TIMER(gt_timer[GTIMER_PHYS], ARMCPU),
         VMSTATE_TIMER(gt_timer[GTIMER_VIRT], ARMCPU),
+        /* Fields required by replay */
+        VMSTATE_UINT32_V(parent_obj.interrupt_request, ARMCPU, 21),
+        VMSTATE_INT32_V(parent_obj.exception_index, ARMCPU, 21),
         VMSTATE_END_OF_LIST()
     },
     .subsections = (VMStateSubsection[]) {
target-i386/machine.c +1 -1
index 16d2f6a..9dfac33 100644
@@ -605,7 +605,7 @@ static const VMStateDescription vmstate_msr_hyperv_time = {
 
 VMStateDescription vmstate_x86_cpu = {
     .name = "cpu",
-    .version_id = 12,
+    .version_id = 13,
     .minimum_version_id = 3,
     .pre_save = cpu_pre_save,
     .post_load = cpu_post_load,

target-i386/machine.c VMStateDescription vmstate_x86_cpu = { · line 710

Quoted 9 lines
target-i386/machine.c +3 -0
@@ -702,6 +702,9 @@ VMStateDescription vmstate_x86_cpu = {
         VMSTATE_UINT64_V(env.xcr0, X86CPU, 12),
         VMSTATE_UINT64_V(env.xstate_bv, X86CPU, 12),
         VMSTATE_YMMH_REGS_VARS(env.ymmh_regs, X86CPU, CPU_NB_REGS, 12),
+        /* Fields required by replay */
+        VMSTATE_UINT32_V(parent_obj.interrupt_request, X86CPU, 13),
+        VMSTATE_INT32_V(parent_obj.exception_index, X86CPU, 13),
         VMSTATE_END_OF_LIST()
         /* The above list is not sorted /wrt version numbers, watch out! */
     },

Instead of messing with parent_obj here and above, can you please add common CPUState stuff to vmstate_cpu_common in exec.c?

Regards,
Andreas

Signature · 2 lines
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg
#36 lore raw

RFC PATCH v2 17/49 target-i386: update fp status fix

In reply to Pavel Dovgalyuk

This patch adds calls to update_fp_status() function from several places where FPU state is changed.

---
 target-i386/cpu.c        |    1 +
 target-i386/cpu.h        |    1 +
 target-i386/fpu_helper.c |    5 ++++-
 target-i386/machine.c    |    2 +-
 4 files changed, 7 insertions(+), 2 deletions(-)
target-i386/cpu.c +1 -0
index 45c662d..27269ad 100644
@@ -2573,6 +2573,7 @@ static void x86_cpu_reset(CPUState *s)
         env->fptags[i] = 1;
     }
     env->fpuc = 0x37f;
+    update_fp_status(env);
 
     env->mxcsr = 0x1f80;
     env->xstate_bv = XSTATE_FP | XSTATE_SSE;
target-i386/cpu.h +1 -0
index e634d83..42bda46 100644
@@ -1249,6 +1249,7 @@ void QEMU_NORETURN raise_interrupt(CPUX86State *nenv, int intno, int is_int,
 /* cc_helper.c */
 extern const uint8_t parity_table[256];
 uint32_t cpu_cc_compute_all(CPUX86State *env1, int op);
+void update_fp_status(CPUX86State *env);
 
 static inline uint32_t cpu_compute_eflags(CPUX86State *env)
 {
target-i386/fpu_helper.c +4 -1
index 1b2900d..a8ffba9 100644
@@ -537,7 +537,7 @@ uint32_t helper_fnstcw(CPUX86State *env)
     return env->fpuc;
 }
 
-static void update_fp_status(CPUX86State *env)
+void update_fp_status(CPUX86State *env)
 {
     int rnd_type;
 
@@ -1006,6 +1006,7 @@ void helper_fstenv(CPUX86State *env, target_ulong ptr, int data32)
         cpu_stw_data(env, ptr + 10, 0);
         cpu_stw_data(env, ptr + 12, 0);
     }
+    update_fp_status(env);
 }
 
 void helper_fldenv(CPUX86State *env, target_ulong ptr, int data32)
@@ -1055,6 +1056,7 @@ void helper_fsave(CPUX86State *env, target_ulong ptr, int data32)
     env->fptags[5] = 1;
     env->fptags[6] = 1;
     env->fptags[7] = 1;
+    update_fp_status(env);
 }
 
 void helper_frstor(CPUX86State *env, target_ulong ptr, int data32)
@@ -1158,6 +1160,7 @@ void helper_fxrstor(CPUX86State *env, target_ulong ptr, int data64)
     }
 
     env->fpuc = cpu_lduw_data(env, ptr);
+    update_fp_status(env);
     fpus = cpu_lduw_data(env, ptr + 2);
     fptag = cpu_lduw_data(env, ptr + 4);
     env->fpstt = (fpus >> 11) & 7;
target-i386/machine.c +1 -1
index 9dfac33..b8f3467 100644
@@ -315,13 +315,13 @@ static int cpu_post_load(void *opaque, int version_id)
     env->hflags &= ~HF_CPL_MASK;
     env->hflags |= (env->segs[R_SS].flags >> DESC_DPL_SHIFT) & HF_CPL_MASK;
 
-    /* XXX: restore FPU round state */
     env->fpstt = (env->fpus_vmstate >> 11) & 7;
     env->fpus = env->fpus_vmstate & ~0x3800;
     env->fptag_vmstate ^= 0xff;
     for(i = 0; i < 8; i++) {
         env->fptags[i] = (env->fptag_vmstate >> i) & 1;
     }
+    update_fp_status(env);
 
     cpu_breakpoint_remove_all(cs, BP_CPU);
     cpu_watchpoint_remove_all(cs, BP_CPU);
#37 lore raw

RFC PATCH v2 18/49 migration: add vmstate for int8 and char arrays

In reply to Pavel Dovgalyuk

This patch adds macros for vmstate int8 and char arrays.

---
 include/migration/vmstate.h |   13 +++++++++++++
 vmstate.c                   |    6 ++++++
 2 files changed, 19 insertions(+), 0 deletions(-)
include/migration/vmstate.h +13 -0
index 9a001bd..2f7207b 100644
@@ -143,6 +143,7 @@ extern const VMStateDescription vmstate_dummy;
 #endif
 
 extern const VMStateInfo vmstate_info_bool;
+extern const VMStateInfo vmstate_info_char;
 
 extern const VMStateInfo vmstate_info_int8;
 extern const VMStateInfo vmstate_info_int16;
@@ -694,6 +695,18 @@ extern const VMStateInfo vmstate_info_bitmap;
 #define VMSTATE_UINT64_ARRAY(_f, _s, _n)                              \
     VMSTATE_UINT64_ARRAY_V(_f, _s, _n, 0)
 
+#define VMSTATE_INT8_ARRAY_V(_f, _s, _n, _v)                          \
+    VMSTATE_ARRAY(_f, _s, _n, _v, vmstate_info_int8, int8_t)
+
+#define VMSTATE_INT8_ARRAY(_f, _s, _n)                                \
+    VMSTATE_INT8_ARRAY_V(_f, _s, _n, 0)
+
+#define VMSTATE_CHAR_ARRAY_V(_f, _s, _n, _v)                          \
+    VMSTATE_ARRAY(_f, _s, _n, _v, vmstate_info_char, char)
+
+#define VMSTATE_CHAR_ARRAY(_f, _s, _n)                                \
+    VMSTATE_CHAR_ARRAY_V(_f, _s, _n, 0)
+
 #define VMSTATE_INT16_ARRAY_V(_f, _s, _n, _v)                         \
     VMSTATE_ARRAY(_f, _s, _n, _v, vmstate_info_int16, int16_t)
 
vmstate.c +6 -0
index ef2f87b..6a7f7c2 100644
@@ -286,6 +286,12 @@ const VMStateInfo vmstate_info_int8 = {
     .put  = put_int8,
 };
 
+const VMStateInfo vmstate_info_char = {
+    .name = "char",
+    .get  = get_int8,
+    .put  = put_int8,
+};
+
 /* 16 bit int */
 
 static int get_int16(QEMUFile *f, void *pv, size_t size)
#38 lore raw

RFC PATCH v2 19/49 replay: global variables and function stubs

In reply to Pavel Dovgalyuk

This patch adds global variables, defines, functions declarations, and function stubs for deterministic VM replay used by external modules.

---
 Makefile.target      |    1 +
 replay/Makefile.objs |    1 +
 replay/replay.c      |   19 +++++++++++++++++++
 replay/replay.h      |   32 ++++++++++++++++++++++++++++++++
 stubs/Makefile.objs  |    1 +
 stubs/replay.c       |    9 +++++++++
 6 files changed, 63 insertions(+), 0 deletions(-)
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c
Makefile.target +1 -0
index 137d0b0..9830313 100644
@@ -76,6 +76,7 @@ all: $(PROGS) stap
 #########################################################
 # cpu emulator library
 obj-y = exec.o translate-all.o cpu-exec.o
+obj-y += replay/
 obj-y += tcg/tcg.o tcg/optimize.o
 obj-$(CONFIG_TCG_INTERPRETER) += tci.o
 obj-$(CONFIG_TCG_INTERPRETER) += disas/tci.o
replay/Makefile.objs new file +1 -0
new file mode 100755
index 0000000..7ea860f
@@ -0,0 +1 @@
+obj-y += replay.o
replay/replay.c new file +19 -0
new file mode 100755
index 0000000..5a69465
@@ -0,0 +1,19 @@
+/*
+ * replay.c
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming 
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "replay.h"
+
+int replay_mode = REPLAY_NONE;
+/*! Stores current submode for PLAY mode */
+int play_submode = REPLAY_PLAY_UNKNOWN;
+
+/* Suffix for the disk images filenames */
+char *replay_image_suffix;
replay/replay.h new file +32 -0
new file mode 100755
index 0000000..a883aa4
@@ -0,0 +1,32 @@
+#ifndef REPLAY_H
+#define REPLAY_H
+
+/*
+ * replay.h
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming 
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+/* replay modes */
+#define REPLAY_NONE 0
+#define REPLAY_SAVE 1
+#define REPLAY_PLAY 2
+/* replay submodes */
+#define REPLAY_PLAY_UNKNOWN 0
+/* Normal log replaying */
+#define REPLAY_PLAY_NORMAL  1
+
+extern int replay_mode;
+extern char *replay_image_suffix;
+/*! Shift value for icount based on replay or zero, if it is disabled. */
+extern int replay_icount;
+
+/*! Returns replay play submode */
+int replay_get_play_submode(void);
+
+#endif
stubs/Makefile.objs +1 -0
index 528e161..b90c60a 100644
@@ -26,6 +26,7 @@ stub-obj-y += notify-event.o
 stub-obj-y += pci-drive-hot-add.o
 stub-obj-$(CONFIG_SPICE) += qemu-chr-open-spice.o
 stub-obj-y += qtest.o
+stub-obj-y += replay.o
 stub-obj-y += reset.o
 stub-obj-y += runstate-check.o
 stub-obj-y += set-fd-handler.o
stubs/replay.c new file +9 -0
new file mode 100755
index 0000000..1e99ca4
@@ -0,0 +1,9 @@
+#include "replay/replay.h"
+
+int replay_mode;
+int replay_icount;
+
+int replay_get_play_submode(void)
+{
+    return 0;
+}
#39 lore raw

RFC PATCH v2 20/49 block: add suffix parameter to bdrv_open functions

In reply to Pavel Dovgalyuk

This patch adds parameter with suffix for overlay filenames. This parameter will be used by replay to automatically create overlay files based on drive images supplied to VM.

---
 block.c               |   44 ++++++++++++++++++++++++++++++--------------
 block/blkdebug.c      |    2 +-
 block/blkverify.c     |    4 ++--
 block/cow.c           |    2 +-
 block/qcow.c          |    2 +-
 block/qcow2.c         |    6 +++---
 block/qed.c           |    2 +-
 block/sheepdog.c      |    4 ++--
 block/vmdk.c          |    8 ++++----
 block/vvfat.c         |    2 +-
 blockdev.c            |   11 ++++++-----
 include/block/block.h |    6 +++---
 qemu-img.c            |    6 +++---
 qemu-io.c             |    4 ++--
 qemu-nbd.c            |    2 +-
 15 files changed, 61 insertions(+), 44 deletions(-)
block.c +30 -14
index 8800a6b..3ff1052 100644
@@ -35,6 +35,7 @@
 #include "qmp-commands.h"
 #include "qemu/timer.h"
 #include "qapi-event.h"
+#include "replay/replay.h"
 
 #ifdef CONFIG_BSD
 #include <sys/types.h>
@@ -1209,7 +1210,7 @@ int bdrv_open_backing_file(BlockDriverState *bs, QDict *options, Error **errp)
     assert(bs->backing_hd == NULL);
     ret = bdrv_open(&backing_hd,
                     *backing_filename ? backing_filename : NULL, NULL, options,
-                    bdrv_backing_flags(bs->open_flags), back_drv, &local_err);
+                    bdrv_backing_flags(bs->open_flags), back_drv, &local_err, NULL);
     if (ret < 0) {
         bdrv_unref(backing_hd);
         backing_hd = NULL;
@@ -1244,7 +1245,7 @@ free_exit:
  */
 int bdrv_open_image(BlockDriverState **pbs, const char *filename,
                     QDict *options, const char *bdref_key, int flags,
-                    bool allow_none, Error **errp)
+                    bool allow_none, Error **errp, const char *fname_suffix)
 {
     QDict *image_options;
     int ret;
@@ -1271,14 +1272,14 @@ int bdrv_open_image(BlockDriverState **pbs, const char *filename,
         goto done;
     }
 
-    ret = bdrv_open(pbs, filename, reference, image_options, flags, NULL, errp);
+    ret = bdrv_open(pbs, filename, reference, image_options, flags, NULL, errp, NULL);
 
 done:
     qdict_del(options, bdref_key);
     return ret;
 }
 
-int bdrv_append_temp_snapshot(BlockDriverState *bs, int flags, Error **errp)
+int bdrv_append_temp_snapshot(BlockDriverState *bs, int flags, Error **errp, const char *fname_suffix)
 {
     /* TODO: extra byte is a hack to ensure MAX_PATH space on Windows. */
     char *tmp_filename = g_malloc0(PATH_MAX + 1);
@@ -1303,17 +1304,29 @@ int bdrv_append_temp_snapshot(BlockDriverState *bs, int flags, Error **errp)
     total_size &= BDRV_SECTOR_MASK;
 
     /* Create the temporary image */
-    ret = get_tmp_filename(tmp_filename, PATH_MAX + 1);
-    if (ret < 0) {
-        error_setg_errno(errp, -ret, "Could not get temporary filename");
-        goto out;
+    /* We will not delete created file in record/replay mode */
+    if (replay_mode != REPLAY_NONE && fname_suffix) {
+        ret = snprintf(tmp_filename, PATH_MAX + 1, "%s.%s", bs->filename, fname_suffix);
+        if (ret < 0) {
+            goto out;
+        }
+    } else {
+        ret = get_tmp_filename(tmp_filename, PATH_MAX + 1);
+        if (ret < 0) {
+            error_setg_errno(errp, -ret, "Could not get temporary filename");
+            goto out;
+        }
     }
 
     bdrv_qcow2 = bdrv_find_format("qcow2");
     opts = qemu_opts_create(bdrv_qcow2->create_opts, NULL, 0,
                             &error_abort);
     qemu_opt_set_number(opts, BLOCK_OPT_SIZE, total_size);
-    ret = bdrv_create(bdrv_qcow2, tmp_filename, opts, &local_err);
+    if (replay_mode == REPLAY_PLAY) {
+        ret = 0;
+    } else {
+        ret = bdrv_create(bdrv_qcow2, tmp_filename, opts, &local_err);
+    }
     qemu_opts_del(opts);
     if (ret < 0) {
         error_setg_errno(errp, -ret, "Could not create temporary overlay "
@@ -1333,7 +1346,7 @@ int bdrv_append_temp_snapshot(BlockDriverState *bs, int flags, Error **errp)
     bs_snapshot = bdrv_new("", &error_abort);
 
     ret = bdrv_open(&bs_snapshot, NULL, NULL, snapshot_options,
-                    flags, bdrv_qcow2, &local_err);
+                    flags, bdrv_qcow2, &local_err, NULL);
     if (ret < 0) {
         error_propagate(errp, local_err);
         goto out;
@@ -1363,7 +1376,7 @@ out:
  */
 int bdrv_open(BlockDriverState **pbs, const char *filename,
               const char *reference, QDict *options, int flags,
-              BlockDriver *drv, Error **errp)
+              BlockDriver *drv, Error **errp, const char *fname_suffix)
 {
     int ret;
     BlockDriverState *file = NULL, *bs;
@@ -1444,13 +1457,16 @@ int bdrv_open(BlockDriverState **pbs, const char *filename,
         }
         if (flags & BDRV_O_SNAPSHOT) {
             snapshot_flags = bdrv_temp_snapshot_flags(flags);
+            if (replay_mode != REPLAY_NONE) {
+                snapshot_flags &= ~BDRV_O_TEMPORARY;
+            }
             flags = bdrv_backing_flags(flags);
         }
 
         assert(file == NULL);
         ret = bdrv_open_image(&file, filename, options, "file",
                               bdrv_inherited_flags(flags),
-                              true, &local_err);
+                              true, &local_err, NULL);
         if (ret < 0) {
             goto fail;
         }
@@ -1493,7 +1509,7 @@ int bdrv_open(BlockDriverState **pbs, const char *filename,
     /* For snapshot=on, create a temporary qcow2 overlay. bs points to the
      * temporary snapshot afterwards. */
     if (snapshot_flags) {
-        ret = bdrv_append_temp_snapshot(bs, snapshot_flags, &local_err);
+        ret = bdrv_append_temp_snapshot(bs, snapshot_flags, &local_err, fname_suffix);
         if (local_err) {
             goto close_and_fail;
         }
@@ -5573,7 +5589,7 @@ void bdrv_img_create(const char *filename, const char *fmt,
 
             bs = NULL;
             ret = bdrv_open(&bs, backing_file, NULL, NULL, back_flags,
-                            backing_drv, &local_err);
+                            backing_drv, &local_err, NULL);
             if (ret < 0) {
                 error_setg_errno(errp, -ret, "Could not open '%s': %s",
                                  backing_file,
block/blkdebug.c +1 -1
index f51407d..151538a 100644
@@ -412,7 +412,7 @@ static int blkdebug_open(BlockDriverState *bs, QDict *options, int flags,
     /* Open the backing file */
     assert(bs->file == NULL);
     ret = bdrv_open_image(&bs->file, qemu_opt_get(opts, "x-image"), options, "image",
-                          flags | BDRV_O_PROTOCOL, false, &local_err);
+                          flags | BDRV_O_PROTOCOL, false, &local_err, NULL);
     if (ret < 0) {
         error_propagate(errp, local_err);
         goto out;
block/blkverify.c +2 -2
index 621b785..94d7db7 100644
@@ -138,7 +138,7 @@ static int blkverify_open(BlockDriverState *bs, QDict *options, int flags,
     /* Open the raw file */
     assert(bs->file == NULL);
     ret = bdrv_open_image(&bs->file, qemu_opt_get(opts, "x-raw"), options,
-                          "raw", flags | BDRV_O_PROTOCOL, false, &local_err);
+                          "raw", flags | BDRV_O_PROTOCOL, false, &local_err, NULL);
     if (ret < 0) {
         error_propagate(errp, local_err);
         goto fail;
@@ -147,7 +147,7 @@ static int blkverify_open(BlockDriverState *bs, QDict *options, int flags,
     /* Open the test file */
     assert(s->test_file == NULL);
     ret = bdrv_open_image(&s->test_file, qemu_opt_get(opts, "x-image"), options,
-                          "test", flags, false, &local_err);
+                          "test", flags, false, &local_err, NULL);
     if (ret < 0) {
         error_propagate(errp, local_err);
         s->test_file = NULL;
block/cow.c +1 -1
index 6ee4833..d99c4d7 100644
@@ -345,7 +345,7 @@ static int cow_create(const char *filename, QemuOpts *opts, Error **errp)
     }
 
     ret = bdrv_open(&cow_bs, filename, NULL, NULL,
-                    BDRV_O_RDWR | BDRV_O_PROTOCOL, NULL, &local_err);
+                    BDRV_O_RDWR | BDRV_O_PROTOCOL, NULL, &local_err, NULL);
     if (ret < 0) {
         error_propagate(errp, local_err);
         goto exit;
block/qcow.c +1 -1
index a874056..6e21cea 100644
@@ -720,7 +720,7 @@ static int qcow_create(const char *filename, QemuOpts *opts, Error **errp)
 
     qcow_bs = NULL;
     ret = bdrv_open(&qcow_bs, filename, NULL, NULL,
-                    BDRV_O_RDWR | BDRV_O_PROTOCOL, NULL, &local_err);
+                    BDRV_O_RDWR | BDRV_O_PROTOCOL, NULL, &local_err, NULL);
     if (ret < 0) {
         error_propagate(errp, local_err);
         goto cleanup;
block/qcow2.c +3 -3
index 67e55c9..d9da238 100644
@@ -1635,7 +1635,7 @@ static int qcow2_create2(const char *filename, int64_t total_size,
 
     bs = NULL;
     ret = bdrv_open(&bs, filename, NULL, NULL, BDRV_O_RDWR | BDRV_O_PROTOCOL,
-                    NULL, &local_err);
+                    NULL, &local_err, NULL);
     if (ret < 0) {
         error_propagate(errp, local_err);
         return ret;
@@ -1697,7 +1697,7 @@ static int qcow2_create2(const char *filename, int64_t total_size,
     BlockDriver* drv = bdrv_find_format("qcow2");
     assert(drv != NULL);
     ret = bdrv_open(&bs, filename, NULL, NULL,
-        BDRV_O_RDWR | BDRV_O_CACHE_WB | BDRV_O_NO_FLUSH, drv, &local_err);
+        BDRV_O_RDWR | BDRV_O_CACHE_WB | BDRV_O_NO_FLUSH, drv, &local_err, NULL);
     if (ret < 0) {
         error_propagate(errp, local_err);
         goto out;
@@ -1749,7 +1749,7 @@ static int qcow2_create2(const char *filename, int64_t total_size,
     /* Reopen the image without BDRV_O_NO_FLUSH to flush it before returning */
     ret = bdrv_open(&bs, filename, NULL, NULL,
                     BDRV_O_RDWR | BDRV_O_CACHE_WB | BDRV_O_NO_BACKING,
-                    drv, &local_err);
+                    drv, &local_err, NULL);
     if (local_err) {
         error_propagate(errp, local_err);
         goto out;
block/qed.c +1 -1
index b69374b..3417187 100644
@@ -595,7 +595,7 @@ static int qed_create(const char *filename, uint32_t cluster_size,
     bs = NULL;
     ret = bdrv_open(&bs, filename, NULL, NULL,
                     BDRV_O_RDWR | BDRV_O_CACHE_WB | BDRV_O_PROTOCOL, NULL,
-                    &local_err);
+                    &local_err, NULL);
     if (ret < 0) {
         error_propagate(errp, local_err);
         return ret;
block/sheepdog.c +2 -2
index 8d9350c..bb3dfef 100644
@@ -1576,7 +1576,7 @@ static int sd_prealloc(const char *filename, Error **errp)
     int ret;
 
     ret = bdrv_open(&bs, filename, NULL, NULL, BDRV_O_RDWR | BDRV_O_PROTOCOL,
-                    NULL, errp);
+                    NULL, errp, NULL);
     if (ret < 0) {
         goto out_with_err_set;
     }
@@ -1739,7 +1739,7 @@ static int sd_create(const char *filename, QemuOpts *opts,
 
         bs = NULL;
         ret = bdrv_open(&bs, backing_file, NULL, NULL, BDRV_O_PROTOCOL, NULL,
-                        errp);
+                        errp, NULL);
         if (ret < 0) {
             goto out;
         }
block/vmdk.c +4 -4
index 27a78da..5bafe31 100644
@@ -809,7 +809,7 @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
                 desc_file_path, fname);
         extent_file = NULL;
         ret = bdrv_open(&extent_file, extent_path, NULL, NULL,
-                        bs->open_flags | BDRV_O_PROTOCOL, NULL, errp);
+                        bs->open_flags | BDRV_O_PROTOCOL, NULL, errp, NULL);
         if (ret) {
             return ret;
         }
@@ -1547,7 +1547,7 @@ static int vmdk_create_extent(const char *filename, int64_t filesize,
 
     assert(bs == NULL);
     ret = bdrv_open(&bs, filename, NULL, NULL, BDRV_O_RDWR | BDRV_O_PROTOCOL,
-                    NULL, &local_err);
+                    NULL, &local_err, NULL);
     if (ret < 0) {
         error_propagate(errp, local_err);
         goto exit;
@@ -1805,7 +1805,7 @@ static int vmdk_create(const char *filename, QemuOpts *opts, Error **errp)
     if (backing_file) {
         BlockDriverState *bs = NULL;
         ret = bdrv_open(&bs, backing_file, NULL, NULL, BDRV_O_NO_BACKING, NULL,
-                        errp);
+                        errp, NULL);
         if (ret != 0) {
             goto exit;
         }
@@ -1881,7 +1881,7 @@ static int vmdk_create(const char *filename, QemuOpts *opts, Error **errp)
     }
     assert(new_bs == NULL);
     ret = bdrv_open(&new_bs, filename, NULL, NULL,
-                    BDRV_O_RDWR | BDRV_O_PROTOCOL, NULL, &local_err);
+                    BDRV_O_RDWR | BDRV_O_PROTOCOL, NULL, &local_err, NULL);
     if (ret < 0) {
         error_propagate(errp, local_err);
         goto exit;
block/vvfat.c +1 -1
index 70176b1..f86f890 100644
@@ -2939,7 +2939,7 @@ static int enable_write_target(BDRVVVFATState *s, Error **errp)
     s->qcow = NULL;
     ret = bdrv_open(&s->qcow, s->qcow_filename, NULL, NULL,
                     BDRV_O_RDWR | BDRV_O_CACHE_WB | BDRV_O_NO_FLUSH,
-                    bdrv_qcow, errp);
+                    bdrv_qcow, errp, NULL);
     if (ret < 0) {
         goto err;
     }
blockdev.c +6 -5
index 48bd9a3..e5d4b4d 100644
@@ -44,6 +44,7 @@
 #include "qmp-commands.h"
 #include "trace.h"
 #include "sysemu/arch_init.h"
+#include "replay/replay.h"
 
 static QTAILQ_HEAD(drivelist, DriveInfo) drives = QTAILQ_HEAD_INITIALIZER(drives);
 
@@ -520,7 +521,7 @@ static DriveInfo *blockdev_init(const char *file, QDict *bs_opts,
     bdrv_flags |= ro ? 0 : BDRV_O_RDWR;
 
     QINCREF(bs_opts);
-    ret = bdrv_open(&dinfo->bdrv, file, NULL, bs_opts, bdrv_flags, drv, &error);
+    ret = bdrv_open(&dinfo->bdrv, file, NULL, bs_opts, bdrv_flags, drv, &error, replay_image_suffix);
 
     if (ret < 0) {
         error_setg(errp, "could not open disk image %s: %s",
@@ -1367,7 +1368,7 @@ static void external_snapshot_prepare(BlkTransactionState *common,
      * extended QMP command? */
     assert(state->new_bs == NULL);
     ret = bdrv_open(&state->new_bs, new_image_file, NULL, options,
-                    flags | BDRV_O_NO_BACKING, drv, &local_err);
+                    flags | BDRV_O_NO_BACKING, drv, &local_err, NULL);
     /* We will manually add the backing_hd field to the bs later */
     if (ret != 0) {
         error_propagate(errp, local_err);
@@ -1616,7 +1617,7 @@ static void qmp_bdrv_open_encrypted(BlockDriverState *bs, const char *filename,
     Error *local_err = NULL;
     int ret;
 
-    ret = bdrv_open(&bs, filename, NULL, NULL, bdrv_flags, drv, &local_err);
+    ret = bdrv_open(&bs, filename, NULL, NULL, bdrv_flags, drv, &local_err, NULL);
     if (ret < 0) {
         error_propagate(errp, local_err);
         return;
@@ -2108,7 +2109,7 @@ void qmp_drive_backup(const char *device, const char *target,
     }
 
     target_bs = NULL;
-    ret = bdrv_open(&target_bs, target, NULL, NULL, flags, drv, &local_err);
+    ret = bdrv_open(&target_bs, target, NULL, NULL, flags, drv, &local_err, NULL);
     if (ret < 0) {
         error_propagate(errp, local_err);
         return;
@@ -2282,7 +2283,7 @@ void qmp_drive_mirror(const char *device, const char *target,
      */
     target_bs = NULL;
     ret = bdrv_open(&target_bs, target, NULL, options,
-                    flags | BDRV_O_NO_BACKING, drv, &local_err);
+                    flags | BDRV_O_NO_BACKING, drv, &local_err, NULL);
     if (ret < 0) {
         error_propagate(errp, local_err);
         return;
include/block/block.h +3 -3
index 32d3676..dd05458 100644
@@ -211,13 +211,13 @@ int bdrv_parse_cache_flags(const char *mode, int *flags);
 int bdrv_parse_discard_flags(const char *mode, int *flags);
 int bdrv_open_image(BlockDriverState **pbs, const char *filename,
                     QDict *options, const char *bdref_key, int flags,
-                    bool allow_none, Error **errp);
+                    bool allow_none, Error **errp, const char *fname_suffix);
 void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd);
 int bdrv_open_backing_file(BlockDriverState *bs, QDict *options, Error **errp);
-int bdrv_append_temp_snapshot(BlockDriverState *bs, int flags, Error **errp);
+int bdrv_append_temp_snapshot(BlockDriverState *bs, int flags, Error **errp, const char *fname_suffix);
 int bdrv_open(BlockDriverState **pbs, const char *filename,
               const char *reference, QDict *options, int flags,
-              BlockDriver *drv, Error **errp);
+              BlockDriver *drv, Error **errp, const char *fname_suffix);
 BlockReopenQueue *bdrv_reopen_queue(BlockReopenQueue *bs_queue,
                                     BlockDriverState *bs, int flags);
 int bdrv_reopen_multiple(BlockReopenQueue *bs_queue, Error **errp);
qemu-img.c +3 -3
index c98896b..d38c398 100644
@@ -321,7 +321,7 @@ static BlockDriverState *bdrv_new_open(const char *id,
         drv = NULL;
     }
 
-    ret = bdrv_open(&bs, filename, NULL, NULL, flags, drv, &local_err);
+    ret = bdrv_open(&bs, filename, NULL, NULL, flags, drv, &local_err, NULL);
     if (ret < 0) {
         error_report("Could not open '%s': %s", filename,
                      error_get_pretty(local_err));
@@ -2384,7 +2384,7 @@ static int img_rebase(int argc, char **argv)
         bs_old_backing = bdrv_new("old_backing", &error_abort);
         bdrv_get_backing_filename(bs, backing_name, sizeof(backing_name));
         ret = bdrv_open(&bs_old_backing, backing_name, NULL, NULL, BDRV_O_FLAGS,
-                        old_backing_drv, &local_err);
+                        old_backing_drv, &local_err, NULL);
         if (ret) {
             error_report("Could not open old backing file '%s': %s",
                          backing_name, error_get_pretty(local_err));
@@ -2394,7 +2394,7 @@ static int img_rebase(int argc, char **argv)
         if (out_baseimg[0]) {
             bs_new_backing = bdrv_new("new_backing", &error_abort);
             ret = bdrv_open(&bs_new_backing, out_baseimg, NULL, NULL,
-                            BDRV_O_FLAGS, new_backing_drv, &local_err);
+                            BDRV_O_FLAGS, new_backing_drv, &local_err, NULL);
             if (ret) {
                 error_report("Could not open new backing file '%s': %s",
                              out_baseimg, error_get_pretty(local_err));
qemu-io.c +2 -2
index b55a550..849067d 100644
@@ -60,7 +60,7 @@ static int openfile(char *name, int flags, int growable, QDict *opts)
 
     if (growable) {
         if (bdrv_open(&qemuio_bs, name, NULL, opts, flags | BDRV_O_PROTOCOL,
-                      NULL, &local_err))
+                      NULL, &local_err, NULL))
         {
             fprintf(stderr, "%s: can't open%s%s: %s\n", progname,
                     name ? " device " : "", name ?: "",
@@ -71,7 +71,7 @@ static int openfile(char *name, int flags, int growable, QDict *opts)
     } else {
         qemuio_bs = bdrv_new("hda", &error_abort);
 
-        if (bdrv_open(&qemuio_bs, name, NULL, opts, flags, NULL, &local_err)
+        if (bdrv_open(&qemuio_bs, name, NULL, opts, flags, NULL, &local_err, NULL)
             < 0)
         {
             fprintf(stderr, "%s: can't open%s%s: %s\n", progname,
qemu-nbd.c +1 -1
index 626e584..e780d1c 100644
@@ -663,7 +663,7 @@ int main(int argc, char **argv)
     bs = bdrv_new("hda", &error_abort);
 
     srcpath = argv[optind];
-    ret = bdrv_open(&bs, srcpath, NULL, NULL, flags, drv, &local_err);
+    ret = bdrv_open(&bs, srcpath, NULL, NULL, flags, drv, &local_err, NULL);
     if (ret < 0) {
         errno = -ret;
         err(EXIT_FAILURE, "Failed to bdrv_open '%s': %s", argv[optind],
#40 lore raw

RFC PATCH v2 21/49 sysemu: system functions for replay

In reply to Pavel Dovgalyuk

This patch removes "static" specifier from several qemu function to make them visible to the replay module. It also invents several system functions that will be used by replay.

---
 arch_init.c             |    8 ++++++++
 cpu-exec.c              |    6 ++++--
 cpus.c                  |    4 ++--
 include/exec/exec-all.h |    4 +++-
 include/sysemu/cpus.h   |    1 +
 include/sysemu/sysemu.h |    1 +
 savevm.c                |   32 ++++++++++++++++++++++++++------
 translate-all.c         |    8 ++++++++
 8 files changed, 53 insertions(+), 11 deletions(-)
arch_init.c +8 -0
index 8ddaf35..5ee33d9 100644
@@ -52,6 +52,7 @@
 #include "exec/ram_addr.h"
 #include "hw/acpi/acpi.h"
 #include "qemu/host-utils.h"
+#include "replay/replay.h"
 
 #ifdef DEBUG_ARCH_INIT
 #define DPRINTF(fmt, ...) \
@@ -1092,6 +1093,13 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
 
                 total_ram_bytes -= length;
             }
+            if (replay_mode == REPLAY_PLAY) {
+                RAMBlock *block;
+                /* Clear the blocks' memory instead of resetting the machine */
+                QTAILQ_FOREACH(block, &ram_list.blocks, next) {
+                    memset(block->host, 0, block->length);
+                }
+            }
         } else if (flags & RAM_SAVE_FLAG_COMPRESS) {
             void *host;
             uint8_t ch;
cpu-exec.c +4 -2
index 38e5f02..66a693c 100644
@@ -189,12 +189,14 @@ static inline TranslationBlock *tb_find_fast(CPUArchState *env)
 
 static CPUDebugExcpHandler *debug_excp_handler;
 
-void cpu_set_debug_excp_handler(CPUDebugExcpHandler *handler)
+CPUDebugExcpHandler *cpu_set_debug_excp_handler(CPUDebugExcpHandler *handler)
 {
+    CPUDebugExcpHandler *old = debug_excp_handler;
     debug_excp_handler = handler;
+    return old;
 }
 
-static void cpu_handle_debug_exception(CPUArchState *env)
+void cpu_handle_debug_exception(CPUArchState *env)
 {
     CPUState *cpu = ENV_GET_CPU(env);
     CPUWatchpoint *wp;
cpus.c +2 -2
index 5e7f2cf..bbad529 100644
@@ -85,7 +85,7 @@ static bool cpu_thread_is_idle(CPUState *cpu)
     return true;
 }
 
-static bool all_cpu_threads_idle(void)
+bool all_cpu_threads_idle(void)
 {
     CPUState *cpu;
 
@@ -1033,7 +1033,7 @@ bool qemu_cpu_is_self(CPUState *cpu)
     return qemu_thread_is_self(cpu->thread);
 }
 
-static bool qemu_in_vcpu_thread(void)
+bool qemu_in_vcpu_thread(void)
 {
     return current_cpu && qemu_cpu_is_self(current_cpu);
 }
include/exec/exec-all.h +3 -1
index 528928f..4b7a241 100644
@@ -212,6 +212,7 @@ static inline unsigned int tb_phys_hash_func(tb_page_addr_t pc)
 
 void tb_free(TranslationBlock *tb);
 void tb_flush(CPUArchState *env);
+void tb_flush_all(void);
 void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr);
 
 #if defined(USE_DIRECT_JUMP)
@@ -385,7 +386,8 @@ tb_page_addr_t get_page_addr_code(CPUArchState *env1, target_ulong addr);
 
 typedef void (CPUDebugExcpHandler)(CPUArchState *env);
 
-void cpu_set_debug_excp_handler(CPUDebugExcpHandler *handler);
+CPUDebugExcpHandler *cpu_set_debug_excp_handler(CPUDebugExcpHandler *handler);
+void cpu_handle_debug_exception(CPUArchState *env);
 
 /* vl.c */
 extern int singlestep;
include/sysemu/cpus.h +1 -0
index 4f79081..8455e0e 100644
@@ -6,6 +6,7 @@ void qemu_init_cpu_loop(void);
 void resume_all_vcpus(void);
 void pause_all_vcpus(void);
 void cpu_stop_current(void);
+bool all_cpu_threads_idle(void);
 
 void cpu_synchronize_all_states(void);
 void cpu_synchronize_all_post_reset(void);
include/sysemu/sysemu.h +1 -0
index d8539fd..1ebfef9 100644
@@ -76,6 +76,7 @@ void qemu_add_machine_init_done_notifier(Notifier *notify);
 
 void do_savevm(Monitor *mon, const QDict *qdict);
 int load_vmstate(const char *name);
+int save_vmstate(Monitor *mon, const char *name);
 void do_delvm(Monitor *mon, const QDict *qdict);
 void do_info_snapshots(Monitor *mon, const QDict *qdict);
 
savevm.c +26 -6
index e19ae0a..9136778 100644
@@ -42,7 +42,7 @@
 #include "qemu/iov.h"
 #include "block/snapshot.h"
 #include "block/qapi.h"
-
+#include "replay/replay.h"
 
 #ifndef ETH_P_RARP
 #define ETH_P_RARP 0x8035
@@ -1039,7 +1039,7 @@ static int del_existing_snapshots(Monitor *mon, const char *name)
     return 0;
 }
 
-void do_savevm(Monitor *mon, const QDict *qdict)
+int save_vmstate(Monitor *mon, const char *name)
 {
     BlockDriverState *bs, *bs1;
     QEMUSnapshotInfo sn1, *sn = &sn1, old_sn1, *old_sn = &old_sn1;
@@ -1049,7 +1049,7 @@ void do_savevm(Monitor *mon, const QDict *qdict)
     uint64_t vm_state_size;
     qemu_timeval tv;
     struct tm tm;
-    const char *name = qdict_get_try_str(qdict, "name");
+    int success = 0;
 
     /* Verify if there is a device that doesn't support snapshots and is writable */
     bs = NULL;
@@ -1062,14 +1062,18 @@ void do_savevm(Monitor *mon, const QDict *qdict)
         if (!bdrv_can_snapshot(bs)) {
             monitor_printf(mon, "Device '%s' is writable but does not support snapshots.\n",
                                bdrv_get_device_name(bs));
-            return;
+            return success;
         }
     }
 
     bs = find_vmstate_bs();
     if (!bs) {
         monitor_printf(mon, "No block device can accept snapshots\n");
-        return;
+        if (replay_mode != REPLAY_NONE) {
+            fprintf(stderr, "At least one hdd should be attached to QEMU for replay\n");
+            exit(1);
+        }
+        return success;
     }
 
     saved_vm_running = runstate_is_running();
@@ -1118,6 +1122,7 @@ void do_savevm(Monitor *mon, const QDict *qdict)
 
     /* create the snapshots */
 
+    success = 1;
     bs1 = NULL;
     while ((bs1 = bdrv_next(bs1))) {
         if (bdrv_can_snapshot(bs1)) {
@@ -1127,6 +1132,7 @@ void do_savevm(Monitor *mon, const QDict *qdict)
             if (ret < 0) {
                 monitor_printf(mon, "Error while creating snapshot on '%s'\n",
                                bdrv_get_device_name(bs1));
+                success = 0;
             }
         }
     }
@@ -1135,6 +1141,14 @@ void do_savevm(Monitor *mon, const QDict *qdict)
     if (saved_vm_running) {
         vm_start();
     }
+
+    return success;
+}
+
+void do_savevm(Monitor *mon, const QDict *qdict)
+{
+    const char *name = qdict_get_try_str(qdict, "name");
+    save_vmstate(mon, name);
 }
 
 void qmp_xen_save_devices_state(const char *filename, Error **errp)
@@ -1231,7 +1245,13 @@ int load_vmstate(const char *name)
         return -EINVAL;
     }
 
-    qemu_system_reset(VMRESET_SILENT);
+    /* Do not reset in replay mode.
+       1. Reset will alter the behavior in play mode compared to save one
+       2. Timers read by reset handlers are not correct, because
+          replay reads them from the unknown part of the log */
+    if (replay_mode == REPLAY_NONE) {
+        qemu_system_reset(VMRESET_SILENT);
+    }
     ret = qemu_loadvm_state(f);
 
     qemu_fclose(f);
translate-all.c +8 -0
index 8f7e11b..fcb0697 100644
@@ -803,6 +803,14 @@ void tb_flush(CPUArchState *env1)
     tcg_ctx.tb_ctx.tb_flush_count++;
 }
 
+void tb_flush_all(void)
+{
+    CPUState *cpu;
+    for (cpu = first_cpu ; cpu != NULL ; cpu = CPU_NEXT(cpu)) {
+        tb_flush(cpu->env_ptr);
+    }
+}
+
 #ifdef DEBUG_TB_CHECK
 
 static void tb_invalidate_check(target_ulong address)
#41 lore raw

RFC PATCH v2 22/49 replay: internal functions for replay log

In reply to Pavel Dovgalyuk

This patch adds functions to perform read and write operations with replay log.

---
 replay/Makefile.objs     |    1 
 replay/replay-internal.c |  141 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   50 ++++++++++++++++
 replay/replay.c          |    6 ++
 4 files changed, 198 insertions(+), 0 deletions(-)
 create mode 100755 replay/replay-internal.c
 create mode 100755 replay/replay-internal.h
replay/Makefile.objs +1 -0
index 7ea860f..1148f45 100755
@@ -1 +1,2 @@
 obj-y += replay.o
+obj-y += replay-internal.o
replay/replay-internal.c new file +141 -0

141 lines not on this page · raw message

replay/replay-internal.h new file +50 -0

50 lines not on this page · raw message

replay/replay.c +6 -0
index 5a69465..2961e17 100755
@@ -17,3 +17,9 @@ int play_submode = REPLAY_PLAY_UNKNOWN;
 
 /* Suffix for the disk images filenames */
 char *replay_image_suffix;
+
+
+int replay_get_play_submode(void)
+{
+    return play_submode;
+}
#42 lore raw

RFC PATCH v2 23/49 cpu: invent instruction count for accurate replay

In reply to Pavel Dovgalyuk

This patch adds instructions count fields to cpu structure and invents several functions for increasing this counter while executing translation blocks.

---
 cpu-exec.c               |   14 ++++-
 cpus.c                   |    5 ++
 exec.c                   |    4 +
 include/exec/cpu-defs.h  |    1 
 include/qom/cpu.h        |    4 +
 replay/Makefile.objs     |    1 
 replay/replay-events.c   |   26 +++++++++
 replay/replay-internal.c |   14 +++++
 replay/replay-internal.h |   37 ++++++++++++-
 replay/replay.c          |  136 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   16 +++++
 translate-all.c          |    7 ++
 12 files changed, 260 insertions(+), 5 deletions(-)
 create mode 100755 replay/replay-events.c
cpu-exec.c +12 -2
index 66a693c..13c0ec6 100644
@@ -22,6 +22,7 @@
 #include "tcg.h"
 #include "qemu/atomic.h"
 #include "sysemu/qtest.h"
+#include "qemu/main-loop.h"
 
 void cpu_loop_exit(CPUState *cpu)
 {
@@ -283,19 +284,24 @@ int cpu_exec(CPUArchState *env)
 #else
 #error unsupported target CPU
 #endif
-    cpu->exception_index = -1;
-
     /* prepare setjmp context for exception handling */
     for(;;) {
         if (sigsetjmp(cpu->jmp_env, 0) == 0) {
             /* if an exception is pending, we execute it here */
             if (cpu->exception_index >= 0) {
+                if (cpu->exception_index == EXCP_REPLAY) {
+                    ret = cpu->exception_index;
+                    cpu->exception_index = -1;
+                    qemu_notify_event();
+                    break;
+                }
                 if (cpu->exception_index >= EXCP_INTERRUPT) {
                     /* exit request from the cpu execution loop */
                     ret = cpu->exception_index;
                     if (ret == EXCP_DEBUG) {
                         cpu_handle_debug_exception(env);
                     }
+                    cpu->exception_index = -1;
                     break;
                 } else {
 #if defined(CONFIG_USER_ONLY)
@@ -601,6 +607,10 @@ int cpu_exec(CPUArchState *env)
                         next_tb = 0;
                     }
                 }
+                if (cpu->exception_index == EXCP_REPLAY) {
+                    /* go to exception_index checking */
+                    break;
+                }
                 if (unlikely(cpu->exit_request)) {
                     cpu->exit_request = 0;
                     cpu->exception_index = EXCP_INTERRUPT;
cpus.c +5 -0
index bbad529..eb2a795 100644
@@ -937,6 +937,8 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)
     CPU_FOREACH(cpu) {
         cpu->thread_id = qemu_get_thread_id();
         cpu->created = true;
+        /* init exception index here */
+        cpu->exception_index = -1;
     }
     qemu_cond_signal(&qemu_cpu_cond);
 
@@ -1186,6 +1188,7 @@ void qemu_init_vcpu(CPUState *cpu)
     cpu->nr_cores = smp_cores;
     cpu->nr_threads = smp_threads;
     cpu->stopped = true;
+    cpu->instructions_count = 0;
     if (kvm_enabled()) {
         qemu_kvm_start_vcpu(cpu);
     } else if (tcg_enabled()) {
@@ -1307,6 +1310,8 @@ static void tcg_exec_all(void)
             if (r == EXCP_DEBUG) {
                 cpu_handle_guest_debug(cpu);
                 break;
+            } else if (r == EXCP_REPLAY) {
+                break;
             }
         } else if (cpu->stop || cpu->stopped) {
             break;
exec.c +4 -0
index 5a2a25e..36adb62 100644
@@ -52,6 +52,7 @@
 #include "exec/ram_addr.h"
 
 #include "qemu/range.h"
+#include "replay/replay.h"
 
 //#define DEBUG_SUBPAGE
 
@@ -1615,6 +1616,9 @@ static void check_watchpoint(int offset, int len_mask, int flags)
             if (!cpu->watchpoint_hit) {
                 cpu->watchpoint_hit = wp;
                 tb_check_watchpoint(cpu);
+                /* Current instruction is already processed by replay.
+                   Set flags that allow skpping this events */
+                replay_undo_last_instruction();
                 if (wp->flags & BP_STOP_BEFORE_ACCESS) {
                     cpu->exception_index = EXCP_DEBUG;
                     cpu_loop_exit(cpu);
include/exec/cpu-defs.h +1 -0
index 2dd6206..db9ae1a 100644
@@ -59,6 +59,7 @@ typedef uint64_t target_ulong;
 #define EXCP_DEBUG      0x10002 /* cpu stopped after a breakpoint or singlestep */
 #define EXCP_HALTED     0x10003 /* cpu is halted (waiting for external event) */
 #define EXCP_YIELD      0x10004 /* cpu wants to yield timeslice to another */
+#define EXCP_REPLAY     0x10005 /* for breaking execution loop to make correct order of events */
 
 /* Only the bottom TB_JMP_PAGE_BITS of the jump cache hash bits vary for
    addresses on the same page.  The top bits are the same.  This allows
include/qom/cpu.h +4 -0
index 1aafbf5..9c7b5c8 100644
@@ -287,6 +287,7 @@ struct CPUState {
        (absolute value) offset as small as possible.  This reduces code
        size, especially for hosts without large memory offsets.  */
     volatile sig_atomic_t tcg_exit_req;
+    uint32_t instructions_count;
 };
 
 QTAILQ_HEAD(CPUTailQ, CPUState);
@@ -466,6 +467,9 @@ static inline bool cpu_has_work(CPUState *cpu)
  */
 bool qemu_cpu_is_self(CPUState *cpu);
 
+//need here for replay work
+bool qemu_in_vcpu_thread(void);
+
 /**
  * qemu_cpu_kick:
  * @cpu: The vCPU to kick.
replay/Makefile.objs +1 -0
index 1148f45..56da09c 100755
@@ -1,2 +1,3 @@
 obj-y += replay.o
 obj-y += replay-internal.o
+obj-y += replay-events.o
replay/replay-events.c new file +26 -0
new file mode 100755
index 0000000..eaffc48
@@ -0,0 +1,26 @@
+/*
+ * replay-events.c
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming 
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "replay.h"
+#include "replay-internal.h"
+
+bool replay_has_events(void)
+{
+    return false;
+}
+
+void replay_save_events(int opt)
+{
+}
+
+void replay_read_events(int opt)
+{
+}
replay/replay-internal.c +14 -0
index 706b46b..f416317 100755
@@ -10,6 +10,7 @@
  */
 
 #include "qemu-common.h"
+#include "replay.h"
 #include "replay-internal.h"
 
 volatile unsigned int replay_data_kind = -1;
@@ -139,3 +140,16 @@ void replay_fetch_data_kind(void)
         }
     }
 }
+
+/*! Saves cached instructions. */
+void replay_save_instructions(void)
+{
+    if (replay_file && replay_mode == REPLAY_SAVE) {
+        if (first_cpu != NULL && first_cpu->instructions_count > 0) {
+            replay_put_event(EVENT_INSTRUCTION);
+            replay_put_dword(first_cpu->instructions_count);
+            replay_state.current_step += first_cpu->instructions_count;
+            first_cpu->instructions_count = 0;
+        }
+    }
+}
replay/replay-internal.h +34 -3
index b959bca..9ebfa83 100755
@@ -13,6 +13,21 @@
  */
 
 #include <stdio.h>
+#include "sysemu/sysemu.h"
+
+/* for async events */
+#define EVENT_ASYNC                 24
+/* for instruction event */
+#define EVENT_INSTRUCTION           32
+
+typedef struct ReplayState {
+    /*! Nonzero, when next instruction is repeated one and was already
+        processed. */
+    int skipping_instruction;
+    /*! Current step - number of processed instructions and timer events. */
+    uint64_t current_step;
+} ReplayState;
+extern ReplayState replay_state;
 
 extern volatile unsigned int replay_data_kind;
 extern volatile unsigned int replay_has_unread_data;
@@ -40,11 +55,27 @@ void replay_check_error(void);
 /*! Reads data type from the file and stores it in the
     replay_data_kind variable. */
 void replay_fetch_data_kind(void);
-
-/*! Saves queued events (like instructions and sound). */
-void replay_save_instructions(void);
 /*! Checks that the next data is corresponding to the desired kind.
     Terminates the program in case of error. */
 void validate_data_kind(int kind);
 
+/*! Saves queued events (like instructions and sound). */
+void replay_save_instructions(void);
+
+/*! Skips async events until some sync event will be found. */
+bool skip_async_events(int stop_event);
+/*! Skips async events invocations from the input,
+    until required data kind is found. If the requested data is not found
+    reports an error and stops the execution. */
+void skip_async_events_until(unsigned int kind);
+
+/* Asynchronous events queue */
+
+/*! Returns true if there are any unsaved events in the queue */
+bool replay_has_events(void);
+/*! Saves events from queue into the file */
+void replay_save_events(int opt);
+/*! Read events from the file into the input queue */
+void replay_read_events(int opt);
+
 #endif
replay/replay.c +136 -0

151 lines not on this page · raw message

replay/replay.h +16 -0
index a883aa4..1ce7f78 100755
@@ -12,6 +12,9 @@
  *
  */
 
+#include <stdbool.h>
+#include <stdint.h>
+
 /* replay modes */
 #define REPLAY_NONE 0
 #define REPLAY_SAVE 1
@@ -29,4 +32,17 @@ extern int replay_icount;
 /*! Returns replay play submode */
 int replay_get_play_submode(void);
 
+/* Processing the instructions */
+
+/*! Returns number of executed instructions. */
+uint64_t replay_get_current_step(void);
+/*! Called before instruction execution */
+void replay_instruction(int process_events);
+/*! Undo last instruction count, when exception occurs */
+void replay_undo_last_instruction(void);
+/*! Returns true if asynchronous event is pending */
+bool replay_has_async_request(void);
+/*! Returns non-zero if next event is instruction. */
+bool replay_has_instruction(void);
+
 #endif
translate-all.c +7 -0
index fcb0697..37582a2 100644
@@ -59,6 +59,7 @@
 #include "exec/cputlb.h"
 #include "translate-all.h"
 #include "qemu/timer.h"
+#include "replay/replay.h"
 
 //#define DEBUG_TB_INVALIDATE
 //#define DEBUG_FLUSH
@@ -1171,6 +1172,9 @@ void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end,
                 cpu_restore_state_from_tb(cpu, current_tb, cpu->mem_io_pc);
                 cpu_get_tb_cpu_state(env, &current_pc, &current_cs_base,
                                      &current_flags);
+                /* Current instruction is already processed by replay.
+                   Set flags that allow skpping this event */
+                replay_undo_last_instruction();
             }
 #endif /* TARGET_HAS_PRECISE_SMC */
             /* we need to do that to handle the case where a signal
@@ -1290,6 +1294,9 @@ static void tb_invalidate_phys_page(tb_page_addr_t addr,
             cpu_restore_state_from_tb(cpu, current_tb, pc);
             cpu_get_tb_cpu_state(env, &current_pc, &current_cs_base,
                                  &current_flags);
+            /* Current instruction is already processed by replay.
+               Set flags that allow skpping this event */
+            replay_undo_last_instruction();
         }
 #endif /* TARGET_HAS_PRECISE_SMC */
         tb_phys_invalidate(tb, addr);
#43 lore raw

RFC PATCH v2 24/49 target-arm: instructions counting code for replay

In reply to Pavel Dovgalyuk

This patch adds instructions counting into the target-specific part of arm simulator. In record/replay mode it inserts replay functions calls and instructions counter increment into the translated code.

---
 target-arm/Makefile.objs   |    1 +
 target-arm/helper.h        |    2 ++
 target-arm/replay_helper.c |   33 ++++++++++++++++++++++++++++++
 target-arm/translate.c     |   48 +++++++++++++++++++++++++++++++++++++++++---
 4 files changed, 81 insertions(+), 3 deletions(-)
 create mode 100755 target-arm/replay_helper.c
target-arm/Makefile.objs +1 -0
index dcd167e..b8f61df 100644
@@ -9,3 +9,4 @@ obj-y += neon_helper.o iwmmxt_helper.o
 obj-y += gdbstub.o
 obj-$(TARGET_AARCH64) += cpu64.o translate-a64.o helper-a64.o gdbstub64.o
 obj-y += crypto_helper.o
+obj-y += replay_helper.o
target-arm/helper.h +2 -0
index facfcd2..0233b64 100644
@@ -529,3 +529,5 @@ DEF_HELPER_FLAGS_2(neon_pmull_64_hi, TCG_CALL_NO_RWG_SE, i64, i64, i64)
 #ifdef TARGET_AARCH64
 #include "helper-a64.h"
 #endif
+
+DEF_HELPER_1(replay_instruction, i32, env)
target-arm/replay_helper.c new file +33 -0
new file mode 100755
index 0000000..736cd51
@@ -0,0 +1,33 @@
+/*
+ * replay_helper.c
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming 
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "cpu.h"
+#include "exec/helper-proto.h"
+#include "replay/replay.h"
+
+uint32_t helper_replay_instruction(CPUARMState *env)
+{
+    CPUState *cpu = ENV_GET_CPU(env);
+    if (replay_mode == REPLAY_PLAY
+        && !replay_has_instruction()) {
+        cpu->exception_index = EXCP_REPLAY;
+        return 1;
+    }
+
+    if (cpu->exit_request) {
+        cpu->exception_index = EXCP_REPLAY;
+        return 1;
+    }
+
+    int timer = replay_has_async_request();
+    replay_instruction(timer);
+    return timer;
+}
target-arm/translate.c +45 -3

80 lines not on this page · raw message

#44 lore raw

RFC PATCH v2 25/49 target-i386: instructions counting code for replay

In reply to Pavel Dovgalyuk

This patch adds instructions counting into the target-specific part of i386/x86_64 simulator. In record/replay mode it inserts replay functions calls and instructions counter increment into the translated code.

---
 target-i386/Makefile.objs   |    1 +
 target-i386/cpu.h           |    7 ++++
 target-i386/helper.h        |    2 +
 target-i386/replay_helper.c |   33 ++++++++++++++++++
 target-i386/translate.c     |   79 +++++++++++++++++++++++++++++++++++++++++--
 5 files changed, 118 insertions(+), 4 deletions(-)
 create mode 100755 target-i386/replay_helper.c
target-i386/Makefile.objs +1 -0
index 027b94e..09a6e8a 100644
@@ -2,6 +2,7 @@ obj-y += translate.o helper.o cpu.o
 obj-y += excp_helper.o fpu_helper.o cc_helper.o int_helper.o svm_helper.o
 obj-y += smm_helper.o misc_helper.o mem_helper.o seg_helper.o
 obj-y += gdbstub.o
+obj-y += replay_helper.o
 obj-$(CONFIG_SOFTMMU) += machine.o arch_memory_mapping.o arch_dump.o
 obj-$(CONFIG_KVM) += kvm.o
 obj-$(call lnot,$(CONFIG_KVM)) += kvm-stub.o
target-i386/cpu.h +7 -0
index 42bda46..c9b92af 100644
@@ -28,6 +28,13 @@
 #define TARGET_LONG_BITS 32
 #endif
 
+/* Maximum instruction code size */
+#ifdef TARGET_X86_64
+#define TARGET_MAX_INSN_SIZE 16
+#else
+#define TARGET_MAX_INSN_SIZE 16
+#endif
+
 /* target supports implicit self modifying code */
 #define TARGET_HAS_SMC
 /* support for self modifying code even if the modified instruction is
target-i386/helper.h +2 -0
index 8eb0145..058302b 100644
@@ -217,3 +217,5 @@ DEF_HELPER_3(rcrl, tl, env, tl, tl)
 DEF_HELPER_3(rclq, tl, env, tl, tl)
 DEF_HELPER_3(rcrq, tl, env, tl, tl)
 #endif
+
+DEF_HELPER_1(replay_instruction, i32, env)
target-i386/replay_helper.c new file +33 -0
new file mode 100755
index 0000000..8b82eee
@@ -0,0 +1,33 @@
+/*
+ * replay_helper.c
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming 
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "cpu.h"
+#include "exec/helper-proto.h"
+#include "replay/replay.h"
+
+uint32_t helper_replay_instruction(CPUX86State *env)
+{
+    CPUState *cpu = ENV_GET_CPU(env);
+    if (replay_mode == REPLAY_PLAY
+        && !replay_has_instruction()) {
+        cpu->exception_index = EXCP_REPLAY;
+        return 1;
+    }
+
+    if (cpu->exit_request) {
+        cpu->exception_index = EXCP_REPLAY;
+        return 1;
+    }
+
+    int timer = replay_has_async_request();
+    replay_instruction(timer);
+    return timer;
+}
target-i386/translate.c +75 -4

135 lines not on this page · raw message

#45 lore raw

RFC PATCH v2 26/49 replay: interrupts and exceptions

In reply to Pavel Dovgalyuk

This patch includes modifications of common cpu files. All interrupts and exceptions occured during recording are written into the replay log. These events allow correct replaying the execution by kicking cpu thread when one of these events is found in the log.

---
 cpu-exec.c               |   14 +++++++++++---
 replay/replay-internal.h |    4 ++++
 replay/replay.c          |   42 ++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   14 ++++++++++++++
 4 files changed, 71 insertions(+), 3 deletions(-)
cpu-exec.c +11 -3
index 13c0ec6..fcde7ce 100644
@@ -23,6 +23,7 @@
 #include "qemu/atomic.h"
 #include "sysemu/qtest.h"
 #include "qemu/main-loop.h"
+#include "replay/replay.h"
 
 void cpu_loop_exit(CPUState *cpu)
 {
@@ -314,8 +315,14 @@ int cpu_exec(CPUArchState *env)
                     ret = cpu->exception_index;
                     break;
 #else
-                    cc->do_interrupt(cpu);
-                    cpu->exception_index = -1;
+                    if (replay_exception()) {
+                        cc->do_interrupt(cpu);
+                        cpu->exception_index = -1;
+                    } else if (!replay_has_interrupt()) {
+                        /* give a chance to iothread in replay mode */
+                        ret = EXCP_REPLAY;
+                        break;
+                    }
 #endif
                 }
             }
@@ -323,7 +330,8 @@ int cpu_exec(CPUArchState *env)
             next_tb = 0; /* force lookup of first TB */
             for(;;) {
                 interrupt_request = cpu->interrupt_request;
-                if (unlikely(interrupt_request)) {
+                if (unlikely((interrupt_request & CPU_INTERRUPT_DEBUG)
+                             || (interrupt_request && replay_interrupt()))) {
                     if (unlikely(cpu->singlestep_enabled & SSTEP_NOIRQ)) {
                         /* Mask out external interrupts for this step. */
                         interrupt_request &= ~CPU_INTERRUPT_SSTEP_MASK;
replay/replay-internal.h +4 -0
index 9ebfa83..019ae20 100755
@@ -15,6 +15,10 @@
 #include <stdio.h>
 #include "sysemu/sysemu.h"
 
+/* for software interrupt */
+#define EVENT_INTERRUPT             15
+/* for emulated exceptions */
+#define EVENT_EXCEPTION             23
 /* for async events */
 #define EVENT_ASYNC                 24
 /* for instruction event */
replay/replay.c +42 -0

45 lines not on this page · raw message

replay/replay.h +14 -0
index 1ce7f78..10e6681 100755
@@ -45,4 +45,18 @@ bool replay_has_async_request(void);
 /*! Returns non-zero if next event is instruction. */
 bool replay_has_instruction(void);
 
+/* Interrupts and exceptions */
+
+/*! Called by exception handler to write or read
+    exception processing events. */
+bool replay_exception(void);
+/*! Called by interrupt handlers to write or read
+    interrupt processing events.
+    \return true if interrupt should be processed */
+bool replay_interrupt(void);
+/*! Tries to read interrupt event from the file.
+    Returns true, when interrupt request is pending */
+bool replay_has_interrupt(void);
+
+
 #endif
#46 lore raw

RFC PATCH v2 27/49 vga: do not use virtual clock for blinking cursor

In reply to Pavel Dovgalyuk

This patch changes virtual clock to realtime one in vga_draw_text and vga_update_display functions. Display update process depends on realtime clock. If we read virtual clock there, virtual machine becomes non-deterministic, because every read to virtual clock is written to the replay log.

---
 hw/display/vga.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
hw/display/vga.c +2 -2
index 4b089a3..02d3be3 100644
@@ -1304,7 +1304,7 @@ static void vga_draw_text(VGACommonState *s, int full_update)
     uint32_t *ch_attr_ptr;
     vga_draw_glyph8_func *vga_draw_glyph8;
     vga_draw_glyph9_func *vga_draw_glyph9;
-    int64_t now = qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL);
+    int64_t now = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
 
     /* compute font data address (in plane 2) */
     v = s->sr[VGA_SEQ_CHARACTER_MAP];
@@ -1906,7 +1906,7 @@ static void vga_update_display(void *opaque)
         }
         if (graphic_mode != s->graphic_mode) {
             s->graphic_mode = graphic_mode;
-            s->cursor_blink_time = qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL);
+            s->cursor_blink_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
             full_update = 1;
         }
         switch(graphic_mode) {
#47 lore raw

RFC PATCH v2 28/49 replay: asynchronous events infrastructure

In reply to Pavel Dovgalyuk

This patch adds module for saving and replaying asynchronous events. These events include network packets, keyboard and mouse input, USB packets, thread pool and bottom halves callbacks.
All events are stored in the queue to be processed at synchronization points such as beginning of TB execution, or checkpoint in the iothread.

---
 replay/replay-events.c   |  198 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   17 ++++
 replay/replay.h          |    4 +
 3 files changed, 218 insertions(+), 1 deletions(-)
replay/replay-events.c +197 -1
index eaffc48..eda574e 100755
@@ -12,15 +12,211 @@
 #include "replay.h"
 #include "replay-internal.h"
 
+typedef struct Event {
+    int event_kind;
+    void *opaque;
+    void *opaque2;
+    uint64_t id;
+
+    QTAILQ_ENTRY(Event) events;
+} Event;
+
+static QTAILQ_HEAD(, Event) events_list = QTAILQ_HEAD_INITIALIZER(events_list);
+
+static QemuMutex lock;
+static unsigned int read_event_kind = -1;
+static uint64_t read_id = -1;
+static int read_opt = -1;
+
+static bool replay_events_enabled = false;
+
+/* Functions */
+
+static void replay_run_event(Event *event)
+{
+    switch (event->event_kind) {
+    default:
+        fprintf(stderr, "Replay: invalid async event ID (%d) in the queue\n", 
+                event->event_kind);
+        exit(1);
+        break;
+    }
+}
+
+void replay_enable_events(void)
+{
+    replay_events_enabled = true;
+}
+
 bool replay_has_events(void)
 {
-    return false;
+    return !QTAILQ_EMPTY(&events_list);
+}
+
+void replay_flush_events(void)
+{
+    qemu_mutex_lock(&lock);
+    while (!QTAILQ_EMPTY(&events_list)) {
+        Event *event = QTAILQ_FIRST(&events_list);
+        replay_run_event(event);
+        QTAILQ_REMOVE(&events_list, event, events);
+        g_free(event);
+    }
+    qemu_mutex_unlock(&lock);
+}
+
+void replay_disable_events(void)
+{
+    replay_events_enabled = false;
+    /* Flush events queue before waiting of completion */
+    replay_flush_events();
+}
+
+void replay_clear_events(void)
+{
+    qemu_mutex_lock(&lock);
+    while (!QTAILQ_EMPTY(&events_list)) {
+        Event *event = QTAILQ_FIRST(&events_list);
+        QTAILQ_REMOVE(&events_list, event, events);
+
+        g_free(event);
+    }
+    qemu_mutex_unlock(&lock);
+}
+
+static void replay_add_event_internal(int event_kind, void *opaque, void *opaque2, uint64_t id)
+{
+    if (event_kind >= REPLAY_ASYNC_COUNT) {
+        fprintf(stderr, "Replay: invalid async event ID (%d)\n", event_kind);
+        exit(1);
+    }
+    if (!replay_file || replay_mode == REPLAY_NONE
+        || !replay_events_enabled) {
+        Event e;
+        e.event_kind = event_kind;
+        e.opaque = opaque;
+        e.opaque2 = opaque2;
+        e.id = id;
+        replay_run_event(&e);
+        return;
+    }
+
+    Event *event = g_malloc0(sizeof(Event));
+    event->event_kind = event_kind;
+    event->opaque = opaque;
+    event->opaque2 = opaque2;
+    event->id = id;
+
+    qemu_mutex_lock(&lock);
+    QTAILQ_INSERT_TAIL(&events_list, event, events);
+    qemu_mutex_unlock(&lock);
+}
+
+void replay_add_event(int event_kind, void *opaque)
+{
+    replay_add_event_internal(event_kind, opaque, NULL, 0);
 }
 
 void replay_save_events(int opt)
 {
+    qemu_mutex_lock(&lock);
+    while (!QTAILQ_EMPTY(&events_list)) {
+        Event *event = QTAILQ_FIRST(&events_list);
+        if (replay_mode != REPLAY_PLAY) {
+            /* put the event into the file */
+            if (opt == -1) {
+                replay_put_event(EVENT_ASYNC);
+            } else {
+                replay_put_event(EVENT_ASYNC_OPT);
+                replay_put_byte(opt);
+            }
+            replay_put_byte(event->event_kind);
+
+            /* save event-specific data */
+            switch (event->event_kind) {
+            }
+        }
+
+        replay_run_event(event);
+        QTAILQ_REMOVE(&events_list, event, events);
+        g_free(event);
+    }
+    qemu_mutex_unlock(&lock);
 }
 
 void replay_read_events(int opt)
 {
+    replay_fetch_data_kind();
+    while ((opt == -1 && replay_data_kind == EVENT_ASYNC)
+        || (opt != -1 && replay_data_kind == EVENT_ASYNC_OPT)) {
+
+        if (read_event_kind == -1) {
+            if (opt != -1) {
+                read_opt = replay_get_byte();
+            }
+            read_event_kind = replay_get_byte();
+            read_id = -1;
+            replay_check_error();
+        }
+        
+        if (opt != read_opt)
+            break;
+        /* Execute some events without searching them in the queue */
+        switch (read_event_kind) {
+        default:
+            fprintf(stderr, "Unknown ID %d of replay event\n", read_event_kind);
+            exit(1);
+            break;
+        }
+
+        qemu_mutex_lock(&lock);
+
+        Event *event = NULL;
+        Event *curr = NULL;
+        QTAILQ_FOREACH(curr, &events_list, events) {
+            if (curr->event_kind == read_event_kind
+                && (read_id == -1 || read_id == curr->id)) {
+                event = curr;
+                break;
+            }
+        }
+
+        if (event) {
+            /* read event-specific reading data */
+
+            QTAILQ_REMOVE(&events_list, event, events);
+
+            qemu_mutex_unlock(&lock);
+
+            /* reset unread data and other parameters to allow
+               reading other data from the log while
+               running the event */
+            replay_has_unread_data = 0;
+            read_event_kind = -1;
+            read_id = -1;
+            read_opt = -1;
+
+            replay_run_event(event);
+            g_free(event);
+
+            replay_fetch_data_kind();
+        } else {
+            qemu_mutex_unlock(&lock);
+            /* No such event found in the queue */
+            break;
+        }
+    }
+}
+
+void replay_init_events(void)
+{
+    read_event_kind = -1;
+    qemu_mutex_init(&lock);
+}
+
+void replay_finish_events(void)
+{
+    replay_events_enabled = false;
+    replay_clear_events();
+    qemu_mutex_destroy(&lock);
 }
replay/replay-internal.h +17 -0
index 019ae20..cffc4e2 100755
@@ -21,9 +21,14 @@
 #define EVENT_EXCEPTION             23
 /* for async events */
 #define EVENT_ASYNC                 24
+#define EVENT_ASYNC_OPT             25
 /* for instruction event */
 #define EVENT_INSTRUCTION           32
 
+/* Asynchronous events IDs */
+
+#define REPLAY_ASYNC_COUNT             0
+
 typedef struct ReplayState {
     /*! Nonzero, when next instruction is repeated one and was already
         processed. */
@@ -75,11 +80,23 @@ void skip_async_events_until(unsigned int kind);
 
 /* Asynchronous events queue */
 
+/*! Initializes events' processing internals */
+void replay_init_events(void);
+/*! Clears internal data structures for events handling */
+void replay_finish_events(void);
+/*! Enables storing events in the queue */
+void replay_enable_events(void);
+/*! Flushes events queue */
+void replay_flush_events(void);
+/*! Clears events list before loading new VM state */
+void replay_clear_events(void);
 /*! Returns true if there are any unsaved events in the queue */
 bool replay_has_events(void);
 /*! Saves events from queue into the file */
 void replay_save_events(int opt);
 /*! Read events from the file into the input queue */
 void replay_read_events(int opt);
+/*! Adds specified async event to the queue */
+void replay_add_event(int event_id, void *opaque);
 
 #endif
replay/replay.h +4 -0
index 10e6681..7f461c5 100755
@@ -58,5 +58,9 @@ bool replay_interrupt(void);
     Returns true, when interrupt request is pending */
 bool replay_has_interrupt(void);
 
+/* Asynchronous events queue */
+
+/*! Disables storing events in the queue */
+void replay_disable_events(void);
 
 #endif
#48 lore raw

RFC PATCH v2 29/49 replay: recording and replaying clock ticks

In reply to Pavel Dovgalyuk

Clock ticks are considered as the sources of non-deterministic data for virtual machine. This patch implements saving the clock values when they are acquired (virtual, host clock, rdtsc, and some other timers). When replaying the execution corresponding values are read from log and transfered to the module, which wants to read the values.
Such a design required the clock polling to be synchronized. Sometimes it is not true - e.g. when timeouts for timer lists are checked. In this case we use a cached value of the clock, passing it to the client code.

---
 cpus.c                   |    9 ++++++
 include/qemu/timer.h     |   42 +++++++++++++++++++++++++-
 replay/Makefile.objs     |    1 +
 replay/replay-internal.h |   11 +++++++
 replay/replay-time.c     |   75 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   17 ++++++++++
 stubs/replay.c           |    9 ++++++
 7 files changed, 162 insertions(+), 2 deletions(-)
 create mode 100755 replay/replay-time.c
cpus.c +9 -0
index eb2a795..15a2106 100644
@@ -40,6 +40,7 @@
 #include "qemu/bitmap.h"
 #include "qemu/seqlock.h"
 #include "qapi-event.h"
+#include "replay/replay.h"
 
 #ifndef _WIN32
 #include "qemu/compatfd.h"
@@ -206,11 +207,19 @@ int64_t cpu_get_clock(void)
     int64_t ti;
     unsigned start;
 
+    if (replay_mode == REPLAY_PLAY) {
+        return replay_read_clock(REPLAY_CLOCK_VIRTUAL);
+    }
+
     do {
         start = seqlock_read_begin(&timers_state.vm_clock_seqlock);
         ti = cpu_get_clock_locked();
     } while (seqlock_read_retry(&timers_state.vm_clock_seqlock, start));
 
+    if (replay_mode == REPLAY_SAVE) {
+        replay_save_clock(REPLAY_CLOCK_VIRTUAL, ti);
+    }
+
     return ti;
 }
 
include/qemu/timer.h +40 -2

72 lines not on this page · raw message

replay/Makefile.objs +1 -0
index 56da09c..257c320 100755
@@ -1,3 +1,4 @@
 obj-y += replay.o
 obj-y += replay-internal.o
 obj-y += replay-events.o
+obj-y += replay-time.o
replay/replay-internal.h +11 -0
index cffc4e2..a4fbc26 100755
@@ -24,12 +24,17 @@
 #define EVENT_ASYNC_OPT             25
 /* for instruction event */
 #define EVENT_INSTRUCTION           32
+/* for clock read/writes */
+#define EVENT_CLOCK                 64
+/* some of grteater codes are reserved for clocks */
 
 /* Asynchronous events IDs */
 
 #define REPLAY_ASYNC_COUNT             0
 
 typedef struct ReplayState {
+    /*! Cached clock values. */
+    int64_t cached_clock[REPLAY_CLOCK_COUNT];
     /*! Nonzero, when next instruction is repeated one and was already
         processed. */
     int skipping_instruction;
@@ -78,6 +83,12 @@ bool skip_async_events(int stop_event);
     reports an error and stops the execution. */
 void skip_async_events_until(unsigned int kind);
 
+/*! Reads next clock value from the file.
+    If clock kind read from the file is different from the parameter,
+    the value is not used.
+    If the parameter is -1, the clock value is read to the cache anyway. */
+void replay_read_next_clock(unsigned int kind);
+
 /* Asynchronous events queue */
 
 /*! Initializes events' processing internals */
replay/replay-time.c new file +75 -0

75 lines not on this page · raw message

replay/replay.h +17 -0
index 7f461c5..3d09ec8 100755
@@ -24,6 +24,16 @@
 /* Normal log replaying */
 #define REPLAY_PLAY_NORMAL  1
 
+/* replay clock kinds */
+/* rdtsc */
+#define REPLAY_CLOCK_REAL_TICKS 0
+/* host_clock */
+#define REPLAY_CLOCK_REALTIME   1
+/* vm_clock */
+#define REPLAY_CLOCK_VIRTUAL    2
+
+#define REPLAY_CLOCK_COUNT      3
+
 extern int replay_mode;
 extern char *replay_image_suffix;
 /*! Shift value for icount based on replay or zero, if it is disabled. */
@@ -58,6 +68,13 @@ bool replay_interrupt(void);
     Returns true, when interrupt request is pending */
 bool replay_has_interrupt(void);
 
+/* Processing clocks and other time sources */
+
+/*! Save the specified clock */
+void replay_save_clock(unsigned int kind, int64_t clock);
+/*! Read the specified clock from the log or return cached data */
+int64_t replay_read_clock(unsigned int kind);
+
 /* Asynchronous events queue */
 
 /*! Disables storing events in the queue */
stubs/replay.c +9 -0
index 1e99ca4..3b88b1a 100755
@@ -7,3 +7,12 @@ int replay_get_play_submode(void)
 {
     return 0;
 }
+
+void replay_save_clock(unsigned int kind, int64_t clock)
+{
+}
+
+int64_t replay_read_clock(unsigned int kind)
+{
+    return 0;
+}
#49 lore raw

RFC PATCH v2 30/49 replay: recording and replaying different timers

In reply to Pavel Dovgalyuk

This patch introduces functions for recording and replaying realtime sources, that do not use qemu-clock interface. These include return value of time() function in time_t and struct tm forms. Patch also adds warning to get_timedate function to prevent its usage in recording mode, because it may lead to non-determinism.

---
 hw/timer/mc146818rtc.c   |    2 -
 hw/timer/pl031.c         |   10 ++++
 include/qemu-common.h    |    1 
 replay/replay-internal.h |    7 +++
 replay/replay-time.c     |  106 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |    8 +++
 vl.c                     |   16 ++++++-
 7 files changed, 145 insertions(+), 5 deletions(-)
hw/timer/mc146818rtc.c +1 -1
index c204abb..b7c842f 100644
@@ -703,7 +703,7 @@ static void rtc_set_date_from_host(ISADevice *dev)
     RTCState *s = MC146818_RTC(dev);
     struct tm tm;
 
-    qemu_get_timedate(&tm, 0);
+    qemu_get_timedate_no_warning(&tm, 0);
 
     s->base_rtc = mktimegm(&tm);
     s->last_update = qemu_clock_get_ns(rtc_clock);
hw/timer/pl031.c +9 -1
index f8e5abc..02c814f 100644
@@ -14,6 +14,7 @@
 #include "hw/sysbus.h"
 #include "qemu/timer.h"
 #include "sysemu/sysemu.h"
+#include "replay/replay.h"
 
 //#define DEBUG_PL031
 
@@ -200,7 +201,14 @@ static int pl031_init(SysBusDevice *dev)
     sysbus_init_mmio(dev, &s->iomem);
 
     sysbus_init_irq(dev, &s->irq);
-    qemu_get_timedate(&tm, 0);
+    if (replay_mode == REPLAY_SAVE) {
+        qemu_get_timedate_no_warning(&tm, 0);
+        replay_save_tm(&tm);
+    } else if (replay_mode == REPLAY_PLAY) {
+        replay_read_tm(&tm);
+    } else {
+        qemu_get_timedate_no_warning(&tm, 0);
+    }
     s->tick_offset = mktimegm(&tm) -
         qemu_clock_get_ns(rtc_clock) / get_ticks_per_sec();
 
include/qemu-common.h +1 -0
index ae76197..48013ce 100644
@@ -121,6 +121,7 @@ extern int use_icount;
 int qemu_main(int argc, char **argv, char **envp);
 #endif
 
+void qemu_get_timedate_no_warning(struct tm *tm, int offset);
 void qemu_get_timedate(struct tm *tm, int offset);
 int qemu_timedate_diff(struct tm *tm);
 
replay/replay-internal.h +7 -0
index a4fbc26..79e23ee 100755
@@ -15,10 +15,17 @@
 #include <stdio.h>
 #include "sysemu/sysemu.h"
 
+<<<<<<< current
 /* for software interrupt */
 #define EVENT_INTERRUPT             15
 /* for emulated exceptions */
 #define EVENT_EXCEPTION             23
+=======
+/* for time_t event */
+#define EVENT_TIME_T                1
+/* for tm event */
+#define EVENT_TM                    2
+>>>>>>> patched
 /* for async events */
 #define EVENT_ASYNC                 24
 #define EVENT_ASYNC_OPT             25
replay/replay-time.c +106 -0

109 lines not on this page · raw message

replay/replay.h +8 -0
index 3d09ec8..16e368c 100755
@@ -14,6 +14,7 @@
 
 #include <stdbool.h>
 #include <stdint.h>
+#include <time.h>
 
 /* replay modes */
 #define REPLAY_NONE 0
@@ -74,6 +75,13 @@ bool replay_has_interrupt(void);
 void replay_save_clock(unsigned int kind, int64_t clock);
 /*! Read the specified clock from the log or return cached data */
 int64_t replay_read_clock(unsigned int kind);
+/*! Returns result of time() function execution in normal and record modes.
+    In play mode returns value read from the log. */
+time_t replay_time(void);
+/*! Saves struct tm value to the log */
+void replay_save_tm(struct tm *tm);
+/*! Reads struct tm value from the log. Stops execution in case of error */
+void replay_read_tm(struct tm *tm);
 
 /* Asynchronous events queue */
 
vl.c +13 -3
index 6e084c2..86aba43 100644
@@ -118,6 +118,7 @@ int main(int argc, char **argv)
 #include "qapi/opts-visitor.h"
 #include "qom/object_interfaces.h"
 #include "qapi-event.h"
+#include "replay/replay.h"
 
 #define DEFAULT_RAM_SIZE 128
 
@@ -757,7 +758,7 @@ void vm_start(void)
 
 /***********************************************************/
 /* host time/date access */
-void qemu_get_timedate(struct tm *tm, int offset)
+void qemu_get_timedate_no_warning(struct tm *tm, int offset)
 {
     time_t ti;
 
@@ -774,6 +775,15 @@ void qemu_get_timedate(struct tm *tm, int offset)
     }
 }
 
+/* host time/date access with replay warning */
+void qemu_get_timedate(struct tm *tm, int offset)
+{
+    if (replay_mode == REPLAY_SAVE) {
+        fprintf(stderr, "REPLAY WARNING! qemu_get_timedate function may lead to non-determinism\n");
+    }
+    qemu_get_timedate_no_warning(tm, offset);
+}
+
 int qemu_timedate_diff(struct tm *tm)
 {
     time_t seconds;
@@ -789,7 +799,7 @@ int qemu_timedate_diff(struct tm *tm)
     else
         seconds = mktimegm(tm) + rtc_date_offset;
 
-    return seconds - time(NULL);
+    return seconds - replay_time();
 }
 
 static void configure_rtc_date_offset(const char *startdate, int legacy)
@@ -827,7 +837,7 @@ static void configure_rtc_date_offset(const char *startdate, int legacy)
                             "'2006-06-17T16:01:21' or '2006-06-17'\n");
             exit(1);
         }
-        rtc_date_offset = time(NULL) - rtc_start_date;
+        rtc_date_offset = replay_time() - rtc_start_date;
     }
 }
 
#50 lore raw

RFC PATCH v2 31/49 replay: shutdown event

In reply to Pavel Dovgalyuk

This patch records and replays simulator shutdown event.

---
 include/sysemu/sysemu.h  |    1 +
 replay/replay-internal.h |   13 ++++++-------
 replay/replay.c          |   11 +++++++++++
 replay/replay.h          |    5 +++++
 vl.c                     |    8 +++++++-
 5 files changed, 30 insertions(+), 8 deletions(-)
include/sysemu/sysemu.h +1 -0
index 1ebfef9..9000feb 100644
@@ -57,6 +57,7 @@ void qemu_register_suspend_notifier(Notifier *notifier);
 void qemu_system_wakeup_request(WakeupReason reason);
 void qemu_system_wakeup_enable(WakeupReason reason, bool enabled);
 void qemu_register_wakeup_notifier(Notifier *notifier);
+void qemu_system_shutdown_request_impl(void);
 void qemu_system_shutdown_request(void);
 void qemu_system_powerdown_request(void);
 void qemu_register_powerdown_notifier(Notifier *notifier);
replay/replay-internal.h +6 -7
index 79e23ee..033dbdc 100755
@@ -15,17 +15,16 @@
 #include <stdio.h>
 #include "sysemu/sysemu.h"
 
-<<<<<<< current
-/* for software interrupt */
-#define EVENT_INTERRUPT             15
-/* for emulated exceptions */
-#define EVENT_EXCEPTION             23
-=======
 /* for time_t event */
 #define EVENT_TIME_T                1
 /* for tm event */
 #define EVENT_TM                    2
->>>>>>> patched
+/* for software interrupt */
+#define EVENT_INTERRUPT             15
+/* for shutdown request */
+#define EVENT_SHUTDOWN              20
+/* for emulated exceptions */
+#define EVENT_EXCEPTION             23
 /* for async events */
 #define EVENT_ASYNC                 24
 #define EVENT_ASYNC_OPT             25
replay/replay.c +11 -0
index 41a27a9..2067d6b 100755
@@ -43,6 +43,10 @@ bool skip_async_events(int stop_event)
             res = true;
         }
         switch (replay_data_kind) {
+        case EVENT_SHUTDOWN:
+            replay_has_unread_data = 0;
+            qemu_system_shutdown_request_impl();
+            break;
         case EVENT_INSTRUCTION:
             first_cpu->instructions_count = replay_get_dword();
             return res;
@@ -201,3 +205,10 @@ bool replay_has_interrupt(void)
     }
     return false;
 }
+
+void replay_shutdown_request(void)
+{
+    if (replay_mode == REPLAY_SAVE) {
+        replay_put_event(EVENT_SHUTDOWN);
+    }
+}
replay/replay.h +5 -0
index 16e368c..a84c57c 100755
@@ -83,6 +83,11 @@ void replay_save_tm(struct tm *tm);
 /*! Reads struct tm value from the log. Stops execution in case of error */
 void replay_read_tm(struct tm *tm);
 
+/* Events */
+
+/*! Called when qemu shutdown is requested. */
+void replay_shutdown_request(void);
+
 /* Asynchronous events queue */
 
 /*! Disables storing events in the queue */
vl.c +7 -1
index 86aba43..1a837e6 100644
@@ -1929,13 +1929,19 @@ void qemu_system_killed(int signal, pid_t pid)
     qemu_system_shutdown_request();
 }
 
-void qemu_system_shutdown_request(void)
+void qemu_system_shutdown_request_impl(void)
 {
     trace_qemu_system_shutdown_request();
     shutdown_requested = 1;
     qemu_notify_event();
 }
 
+void qemu_system_shutdown_request(void)
+{
+    replay_shutdown_request();
+    qemu_system_shutdown_request_impl();
+}
+
 static void qemu_system_powerdown(void)
 {
     qapi_event_send_powerdown(&error_abort);
#51 lore raw

RFC PATCH v2 32/49 replay: checkpoints

In reply to Pavel Dovgalyuk

This patch introduces checkpoints that synchronize cpu thread and iothread. When checkpoint is met in the code all asynchronous events from the queue are executed.

---
 block.c                  |   11 +++++++++++
 cpus.c                   |    2 +-
 include/qemu/timer.h     |    6 ++++--
 qemu-timer.c             |   41 +++++++++++++++++++++++++++++++++--------
 replay/replay-internal.h |    3 +++
 replay/replay.c          |   28 ++++++++++++++++++++++++++++
 replay/replay.h          |    6 ++++++
 stubs/replay.c           |   11 +++++++++++
 vl.c                     |    3 ++-
 9 files changed, 99 insertions(+), 12 deletions(-)
block.c +11 -0
index 3ff1052..fb6c849 100644
@@ -1914,6 +1914,11 @@ void bdrv_drain_all(void)
     BlockDriverState *bs;
 
     while (busy) {
+        if (!replay_checkpoint(8)) {
+            /* Do not wait anymore, we stopped at some place in 
+               the middle of execution during replay */
+            return;
+        }
         busy = false;
 
         QTAILQ_FOREACH(bs, &bdrv_states, device_list) {
@@ -1930,6 +1935,12 @@ void bdrv_drain_all(void)
             busy |= bs_busy;
         }
     }
+    if (replay_mode == REPLAY_PLAY) {
+        /* Skip checkpoints from the log */
+        while (replay_checkpoint(8)) {
+            /* Nothing */
+        }
+    }
 }
 
 /* make a BlockDriverState anonymous by removing from bdrv_state and
cpus.c +1 -1
index 15a2106..4f517ee 100644
@@ -363,7 +363,7 @@ void qtest_clock_warp(int64_t dest)
         qemu_icount_bias += warp;
         seqlock_write_unlock(&timers_state.vm_clock_seqlock);
 
-        qemu_clock_run_timers(QEMU_CLOCK_VIRTUAL);
+        qemu_clock_run_timers(QEMU_CLOCK_VIRTUAL, false);
         clock = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
     }
     qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
include/qemu/timer.h +4 -2
index 9eaed56..6b00341 100644
@@ -232,13 +232,14 @@ void qemu_clock_unregister_reset_notifier(QEMUClockType type,
 /**
  * qemu_clock_run_timers:
  * @type: clock on which to operate
+ * @run_all: true, when called from qemu_clock_run_all_timers
  *
  * Run all the timers associated with the default timer list
  * of a clock.
  *
  * Returns: true if any timer ran.
  */
-bool qemu_clock_run_timers(QEMUClockType type);
+bool qemu_clock_run_timers(QEMUClockType type, bool run_all);
 
 /**
  * qemu_clock_run_all_timers:
@@ -329,12 +330,13 @@ QEMUClockType timerlist_get_clock(QEMUTimerList *timer_list);
 /**
  * timerlist_run_timers:
  * @timer_list: the timer list to use
+ * @run_all: true, when called from qemu_clock_run_all_timers
  *
  * Call all expired timers associated with the timer list.
  *
  * Returns: true if any timer expired
  */
-bool timerlist_run_timers(QEMUTimerList *timer_list);
+bool timerlist_run_timers(QEMUTimerList *timer_list, bool run_all);
 
 /**
  * timerlist_notify:
qemu-timer.c +33 -8

80 lines not on this page · raw message

replay/replay-internal.h +3 -0
index 033dbdc..95cbde6 100755
@@ -34,6 +34,9 @@
 #define EVENT_CLOCK                 64
 /* some of grteater codes are reserved for clocks */
 
+/* for checkpoint event */
+#define EVENT_CHECKPOINT            96
+
 /* Asynchronous events IDs */
 
 #define REPLAY_ASYNC_COUNT             0
replay/replay.c +28 -0
index 2067d6b..225d8ec 100755
@@ -212,3 +212,31 @@ void replay_shutdown_request(void)
         replay_put_event(EVENT_SHUTDOWN);
     }
 }
+
+/* Used checkpoints: 2 3 5 6 7 8 9 */
+int replay_checkpoint(unsigned int checkpoint)
+{
+    replay_save_instructions();
+
+    if (replay_file) {
+        if (replay_mode == REPLAY_PLAY) {
+            if (!skip_async_events(EVENT_CHECKPOINT + checkpoint)) {
+                if (replay_data_kind == EVENT_ASYNC_OPT) {
+                    replay_read_events(checkpoint);
+                    replay_fetch_data_kind();
+                    return replay_data_kind != EVENT_ASYNC_OPT;
+                }
+                return 0;
+            }
+            replay_has_unread_data = 0;
+            replay_read_events(checkpoint);
+            replay_fetch_data_kind();
+            return replay_data_kind != EVENT_ASYNC_OPT;
+        } else if (replay_mode == REPLAY_SAVE) {
+            replay_put_event(EVENT_CHECKPOINT + checkpoint);
+            replay_save_events(checkpoint);
+        }
+    }
+
+    return 1;
+}
replay/replay.h +6 -0
index a84c57c..69a20af 100755
@@ -87,6 +87,12 @@ void replay_read_tm(struct tm *tm);
 
 /*! Called when qemu shutdown is requested. */
 void replay_shutdown_request(void);
+/*! Should be called at check points in the execution.
+    These check points are skipped, if they were not met.
+    Saves checkpoint in the SAVE mode and validates in the PLAY mode.
+    Returns 0 in PLAY mode if checkpoint was not found.
+    Returns 1 in all other cases. */
+int replay_checkpoint(unsigned int checkpoint);
 
 /* Asynchronous events queue */
 
stubs/replay.c +11 -0
index 3b88b1a..1b0b78f 100755
@@ -1,4 +1,5 @@
 #include "replay/replay.h"
+#include "sysemu/sysemu.h"
 
 int replay_mode;
 int replay_icount;
@@ -16,3 +17,13 @@ int64_t replay_read_clock(unsigned int kind)
 {
     return 0;
 }
+
+int replay_checkpoint(unsigned int checkpoint)
+{
+    return 0;
+}
+
+int runstate_is_running(void)
+{
+    return 0;
+}
vl.c +2 -1
index 1a837e6..053b2ba 100644
@@ -1984,7 +1984,8 @@ static bool main_loop_should_exit(void)
             return true;
         }
     }
-    if (qemu_reset_requested()) {
+    if (qemu_reset_requested_get() && replay_checkpoint(7)) {
+        qemu_reset_requested();
         pause_all_vcpus();
         cpu_synchronize_all_states();
         qemu_system_reset(VMRESET_REPORT);
#52 lore raw

RFC PATCH v2 33/49 replay: bottom halves

In reply to Pavel Dovgalyuk

This patch introduces bottom half event for replay queue. It saves the events into the queue and process them at the checkpoints and instructions execution.

---
 async.c                  |   45 +++++++++++++++++++++++++++++++++++++++------
 dma-helpers.c            |    4 +++-
 hw/ide/ahci.c            |    4 +++-
 hw/ide/core.c            |    4 +++-
 hw/timer/arm_timer.c     |    2 +-
 hw/usb/hcd-uhci.c        |    2 +-
 include/block/aio.h      |   17 +++++++++++++++++
 include/qemu/main-loop.h |    1 +
 main-loop.c              |    5 +++++
 replay/replay-events.c   |   16 ++++++++++++++++
 replay/replay-internal.h |    3 ++-
 replay/replay.h          |    2 ++
 stubs/replay.c           |    4 ++++
 13 files changed, 97 insertions(+), 12 deletions(-)
async.c +39 -6
index 5b6fe6b..687cdb2 100644
@@ -26,6 +26,7 @@
 #include "block/aio.h"
 #include "block/thread-pool.h"
 #include "qemu/main-loop.h"
+#include "replay/replay.h"
 
 /***********************************************************/
 /* bottom halves (can be seen as timers which expire ASAP) */
@@ -38,24 +39,52 @@ struct QEMUBH {
     bool scheduled;
     bool idle;
     bool deleted;
+    bool replay;
+    uint64_t id;
 };
 
 QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque)
 {
-    QEMUBH *bh;
+    QEMUBH *bh, **last;
     bh = g_malloc0(sizeof(QEMUBH));
     bh->ctx = ctx;
     bh->cb = cb;
     bh->opaque = opaque;
     qemu_mutex_lock(&ctx->bh_lock);
-    bh->next = ctx->first_bh;
-    /* Make sure that the members are ready before putting bh into list */
-    smp_wmb();
-    ctx->first_bh = bh;
+    if (replay_mode != REPLAY_NONE) {
+        /* Slower way, but this is a queue and not a stack.
+           Replay will process the BH in the same order they 
+           came into the queue. */
+        last = &ctx->first_bh;
+        while (*last) {
+            last = &(*last)->next;
+        }
+        smp_wmb();
+        *last = bh;
+    } else {
+        bh->next = ctx->first_bh;
+        /* Make sure that the members are ready before putting bh into list */
+        smp_wmb();
+        ctx->first_bh = bh;
+    }
     qemu_mutex_unlock(&ctx->bh_lock);
     return bh;
 }
 
+QEMUBH *aio_bh_new_replay(AioContext *ctx, QEMUBHFunc *cb, void *opaque, uint64_t id)
+{
+    QEMUBH *bh = aio_bh_new(ctx, cb, opaque);
+    bh->replay = true;
+    bh->id = id;
+    return bh;
+}
+
+void aio_bh_call(void *opaque)
+{
+    QEMUBH *bh = (QEMUBH *)opaque;
+    bh->cb(bh->opaque);
+}
+
 /* Multiple occurrences of aio_bh_poll cannot be called concurrently */
 int aio_bh_poll(AioContext *ctx)
 {
@@ -78,7 +107,11 @@ int aio_bh_poll(AioContext *ctx)
             if (!bh->idle)
                 ret = 1;
             bh->idle = 0;
-            bh->cb(bh->opaque);
+            if (!bh->replay) {
+                aio_bh_call(bh);
+            } else {
+                replay_add_bh_event(bh, bh->id);
+            }
         }
     }
 
dma-helpers.c +3 -1
index 53cbe92..05aadea 100644
@@ -12,6 +12,7 @@
 #include "qemu/range.h"
 #include "qemu/thread.h"
 #include "qemu/main-loop.h"
+#include "replay/replay.h"
 
 /* #define DEBUG_IOMMU */
 
@@ -96,7 +97,8 @@ static void continue_after_map_failure(void *opaque)
 {
     DMAAIOCB *dbs = (DMAAIOCB *)opaque;
 
-    dbs->bh = qemu_bh_new(reschedule_dma, dbs);
+    dbs->bh = qemu_bh_new_replay(reschedule_dma, dbs,
+                                 replay_get_current_step());
     qemu_bh_schedule(dbs->bh);
 }
 
hw/ide/ahci.c +3 -1
index 604152a..0762743 100644
@@ -32,6 +32,7 @@
 #include "internal.h"
 #include <hw/ide/pci.h>
 #include <hw/ide/ahci.h>
+#include "replay/replay.h"
 
 /* #define DEBUG_AHCI */
 
@@ -1132,7 +1133,8 @@ static int ahci_async_cmd_done(IDEDMA *dma)
 
     if (!ad->check_bh) {
         /* maybe we still have something to process, check later */
-        ad->check_bh = qemu_bh_new(ahci_check_cmd_bh, ad);
+        ad->check_bh = qemu_bh_new_replay(ahci_check_cmd_bh, ad,
+                                          replay_get_current_step());
         qemu_bh_schedule(ad->check_bh);
     }
 
hw/ide/core.c +3 -1
index 3a38f1e..3df26f0 100644
@@ -32,6 +32,7 @@
 #include "sysemu/dma.h"
 #include "hw/block/block.h"
 #include "sysemu/blockdev.h"
+#include "replay/replay.h"
 
 #include <hw/ide/internal.h>
 
@@ -409,7 +410,8 @@ BlockDriverAIOCB *ide_issue_trim(BlockDriverState *bs,
     TrimAIOCB *iocb;
 
     iocb = qemu_aio_get(&trim_aiocb_info, bs, cb, opaque);
-    iocb->bh = qemu_bh_new(ide_trim_bh_cb, iocb);
+    iocb->bh = qemu_bh_new_replay(ide_trim_bh_cb, iocb,
+                                  replay_get_current_step());
     iocb->ret = 0;
     iocb->qiov = qiov;
     iocb->i = -1;
hw/timer/arm_timer.c +1 -1
index 1452910..97784a0 100644
@@ -168,7 +168,7 @@ static arm_timer_state *arm_timer_init(uint32_t freq)
     s->freq = freq;
     s->control = TIMER_CTRL_IE;
 
-    bh = qemu_bh_new(arm_timer_tick, s);
+    bh = qemu_bh_new_replay(arm_timer_tick, s, 0);
     s->timer = ptimer_init(bh);
     vmstate_register(NULL, -1, &vmstate_arm_timer, s);
     return s;
hw/usb/hcd-uhci.c +1 -1
index c3bf72c..3aaf506 100644
@@ -1224,7 +1224,7 @@ static int usb_uhci_common_initfn(PCIDevice *dev)
                               USB_SPEED_MASK_LOW | USB_SPEED_MASK_FULL);
         }
     }
-    s->bh = qemu_bh_new(uhci_bh, s);
+    s->bh = qemu_bh_new_replay(uhci_bh, s, 0);
     s->frame_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, uhci_frame_timer, s);
     s->num_ports_vmstate = NB_PORTS;
     QTAILQ_INIT(&s->queues);
include/block/aio.h +17 -0
index a92511b..3268ffa 100644
@@ -34,6 +34,8 @@ struct BlockDriverAIOCB {
     const AIOCBInfo *aiocb_info;
     BlockDriverState *bs;
     BlockDriverCompletionFunc *cb;
+    bool replay;
+    uint64_t replay_step;
     void *opaque;
 };
 
@@ -132,6 +134,16 @@ void aio_context_release(AioContext *ctx);
 QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque);
 
 /**
+ * aio_bh_new_replay: Allocate a new bottom half structure for replay.
+ *
+ * This function calls aio_bh_new function and also fills replay parameters
+ * of the BH structure. BH created with this function in record/replay mode
+ * are executed through the replay queue only at checkpoints and instructions
+ * executions.
+ */
+QEMUBH *aio_bh_new_replay(AioContext *ctx, QEMUBHFunc *cb, void *opaque, uint64_t id);
+
+/**
  * aio_notify: Force processing of pending events.
  *
  * Similar to signaling a condition variable, aio_notify forces
@@ -147,6 +159,11 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque);
 void aio_notify(AioContext *ctx);
 
 /**
+ * aio_bh_call: Executes callback function of the specified BH.
+ */
+void aio_bh_call(void *opaque);
+
+/**
  * aio_bh_poll: Poll bottom halves for an AioContext.
  *
  * These are internal functions used by the QEMU main loop.
include/qemu/main-loop.h +1 -0
index 6f0200a..c221ddd 100644
@@ -306,6 +306,7 @@ void qemu_iohandler_fill(GArray *pollfds);
 void qemu_iohandler_poll(GArray *pollfds, int rc);
 
 QEMUBH *qemu_bh_new(QEMUBHFunc *cb, void *opaque);
+QEMUBH *qemu_bh_new_replay(QEMUBHFunc *cb, void *opaque, uint64_t id);
 void qemu_bh_schedule_idle(QEMUBH *bh);
 
 #endif
main-loop.c +5 -0
index 8a85493..4e1af51 100644
@@ -499,6 +499,11 @@ QEMUBH *qemu_bh_new(QEMUBHFunc *cb, void *opaque)
     return aio_bh_new(qemu_aio_context, cb, opaque);
 }
 
+QEMUBH *qemu_bh_new_replay(QEMUBHFunc *cb, void *opaque, uint64_t id)
+{
+    return aio_bh_new_replay(qemu_aio_context, cb, opaque, id);
+}
+
 bool qemu_aio_wait(void)
 {
     return aio_poll(qemu_aio_context, true);
replay/replay-events.c +16 -0
index eda574e..81a5a6b 100755
@@ -35,6 +35,9 @@ static bool replay_events_enabled = false;
 static void replay_run_event(Event *event)
 {
     switch (event->event_kind) {
+    case REPLAY_ASYNC_EVENT_BH:
+        aio_bh_call(event->opaque);
+        break;
     default:
         fprintf(stderr, "Replay: invalid async event ID (%d) in the queue\n", 
                 event->event_kind);
@@ -117,6 +120,11 @@ void replay_add_event(int event_kind, void *opaque)
     replay_add_event_internal(event_kind, opaque, NULL, 0);
 }
 
+void replay_add_bh_event(void *bh, uint64_t id)
+{
+    replay_add_event_internal(REPLAY_ASYNC_EVENT_BH, bh, NULL, id);
+}
+
 void replay_save_events(int opt)
 {
     qemu_mutex_lock(&lock);
@@ -134,6 +142,9 @@ void replay_save_events(int opt)
 
             /* save event-specific data */
             switch (event->event_kind) {
+            case REPLAY_ASYNC_EVENT_BH:
+                replay_put_qword(event->id);
+                break;
             }
         }
 
@@ -163,6 +174,11 @@ void replay_read_events(int opt)
             break;
         /* Execute some events without searching them in the queue */
         switch (read_event_kind) {
+        case REPLAY_ASYNC_EVENT_BH:
+            if (read_id == -1) {
+                read_id = replay_get_qword();
+            }
+            break;
         default:
             fprintf(stderr, "Unknown ID %d of replay event\n", read_event_kind);
             exit(1);
replay/replay-internal.h +2 -1
index 95cbde6..bbb117e 100755
@@ -39,7 +39,8 @@
 
 /* Asynchronous events IDs */
 
-#define REPLAY_ASYNC_COUNT             0
+#define REPLAY_ASYNC_EVENT_BH          0
+#define REPLAY_ASYNC_COUNT             1
 
 typedef struct ReplayState {
     /*! Cached clock values. */
replay/replay.h +2 -0
index 69a20af..3040b6b 100755
@@ -98,5 +98,7 @@ int replay_checkpoint(unsigned int checkpoint);
 
 /*! Disables storing events in the queue */
 void replay_disable_events(void);
+/*! Adds BH event to the queue */
+void replay_add_bh_event(void *bh, uint64_t id);
 
 #endif
stubs/replay.c +4 -0
index 1b0b78f..8a4d69e 100755
@@ -27,3 +27,7 @@ int runstate_is_running(void)
 {
     return 0;
 }
+
+void replay_add_bh_event(void *bh, uint64_t id)
+{
+}
#53 lore raw

RFC PATCH v2 34/49 replay: replay aio requests

In reply to Pavel Dovgalyuk

This patch adds identifier to aio requests. ID is used for creating bottom halves and identifying them while replaying.
The patch also introduces several functions that make possible replaying of the aio requests.

---
 block.c               |   79 +++++++++++++++++++++++++++++++++++++++++++------
 block/qcow2.c         |    4 ++
 dma-helpers.c         |    6 ++--
 hw/block/virtio-blk.c |   10 +++---
 hw/ide/atapi.c        |    9 ++++--
 hw/ide/core.c         |   14 ++++++---
 include/block/block.h |   15 +++++++++
 include/qemu-common.h |    2 +
 qemu-io-cmds.c        |    2 +
 stubs/replay.c        |    5 +++
 trace-events          |    2 +
 util/iov.c            |    4 ++
 12 files changed, 126 insertions(+), 26 deletions(-)
block.c +70 -9
index fb6c849..849b75a 100644
@@ -84,7 +84,8 @@ static BlockDriverAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
                                                BdrvRequestFlags flags,
                                                BlockDriverCompletionFunc *cb,
                                                void *opaque,
-                                               bool is_write);
+                                               bool is_write,
+                                               bool aio_replay);
 static void coroutine_fn bdrv_co_do_rw(void *opaque);
 static int coroutine_fn bdrv_co_do_write_zeroes(BlockDriverState *bs,
     int64_t sector_num, int nb_sectors, BdrvRequestFlags flags);
@@ -4370,7 +4371,19 @@ BlockDriverAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,
     trace_bdrv_aio_readv(bs, sector_num, nb_sectors, opaque);
 
     return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
-                                 cb, opaque, false);
+                                 cb, opaque, false, false);
+}
+
+BlockDriverAIOCB *bdrv_aio_readv_replay(BlockDriverState *bs,
+                                        int64_t sector_num,
+                                        QEMUIOVector *qiov, int nb_sectors,
+                                        BlockDriverCompletionFunc *cb,
+                                        void *opaque)
+{
+    trace_bdrv_aio_readv_replay(bs, sector_num, nb_sectors, opaque);
+
+    return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
+                                 cb, opaque, false, true);
 }
 
 BlockDriverAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
@@ -4380,7 +4393,19 @@ BlockDriverAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
     trace_bdrv_aio_writev(bs, sector_num, nb_sectors, opaque);
 
     return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
-                                 cb, opaque, true);
+                                 cb, opaque, true, false);
+}
+
+BlockDriverAIOCB *bdrv_aio_writev_replay(BlockDriverState *bs,
+                                         int64_t sector_num,
+                                         QEMUIOVector *qiov, int nb_sectors,
+                                         BlockDriverCompletionFunc *cb,
+                                         void *opaque)
+{
+    trace_bdrv_aio_writev_replay(bs, sector_num, nb_sectors, opaque);
+
+    return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
+                                 cb, opaque, true, false);
 }
 
 BlockDriverAIOCB *bdrv_aio_write_zeroes(BlockDriverState *bs,
@@ -4391,7 +4416,7 @@ BlockDriverAIOCB *bdrv_aio_write_zeroes(BlockDriverState *bs,
 
     return bdrv_co_aio_rw_vector(bs, sector_num, NULL, nb_sectors,
                                  BDRV_REQ_ZERO_WRITE | flags,
-                                 cb, opaque, true);
+                                 cb, opaque, true, true);
 }
 
 
@@ -4527,7 +4552,8 @@ static int multiwrite_merge(BlockDriverState *bs, BlockRequest *reqs,
  * requests. However, the fields opaque and error are left unmodified as they
  * are used to signal failure for a single request to the caller.
  */
-int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs)
+int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs,
+                        bool replay)
 {
     MultiwriteCB *mcb;
     int i;
@@ -4565,7 +4591,7 @@ int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs)
         bdrv_co_aio_rw_vector(bs, reqs[i].sector, reqs[i].qiov,
                               reqs[i].nb_sectors, reqs[i].flags,
                               multiwrite_cb, mcb,
-                              true);
+                              true, replay);
     }
 
     return 0;
@@ -4714,7 +4740,11 @@ static void coroutine_fn bdrv_co_do_rw(void *opaque)
             acb->req.nb_sectors, acb->req.qiov, acb->req.flags);
     }
 
-    acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    if (acb->common.replay) {
+        acb->bh = aio_bh_new_replay(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb, acb->common.replay_step);
+    } else {
+        acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    }
     qemu_bh_schedule(acb->bh);
 }
 
@@ -4725,7 +4755,8 @@ static BlockDriverAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
                                                BdrvRequestFlags flags,
                                                BlockDriverCompletionFunc *cb,
                                                void *opaque,
-                                               bool is_write)
+                                               bool is_write,
+                                               bool aio_replay)
 {
     Coroutine *co;
     BlockDriverAIOCBCoroutine *acb;
@@ -4737,6 +4768,10 @@ static BlockDriverAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
     acb->req.flags = flags;
     acb->is_write = is_write;
     acb->done = NULL;
+    acb->common.replay = aio_replay;
+    if (aio_replay) {
+        acb->common.replay_step = replay_get_current_step();
+    }
 
     co = qemu_coroutine_create(bdrv_co_do_rw);
     qemu_coroutine_enter(co, acb);
@@ -4750,7 +4785,12 @@ static void coroutine_fn bdrv_aio_flush_co_entry(void *opaque)
     BlockDriverState *bs = acb->common.bs;
 
     acb->req.error = bdrv_co_flush(bs);
-    acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    if (acb->common.replay) {
+        acb->bh = aio_bh_new_replay(bdrv_get_aio_context(bs), bdrv_co_em_bh,
+                                    acb, acb->common.replay_step);
+    } else {
+        acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    }
     qemu_bh_schedule(acb->bh);
 }
 
@@ -4771,6 +4811,25 @@ BlockDriverAIOCB *bdrv_aio_flush(BlockDriverState *bs,
     return &acb->common;
 }
 
+BlockDriverAIOCB *bdrv_aio_flush_replay(BlockDriverState *bs,
+        BlockDriverCompletionFunc *cb, void *opaque)
+{
+    trace_bdrv_aio_flush(bs, opaque);
+
+    Coroutine *co;
+    BlockDriverAIOCBCoroutine *acb;
+
+    acb = qemu_aio_get(&bdrv_em_co_aiocb_info, bs, cb, opaque);
+    acb->done = NULL;
+    acb->common.replay = true;
+    acb->common.replay_step = replay_get_current_step();
+
+    co = qemu_coroutine_create(bdrv_aio_flush_co_entry);
+    qemu_coroutine_enter(co, acb);
+
+    return &acb->common;
+}
+
 static void coroutine_fn bdrv_aio_discard_co_entry(void *opaque)
 {
     BlockDriverAIOCBCoroutine *acb = opaque;
@@ -4821,6 +4880,8 @@ void *qemu_aio_get(const AIOCBInfo *aiocb_info, BlockDriverState *bs,
     acb->bs = bs;
     acb->cb = cb;
     acb->opaque = opaque;
+    acb->replay_step = 0;
+    acb->replay = false;
     return acb;
 }
 
block/qcow2.c +4 -0
index d9da238..358a144 100644
@@ -988,6 +988,8 @@ static coroutine_fn int qcow2_co_readv(BlockDriverState *bs, int64_t sector_num,
     uint8_t *cluster_data = NULL;
 
     qemu_iovec_init(&hd_qiov, qiov->niov);
+    hd_qiov.replay = qiov->replay;
+    hd_qiov.replay_step = qiov->replay_step;
 
     qemu_co_mutex_lock(&s->lock);
 
@@ -1131,6 +1133,8 @@ static coroutine_fn int qcow2_co_writev(BlockDriverState *bs,
                                  remaining_sectors);
 
     qemu_iovec_init(&hd_qiov, qiov->niov);
+    hd_qiov.replay = qiov->replay;
+    hd_qiov.replay_step = qiov->replay_step;
 
     s->cluster_cache_offset = -1; /* disable compressed cache */
 
dma-helpers.c +4 -2
index 05aadea..1606a19 100644
@@ -219,6 +219,8 @@ BlockDriverAIOCB *dma_bdrv_io(
     dbs->io_func = io_func;
     dbs->bh = NULL;
     qemu_iovec_init(&dbs->iov, sg->nsg);
+    dbs->iov.replay = true;
+    dbs->iov.replay_step = replay_get_current_step();
     dma_bdrv_cb(dbs, 0);
     return &dbs->common;
 }
@@ -228,7 +230,7 @@ BlockDriverAIOCB *dma_bdrv_read(BlockDriverState *bs,
                                 QEMUSGList *sg, uint64_t sector,
                                 void (*cb)(void *opaque, int ret), void *opaque)
 {
-    return dma_bdrv_io(bs, sg, sector, bdrv_aio_readv, cb, opaque,
+    return dma_bdrv_io(bs, sg, sector, bdrv_aio_readv_replay, cb, opaque,
                        DMA_DIRECTION_FROM_DEVICE);
 }
 
@@ -236,7 +238,7 @@ BlockDriverAIOCB *dma_bdrv_write(BlockDriverState *bs,
                                  QEMUSGList *sg, uint64_t sector,
                                  void (*cb)(void *opaque, int ret), void *opaque)
 {
-    return dma_bdrv_io(bs, sg, sector, bdrv_aio_writev, cb, opaque,
+    return dma_bdrv_io(bs, sg, sector, bdrv_aio_writev_replay, cb, opaque,
                        DMA_DIRECTION_TO_DEVICE);
 }
 
hw/block/virtio-blk.c +5 -5
index aec3146..adf8dd2 100644
@@ -265,7 +265,7 @@ void virtio_submit_multiwrite(BlockDriverState *bs, MultiReqBuffer *mrb)
         return;
     }
 
-    ret = bdrv_aio_multiwrite(bs, mrb->blkreq, mrb->num_writes);
+    ret = bdrv_aio_multiwrite(bs, mrb->blkreq, mrb->num_writes, true);
     if (ret != 0) {
         for (i = 0; i < mrb->num_writes; i++) {
             if (mrb->blkreq[i].error) {
@@ -285,7 +285,7 @@ static void virtio_blk_handle_flush(VirtIOBlockReq *req, MultiReqBuffer *mrb)
      * Make sure all outstanding writes are posted to the backing device.
      */
     virtio_submit_multiwrite(req->dev->bs, mrb);
-    bdrv_aio_flush(req->dev->bs, virtio_blk_flush_complete, req);
+    bdrv_aio_flush_replay(req->dev->bs, virtio_blk_flush_complete, req);
 }
 
 static void virtio_blk_handle_write(VirtIOBlockReq *req, MultiReqBuffer *mrb)
@@ -341,9 +341,9 @@ static void virtio_blk_handle_read(VirtIOBlockReq *req)
         virtio_blk_rw_complete(req, -EIO);
         return;
     }
-    bdrv_aio_readv(req->dev->bs, sector, &req->qiov,
-                   req->qiov.size / BDRV_SECTOR_SIZE,
-                   virtio_blk_rw_complete, req);
+    bdrv_aio_readv_replay(req->dev->bs, sector, &req->qiov,
+                          req->qiov.size / BDRV_SECTOR_SIZE,
+                          virtio_blk_rw_complete, req);
 }
 
 void virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb)
hw/ide/atapi.c +6 -3
index f7d2009..e94cd93 100644
@@ -25,6 +25,7 @@
 
 #include "hw/ide/internal.h"
 #include "hw/scsi/scsi.h"
+#include "replay/replay.h"
 
 static void ide_atapi_cmd_read_dma_cb(void *opaque, int ret);
 
@@ -348,10 +349,12 @@ static void ide_atapi_cmd_read_dma_cb(void *opaque, int ret)
     s->bus->dma->iov.iov_base = (void *)(s->io_buffer + data_offset);
     s->bus->dma->iov.iov_len = n * 4 * 512;
     qemu_iovec_init_external(&s->bus->dma->qiov, &s->bus->dma->iov, 1);
+    s->bus->dma->qiov.replay = true;
+    s->bus->dma->qiov.replay_step = replay_get_current_step();
 
-    s->bus->dma->aiocb = bdrv_aio_readv(s->bs, (int64_t)s->lba << 2,
-                                       &s->bus->dma->qiov, n * 4,
-                                       ide_atapi_cmd_read_dma_cb, s);
+    s->bus->dma->aiocb = bdrv_aio_readv_replay(s->bs, (int64_t)s->lba << 2,
+                                               &s->bus->dma->qiov, n * 4,
+                                               ide_atapi_cmd_read_dma_cb, s);
     return;
 
 eot:
hw/ide/core.c +9 -5
index 3df26f0..8c3cfe2 100644
@@ -559,10 +559,12 @@ void ide_sector_read(IDEState *s)
     s->iov.iov_base = s->io_buffer;
     s->iov.iov_len  = n * BDRV_SECTOR_SIZE;
     qemu_iovec_init_external(&s->qiov, &s->iov, 1);
+    s->qiov.replay = true;
+    s->qiov.replay_step = replay_get_current_step();
 
     bdrv_acct_start(s->bs, &s->acct, n * BDRV_SECTOR_SIZE, BDRV_ACCT_READ);
-    s->pio_aiocb = bdrv_aio_readv(s->bs, sector_num, &s->qiov, n,
-                                  ide_sector_read_cb, s);
+    s->pio_aiocb = bdrv_aio_readv_replay(s->bs, sector_num, &s->qiov, n,
+                                         ide_sector_read_cb, s);
 }
 
 static void dma_buf_commit(IDEState *s)
@@ -795,10 +797,12 @@ void ide_sector_write(IDEState *s)
     s->iov.iov_base = s->io_buffer;
     s->iov.iov_len  = n * BDRV_SECTOR_SIZE;
     qemu_iovec_init_external(&s->qiov, &s->iov, 1);
+    s->qiov.replay = true;
+    s->qiov.replay_step = replay_get_current_step();
 
     bdrv_acct_start(s->bs, &s->acct, n * BDRV_SECTOR_SIZE, BDRV_ACCT_READ);
-    s->pio_aiocb = bdrv_aio_writev(s->bs, sector_num, &s->qiov, n,
-                                   ide_sector_write_cb, s);
+    s->pio_aiocb = bdrv_aio_writev_replay(s->bs, sector_num, &s->qiov, n,
+                                          ide_sector_write_cb, s);
 }
 
 static void ide_flush_cb(void *opaque, int ret)
@@ -827,7 +831,7 @@ void ide_flush_cache(IDEState *s)
 
     s->status |= BUSY_STAT;
     bdrv_acct_start(s->bs, &s->acct, 0, BDRV_ACCT_FLUSH);
-    bdrv_aio_flush(s->bs, ide_flush_cb, s);
+    bdrv_aio_flush_replay(s->bs, ide_flush_cb, s);
 }
 
 static void ide_cfata_metadata_inquiry(IDEState *s)
include/block/block.h +14 -1
index dd05458..636589a 100644
@@ -325,11 +325,24 @@ typedef void BlockDriverDirtyHandler(BlockDriverState *bs, int64_t sector,
 BlockDriverAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,
                                  QEMUIOVector *iov, int nb_sectors,
                                  BlockDriverCompletionFunc *cb, void *opaque);
+BlockDriverAIOCB *bdrv_aio_readv_replay(BlockDriverState *bs,
+                                        int64_t sector_num,
+                                        QEMUIOVector *iov, int nb_sectors,
+                                        BlockDriverCompletionFunc *cb,
+                                        void *opaque);
 BlockDriverAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
                                   QEMUIOVector *iov, int nb_sectors,
                                   BlockDriverCompletionFunc *cb, void *opaque);
+BlockDriverAIOCB *bdrv_aio_writev_replay(BlockDriverState *bs,
+                                         int64_t sector_num,
+                                         QEMUIOVector *iov, int nb_sectors,
+                                         BlockDriverCompletionFunc *cb,
+                                         void *opaque);
 BlockDriverAIOCB *bdrv_aio_flush(BlockDriverState *bs,
                                  BlockDriverCompletionFunc *cb, void *opaque);
+BlockDriverAIOCB *bdrv_aio_flush_replay(BlockDriverState *bs,
+                                        BlockDriverCompletionFunc *cb,
+                                        void *opaque);
 BlockDriverAIOCB *bdrv_aio_discard(BlockDriverState *bs,
                                    int64_t sector_num, int nb_sectors,
                                    BlockDriverCompletionFunc *cb, void *opaque);
@@ -349,7 +362,7 @@ typedef struct BlockRequest {
 } BlockRequest;
 
 int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs,
-    int num_reqs);
+                        int num_reqs, bool replay);
 
 /* sg packet commands */
 int bdrv_ioctl(BlockDriverState *bs, unsigned long int req, void *buf);
include/qemu-common.h +2 -0
index 48013ce..50ab79c 100644
@@ -309,6 +309,8 @@ typedef struct QEMUIOVector {
     int niov;
     int nalloc;
     size_t size;
+    bool replay;
+    uint64_t replay_step;
 } QEMUIOVector;
 
 void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint);
qemu-io-cmds.c +1 -1
index 60c1ceb..f8a809a 100644
@@ -592,7 +592,7 @@ static int do_aio_multiwrite(BlockDriverState *bs, BlockRequest* reqs,
         *total += reqs[i].qiov->size;
     }
 
-    ret = bdrv_aio_multiwrite(bs, reqs, num_reqs);
+    ret = bdrv_aio_multiwrite(bs, reqs, num_reqs, false);
     if (ret < 0) {
         return ret;
     }
stubs/replay.c +5 -0
index 8a4d69e..1dea1f6 100755
@@ -31,3 +31,8 @@ int runstate_is_running(void)
 void replay_add_bh_event(void *bh, uint64_t id)
 {
 }
+
+uint64_t replay_get_current_step(void)
+{
+    return 0;
+}
trace-events +2 -0
index 11a17a8..8382348 100644
@@ -59,7 +59,9 @@ bdrv_aio_multiwrite(void *mcb, int num_callbacks, int num_reqs) "mcb %p num_call
 bdrv_aio_discard(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_flush(void *bs, void *opaque) "bs %p opaque %p"
 bdrv_aio_readv(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
+bdrv_aio_readv_replay(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_writev(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
+bdrv_aio_writev_replay(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_write_zeroes(void *bs, int64_t sector_num, int nb_sectors, int flags, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d flags %#x opaque %p"
 bdrv_lock_medium(void *bs, bool locked) "bs %p locked %d"
 bdrv_co_readv(void *bs, int64_t sector_num, int nb_sector) "bs %p sector_num %"PRId64" nb_sectors %d"
util/iov.c +4 -0
index 2b4f46d..3bf7092 100644
@@ -257,6 +257,8 @@ void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint)
     qiov->niov = 0;
     qiov->nalloc = alloc_hint;
     qiov->size = 0;
+    qiov->replay = false;
+    qiov->replay_step = 0;
 }
 
 void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov)
@@ -267,6 +269,8 @@ void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov)
     qiov->niov = niov;
     qiov->nalloc = -1;
     qiov->size = 0;
+    qiov->replay = false;
+    qiov->replay_step = 0;
     for (i = 0; i < niov; i++)
         qiov->size += iov[i].iov_len;
 }
#54 lore raw

RFC PATCH v2 35/49 replay: thread pool

In reply to Pavel Dovgalyuk

This patch modifies thread pool to allow replaying asynchronous thread tasks synchronously in replay mode.

---
 block/raw-posix.c           |    6 +++--
 block/raw-win32.c           |    4 ++-
 include/block/thread-pool.h |    4 ++-
 replay/replay-events.c      |   11 +++++++++
 replay/replay-internal.h    |    3 ++
 replay/replay.h             |    2 ++
 stubs/replay.c              |    4 +++
 tests/test-thread-pool.c    |    7 +++---
 thread-pool.c               |   53 +++++++++++++++++++++++++++++--------------
 9 files changed, 69 insertions(+), 25 deletions(-)
block/raw-posix.c +4 -2
index a857def..f600736 100644
@@ -1025,7 +1025,9 @@ static BlockDriverAIOCB *paio_submit(BlockDriverState *bs, int fd,
 
     trace_paio_submit(acb, opaque, sector_num, nb_sectors, type);
     pool = aio_get_thread_pool(bdrv_get_aio_context(bs));
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque,
+                                  qiov ? qiov->replay : false,
+                                  qiov ? qiov->replay_step : 0);
 }
 
 static BlockDriverAIOCB *raw_aio_submit(BlockDriverState *bs,
@@ -1853,7 +1855,7 @@ static BlockDriverAIOCB *hdev_aio_ioctl(BlockDriverState *bs,
     acb->aio_ioctl_buf = buf;
     acb->aio_ioctl_cmd = req;
     pool = aio_get_thread_pool(bdrv_get_aio_context(bs));
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque, false, 0);
 }
 
 #elif defined(__FreeBSD__) || defined(__FreeBSD_kernel__)
block/raw-win32.c +3 -1
index 902eab6..212307c 100644
@@ -158,7 +158,9 @@ static BlockDriverAIOCB *paio_submit(BlockDriverState *bs, HANDLE hfile,
 
     trace_paio_submit(acb, opaque, sector_num, nb_sectors, type);
     pool = aio_get_thread_pool(bdrv_get_aio_context(bs));
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque,
+                                  qiov ? qiov->replay : false,
+                                  qiov ? qiov->replay_step : 0);
 }
 
 int qemu_ftruncate64(int fd, int64_t length)
include/block/thread-pool.h +3 -1
index 32afcdd..df74f8d 100644
@@ -33,9 +33,11 @@ void thread_pool_free(ThreadPool *pool);
 
 BlockDriverAIOCB *thread_pool_submit_aio(ThreadPool *pool,
         ThreadPoolFunc *func, void *arg,
-        BlockDriverCompletionFunc *cb, void *opaque);
+        BlockDriverCompletionFunc *cb, void *opaque,
+        bool replay, uint64_t replay_step);
 int coroutine_fn thread_pool_submit_co(ThreadPool *pool,
         ThreadPoolFunc *func, void *arg);
 void thread_pool_submit(ThreadPool *pool, ThreadPoolFunc *func, void *arg);
+void thread_pool_work(ThreadPool *pool, void *r);
 
 #endif
replay/replay-events.c +11 -0
index 81a5a6b..f39889d 100755
@@ -11,6 +11,7 @@
 
 #include "replay.h"
 #include "replay-internal.h"
+#include "block/thread-pool.h"
 
 typedef struct Event {
     int event_kind;
@@ -38,6 +39,9 @@ static void replay_run_event(Event *event)
     case REPLAY_ASYNC_EVENT_BH:
         aio_bh_call(event->opaque);
         break;
+    case REPLAY_ASYNC_EVENT_THREAD:
+        thread_pool_work((ThreadPool *)event->opaque, event->opaque2);
+        break;
     default:
         fprintf(stderr, "Replay: invalid async event ID (%d) in the queue\n", 
                 event->event_kind);
@@ -125,6 +129,11 @@ void replay_add_bh_event(void *bh, uint64_t id)
     replay_add_event_internal(REPLAY_ASYNC_EVENT_BH, bh, NULL, id);
 }
 
+void replay_add_thread_event(void *opaque, void *opaque2, uint64_t id)
+{
+    replay_add_event_internal(REPLAY_ASYNC_EVENT_THREAD, opaque, opaque2, id);
+}
+
 void replay_save_events(int opt)
 {
     qemu_mutex_lock(&lock);
@@ -143,6 +152,7 @@ void replay_save_events(int opt)
             /* save event-specific data */
             switch (event->event_kind) {
             case REPLAY_ASYNC_EVENT_BH:
+            case REPLAY_ASYNC_EVENT_THREAD:
                 replay_put_qword(event->id);
                 break;
             }
@@ -175,6 +185,7 @@ void replay_read_events(int opt)
         /* Execute some events without searching them in the queue */
         switch (read_event_kind) {
         case REPLAY_ASYNC_EVENT_BH:
+        case REPLAY_ASYNC_EVENT_THREAD:
             if (read_id == -1) {
                 read_id = replay_get_qword();
             }
replay/replay-internal.h +2 -1
index bbb117e..3f97fd7 100755
@@ -40,7 +40,8 @@
 /* Asynchronous events IDs */
 
 #define REPLAY_ASYNC_EVENT_BH          0
-#define REPLAY_ASYNC_COUNT             1
+#define REPLAY_ASYNC_EVENT_THREAD      1
+#define REPLAY_ASYNC_COUNT             2
 
 typedef struct ReplayState {
     /*! Cached clock values. */
replay/replay.h +2 -0
index 3040b6b..e96c74f 100755
@@ -100,5 +100,7 @@ int replay_checkpoint(unsigned int checkpoint);
 void replay_disable_events(void);
 /*! Adds BH event to the queue */
 void replay_add_bh_event(void *bh, uint64_t id);
+/*! Adds thread event to the queue */
+void replay_add_thread_event(void *pool, void *req, uint64_t id);
 
 #endif
stubs/replay.c +4 -0
index 1dea1f6..c269b59 100755
@@ -36,3 +36,7 @@ uint64_t replay_get_current_step(void)
 {
     return 0;
 }
+
+void replay_add_thread_event(void *opaque, void *opaque2, uint64_t id)
+{
+}
tests/test-thread-pool.c +4 -3
index aa156bc..6dfbb55 100644
@@ -55,7 +55,7 @@ static void test_submit_aio(void)
 {
     WorkerTestData data = { .n = 0, .ret = -EINPROGRESS };
     data.aiocb = thread_pool_submit_aio(pool, worker_cb, &data,
-                                        done_cb, &data);
+                                        done_cb, &data, false, 0);
 
     /* The callbacks are not called until after the first wait.  */
     active = 1;
@@ -119,7 +119,8 @@ static void test_submit_many(void)
     for (i = 0; i < 100; i++) {
         data[i].n = 0;
         data[i].ret = -EINPROGRESS;
-        thread_pool_submit_aio(pool, worker_cb, &data[i], done_cb, &data[i]);
+        thread_pool_submit_aio(pool, worker_cb, &data[i], done_cb, &data[i],
+                               false, 0);
     }
 
     active = 100;
@@ -148,7 +149,7 @@ static void test_cancel(void)
         data[i].n = 0;
         data[i].ret = -EINPROGRESS;
         data[i].aiocb = thread_pool_submit_aio(pool, long_cb, &data[i],
-                                               done_cb, &data[i]);
+                                               done_cb, &data[i], false, 0);
     }
 
     /* Starting the threads may be left to a bottom half.  Let it
thread-pool.c +36 -17

104 lines not on this page · raw message

#55 lore raw

RFC PATCH v2 36/49 pl031: vmstate in replay mode

In reply to Pavel Dovgalyuk

This patch modifies vmstate for PL031 RTC. It removes querying of the rtc and virtual clocks while saving and restoring VM state, because in replay mode these clocks are stopped while saving. And reading the clock while restoring the VM state may lead to read of the incorrect values, because clocks cache in replay module could not be loaded yet.

---
 hw/timer/pl031.c |   22 +++++++++++++++-------
 1 files changed, 15 insertions(+), 7 deletions(-)
hw/timer/pl031.c +15 -7
index 02c814f..017b1ce 100644
@@ -220,19 +220,27 @@ static void pl031_pre_save(void *opaque)
 {
     PL031State *s = opaque;
 
-    /* tick_offset is base_time - rtc_clock base time.  Instead, we want to
-     * store the base time relative to the QEMU_CLOCK_VIRTUAL for backwards-compatibility.  */
-    int64_t delta = qemu_clock_get_ns(rtc_clock) - qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
-    s->tick_offset_vmstate = s->tick_offset + delta / get_ticks_per_sec();
+    if (replay_mode == REPLAY_NONE) {
+        /* tick_offset is base_time - rtc_clock base time.  Instead, we want to
+         * store the base time relative to the QEMU_CLOCK_VIRTUAL for backwards-compatibility.  */
+        int64_t delta = qemu_clock_get_ns(rtc_clock) - qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
+        s->tick_offset_vmstate = s->tick_offset + delta / get_ticks_per_sec();
+    } else {
+        s->tick_offset_vmstate = s->tick_offset;
+    }
 }
 
 static int pl031_post_load(void *opaque, int version_id)
 {
     PL031State *s = opaque;
 
-    int64_t delta = qemu_clock_get_ns(rtc_clock) - qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
-    s->tick_offset = s->tick_offset_vmstate - delta / get_ticks_per_sec();
-    pl031_set_alarm(s);
+    if (replay_mode == REPLAY_NONE) {
+        int64_t delta = qemu_clock_get_ns(rtc_clock) - qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
+        s->tick_offset = s->tick_offset_vmstate - delta / get_ticks_per_sec();
+        pl031_set_alarm(s);
+    } else {
+        s->tick_offset = s->tick_offset_vmstate;
+    }
     return 0;
 }
 
#56 lore raw

RFC PATCH v2 37/49 replay: initialization and deinitialization

In reply to Pavel Dovgalyuk

This patch introduces the functions for enabling the record/replay and for freeing the resources when simulator closes.

---
 exec.c                   |    1 
 replay/replay-internal.h |    2 +
 replay/replay.c          |  127 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   11 ++++
 vl.c                     |    5 ++
 5 files changed, 146 insertions(+), 0 deletions(-)
exec.c +1 -0
index 36adb62..050e0a8 100644
@@ -732,6 +732,7 @@ void cpu_abort(CPUState *cpu, const char *fmt, ...)
     }
     va_end(ap2);
     va_end(ap);
+    replay_finish();
 #if defined(CONFIG_USER_ONLY)
     {
         struct sigaction act;
replay/replay-internal.h +2 -0
index 3f97fd7..7e5b3af 100755
@@ -36,6 +36,8 @@
 
 /* for checkpoint event */
 #define EVENT_CHECKPOINT            96
+/* end of log event */
+#define EVENT_END                   127
 
 /* Asynchronous events IDs */
 
replay/replay.c +127 -0

140 lines not on this page · raw message

replay/replay.h +11 -0
index e96c74f..e066cb2 100755
@@ -16,6 +16,8 @@
 #include <stdint.h>
 #include <time.h>
 
+struct QemuOpts;
+
 /* replay modes */
 #define REPLAY_NONE 0
 #define REPLAY_SAVE 1
@@ -43,6 +45,15 @@ extern int replay_icount;
 /*! Returns replay play submode */
 int replay_get_play_submode(void);
 
+/* Replay process control functions */
+
+/*! Enables recording or saving event log with specified parameters */
+void replay_configure(struct QemuOpts *opts, int mode);
+/*! Initializes timers used for snapshotting and enables events recording */
+void replay_init_timer(void);
+/*! Closes replay log file and frees other resources. */
+void replay_finish(void);
+
 /* Processing the instructions */
 
 /*! Returns number of executed instructions. */
vl.c +5 -0
index 053b2ba..6050faf 100644
@@ -4544,7 +4544,12 @@ int main(int argc, char **argv, char **envp)
         }
     }
 
+    replay_init_timer();
+
     main_loop();
+    if (replay_mode != REPLAY_NONE) {
+        replay_disable_events();
+    }
     bdrv_close_all();
     pause_all_vcpus();
     res_free();
#57 lore raw

RFC PATCH v2 38/49 replay: command line options

In reply to Pavel Dovgalyuk

This patch introduces command line options for enabling recording or replaying virtual machine behavior. "-record" option starts recording of the execution and saves it into the log, specified with "fname" parameter. "-replay" option is intended for replaying previously saved log.

---
 cpus.c          |   15 +++++++++--
 qemu-options.hx |   27 +++++++++++++++++++++
 vl.c            |   72 +++++++++++++++++++++++++++++++++++++++++++++++++++++--
 3 files changed, 109 insertions(+), 5 deletions(-)
cpus.c +12 -3
index 4f517ee..e16d8f8 100644
@@ -829,12 +829,21 @@ static void qemu_wait_io_event_common(CPUState *cpu)
 static void qemu_tcg_wait_io_event(void)
 {
     CPUState *cpu;
+    GMainContext *context = g_main_context_default();
 
-    while (all_cpu_threads_idle()) {
-       /* Start accounting real time to the virtual clock if the CPUs
-          are idle.  */
+    if (replay_mode == REPLAY_PLAY
+        && all_cpu_threads_idle() && first_cpu->halted) {
+        /* wakeup iothread when there is no code to execute in replay mode */
         qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
+        g_main_context_wakeup(context);
         qemu_cond_wait(tcg_halt_cond, &qemu_global_mutex);
+    } else {
+        while (all_cpu_threads_idle()) {
+           /* Start accounting real time to the virtual clock if the CPUs
+              are idle.  */
+            qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
+            qemu_cond_wait(tcg_halt_cond, &qemu_global_mutex); 
+        }
     }
 
     while (iothread_requesting_mutex) {
qemu-options.hx +27 -0
index 9e54686..7548f87 100644
@@ -3348,6 +3348,33 @@ Dump json-encoded vmstate information for current machine type to file
 in @var{file}
 ETEXI
 
+DEF("record", HAS_ARG, QEMU_OPTION_record,
+    "-record fname=<filename>[,suffix=<suffix>,snapshot=<on/off>]\n"
+    "                writes replay file for latter replaying\n",
+    QEMU_ARCH_ALL)
+STEXI
+@item -record fname=@var{file}[,suffix=@var{suffix},snapshot=@var{snapshot}]
+Writes compact execution trace into @var{file}.
+Changes for disk images are written
+into separate files with @var{suffix} added. If no @var{suffix} is
+specified, "replay_qcow" is used as suffix.
+If @var{snapshot} parameter is set as off, then original disk image will be
+modified. Default value is on.
+ETEXI
+
+DEF("replay", HAS_ARG, QEMU_OPTION_replay,
+    "-replay fname=<filename>[,suffix=<suffix>,snapshot=<on/off>]\n"
+    "                plays saved replay file\n", QEMU_ARCH_ALL)
+STEXI
+@item -replay fname=@var{filename}[,suffix=@var{suffix},snapshot=@var{snapshot}]
+Plays compact execution trace from @var{filename}.
+Changes for disk images and VM states are read
+from separate files with @var{suffix} added. If no @var{suffix} is
+specified, "replay_qcow" is used as suffix.
+If @var{snapshot} parameter is set as off, then original disk image will be
+modified. Default value is on.
+ETEXI
+
 HXCOMM This is the last statement. Insert new options before this line!
 STEXI
 @end table
vl.c +70 -2

144 lines not on this page · raw message

#58 lore raw

RFC PATCH v2 39/49 replay: snapshotting the virtual machine

In reply to Pavel Dovgalyuk

This patch adds 'period' parameter to the 'record' command line option. This parameters turns on periodic snapshotting of the VM which could be used by replay to move forward and backward in time.
If 'period' parameter is not specified, only one snapshot is made at the start of the virtual machine.

---
 cpus.c                   |   46 ++++++++-
 qemu-options.hx          |    3 -
 qemu-timer.c             |    4 +
 replay/replay-internal.h |   18 ++++
 replay/replay.c          |  223 ++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 287 insertions(+), 7 deletions(-)
cpus.c +41 -5
index e16d8f8..019cc0a 100644
@@ -228,11 +228,22 @@ int64_t cpu_get_clock(void)
  */
 void cpu_enable_ticks(void)
 {
+    int64_t ti;
     /* Here, the really thing protected by seqlock is cpu_clock_offset. */
     seqlock_write_lock(&timers_state.vm_clock_seqlock);
     if (!timers_state.cpu_ticks_enabled) {
         timers_state.cpu_ticks_offset -= cpu_get_real_ticks();
-        timers_state.cpu_clock_offset -= get_clock();
+
+        if (replay_mode == REPLAY_SAVE) {
+            ti = get_clock();
+            replay_save_clock(REPLAY_CLOCK_VIRTUAL, ti);
+        } else if (replay_mode == REPLAY_PLAY) {
+            ti = replay_read_clock(REPLAY_CLOCK_VIRTUAL);
+        } else {
+            ti = get_clock();
+        }
+
+        timers_state.cpu_clock_offset -= ti;
         timers_state.cpu_ticks_enabled = 1;
     }
     seqlock_write_unlock(&timers_state.vm_clock_seqlock);
@@ -437,6 +448,22 @@ void qemu_clock_warp(QEMUClockType type)
     }
 }
 
+static bool is_replay_enabled(void *opaque)
+{
+    return replay_mode != REPLAY_NONE;
+}
+
+static const VMStateDescription vmstate_timers_for_replay = {
+    .name = "timer for replay",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .minimum_version_id_old = 1,
+    .fields      = (VMStateField []) {
+        VMSTATE_INT64(cpu_ticks_prev, TimersState),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
 static const VMStateDescription vmstate_timers = {
     .name = "timer",
     .version_id = 2,
@@ -446,6 +473,14 @@ static const VMStateDescription vmstate_timers = {
         VMSTATE_INT64(dummy, TimersState),
         VMSTATE_INT64_V(cpu_clock_offset, TimersState, 2),
         VMSTATE_END_OF_LIST()
+    },
+    .subsections = (VMStateSubsection []) {
+        {
+            .vmsd = &vmstate_timers_for_replay,
+            .needed = is_replay_enabled,
+        }, {
+            /* empty */
+        }
     }
 };
 
@@ -537,9 +572,11 @@ static int do_vm_stop(RunState state)
     int ret = 0;
 
     if (runstate_is_running()) {
+        runstate_set(state);
+        /* Disable ticks can cause recursive call of vm_stop.
+           Stopping before calling functions prevents infinite recursion. */
         cpu_disable_ticks();
         pause_all_vcpus();
-        runstate_set(state);
         vm_state_notify(0, state);
         qapi_event_send_stop(&error_abort);
     }
@@ -1320,10 +1357,9 @@ static void tcg_exec_all(void)
         CPUState *cpu = next_cpu;
         CPUArchState *env = cpu->env_ptr;
 
-        qemu_clock_enable(QEMU_CLOCK_VIRTUAL,
-                          (cpu->singlestep_enabled & SSTEP_NOTIMER) == 0);
-
         if (cpu_can_run(cpu)) {
+            qemu_clock_enable(QEMU_CLOCK_VIRTUAL,
+                              (cpu->singlestep_enabled & SSTEP_NOTIMER) == 0);
             r = tcg_cpu_exec(env);
             if (r == EXCP_DEBUG) {
                 cpu_handle_guest_debug(cpu);
qemu-options.hx +2 -1
index 7548f87..fcfc8d1 100644
@@ -3353,8 +3353,9 @@ DEF("record", HAS_ARG, QEMU_OPTION_record,
     "                writes replay file for latter replaying\n",
     QEMU_ARCH_ALL)
 STEXI
-@item -record fname=@var{file}[,suffix=@var{suffix},snapshot=@var{snapshot}]
+@item -record fname=@var{file}[,period=@var{period},suffix=@var{suffix},snapshot=@var{snapshot}]
 Writes compact execution trace into @var{file}.
+VM state is auto saved every @var{period} second, if this parameter is specified.
 Changes for disk images are written
 into separate files with @var{suffix} added. If no @var{suffix} is
 specified, "replay_qcow" is used as suffix.
qemu-timer.c +3 -1
index dcf9b14..0dcba4c 100644
@@ -103,7 +103,9 @@ QEMUTimerList *timerlist_new(QEMUClockType type,
     QEMUClock *clock = qemu_clock_ptr(type);
 
     timer_list = g_malloc0(sizeof(QEMUTimerList));
-    qemu_event_init(&timer_list->timers_done_ev, false);
+    /* Create signaled event, because they should be signaled 
+       outside the timerlist_run_timers function */
+    qemu_event_init(&timer_list->timers_done_ev, true);
     timer_list->clock = clock;
     timer_list->notify_cb = cb;
     timer_list->notify_opaque = opaque;
replay/replay-internal.h +18 -0
index 7e5b3af..03a9cec 100755
@@ -15,6 +15,8 @@
 #include <stdio.h>
 #include "sysemu/sysemu.h"
 
+/* internal data for savevm */
+#define EVENT_END_STARTUP           0
 /* for time_t event */
 #define EVENT_TIME_T                1
 /* for tm event */
@@ -23,6 +25,10 @@
 #define EVENT_INTERRUPT             15
 /* for shutdown request */
 #define EVENT_SHUTDOWN              20
+/* for save VM event */
+#define EVENT_SAVE_VM_BEGIN         21
+/* for save VM event */
+#define EVENT_SAVE_VM_END           22
 /* for emulated exceptions */
 #define EVENT_EXCEPTION             23
 /* for async events */
@@ -53,9 +59,21 @@ typedef struct ReplayState {
     int skipping_instruction;
     /*! Current step - number of processed instructions and timer events. */
     uint64_t current_step;
+    /*! Temporary data for saving/loading replay file position. */
+    uint64_t file_offset;
 } ReplayState;
 extern ReplayState replay_state;
 
+/*! Information about saved VM state */
+struct SavedStateInfo {
+    /* Offset in the replay log file where state is saved. */
+    uint64_t file_offset;
+    /* Step number, corresponding to the saved state. */
+    uint64_t step;
+};
+/*! Reference to the saved state */
+typedef struct SavedStateInfo SavedStateInfo;
+
 extern volatile unsigned int replay_data_kind;
 extern volatile unsigned int replay_has_unread_data;
 
replay/replay.c +223 -0

288 lines not on this page · raw message

#59 lore raw

RFC PATCH v2 40/49 replay: recording of the user input

In reply to Pavel Dovgalyuk

This records user input (keyboard and mouse events) in record mode and replays these input events in replay mode.

---
 include/ui/input.h       |    2 +
 replay/Makefile.objs     |    1 
 replay/replay-events.c   |   48 +++++++++++++++++++++
 replay/replay-input.c    |  107 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   11 ++++-
 replay/replay.h          |    5 ++
 ui/input.c               |   79 ++++++++++++++++++++++++++--------
 7 files changed, 233 insertions(+), 20 deletions(-)
 create mode 100755 replay/replay-input.c
include/ui/input.h +2 -0
index 5d5ac00..d06a12d 100644
@@ -33,7 +33,9 @@ void qemu_input_handler_bind(QemuInputHandlerState *s,
                              const char *device_id, int head,
                              Error **errp);
 void qemu_input_event_send(QemuConsole *src, InputEvent *evt);
+void qemu_input_event_send_impl(QemuConsole *src, InputEvent *evt);
 void qemu_input_event_sync(void);
+void qemu_input_event_sync_impl(void);
 
 InputEvent *qemu_input_event_new_key(KeyValue *key, bool down);
 void qemu_input_event_send_key(QemuConsole *src, KeyValue *key, bool down);
replay/Makefile.objs +1 -0
index 257c320..3936296 100755
@@ -2,3 +2,4 @@ obj-y += replay.o
 obj-y += replay-internal.o
 obj-y += replay-events.o
 obj-y += replay-time.o
+obj-y += replay-input.o
replay/replay-events.c +48 -0

84 lines not on this page · raw message

replay/replay-input.c new file +107 -0

107 lines not on this page · raw message

replay/replay-internal.h +10 -1
index 03a9cec..6aea0f0 100755
@@ -49,7 +49,9 @@
 
 #define REPLAY_ASYNC_EVENT_BH          0
 #define REPLAY_ASYNC_EVENT_THREAD      1
-#define REPLAY_ASYNC_COUNT             2
+#define REPLAY_ASYNC_EVENT_INPUT       2
+#define REPLAY_ASYNC_EVENT_INPUT_SYNC  3
+#define REPLAY_ASYNC_COUNT             4
 
 typedef struct ReplayState {
     /*! Cached clock values. */
@@ -141,4 +143,11 @@ void replay_read_events(int opt);
 /*! Adds specified async event to the queue */
 void replay_add_event(int event_id, void *opaque);
 
+/* Input events */
+
+/*! Saves input event to the log */
+void replay_save_input_event(InputEvent *evt);
+/*! Reads input event from the log */
+InputEvent *replay_read_input_event(void);
+
 #endif
replay/replay.h +5 -0
index e066cb2..cc0b4f4 100755
@@ -17,6 +17,7 @@
 #include <time.h>
 
 struct QemuOpts;
+struct InputEvent;
 
 /* replay modes */
 #define REPLAY_NONE 0
@@ -113,5 +114,9 @@ void replay_disable_events(void);
 void replay_add_bh_event(void *bh, uint64_t id);
 /*! Adds thread event to the queue */
 void replay_add_thread_event(void *pool, void *req, uint64_t id);
+/*! Adds input event to the queue */
+void replay_add_input_event(struct InputEvent *event);
+/*! Adds input sync event to the queue */
+void replay_add_input_sync_event(void);
 
 #endif
ui/input.c +60 -19

141 lines not on this page · raw message

#60 lore raw

RFC PATCH v2 41/49 tap-win32: destroy the thread at exit

In reply to Pavel Dovgalyuk

This patch fixes resource leak caused by created thread which is not destroyed at exit.

---
 net/tap-win32.c |   11 ++++++-----
 1 files changed, 6 insertions(+), 5 deletions(-)
net/tap-win32.c +6 -5
index 8aee611..efd1c75 100644
@@ -99,6 +99,7 @@ typedef struct tap_win32_overlapped {
     HANDLE output_queue_semaphore;
     HANDLE free_list_semaphore;
     HANDLE tap_semaphore;
+    HANDLE hThread;
     CRITICAL_SECTION output_queue_cs;
     CRITICAL_SECTION free_list_cs;
     OVERLAPPED read_overlapped;
@@ -625,8 +626,9 @@ static int tap_win32_open(tap_win32_overlapped_t **phandle,
 
     *phandle = &tap_overlapped;
 
-    CreateThread(NULL, 0, tap_win32_thread_entry,
-                 (LPVOID)&tap_overlapped, 0, &idThread);
+    tap_overlapped.hThread = CreateThread(NULL, 0, tap_win32_thread_entry,
+                                          (LPVOID)&tap_overlapped,
+                                          0, &idThread);
     return 0;
 }
 
@@ -643,9 +645,8 @@ static void tap_cleanup(NetClientState *nc)
 
     qemu_del_wait_object(s->handle->tap_semaphore, NULL, NULL);
 
-    /* FIXME: need to kill thread and close file handle:
-       tap_win32_close(s);
-    */
+    TerminateThread(s->handle->hThread, 0);
+    CloseHandle(s->handle->handle);
 }
 
 static ssize_t tap_receive(NetClientState *nc, const uint8_t *buf, size_t size)
#61 lore raw

RFC PATCH v2 42/49 replay: network packets record/replay

In reply to Pavel Dovgalyuk

This patch implements passing network packets to replay module in record mode. New virtual network adapter is implemented to replay the packets when they are read from the log file.

---
 net/Makefile.objs        |    1 
 net/clients.h            |    3 +
 net/dump.c               |    6 +
 net/hub.c                |    1 
 net/net-replay.c         |   66 ++++++++++++++++
 net/net.c                |    7 +-
 net/slirp.c              |   14 +++
 net/socket.c             |   35 ++++++++
 net/tap-win32.c          |   14 +++
 net/tap.c                |   23 +++++-
 net/vde.c                |   14 +++
 qapi-schema.json         |   13 +++
 replay/Makefile.objs     |    1 
 replay/replay-events.c   |   17 ++++
 replay/replay-internal.h |   21 +++++
 replay/replay-net.c      |  190 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.c          |    5 +
 replay/replay.h          |   10 ++
 slirp/slirp.c            |    9 ++
 19 files changed, 436 insertions(+), 14 deletions(-)
 create mode 100755 net/net-replay.c
 create mode 100755 replay/replay-net.c
net/Makefile.objs +1 -0
index ec19cb3..def44e3 100644
@@ -2,6 +2,7 @@ common-obj-y = net.o queue.o checksum.o util.o hub.o
 common-obj-y += socket.o
 common-obj-y += dump.o
 common-obj-y += eth.o
+common-obj-y += net-replay.o
 common-obj-$(CONFIG_L2TPV3) += l2tpv3.o
 common-obj-$(CONFIG_POSIX) += tap.o vhost-user.o
 common-obj-$(CONFIG_LINUX) += tap-linux.o
net/clients.h +3 -0
index 2e8feda..f75f43b 100644
@@ -62,4 +62,7 @@ int net_init_netmap(const NetClientOptions *opts, const char *name,
 int net_init_vhost_user(const NetClientOptions *opts, const char *name,
                         NetClientState *peer);
 
+int net_init_replay(const NetClientOptions *opts, const char *name,
+                    NetClientState *peer);
+
 #endif /* QEMU_NET_CLIENTS_H */
net/dump.c +6 -0
index 9d3a09e..962f2cd 100644
@@ -28,6 +28,7 @@
 #include "qemu/log.h"
 #include "qemu/timer.h"
 #include "hub.h"
+#include "replay/replay.h"
 
 typedef struct DumpState {
     NetClientState nc;
@@ -158,6 +159,11 @@ int net_init_dump(const NetClientOptions *opts, const char *name,
 
     assert(peer);
 
+    if (replay_mode == REPLAY_SAVE) {
+        fprintf(stderr, "-net dump is not permitted in replay mode\n");
+        exit(1);
+    }
+
     if (dump->has_file) {
         file = dump->file;
     } else {
net/hub.c +1 -0
index 7e0f2d6..91fdfb6 100644
@@ -323,6 +323,7 @@ void net_hub_check_clients(void)
             case NET_CLIENT_OPTIONS_KIND_SOCKET:
             case NET_CLIENT_OPTIONS_KIND_VDE:
             case NET_CLIENT_OPTIONS_KIND_VHOST_USER:
+            case NET_CLIENT_OPTIONS_KIND_REPLAY:
                 has_host_dev = 1;
                 break;
             default:
net/net-replay.c new file +66 -0

66 lines not on this page · raw message

net/net.c +6 -1
index 6d930ea..62067f5 100644
@@ -809,6 +809,7 @@ static int (* const net_client_init_fun[NET_CLIENT_OPTIONS_KIND_MAX])(
 #ifdef CONFIG_L2TPV3
         [NET_CLIENT_OPTIONS_KIND_L2TPV3]    = net_init_l2tpv3,
 #endif
+        [NET_CLIENT_OPTIONS_KIND_REPLAY]    = net_init_replay,
 };
 
 
@@ -1264,7 +1265,11 @@ int net_init_clients(void)
         /* if no clients, we use a default config */
         qemu_opts_set(net, NULL, "type", "nic");
 #ifdef CONFIG_SLIRP
-        qemu_opts_set(net, NULL, "type", "user");
+        if (replay_mode != REPLAY_PLAY) {
+            qemu_opts_set(net, NULL, "type", "user");
+        } else {
+            qemu_opts_set(net, NULL, "type", "replay");
+        }
 #endif
     }
 
net/slirp.c +13 -1
index 8fddc03..2ac3b6b 100644
@@ -36,6 +36,7 @@
 #include "qemu/sockets.h"
 #include "slirp/libslirp.h"
 #include "sysemu/char.h"
+#include "replay/replay.h"
 
 static int get_str_sep(char *buf, int buf_size, const char **pp, int sep)
 {
@@ -103,7 +104,11 @@ void slirp_output(void *opaque, const uint8_t *pkt, int pkt_len)
 {
     SlirpState *s = opaque;
 
-    qemu_send_packet(&s->nc, pkt, pkt_len);
+    if (replay_mode == REPLAY_SAVE) {
+        replay_save_net_packet(&s->nc, pkt, pkt_len);
+    } else {
+        qemu_send_packet(&s->nc, pkt, pkt_len);
+    }
 }
 
 static ssize_t net_slirp_receive(NetClientState *nc, const uint8_t *buf, size_t size)
@@ -267,6 +272,13 @@ static int net_slirp_init(NetClientState *peer, const char *model,
     }
 #endif
 
+    if (replay_mode == REPLAY_PLAY) {
+        fprintf(stderr, "-net user is not permitted in replay mode\n");
+        exit(1);
+    } else {
+        replay_add_network_client(nc);
+    }
+
     return 0;
 
 error:
net/socket.c +33 -2
index fb21e20..bbca37b 100644
@@ -32,6 +32,7 @@
 #include "qemu/sockets.h"
 #include "qemu/iov.h"
 #include "qemu/main-loop.h"
+#include "replay/replay.h"
 
 typedef struct NetSocketState {
     NetClientState nc;
@@ -211,7 +212,11 @@ static void net_socket_send(void *opaque)
             buf += l;
             size -= l;
             if (s->index >= s->packet_len) {
-                qemu_send_packet(&s->nc, s->buf, s->packet_len);
+                if (replay_mode == REPLAY_SAVE) {
+                    replay_save_net_packet(&s->nc, s->buf, s->packet_len);
+                } else {
+                    qemu_send_packet(&s->nc, s->buf, s->packet_len);
+                }
                 s->index = 0;
                 s->state = 0;
             }
@@ -234,7 +239,11 @@ static void net_socket_send_dgram(void *opaque)
         net_socket_write_poll(s, false);
         return;
     }
-    qemu_send_packet(&s->nc, s->buf, size);
+    if (replay_mode == REPLAY_SAVE) {
+        replay_save_net_packet(&s->nc, s->buf, size);
+    } else {
+        qemu_send_packet(&s->nc, s->buf, size);
+    }
 }
 
 static int net_socket_mcast_create(struct sockaddr_in *mcastaddr, struct in_addr *localaddr)
@@ -406,6 +415,13 @@ static NetSocketState *net_socket_fd_init_dgram(NetClientState *peer,
         s->dgram_dst = saddr;
     }
 
+    if (replay_mode == REPLAY_PLAY) {
+        fprintf(stderr, "-net socket is not permitted in replay mode\n");
+        exit(1);
+    } else {
+        replay_add_network_client(nc);
+    }
+
     return s;
 
 err:
@@ -452,6 +468,14 @@ static NetSocketState *net_socket_fd_init_stream(NetClientState *peer,
     } else {
         qemu_set_fd_handler(s->fd, NULL, net_socket_connect, s);
     }
+
+    if (replay_mode == REPLAY_PLAY) {
+        printf("-net socket is not permitted in replay mode\n");
+        exit(1);
+    } else {
+        replay_add_network_client(nc);
+    }
+
     return s;
 }
 
@@ -548,6 +572,13 @@ static int net_socket_listen_init(NetClientState *peer,
     s->listen_fd = fd;
     s->nc.link_down = true;
 
+    if (replay_mode == REPLAY_PLAY) {
+        printf("-net socket is not permitted in replay mode\n");
+        exit(1);
+    } else {
+        replay_add_network_client(nc);
+    }
+
     qemu_set_fd_handler(s->listen_fd, net_socket_accept, NULL, s);
     return 0;
 }
net/tap-win32.c +13 -1
index efd1c75..1f2a2f3 100644
@@ -34,6 +34,7 @@
 #include "net/tap.h"            /* tap_has_ufo, ... */
 #include "sysemu/sysemu.h"
 #include "qemu/error-report.h"
+#include "replay/replay.h"
 #include <stdio.h>
 #include <windows.h>
 #include <winioctl.h>
@@ -665,7 +666,11 @@ static void tap_win32_send(void *opaque)
 
     size = tap_win32_read(s->handle, &buf, max_size);
     if (size > 0) {
-        qemu_send_packet(&s->nc, buf, size);
+        if (replay_mode == REPLAY_SAVE) {
+            replay_save_net_packet(&s->nc, buf, size);
+        } else {
+            qemu_send_packet(&s->nc, buf, size);
+        }
         tap_win32_free_buffer(s->handle, buf);
     }
 }
@@ -749,6 +754,13 @@ static int tap_win32_init(NetClientState *peer, const char *model,
 
     qemu_add_wait_object(s->handle->tap_semaphore, tap_win32_send, s);
 
+    if (replay_mode == REPLAY_PLAY) {
+        fprintf(stderr, "-net tap is not permitted in replay mode\n");
+        exit(1);
+    } else {
+        replay_add_network_client(nc);
+    }
+
     return 0;
 }
 
net/tap.c +19 -4
index a40f7f0..8b2094e 100644
@@ -39,6 +39,7 @@
 #include "sysemu/sysemu.h"
 #include "qemu-common.h"
 #include "qemu/error-report.h"
+#include "replay/replay.h"
 
 #include "net/tap.h"
 
@@ -203,11 +204,17 @@ static void tap_send(void *opaque)
             size -= s->host_vnet_hdr_len;
         }
 
-        size = qemu_send_packet_async(&s->nc, buf, size, tap_send_completed);
-        if (size == 0) {
-            tap_read_poll(s, false);
+        if (replay_mode == REPLAY_SAVE) {
+            replay_save_net_packet(&s->nc, buf, size);
             break;
-        } else if (size < 0) {
+        } else {
+            size = qemu_send_packet_async(&s->nc, buf, size, tap_send_completed);
+            if (size == 0) {
+                tap_read_poll(s, false);
+                break;
+            } else if (size < 0) {
+                break;
+            }
             break;
         }
     }
@@ -353,6 +360,14 @@ static TAPState *net_tap_fd_init(NetClientState *peer,
     }
     tap_read_poll(s, true);
     s->vhost_net = NULL;
+
+    if (replay_mode == REPLAY_PLAY) {
+        fprintf(stderr, "-net tap is not permitted in replay mode\n");
+        exit(1);
+    } else {
+        replay_add_network_client(nc);
+    }
+
     return s;
 }
 
net/vde.c +13 -1
index 2a619fb..ca34c4b 100644
@@ -30,6 +30,7 @@
 #include "qemu-common.h"
 #include "qemu/option.h"
 #include "qemu/main-loop.h"
+#include "replay/replay.h"
 
 typedef struct VDEState {
     NetClientState nc;
@@ -44,7 +45,11 @@ static void vde_to_qemu(void *opaque)
 
     size = vde_recv(s->vde, (char *)buf, sizeof(buf), 0);
     if (size > 0) {
-        qemu_send_packet(&s->nc, buf, size);
+        if (replay_mode == REPLAY_SAVE) {
+            replay_save_net_packet(&s->nc, buf, size);
+        } else {
+            qemu_send_packet(&s->nc, buf, size);
+        }
     }
 }
 
@@ -106,6 +111,13 @@ static int net_vde_init(NetClientState *peer, const char *model,
 
     qemu_set_fd_handler(vde_datafd(s->vde), vde_to_qemu, NULL, s);
 
+    if (replay_mode == REPLAY_PLAY) {
+        fprintf(stderr, "-net vde is not permitted in replay mode\n");
+        exit(1);
+    } else {
+        replay_add_network_client(nc);
+    }
+
     return 0;
 }
 
qapi-schema.json +12 -1
index b11aad2..fee541a 100644
@@ -2211,6 +2211,16 @@
     '*vhostforce':    'bool' } }
 
 ##
+# @NetdevReplayOptions
+#
+# Reads network traffic from the log in replay mode.
+#
+# Since: 2.2
+##
+{ 'type': 'NetdevReplayOptions',
+  'data': { } }
+
+##
 # @NetClientOptions
 #
 # A discriminated record of network device traits.
@@ -2233,7 +2243,8 @@
     'bridge':   'NetdevBridgeOptions',
     'hubport':  'NetdevHubPortOptions',
     'netmap':   'NetdevNetmapOptions',
-    'vhost-user': 'NetdevVhostUserOptions' } }
+    'vhost-user': 'NetdevVhostUserOptions',
+    'replay':   'NetdevReplayOptions' } }
 
 ##
 # @NetLegacy
replay/Makefile.objs +1 -0
index 3936296..43bafbb 100755
@@ -3,3 +3,4 @@ obj-y += replay-internal.o
 obj-y += replay-events.o
 obj-y += replay-time.o
 obj-y += replay-input.o
+obj-y += replay-net.o
replay/replay-events.c +17 -0
index d7ed0b2..077b012 100755
@@ -53,6 +53,9 @@ static void replay_run_event(Event *event)
     case REPLAY_ASYNC_EVENT_INPUT_SYNC:
         qemu_input_event_sync_impl();
         break;
+    case REPLAY_ASYNC_EVENT_NETWORK:
+        replay_net_send_packet(event->opaque);
+        break;
     default:
         fprintf(stderr, "Replay: invalid async event ID (%d) in the queue\n", 
                 event->event_kind);
@@ -179,6 +182,9 @@ void replay_save_events(int opt)
             case REPLAY_ASYNC_EVENT_INPUT:
                 replay_save_input_event(event->opaque);
                 break;
+            case REPLAY_ASYNC_EVENT_NETWORK:
+                replay_net_save_packet(event->opaque);
+                break;
             }
         }
 
@@ -238,6 +244,17 @@ void replay_read_events(int opt)
             replay_fetch_data_kind();
             /* continue with the next event */
             continue;
+        case REPLAY_ASYNC_EVENT_NETWORK:
+            e.opaque = replay_net_read_packet();
+            e.event_kind = read_event_kind;
+            replay_run_event(&e);
+
+            replay_has_unread_data = 0;
+            read_event_kind = -1;
+            read_opt = -1;
+            replay_fetch_data_kind();
+            /* continue with the next event */
+            continue;
         default:
             fprintf(stderr, "Unknown ID %d of replay event\n", read_event_kind);
             exit(1);
replay/replay-internal.h +20 -1
index 6aea0f0..254a386 100755
@@ -51,7 +51,8 @@
 #define REPLAY_ASYNC_EVENT_THREAD      1
 #define REPLAY_ASYNC_EVENT_INPUT       2
 #define REPLAY_ASYNC_EVENT_INPUT_SYNC  3
-#define REPLAY_ASYNC_COUNT             4
+#define REPLAY_ASYNC_EVENT_NETWORK     4
+#define REPLAY_ASYNC_COUNT             5
 
 typedef struct ReplayState {
     /*! Cached clock values. */
@@ -150,4 +151,22 @@ void replay_save_input_event(InputEvent *evt);
 /*! Reads input event from the log */
 InputEvent *replay_read_input_event(void);
 
+/* Network events */
+
+/*! Initializes network data structures. */
+void replay_net_init(void);
+/*! Cleans up network data structures. */
+void replay_net_free(void);
+/*! Reads packets offsets array from the log. */
+void replay_net_read_packets_data(void);
+/*! Writes packets offsets array to the log. */
+void replay_net_write_packets_data(void);
+/*! Saves network packet into the log. */
+void replay_net_save_packet(void *opaque);
+/*! Reads network packet from the log. */
+void *replay_net_read_packet(void);
+/*! Called to send packet that was read or received from external input
+    to the net queue. */
+void replay_net_send_packet(void *opaque);
+
 #endif
replay/replay-net.c new file +190 -0

190 lines not on this page · raw message

replay/replay.c +5 -0
index 67b32e0..47a5809 100755
@@ -461,6 +461,8 @@ static void replay_enable(const char *fname, int mode)
     replay_state.current_step = 0;
     current_saved_state = 0;
 
+    replay_net_init();
+
     /* skip file header for SAVE and check it for PLAY */
     if (replay_mode == REPLAY_SAVE) {
         fseek(replay_file, HEADER_SIZE, SEEK_SET);
@@ -479,6 +481,7 @@ static void replay_enable(const char *fname, int mode)
             saved_states = g_malloc(sizeof(SavedStateInfo) * saved_states_count);
             fread(saved_states, sizeof(SavedStateInfo), saved_states_count, replay_file);
         }
+        replay_net_read_packets_data();
         /* go to the beginning */
         fseek(replay_file, 12, SEEK_SET);
     }
@@ -560,6 +563,7 @@ void replay_finish(void)
                 fwrite(saved_states, sizeof(SavedStateInfo),
                        saved_states_count, replay_file);
             }
+            replay_net_write_packets_data();
 
             /* write header */
             fseek(replay_file, 0, SEEK_SET);
@@ -588,5 +592,6 @@ void replay_finish(void)
         replay_image_suffix = NULL;
     }
 
+    replay_net_free();
     replay_finish_events();
 }
replay/replay.h +10 -0
index cc0b4f4..f654673 100755
@@ -18,6 +18,7 @@
 
 struct QemuOpts;
 struct InputEvent;
+struct NetClientState;
 
 /* replay modes */
 #define REPLAY_NONE 0
@@ -119,4 +120,13 @@ void replay_add_input_event(struct InputEvent *event);
 /*! Adds input sync event to the queue */
 void replay_add_input_sync_event(void);
 
+/* Network */
+
+/*! Registers net client in the replay module. */
+void replay_add_network_client(struct NetClientState *nc);
+/*! Saves incoming network packet in the replay log. */
+void replay_save_net_packet(struct NetClientState *nc, const uint8_t *buf,
+                            size_t size);
+
+
 #endif
slirp/slirp.c +7 -2
index 35f819a..034c268 100644
@@ -26,6 +26,7 @@
 #include "sysemu/char.h"
 #include "slirp.h"
 #include "hw/hw.h"
+#include "replay/replay.h"
 
 /* host loopback address */
 struct in_addr loopback_addr;
@@ -234,8 +235,12 @@ Slirp *slirp_init(int restricted, struct in_addr vnetwork,
 
     slirp->opaque = opaque;
 
-    register_savevm(NULL, "slirp", 0, 3,
-                    slirp_state_save, slirp_state_load, slirp);
+    /* Do not save slirp state in record mode, because slirp device
+       will not be created in replay mode */
+    if (replay_mode == REPLAY_NONE) {
+        register_savevm(NULL, "slirp", 0, 3,
+                        slirp_state_save, slirp_state_load, slirp);
+    }
 
     QTAILQ_INSERT_TAIL(&slirp_instances, slirp, entry);
 
#62 lore raw

RFC PATCH v2 43/49 replay: audio data record/replay

In reply to Pavel Dovgalyuk

This patch adds deterministic replay for audio adapter. Replay module saves data from the microphone and "end-of-playback" events.
Support of audio record and replay is implemented only for Win32 hosts.

---
 audio/audio.c            |   14 ++-
 audio/audio_win_int.h    |    3 +
 audio/winwaveaudio.c     |  167 ++++++++++++++++++++++++++--------
 replay/Makefile.objs     |    1 
 replay/replay-audio.c    |  228 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.c |    4 +
 replay/replay-internal.h |   14 +++
 replay/replay.h          |   20 ++++
 8 files changed, 408 insertions(+), 43 deletions(-)
 create mode 100755 replay/replay-audio.c
audio/audio.c +10 -4
index 9d018e9..7b272b7 100644
@@ -26,6 +26,7 @@
 #include "monitor/monitor.h"
 #include "qemu/timer.h"
 #include "sysemu/sysemu.h"
+#include "replay/replay.h"
 
 #define AUDIO_CAP "audio"
 #include "audio_int.h"
@@ -1201,7 +1202,9 @@ void AUD_set_active_out (SWVoiceOut *sw, int on)
             if (!hw->enabled) {
                 hw->enabled = 1;
                 if (s->vm_running) {
-                    hw->pcm_ops->ctl_out (hw, VOICE_ENABLE, conf.try_poll_out);
+                    hw->pcm_ops->ctl_out(hw, VOICE_ENABLE,
+                                         replay_mode == REPLAY_NONE
+                                             ? conf.try_poll_out : 0);
                     audio_reset_timer (s);
                 }
             }
@@ -1763,11 +1766,13 @@ static void audio_vm_change_state_handler (void *opaque, int running,
 
     s->vm_running = running;
     while ((hwo = audio_pcm_hw_find_any_enabled_out (hwo))) {
-        hwo->pcm_ops->ctl_out (hwo, op, conf.try_poll_out);
+        hwo->pcm_ops->ctl_out(hwo, op, replay_mode == REPLAY_NONE
+                                           ? conf.try_poll_out : 0);
     }
 
     while ((hwi = audio_pcm_hw_find_any_enabled_in (hwi))) {
-        hwi->pcm_ops->ctl_in (hwi, op, conf.try_poll_in);
+        hwi->pcm_ops->ctl_in(hwi, op, replay_mode == REPLAY_NONE
+                                          ? conf.try_poll_in : 0);
     }
     audio_reset_timer (s);
 }
@@ -1810,9 +1815,10 @@ static void audio_atexit (void)
 
 static const VMStateDescription vmstate_audio = {
     .name = "audio",
-    .version_id = 1,
+    .version_id = 2,
     .minimum_version_id = 1,
     .fields = (VMStateField[]) {
+        VMSTATE_TIMER_V(ts, AudioState, 2),
         VMSTATE_END_OF_LIST()
     }
 };
audio/audio_win_int.h +3 -0
index fa5b3cb..0525ae6 100644
@@ -7,4 +7,7 @@ int waveformat_from_audio_settings (WAVEFORMATEX *wfx,
 int waveformat_to_audio_settings (WAVEFORMATEX *wfx,
                                   struct audsettings *as);
 
+void winwave_callback_out_impl(void *dwInstance, WAVEHDR *h);
+void winwave_callback_in_impl(void *dwInstance, WAVEHDR *h);
+
 #endif /* AUDIO_WIN_INT_H */
audio/winwaveaudio.c +128 -39

284 lines not on this page · raw message

replay/Makefile.objs +1 -0
index 43bafbb..1d57e71 100755
@@ -4,3 +4,4 @@ obj-y += replay-events.o
 obj-y += replay-time.o
 obj-y += replay-input.o
 obj-y += replay-net.o
+obj-y += replay-audio.o
replay/replay-audio.c new file +228 -0

228 lines not on this page · raw message

replay/replay-internal.c +4 -0
index f416317..fab9c0c 100755
@@ -151,5 +151,9 @@ void replay_save_instructions(void)
             replay_state.current_step += first_cpu->instructions_count;
             first_cpu->instructions_count = 0;
         }
+#ifdef _WIN32
+        replay_save_sound_in();
+        replay_save_sound_out();
+#endif
     }
 }
replay/replay-internal.h +14 -0
index 254a386..0f001f4 100755
@@ -21,6 +21,10 @@
 #define EVENT_TIME_T                1
 /* for tm event */
 #define EVENT_TM                    2
+/* for outgoing sound event */
+#define EVENT_SOUND_OUT             7
+/* for incoming sound event */
+#define EVENT_SOUND_IN              8
 /* for software interrupt */
 #define EVENT_INTERRUPT             15
 /* for shutdown request */
@@ -169,4 +173,14 @@ void *replay_net_read_packet(void);
     to the net queue. */
 void replay_net_send_packet(void *opaque);
 
+/* Sound events */
+
+/*! Returns true, when there are any pending sound events. */
+bool replay_has_sound_events(void);
+void replay_save_sound_out(void);
+void replay_save_sound_in(void);
+void replay_read_sound_out(void);
+void replay_read_sound_in(void);
+void replay_sound_flush_queue(void);
+
 #endif
replay/replay.h +20 -0
index f654673..700b19a 100755
@@ -15,6 +15,10 @@
 #include <stdbool.h>
 #include <stdint.h>
 #include <time.h>
+#ifdef _WIN32
+#include <windows.h>
+#include <mmsystem.h>
+#endif
 
 struct QemuOpts;
 struct InputEvent;
@@ -128,5 +132,21 @@ void replay_add_network_client(struct NetClientState *nc);
 void replay_save_net_packet(struct NetClientState *nc, const uint8_t *buf,
                             size_t size);
 
+/* Audio */
+
+#ifdef _WIN32
+/*! Microphone event. */
+void replay_sound_in_event(WAVEHDR *hdr);
+/*! Adds header to the queue.
+    In record mode this header is queued for saving into log.
+    In replay mode this header is queued for reading from log.
+    Returns 1 in replay mode when queue is full.
+    Otherwise returns 0. */
+int replay_sound_out_event(WAVEHDR *hdr);
+/*! Initializes queue for sound input. */
+void replay_init_sound_in(void *instance, WAVEHDR *hdrs, int sz);
+/*! Initializes queue for sound output. */
+void replay_init_sound_out(void *instance, WAVEHDR *hdrs, int sz);
+#endif
 
 #endif
#63 lore raw

RFC PATCH v2 44/49 replay: serial port

In reply to Pavel Dovgalyuk

This patch implements record and replay of serial ports.
In record mode serial port can be connected to the source of the data:
-serial tcp:127.0.0.1:23,nowait,server
In replay mode it should not be connected to any data source, but should be initialized with null: -serial null

---
 include/sysemu/char.h    |   25 ++++++++++++
 qemu-char.c              |   55 +++++++++++++++++++++++---
 replay/Makefile.objs     |    1 
 replay/replay-char.c     |   99 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-events.c   |   18 ++++++++
 replay/replay-internal.h |   14 ++++++-
 replay/replay.c          |   14 +++++++
 replay/replay.h          |   13 ++++++
 vl.c                     |    4 +-
 9 files changed, 236 insertions(+), 7 deletions(-)
 create mode 100755 replay/replay-char.c
include/sysemu/char.h +25 -0
index 0bbd631..a7eb578 100644
@@ -85,6 +85,7 @@ struct CharDriverState {
     int is_mux;
     guint fd_in_tag;
     QemuOpts *opts;
+    bool replay;
     QTAILQ_ENTRY(CharDriverState) next;
 };
 
@@ -126,6 +127,21 @@ CharDriverState *qemu_chr_new(const char *label, const char *filename,
                               void (*init)(struct CharDriverState *s));
 
 /**
+ * @qemu_chr_new_replay:
+ *
+ * Create a new character backend from a URI.
+ * All recieved data will be logged by replay module.
+ *
+ * @label the name of the backend
+ * @filename the URI
+ * @init not sure..
+ *
+ * Returns: a new character backend
+ */
+CharDriverState *qemu_chr_new_replay(const char *label, const char *filename,
+                                     void (*init)(struct CharDriverState *s));
+
+/**
  * @qemu_chr_delete:
  *
  * Destroy a character backend.
@@ -320,6 +336,15 @@ int qemu_chr_be_can_write(CharDriverState *s);
  */
 void qemu_chr_be_write(CharDriverState *s, uint8_t *buf, int len);
 
+/**
+ * @qemu_chr_be_write_impl:
+ *
+ * Implementation of back end writing. Used by replay module.
+ *
+ * @buf a buffer to receive data from the front end
+ * @len the number of bytes to receive from the front end
+ */
+void qemu_chr_be_write_impl(CharDriverState *s, uint8_t *buf, int len);
 
 /**
  * @qemu_chr_be_event:
qemu-char.c +50 -5

94 lines not on this page · raw message

replay/Makefile.objs +1 -0
index 1d57e71..c54e550 100755
@@ -5,3 +5,4 @@ obj-y += replay-time.o
 obj-y += replay-input.o
 obj-y += replay-net.o
 obj-y += replay-audio.o
+obj-y += replay-char.o
replay/replay-char.c new file +99 -0

99 lines not on this page · raw message

replay/replay-events.c +18 -0
index 077b012..6e0120e 100755
@@ -56,6 +56,9 @@ static void replay_run_event(Event *event)
     case REPLAY_ASYNC_EVENT_NETWORK:
         replay_net_send_packet(event->opaque);
         break;
+    case REPLAY_ASYNC_EVENT_CHAR:
+        replay_event_char_run(event->opaque);
+        break;
     default:
         fprintf(stderr, "Replay: invalid async event ID (%d) in the queue\n", 
                 event->event_kind);
@@ -185,6 +188,9 @@ void replay_save_events(int opt)
             case REPLAY_ASYNC_EVENT_NETWORK:
                 replay_net_save_packet(event->opaque);
                 break;
+            case REPLAY_ASYNC_EVENT_CHAR:
+                replay_event_char_save(event->opaque);
+                break;
             }
         }
 
@@ -255,6 +261,18 @@ void replay_read_events(int opt)
             replay_fetch_data_kind();
             /* continue with the next event */
             continue;
+        case REPLAY_ASYNC_EVENT_CHAR:
+            e.event_kind = read_event_kind;
+            e.opaque = replay_event_char_read();
+
+            replay_has_unread_data = 0;
+            read_event_kind = -1;
+            read_opt = -1;
+            replay_fetch_data_kind();
+
+            replay_run_event(&e);
+            /* continue with the next event */
+            continue;
         default:
             fprintf(stderr, "Unknown ID %d of replay event\n", read_event_kind);
             exit(1);
replay/replay-internal.h +13 -1
index 0f001f4..dfcd5fd 100755
@@ -38,6 +38,8 @@
 /* for async events */
 #define EVENT_ASYNC                 24
 #define EVENT_ASYNC_OPT             25
+/* for int data */
+#define EVENT_DATA_INT              26
 /* for instruction event */
 #define EVENT_INSTRUCTION           32
 /* for clock read/writes */
@@ -56,7 +58,8 @@
 #define REPLAY_ASYNC_EVENT_INPUT       2
 #define REPLAY_ASYNC_EVENT_INPUT_SYNC  3
 #define REPLAY_ASYNC_EVENT_NETWORK     4
-#define REPLAY_ASYNC_COUNT             5
+#define REPLAY_ASYNC_EVENT_CHAR        5
+#define REPLAY_ASYNC_COUNT             6
 
 typedef struct ReplayState {
     /*! Cached clock values. */
@@ -183,4 +186,13 @@ void replay_read_sound_out(void);
 void replay_read_sound_in(void);
 void replay_sound_flush_queue(void);
 
+/* Character devices */
+
+/*! Called to run char device event. */
+void replay_event_char_run(void *opaque);
+/*! Writes char event to the file. */
+void replay_event_char_save(void *opaque);
+/*! Reads char event from the file. */
+void *replay_event_char_read(void);
+
 #endif
replay/replay.c +14 -0
index 47a5809..8e15b04 100755
@@ -595,3 +595,17 @@ void replay_finish(void)
     replay_net_free();
     replay_finish_events();
 }
+
+void replay_data_int(int *data)
+{
+    if (replay_file && replay_mode == REPLAY_PLAY) {
+        skip_async_events_until(EVENT_DATA_INT);
+        *data = replay_get_dword();
+        replay_check_error();
+        replay_has_unread_data = 0;
+    } else if (replay_file && replay_mode == REPLAY_SAVE) {
+        replay_save_instructions();
+        replay_put_event(EVENT_DATA_INT);
+        replay_put_dword(*data);
+    }
+}
replay/replay.h +13 -0
index 700b19a..533fa0a 100755
@@ -23,6 +23,7 @@
 struct QemuOpts;
 struct InputEvent;
 struct NetClientState;
+struct CharDriverState;
 
 /* replay modes */
 #define REPLAY_NONE 0
@@ -149,4 +150,16 @@ void replay_init_sound_in(void *instance, WAVEHDR *hdrs, int sz);
 void replay_init_sound_out(void *instance, WAVEHDR *hdrs, int sz);
 #endif
 
+/* Character device */
+
+/*! Registers char driver to save it's events */
+void replay_register_char_driver(struct CharDriverState *chr);
+/*! Saves write to char device event to the log */
+void replay_chr_be_write(struct CharDriverState *s, uint8_t *buf, int len);
+
+/* Other data */
+
+/*! Writes or reads integer value to/from replay log. */
+void replay_data_int(int *data);
+
 #endif
vl.c +3 -1
index f8c1e9e..9957bfe 100644
@@ -2559,7 +2559,9 @@ static int serial_parse(const char *devname)
         exit(1);
     }
     snprintf(label, sizeof(label), "serial%d", index);
-    serial_hds[index] = qemu_chr_new(label, devname, NULL);
+    serial_hds[index] = (replay_mode == REPLAY_NONE ? qemu_chr_new
+                                                    : qemu_chr_new_replay)
+                        (label, devname, NULL);
     if (!serial_hds[index]) {
         fprintf(stderr, "qemu: could not connect serial device"
                 " to character backend '%s'\n", devname);
#64 lore raw

RFC PATCH v2 45/49 replay: USB passthrough

In reply to Pavel Dovgalyuk

It writes all external data, returned by libusb,
to the log. This data is read in replay mode instead of calling libusb functions.
Command line option for connecting USB device to simulator should be specified in both record and replay modes. In replay mode only log file is read and USB device could be not connected to the host machine.

---
 hw/usb/host-libusb.c     |  525 ++++++++++++++++++++++++++++++----------------
 include/hw/host-libusb.h |  105 +++++++++
 replay/Makefile.objs     |    1 
 replay/replay-events.c   |   49 ++++
 replay/replay-internal.h |   21 ++
 replay/replay-usb.c      |  188 ++++++++++++++++
 replay/replay.c          |   29 +++
 replay/replay.h          |   30 +++
 8 files changed, 765 insertions(+), 183 deletions(-)
 create mode 100755 include/hw/host-libusb.h
 create mode 100755 replay/replay-usb.c
hw/usb/host-libusb.c +343 -182
index c189147..5283c64 100644
@@ -40,98 +40,15 @@
 #include "monitor/monitor.h"
 #include "sysemu/sysemu.h"
 #include "trace.h"
+#include "qemu/log.h"
+#include "replay/replay.h"
 
 #include "hw/usb.h"
+#include "hw/host-libusb.h"
 
 /* ------------------------------------------------------------------------ */
 
-#define TYPE_USB_HOST_DEVICE "usb-host"
-#define USB_HOST_DEVICE(obj) \
-     OBJECT_CHECK(USBHostDevice, (obj), TYPE_USB_HOST_DEVICE)
-
-typedef struct USBHostDevice USBHostDevice;
-typedef struct USBHostRequest USBHostRequest;
-typedef struct USBHostIsoXfer USBHostIsoXfer;
-typedef struct USBHostIsoRing USBHostIsoRing;
-
-struct USBAutoFilter {
-    uint32_t bus_num;
-    uint32_t addr;
-    char     *port;
-    uint32_t vendor_id;
-    uint32_t product_id;
-};
-
-enum USBHostDeviceOptions {
-    USB_HOST_OPT_PIPELINE,
-};
-
-struct USBHostDevice {
-    USBDevice parent_obj;
-
-    /* properties */
-    struct USBAutoFilter             match;
-    int32_t                          bootindex;
-    uint32_t                         iso_urb_count;
-    uint32_t                         iso_urb_frames;
-    uint32_t                         options;
-    uint32_t                         loglevel;
-
-    /* state */
-    QTAILQ_ENTRY(USBHostDevice)      next;
-    int                              seen, errcount;
-    int                              bus_num;
-    int                              addr;
-    char                             port[16];
-
-    libusb_device                    *dev;
-    libusb_device_handle             *dh;
-    struct libusb_device_descriptor  ddesc;
-
-    struct {
-        bool                         detached;
-        bool                         claimed;
-    } ifs[USB_MAX_INTERFACES];
-
-    /* callbacks & friends */
-    QEMUBH                           *bh_nodev;
-    QEMUBH                           *bh_postld;
-    Notifier                         exit;
-
-    /* request queues */
-    QTAILQ_HEAD(, USBHostRequest)    requests;
-    QTAILQ_HEAD(, USBHostIsoRing)    isorings;
-};
-
-struct USBHostRequest {
-    USBHostDevice                    *host;
-    USBPacket                        *p;
-    bool                             in;
-    struct libusb_transfer           *xfer;
-    unsigned char                    *buffer;
-    unsigned char                    *cbuf;
-    unsigned int                     clen;
     bool                             usb3ep0quirk;
-    QTAILQ_ENTRY(USBHostRequest)     next;
-};
-
-struct USBHostIsoXfer {
-    USBHostIsoRing                   *ring;
-    struct libusb_transfer           *xfer;
-    bool                             copy_complete;
-    unsigned int                     packet;
-    QTAILQ_ENTRY(USBHostIsoXfer)     next;
-};
-
-struct USBHostIsoRing {
-    USBHostDevice                    *host;
-    USBEndpoint                      *ep;
-    QTAILQ_HEAD(, USBHostIsoXfer)    unused;
-    QTAILQ_HEAD(, USBHostIsoXfer)    inflight;
-    QTAILQ_HEAD(, USBHostIsoXfer)    copy;
-    QTAILQ_ENTRY(USBHostIsoRing)     next;
-};
-
 static QTAILQ_HEAD(, USBHostDevice) hostdevs =
     QTAILQ_HEAD_INITIALIZER(hostdevs);
 
@@ -143,6 +60,24 @@ static void usb_host_attach_kernel(USBHostDevice *s);
 
 /* ------------------------------------------------------------------------ */
 
+typedef struct USBHostTimer {
+    QEMUTimer *timer;
+} USBHostTimer;
+
+USBHostTimer usb_auto_timer;
+static int submitted_xfers = 0;
+
+static const VMStateDescription vmstate_usb_host_timer = {
+    .name = "usb-host-timer",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .fields = (VMStateField[]) {
+        VMSTATE_TIMER(timer, USBHostTimer),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
+
 #define CONTROL_TIMEOUT  10000        /* 10 sec    */
 #define BULK_TIMEOUT         0        /* unlimited */
 #define INTR_TIMEOUT         0        /* unlimited */
@@ -214,6 +149,11 @@ static void usb_host_del_fd(int fd, void *user_data)
     qemu_set_fd_handler(fd, NULL, NULL, NULL);
 }
 
+bool usb_host_has_xfers(void)
+{
+    return submitted_xfers != 0;
+}
+
 static int usb_host_init(void)
 {
     const struct libusb_pollfd **poll;
@@ -226,18 +166,34 @@ static int usb_host_init(void)
     if (rc != 0) {
         return -1;
     }
-    libusb_set_debug(ctx, loglevel);
-
-    libusb_set_pollfd_notifiers(ctx, usb_host_add_fd,
+    if (replay_mode != REPLAY_PLAY) {
+        libusb_set_debug(ctx, loglevel);
+        libusb_set_pollfd_notifiers(ctx, usb_host_add_fd,
                                 usb_host_del_fd,
                                 ctx);
-    poll = libusb_get_pollfds(ctx);
-    if (poll) {
-        for (i = 0; poll[i] != NULL; i++) {
-            usb_host_add_fd(poll[i]->fd, poll[i]->events, ctx);
+
+        poll = libusb_get_pollfds(ctx);
+        if (poll) {
+            for (i = 0; poll[i] != NULL; i++) {
+                usb_host_add_fd(poll[i]->fd, poll[i]->events, ctx);
+            }
+        }
+        free(poll);
+    }
+
+    /* replay: changed timer for working with VM clock instead of RT */
+    if (!usb_auto_timer.timer) {
+        usb_auto_timer.timer = timer_new_ms(QEMU_CLOCK_VIRTUAL, usb_host_auto_check, NULL);
+        if (!usb_auto_timer.timer) {
+            return -1;
+        }
+        trace_usb_host_auto_scan_enabled();
+
+        if (replay_mode != REPLAY_NONE) {
+            vmstate_register(NULL, 0, &vmstate_usb_host_timer, &usb_auto_timer);
         }
     }
-    free(poll);
+
     return 0;
 }
 
@@ -252,6 +208,7 @@ static int usb_host_get_port(libusb_device *dev, char *port, size_t len)
 #else
     rc = libusb_get_port_path(ctx, dev, path, 7);
 #endif
+
     if (rc < 0) {
         return 0;
     }
@@ -307,6 +264,7 @@ static USBHostRequest *usb_host_req_alloc(USBHostDevice *s, USBPacket *p,
     r->host = s;
     r->p = p;
     r->in = in;
+    // allocate xfer's in REPLAY_PLAY too
     r->xfer = libusb_alloc_transfer(0);
     if (bufsize) {
         r->buffer = g_malloc(bufsize);
@@ -320,6 +278,7 @@ static void usb_host_req_free(USBHostRequest *r)
     if (r->host) {
         QTAILQ_REMOVE(&r->host->requests, r, next);
     }
+    // free xfer's in REPLAY_PLAY too
     libusb_free_transfer(r->xfer);
     g_free(r->buffer);
     g_free(r);
@@ -337,12 +296,13 @@ static USBHostRequest *usb_host_req_find(USBHostDevice *s, USBPacket *p)
     return NULL;
 }
 
-static void usb_host_req_complete_ctrl(struct libusb_transfer *xfer)
+void usb_host_req_complete_ctrl(struct libusb_transfer *xfer)
 {
     USBHostRequest *r = xfer->user_data;
     USBHostDevice  *s = r->host;
     bool disconnect = (xfer->status == LIBUSB_TRANSFER_NO_DEVICE);
 
+    --submitted_xfers;
     if (r->p == NULL) {
         goto out; /* request was canceled */
     }
@@ -354,7 +314,7 @@ static void usb_host_req_complete_ctrl(struct libusb_transfer *xfer)
 
         /* Fix up USB-3 ep0 maxpacket size to allow superspeed connected devices
          * to work redirected to a not superspeed capable hcd */
-        if (r->usb3ep0quirk && xfer->actual_length >= 18 &&
+        if (usb3ep0quirk && xfer->actual_length >= 18 &&
             r->cbuf[7] == 9) {
             r->cbuf[7] = 64;
         }
@@ -370,12 +330,13 @@ out:
     }
 }
 
-static void usb_host_req_complete_data(struct libusb_transfer *xfer)
+void usb_host_req_complete_data(struct libusb_transfer *xfer)
 {
     USBHostRequest *r = xfer->user_data;
     USBHostDevice  *s = r->host;
     bool disconnect = (xfer->status == LIBUSB_TRANSFER_NO_DEVICE);
 
+    --submitted_xfers;
     if (r->p == NULL) {
         goto out; /* request was canceled */
     }
@@ -419,17 +380,17 @@ static void usb_host_req_abort(USBHostRequest *r)
     QTAILQ_REMOVE(&r->host->requests, r, next);
     r->host = NULL;
 
-    if (inflight) {
+    if (inflight && replay_mode != REPLAY_PLAY) {
         libusb_cancel_transfer(r->xfer);
     }
 }
 
 /* ------------------------------------------------------------------------ */
 
-static void usb_host_req_complete_iso(struct libusb_transfer *transfer)
+void usb_host_req_complete_iso(struct libusb_transfer *transfer)
 {
     USBHostIsoXfer *xfer = transfer->user_data;
-
+    --submitted_xfers;
     if (!xfer) {
         /* USBHostIsoXfer released while inflight */
         g_free(transfer->buffer);
@@ -467,6 +428,7 @@ static USBHostIsoRing *usb_host_iso_alloc(USBHostDevice *s, USBEndpoint *ep)
     for (i = 0; i < s->iso_urb_count; i++) {
         xfer = g_new0(USBHostIsoXfer, 1);
         xfer->ring = ring;
+        // OK for replay too
         xfer->xfer = libusb_alloc_transfer(packets);
         xfer->xfer->dev_handle = s->dh;
         xfer->xfer->type = LIBUSB_TRANSFER_TYPE_ISOCHRONOUS;
@@ -475,7 +437,7 @@ static USBHostIsoRing *usb_host_iso_alloc(USBHostDevice *s, USBEndpoint *ep)
         if (ring->ep->pid == USB_TOKEN_IN) {
             xfer->xfer->endpoint |= USB_DIR_IN;
         }
-        xfer->xfer->callback = usb_host_req_complete_iso;
+        xfer->xfer->callback = replay_mode == REPLAY_NONE ? usb_host_req_complete_iso : replay_req_complete_iso;
         xfer->xfer->user_data = xfer;
 
         xfer->xfer->num_iso_packets = packets;
@@ -514,6 +476,7 @@ static void usb_host_iso_free_xfer(USBHostIsoXfer *xfer, bool inflight)
         xfer->xfer->user_data = NULL;
     } else {
         g_free(xfer->xfer->buffer);
+        // REPLAY_PLAY also allocates xfer
         libusb_free_transfer(xfer->xfer);
     }
     g_free(xfer);
@@ -549,12 +512,19 @@ static void usb_host_iso_free_all(USBHostDevice *s)
     }
 }
 
+
+unsigned char *usb_host_get_iso_packet_buffer(USBHostIsoXfer *xfer, int packet)
+{
+    // replay OK
+    return libusb_get_iso_packet_buffer_simple(xfer->xfer, packet);
+}
+
 static bool usb_host_iso_data_copy(USBHostIsoXfer *xfer, USBPacket *p)
 {
     unsigned int psize;
     unsigned char *buf;
 
-    buf = libusb_get_iso_packet_buffer_simple(xfer->xfer, xfer->packet);
+    buf = usb_host_get_iso_packet_buffer(xfer, xfer->packet);
     if (p->pid == USB_TOKEN_OUT) {
         psize = p->iov.size;
         if (psize > xfer->ring->ep->max_packet_size) {
@@ -600,7 +570,7 @@ static void usb_host_iso_data_in(USBHostDevice *s, USBPacket *p)
     while ((xfer = QTAILQ_FIRST(&ring->unused)) != NULL) {
         QTAILQ_REMOVE(&ring->unused, xfer, next);
         usb_host_iso_reset_xfer(xfer);
-        rc = libusb_submit_transfer(xfer->xfer);
+        REPLAY_DATA_INT(rc, libusb_submit_transfer(xfer->xfer));
         if (rc != 0) {
             usb_host_libusb_error("libusb_submit_transfer [iso]", rc);
             QTAILQ_INSERT_TAIL(&ring->unused, xfer, next);
@@ -608,6 +578,9 @@ static void usb_host_iso_data_in(USBHostDevice *s, USBPacket *p)
                 disconnect = true;
             }
             break;
+        } else {
+            replay_req_register_iso(xfer->xfer);
+            ++submitted_xfers;
         }
         if (QTAILQ_EMPTY(&ring->inflight)) {
             trace_usb_host_iso_start(s->bus_num, s->addr, p->ep->nr);
@@ -662,7 +635,7 @@ static void usb_host_iso_data_out(USBHostDevice *s, USBPacket *p)
     while ((xfer = QTAILQ_FIRST(&ring->copy)) != NULL &&
            xfer->copy_complete) {
         QTAILQ_REMOVE(&ring->copy, xfer, next);
-        rc = libusb_submit_transfer(xfer->xfer);
+        REPLAY_DATA_INT(rc, libusb_submit_transfer(xfer->xfer));
         if (rc != 0) {
             usb_host_libusb_error("libusb_submit_transfer [iso]", rc);
             QTAILQ_INSERT_TAIL(&ring->unused, xfer, next);
@@ -670,6 +643,9 @@ static void usb_host_iso_data_out(USBHostDevice *s, USBPacket *p)
                 disconnect = true;
             }
             break;
+        } else {
+            replay_req_register_iso(xfer->xfer);
+            ++submitted_xfers;
         }
         if (QTAILQ_EMPTY(&ring->inflight)) {
             trace_usb_host_iso_start(s->bus_num, s->addr, p->ep->nr);
@@ -697,6 +673,11 @@ static void usb_host_speed_compat(USBHostDevice *s)
     bool compat_full = true;
     uint8_t type;
     int rc, c, i, a, e;
+    bool ok = false;
+    if (replay_mode == REPLAY_PLAY) {
+        REPLAY_DATA_VAR(ok);
+        return;
+    }
 
     for (c = 0;; c++) {
         rc = libusb_get_config_descriptor(s->dev, c, &conf);
@@ -762,9 +743,12 @@ static void usb_host_ep_update(USBHostDevice *s)
         [USB_ENDPOINT_XFER_INT]     = "int",
     };
     USBDevice *udev = USB_DEVICE(s);
-    struct libusb_config_descriptor *conf;
-    const struct libusb_interface_descriptor *intf;
-    const struct libusb_endpoint_descriptor *endp;
+    struct libusb_config_descriptor dummy_conf;
+    struct libusb_interface_descriptor dummy_intf;
+    struct libusb_endpoint_descriptor dummy_endp;
+    struct libusb_config_descriptor *conf = &dummy_conf;
+    struct libusb_interface_descriptor *intf;
+    struct libusb_endpoint_descriptor *endp;
 #ifdef HAVE_STREAMS
     struct libusb_ss_endpoint_companion_descriptor *endp_ss_comp;
 #endif
@@ -773,21 +757,36 @@ static void usb_host_ep_update(USBHostDevice *s)
     int rc, i, e;
 
     usb_ep_reset(udev);
-    rc = libusb_get_active_config_descriptor(s->dev, &conf);
+    REPLAY_DATA_INT(rc, libusb_get_active_config_descriptor(s->dev, &conf));
     if (rc != 0) {
         return;
     }
     trace_usb_host_parse_config(s->bus_num, s->addr,
                                 conf->bConfigurationValue, true);
 
+    REPLAY_DATA_VAR(conf->bNumInterfaces);
     for (i = 0; i < conf->bNumInterfaces; i++) {
-        assert(udev->altsetting[i] < conf->interface[i].num_altsetting);
-        intf = &conf->interface[i].altsetting[udev->altsetting[i]];
+        if (replay_mode != REPLAY_PLAY) {
+            assert(udev->altsetting[i] < conf->interface[i].num_altsetting);
+            intf = (struct libusb_interface_descriptor *)&conf->interface[i].altsetting[udev->altsetting[i]];
+        } else {
+            intf = &dummy_intf;
+        }
+        REPLAY_DATA_VAR(intf->bInterfaceNumber);
+        REPLAY_DATA_VAR(intf->bAlternateSetting);
+        REPLAY_DATA_VAR(intf->bNumEndpoints);
         trace_usb_host_parse_interface(s->bus_num, s->addr,
                                        intf->bInterfaceNumber,
                                        intf->bAlternateSetting, true);
         for (e = 0; e < intf->bNumEndpoints; e++) {
-            endp = &intf->endpoint[e];
+            if (replay_mode != REPLAY_PLAY) {
+                endp = (struct libusb_endpoint_descriptor *)&intf->endpoint[e];
+            } else {
+                endp = &dummy_endp;
+            }
+            REPLAY_DATA_VAR(endp->bEndpointAddress);
+            REPLAY_DATA_VAR(endp->bmAttributes);
+            REPLAY_DATA_VAR(endp->wMaxPacketSize);
 
             devep = endp->bEndpointAddress;
             pid = (devep & USB_DIR_IN) ? USB_TOKEN_IN : USB_TOKEN_OUT;
@@ -825,45 +824,76 @@ static void usb_host_ep_update(USBHostDevice *s)
         }
     }
 
-    libusb_free_config_descriptor(conf);
+    if (replay_mode != REPLAY_PLAY) {
+        libusb_free_config_descriptor(conf);
+    }
 }
 
 static int usb_host_open(USBHostDevice *s, libusb_device *dev)
 {
     USBDevice *udev = USB_DEVICE(s);
-    int bus_num = libusb_get_bus_number(dev);
-    int addr    = libusb_get_device_address(dev);
+    int bus_num;
+    REPLAY_DATA_INT(bus_num, libusb_get_bus_number(dev));
+    int addr;
+    REPLAY_DATA_INT(addr, libusb_get_device_address(dev));
     int rc;
+    int speed;
 
     trace_usb_host_open_started(bus_num, addr);
 
-    if (s->dh != NULL) {
+    if (s->is_open) {
         goto fail;
     }
-    rc = libusb_open(dev, &s->dh);
+    REPLAY_DATA_INT(rc, libusb_open(dev, &s->dh));
     if (rc != 0) {
         goto fail;
     }
+    if (replay_mode == REPLAY_PLAY) {
+        // invalid non-NULL pointer for checking conditions
+        s->dh = (void*)0xbad;
+    }
+    s->is_open = true;
 
+    usb_host_detach_kernel(s);
+    if (replay_mode != REPLAY_PLAY) {
+        libusb_get_device_descriptor(dev, &s->ddesc);
+    }
+    // TODO: all fields are read and written as int
+    REPLAY_DATA_VAR(s->ddesc.bLength);
+    REPLAY_DATA_VAR(s->ddesc.bDescriptorType);
+    REPLAY_DATA_VAR(s->ddesc.bcdUSB);
+    REPLAY_DATA_VAR(s->ddesc.bDeviceClass);
+    REPLAY_DATA_VAR(s->ddesc.bDeviceSubClass);
+    REPLAY_DATA_VAR(s->ddesc.bDeviceProtocol);
+    REPLAY_DATA_VAR(s->ddesc.bMaxPacketSize0);
+    REPLAY_DATA_VAR(s->ddesc.idVendor);
+    REPLAY_DATA_VAR(s->ddesc.idProduct);
+    REPLAY_DATA_VAR(s->ddesc.bcdDevice);
+    REPLAY_DATA_VAR(s->ddesc.iManufacturer);
+    REPLAY_DATA_VAR(s->ddesc.iProduct);
+    REPLAY_DATA_VAR(s->ddesc.iSerialNumber);
+    REPLAY_DATA_VAR(s->ddesc.bNumConfigurations);
     s->dev     = dev;
     s->bus_num = bus_num;
     s->addr    = addr;
-
-    usb_host_detach_kernel(s);
-
-    libusb_get_device_descriptor(dev, &s->ddesc);
-    usb_host_get_port(s->dev, s->port, sizeof(s->port));
+    if (replay_mode != REPLAY_PLAY) {
+        usb_host_get_port(s->dev, s->port, sizeof(s->port));
+    }
+    replay_data_buffer((unsigned char *)s->port, sizeof(s->port));
 
     usb_ep_init(udev);
     usb_host_ep_update(s);
 
-    udev->speed     = speed_map[libusb_get_device_speed(dev)];
+    udev->speed     = speed_map[REPLAY_DATA_INT(speed, libusb_get_device_speed(dev))];
     usb_host_speed_compat(s);
 
     if (s->ddesc.iProduct) {
-        libusb_get_string_descriptor_ascii(s->dh, s->ddesc.iProduct,
+        if (replay_mode != REPLAY_PLAY) {
+            libusb_get_string_descriptor_ascii(s->dh, s->ddesc.iProduct,
                                            (unsigned char *)udev->product_desc,
                                            sizeof(udev->product_desc));
+        }
+        replay_data_buffer((unsigned char*)udev->product_desc, sizeof(udev->product_desc));
     } else {
         snprintf(udev->product_desc, sizeof(udev->product_desc),
                  "host:%d.%d", bus_num, addr);
@@ -879,10 +909,13 @@ static int usb_host_open(USBHostDevice *s, libusb_device *dev)
 
 fail:
     trace_usb_host_open_failure(bus_num, addr);
-    if (s->dh != NULL) {
-        libusb_close(s->dh);
+    if (s->is_open) {
+        if (replay_mode != REPLAY_PLAY) {
+            libusb_close(s->dh);
+        }
         s->dh = NULL;
         s->dev = NULL;
+        s->is_open = false;
     }
     return -1;
 }
@@ -896,11 +929,20 @@ static void usb_host_abort_xfers(USBHostDevice *s)
     }
 }
 
+static void usb_host_free_xfers(USBHostDevice *s)
+{
+    USBHostRequest *r;
+
+    while ((r = QTAILQ_FIRST(&s->requests)) != NULL) {
+        usb_host_req_free(r);
+    }
+}
+
 static int usb_host_close(USBHostDevice *s)
 {
     USBDevice *udev = USB_DEVICE(s);
 
-    if (s->dh == NULL) {
+    if (!s->is_open) {
         return -1;
     }
 
@@ -914,11 +956,20 @@ static int usb_host_close(USBHostDevice *s)
     }
 
     usb_host_release_interfaces(s);
-    libusb_reset_device(s->dh);
+    if (replay_mode != REPLAY_PLAY) {
+        libusb_reset_device(s->dh);
+    }
+
     usb_host_attach_kernel(s);
-    libusb_close(s->dh);
+    if (replay_mode != REPLAY_PLAY) {
+        libusb_close(s->dh);
+    }
+
+    //replay_unregister_usb_device(udev);
+
     s->dh = NULL;
     s->dev = NULL;
+    s->is_open = false;
 
     usb_host_auto_check(NULL);
     return 0;
@@ -933,7 +984,7 @@ static void usb_host_nodev_bh(void *opaque)
 static void usb_host_nodev(USBHostDevice *s)
 {
     if (!s->bh_nodev) {
-        s->bh_nodev = qemu_bh_new(usb_host_nodev_bh, s);
+        s->bh_nodev = qemu_bh_new_replay(usb_host_nodev_bh, s, s->bus_num);
     }
     qemu_bh_schedule(s->bh_nodev);
 }
@@ -942,7 +993,7 @@ static void usb_host_exit_notifier(struct Notifier *n, void *data)
 {
     USBHostDevice *s = container_of(n, USBHostDevice, exit);
 
-    if (s->dh) {
+    if (s->is_open) {
         usb_host_release_interfaces(s);
         usb_host_attach_kernel(s);
     }
@@ -1004,57 +1055,70 @@ static void usb_host_cancel_packet(USBDevice *udev, USBPacket *p)
     r = usb_host_req_find(s, p);
     if (r && r->p) {
         r->p = NULL; /* mark as dead */
-        libusb_cancel_transfer(r->xfer);
+        if (replay_mode != REPLAY_PLAY) {
+            libusb_cancel_transfer(r->xfer);
+        }
     }
 }
 
 static void usb_host_detach_kernel(USBHostDevice *s)
 {
-    struct libusb_config_descriptor *conf;
+    struct libusb_config_descriptor dummy_conf;
+    struct libusb_config_descriptor *conf = &dummy_conf;
     int rc, i;
 
-    rc = libusb_get_active_config_descriptor(s->dev, &conf);
+    REPLAY_DATA_INT(rc, libusb_get_active_config_descriptor(s->dev, &conf));
     if (rc != 0) {
         return;
     }
+    REPLAY_DATA_VAR(conf->bNumInterfaces);
     for (i = 0; i < conf->bNumInterfaces; i++) {
-        rc = libusb_kernel_driver_active(s->dh, i);
+        REPLAY_DATA_INT(rc, libusb_kernel_driver_active(s->dh, i));
         usb_host_libusb_error("libusb_kernel_driver_active", rc);
         if (rc != 1) {
             continue;
         }
         trace_usb_host_detach_kernel(s->bus_num, s->addr, i);
-        rc = libusb_detach_kernel_driver(s->dh, i);
+        REPLAY_DATA_INT(rc, libusb_detach_kernel_driver(s->dh, i));
         usb_host_libusb_error("libusb_detach_kernel_driver", rc);
         s->ifs[i].detached = true;
     }
-    libusb_free_config_descriptor(conf);
+    if (replay_mode != REPLAY_PLAY) {
+        libusb_free_config_descriptor(conf);
+    }
 }
 
 static void usb_host_attach_kernel(USBHostDevice *s)
 {
-    struct libusb_config_descriptor *conf;
+    struct libusb_config_descriptor dummy_conf;
+    struct libusb_config_descriptor *conf = &dummy_conf;
     int rc, i;
 
-    rc = libusb_get_active_config_descriptor(s->dev, &conf);
+    REPLAY_DATA_INT(rc, libusb_get_active_config_descriptor(s->dev, &conf));
     if (rc != 0) {
         return;
     }
-    for (i = 0; i < conf->bNumInterfaces; i++) {
+    REPLAY_DATA_VAR(conf->bNumInterfaces);
+    for (i = 0; i < conf->bNumInterfaces ; i++) {
         if (!s->ifs[i].detached) {
             continue;
         }
         trace_usb_host_attach_kernel(s->bus_num, s->addr, i);
-        libusb_attach_kernel_driver(s->dh, i);
+        if (replay_mode != REPLAY_PLAY) {
+            libusb_attach_kernel_driver(s->dh, i);
+        }
         s->ifs[i].detached = false;
     }
-    libusb_free_config_descriptor(conf);
+    if (replay_mode != REPLAY_PLAY) {
+        libusb_free_config_descriptor(conf);
+    }
 }
 
 static int usb_host_claim_interfaces(USBHostDevice *s, int configuration)
 {
     USBDevice *udev = USB_DEVICE(s);
-    struct libusb_config_descriptor *conf;
+    struct libusb_config_descriptor dummy_conf;
+    struct libusb_config_descriptor *conf = &dummy_conf;
     int rc, i;
 
     for (i = 0; i < USB_MAX_INTERFACES; i++) {
@@ -1065,7 +1129,7 @@ static int usb_host_claim_interfaces(USBHostDevice *s, int configuration)
 
     usb_host_detach_kernel(s);
 
-    rc = libusb_get_active_config_descriptor(s->dev, &conf);
+    REPLAY_DATA_INT(rc, libusb_get_active_config_descriptor(s->dev, &conf));
     if (rc != 0) {
         if (rc == LIBUSB_ERROR_NOT_FOUND) {
             /* address state - ignore */
@@ -1073,10 +1137,11 @@ static int usb_host_claim_interfaces(USBHostDevice *s, int configuration)
         }
         return USB_RET_STALL;
     }
+    REPLAY_DATA_VAR(conf->bNumInterfaces);
 
     for (i = 0; i < conf->bNumInterfaces; i++) {
         trace_usb_host_claim_interface(s->bus_num, s->addr, configuration, i);
-        rc = libusb_claim_interface(s->dh, i);
+        REPLAY_DATA_INT(rc, libusb_claim_interface(s->dh, i));
         usb_host_libusb_error("libusb_claim_interface", rc);
         if (rc != 0) {
             return USB_RET_STALL;
@@ -1087,7 +1152,9 @@ static int usb_host_claim_interfaces(USBHostDevice *s, int configuration)
     udev->ninterfaces   = conf->bNumInterfaces;
     udev->configuration = configuration;
 
-    libusb_free_config_descriptor(conf);
+    if (replay_mode != REPLAY_PLAY) {
+        libusb_free_config_descriptor(conf);
+    }
     return USB_RET_SUCCESS;
 }
 
@@ -1101,7 +1168,7 @@ static void usb_host_release_interfaces(USBHostDevice *s)
             continue;
         }
         trace_usb_host_release_interface(s->bus_num, s->addr, i);
-        rc = libusb_release_interface(s->dh, i);
+        REPLAY_DATA_INT(rc, libusb_release_interface(s->dh, i));
         usb_host_libusb_error("libusb_release_interface", rc);
         s->ifs[i].claimed = false;
     }
@@ -1122,7 +1189,7 @@ static void usb_host_set_config(USBHostDevice *s, int config, USBPacket *p)
     trace_usb_host_set_config(s->bus_num, s->addr, config);
 
     usb_host_release_interfaces(s);
-    rc = libusb_set_configuration(s->dh, config);
+    REPLAY_DATA_INT(rc, libusb_set_configuration(s->dh, config));
     if (rc != 0) {
         usb_host_libusb_error("libusb_set_configuration", rc);
         p->status = USB_RET_STALL;
@@ -1153,7 +1220,7 @@ static void usb_host_set_interface(USBHostDevice *s, int iface, int alt,
         return;
     }
 
-    rc = libusb_set_interface_alt_setting(s->dh, iface, alt);
+    REPLAY_DATA_INT(rc, libusb_set_interface_alt_setting(s->dh, iface, alt));
     if (rc != 0) {
         usb_host_libusb_error("libusb_set_interface_alt_setting", rc);
         p->status = USB_RET_STALL;
@@ -1177,7 +1244,7 @@ static void usb_host_handle_control(USBDevice *udev, USBPacket *p,
 
     trace_usb_host_req_control(s->bus_num, s->addr, p, request, value, index);
 
-    if (s->dh == NULL) {
+    if (!s->is_open) {
         p->status = USB_RET_NODEV;
         trace_usb_host_req_emulated(s->bus_num, s->addr, p, p->status);
         return;
@@ -1202,7 +1269,9 @@ static void usb_host_handle_control(USBDevice *udev, USBPacket *p,
     case EndpointOutRequest | USB_REQ_CLEAR_FEATURE:
         if (value == 0) { /* clear halt */
             int pid = (index & USB_DIR_IN) ? USB_TOKEN_IN : USB_TOKEN_OUT;
-            libusb_clear_halt(s->dh, index);
+            if (replay_mode != REPLAY_PLAY) {
+                libusb_clear_halt(s->dh, index);
+            }
             usb_ep_set_halted(udev, pid, index & 0x0f, 0);
             trace_usb_host_req_emulated(s->bus_num, s->addr, p, p->status);
             return;
@@ -1222,13 +1291,15 @@ static void usb_host_handle_control(USBDevice *udev, USBPacket *p,
     if (udev->speed == USB_SPEED_SUPER &&
         !((udev->port->speedmask & USB_SPEED_MASK_SUPER)) &&
         request == 0x8006 && value == 0x100 && index == 0) {
-        r->usb3ep0quirk = true;
+        usb3ep0quirk = true;
     }
 
+    // call in REPLAY_PLAY too - this function just copies data
     libusb_fill_control_transfer(r->xfer, s->dh, r->buffer,
-                                 usb_host_req_complete_ctrl, r,
-                                 CONTROL_TIMEOUT);
-    rc = libusb_submit_transfer(r->xfer);
+                                 replay_mode == REPLAY_NONE ? usb_host_req_complete_ctrl : replay_req_complete_ctrl,
+                                 r, CONTROL_TIMEOUT);
+
+    REPLAY_DATA_INT(rc, libusb_submit_transfer(r->xfer));
     if (rc != 0) {
         p->status = USB_RET_NODEV;
         trace_usb_host_req_complete(s->bus_num, s->addr, p,
@@ -1237,6 +1308,9 @@ static void usb_host_handle_control(USBDevice *udev, USBPacket *p,
             usb_host_nodev(s);
         }
         return;
+    } else {
+        replay_req_register_ctrl(r->xfer);
+        ++submitted_xfers;
     }
 
     p->status = USB_RET_ASYNC;
@@ -1258,7 +1332,7 @@ static void usb_host_handle_data(USBDevice *udev, USBPacket *p)
                             p->pid == USB_TOKEN_IN,
                             p->ep->nr, p->iov.size);
 
-    if (s->dh == NULL) {
+    if (!s->is_open) {
         p->status = USB_RET_NODEV;
         trace_usb_host_req_emulated(s->bus_num, s->addr, p, p->status);
         return;
@@ -1291,7 +1365,7 @@ static void usb_host_handle_data(USBDevice *udev, USBPacket *p)
         } else {
             libusb_fill_bulk_transfer(r->xfer, s->dh, ep,
                                       r->buffer, size,
-                                      usb_host_req_complete_data, r,
+                                      replay_mode == REPLAY_NONE ? usb_host_req_complete_data : replay_req_complete_data, r,
                                       BULK_TIMEOUT);
         }
         break;
@@ -1303,7 +1377,7 @@ static void usb_host_handle_data(USBDevice *udev, USBPacket *p)
         ep = p->ep->nr | (r->in ? USB_DIR_IN : 0);
         libusb_fill_interrupt_transfer(r->xfer, s->dh, ep,
                                        r->buffer, p->iov.size,
-                                       usb_host_req_complete_data, r,
+                                       replay_mode == REPLAY_NONE ? usb_host_req_complete_data : replay_req_complete_data, r,
                                        INTR_TIMEOUT);
         break;
     case USB_ENDPOINT_XFER_ISOC:
@@ -1322,7 +1396,7 @@ static void usb_host_handle_data(USBDevice *udev, USBPacket *p)
         return;
     }
 
-    rc = libusb_submit_transfer(r->xfer);
+    REPLAY_DATA_INT(rc, libusb_submit_transfer(r->xfer));
     if (rc != 0) {
         p->status = USB_RET_NODEV;
         trace_usb_host_req_complete(s->bus_num, s->addr, p,
@@ -1331,6 +1405,9 @@ static void usb_host_handle_data(USBDevice *udev, USBPacket *p)
             usb_host_nodev(s);
         }
         return;
+    } else {
+        replay_req_register_data(r->xfer);
+        ++submitted_xfers;
     }
 
     p->status = USB_RET_ASYNC;
@@ -1350,7 +1427,7 @@ static void usb_host_handle_reset(USBDevice *udev)
 
     trace_usb_host_reset(s->bus_num, s->addr);
 
-    rc = libusb_reset_device(s->dh);
+    REPLAY_DATA_INT(rc, libusb_reset_device(s->dh));
     if (rc != 0) {
         usb_host_nodev(s);
     }
@@ -1425,7 +1502,7 @@ static void usb_host_post_load_bh(void *opaque)
     USBHostDevice *dev = opaque;
     USBDevice *udev = USB_DEVICE(dev);
 
-    if (dev->dh != NULL) {
+    if (dev->is_open) {
         usb_host_close(dev);
     }
     if (udev->attached) {
@@ -1445,6 +1522,37 @@ static int usb_host_post_load(void *opaque, int version_id)
     return 0;
 }
 
+static int usb_host_post_load_replay(void *opaque, int version_id)
+{
+    USBHostDevice *dev = opaque;
+
+    usb_host_free_xfers(dev);
+    usb_host_iso_free_all(dev);
+    submitted_xfers = 0;
+    return 0;
+}
+
+static void usb_host_pre_save_replay(void *opaque)
+{
+    USBHostDevice *dev = opaque;
+
+    if (!QTAILQ_EMPTY(&dev->requests)) {
+        fprintf(stderr, "Replay: Saving VM state with non-empty USB requests queue\n");
+        exit(1);
+    }
+
+    if (!QTAILQ_EMPTY(&dev->isorings)) {
+        fprintf(stderr, "Replay: Saving VM state with non-empty USB iso rings queue\n");
+        exit(1);
+    }
+
+    // paranoidal checking
+    if (submitted_xfers != 0) {
+        fprintf(stderr, "Replay: Saving VM state with non-empty USB iso rings queue\n");
+        exit(1);
+    }
+}
+
 static const VMStateDescription vmstate_usb_host = {
     .name = "usb-host",
     .version_id = 1,
@@ -1456,6 +1564,51 @@ static const VMStateDescription vmstate_usb_host = {
     }
 };
 
+static const VMStateDescription vmstate_usb_host_interface = {
+    .name = "usb-host-interface",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .fields = (VMStateField[]) {
+        VMSTATE_BOOL(detached, USBHostInterface),
+        VMSTATE_BOOL(claimed, USBHostInterface),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
+static const VMStateDescription vmstate_usb_host_replay = {
+    .name = "usb-host",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .post_load = usb_host_post_load_replay,
+    .pre_save = usb_host_pre_save_replay,
+    .fields = (VMStateField[]) {
+        VMSTATE_USB_DEVICE(parent_obj, USBHostDevice),
+        VMSTATE_INT32(seen, USBHostDevice),
+        VMSTATE_INT32(errcount, USBHostDevice),
+        VMSTATE_INT32(bus_num, USBHostDevice),
+        VMSTATE_INT32(addr, USBHostDevice),
+        VMSTATE_INT32(bus_num, USBHostDevice),
+        VMSTATE_BOOL(is_open, USBHostDevice),
+        VMSTATE_CHAR_ARRAY(port, USBHostDevice, 16),
+        VMSTATE_UINT8(ddesc.bLength, USBHostDevice),
+        VMSTATE_UINT8(ddesc.bDescriptorType, USBHostDevice),
+        VMSTATE_UINT16(ddesc.bcdUSB, USBHostDevice),
+        VMSTATE_UINT8(ddesc.bDeviceClass, USBHostDevice),
+        VMSTATE_UINT8(ddesc.bDeviceSubClass, USBHostDevice),
+        VMSTATE_UINT8(ddesc.bDeviceProtocol, USBHostDevice),
+        VMSTATE_UINT8(ddesc.bMaxPacketSize0, USBHostDevice),
+        VMSTATE_UINT16(ddesc.idVendor, USBHostDevice),
+        VMSTATE_UINT16(ddesc.idProduct, USBHostDevice),
+        VMSTATE_UINT16(ddesc.bcdDevice, USBHostDevice),
+        VMSTATE_UINT8(ddesc.iManufacturer, USBHostDevice),
+        VMSTATE_UINT8(ddesc.iProduct, USBHostDevice),
+        VMSTATE_UINT8(ddesc.iSerialNumber, USBHostDevice),
+        VMSTATE_UINT8(ddesc.bNumConfigurations, USBHostDevice),
+        VMSTATE_STRUCT_ARRAY(ifs, USBHostDevice, USB_MAX_INTERFACES, 1, vmstate_usb_host_interface, USBHostInterface),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
 static Property usb_host_dev_properties[] = {
     DEFINE_PROP_UINT32("hostbus",  USBHostDevice, match.bus_num,    0),
     DEFINE_PROP_UINT32("hostaddr", USBHostDevice, match.addr,       0),
@@ -1487,7 +1640,10 @@ static void usb_host_class_initfn(ObjectClass *klass, void *data)
     uc->flush_ep_queue = usb_host_flush_ep_queue;
     uc->alloc_streams  = usb_host_alloc_streams;
     uc->free_streams   = usb_host_free_streams;
-    dc->vmsd = &vmstate_usb_host;
+    if (replay_mode == REPLAY_NONE)
+        dc->vmsd = &vmstate_usb_host;
+    else
+        dc->vmsd = &vmstate_usb_host_replay;
     dc->props = usb_host_dev_properties;
     set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
 }
@@ -1508,7 +1664,6 @@ type_init(usb_host_register_types)
 
 /* ------------------------------------------------------------------------ */
 
-static QEMUTimer *usb_auto_timer;
 static VMChangeStateEntry *usb_vmstate;
 
 static void usb_host_vm_state(void *unused, int running, RunState state)
@@ -1525,70 +1680,74 @@ static void usb_host_auto_check(void *unused)
     libusb_device **devs = NULL;
     struct libusb_device_descriptor ddesc;
     int unconnected = 0;
-    int i, n;
+    int i, n, tmp;
 
     if (usb_host_init() != 0) {
         return;
     }
 
     if (runstate_is_running()) {
-        n = libusb_get_device_list(ctx, &devs);
+        REPLAY_DATA_INT(n, libusb_get_device_list(ctx, &devs));
         for (i = 0; i < n; i++) {
-            if (libusb_get_device_descriptor(devs[i], &ddesc) != 0) {
+            if (REPLAY_DATA_INT(tmp, libusb_get_device_descriptor(devs[i], &ddesc)) != 0) {
                 continue;
             }
-            if (ddesc.bDeviceClass == LIBUSB_CLASS_HUB) {
+            if (REPLAY_DATA_INT(tmp, ddesc.bDeviceClass) == LIBUSB_CLASS_HUB) {
                 continue;
             }
             QTAILQ_FOREACH(s, &hostdevs, next) {
                 f = &s->match;
                 if (f->bus_num > 0 &&
-                    f->bus_num != libusb_get_bus_number(devs[i])) {
+                    f->bus_num != REPLAY_DATA_INT(tmp, libusb_get_bus_number(devs[i]))) {
                     continue;
                 }
                 if (f->addr > 0 &&
-                    f->addr != libusb_get_device_address(devs[i])) {
+                    f->addr != REPLAY_DATA_INT(tmp, libusb_get_device_address(devs[i]))) {
                     continue;
                 }
                 if (f->port != NULL) {
                     char port[16] = "-";
-                    usb_host_get_port(devs[i], port, sizeof(port));
+                    if (replay_mode != REPLAY_PLAY) {
+                        usb_host_get_port(devs[i], port, sizeof(port));
+                    }
+                    replay_data_buffer((unsigned char *)port, sizeof(port));
                     if (strcmp(f->port, port) != 0) {
                         continue;
                     }
                 }
                 if (f->vendor_id > 0 &&
-                    f->vendor_id != ddesc.idVendor) {
+                    f->vendor_id != REPLAY_DATA_INT(tmp, ddesc.idVendor)) {
                     continue;
                 }
                 if (f->product_id > 0 &&
-                    f->product_id != ddesc.idProduct) {
+                    f->product_id != REPLAY_DATA_INT(tmp, ddesc.idProduct)) {
                     continue;
                 }
-
                 /* We got a match */
                 s->seen++;
                 if (s->errcount >= 3) {
                     continue;
                 }
-                if (s->dh != NULL) {
+                if (s->is_open) {
                     continue;
                 }
-                if (usb_host_open(s, devs[i]) < 0) {
+                if (usb_host_open(s, replay_mode == REPLAY_PLAY ? (libusb_device*)0xbad : devs[i]) < 0) {
                     s->errcount++;
                     continue;
                 }
                 break;
             }
         }
-        libusb_free_device_list(devs, 1);
+        if (replay_mode != REPLAY_PLAY) {
+            libusb_free_device_list(devs, 1);
+        }
 
         QTAILQ_FOREACH(s, &hostdevs, next) {
-            if (s->dh == NULL) {
+            if (!s->is_open) {
                 unconnected++;
             }
             if (s->seen == 0) {
-                if (s->dh) {
+                if (s->is_open) {
                     usb_host_close(s);
                 }
                 s->errcount = 0;
@@ -1608,17 +1767,19 @@ static void usb_host_auto_check(void *unused)
 #endif
     }
 
-    if (!usb_vmstate) {
+    if (!usb_vmstate && replay_mode == REPLAY_NONE) {
         usb_vmstate = qemu_add_vm_change_state_handler(usb_host_vm_state, NULL);
     }
-    if (!usb_auto_timer) {
-        usb_auto_timer = timer_new_ms(QEMU_CLOCK_REALTIME, usb_host_auto_check, NULL);
+
+
+    /*if (!usb_auto_timer) {
+        usb_auto_timer.timer = timer_new_ms(QEMU_CLOCK_VIRTUAL, usb_host_auto_check, NULL);
         if (!usb_auto_timer) {
             return;
         }
         trace_usb_host_auto_scan_enabled();
-    }
-    timer_mod(usb_auto_timer, qemu_clock_get_ms(QEMU_CLOCK_REALTIME) + 2000);
+    }*/
+    timer_mod(usb_auto_timer.timer, qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL) + 2000);
 }
 
 void usb_host_info(Monitor *mon, const QDict *qdict)
include/hw/host-libusb.h new file +105 -0

105 lines not on this page · raw message

replay/Makefile.objs +1 -0
index c54e550..f2e3bdc 100755
@@ -6,3 +6,4 @@ obj-y += replay-input.o
 obj-y += replay-net.o
 obj-y += replay-audio.o
 obj-y += replay-char.o
+obj-y += replay-usb.o
replay/replay-events.c +49 -0

79 lines not on this page · raw message

replay/replay-internal.h +20 -1
index dfcd5fd..56b6ad9 100755
@@ -40,6 +40,8 @@
 #define EVENT_ASYNC_OPT             25
 /* for int data */
 #define EVENT_DATA_INT              26
+/* for data buffer */
+#define EVENT_DATA_BUFFER           27
 /* for instruction event */
 #define EVENT_INSTRUCTION           32
 /* for clock read/writes */
@@ -59,7 +61,10 @@
 #define REPLAY_ASYNC_EVENT_INPUT_SYNC  3
 #define REPLAY_ASYNC_EVENT_NETWORK     4
 #define REPLAY_ASYNC_EVENT_CHAR        5
-#define REPLAY_ASYNC_COUNT             6
+#define REPLAY_ASYNC_EVENT_USB_CTRL    6
+#define REPLAY_ASYNC_EVENT_USB_DATA    7
+#define REPLAY_ASYNC_EVENT_USB_ISO     8
+#define REPLAY_ASYNC_COUNT             9
 
 typedef struct ReplayState {
     /*! Cached clock values. */
@@ -195,4 +200,18 @@ void replay_event_char_save(void *opaque);
 /*! Reads char event from the file. */
 void *replay_event_char_read(void);
 
+/* USB events */
+
+void replay_event_usb_ctrl(void *opaque);
+void replay_event_usb_data(void *opaque);
+void replay_event_usb_iso(void *opaque);
+void replay_event_save_usb_xfer(void *opaque);
+void replay_event_save_usb_iso_xfer(void *opaque);
+void replay_event_read_usb_xfer(void *opaque);
+void replay_event_read_usb_iso_xfer(void *opaque);
+void replay_event_skip_usb_xfer(void);
+void replay_event_skip_usb_iso_xfer(void);
+bool replay_usb_has_xfers(void);
+void replay_add_usb_event(unsigned int event_id, uint64_t id, void *opaque);
+
 #endif
replay/replay-usb.c new file +188 -0

188 lines not on this page · raw message

replay/replay.c +29 -0
index 8e15b04..a89ba91 100755
@@ -101,6 +101,16 @@ static void replay_savevm(void *opaque)
     char name[128];
     uint64_t offset;
 
+#ifdef CONFIG_USB_LIBUSB
+    if (replay_usb_has_xfers()) {
+        /* Retry of saving VM state, when USB host controller is not ready.
+           We cannot save or interrupt non-finished transfers, so
+           just wait for finishing them later. */
+        timer_mod(save_timer, qemu_clock_get_ms(QEMU_CLOCK_REALTIME) + 1);
+        return;
+    }
+#endif
+
     offset = ftello64(replay_file);
 
     replay_save_instructions();
@@ -609,3 +619,22 @@ void replay_data_int(int *data)
         replay_put_dword(*data);
     }
 }
+
+void replay_data_buffer(unsigned char *data, size_t size)
+{
+    if (replay_file && replay_mode == REPLAY_PLAY) {
+        size_t read_size = 0;
+        skip_async_events_until(EVENT_DATA_BUFFER);
+        replay_get_array(data, &read_size);
+        replay_check_error();
+        if (read_size != size) {
+            fprintf(stderr, "Replay: read non-matching size of the buffer\n");
+            exit(1);
+        }
+        replay_has_unread_data = 0;
+    } else if (replay_file && replay_mode == REPLAY_SAVE) {
+        replay_save_instructions();
+        replay_put_event(EVENT_DATA_BUFFER);
+        replay_put_array(data, size);
+    }
+}
replay/replay.h +30 -0
index 533fa0a..aa15803 100755
@@ -24,6 +24,7 @@ struct QemuOpts;
 struct InputEvent;
 struct NetClientState;
 struct CharDriverState;
+struct libusb_transfer;
 
 /* replay modes */
 #define REPLAY_NONE 0
@@ -157,9 +158,38 @@ void replay_register_char_driver(struct CharDriverState *chr);
 /*! Saves write to char device event to the log */
 void replay_chr_be_write(struct CharDriverState *s, uint8_t *buf, int len);
 
+/* USB events */
+
+/*! Called in save mode when xfer for ctrl usb data is completed */
+void replay_req_complete_ctrl(struct libusb_transfer *xfer);
+/*! Called in play mode to register xfer 
+    to be completed by reading it from the log */
+void replay_req_register_ctrl(struct libusb_transfer *xfer);
+/*! Called in save mode when xfer for usb data request is completed */
+void replay_req_complete_data(struct libusb_transfer *xfer);
+/*! Called in play mode to register xfer 
+    to be completed by reading it from the log */
+void replay_req_register_data(struct libusb_transfer *xfer);
+/*! Called in save mode when iso xfer for usb data request is completed */
+void replay_req_complete_iso(struct libusb_transfer *xfer);
+/*! Called in play mode to register iso xfer 
+    to be completed by reading it from the log */
+void replay_req_register_iso(struct libusb_transfer *xfer);
+
 /* Other data */
 
 /*! Writes or reads integer value to/from replay log. */
 void replay_data_int(int *data);
+/*! Macro for using replay_data_int function */
+#define REPLAY_DATA_INT(var, value) \
+        ((replay_mode == REPLAY_NONE ? var = (value) \
+            : replay_mode == REPLAY_SAVE \
+                ? (var = (value), replay_data_int(&var), var) \
+                : /* PLAY */ (replay_data_int(&var), var)))
+/*! Writes or reads buffer to/from replay log. */
+void replay_data_buffer(unsigned char *data, size_t size);
+/*! Macro for using replay_data_buffer function */
+#define REPLAY_DATA_VAR(var) \
+                        replay_data_buffer((unsigned char*)&(var), sizeof(var))
 
 #endif
#65 lore raw

RFC PATCH v2 46/49 replay: replay_info command

In reply to Pavel Dovgalyuk

This patch adds support for replay_info monitor command. This command returns the information about replay execution (replay mode and current step).

---
 hmp-commands.hx      |   13 +++++++++++++
 monitor.c            |   17 +++++++++++++++++
 qapi-schema.json     |   27 +++++++++++++++++++++++++++
 qmp-commands.hx      |   14 ++++++++++++++
 replay/Makefile.objs |    1 +
 replay/replay-qmp.c  |   29 +++++++++++++++++++++++++++++
 replay/replay.c      |   25 +++++++++++++++++++++++++
 replay/replay.h      |    4 ++++
 8 files changed, 130 insertions(+), 0 deletions(-)
 create mode 100755 replay/replay-qmp.c
hmp-commands.hx +13 -0
index d0943b1..19174f1 100644
@@ -1788,6 +1788,19 @@ STEXI
 show available trace events and their state
 ETEXI
 
+    {
+        .name       = "replay_info",
+        .args_type  = "",
+        .params     = "",
+        .help       = "show replay info",
+        .mhandler.cmd = do_replay_info,
+    },
+
+STEXI
+@item replay_info
+Shows information about replay process.
+ETEXI
+
 STEXI
 @end table
 ETEXI
monitor.c +17 -0
index 5bc70a6..f336b91 100644
@@ -73,6 +73,7 @@
 #include "block/qapi.h"
 #include "qapi/qmp-event.h"
 #include "qapi-event.h"
+#include "replay/replay.h"
 
 /* for pic/irq_info */
 #if defined(TARGET_SPARC)
@@ -1173,6 +1174,22 @@ static void do_watchdog_action(Monitor *mon, const QDict *qdict)
     }
 }
 
+static void do_replay_info(Monitor *mon, const QDict *qdict)
+{
+    switch (replay_mode) {
+    case REPLAY_NONE:
+        monitor_printf(mon, "Replay is not enabled\n");
+        break;
+    default:
+        monitor_printf(mon, "Replay mode: %s ", replay_get_mode_name());
+        if (replay_mode == REPLAY_PLAY) {
+            monitor_printf(mon, "(%s)", replay_get_play_submode_name());
+        }
+        monitor_printf(mon, "\n\tcurrent step: %" PRId64 "\n", replay_get_current_step());
+        break;
+    }
+}
+
 static void monitor_printc(Monitor *mon, int c)
 {
     monitor_printf(mon, "'");
qapi-schema.json +27 -0
index fee541a..c45f795 100644
@@ -3491,3 +3491,30 @@
 # Since: 2.1
 ##
 { 'command': 'rtc-reset-reinjection' }
+
+##
+# @ReplayInfo:
+#
+# Information about replay process
+#
+# @mode: replay mode (none, play, record)
+#
+# @submode: play submode
+#
+# @step: current step of record or play
+#
+# Since:  2.2
+##
+{ 'type': 'ReplayInfo',
+  'data': {'mode': 'str', 'submode': 'str', 'step': 'uint64'} }
+
+##
+# @replay_info
+#
+# Query the status of replay engine
+#
+# Returns: @ReplayInfo reflecting the status
+#
+# Since:  2.2
+##
+{ 'command': 'replay_info', 'returns': 'ReplayInfo' }
qmp-commands.hx +14 -0
index 4be4765..d475633 100644
@@ -3755,3 +3755,17 @@ Example:
 <- { "return": {} }
 
 EQMP
+
+    {
+        .name       = "replay_info",
+        .args_type  = "",
+        .mhandler.cmd_new = qmp_marshal_input_replay_info,
+    },
+
+SQMP
+replay_info
+-----------
+
+Shows information about replay process.
+
+EQMP
replay/Makefile.objs +1 -0
index f2e3bdc..1bc9621 100755
@@ -7,3 +7,4 @@ obj-y += replay-net.o
 obj-y += replay-audio.o
 obj-y += replay-char.o
 obj-y += replay-usb.o
+obj-y += replay-qmp.o
replay/replay-qmp.c new file +29 -0
new file mode 100755
index 0000000..966bd4d
@@ -0,0 +1,29 @@
+/*
+ * replay-qmp.c
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "sysemu/sysemu.h"
+#include "qmp-commands.h"
+#include "qapi/qmp/qobject.h"
+#include "qapi/qmp-input-visitor.h"
+#include "replay/replay.h"
+#include "replay/replay-internal.h"
+
+ReplayInfo *qmp_replay_info(Error **errp)
+{
+    ReplayInfo *info = g_malloc0(sizeof(*info));
+
+    info->mode = g_strdup(replay_get_mode_name());
+    info->submode = g_strdup(replay_get_play_submode_name());
+    info->step = replay_get_current_step();
+
+    return info;
+}
replay/replay.c +25 -0
index a89ba91..f711c26 100755
@@ -638,3 +638,28 @@ void replay_data_buffer(unsigned char *data, size_t size)
         replay_put_array(data, size);
     }
 }
+
+const char *replay_get_mode_name(void)
+{
+    switch (replay_mode) {
+    case REPLAY_NONE:
+        return "none";
+    case REPLAY_PLAY:
+        return "replay";
+    case REPLAY_SAVE:
+        return "record";
+    default:
+        return "unknown";
+    }
+}
+
+const char *replay_get_play_submode_name(void)
+{
+    switch (play_submode) {
+    case REPLAY_PLAY_NORMAL:
+        return "normal";
+    case REPLAY_PLAY_UNKNOWN:
+    default:
+        return "unknown";
+    }
+}
replay/replay.h +4 -0
index aa15803..ee0460f 100755
@@ -52,6 +52,10 @@ extern int replay_icount;
 
 /*! Returns replay play submode */
 int replay_get_play_submode(void);
+/*! Returns current mode name */
+const char *replay_get_mode_name(void);
+/*! Returns replay-play submode name */
+const char *replay_get_play_submode_name(void);
 
 /* Replay process control functions */
 
#66 lore raw

Re: [Qemu-devel] [RFC PATCH v2 46/49] replay: replay_info command

Quoted 10 lines · Pavel Dovgalyuk

This patch adds support for replay_info monitor command. This command returns the information about replay execution (replay mode and current step).

---

+++ b/qapi-schema.json

qapi-schema.json +0 -0
@@ -3491,3 +3491,30 @@
 # Since: 2.1
 ##

qapi-schema.json line 3502

Quoted 10 lines
qapi-schema.json +9 -0
 { 'command': 'rtc-reset-reinjection' }
+
+##
+# @ReplayInfo:
+#
+# Information about replay process
+#
+# @mode: replay mode (none, play, record)
+#
+# @submode: play submode

What are the valid submodes? Does the set of submodes differ according to the main mode? If so, is it better to use a flat union, where the subtype is strongly typed to the correct limited possibilities according to the mode?

qapi-schema.json line 3509

Quoted 7 lines
qapi-schema.json +7 -0
+#
+# @step: current step of record or play
+#
+# Since:  2.2
+##
+{ 'type': 'ReplayInfo',
+  'data': {'mode': 'str', 'submode': 'str', 'step': 'uint64'} }

Using 'str' to open-code a finite set of strings is not nice. Please add:
{ 'enum': 'ReplayMode', 'data': [ 'none', 'play', 'record' ] }
then use 'mode': 'ReplayMode'.

Quoted 8 lines
qmp-commands.hx +4 -0
+++ b/qmp-commands.hx
@@ -3755,3 +3755,17 @@ Example:
 <- { "return": {} }
 
 EQMP
+
+    {
+        .name       = "replay_info",

qmp-commands.hx Example: · line 3770

Quoted 10 lines
qmp-commands.hx +10 -0
+        .args_type  = "",
+        .mhandler.cmd_new = qmp_marshal_input_replay_info,
+    },
+
+SQMP
+replay_info
+-----------
+
+Shows information about replay process.
+

No example?

Quoted 5 lines
replay/replay.c +5 -0
+
+const char *replay_get_mode_name(void)
+{
+    switch (replay_mode) {
+    case REPLAY_NONE:

replay/replay.c void replay_data_buffer(unsigned char *data, size_t size) · line 655

Quoted 10 lines
replay/replay.c +10 -0
+        return "none";
+    case REPLAY_PLAY:
+        return "replay";
+    case REPLAY_SAVE:
+        return "record";
+    default:
+        return "unknown";
+    }
+}
+

No need to open-code this. If you properly set up an enum type in the .json file, then you get the conversions between enum values and strings for free from the generated QAPI framework.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#67 lore raw

Re: [Qemu-devel] [RFC PATCH v2 46/49] replay: replay_info command

In reply to Pavel Dovgalyuk
Quoted 5 lines · Pavel Dovgalyuk

This patch adds support for replay_info monitor command. This command returns the information about replay execution (replay mode and current step).

---

Oh, missed one.

Review

Quoted 4 lines

+++ b/qapi-schema.json

(quoted diff) +2 -0
+##
+# @replay_info

Please name this replay-info. HMP still uses '_', but QMP should prefer '-'.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#68 lore raw

RFC PATCH v2 47/49 replay: replay_break command

In reply to Pavel Dovgalyuk

This patch adds support for replay_break monitor command. This command sets the step (measured in executed instructions) where replay should be stopped.

---
 hmp-commands.hx     |   14 ++++++++++++++
 monitor.c           |   15 +++++++++++++++
 qapi-schema.json    |   11 +++++++++++
 qmp-commands.hx     |   23 +++++++++++++++++++++++
 replay/replay-qmp.c |   13 +++++++++++++
 replay/replay.c     |   24 +++++++++++++++++++++++-
 replay/replay.h     |    2 ++
 7 files changed, 101 insertions(+), 1 deletions(-)
hmp-commands.hx +14 -0
index 19174f1..3eaa80e 100644
@@ -1801,6 +1801,20 @@ STEXI
 Shows information about replay process.
 ETEXI
 
+    {
+        .name       = "replay_break",
+        .args_type  = "step:l",
+        .params     = "step",
+        .help       = "stop replaying at the specified replay step",
+        .mhandler.cmd = do_replay_break,
+    },
+
+STEXI
+@item replay_break @var{step}
+Stops replaying at the specified @var{step}.
+
+ETEXI
+
 STEXI
 @end table
 ETEXI
monitor.c +15 -0
index f336b91..4710a59 100644
@@ -1190,6 +1190,21 @@ static void do_replay_info(Monitor *mon, const QDict *qdict)
     }
 }
 
+static void do_replay_break(Monitor *mon, const QDict *qdict)
+{
+    if (replay_mode == REPLAY_PLAY) {
+        uint64_t step = qdict_get_int(qdict, "step");
+        if (step >= replay_get_current_step()) {
+            monitor_printf(mon, "Setting break at step: %" PRId64 "\n", step);
+            replay_set_break(step);
+        } else {
+            monitor_printf(mon, "Cannot stop on the preceding step.\n");
+        }
+    } else {
+        monitor_printf(mon, "You can stop at the specific step only in PLAY mode.\n");
+    }
+}
+
 static void monitor_printc(Monitor *mon, int c)
 {
     monitor_printf(mon, "'");
qapi-schema.json +11 -0
index c45f795..d0a4651 100644
@@ -3518,3 +3518,14 @@
 # Since:  2.2
 ##
 { 'command': 'replay_info', 'returns': 'ReplayInfo' }
+
+##
+# @replay_break
+#
+# Sets breakpoint at the specified step of replaying
+#
+# @step: step where breakpoint should be set
+#
+# Since:  2.2
+##
+{ 'command': 'replay_break', 'data': { 'step': 'uint64' } }
qmp-commands.hx +23 -0
index d475633..575db76 100644
@@ -3769,3 +3769,26 @@ replay_info
 Shows information about replay process.
 
 EQMP
+
+    {
+        .name       = "replay_break",
+        .args_type  = "step:l",
+        .mhandler.cmd_new = qmp_marshal_input_replay_break,
+    },
+
+SQMP
+replay_break
+------------
+
+Sets breakpoint at the specified replay step.
+
+Arguments:
+
+- "step": step where breakpoint should be set
+
+Example:
+
+-> { "execute": "replay_break", "arguments": { "step": 1024 } }
+<- { "return": {} }
+
+EQMP
replay/replay-qmp.c +13 -0
index 966bd4d..a9f30da 100755
@@ -27,3 +27,16 @@ ReplayInfo *qmp_replay_info(Error **errp)
 
     return info;
 }
+
+void qmp_replay_break(uint64_t step, Error **errp)
+{
+    if (replay_mode == REPLAY_PLAY) {
+        if (step >= replay_get_current_step()) {
+            replay_set_break(step);
+        } else {
+            error_setg(errp, "Cannot stop on the preceding step");
+        }
+    } else {
+        error_setg(errp, "replay_break can be used only in PLAY mode");
+    }
+}
replay/replay.c +23 -1
index f711c26..d6949d6 100755
@@ -31,6 +31,8 @@ static char *replay_filename;
 char *replay_image_suffix;
 
 ReplayState replay_state;
+/*! Step for stopping execution at. */
+static uint64_t replay_break_step = -1;
 
 /* 
     Auto-saving for VM states data 
@@ -284,6 +286,19 @@ void replay_instruction(int process_events)
         } else if (replay_mode == REPLAY_PLAY) {
             skip_async_events_until(EVENT_INSTRUCTION);
             if (first_cpu->instructions_count >= 1) {
+                if (replay_get_current_step() == replay_break_step) {
+                    replay_break_step = -1;
+
+                    /* for stopping VM */
+                    if (play_submode == REPLAY_PLAY_NORMAL) {
+                        first_cpu->exception_index = EXCP_DEBUG;
+                        monitor_printf(default_mon, "Execution has stopped.\n");
+                        vm_stop(EXCP_DEBUG);
+                    }
+                    /* for breaking execution loop */
+                    cpu_exit(first_cpu);
+                    return;
+                }
                 ++replay_state.current_step;
                 --first_cpu->instructions_count;
                 if (first_cpu->instructions_count == 0) {
@@ -312,7 +327,8 @@ bool replay_has_async_request(void)
     }
 
     if (replay_mode == REPLAY_PLAY) {
-        if (skip_async_events(EVENT_ASYNC)) {
+        if (skip_async_events(EVENT_ASYNC)
+            || replay_get_current_step() == replay_break_step) {
             return true;
         }
 
@@ -470,6 +486,7 @@ static void replay_enable(const char *fname, int mode)
     replay_state.skipping_instruction = 0;
     replay_state.current_step = 0;
     current_saved_state = 0;
+    replay_break_step = -1;
 
     replay_net_init();
 
@@ -663,3 +680,8 @@ const char *replay_get_play_submode_name(void)
         return "unknown";
     }
 }
+
+void replay_set_break(uint64_t step)
+{
+    replay_break_step = step;
+}
replay/replay.h +2 -0
index ee0460f..9a20a3b 100755
@@ -65,6 +65,8 @@ void replay_configure(struct QemuOpts *opts, int mode);
 void replay_init_timer(void);
 /*! Closes replay log file and frees other resources. */
 void replay_finish(void);
+/*! Sets step where execution should be stopped. */
+void replay_set_break(uint64_t step);
 
 /* Processing the instructions */
 
#69 lore raw

Re: [Qemu-devel] [RFC PATCH v2 47/49] replay: replay_break command

Quoted 18 lines · Pavel Dovgalyuk

This patch adds support for replay_break monitor command. This command sets the step (measured in executed instructions) where replay should be stopped.

---
 hmp-commands.hx     |   14 ++++++++++++++
 monitor.c           |   15 +++++++++++++++
 qapi-schema.json    |   11 +++++++++++
 qmp-commands.hx     |   23 +++++++++++++++++++++++
 replay/replay-qmp.c |   13 +++++++++++++
 replay/replay.c     |   24 +++++++++++++++++++++++-
 replay/replay.h     |    2 ++
 7 files changed, 101 insertions(+), 1 deletions(-)

+++ b/qapi-schema.json

qapi-schema.json line 3523

Quoted 6 lines
qapi-schema.json +3 -0
@@ -3518,3 +3518,14 @@
 # Since:  2.2
 ##
 { 'command': 'replay_info', 'returns': 'ReplayInfo' }
+
+##
+# @replay_break

replay-break in QMP

qapi-schema.json line 3529

Quoted 6 lines
qapi-schema.json +6 -0
+#
+# Sets breakpoint at the specified step of replaying
+#
+# @step: step where breakpoint should be set
+#
+# Since:  2.2

Is the double space intentional?

This is a very high-level description - reading just this definition, I don't know what typical settings are, and whether changing the number to be larger or smaller makes a difference. Might be worth a bit more docs.

I hate write-only interfaces - how do I query what the current breakpoint step is?

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#70 lore raw

RFC PATCH v2 48/49 replay: replay_seek_step command

In reply to Pavel Dovgalyuk

This patch adds support for replay_seek_step monitor command. This command loads one of the snapshots and replays the execution until the specified step is met.

---
 hmp-commands.hx          |   14 ++++++++++++++
 monitor.c                |   17 +++++++++++++++++
 qapi-schema.json         |   11 +++++++++++
 qmp-commands.hx          |   23 +++++++++++++++++++++++
 replay/replay-internal.h |    3 +++
 replay/replay-qmp.c      |   15 +++++++++++++++
 replay/replay.c          |   43 +++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |    4 ++++
 8 files changed, 130 insertions(+), 0 deletions(-)
hmp-commands.hx +14 -0
index 3eaa80e..57ce5da 100644
@@ -1815,6 +1815,20 @@ Stops replaying at the specified @var{step}.
 
 ETEXI
 
+    {
+        .name       = "replay_seek",
+        .args_type  = "step:l",
+        .params     = "step",
+        .help       = "seeks the replay log to the specified step",
+        .mhandler.cmd = do_replay_seek,
+    },
+
+STEXI
+@item replay_seek @var{step}
+Seeks the replay log to the specified @var{step}.
+
+ETEXI
+
 STEXI
 @end table
 ETEXI
monitor.c +17 -0
index 4710a59..031decf 100644
@@ -1205,6 +1205,23 @@ static void do_replay_break(Monitor *mon, const QDict *qdict)
     }
 }
 
+static void do_replay_seek(Monitor *mon, const QDict *qdict)
+{
+    if (replay_mode == REPLAY_PLAY) {
+        uint64_t step = qdict_get_int(qdict, "step");
+        if (replay_seek_step(step)) {
+            monitor_printf(mon, "Seeking for step: %" PRId64 "\n", step);
+            if (!runstate_is_running()) {
+                vm_start();
+            }
+        } else {
+            monitor_printf(mon, "Cannot seek to the specified step.\n");
+        }
+    } else {
+        monitor_printf(mon, "You can go to the specific step only in PLAY mode.\n");
+    }
+}
+
 static void monitor_printc(Monitor *mon, int c)
 {
     monitor_printf(mon, "'");
qapi-schema.json +11 -0
index d0a4651..9d3fda3 100644
@@ -3529,3 +3529,14 @@
 # Since:  2.2
 ##
 { 'command': 'replay_break', 'data': { 'step': 'uint64' } }
+
+##
+# @replay_seek
+#
+# Seeks the replay log to the specified step
+#
+# @step: destination replay step 
+#
+# Since:  2.2
+##
+{ 'command': 'replay_seek', 'data': { 'step': 'uint64' } }
qmp-commands.hx +23 -0
index 575db76..a144327 100644
@@ -3792,3 +3792,26 @@ Example:
 <- { "return": {} }
 
 EQMP
+
+    {
+        .name       = "replay_seek",
+        .args_type  = "step:l",
+        .mhandler.cmd_new = qmp_marshal_input_replay_seek,
+    },
+
+SQMP
+replay_seek
+-----------
+
+Seeks the replay log to the specified step.
+
+Arguments:
+
+- "step": destination replay step
+
+Example:
+
+-> { "execute": "replay_seek", "arguments": { "step": 1024 } }
+<- { "return": {} }
+
+EQMP
replay/replay-internal.h +3 -0
index 56b6ad9..c7655e7 100755
@@ -135,6 +135,9 @@ void skip_async_events_until(unsigned int kind);
     If the parameter is -1, the clock value is read to the cache anyway. */
 void replay_read_next_clock(unsigned int kind);
 
+/*! Finds saved state info which is nearest before the specified step. */
+SavedStateInfo *find_nearest_state(uint64_t step);
+
 /* Asynchronous events queue */
 
 /*! Initializes events' processing internals */
replay/replay-qmp.c +15 -0
index a9f30da..5cb04fb 100755
@@ -40,3 +40,18 @@ void qmp_replay_break(uint64_t step, Error **errp)
         error_setg(errp, "replay_break can be used only in PLAY mode");
     }
 }
+
+void qmp_replay_seek(uint64_t step, Error **errp)
+{
+    if (replay_mode == REPLAY_PLAY) {
+        if (replay_seek_step(step)) {
+            if (!runstate_is_running()) {
+                vm_start();
+            }
+        } else {
+            error_setg(errp, "Cannot seek to the specified step");
+        }
+    } else {
+        error_setg(errp, "replay_seek can be used only in PLAY mode");
+    }
+}
replay/replay.c +43 -0

46 lines not on this page · raw message

replay/replay.h +4 -0
index 9a20a3b..356e99b 100755
@@ -67,6 +67,10 @@ void replay_init_timer(void);
 void replay_finish(void);
 /*! Sets step where execution should be stopped. */
 void replay_set_break(uint64_t step);
+/*! Seeks to the specified step. 
+    Loads VM state, if possible, and sets break to specified step.
+    Returns nonzero, when seeking was successful. */
+int replay_seek_step(uint64_t step);
 
 /* Processing the instructions */
 
#71 lore raw

Re: [Qemu-devel] [RFC PATCH v2 48/49] replay: replay_seek_step command

Quoted 19 lines · Pavel Dovgalyuk

This patch adds support for replay_seek_step monitor command. This command loads one of the snapshots and replays the execution until the specified step is met.

---
 hmp-commands.hx          |   14 ++++++++++++++
 monitor.c                |   17 +++++++++++++++++
 qapi-schema.json         |   11 +++++++++++
 qmp-commands.hx          |   23 +++++++++++++++++++++++
 replay/replay-internal.h |    3 +++
 replay/replay-qmp.c      |   15 +++++++++++++++
 replay/replay.c          |   43 +++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |    4 ++++
 8 files changed, 130 insertions(+), 0 deletions(-)

+++ b/qapi-schema.json

qapi-schema.json line 3534

Quoted 6 lines
qapi-schema.json +3 -0
@@ -3529,3 +3529,14 @@
 # Since:  2.2
 ##
 { 'command': 'replay_break', 'data': { 'step': 'uint64' } }
+
+##
+# @replay_seek

replay-seek

Quoted 7 lines
qapi-schema.json +7 -0
+#
+# Seeks the replay log to the specified step
+#
+# @step: destination replay step 
+#
+# Since:  2.2
+##
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#72 lore raw

RFC PATCH v2 49/49 gdbstub: reverse debugging

In reply to Pavel Dovgalyuk

This patch introduces support of reverse debugging through the gdb remote protocol. Patch adds reverse-stepi and reverse-continue commands support to qemu. Other reverse commands should also work, because they reuse these ones.

---
 exec.c                      |    7 ++
 gdbstub.c                   |   79 ++++++++++++++++++-----
 replay/Makefile.objs        |    1 
 replay/replay-debug.c       |  148 +++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h    |    4 +
 replay/replay.c             |    5 +
 replay/replay.h             |   13 ++++
 target-arm/helper.h         |    1 
 target-arm/replay_helper.c  |    5 +
 target-arm/translate.c      |   14 +++-
 target-i386/helper.h        |    1 
 target-i386/replay_helper.c |    5 +
 target-i386/translate.c     |   12 ++-
 13 files changed, 269 insertions(+), 26 deletions(-)
 create mode 100755 replay/replay-debug.c
exec.c +7 -0
index 050e0a8..8255d96 100644
@@ -1613,6 +1613,13 @@ static void check_watchpoint(int offset, int len_mask, int flags)
     QTAILQ_FOREACH(wp, &cpu->watchpoints, entry) {
         if ((vaddr == (wp->vaddr & len_mask) ||
              (vaddr & wp->len_mask) == wp->vaddr) && (wp->flags & flags)) {
+            /* Don't actually process a watchpoint, it will be processed,
+               when reverse execution stops. */
+            if (replay_get_play_submode() == REPLAY_PLAY_REVERSE) {
+                wp->flags &= ~BP_WATCHPOINT_HIT;
+                replay_reverse_breakpoint();
+                continue;
+            }
             wp->flags |= BP_WATCHPOINT_HIT;
             if (!cpu->watchpoint_hit) {
                 cpu->watchpoint_hit = wp;
gdbstub.c +62 -17

130 lines not on this page · raw message

replay/Makefile.objs +1 -0
index 1bc9621..af3baa4 100755
@@ -8,3 +8,4 @@ obj-y += replay-audio.o
 obj-y += replay-char.o
 obj-y += replay-usb.o
 obj-y += replay-qmp.o
+obj-y += replay-debug.o
replay/replay-debug.c new file +148 -0

148 lines not on this page · raw message

replay/replay-internal.h +4 -0
index c7655e7..e53913b 100755
@@ -88,7 +88,11 @@ struct SavedStateInfo {
 };
 /*! Reference to the saved state */
 typedef struct SavedStateInfo SavedStateInfo;
+/* List of the saved states information */
+extern SavedStateInfo *saved_states;
 
+/*! Stores current submode for PLAY mode */
+extern int play_submode;
 extern volatile unsigned int replay_data_kind;
 extern volatile unsigned int replay_has_unread_data;
 
replay/replay.c +5 -0
index e6b3e39..b4162f3 100755
@@ -294,6 +294,9 @@ void replay_instruction(int process_events)
                         first_cpu->exception_index = EXCP_DEBUG;
                         monitor_printf(default_mon, "Execution has stopped.\n");
                         vm_stop(EXCP_DEBUG);
+                    } else if (play_submode == REPLAY_PLAY_REVERSE) {
+                        cpu_handle_debug_exception(first_cpu->env_ptr);
+                        return;
                     }
                     /* for breaking execution loop */
                     cpu_exit(first_cpu);
@@ -675,6 +678,8 @@ const char *replay_get_play_submode_name(void)
     switch (play_submode) {
     case REPLAY_PLAY_NORMAL:
         return "normal";
+    case REPLAY_PLAY_REVERSE:
+        return "reverse";
     case REPLAY_PLAY_UNKNOWN:
     default:
         return "unknown";
replay/replay.h +13 -0
index 356e99b..565af2f 100755
@@ -34,6 +34,8 @@ struct libusb_transfer;
 #define REPLAY_PLAY_UNKNOWN 0
 /* Normal log replaying */
 #define REPLAY_PLAY_NORMAL  1
+/* Replaying for reverse execution */
+#define REPLAY_PLAY_REVERSE 2
 
 /* replay clock kinds */
 /* rdtsc */
@@ -186,6 +188,17 @@ void replay_req_complete_iso(struct libusb_transfer *xfer);
     to be completed by reading it from the log */
 void replay_req_register_iso(struct libusb_transfer *xfer);
 
+/* Reverse debugging */
+
+/*! Initializes reverse continue execution.
+    Called by debugger module. */
+void replay_reverse_continue(void);
+/*! Initializes reverse step execution.
+    Called by debugger module. */
+void replay_reverse_step(void);
+/* Called instead of invoking breakpoint in reverse continue mode. */
+void replay_reverse_breakpoint(void);
+
 /* Other data */
 
 /*! Writes or reads integer value to/from replay log. */
target-arm/helper.h +1 -0
index 0233b64..802ff9d 100644
@@ -531,3 +531,4 @@ DEF_HELPER_FLAGS_2(neon_pmull_64_hi, TCG_CALL_NO_RWG_SE, i64, i64, i64)
 #endif
 
 DEF_HELPER_1(replay_instruction, i32, env)
+DEF_HELPER_0(reverse_breakpoint, void)
target-arm/replay_helper.c +5 -0
index 736cd51..d342326 100755
@@ -31,3 +31,8 @@ uint32_t helper_replay_instruction(CPUARMState *env)
     replay_instruction(timer);
     return timer;
 }
+
+void helper_reverse_breakpoint(void)
+{
+    replay_reverse_breakpoint();
+}
target-arm/translate.c +9 -5
index dd40998..bd04cd6 100644
@@ -11021,11 +11021,15 @@ static inline void gen_intermediate_code_internal(ARMCPU *cpu,
         if (unlikely(!QTAILQ_EMPTY(&cs->breakpoints))) {
             QTAILQ_FOREACH(bp, &cs->breakpoints, entry) {
                 if (bp->pc == dc->pc) {
-                    gen_exception_internal_insn(dc, 0, EXCP_DEBUG);
-                    /* Advance PC so that clearing the breakpoint will
-                       invalidate this TB.  */
-                    dc->pc += 2;
-                    goto done_generating;
+                    if (replay_get_play_submode() == REPLAY_PLAY_REVERSE) {
+                        gen_helper_reverse_breakpoint();
+                    } else {
+                        gen_exception_internal_insn(dc, 0, EXCP_DEBUG);
+                        /* Advance PC so that clearing the breakpoint will
+                           invalidate this TB.  */
+                        dc->pc += 2;
+                        goto done_generating;
+                    }
                 }
             }
         }
target-i386/helper.h +1 -0
index 058302b..9aacef1 100644
@@ -219,3 +219,4 @@ DEF_HELPER_3(rcrq, tl, env, tl, tl)
 #endif
 
 DEF_HELPER_1(replay_instruction, i32, env)
+DEF_HELPER_0(reverse_breakpoint, void)
target-i386/replay_helper.c +5 -0
index 8b82eee..675bc17 100755
@@ -31,3 +31,8 @@ uint32_t helper_replay_instruction(CPUX86State *env)
     replay_instruction(timer);
     return timer;
 }
+
+void helper_reverse_breakpoint(void)
+{
+    replay_reverse_breakpoint();
+}
target-i386/translate.c +8 -4
index 6ae3fc3..104a569 100644
@@ -2534,10 +2534,14 @@ static void gen_interrupt(DisasContext *s, int intno,
 
 static void gen_debug(DisasContext *s, target_ulong cur_eip)
 {
-    gen_update_cc_op(s);
-    gen_jmp_im(cur_eip);
-    gen_helper_debug(cpu_env);
-    s->is_jmp = DISAS_TB_JUMP;
+    if (replay_get_play_submode() == REPLAY_PLAY_REVERSE) {
+        gen_helper_reverse_breakpoint();
+    } else {
+        gen_update_cc_op(s);
+        gen_jmp_im(cur_eip);
+        gen_helper_debug(cpu_env);
+        s->is_jmp = DISAS_TB_JUMP;
+    }
 }
 
 /* generate a generic end of block. Trace exception is also generated
#73 lore raw
In reply to Pavel Dovgalyuk
Quoted 67 lines · Pavel Dovgalyuk

This set of patches is related to the reverse execution and deterministic replay of qemu execution Our implementation of deterministic replay can be used for deterministic and reverse debugging of guest code through gdb remote interface.

Execution recording writes non-deterministic events log, which can be later used for replaying the execution anywhere and for unlimited number of times. It also supports checkpointing for faster rewinding during reverse debugging. Execution replaying reads the log and replays all non-deterministic events including external input, hardware clocks, and interrupts.

Reverse execution has the following features:
* Deterministically replays whole system execution and all contents of the memory, state of the hadrware devices, clocks, and screen of the VM.
* Writes execution log into the file for latter replaying for multiple times on different machines.
* Supports i386, x86_64, and ARM hardware platforms.
* Performs deterministic replay of all operations with keyboard, mouse, network adapters, audio devices, serial interfaces, and physical USB devices connected to the emulator.
* Provides support for gdb reverse debugging commands like reverse-step and reverse-continue.
* Supports auto-checkpointing for convenient reverse debugging.

Usage of the record/replay:
* First, record the execution, by adding '-record fname=replay.bin' to the command line.
* Then you can replay it for the multiple times by using another command line option: '-replay fname=replay.bin'
* Virtual machine should have at least one virtual disk, which is used to store checkpoints. If you want to enable automatic checkpointing, simply add ',period=XX' to record options, where XX is the checkpointing period in seconds.
* Using of the network adapters in record/replay mode is possible with the following command-line options:
- '-net user' (or another host adapter) in record mode
- '-net replay' in replay mode. Every host network adapter should be replaced by 'replay' when replaying the execution.
* Reverse debugging can be used through gdb remote interface. reverse-stepi and reverse-continue commands are supported. Other reverse commands should also work, because they reuse these ones.
* Monitor is extended by the following commands:
- replay_info - prints information about replay mode and current step (number of instructions executed)
- replay_break - sets "breakpoint" at the specified instructions count.
- replay_seek - rewinds (using the checkpoints, if possible) to the specified step of replay log.

Paper with short description of deterministic replay implementation:
http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html

Modifications of qemu include:
* adding missed fields of the virtual devices' states to the vmstate structures to allow deterministic saving and restoring the VM state
* adding virtual clock-based timers to vmstate structures, because virtual clock is the part of the virtual machine state
* modification of block layer to support automatic creation of the overlay files to store the changes and snapshots while recording
* disabling of system reset while loading VM state to avoid generating of interrupts by reset handlers
* adding warpers for clock and time functions to save their return values in the log
* saving different asynchronous events (e.g. system shutdown) into the log
* synchronization of the bottom halves execution
* synchronization of the threads from thread pool
* recording/replaying user input (mouse and keyboard), input from virtual serial ports, incoming network packets, input from connected USB devices
* adding HMP/QMP commands to monitor for controlling replay execution

Quoted 5 lines

v2 changes:
* Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
* Added QMP versions of replay commands (as suggested by Eric Blake)
* Removed some optional features of replay to make patches cleaner
* Minor changes and code cleanup were made

Thanks for reshaping the patches.
We are a little busy just now, but I will review that as soon as possible.

Thanks,
Fred

Quoted 202 lines

---

Pavel Dovgalyuk (49):
acpi: accurate overflow check
integratorcp: adding vmstate for save/restore
pcspk: adding vmstate for save/restore
fdc: adding vmstate for save/restore
parallel: adding vmstate for save/restore
serial: fixing vmstate for save/restore
kvmapic: fixing loading vmstate
hpet: fixing saving and loading process
pckbd: adding new fields to vmstate
rtl8139: adding new fields to vmstate
piix: do not raise irq while loading vmstate
mc146818rtc: add missed field to vmstate
pl031: add missed field to vmstate
ide pci: reset status field before loading the vmstate
softmmu: fixing usage of cpu_st/ld* from helpers
target: save cpu state fields
target-i386: update fp status fix
migration: add vmstate for int8 and char arrays
replay: global variables and function stubs
block: add suffix parameter to bdrv_open functions
sysemu: system functions for replay
replay: internal functions for replay log
cpu: invent instruction count for accurate replay
target-arm: instructions counting code for replay
target-i386: instructions counting code for replay
replay: interrupts and exceptions
vga: do not use virtual clock for blinking cursor
replay: asynchronous events infrastructure
replay: recording and replaying clock ticks
replay: recording and replaying different timers
replay: shutdown event
replay: checkpoints
replay: bottom halves
replay: replay aio requests
replay: thread pool
pl031: vmstate in replay mode
replay: initialization and deinitialization
replay: command line options
replay: snapshotting the virtual machine
replay: recording of the user input
tap-win32: destroy the thread at exit
replay: network packets record/replay
replay: audio data record/replay
replay: serial port
replay: USB passthrough
replay: replay_info command
replay: replay_break command
replay: replay_seek_step command
gdbstub: reverse debugging

  Makefile.target                  |    1
  arch_init.c                      |    8
  async.c                          |   45 ++
  audio/audio.c                    |   14 +
  audio/audio_win_int.h            |    3
  audio/winwaveaudio.c             |  167 +++++++--
  block.c                          |  134 ++++++-
  block/blkdebug.c                 |    2
  block/blkverify.c                |    4
  block/cow.c                      |    2
  block/qcow.c                     |    2
  block/qcow2.c                    |   10 -
  block/qed.c                      |    2
  block/raw-posix.c                |    6
  block/raw-win32.c                |    4
  block/sheepdog.c                 |    4
  block/vmdk.c                     |    8
  block/vvfat.c                    |    2
  blockdev.c                       |   11 -
  cpu-exec.c                       |   34 +-
  cpus.c                           |   81 ++++
  dma-helpers.c                    |   10 -
  exec.c                           |   12 +
  gdbstub.c                        |   79 +++-
  hmp-commands.hx                  |   41 ++
  hw/acpi/core.c                   |    7
  hw/arm/integratorcp.c            |   38 ++
  hw/audio/pcspk.c                 |   19 +
  hw/block/fdc.c                   |   11 -
  hw/block/virtio-blk.c            |   10 -
  hw/char/parallel.c               |   22 +
  hw/char/serial.c                 |  115 ++++--
  hw/display/vga.c                 |    4
  hw/i386/kvmvapic.c               |   22 +
  hw/ide/ahci.c                    |    4
  hw/ide/atapi.c                   |    9
  hw/ide/core.c                    |   18 +
  hw/ide/pci.c                     |   10 +
  hw/input/pckbd.c                 |    3
  hw/intc/apic_common.c            |    5
  hw/net/rtl8139.c                 |    5
  hw/pci-host/piix.c               |   22 +
  hw/timer/arm_timer.c             |    2
  hw/timer/hpet.c                  |   13 -
  hw/timer/mc146818rtc.c           |    5
  hw/timer/pl031.c                 |   35 +-
  hw/usb/hcd-uhci.c                |    2
  hw/usb/host-libusb.c             |  525 ++++++++++++++++++---------
  include/block/aio.h              |   17 +
  include/block/block.h            |   21 +
  include/block/thread-pool.h      |    4
  include/exec/cpu-defs.h          |    1
  include/exec/cpu_ldst_template.h |   28 +
  include/exec/exec-all.h          |   31 ++
  include/hw/host-libusb.h         |  105 +++++
  include/migration/vmstate.h      |   13 +
  include/qemu-common.h            |    3
  include/qemu/main-loop.h         |    1
  include/qemu/timer.h             |   48 ++
  include/qom/cpu.h                |    4
  include/sysemu/char.h            |   25 +
  include/sysemu/cpus.h            |    1
  include/sysemu/sysemu.h          |    2
  include/ui/input.h               |    2
  main-loop.c                      |    5
  monitor.c                        |   49 +++
  net/Makefile.objs                |    1
  net/clients.h                    |    3
  net/dump.c                       |    6
  net/hub.c                        |    1
  net/net-replay.c                 |   66 +++
  net/net.c                        |    7
  net/slirp.c                      |   14 +
  net/socket.c                     |   35 ++
  net/tap-win32.c                  |   25 +
  net/tap.c                        |   23 +
  net/vde.c                        |   14 +
  qapi-schema.json                 |   62 +++
  qemu-char.c                      |   55 +++
  qemu-img.c                       |    6
  qemu-io-cmds.c                   |    2
  qemu-io.c                        |    4
  qemu-nbd.c                       |    2
  qemu-options.hx                  |   28 +
  qemu-timer.c                     |   45 ++
  qmp-commands.hx                  |   60 +++
  replay/Makefile.objs             |   11 +
  replay/replay-audio.c            |  228 ++++++++++++
  replay/replay-char.c             |   99 +++++
  replay/replay-debug.c            |  148 ++++++++
  replay/replay-events.c           |  381 ++++++++++++++++++++
  replay/replay-input.c            |  107 ++++++
  replay/replay-internal.c         |  159 ++++++++
  replay/replay-internal.h         |  224 ++++++++++++
  replay/replay-net.c              |  190 ++++++++++
  replay/replay-qmp.c              |   57 +++
  replay/replay-time.c             |  181 +++++++++
  replay/replay-usb.c              |  188 ++++++++++
  replay/replay.c                  |  735 ++++++++++++++++++++++++++++++++++++++
  replay/replay.h                  |  218 +++++++++++
  savevm.c                         |   32 +-
  slirp/slirp.c                    |    9
  softmmu_template.h               |   18 +
  stubs/Makefile.objs              |    1
  stubs/replay.c                   |   42 ++
  target-arm/Makefile.objs         |    1
  target-arm/helper.h              |    3
  target-arm/machine.c             |    5
  target-arm/replay_helper.c       |   38 ++
  target-arm/translate.c           |   62 +++
  target-i386/Makefile.objs        |    1
  target-i386/cpu.c                |    1
  target-i386/cpu.h                |    8
  target-i386/fpu_helper.c         |    5
  target-i386/helper.h             |    3
  target-i386/machine.c            |    7
  target-i386/replay_helper.c      |   38 ++
  target-i386/translate.c          |   91 ++++-
  tests/test-thread-pool.c         |    7
  thread-pool.c                    |   53 ++-
  trace-events                     |    2
  translate-all.c                  |   15 +
  ui/input.c                       |   79 +++-
  util/iov.c                       |    4
  vl.c                             |  108 +++++-
  vmstate.c                        |    6
  126 files changed, 5433 insertions(+), 543 deletions(-)

create mode 100755 include/hw/host-libusb.h
create mode 100755 net/net-replay.c
create mode 100755 replay/Makefile.objs
create mode 100755 replay/replay-audio.c
create mode 100755 replay/replay-char.c
create mode 100755 replay/replay-debug.c
create mode 100755 replay/replay-events.c
create mode 100755 replay/replay-input.c
create mode 100755 replay/replay-internal.c
create mode 100755 replay/replay-internal.h
create mode 100755 replay/replay-net.c
create mode 100755 replay/replay-qmp.c
create mode 100755 replay/replay-time.c
create mode 100755 replay/replay-usb.c
create mode 100755 replay/replay.c
create mode 100755 replay/replay.h
create mode 100755 stubs/replay.c
create mode 100755 target-arm/replay_helper.c
create mode 100755 target-i386/replay_helper.c

#74 lore raw
In reply to Pavel Dovgalyuk

Il 17/07/2014 13:01, Pavel Dovgalyuk ha scritto:

Quoted 5 lines

This set of patches is related to the reverse execution and deterministic replay of qemu execution Our implementation of deterministic replay can be used for deterministic and reverse debugging of guest code through gdb remote interface.

Quoted 7 lines

Execution recording writes non-deterministic events log, which can be later used for replaying the execution anywhere and for unlimited number of times. It also supports checkpointing for faster rewinding during reverse debugging. Execution replaying reads the log and replays all non-deterministic events including external input, hardware clocks, and interrupts.

From a first look:

- patches 1-14-15-17-41 are okay :)

- patches 2 to 13 probably should try to use subsections, so that VMs that do not use the devices try not to save the extra data and keep backwards migration compatibility (at least try to)

- patch 16 should also use subsections, and perhaps apply to all other CPUs too?

- patches 23-24-25 perhaps could try using icount, like Konrad's patch do?

- patch 27 makes sense but VIRTUAL is used to skip blinking when the VM is stopped

- the others I haven't yet looked at, but they look like something that would bitrot really, really fast. For patch 33, I think changing INSERT_HEAD to INSERT_TAIL would be just fine, and I wonder if it's the same for other patches here. How do you plan on testing them and keeping them up to date?

Paolo

Quoted 263 lines

Reverse execution has the following features:
* Deterministically replays whole system execution and all contents of the memory, state of the hadrware devices, clocks, and screen of the VM.
* Writes execution log into the file for latter replaying for multiple times on different machines.
* Supports i386, x86_64, and ARM hardware platforms.
* Performs deterministic replay of all operations with keyboard, mouse, network adapters, audio devices, serial interfaces, and physical USB devices connected to the emulator.
* Provides support for gdb reverse debugging commands like reverse-step and reverse-continue.
* Supports auto-checkpointing for convenient reverse debugging.

Usage of the record/replay:
* First, record the execution, by adding '-record fname=replay.bin' to the command line.
* Then you can replay it for the multiple times by using another command line option: '-replay fname=replay.bin'
* Virtual machine should have at least one virtual disk, which is used to store checkpoints. If you want to enable automatic checkpointing, simply add ',period=XX' to record options, where XX is the checkpointing period in seconds.
* Using of the network adapters in record/replay mode is possible with the following command-line options:
- '-net user' (or another host adapter) in record mode
- '-net replay' in replay mode. Every host network adapter should be replaced by 'replay' when replaying the execution.
* Reverse debugging can be used through gdb remote interface. reverse-stepi and reverse-continue commands are supported. Other reverse commands should also work, because they reuse these ones.
* Monitor is extended by the following commands:
- replay_info - prints information about replay mode and current step (number of instructions executed)
- replay_break - sets "breakpoint" at the specified instructions count.
- replay_seek - rewinds (using the checkpoints, if possible) to the specified step of replay log.

Paper with short description of deterministic replay implementation:
http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html

Modifications of qemu include:
* adding missed fields of the virtual devices' states to the vmstate structures to allow deterministic saving and restoring the VM state
* adding virtual clock-based timers to vmstate structures, because virtual clock is the part of the virtual machine state
* modification of block layer to support automatic creation of the overlay files to store the changes and snapshots while recording
* disabling of system reset while loading VM state to avoid generating of interrupts by reset handlers
* adding warpers for clock and time functions to save their return values in the log
* saving different asynchronous events (e.g. system shutdown) into the log
* synchronization of the bottom halves execution
* synchronization of the threads from thread pool
* recording/replaying user input (mouse and keyboard), input from virtual serial ports, incoming network packets, input from connected USB devices
* adding HMP/QMP commands to monitor for controlling replay execution

v2 changes:
* Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
* Added QMP versions of replay commands (as suggested by Eric Blake)
* Removed some optional features of replay to make patches cleaner
* Minor changes and code cleanup were made

---

Pavel Dovgalyuk (49):
acpi: accurate overflow check
integratorcp: adding vmstate for save/restore
pcspk: adding vmstate for save/restore
fdc: adding vmstate for save/restore
parallel: adding vmstate for save/restore
serial: fixing vmstate for save/restore
kvmapic: fixing loading vmstate
hpet: fixing saving and loading process
pckbd: adding new fields to vmstate
rtl8139: adding new fields to vmstate
piix: do not raise irq while loading vmstate
mc146818rtc: add missed field to vmstate
pl031: add missed field to vmstate
ide pci: reset status field before loading the vmstate
softmmu: fixing usage of cpu_st/ld* from helpers
target: save cpu state fields
target-i386: update fp status fix
migration: add vmstate for int8 and char arrays
replay: global variables and function stubs
block: add suffix parameter to bdrv_open functions
sysemu: system functions for replay
replay: internal functions for replay log
cpu: invent instruction count for accurate replay
target-arm: instructions counting code for replay
target-i386: instructions counting code for replay
replay: interrupts and exceptions
vga: do not use virtual clock for blinking cursor
replay: asynchronous events infrastructure
replay: recording and replaying clock ticks
replay: recording and replaying different timers
replay: shutdown event
replay: checkpoints
replay: bottom halves
replay: replay aio requests
replay: thread pool
pl031: vmstate in replay mode
replay: initialization and deinitialization
replay: command line options
replay: snapshotting the virtual machine
replay: recording of the user input
tap-win32: destroy the thread at exit
replay: network packets record/replay
replay: audio data record/replay
replay: serial port
replay: USB passthrough
replay: replay_info command
replay: replay_break command
replay: replay_seek_step command
gdbstub: reverse debugging

 Makefile.target                  |    1 
 arch_init.c                      |    8 
 async.c                          |   45 ++
 audio/audio.c                    |   14 +
 audio/audio_win_int.h            |    3 
 audio/winwaveaudio.c             |  167 +++++++--
 block.c                          |  134 ++++++-
 block/blkdebug.c                 |    2 
 block/blkverify.c                |    4 
 block/cow.c                      |    2 
 block/qcow.c                     |    2 
 block/qcow2.c                    |   10 -
 block/qed.c                      |    2 
 block/raw-posix.c                |    6 
 block/raw-win32.c                |    4 
 block/sheepdog.c                 |    4 
 block/vmdk.c                     |    8 
 block/vvfat.c                    |    2 
 blockdev.c                       |   11 -
 cpu-exec.c                       |   34 +-
 cpus.c                           |   81 ++++
 dma-helpers.c                    |   10 -
 exec.c                           |   12 +
 gdbstub.c                        |   79 +++-
 hmp-commands.hx                  |   41 ++
 hw/acpi/core.c                   |    7 
 hw/arm/integratorcp.c            |   38 ++
 hw/audio/pcspk.c                 |   19 +
 hw/block/fdc.c                   |   11 -
 hw/block/virtio-blk.c            |   10 -
 hw/char/parallel.c               |   22 +
 hw/char/serial.c                 |  115 ++++--
 hw/display/vga.c                 |    4 
 hw/i386/kvmvapic.c               |   22 +
 hw/ide/ahci.c                    |    4 
 hw/ide/atapi.c                   |    9 
 hw/ide/core.c                    |   18 +
 hw/ide/pci.c                     |   10 +
 hw/input/pckbd.c                 |    3 
 hw/intc/apic_common.c            |    5 
 hw/net/rtl8139.c                 |    5 
 hw/pci-host/piix.c               |   22 +
 hw/timer/arm_timer.c             |    2 
 hw/timer/hpet.c                  |   13 -
 hw/timer/mc146818rtc.c           |    5 
 hw/timer/pl031.c                 |   35 +-
 hw/usb/hcd-uhci.c                |    2 
 hw/usb/host-libusb.c             |  525 ++++++++++++++++++---------
 include/block/aio.h              |   17 +
 include/block/block.h            |   21 +
 include/block/thread-pool.h      |    4 
 include/exec/cpu-defs.h          |    1 
 include/exec/cpu_ldst_template.h |   28 +
 include/exec/exec-all.h          |   31 ++
 include/hw/host-libusb.h         |  105 +++++
 include/migration/vmstate.h      |   13 +
 include/qemu-common.h            |    3 
 include/qemu/main-loop.h         |    1 
 include/qemu/timer.h             |   48 ++
 include/qom/cpu.h                |    4 
 include/sysemu/char.h            |   25 +
 include/sysemu/cpus.h            |    1 
 include/sysemu/sysemu.h          |    2 
 include/ui/input.h               |    2 
 main-loop.c                      |    5 
 monitor.c                        |   49 +++
 net/Makefile.objs                |    1 
 net/clients.h                    |    3 
 net/dump.c                       |    6 
 net/hub.c                        |    1 
 net/net-replay.c                 |   66 +++
 net/net.c                        |    7 
 net/slirp.c                      |   14 +
 net/socket.c                     |   35 ++
 net/tap-win32.c                  |   25 +
 net/tap.c                        |   23 +
 net/vde.c                        |   14 +
 qapi-schema.json                 |   62 +++
 qemu-char.c                      |   55 +++
 qemu-img.c                       |    6 
 qemu-io-cmds.c                   |    2 
 qemu-io.c                        |    4 
 qemu-nbd.c                       |    2 
 qemu-options.hx                  |   28 +
 qemu-timer.c                     |   45 ++
 qmp-commands.hx                  |   60 +++
 replay/Makefile.objs             |   11 +
 replay/replay-audio.c            |  228 ++++++++++++
 replay/replay-char.c             |   99 +++++
 replay/replay-debug.c            |  148 ++++++++
 replay/replay-events.c           |  381 ++++++++++++++++++++
 replay/replay-input.c            |  107 ++++++
 replay/replay-internal.c         |  159 ++++++++
 replay/replay-internal.h         |  224 ++++++++++++
 replay/replay-net.c              |  190 ++++++++++
 replay/replay-qmp.c              |   57 +++
 replay/replay-time.c             |  181 +++++++++
 replay/replay-usb.c              |  188 ++++++++++
 replay/replay.c                  |  735 ++++++++++++++++++++++++++++++++++++++
 replay/replay.h                  |  218 +++++++++++
 savevm.c                         |   32 +-
 slirp/slirp.c                    |    9 
 softmmu_template.h               |   18 +
 stubs/Makefile.objs              |    1 
 stubs/replay.c                   |   42 ++
 target-arm/Makefile.objs         |    1 
 target-arm/helper.h              |    3 
 target-arm/machine.c             |    5 
 target-arm/replay_helper.c       |   38 ++
 target-arm/translate.c           |   62 +++
 target-i386/Makefile.objs        |    1 
 target-i386/cpu.c                |    1 
 target-i386/cpu.h                |    8 
 target-i386/fpu_helper.c         |    5 
 target-i386/helper.h             |    3 
 target-i386/machine.c            |    7 
 target-i386/replay_helper.c      |   38 ++
 target-i386/translate.c          |   91 ++++-
 tests/test-thread-pool.c         |    7 
 thread-pool.c                    |   53 ++-
 trace-events                     |    2 
 translate-all.c                  |   15 +
 ui/input.c                       |   79 +++-
 util/iov.c                       |    4 
 vl.c                             |  108 +++++-
 vmstate.c                        |    6 
 126 files changed, 5433 insertions(+), 543 deletions(-)
 create mode 100755 include/hw/host-libusb.h
 create mode 100755 net/net-replay.c
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay-audio.c
 create mode 100755 replay/replay-char.c
 create mode 100755 replay/replay-debug.c
 create mode 100755 replay/replay-events.c
 create mode 100755 replay/replay-input.c
 create mode 100755 replay/replay-internal.c
 create mode 100755 replay/replay-internal.h
 create mode 100755 replay/replay-net.c
 create mode 100755 replay/replay-qmp.c
 create mode 100755 replay/replay-time.c
 create mode 100755 replay/replay-usb.c
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c
 create mode 100755 target-arm/replay_helper.c
 create mode 100755 target-i386/replay_helper.c
#75 lore raw
Quoted 13 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini Il 17/07/2014 13:01, Pavel Dovgalyuk ha scritto:

Quoted 10 lines

This set of patches is related to the reverse execution and deterministic replay of qemu execution Our implementation of deterministic replay can be used for deterministic and reverse debugging of guest code through gdb remote interface.

Execution recording writes non-deterministic events log, which can be later used for replaying the execution anywhere and for unlimited number of times. It also supports checkpointing for faster rewinding during reverse debugging. Execution replaying reads the log and replays all non-deterministic events including external input, hardware clocks, and interrupts.

Quoted 5 lines

From a first look:

- patches 2 to 13 probably should try to use subsections, so that VMs that do not use the devices try not to save the extra data and keep backwards migration compatibility (at least try to)

Could you give me and example?
As I know, subsection is loaded when some predicate function returns true. How can I
construct such a function for integratorcp module? What kind of condition should it check?
In this module I just added missed vmstates (it does not saved/restored at all
by the master version).

Quoted 2 lines

- patch 16 should also use subsections, and perhaps apply to all other CPUs too?

We implemented replay only for i386 and ARM. If we'll change other targets, it will not
add record/replay capabilities to them, but can confuse the reviewers.

Quoted 1 line

- patches 23-24-25 perhaps could try using icount, like Konrad's patch do?

Using faster icount (like in Konrad's patches) is the our next aim. It obviously will
increase the speed of recording process. But now I submitted slower, but more conservative version of icount which we had already tested.

Quoted 2 lines

- patch 27 makes sense but VIRTUAL is used to skip blinking when the VM is stopped

Right, this is kind of hack. I haven't found better solution yet.

Quoted 5 lines

- the others I haven't yet looked at, but they look like something that would bitrot really, really fast. For patch 33, I think changing INSERT_HEAD to INSERT_TAIL would be just fine, and I wonder if it's the same for other patches here. How do you plan on testing them and keeping them up to date?

We're constantly keeping these patches up to date, because we are using deterministic replay
and reverse debugging for solving our tasks. We re-test all the features when pulling new patches from the master branch.

Pavel Dovgalyuk

Quoted 265 lines
Quoted 10 lines

Reverse execution has the following features:
* Deterministically replays whole system execution and all contents of the memory, state of the hadrware devices, clocks, and screen of the VM.
* Writes execution log into the file for latter replaying for multiple times on different machines.
* Supports i386, x86_64, and ARM hardware platforms.
* Performs deterministic replay of all operations with keyboard, mouse, network adapters, audio devices, serial interfaces, and physical USB devices connected to the emulator.
* Provides support for gdb reverse debugging commands like reverse-step and reverse-

continue.

Quoted 254 lines

* Supports auto-checkpointing for convenient reverse debugging.

Usage of the record/replay:
* First, record the execution, by adding '-record fname=replay.bin' to the command line.
* Then you can replay it for the multiple times by using another command line option: '-replay fname=replay.bin'
* Virtual machine should have at least one virtual disk, which is used to store checkpoints. If you want to enable automatic checkpointing, simply add ',period=XX' to record options, where XX is the checkpointing period in seconds.
* Using of the network adapters in record/replay mode is possible with the following command-line options:
- '-net user' (or another host adapter) in record mode
- '-net replay' in replay mode. Every host network adapter should be replaced by 'replay' when replaying the execution.
* Reverse debugging can be used through gdb remote interface. reverse-stepi and reverse-continue commands are supported. Other reverse commands should also work, because they reuse these ones.
* Monitor is extended by the following commands:
- replay_info - prints information about replay mode and current step (number of instructions executed)
- replay_break - sets "breakpoint" at the specified instructions count.
- replay_seek - rewinds (using the checkpoints, if possible) to the specified step of replay log.

Paper with short description of deterministic replay implementation:
http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html

Modifications of qemu include:
* adding missed fields of the virtual devices' states to the vmstate structures to allow deterministic saving and restoring the VM state
* adding virtual clock-based timers to vmstate structures, because virtual clock is the part of the virtual machine state
* modification of block layer to support automatic creation of the overlay files to store the changes and snapshots while recording
* disabling of system reset while loading VM state to avoid generating of interrupts by reset handlers
* adding warpers for clock and time functions to save their return values in the log
* saving different asynchronous events (e.g. system shutdown) into the log
* synchronization of the bottom halves execution
* synchronization of the threads from thread pool
* recording/replaying user input (mouse and keyboard), input from virtual serial ports, incoming network packets, input from connected USB devices
* adding HMP/QMP commands to monitor for controlling replay execution

v2 changes:
* Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
* Added QMP versions of replay commands (as suggested by Eric Blake)
* Removed some optional features of replay to make patches cleaner
* Minor changes and code cleanup were made

---

Pavel Dovgalyuk (49):
acpi: accurate overflow check
integratorcp: adding vmstate for save/restore
pcspk: adding vmstate for save/restore
fdc: adding vmstate for save/restore
parallel: adding vmstate for save/restore
serial: fixing vmstate for save/restore
kvmapic: fixing loading vmstate
hpet: fixing saving and loading process
pckbd: adding new fields to vmstate
rtl8139: adding new fields to vmstate
piix: do not raise irq while loading vmstate
mc146818rtc: add missed field to vmstate
pl031: add missed field to vmstate
ide pci: reset status field before loading the vmstate
softmmu: fixing usage of cpu_st/ld* from helpers
target: save cpu state fields
target-i386: update fp status fix
migration: add vmstate for int8 and char arrays
replay: global variables and function stubs
block: add suffix parameter to bdrv_open functions
sysemu: system functions for replay
replay: internal functions for replay log
cpu: invent instruction count for accurate replay
target-arm: instructions counting code for replay
target-i386: instructions counting code for replay
replay: interrupts and exceptions
vga: do not use virtual clock for blinking cursor
replay: asynchronous events infrastructure
replay: recording and replaying clock ticks
replay: recording and replaying different timers
replay: shutdown event
replay: checkpoints
replay: bottom halves
replay: replay aio requests
replay: thread pool
pl031: vmstate in replay mode
replay: initialization and deinitialization
replay: command line options
replay: snapshotting the virtual machine
replay: recording of the user input
tap-win32: destroy the thread at exit
replay: network packets record/replay
replay: audio data record/replay
replay: serial port
replay: USB passthrough
replay: replay_info command
replay: replay_break command
replay: replay_seek_step command
gdbstub: reverse debugging

 Makefile.target                  |    1
 arch_init.c                      |    8
 async.c                          |   45 ++
 audio/audio.c                    |   14 +
 audio/audio_win_int.h            |    3
 audio/winwaveaudio.c             |  167 +++++++--
 block.c                          |  134 ++++++-
 block/blkdebug.c                 |    2
 block/blkverify.c                |    4
 block/cow.c                      |    2
 block/qcow.c                     |    2
 block/qcow2.c                    |   10 -
 block/qed.c                      |    2
 block/raw-posix.c                |    6
 block/raw-win32.c                |    4
 block/sheepdog.c                 |    4
 block/vmdk.c                     |    8
 block/vvfat.c                    |    2
 blockdev.c                       |   11 -
 cpu-exec.c                       |   34 +-
 cpus.c                           |   81 ++++
 dma-helpers.c                    |   10 -
 exec.c                           |   12 +
 gdbstub.c                        |   79 +++-
 hmp-commands.hx                  |   41 ++
 hw/acpi/core.c                   |    7
 hw/arm/integratorcp.c            |   38 ++
 hw/audio/pcspk.c                 |   19 +
 hw/block/fdc.c                   |   11 -
 hw/block/virtio-blk.c            |   10 -
 hw/char/parallel.c               |   22 +
 hw/char/serial.c                 |  115 ++++--
 hw/display/vga.c                 |    4
 hw/i386/kvmvapic.c               |   22 +
 hw/ide/ahci.c                    |    4
 hw/ide/atapi.c                   |    9
 hw/ide/core.c                    |   18 +
 hw/ide/pci.c                     |   10 +
 hw/input/pckbd.c                 |    3
 hw/intc/apic_common.c            |    5
 hw/net/rtl8139.c                 |    5
 hw/pci-host/piix.c               |   22 +
 hw/timer/arm_timer.c             |    2
 hw/timer/hpet.c                  |   13 -
 hw/timer/mc146818rtc.c           |    5
 hw/timer/pl031.c                 |   35 +-
 hw/usb/hcd-uhci.c                |    2
 hw/usb/host-libusb.c             |  525 ++++++++++++++++++---------
 include/block/aio.h              |   17 +
 include/block/block.h            |   21 +
 include/block/thread-pool.h      |    4
 include/exec/cpu-defs.h          |    1
 include/exec/cpu_ldst_template.h |   28 +
 include/exec/exec-all.h          |   31 ++
 include/hw/host-libusb.h         |  105 +++++
 include/migration/vmstate.h      |   13 +
 include/qemu-common.h            |    3
 include/qemu/main-loop.h         |    1
 include/qemu/timer.h             |   48 ++
 include/qom/cpu.h                |    4
 include/sysemu/char.h            |   25 +
 include/sysemu/cpus.h            |    1
 include/sysemu/sysemu.h          |    2
 include/ui/input.h               |    2
 main-loop.c                      |    5
 monitor.c                        |   49 +++
 net/Makefile.objs                |    1
 net/clients.h                    |    3
 net/dump.c                       |    6
 net/hub.c                        |    1
 net/net-replay.c                 |   66 +++
 net/net.c                        |    7
 net/slirp.c                      |   14 +
 net/socket.c                     |   35 ++
 net/tap-win32.c                  |   25 +
 net/tap.c                        |   23 +
 net/vde.c                        |   14 +
 qapi-schema.json                 |   62 +++
 qemu-char.c                      |   55 +++
 qemu-img.c                       |    6
 qemu-io-cmds.c                   |    2
 qemu-io.c                        |    4
 qemu-nbd.c                       |    2
 qemu-options.hx                  |   28 +
 qemu-timer.c                     |   45 ++
 qmp-commands.hx                  |   60 +++
 replay/Makefile.objs             |   11 +
 replay/replay-audio.c            |  228 ++++++++++++
 replay/replay-char.c             |   99 +++++
 replay/replay-debug.c            |  148 ++++++++
 replay/replay-events.c           |  381 ++++++++++++++++++++
 replay/replay-input.c            |  107 ++++++
 replay/replay-internal.c         |  159 ++++++++
 replay/replay-internal.h         |  224 ++++++++++++
 replay/replay-net.c              |  190 ++++++++++
 replay/replay-qmp.c              |   57 +++
 replay/replay-time.c             |  181 +++++++++
 replay/replay-usb.c              |  188 ++++++++++
 replay/replay.c                  |  735 ++++++++++++++++++++++++++++++++++++++
 replay/replay.h                  |  218 +++++++++++
 savevm.c                         |   32 +-
 slirp/slirp.c                    |    9
 softmmu_template.h               |   18 +
 stubs/Makefile.objs              |    1
 stubs/replay.c                   |   42 ++
 target-arm/Makefile.objs         |    1
 target-arm/helper.h              |    3
 target-arm/machine.c             |    5
 target-arm/replay_helper.c       |   38 ++
 target-arm/translate.c           |   62 +++
 target-i386/Makefile.objs        |    1
 target-i386/cpu.c                |    1
 target-i386/cpu.h                |    8
 target-i386/fpu_helper.c         |    5
 target-i386/helper.h             |    3
 target-i386/machine.c            |    7
 target-i386/replay_helper.c      |   38 ++
 target-i386/translate.c          |   91 ++++-
 tests/test-thread-pool.c         |    7
 thread-pool.c                    |   53 ++-
 trace-events                     |    2
 translate-all.c                  |   15 +
 ui/input.c                       |   79 +++-
 util/iov.c                       |    4
 vl.c                             |  108 +++++-
 vmstate.c                        |    6
 126 files changed, 5433 insertions(+), 543 deletions(-)
 create mode 100755 include/hw/host-libusb.h
 create mode 100755 net/net-replay.c
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay-audio.c
 create mode 100755 replay/replay-char.c
 create mode 100755 replay/replay-debug.c
 create mode 100755 replay/replay-events.c
 create mode 100755 replay/replay-input.c
 create mode 100755 replay/replay-internal.c
 create mode 100755 replay/replay-internal.h
 create mode 100755 replay/replay-net.c
 create mode 100755 replay/replay-qmp.c
 create mode 100755 replay/replay-time.c
 create mode 100755 replay/replay-usb.c
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c
 create mode 100755 target-arm/replay_helper.c
 create mode 100755 target-i386/replay_helper.c

A message that is not in the archive (sent off-list or not yet received): [email protected]

#77 lore raw

Il 28/07/2014 09:50, Pavel Dovgaluk ha scritto:

Quoted 9 lines
Quoted 3 lines

- patches 2 to 13 probably should try to use subsections, so that VMs that do not use the devices try not to save the extra data and keep backwards migration compatibility (at least try to)

Could you give me and example?
As I know, subsection is loaded when some predicate function returns true. How can I
construct such a function for integratorcp module? What kind of condition should it check?
In this module I just added missed vmstates (it does not saved/restored at all
by the master version).

I tried to review the x86 ones. These are the most interesting ones, because it's the only board where we support cross-version migration.

Parallel is (quite obviously) unsalvageable, and I think HPET is too.

Quoted 5 lines
Quoted 2 lines

- patch 16 should also use subsections, and perhaps apply to all other CPUs too?

We implemented replay only for i386 and ARM. If we'll change other targets, it will not
add record/replay capabilities to them, but can confuse the reviewers.

Why are these fields only required by record/replay?

Quoted 5 lines
Quoted 1 line

- patches 23-24-25 perhaps could try using icount, like Konrad's patch do?

Using faster icount (like in Konrad's patches) is the our next aim. It obviously will
increase the speed of recording process. But now I submitted slower, but more conservative version of icount which we had already tested.

I see.

Quoted 4 lines
Quoted 2 lines

- patch 27 makes sense but VIRTUAL is used to skip blinking when the VM is stopped

Right, this is kind of hack. I haven't found better solution yet.

I think it's okay to use REALTIME, just add runstate_is_running() to the "if (now >= s->cursor_blink_time)".

That said, "every read to virtual clock is written to the replay log" worries me a bit from the point of view of thread-safety. Do you need to log all reads because you don't use icount? Reads can only happen at given points if you use icount, and you could simply log the icount value at each synchronization point.

Paolo

#78 lore raw
Quoted 8 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

Quoted 5 lines
Quoted 2 lines

- patch 16 should also use subsections, and perhaps apply to all other CPUs too?

We implemented replay only for i386 and ARM. If we'll change other targets, it will not
add record/replay capabilities to them, but can confuse the reviewers.

Why are these fields only required by record/replay?

No, these fields can cause non-determinism for normal execution mode too. But these changes are not related to reverse execution. I can make patches for them and submit them separately, if needed.

Quoted 13 lines
Quoted 5 lines
Quoted 1 line

- patches 23-24-25 perhaps could try using icount, like Konrad's patch do?

Using faster icount (like in Konrad's patches) is the our next aim. It obviously will
increase the speed of recording process. But now I submitted slower, but more conservative version of icount which we had already tested.

I see.

Quoted 4 lines
Quoted 2 lines

- patch 27 makes sense but VIRTUAL is used to skip blinking when the VM is stopped

Right, this is kind of hack. I haven't found better solution yet.

Quoted 8 lines

I think it's okay to use REALTIME, just add runstate_is_running() to the "if (now >= s->cursor_blink_time)".

That said, "every read to virtual clock is written to the replay log" worries me a bit from the point of view of thread-safety. Do you need to log all reads because you don't use icount? Reads can only happen at given points if you use icount, and you could simply log the icount value at each synchronization point.

We made two implementations of virtual clock. One uses host realtime clock and thus should be written into the log. Another one uses icount and calculated in one of the replay modules (similar to regular icount implementation). This patch series contains only first implementation, because it looks more convenient:
user does not have to specify icount shift value, when starting the recording or replaying.

Pavel Dovgalyuk

#79 lore raw

Il 30/07/2014 09:44, Pavel Dovgaluk ha scritto:

Quoted 4 lines
Quoted 4 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

Quoted 3 lines
Quoted 2 lines

- patch 16 should also use subsections, and perhaps apply to all other CPUs too?

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

We implemented replay only for i386 and ARM. If we'll change other targets, it will not
add record/replay capabilities to them, but can confuse the reviewers.

Why are these fields only required by record/replay?

No, these fields can cause non-determinism for normal execution mode too. But these changes are not related to reverse execution. I can make patches for them and submit them separately, if needed.

Yes, thanks. If it causes nondeterminism for normal execution mode too, it's better to apply them to all targets. If possible use a subsection, though.

Quoted 14 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

- patch 27 makes sense but VIRTUAL is used to skip blinking when the VM is stopped

Right, this is kind of hack. I haven't found better solution yet.

Quoted 8 lines

I think it's okay to use REALTIME, just add runstate_is_running() to the "if (now >= s->cursor_blink_time)".

That said, "every read to virtual clock is written to the replay log" worries me a bit from the point of view of thread-safety. Do you need to log all reads because you don't use icount? Reads can only happen at given points if you use icount, and you could simply log the icount value at each synchronization point.

Quoted 5 lines

We made two implementations of virtual clock. One uses host realtime clock and thus should be written into the log. Another one uses icount and calculated in one of the replay modules (similar to regular icount implementation). This patch series contains only first implementation, because it looks more convenient:
user does not have to specify icount shift value, when starting the recording or replaying.

I think from the upstreaming point of view it's better to stick with what makes the code simpler. Start by submitting only the icount-based implementation, the other can come later.

Paolo

#80 lore raw
Quoted 30 lines · Paolo Bonzini

Il 30/07/2014 09:44, Pavel Dovgaluk ha scritto:

Quoted 9 lines
Quoted 6 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

Quoted 4 lines
Quoted 2 lines

- patch 16 should also use subsections, and perhaps apply to all other CPUs too?

We implemented replay only for i386 and ARM. If we'll change other targets, it will not
add record/replay capabilities to them, but can confuse the reviewers.

Why are these fields only required by record/replay?

No, these fields can cause non-determinism for normal execution mode too. But these changes are not related to reverse execution. I can make patches for them and submit them separately, if needed.

Yes, thanks. If it causes nondeterminism for normal execution mode too, it's better to apply them to all targets. If possible use a subsection, though.

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
Quoted 2 lines

- patch 27 makes sense but VIRTUAL is used to skip blinking when the VM is stopped

Right, this is kind of hack. I haven't found better solution yet.

I think it's okay to use REALTIME, just add runstate_is_running() to the "if (now >= s->cursor_blink_time)".

That said, "every read to virtual clock is written to the replay log" worries me a bit from the point of view of thread-safety. Do you need to log all reads because you don't use icount? Reads can only happen at given points if you use icount, and you could simply log the icount value at each synchronization point.

Quoted 5 lines

We made two implementations of virtual clock. One uses host realtime clock and thus should be written into the log. Another one uses icount and calculated in one of the replay modules (similar to regular icount implementation). This patch series contains only first implementation, because it looks more convenient:
user does not have to specify icount shift value, when starting the recording or replaying.

Quoted 5 lines

I think from the upstreaming point of view it's better to stick with what makes the code simpler. Start by submitting only the icount-based implementation, the other can come later.

Paolo

Hi all,

I think that's actually our implementation cover no?
(http://lists.gnu.org/archive/html/qemu-devel/2014-07/msg00677.html)

Fred

#82 lore raw
Quoted 4 lines · Paolo Bonzini

Il 30/07/2014 15:19, Frederic Konrad ha scritto:

Quoted 3 lines
Quoted 3 lines

Start by submitting only the icount-based
implementation, the other can come later.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Paolo

Hi all,

I think that's actually our implementation cover no?
(http://lists.gnu.org/archive/html/qemu-devel/2014-07/msg00677.html)

Yes, Pavel's series covers the missing bits of yours AFAIU.

Paolo

I think that our first patch set part (icount fixes and icount based clock) is fully reviewed and can be upstreamed, but since the reverse execution mechanism itself is very similar between our series and Pavels, while Pavel’s set goes further in terms of addressing IO replay, I suggest Pavel uses our icount counter with his mechanism.

In addition maybe this series can be split eg:
the migration related series/the mechanism itself so it will be easier for me to review your mechanism?

Thanks,
Fred

#83 lore raw
Quoted 18 lines · Frederic Konrad
Quoted 4 lines · Paolo Bonzini

Il 30/07/2014 15:19, Frederic Konrad ha scritto:

Quoted 3 lines
Quoted 3 lines

Start by submitting only the icount-based
implementation, the other can come later.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Paolo

Hi all,

I think that's actually our implementation cover no?
(http://lists.gnu.org/archive/html/qemu-devel/2014-07/msg00677.html)

Yes, Pavel's series covers the missing bits of yours AFAIU.

Paolo

I think that our first patch set part (icount fixes and icount based clock) is fully reviewed and can be upstreamed, but since the reverse execution mechanism itself is very similar between our series and Pavels, while Pavel’s set goes further in terms of addressing IO replay, I suggest Pavel uses our icount counter with his mechanism.

Quoted 6 lines

In addition maybe this series can be split eg:
the migration related series/the mechanism itself so it will be easier for me to review your mechanism?

Thanks,
Fred

Is that making sense?

Fred

#84 lore raw

Il 31/07/2014 15:05, Frederic Konrad ha scritto:

Quoted 9 lines
Quoted 9 lines
Quoted 1 line

I think that our first patch set part (icount fixes and icount based clock) is fully reviewed and can be upstreamed,
but since the reverse execution mechanism itself is very similar between our series and Pavels, while Pavel’s
set goes further in terms of addressing IO replay, I suggest Pavel uses our icount counter with his mechanism.

Quoted 5 lines
Quoted 3 lines

In addition maybe this series can be split eg:
the migration related series/the mechanism itself so it will be easier for me to review your mechanism?

Is that making sense?

Yes, definitely. If you rebase reverse execution on top of branch uq/master of git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git I can include it in the same pull request as icount alignment (which is now ready for inclusion in 2.2).

Paolo

#85 lore raw
In reply to Paolo Bonzini
Quoted 11 lines

From: Paolo Bonzini [mailto:[email protected]]
Il 30/07/2014 09:44, Pavel Dovgaluk ha scritto:

Quoted 9 lines
Quoted 8 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

Quoted 5 lines
Quoted 2 lines

- patch 16 should also use subsections, and perhaps apply to all other CPUs too?

We implemented replay only for i386 and ARM. If we'll change other targets, it will not
add record/replay capabilities to them, but can confuse the reviewers.

Why are these fields only required by record/replay?

Quoted 7 lines
Quoted 3 lines

No, these fields can cause non-determinism for normal execution mode too. But these changes are not related to reverse execution. I can make patches for them and submit them separately, if needed.

Yes, thanks. If it causes nondeterminism for normal execution mode too, it's better to apply them to all targets. If possible use a subsection, though.

I've sent this change as a separate patch.
Instead of changing every virtual machine's vmstate, I added exception_index to the common exec.c module.

Pavel Dovgalyuk