Skip to contentSkip to conversation

Deterministic replay core

RFC PATCH Superseded Pavel Dovgalyuk 4 participants 82 messages Jan 22, 2015 – Feb 17, 2015 6 messages not in the archive

Subsystems (2)

v8 of 18 Superseded Latest is v18 →

lore
Share & tools
lore
#0 lore raw

v8 cover Deterministic replay core

This set of patches is related to the reverse execution and deterministic replay of qemu execution This implementation of deterministic replay can be used for deterministic debugging of guest code through gdb remote interface.

These patches include only core function of the replay, excluding the support for replaying serial, audio, network, and USB devices' operations. Reverse debugging and monitor commands were also excluded to be submitted later as separate patches.

Execution recording writes non-deterministic events log, which can be later used for replaying the execution anywhere and for unlimited number of times. It also supports checkpointing for faster rewinding during reverse debugging. Execution replaying reads the log and replays all non-deterministic events including external input, hardware clocks, and interrupts.

Deterministic replay has the following features:
* Deterministically replays whole system execution and all contents of the memory, state of the hadrware devices, clocks, and screen of the VM.
* Writes execution log into the file for latter replaying for multiple times on different machines.
* Supports i386, x86_64, and ARM hardware platforms.
* Performs deterministic replay of all operations with keyboard and mouse input devices.
* Supports auto-checkpointing for convenient reverse debugging.

Usage of the record/replay:
* First, record the execution, by adding the following string to the command line:
'-record fname=replay.bin -icount 7 -net none'. Block devices' images are not actually changed in the recording mode, because all of the changes are written to the temporary overlay file.
* Then you can replay it for the multiple times by using another command line option: '-replay fname=replay.bin -icount 7 -net none'
* '-net none' option should also be specified if network replay patches are not applied.

Paper with short description of deterministic replay implementation:
http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html

Modifications of qemu include:
* wrappers for clock and time functions to save their return values in the log
* saving different asynchronous events (e.g. system shutdown) into the log
* synchronization of the bottom halves execution
* synchronization of the threads from thread pool
* recording/replaying user input (mouse and keyboard)
* adding internal events for cpu and io synchronization

v8 changes:
* Simplified processing of the shutdown event (as suggested by Paolo Bonzini)
* Replaced stack of bottom halves in AIO context with QSIMPLEQ (as suggested by Paolo Bonzini)
* Moved replay_submode out of the series (as suggested by Paolo Bonzini)
* Moved suffix option out of the series
* Converted some of the defines into enums (as suggested by Paolo Bonzini)
* Encapsulated save_tm/read_tm calls into the single function (as suggested by Paolo Bonzini)
* Moved record/replay options to icount group (as suggested by Paolo Bonzini)
* Updated mutex protection for the events queue (as suggested by Paolo Bonzini)
* Added mutex to protect replay log file (as suggested by Paolo Bonzini)
* Minor cleanups

v7 changes:
* Removed patches that were applied to upstream.

v6 changes:
* Fixed replay stub return value (as suggested by Eric Blake)
* Fixed icount warping.
* Virtual rt clock now uses cpu_get_clock() (as suggested by Paolo Bonzini)
* Replated get_clock_realtime and get_clock calls with qemu clock requests (as suggested by Paolo Bonzini)
* Modified can_do_io logic to allow requesting icount from cpu_exec function (as suggested by Paolo Bonzini)
* Removed applied patches.

v5 changes:
* Minor changes.
* Used fixed-width integer types for read/write functions (as suggested by Alex Bennee)
* Moved savevm-related code out of the core.
* Added new traced clock for deterministic virtual clock warping (as suggested by Paolo Bonzini)
* Fixed exception_index reset for user mode (as suggested by Paolo Bonzini)
* Adopted Paolo's icount patches
* Fixed hardware interrupts replaying

v4 changes:
* Updated block drivers to support new bdrv_open interface.
* Moved migration patches into separate series (as suggested by Paolo Bonzini)
* Fixed a bug in replay_break operation.
* Fixed rtl8139 migration for replay.
* Fixed 'period' parameter processing for record mode.
* Fixed bug in 'reverse-stepi' implementation.
* Fixed replay without making any snapshots (even the starting one).
* Moved core replay patches into the separate series.
* Fixed reverse step and reverse continue support.
* Fixed several bugs in icount subsystem.
* Reusing native qemu icount for replay instructions counting.
* Separated core patches into their own series.

v3 changes:
* Fixed bug with replay of the aio write operations.
* Added virtual clock based on replay icount.
* Removed duplicated saving of interrupt_request CPU field.
* Fixed some coding style issues.
* Renamed QMP commands for controlling reverse execution (as suggested by Eric Blake)
* Replay mode and submode implemented as QAPI enumerations (as suggested by Eric Blake)
* Added description and example for replay-info command (as suggested by Eric Blake)
* Added information about the current breakpoint to the output of replay-info (as suggested by Eric Blake)
* Updated version id for HPET vmstate (as suggested by Paolo Bonzini)
* Removed static fields from parallel vmstate (as suggested by Paolo Bonzini)
* New vmstate fields for mc146818rtc, pckbd, kvmapic, serial, fdc, rtl8139 moved to subsection (as suggested by Paolo Bonzini)
* Disabled textmode cursor blinking, when virtual machine is stopped (as suggested by Paolo Bonzini)
* Extracted saving of exception_index to separate patch (as suggested by Paolo Bonzini)

v2 changes:
* Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
* Added QMP versions of replay commands (as suggested by Eric Blake)
* Removed some optional features of replay to make patches cleaner
* Minor changes and code cleanup were made

---

Pavel Dovgalyuk (21):
i386: partial revert of interrupt poll fix
replay: global variables and function stubs
sysemu: system functions for replay
replay: internal functions for replay log
replay: introduce mutex to protect the replay log
replay: introduce icount event
cpu-exec: allow temporary disabling icount
cpu: replay instructions sequence
replay: interrupts and exceptions
replay: asynchronous events infrastructure
replay: recording and replaying clock ticks
replay: recording and replaying different timers
replay: shutdown event
replay: checkpoints
aio: replace stack of bottom halves with queue
replay: bottom halves
replay: replay aio requests
replay: thread pool
replay: initialization and deinitialization
replay: command line options
replay: recording of the user input

 Makefile.target                |    1 
 async.c                        |   49 ++++--
 block.c                        |   92 ++++++++++-
 block/block-backend.c          |   30 +++-
 block/qcow2.c                  |    4 
 block/raw-posix.c              |    6 -
 block/raw-win32.c              |    4 
 cpu-exec.c                     |   60 ++++++-
 cpus.c                         |   45 ++++-
 dma-helpers.c                  |   10 +
 exec.c                         |    1 
 hw/block/virtio-blk.c          |   10 +
 hw/ide/ahci.c                  |    4 
 hw/ide/atapi.c                 |   10 +
 hw/ide/core.c                  |   18 +-
 hw/timer/arm_timer.c           |    2 
 hw/timer/mc146818rtc.c         |    3 
 hw/timer/pl031.c               |    3 
 hw/usb/hcd-uhci.c              |    2 
 include/block/aio.h            |   22 ++-
 include/block/block.h          |   15 ++
 include/block/thread-pool.h    |    4 
 include/exec/exec-all.h        |    2 
 include/qemu-common.h          |    3 
 include/qemu/main-loop.h       |    1 
 include/qemu/queue.h           |    7 +
 include/qemu/timer.h           |   16 ++
 include/qom/cpu.h              |   10 +
 include/sysemu/block-backend.h |   10 +
 include/sysemu/cpus.h          |    1 
 include/ui/input.h             |    2 
 main-loop.c                    |   10 +
 qapi-schema.json               |   18 ++
 qemu-io-cmds.c                 |    2 
 qemu-options.hx                |    8 +
 qemu-timer.c                   |   56 +++++--
 replay/Makefile.objs           |    5 +
 replay/replay-events.c         |  310 ++++++++++++++++++++++++++++++++++++
 replay/replay-input.c          |  130 +++++++++++++++
 replay/replay-internal.c       |  183 +++++++++++++++++++++
 replay/replay-internal.h       |  149 +++++++++++++++++
 replay/replay-time.c           |  216 +++++++++++++++++++++++++
 replay/replay.c                |  341 ++++++++++++++++++++++++++++++++++++++++
 replay/replay.h                |  132 +++++++++++++++
 stubs/Makefile.objs            |    1 
 stubs/replay.c                 |   37 ++++
 target-i386/cpu.c              |   10 -
 target-i386/seg_helper.c       |    4 
 tests/test-thread-pool.c       |    7 -
 thread-pool.c                  |   49 ++++--
 trace-events                   |    2 
 translate-all.c                |   10 +
 ui/input.c                     |   68 ++++++--
 util/iov.c                     |    4 
 vl.c                           |   54 ++++++
 55 files changed, 2101 insertions(+), 152 deletions(-)
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay-events.c
 create mode 100755 replay/replay-input.c
 create mode 100755 replay/replay-internal.c
 create mode 100755 replay/replay-internal.h
 create mode 100755 replay/replay-time.c
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c
Signature · 1 line
Pavel Dovgalyuk
#1 lore raw

RFC PATCH v8 1/21 i386: partial revert of interrupt poll fix

Processing CPU_INTERRUPT_POLL requests in cpu_has_work functions break the determinism of cpu_exec. This patch is required to make interrupts processing deterministic.

---
 cpu-exec.c        |    6 ++++++
 target-i386/cpu.c |   10 ++--------
 2 files changed, 8 insertions(+), 8 deletions(-)
cpu-exec.c +6 -0
index a4f0eff..0a485bb 100644
@@ -338,6 +338,12 @@ int cpu_exec(CPUArchState *env)
     volatile bool have_tb_lock = false;
 
     if (cpu->halted) {
+#ifdef TARGET_I386
+        if (cpu->interrupt_request & CPU_INTERRUPT_POLL) {
+            apic_poll_irq(x86_cpu->apic_state);
+            cpu_reset_interrupt(cpu, CPU_INTERRUPT_POLL);
+        }
+#endif
         if (!cpu_has_work(cpu)) {
             return EXCP_HALTED;
         }
target-i386/cpu.c +2 -8
index b81ac5c..8f1f7aa 100644
@@ -2970,14 +2970,8 @@ static bool x86_cpu_has_work(CPUState *cs)
     X86CPU *cpu = X86_CPU(cs);
     CPUX86State *env = &cpu->env;
 
-#if !defined(CONFIG_USER_ONLY)
-    if (cs->interrupt_request & CPU_INTERRUPT_POLL) {
-        apic_poll_irq(cpu->apic_state);
-        cpu_reset_interrupt(cs, CPU_INTERRUPT_POLL);
-    }
-#endif
-
-    return ((cs->interrupt_request & CPU_INTERRUPT_HARD) &&
+    return ((cs->interrupt_request & (CPU_INTERRUPT_HARD |
+                                      CPU_INTERRUPT_POLL)) &&
             (env->eflags & IF_MASK)) ||
            (cs->interrupt_request & (CPU_INTERRUPT_NMI |
                                      CPU_INTERRUPT_INIT |
#2 lore raw

RFC PATCH v8 2/21 replay: global variables and function stubs

In reply to Pavel Dovgalyuk

This patch adds global variables, defines, functions declarations, and function stubs for deterministic VM replay used by external modules.

---
 Makefile.target      |    1 +
 qapi-schema.json     |   18 ++++++++++++++++++
 replay/Makefile.objs |    1 +
 replay/replay.c      |   14 ++++++++++++++
 replay/replay.h      |   19 +++++++++++++++++++
 stubs/Makefile.objs  |    1 +
 stubs/replay.c       |    3 +++
 7 files changed, 57 insertions(+), 0 deletions(-)
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c
Makefile.target +1 -0
index e9ff1ee..a45378f 100644
@@ -83,6 +83,7 @@ all: $(PROGS) stap
 #########################################################
 # cpu emulator library
 obj-y = exec.o translate-all.o cpu-exec.o
+obj-y += replay/
 obj-y += tcg/tcg.o tcg/optimize.o
 obj-$(CONFIG_TCG_INTERPRETER) += tci.o
 obj-$(CONFIG_TCG_INTERPRETER) += disas/tci.o
qapi-schema.json +18 -0
index fbfc52f..7e3177f 100644
@@ -3523,3 +3523,21 @@
 # Since: 2.1
 ##
 { 'command': 'rtc-reset-reinjection' }
+
+##
+# ReplayMode:
+#
+# Mode of the replay subsystem.
+#
+# @none: normal execution mode. Replay or record are not enabled.
+#
+# @record: record mode. All non-deterministic data is written into the
+#          replay log.
+#
+# @play: replay mode. Non-deterministic data required for system execution
+#        is read from the log.
+#
+# Since: 2.3
+##
+{ 'enum': 'ReplayMode',
+  'data': [ 'none', 'record', 'play' ] }
replay/Makefile.objs new file +1 -0
new file mode 100755
index 0000000..7ea860f
@@ -0,0 +1 @@
+obj-y += replay.o
replay/replay.c new file +14 -0
new file mode 100755
index 0000000..5ce066f
@@ -0,0 +1,14 @@
+/*
+ * replay.c
+ *
+ * Copyright (c) 2010-2015 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "replay.h"
+
+ReplayMode replay_mode = REPLAY_MODE_NONE;
replay/replay.h new file +19 -0
new file mode 100755
index 0000000..d6b73c3
@@ -0,0 +1,19 @@
+#ifndef REPLAY_H
+#define REPLAY_H
+
+/*
+ * replay.h
+ *
+ * Copyright (c) 2010-2015 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qapi-types.h"
+
+extern ReplayMode replay_mode;
+
+#endif
stubs/Makefile.objs +1 -0
index 5e347d0..45a6c71 100644
@@ -27,6 +27,7 @@ stub-obj-y += notify-event.o
 stub-obj-y += pci-drive-hot-add.o
 stub-obj-$(CONFIG_SPICE) += qemu-chr-open-spice.o
 stub-obj-y += qtest.o
+stub-obj-y += replay.o
 stub-obj-y += reset.o
 stub-obj-y += runstate-check.o
 stub-obj-y += set-fd-handler.o
stubs/replay.c new file +3 -0
new file mode 100755
index 0000000..563c777
@@ -0,0 +1,3 @@
+#include "replay/replay.h"
+
+ReplayMode replay_mode;
Reviewed-by on v8 2/21
#3 lore raw

Re: [Qemu-devel] [RFC PATCH v8 02/21] replay: global variables and function stubs

Quoted 127 lines · Pavel Dovgalyuk

This patch adds global variables, defines, functions declarations, and function stubs for deterministic VM replay used by external modules.

---
 Makefile.target      |    1 +
 qapi-schema.json     |   18 ++++++++++++++++++
 replay/Makefile.objs |    1 +
 replay/replay.c      |   14 ++++++++++++++
 replay/replay.h      |   19 +++++++++++++++++++
 stubs/Makefile.objs  |    1 +
 stubs/replay.c       |    3 +++
 7 files changed, 57 insertions(+), 0 deletions(-)
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c
Makefile.target +1 -0
index e9ff1ee..a45378f 100644
@@ -83,6 +83,7 @@ all: $(PROGS) stap
 #########################################################
 # cpu emulator library
 obj-y = exec.o translate-all.o cpu-exec.o
+obj-y += replay/
 obj-y += tcg/tcg.o tcg/optimize.o
 obj-$(CONFIG_TCG_INTERPRETER) += tci.o
 obj-$(CONFIG_TCG_INTERPRETER) += disas/tci.o
qapi-schema.json +18 -0
index fbfc52f..7e3177f 100644
@@ -3523,3 +3523,21 @@
 # Since: 2.1
 ##
 { 'command': 'rtc-reset-reinjection' }
+
+##
+# ReplayMode:
+#
+# Mode of the replay subsystem.
+#
+# @none: normal execution mode. Replay or record are not enabled.
+#
+# @record: record mode. All non-deterministic data is written into the
+#          replay log.
+#
+# @play: replay mode. Non-deterministic data required for system execution
+#        is read from the log.
+#
+# Since: 2.3
+##
+{ 'enum': 'ReplayMode',
+  'data': [ 'none', 'record', 'play' ] }
replay/Makefile.objs new file +1 -0
new file mode 100755
index 0000000..7ea860f
@@ -0,0 +1 @@
+obj-y += replay.o
replay/replay.c new file +14 -0
new file mode 100755
index 0000000..5ce066f
@@ -0,0 +1,14 @@
+/*
+ * replay.c
+ *
+ * Copyright (c) 2010-2015 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "replay.h"
+
+ReplayMode replay_mode = REPLAY_MODE_NONE;
replay/replay.h new file +19 -0
new file mode 100755
index 0000000..d6b73c3
@@ -0,0 +1,19 @@
+#ifndef REPLAY_H
+#define REPLAY_H
+
+/*
+ * replay.h
+ *
+ * Copyright (c) 2010-2015 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qapi-types.h"
+
+extern ReplayMode replay_mode;
+
+#endif
stubs/Makefile.objs +1 -0
index 5e347d0..45a6c71 100644
@@ -27,6 +27,7 @@ stub-obj-y += notify-event.o
 stub-obj-y += pci-drive-hot-add.o
 stub-obj-$(CONFIG_SPICE) += qemu-chr-open-spice.o
 stub-obj-y += qtest.o
+stub-obj-y += replay.o
 stub-obj-y += reset.o
 stub-obj-y += runstate-check.o
 stub-obj-y += set-fd-handler.o
stubs/replay.c new file +0 -0
new file mode 100755
index 0000000..563c777

stubs/replay.c line 3

Quoted 3 lines
stubs/replay.c new file +3 -0
@@ -0,0 +1,3 @@
+#include "replay/replay.h"
+
+ReplayMode replay_mode;
Reviewed-by on v8 2/21
#4 lore raw

Re: [Qemu-devel] [RFC PATCH v8 02/21] replay: global variables and function stubs

In reply to Pavel Dovgalyuk
Quoted 4 lines · Pavel Dovgalyuk

This patch adds global variables, defines, functions declarations, and function stubs for deterministic VM replay used by external modules.

Quoted 13 lines
---
 Makefile.target      |    1 +
 qapi-schema.json     |   18 ++++++++++++++++++
 replay/Makefile.objs |    1 +
 replay/replay.c      |   14 ++++++++++++++
 replay/replay.h      |   19 +++++++++++++++++++
 stubs/Makefile.objs  |    1 +
 stubs/replay.c       |    3 +++
 7 files changed, 57 insertions(+), 0 deletions(-)
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#5 lore raw

RFC PATCH v8 3/21 sysemu: system functions for replay

In reply to Pavel Dovgalyuk

This patch removes "static" specifier from several qemu function to make them visible to the replay module. It also invents several system functions that will be used by replay.

---
 cpus.c                  |    4 ++--
 include/exec/exec-all.h |    1 +
 include/qom/cpu.h       |   10 ++++++++++
 include/sysemu/cpus.h   |    1 +
 translate-all.c         |    8 ++++++++
 5 files changed, 22 insertions(+), 2 deletions(-)
cpus.c +2 -2
index 3a5323b..c513275 100644
@@ -88,7 +88,7 @@ static bool cpu_thread_is_idle(CPUState *cpu)
     return true;
 }
 
-static bool all_cpu_threads_idle(void)
+bool all_cpu_threads_idle(void)
 {
     CPUState *cpu;
 
@@ -1121,7 +1121,7 @@ bool qemu_cpu_is_self(CPUState *cpu)
     return qemu_thread_is_self(cpu->thread);
 }
 
-static bool qemu_in_vcpu_thread(void)
+bool qemu_in_vcpu_thread(void)
 {
     return current_cpu && qemu_cpu_is_self(current_cpu);
 }
include/exec/exec-all.h +1 -0
index 4a6237f..63fb1a3 100644
@@ -214,6 +214,7 @@ static inline unsigned int tb_phys_hash_func(tb_page_addr_t pc)
 
 void tb_free(TranslationBlock *tb);
 void tb_flush(CPUArchState *env);
+void tb_flush_all(void);
 void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr);
 
 #if defined(USE_DIRECT_JUMP)
include/qom/cpu.h +10 -0
index 2098f1c..5afb44c 100644
@@ -480,6 +480,16 @@ static inline bool cpu_has_work(CPUState *cpu)
 bool qemu_cpu_is_self(CPUState *cpu);
 
 /**
+ * qemu_in_vcpu_thread:
+ *
+ * Checks whether the caller is executing on the vCPU thread
+ * of the current vCPU.
+ *
+ * Returns: %true if called from vCPU's thread, %false otherwise.
+ */
+bool qemu_in_vcpu_thread(void);
+
+/**
  * qemu_cpu_kick:
  * @cpu: The vCPU to kick.
  *
include/sysemu/cpus.h +1 -0
index 3f162a9..86ae556 100644
@@ -6,6 +6,7 @@ void qemu_init_cpu_loop(void);
 void resume_all_vcpus(void);
 void pause_all_vcpus(void);
 void cpu_stop_current(void);
+bool all_cpu_threads_idle(void);
 
 void cpu_synchronize_all_states(void);
 void cpu_synchronize_all_post_reset(void);
translate-all.c +8 -0
index 4a1b64f..c0cba7d 100644
@@ -812,6 +812,14 @@ void tb_flush(CPUArchState *env1)
     tcg_ctx.tb_ctx.tb_flush_count++;
 }
 
+void tb_flush_all(void)
+{
+    CPUState *cpu;
+    for (cpu = first_cpu ; cpu != NULL ; cpu = CPU_NEXT(cpu)) {
+        tb_flush(cpu->env_ptr);
+    }
+}
+
 #ifdef DEBUG_TB_CHECK
 
 static void tb_invalidate_check(target_ulong address)
Reviewed-by on v8 3/21
#6 lore raw

Re: [Qemu-devel] [RFC PATCH v8 03/21] sysemu: system functions for replay

Quoted 90 lines · Pavel Dovgalyuk

This patch removes "static" specifier from several qemu function to make them visible to the replay module. It also invents several system functions that will be used by replay.

---
 cpus.c                  |    4 ++--
 include/exec/exec-all.h |    1 +
 include/qom/cpu.h       |   10 ++++++++++
 include/sysemu/cpus.h   |    1 +
 translate-all.c         |    8 ++++++++
 5 files changed, 22 insertions(+), 2 deletions(-)
cpus.c +2 -2
index 3a5323b..c513275 100644
@@ -88,7 +88,7 @@ static bool cpu_thread_is_idle(CPUState *cpu)
     return true;
 }
 
-static bool all_cpu_threads_idle(void)
+bool all_cpu_threads_idle(void)
 {
     CPUState *cpu;
 
@@ -1121,7 +1121,7 @@ bool qemu_cpu_is_self(CPUState *cpu)
     return qemu_thread_is_self(cpu->thread);
 }
 
-static bool qemu_in_vcpu_thread(void)
+bool qemu_in_vcpu_thread(void)
 {
     return current_cpu && qemu_cpu_is_self(current_cpu);
 }
include/exec/exec-all.h +1 -0
index 4a6237f..63fb1a3 100644
@@ -214,6 +214,7 @@ static inline unsigned int tb_phys_hash_func(tb_page_addr_t pc)
 
 void tb_free(TranslationBlock *tb);
 void tb_flush(CPUArchState *env);
+void tb_flush_all(void);
 void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr);
 
 #if defined(USE_DIRECT_JUMP)
include/qom/cpu.h +10 -0
index 2098f1c..5afb44c 100644
@@ -480,6 +480,16 @@ static inline bool cpu_has_work(CPUState *cpu)
 bool qemu_cpu_is_self(CPUState *cpu);
 
 /**
+ * qemu_in_vcpu_thread:
+ *
+ * Checks whether the caller is executing on the vCPU thread
+ * of the current vCPU.
+ *
+ * Returns: %true if called from vCPU's thread, %false otherwise.
+ */
+bool qemu_in_vcpu_thread(void);
+
+/**
  * qemu_cpu_kick:
  * @cpu: The vCPU to kick.
  *
include/sysemu/cpus.h +1 -0
index 3f162a9..86ae556 100644
@@ -6,6 +6,7 @@ void qemu_init_cpu_loop(void);
 void resume_all_vcpus(void);
 void pause_all_vcpus(void);
 void cpu_stop_current(void);
+bool all_cpu_threads_idle(void);
 
 void cpu_synchronize_all_states(void);
 void cpu_synchronize_all_post_reset(void);
translate-all.c +1 -0
index 4a1b64f..c0cba7d 100644
@@ -812,6 +812,14 @@ void tb_flush(CPUArchState *env1)
     tcg_ctx.tb_ctx.tb_flush_count++;
 }
 
+void tb_flush_all(void)

translate-all.c void tb_flush(CPUArchState *env1) · line 825

Quoted 10 lines
translate-all.c +7 -0
+{
+    CPUState *cpu;
+    for (cpu = first_cpu ; cpu != NULL ; cpu = CPU_NEXT(cpu)) {
+        tb_flush(cpu->env_ptr);
+    }
+}
+
 #ifdef DEBUG_TB_CHECK
 
 static void tb_invalidate_check(target_ulong address)
#7 lore raw

RFC PATCH v8 4/21 replay: internal functions for replay log

In reply to Pavel Dovgalyuk

This patch adds functions to perform read and write operations with replay log.

---
 replay/Makefile.objs     |    1 
 replay/replay-internal.c |  141 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   50 ++++++++++++++++
 3 files changed, 192 insertions(+), 0 deletions(-)
 create mode 100755 replay/replay-internal.c
 create mode 100755 replay/replay-internal.h
replay/Makefile.objs +1 -0
index 7ea860f..1148f45 100755
@@ -1 +1,2 @@
 obj-y += replay.o
+obj-y += replay-internal.o
replay/replay-internal.c new file +141 -0

141 lines not on this page · raw message

replay/replay-internal.h new file +50 -0

50 lines not on this page · raw message

#8 lore raw

Re: [Qemu-devel] [RFC PATCH v8 04/21] replay: internal functions for replay log

Quoted 39 lines · Pavel Dovgalyuk

This patch adds functions to perform read and write operations with replay log.

---
 replay/Makefile.objs     |    1 
 replay/replay-internal.c |  141 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   50 ++++++++++++++++
 3 files changed, 192 insertions(+), 0 deletions(-)
 create mode 100755 replay/replay-internal.c
 create mode 100755 replay/replay-internal.h
replay/Makefile.objs +1 -0
index 7ea860f..1148f45 100755
@@ -1 +1,2 @@
 obj-y += replay.o
+obj-y += replay-internal.o
replay/replay-internal.c new file +14 -0
new file mode 100755
index 0000000..3865fa5
@@ -0,0 +1,141 @@
+/*
+ * replay-internal.c
+ *
+ * Copyright (c) 2010-2015 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "replay-internal.h"
+

replay/replay-internal.c line 24

Quoted 10 lines
replay/replay-internal.c new file +10 -0
+unsigned int replay_data_kind = -1;
+unsigned int replay_has_unread_data;
+
+/* File for replay writing */
+FILE *replay_file;
+
+void replay_put_byte(uint8_t byte)
+{
+    if (replay_file) {
+        fwrite(&byte, sizeof(byte), 1, replay_file);

putc(byte, replay_file);

replay/replay-internal.c line 37

Quoted 13 lines
replay/replay-internal.c new file +13 -0
+    }
+}
+
+void replay_put_event(uint8_t event)
+{
+    replay_put_byte(event);
+}
+
+
+void replay_put_word(uint16_t word)
+{
+    if (replay_file) {
+        fwrite(&word, sizeof(word), 1, replay_file);

Let's make this use a guaranteed endianness. QEMU usually uses big endian. So either use putc, or use cpu_to_be16 and fwrite.

replay/replay-internal.c line 44

Quoted 7 lines
replay/replay-internal.c new file +7 -0
+    }
+}
+
+void replay_put_dword(uint32_t dword)
+{
+    if (replay_file) {
+        fwrite(&dword, sizeof(dword), 1, replay_file);

Same here.

replay/replay-internal.c line 51

Quoted 7 lines
replay/replay-internal.c new file +7 -0
+    }
+}
+
+void replay_put_qword(int64_t qword)
+{
+    if (replay_file) {
+        fwrite(&qword, sizeof(qword), 1, replay_file);

And same here.

replay/replay-internal.c line 58

Quoted 7 lines
replay/replay-internal.c new file +7 -0
+    }
+}
+
+void replay_put_array(const uint8_t *buf, size_t size)
+{
+    if (replay_file) {
+        fwrite(&size, sizeof(size), 1, replay_file);

replay_put_dword (a limit of 2^32 elements should not be a problem :)).

replay/replay-internal.c line 67

Quoted 9 lines
replay/replay-internal.c new file +9 -0
+        fwrite(buf, 1, size, replay_file);
+    }
+}
+
+uint8_t replay_get_byte(void)
+{
+    uint8_t byte;
+    if (replay_file) {
+        fread(&byte, sizeof(byte), 1, replay_file);

byte = getc(replay_file)

replay/replay-internal.c line 76

Quoted 9 lines
replay/replay-internal.c new file +9 -0
+    }
+    return byte;
+}
+
+uint16_t replay_get_word(void)
+{
+    uint16_t word;
+    if (replay_file) {
+        fread(&word, sizeof(word), 1, replay_file);

Same comment about endianness.

Quoted 33 lines
replay/replay-internal.c new file +33 -0
+    }
+
+    return word;
+}
+
+uint32_t replay_get_dword(void)
+{
+    uint32_t dword;
+    if (replay_file) {
+        fread(&dword, sizeof(dword), 1, replay_file);
+    }
+
+    return dword;
+}
+
+int64_t replay_get_qword(void)
+{
+    int64_t qword;
+    if (replay_file) {
+        fread(&qword, sizeof(qword), 1, replay_file);
+    }
+
+    return qword;
+}
+
+void replay_get_array(uint8_t *buf, size_t *size)
+{
+    if (replay_file) {
+        fread(size, sizeof(*size), 1, replay_file);
+        fread(buf, 1, *size, replay_file);
+    }
+}
+

replay/replay-internal.c line 119

Quoted 10 lines
replay/replay-internal.c new file +10 -0
+void replay_get_array_alloc(uint8_t **buf, size_t *size)
+{
+    if (replay_file) {
+        fread(size, sizeof(*size), 1, replay_file);
+        *buf = g_malloc(*size);
+        fread(*buf, 1, *size, replay_file);
+    }
+}
+
+void replay_check_error(void)

Could this be static? (I haven't checked).

replay/replay-internal.c line 124

Quoted 5 lines
replay/replay-internal.c new file +5 -0
+{
+    if (replay_file) {
+        if (feof(replay_file)) {
+            fprintf(stderr, "replay file is over\n");
+            exit(1);

Perhaps qemu_system_vmstop_request_prepare + qemu_system_vmstop_request(RUN_STATE_PAUSED) instead of exit? Those two functions are thread-safe.

replay/replay-internal.c line 126

Quoted 2 lines
replay/replay-internal.c new file +2 -0
+        } else if (ferror(replay_file)) {
+            fprintf(stderr, "replay file is over or something goes wrong\n");

Same here (RUN_STATE_INTERNAL_ERROR?).

Quoted 59 lines
replay/replay-internal.c new file +15 -0
+            exit(1);
+        }
+    }
+}
+
+void replay_fetch_data_kind(void)
+{
+    if (replay_file) {
+        if (!replay_has_unread_data) {
+            replay_data_kind = replay_get_byte();
+            replay_check_error();
+            replay_has_unread_data = 1;
+        }
+    }
+}
replay/replay-internal.h new file +38 -0
new file mode 100755
index 0000000..dad5bc8
@@ -0,0 +1,50 @@
+#ifndef REPLAY_INTERNAL_H
+#define REPLAY_INTERNAL_H
+
+/*
+ * replay-internal.h
+ *
+ * Copyright (c) 2010-2015 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include <stdio.h>
+
+extern unsigned int replay_data_kind;
+extern unsigned int replay_has_unread_data;
+
+/* File for replay writing */
+extern FILE *replay_file;
+
+void replay_put_byte(uint8_t byte);
+void replay_put_event(uint8_t event);
+void replay_put_word(uint16_t word);
+void replay_put_dword(uint32_t dword);
+void replay_put_qword(int64_t qword);
+void replay_put_array(const uint8_t *buf, size_t size);
+
+uint8_t replay_get_byte(void);
+uint16_t replay_get_word(void);
+uint32_t replay_get_dword(void);
+int64_t replay_get_qword(void);
+void replay_get_array(uint8_t *buf, size_t *size);
+void replay_get_array_alloc(uint8_t **buf, size_t *size);
+
+/*! Checks error status of the file. */
+void replay_check_error(void);

replay/replay-internal.h line 48

Quoted 10 lines
replay/replay-internal.h new file +10 -0
+
+/*! Reads data type from the file and stores it in the
+    replay_data_kind variable. */
+void replay_fetch_data_kind(void);
+
+/*! Saves queued events (like instructions and sound). */
+void replay_save_instructions(void);
+/*! Checks that the next data is corresponding to the desired kind.
+    Terminates the program in case of error. */
+void validate_data_kind(int kind);

Please move the last two prototypes to the patches that introduce the functions.

Quoted 3 lines
replay/replay-internal.h new file +2 -0
+
+#endif
#9 lore raw

Re: [Qemu-devel] [RFC PATCH v8 04/21] replay: internal functions for replay log

Quoted 10 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 6 lines · Pavel Dovgalyuk

This patch adds functions to perform read and write operations with replay log.

---
+void replay_check_error(void)

Could this be static? (I haven't checked).

No, because it is used from several replay files.

Quoted 9 lines

replay/replay-internal.c line 124

Quoted 5 lines
replay/replay-internal.c new file +5 -0
+{
+    if (replay_file) {
+        if (feof(replay_file)) {
+            fprintf(stderr, "replay file is over\n");
+            exit(1);

Perhaps qemu_system_vmstop_request_prepare + qemu_system_vmstop_request(RUN_STATE_PAUSED) instead of exit? Those two functions are thread-safe.

There is no need to stop when replay file is over (because we cannot replay more). Should we send shutdown request instead?

Pavel Dovgalyuk

#10 lore raw

Re: [Qemu-devel] [RFC PATCH v8 04/21] replay: internal functions for replay log

Quoted 3 lines · Pavel Dovgaluk
Quoted 1 line

Could this be static? (I haven't checked).

No, because it is used from several replay files.

I wonder if that's a layering violation.

Quoted 6 lines
Quoted 3 lines

Perhaps qemu_system_vmstop_request_prepare + qemu_system_vmstop_request(RUN_STATE_PAUSED) instead of exit? Those two functions are thread-safe.

There is no need to stop when replay file is over (because we cannot replay more). Should we send shutdown request instead?

I thought about it. I think no, because shutdown is irreversible (see runstate_needs_reset). Just pausing seemed to be the right compromise, and then the next "cont" can run the VM out of replay mode.

Paolo

#11 lore raw

Re: [Qemu-devel] [RFC PATCH v8 04/21] replay: internal functions for replay log

I believe thats what we concluded too
Cheers
Mark.

Quoted 15 lines · Paolo Bonzini
Quoted 3 lines · Pavel Dovgaluk
Quoted 1 line

Could this be static? (I haven't checked).

No, because it is used from several replay files.

I wonder if that's a layering violation.

Quoted 4 lines
Quoted 3 lines

Perhaps qemu_system_vmstop_request_prepare + qemu_system_vmstop_request(RUN_STATE_PAUSED) instead of exit? Those two functions are thread-safe.

Quoted 8 lines
Quoted 2 lines

There is no need to stop when replay file is over (because we cannot replay more). Should we send shutdown request instead?

I thought about it. I think no, because shutdown is irreversible (see runstate_needs_reset). Just pausing seemed to be the right compromise, and then the next "cont" can run the VM out of replay mode.

Paolo

	 +44 (0)20 7100 3485 x 210

+33 (0)5 33 52 01 77x 210

	+33 (0)603762104
	mark.burton
#12 lore raw

RFC PATCH v8 5/21 replay: introduce mutex to protect the replay log

In reply to Pavel Dovgalyuk

This mutex will protect read/write operations for replay log. Using mutex is necessary because most of the events consist of several fields stored in the log. The mutex will help to avoid races.

---
 replay/replay-internal.c |   25 +++++++++++++++++++++++++
 replay/replay-internal.h |    7 +++++++
 2 files changed, 32 insertions(+), 0 deletions(-)
replay/replay-internal.c +25 -0
index 3865fa5..f552dad 100755
@@ -15,6 +15,11 @@
 unsigned int replay_data_kind = -1;
 unsigned int replay_has_unread_data;
 
+/* Mutex to protect reading and writing events to the log.
+   replay_data_kind and replay_has_unread_data are also protected
+   by this mutex. */
+static QemuMutex lock;
+
 /* File for replay writing */
 FILE *replay_file;
 
@@ -139,3 +144,23 @@ void replay_fetch_data_kind(void)
         }
     }
 }
+
+void replay_mutex_init(void)
+{
+    qemu_mutex_init(&lock);
+}
+
+void replay_mutex_destroy(void)
+{
+    qemu_mutex_destroy(&lock);
+}
+
+void replay_mutex_lock(void)
+{
+    qemu_mutex_lock(&lock);
+}
+
+void replay_mutex_unlock(void)
+{
+    qemu_mutex_unlock(&lock);
+}
replay/replay-internal.h +7 -0
index dad5bc8..50ce29b 100755
@@ -34,6 +34,13 @@ int64_t replay_get_qword(void);
 void replay_get_array(uint8_t *buf, size_t *size);
 void replay_get_array_alloc(uint8_t **buf, size_t *size);
 
+/* Mutex functions for protecting replay log file */
+
+void replay_mutex_init(void);
+void replay_mutex_destroy(void);
+void replay_mutex_lock(void);
+void replay_mutex_unlock(void);
+
 /*! Checks error status of the file. */
 void replay_check_error(void);
 
Reviewed-by on v8 5/21
#13 lore raw

Re: [Qemu-devel] [RFC PATCH v8 05/21] replay: introduce mutex to protect the replay log

Quoted 30 lines · Pavel Dovgalyuk

This mutex will protect read/write operations for replay log. Using mutex is necessary because most of the events consist of several fields stored in the log. The mutex will help to avoid races.

---
 replay/replay-internal.c |   25 +++++++++++++++++++++++++
 replay/replay-internal.h |    7 +++++++
 2 files changed, 32 insertions(+), 0 deletions(-)
replay/replay-internal.c +5 -0
index 3865fa5..f552dad 100755
@@ -15,6 +15,11 @@
 unsigned int replay_data_kind = -1;
 unsigned int replay_has_unread_data;
 
+/* Mutex to protect reading and writing events to the log.
+   replay_data_kind and replay_has_unread_data are also protected
+   by this mutex. */
+static QemuMutex lock;
+
 /* File for replay writing */
 FILE *replay_file;
 
@@ -139,3 +144,23 @@ void replay_fetch_data_kind(void)
         }
     }
 }

replay/replay-internal.c void replay_fetch_data_kind(void) · line 156

Quoted 10 lines
replay/replay-internal.c +10 -0
+
+void replay_mutex_init(void)
+{
+    qemu_mutex_init(&lock);
+}
+
+void replay_mutex_destroy(void)
+{
+    qemu_mutex_destroy(&lock);
+}

Destroy is probably not necessary.

Quoted 18 lines
replay/replay-internal.c +9 -0
+void replay_mutex_lock(void)
+{
+    qemu_mutex_lock(&lock);
+}
+
+void replay_mutex_unlock(void)
+{
+    qemu_mutex_unlock(&lock);
+}
replay/replay-internal.h +0 -0
index dad5bc8..50ce29b 100755
@@ -34,6 +34,13 @@ int64_t replay_get_qword(void);
 void replay_get_array(uint8_t *buf, size_t *size);
 void replay_get_array_alloc(uint8_t **buf, size_t *size);
 

replay/replay-internal.h int64_t replay_get_qword(void); · line 46

Quoted 10 lines
replay/replay-internal.h +7 -0
+/* Mutex functions for protecting replay log file */
+
+void replay_mutex_init(void);
+void replay_mutex_destroy(void);
+void replay_mutex_lock(void);
+void replay_mutex_unlock(void);
+
 /*! Checks error status of the file. */
 void replay_check_error(void);
 
#14 lore raw

RFC PATCH v8 6/21 replay: introduce icount event

In reply to Pavel Dovgalyuk

This patch adds icount event to the replay subsystem. This event corresponds to execution of several instructions and used to synchronize input events in the replay phase.

---
 replay/replay-internal.c |   16 ++++++++++++++++
 replay/replay-internal.h |   20 ++++++++++++++++++++
 replay/replay.c          |   46 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |    7 +++++++
 4 files changed, 89 insertions(+), 0 deletions(-)
replay/replay-internal.c +16 -0
index f552dad..49b37a6 100755
@@ -10,6 +10,7 @@
  */
 
 #include "qemu-common.h"
+#include "replay.h"
 #include "replay-internal.h"
 
 unsigned int replay_data_kind = -1;
@@ -164,3 +165,18 @@ void replay_mutex_unlock(void)
 {
     qemu_mutex_unlock(&lock);
 }
+
+/*! Saves cached instructions. */
+void replay_save_instructions(void)
+{
+    if (replay_file && replay_mode == REPLAY_MODE_RECORD) {
+        int diff = (int)(replay_get_current_step() - replay_state.current_step);
+        if (first_cpu != NULL && diff > 0) {
+            replay_mutex_lock();
+            replay_put_event(EVENT_INSTRUCTION);
+            replay_put_dword(diff);
+            replay_state.current_step += diff;
+            replay_mutex_unlock();
+        }
+    }
+}
replay/replay-internal.h +20 -0
index 50ce29b..4d661a1 100755
@@ -14,6 +14,19 @@
 
 #include <stdio.h>
 
+enum ReplayEvents {
+    /* for instruction event */
+    EVENT_INSTRUCTION
+};
+
+typedef struct ReplayState {
+    /*! Current step - number of processed instructions and timer events. */
+    uint64_t current_step;
+    /*! Number of instructions to be executed before other events happen. */
+    int instructions_count;
+} ReplayState;
+extern ReplayState replay_state;
+
 extern unsigned int replay_data_kind;
 extern unsigned int replay_has_unread_data;
 
@@ -54,4 +67,11 @@ void replay_save_instructions(void);
     Terminates the program in case of error. */
 void validate_data_kind(int kind);
 
+/*! Skips async events until some sync event will be found. */
+bool skip_async_events(int stop_event);
+/*! Skips async events invocations from the input,
+    until required data kind is found. If the requested data is not found
+    reports an error and stops the execution. */
+void skip_async_events_until(unsigned int kind);
+
 #endif
replay/replay.c +46 -0

52 lines not on this page · raw message

replay/replay.h +7 -0
index d6b73c3..a03c748 100755
@@ -12,8 +12,15 @@
  *
  */
 
+#include <stdbool.h>
+#include <stdint.h>
 #include "qapi-types.h"
 
 extern ReplayMode replay_mode;
 
+/* Processing the instructions */
+
+/*! Returns number of executed instructions. */
+uint64_t replay_get_current_step(void);
+
 #endif
Reviewed-by on v8 6/21
#15 lore raw

Re: [Qemu-devel] [RFC PATCH v8 06/21] replay: introduce icount event

Quoted 147 lines · Pavel Dovgalyuk

This patch adds icount event to the replay subsystem. This event corresponds to execution of several instructions and used to synchronize input events in the replay phase.

---
 replay/replay-internal.c |   16 ++++++++++++++++
 replay/replay-internal.h |   20 ++++++++++++++++++++
 replay/replay.c          |   46 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |    7 +++++++
 4 files changed, 89 insertions(+), 0 deletions(-)
replay/replay-internal.c +16 -0
index f552dad..49b37a6 100755
@@ -10,6 +10,7 @@
  */
 
 #include "qemu-common.h"
+#include "replay.h"
 #include "replay-internal.h"
 
 unsigned int replay_data_kind = -1;
@@ -164,3 +165,18 @@ void replay_mutex_unlock(void)
 {
     qemu_mutex_unlock(&lock);
 }
+
+/*! Saves cached instructions. */
+void replay_save_instructions(void)
+{
+    if (replay_file && replay_mode == REPLAY_MODE_RECORD) {
+        int diff = (int)(replay_get_current_step() - replay_state.current_step);
+        if (first_cpu != NULL && diff > 0) {
+            replay_mutex_lock();
+            replay_put_event(EVENT_INSTRUCTION);
+            replay_put_dword(diff);
+            replay_state.current_step += diff;
+            replay_mutex_unlock();
+        }
+    }
+}
replay/replay-internal.h +20 -0
index 50ce29b..4d661a1 100755
@@ -14,6 +14,19 @@
 
 #include <stdio.h>
 
+enum ReplayEvents {
+    /* for instruction event */
+    EVENT_INSTRUCTION
+};
+
+typedef struct ReplayState {
+    /*! Current step - number of processed instructions and timer events. */
+    uint64_t current_step;
+    /*! Number of instructions to be executed before other events happen. */
+    int instructions_count;
+} ReplayState;
+extern ReplayState replay_state;
+
 extern unsigned int replay_data_kind;
 extern unsigned int replay_has_unread_data;
 
@@ -54,4 +67,11 @@ void replay_save_instructions(void);
     Terminates the program in case of error. */
 void validate_data_kind(int kind);
 
+/*! Skips async events until some sync event will be found. */
+bool skip_async_events(int stop_event);
+/*! Skips async events invocations from the input,
+    until required data kind is found. If the requested data is not found
+    reports an error and stops the execution. */
+void skip_async_events_until(unsigned int kind);
+
 #endif
replay/replay.c +46 -0
index 5ce066f..a43bbbc 100755
@@ -9,6 +9,52 @@
  *
  */
 
+#include "qemu-common.h"
 #include "replay.h"
+#include "replay-internal.h"
+#include "qemu/timer.h"
 
 ReplayMode replay_mode = REPLAY_MODE_NONE;
+
+ReplayState replay_state;
+
+bool skip_async_events(int stop_event)
+{
+    bool res = false;
+
+    /* nothing to skip - not all instructions used */
+    if (replay_state.instructions_count != 0
+        && replay_has_unread_data) {
+        return stop_event == EVENT_INSTRUCTION;
+    }
+
+    while (true) {
+        replay_fetch_data_kind();
+        if (stop_event == replay_data_kind) {
+            res = true;
+        }
+        switch (replay_data_kind) {
+        case EVENT_INSTRUCTION:
+            replay_state.instructions_count = replay_get_dword();
+            return res;
+        default:
+            /* clock, time_t, checkpoint and other events */
+            return res;
+        }
+    }
+    return res;
+}
+
+void skip_async_events_until(unsigned int kind)
+{
+    if (!skip_async_events(kind)) {
+        fprintf(stderr, "%"PRId64": Read data kind %d instead of expected %d\n",
+            replay_get_current_step(), replay_data_kind, kind);
+        exit(1);
+    }
+}
+
+uint64_t replay_get_current_step(void)
+{
+    return cpu_get_icount_raw();
+}
replay/replay.h +2 -0
index d6b73c3..a03c748 100755
@@ -12,8 +12,15 @@
  *
  */
 
+#include <stdbool.h>
+#include <stdint.h>

replay/replay.h line 26

Quoted 10 lines
replay/replay.h +5 -0
 #include "qapi-types.h"
 
 extern ReplayMode replay_mode;
 
+/* Processing the instructions */
+
+/*! Returns number of executed instructions. */
+uint64_t replay_get_current_step(void);
+
 #endif
#16 lore raw

RFC PATCH v8 7/21 cpu-exec: allow temporary disabling icount

In reply to Pavel Dovgalyuk

This patch is required for deterministic replay to generate an exception by trying executing an instruction without changing icount. It adds new flag to TB for disabling icount while translating it.

---
 cpu-exec.c              |    7 ++++---
 include/exec/exec-all.h |    1 +
 translate-all.c         |    2 +-
 3 files changed, 6 insertions(+), 4 deletions(-)
cpu-exec.c +4 -3
index 0a485bb..49f01f5 100644
@@ -200,7 +200,7 @@ static inline tcg_target_ulong cpu_tb_exec(CPUState *cpu, uint8_t *tb_ptr)
 /* Execute the code without caching the generated code. An interpreter
    could be used if available. */
 static void cpu_exec_nocache(CPUArchState *env, int max_cycles,
-                             TranslationBlock *orig_tb)
+                             TranslationBlock *orig_tb, bool ignore_icount)
 {
     CPUState *cpu = ENV_GET_CPU(env);
     TranslationBlock *tb;
@@ -216,7 +216,8 @@ static void cpu_exec_nocache(CPUArchState *env, int max_cycles,
     /* tb_gen_code can flush our orig_tb, invalidate it now */
     tb_phys_invalidate(orig_tb, -1);
     tb = tb_gen_code(cpu, pc, cs_base, flags,
-                     max_cycles | CF_NOCACHE);
+                     max_cycles | CF_NOCACHE
+                         | (ignore_icount ? CF_IGNORE_ICOUNT : 0));
     cpu->current_tb = tb;
     /* execute the generated code */
     trace_exec_tb_nocache(tb, tb->pc);
@@ -525,7 +526,7 @@ int cpu_exec(CPUArchState *env)
                         } else {
                             if (insns_left > 0) {
                                 /* Execute remaining instructions.  */
-                                cpu_exec_nocache(env, insns_left, tb);
+                                cpu_exec_nocache(env, insns_left, tb, false);
                                 align_clocks(&sc, cpu);
                             }
                             cpu->exception_index = EXCP_INTERRUPT;
include/exec/exec-all.h +1 -0
index 63fb1a3..48a7541 100644
@@ -148,6 +148,7 @@ struct TranslationBlock {
 #define CF_LAST_IO     0x8000 /* Last insn may be an IO access.  */
 #define CF_NOCACHE     0x10000 /* To be freed after execution */
 #define CF_USE_ICOUNT  0x20000
+#define CF_IGNORE_ICOUNT 0x40000 /* Do not generate icount code */
 
     void *tc_ptr;    /* pointer to the translated code */
     /* next matching tb for physical address. */
translate-all.c +1 -1
index c0cba7d..49b3403 100644
@@ -1053,7 +1053,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
     int code_gen_size;
 
     phys_pc = get_page_addr_code(env, pc);
-    if (use_icount) {
+    if (use_icount && !(cflags & CF_IGNORE_ICOUNT)) {
         cflags |= CF_USE_ICOUNT;
     }
     tb = tb_alloc(pc);
#17 lore raw

RFC PATCH v8 8/21 cpu: replay instructions sequence

In reply to Pavel Dovgalyuk

This patch adds calls to replay functions into the icount setup block. In record mode number of executed instructions is written to the log. In replay mode number of istructions to execute is taken from the replay log.

---
 cpu-exec.c      |    1 +
 cpus.c          |   28 ++++++++++++++++++----------
 replay/replay.c |   24 ++++++++++++++++++++++++
 replay/replay.h |    4 ++++
 4 files changed, 47 insertions(+), 10 deletions(-)
cpu-exec.c +1 -0
index 49f01f5..99a0993 100644
@@ -531,6 +531,7 @@ int cpu_exec(CPUArchState *env)
                             }
                             cpu->exception_index = EXCP_INTERRUPT;
                             next_tb = 0;
+                            qemu_notify_event();
                             cpu_loop_exit(cpu);
                         }
                         break;
cpus.c +18 -10
index c513275..8787277 100644
@@ -41,6 +41,7 @@
 #include "qemu/seqlock.h"
 #include "qapi-event.h"
 #include "hw/nmi.h"
+#include "replay/replay.h"
 
 #ifndef _WIN32
 #include "qemu/compatfd.h"
@@ -1342,18 +1343,22 @@ static int tcg_cpu_exec(CPUArchState *env)
                                     + cpu->icount_extra);
         cpu->icount_decr.u16.low = 0;
         cpu->icount_extra = 0;
-        deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
+        if (replay_mode != REPLAY_MODE_PLAY) {
+            deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
 
-        /* Maintain prior (possibly buggy) behaviour where if no deadline
-         * was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than
-         * INT32_MAX nanoseconds ahead, we still use INT32_MAX
-         * nanoseconds.
-         */
-        if ((deadline < 0) || (deadline > INT32_MAX)) {
-            deadline = INT32_MAX;
-        }
+            /* Maintain prior (possibly buggy) behaviour where if no deadline
+             * was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than
+             * INT32_MAX nanoseconds ahead, we still use INT32_MAX
+             * nanoseconds.
+             */
+            if ((deadline < 0) || (deadline > INT32_MAX)) {
+                deadline = INT32_MAX;
+            }
 
-        count = qemu_icount_round(deadline);
+            count = qemu_icount_round(deadline);
+        } else {
+            count = replay_get_instructions();
+        }
         timers_state.qemu_icount += count;
         decr = (count > 0xffff) ? 0xffff : count;
         count -= decr;
@@ -1371,6 +1376,9 @@ static int tcg_cpu_exec(CPUArchState *env)
                         + cpu->icount_extra);
         cpu->icount_decr.u32 = 0;
         cpu->icount_extra = 0;
+        if (replay_mode == REPLAY_MODE_PLAY) {
+            replay_exec_instructions();
+        }
     }
     return ret;
 }
replay/replay.c +24 -0
index a43bbbc..d6f5c4b 100755
@@ -58,3 +58,27 @@ uint64_t replay_get_current_step(void)
 {
     return cpu_get_icount_raw();
 }
+
+int replay_get_instructions(void)
+{
+    int res = 0;
+    replay_mutex_lock();
+    if (skip_async_events(EVENT_INSTRUCTION)) {
+        res = replay_state.instructions_count;
+    }
+    replay_mutex_unlock();
+    return res;
+}
+
+void replay_exec_instructions(void)
+{
+    if (replay_state.instructions_count > 0) {
+        int count = (int)(replay_get_current_step()
+                          - replay_state.current_step);
+        replay_state.instructions_count -= count;
+        replay_state.current_step += count;
+        if (replay_state.instructions_count == 0 && count != 0) {
+            replay_has_unread_data = 0;
+        }
+    }
+}
replay/replay.h +4 -0
index a03c748..e425dea 100755
@@ -22,5 +22,9 @@ extern ReplayMode replay_mode;
 
 /*! Returns number of executed instructions. */
 uint64_t replay_get_current_step(void);
+/*! Returns number of instructions to execute in replay mode. */
+int replay_get_instructions(void);
+/*! Updates instructions counter in replay mode. */
+void replay_exec_instructions(void);
 
 #endif
#18 lore raw

Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence

Quoted 17 lines · Pavel Dovgalyuk

This patch adds calls to replay functions into the icount setup block. In record mode number of executed instructions is written to the log. In replay mode number of istructions to execute is taken from the replay log.

---
 cpu-exec.c      |    1 +
 cpus.c          |   28 ++++++++++++++++++----------
 replay/replay.c |   24 ++++++++++++++++++++++++
 replay/replay.h |    4 ++++
 4 files changed, 47 insertions(+), 10 deletions(-)
cpu-exec.c +0 -0
index 49f01f5..99a0993 100644

cpu-exec.c int cpu_exec(CPUArchState *env) · line 534

Quoted 4 lines
cpu-exec.c +1 -0
@@ -531,6 +531,7 @@ int cpu_exec(CPUArchState *env)
                             }
                             cpu->exception_index = EXCP_INTERRUPT;
                             next_tb = 0;
+                            qemu_notify_event();

Why is this needed?

Quoted 35 lines
cpu-exec.c +0 -0
                             cpu_loop_exit(cpu);
                         }
                         break;
cpus.c +7 -9
index c513275..8787277 100644
@@ -41,6 +41,7 @@
 #include "qemu/seqlock.h"
 #include "qapi-event.h"
 #include "hw/nmi.h"
+#include "replay/replay.h"
 
 #ifndef _WIN32
 #include "qemu/compatfd.h"
@@ -1342,18 +1343,22 @@ static int tcg_cpu_exec(CPUArchState *env)
                                     + cpu->icount_extra);
         cpu->icount_decr.u16.low = 0;
         cpu->icount_extra = 0;
-        deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
+        if (replay_mode != REPLAY_MODE_PLAY) {
+            deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
 
-        /* Maintain prior (possibly buggy) behaviour where if no deadline
-         * was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than
-         * INT32_MAX nanoseconds ahead, we still use INT32_MAX
-         * nanoseconds.
-         */
-        if ((deadline < 0) || (deadline > INT32_MAX)) {
-            deadline = INT32_MAX;
-        }
+            /* Maintain prior (possibly buggy) behaviour where if no deadline
+             * was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than
+             * INT32_MAX nanoseconds ahead, we still use INT32_MAX
+             * nanoseconds.

cpus.c static int tcg_cpu_exec(CPUArchState *env) · line 1361

Quoted 10 lines
cpus.c +8 -1
+             */
+            if ((deadline < 0) || (deadline > INT32_MAX)) {
+                deadline = INT32_MAX;
+            }
 
-        count = qemu_icount_round(deadline);
+            count = qemu_icount_round(deadline);
+        } else {
+            count = replay_get_instructions();
+        }

Please extract the "if" to a separate function tcg_get_icount_limit().

cpus.c static int tcg_cpu_exec(CPUArchState *env) · line 1380

Quoted 9 lines
cpus.c +2 -0
         timers_state.qemu_icount += count;
         decr = (count > 0xffff) ? 0xffff : count;
         count -= decr;
@@ -1371,6 +1376,9 @@ static int tcg_cpu_exec(CPUArchState *env)
                         + cpu->icount_extra);
         cpu->icount_decr.u32 = 0;
         cpu->icount_extra = 0;
+        if (replay_mode == REPLAY_MODE_PLAY) {
+            replay_exec_instructions();

replay_account_executed_instructions()

Quoted 22 lines
cpus.c +1 -0
+        }
     }
     return ret;
 }
replay/replay.c +10 -0
index a43bbbc..d6f5c4b 100755
@@ -58,3 +58,27 @@ uint64_t replay_get_current_step(void)
 {
     return cpu_get_icount_raw();
 }
+
+int replay_get_instructions(void)
+{
+    int res = 0;
+    replay_mutex_lock();
+    if (skip_async_events(EVENT_INSTRUCTION)) {
+        res = replay_state.instructions_count;
+    }
+    replay_mutex_unlock();
+    return res;

replay/replay.c uint64_t replay_get_current_step(void) · line 80

Quoted 10 lines
replay/replay.c +10 -0
+}
+
+void replay_exec_instructions(void)
+{
+    if (replay_state.instructions_count > 0) {
+        int count = (int)(replay_get_current_step()
+                          - replay_state.current_step);
+        replay_state.instructions_count -= count;
+        replay_state.current_step += count;
+        if (replay_state.instructions_count == 0 && count != 0) {

If replay_state.instructions_count is now zero, count must be nonzero (because replay_state.instructions_count was > 0) before.

replay/replay.c uint64_t replay_get_current_step(void) · line 82

Quoted 2 lines
replay/replay.c +2 -0
+            replay_has_unread_data = 0;
+        }

Can replay_state.instructions_count be < count at all? If not, and if replay_state.instructions_count is zero, then count must also be zero.

If so, I suggest rewriting as

        int count = (int)(replay_get_current_step()
                          - replay_state.current_step);
        assert(replay_state.instructions_count >= count);
        replay_state.instructions_count -= count;
        replay_state.current_step += count;
        if (replay_state.instructions_count == 0) {
            replay_has_unread_data = 0;
        }

Paolo

Quoted 17 lines
replay/replay.c +2 -0
+    }
+}
replay/replay.h +4 -0
index a03c748..e425dea 100755
@@ -22,5 +22,9 @@ extern ReplayMode replay_mode;
 
 /*! Returns number of executed instructions. */
 uint64_t replay_get_current_step(void);
+/*! Returns number of instructions to execute in replay mode. */
+int replay_get_instructions(void);
+/*! Updates instructions counter in replay mode. */
+void replay_exec_instructions(void);
 
 #endif
#19 lore raw

Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence

Quoted 14 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 12 lines · Pavel Dovgalyuk

This patch adds calls to replay functions into the icount setup block. In record mode number of executed instructions is written to the log. In replay mode number of istructions to execute is taken from the replay log.

---
 cpu-exec.c      |    1 +
 cpus.c          |   28 ++++++++++++++++++----------
 replay/replay.c |   24 ++++++++++++++++++++++++
 replay/replay.h |    4 ++++
 4 files changed, 47 insertions(+), 10 deletions(-)
Quoted 11 lines

cpu-exec.c int cpu_exec(CPUArchState *env) · line 534

Quoted 9 lines
cpu-exec.c +1 -0
index 49f01f5..99a0993 100644
@@ -531,6 +531,7 @@ int cpu_exec(CPUArchState *env)
                             }
                             cpu->exception_index = EXCP_INTERRUPT;
                             next_tb = 0;
+                            qemu_notify_event();

Why is this needed?

It is needed to wake up iothread in replay mode. Otherwise it waits for additional time and replay becomes too slow.

Quoted 4 lines

cpu-exec.c int cpu_exec(CPUArchState *env) · line 537

Quoted 4 lines
cpu-exec.c +0 -0
                             cpu_loop_exit(cpu);
                         }
                         break;

Pavel Dovgalyuk

#20 lore raw

Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence

Quoted 7 lines · Pavel Dovgaluk
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
                             cpu->exception_index = EXCP_INTERRUPT;
                             next_tb = 0;

+ qemu_notify_event();

Why is this needed?

It is needed to wake up iothread in replay mode. Otherwise it waits for additional time and replay becomes too slow.

What event (something from a timerlist?) is ready, that has not been notified to the iothread yet? qemu_notify_event() should never be needed in common case. It's probably missing somewhere else.

Paolo

#21 lore raw

Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence

Quoted 6 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 4 lines · Pavel Dovgaluk
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
                             cpu->exception_index = EXCP_INTERRUPT;
                             next_tb = 0;

+ qemu_notify_event();

Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

Why is this needed?

It is needed to wake up iothread in replay mode. Otherwise it waits for additional time and replay becomes too slow.

What event (something from a timerlist?) is ready, that has not been notified to the iothread yet? qemu_notify_event() should never be needed in common case. It's probably missing somewhere else.

I think in this case there are no events at all - just reading timers values that were made while recording.
We have to replay these reads by waking iothread.

Pavel Dovgalyuk

A message that is not in the archive (sent off-list or not yet received): [email protected]

#23 lore raw

Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence

Quoted 10 lines · Pavel Dovgaluk
Quoted 10 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 7 lines · Pavel Dovgaluk
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
                             cpu->exception_index = EXCP_INTERRUPT;
                             next_tb = 0;

+ qemu_notify_event();

Why is this needed?

It is needed to wake up iothread in replay mode. Otherwise it waits for additional time and replay becomes too slow.

Quoted 7 lines
Quoted 3 lines

What event (something from a timerlist?) is ready, that has not been notified to the iothread yet? qemu_notify_event() should never be needed in common case. It's probably missing somewhere else.

I think in this case there are no events at all - just reading timers values that were made while recording.
We have to replay these reads by waking iothread.

I think the right place for this is in replay_read_next_clock then.

Paolo

#24 lore raw

Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence

Quoted 16 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

Quoted 14 lines · Pavel Dovgaluk
Quoted 13 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 7 lines · Pavel Dovgaluk
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
                             cpu->exception_index = EXCP_INTERRUPT;
                             next_tb = 0;

+ qemu_notify_event();

Why is this needed?

It is needed to wake up iothread in replay mode. Otherwise it waits for additional time and replay becomes too slow.

What event (something from a timerlist?) is ready, that has not been notified to the iothread yet? qemu_notify_event() should never be needed in common case. It's probably missing somewhere else.

Quoted 5 lines
Quoted 3 lines

I think in this case there are no events at all - just reading timers values that were made while recording.
We have to replay these reads by waking iothread.

I think the right place for this is in replay_read_next_clock then.

It doesn't fit. Log file is not read until all instructions are executed. And the next read from the file should be performed by iothread which should be notified and waked up.

Pavel Dovgalyuk

A message that is not in the archive (sent off-list or not yet received): [email protected]

#26 lore raw

Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence

Quoted 8 lines · Pavel Dovgaluk
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

I think in this case there are no events at all - just reading timers values that were made while recording.
We have to replay these reads by waking iothread.

I think the right place for this is in replay_read_next_clock then.

It doesn't fit. Log file is not read until all instructions are executed. And the next read from the file should be performed by iothread which should be notified and waked up.

I still don't understand. If you're getting EXCP_INTERRUPT it means:

1) that cpu_signal was called

2) in turn this means that qemu_cpu_kick was called

3) in turn this means that the iothread is trying to run via qemu_mutex_lock_iothread (the iothread_requesting_mutex stuff). So why do you need an explicit qemu_notify_event?

Paolo

#27 lore raw

Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence

Quoted 6 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

Quoted 4 lines · Pavel Dovgaluk
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines

I think in this case there are no events at all - just reading timers values that were made while recording.
We have to replay these reads by waking iothread.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

I think the right place for this is in replay_read_next_clock then.

It doesn't fit. Log file is not read until all instructions are executed. And the next read from the file should be performed by iothread which should be notified and waked up.

I still don't understand. If you're getting EXCP_INTERRUPT it means:

1) that cpu_signal was called

No, it isn't. That is the branch when icount is expired. And when it is expired in replay mode we have to wake up iothread, because nobody will care about this.

Quoted 6 lines

2) in turn this means that qemu_cpu_kick was called

3) in turn this means that the iothread is trying to run via qemu_mutex_lock_iothread (the iothread_requesting_mutex stuff). So why do you need an explicit qemu_notify_event?

Pavel Dovgalyuk

A message that is not in the archive (sent off-list or not yet received): [email protected]

#29 lore raw

Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence

Quoted 11 lines · Pavel Dovgaluk
Quoted 11 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

Quoted 8 lines · Pavel Dovgaluk
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

I think in this case there are no events at all - just reading timers values that were made while recording.
We have to replay these reads by waking iothread.

I think the right place for this is in replay_read_next_clock then.

It doesn't fit. Log file is not read until all instructions are executed. And the next read from the file should be performed by iothread which should be notified and waked up.

Quoted 7 lines
Quoted 3 lines

I still don't understand. If you're getting EXCP_INTERRUPT it means:

1) that cpu_signal was called

No, it isn't. That is the branch when icount is expired. And when it is expired in replay mode we have to wake up iothread, because nobody will care about this.

Then it's done here in qemu_tcg_cpu_thread_fn:

        if (use_icount) {
            int64_t deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);

            if (deadline == 0) {
                qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
            }
        }

If you need to move the 4 lines inside the if elsewhere, that I guess it's okay.

Paolo

Quoted 11 lines
Quoted 6 lines

2) in turn this means that qemu_cpu_kick was called

3) in turn this means that the iothread is trying to run via qemu_mutex_lock_iothread (the iothread_requesting_mutex stuff). So why do you need an explicit qemu_notify_event?

Pavel Dovgalyuk

#30 lore raw

Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence

Quoted 15 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

Quoted 13 lines · Pavel Dovgaluk
Quoted 13 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

Quoted 8 lines · Pavel Dovgaluk
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

I think in this case there are no events at all - just reading timers values that were made while recording.
We have to replay these reads by waking iothread.

I think the right place for this is in replay_read_next_clock then.

It doesn't fit. Log file is not read until all instructions are executed. And the next read from the file should be performed by iothread which should be notified and waked up.

I still don't understand. If you're getting EXCP_INTERRUPT it means:

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

1) that cpu_signal was called

No, it isn't. That is the branch when icount is expired. And when it is expired in replay mode we have to wake up iothread, because nobody will care about this.

Then it's done here in qemu_tcg_cpu_thread_fn:

Do you mean that I should put iothread notification right here?
Or that this code duplicates my patch?
If the second one then I guess that it doesn't help and I need to make additional checks about it.

Quoted 10 lines
        if (use_icount) {
            int64_t deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);

            if (deadline == 0) {
                qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
            }
        }

If you need to move the 4 lines inside the if elsewhere, that I guess it's okay.

Pavel Dovgalyuk

A message that is not in the archive (sent off-list or not yet received): [email protected]

#32 lore raw

Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence

Quoted 17 lines · Pavel Dovgaluk
Quoted 17 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

Quoted 15 lines · Pavel Dovgaluk
Quoted 14 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

Quoted 8 lines · Pavel Dovgaluk
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

I think in this case there are no events at all - just reading timers values that were made while recording.
We have to replay these reads by waking iothread.

I think the right place for this is in replay_read_next_clock then.

It doesn't fit. Log file is not read until all instructions are executed. And the next read from the file should be performed by iothread which should be notified and waked up.

I still don't understand. If you're getting EXCP_INTERRUPT it means:

1) that cpu_signal was called

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

No, it isn't. That is the branch when icount is expired. And when it is expired in replay mode we have to wake up iothread, because nobody will care about this.

Then it's done here in qemu_tcg_cpu_thread_fn:

Do you mean that I should put iothread notification right here?

It already notifies the iothread.

Quoted 2 lines

Or that this code duplicates my patch?
If the second one then I guess that it doesn't help and I need to make additional checks about it.

Yes. You can modify your patch to do:

     int64_t deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
     if (deadline == 0) {
         qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
     }

instead of qemu_notify_event(), and remove these lines from qemu_tcg_cpu_thread_fn.

Paolo

Quoted 15 lines
Quoted 10 lines
        if (use_icount) {
            int64_t deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);

            if (deadline == 0) {
                qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
            }
        }

If you need to move the 4 lines inside the if elsewhere, that I guess it's okay.

Pavel Dovgalyuk

#33 lore raw

Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence

Quoted 35 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

Quoted 17 lines · Pavel Dovgaluk
Quoted 17 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

Quoted 15 lines · Pavel Dovgaluk
Quoted 14 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

Quoted 8 lines · Pavel Dovgaluk
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

I think in this case there are no events at all - just reading timers values that were made while recording.
We have to replay these reads by waking iothread.

I think the right place for this is in replay_read_next_clock then.

It doesn't fit. Log file is not read until all instructions are executed. And the next read from the file should be performed by iothread which should be notified and waked up.

I still don't understand. If you're getting EXCP_INTERRUPT it means:

1) that cpu_signal was called

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

No, it isn't. That is the branch when icount is expired. And when it is expired in replay mode we have to wake up iothread, because nobody will care about this.

Then it's done here in qemu_tcg_cpu_thread_fn:

Do you mean that I should put iothread notification right here?

It already notifies the iothread.

Quoted 2 lines

Or that this code duplicates my patch?
If the second one then I guess that it doesn't help and I need to make additional checks

about it.

Yes. You can modify your patch to do:

Quoted 7 lines
     int64_t deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
     if (deadline == 0) {
         qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
     }

instead of qemu_notify_event(), and remove these lines from qemu_tcg_cpu_thread_fn.

I tried this one. But there is one problem.
Expiring of the virtual timers is not the only reason of icount expiration in replay mode.
It may be caused by host timers deadline or poll timeout in record mode. In this case qemu_clock_notify(QEMU_CLOCK_VIRTUAL) will not be called in replay mode and we'll waste time for iothread sleeping.

Pavel Dovgalyuk

#34 lore raw

Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence

Quoted 11 lines · Pavel Dovgaluk
Quoted 7 lines
Quoted 7 lines
     int64_t deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
     if (deadline == 0) {
         qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
     }

instead of qemu_notify_event(), and remove these lines from qemu_tcg_cpu_thread_fn.

I tried this one. But there is one problem.
Expiring of the virtual timers is not the only reason of icount expiration in replay mode.
It may be caused by host timers deadline or poll timeout in record mode. In this case qemu_clock_notify(QEMU_CLOCK_VIRTUAL) will not be called in replay mode and we'll waste time for iothread sleeping.

Sure, but unconditional qemu_notify_event() is also wrong. So, to sum up:

- it's okay to move code from qemu_tcg_cpu_thread_fn to cpu-exec.c

- it's okay to add more qemu_clock_notify calls than just qemu_clock_notify(QEMU_CLOCK_VIRTUAL), each with its own condition

- it's better if all these, after being moved to cpu-exec.c, are also extracted in a separate function

- it's not okay to do an unconditional qemu_notify_event() in cpu-exec.c, even if it's under "if (replay_mode != NONE)".

Thanks for your understanding! :)

Paolo

#35 lore raw

Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence

Quoted 19 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 11 lines · Pavel Dovgaluk
Quoted 7 lines
Quoted 7 lines
     int64_t deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
     if (deadline == 0) {
         qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
     }

instead of qemu_notify_event(), and remove these lines from qemu_tcg_cpu_thread_fn.

I tried this one. But there is one problem.
Expiring of the virtual timers is not the only reason of icount expiration in replay mode. It may be caused by host timers deadline or poll timeout in record mode. In this case qemu_clock_notify(QEMU_CLOCK_VIRTUAL) will not be called in replay mode and we'll waste time

for iothread sleeping.

Sure, but unconditional qemu_notify_event() is also wrong. So, to sum up:

- it's okay to move code from qemu_tcg_cpu_thread_fn to cpu-exec.c

Quoted 8 lines

- it's okay to add more qemu_clock_notify calls than just qemu_clock_notify(QEMU_CLOCK_VIRTUAL), each with its own condition

- it's better if all these, after being moved to cpu-exec.c, are also extracted in a separate function

- it's not okay to do an unconditional qemu_notify_event() in cpu-exec.c, even if it's under "if (replay_mode != NONE)".

How can I wake up iothread if there are no pending timers? deadline will (almost) never become zero in my case, because there is another kind of event in the log (not the timer one).
Should I fetch the event and call qemu_notify_event() from replay module?

Quoted 2 lines

Thanks for your understanding! :)

Pavel Dovgalyuk

#36 lore raw

Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence

Quoted 4 lines · Pavel Dovgaluk

How can I wake up iothread if there are no pending timers? deadline will (almost) never become zero in my case, because there is another kind of event in the log (not the timer one).
Should I fetch the event and call qemu_notify_event() from replay module?

Yes, a few mails ago I suggested that (maybe I got the wrong function in the replay module).

Paolo

#37 lore raw

RFC PATCH v8 9/21 replay: interrupts and exceptions

In reply to Pavel Dovgalyuk

This patch includes modifications of common cpu files. All interrupts and exceptions occured during recording are written into the replay log. These events allow correct replaying the execution by kicking cpu thread when one of these events is found in the log.

---
 cpu-exec.c               |   48 +++++++++++++++++++++++++-------
 replay/replay-internal.h |    6 +++-
 replay/replay.c          |   69 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   17 +++++++++++
 target-i386/seg_helper.c |    4 +++
 5 files changed, 132 insertions(+), 12 deletions(-)
cpu-exec.c +37 -11
index 99a0993..16d1faa 100644
@@ -24,6 +24,7 @@
 #include "qemu/atomic.h"
 #include "sysemu/qtest.h"
 #include "qemu/timer.h"
+#include "replay/replay.h"
 
 /* -icount align implementation. */
 
@@ -338,22 +339,25 @@ int cpu_exec(CPUArchState *env)
     /* This must be volatile so it is not trashed by longjmp() */
     volatile bool have_tb_lock = false;
 
+    /* replay_interrupt may need current_cpu */
+    current_cpu = cpu;
+
     if (cpu->halted) {
 #ifdef TARGET_I386
-        if (cpu->interrupt_request & CPU_INTERRUPT_POLL) {
+        if ((cpu->interrupt_request & CPU_INTERRUPT_POLL)
+            && replay_interrupt()) {
             apic_poll_irq(x86_cpu->apic_state);
             cpu_reset_interrupt(cpu, CPU_INTERRUPT_POLL);
         }
 #endif
         if (!cpu_has_work(cpu)) {
+            current_cpu = NULL;
             return EXCP_HALTED;
         }
 
         cpu->halted = 0;
     }
 
-    current_cpu = cpu;
-
     /* As long as current_cpu is null, up to the assignment just above,
      * requests by other threads to exit the execution loop are expected to
      * be issued using the exit_request global. We must make sure that our
@@ -400,10 +404,21 @@ int cpu_exec(CPUArchState *env)
                     cpu->exception_index = -1;
                     break;
 #else
-                    cc->do_interrupt(cpu);
-                    cpu->exception_index = -1;
+                    if (replay_exception()) {
+                        cc->do_interrupt(cpu);
+                        cpu->exception_index = -1;
+                    } else if (!replay_has_interrupt()) {
+                        /* give a chance to iothread in replay mode */
+                        ret = EXCP_INTERRUPT;
+                        break;
+                    }
 #endif
                 }
+            } else if (replay_has_exception()
+                       && cpu->icount_decr.u16.low + cpu->icount_extra == 0) {
+                /* try to cause an exception pending in the log */
+                cpu_exec_nocache(env, 1, tb_find_fast(env), true);
+                break;
             }
 
             next_tb = 0; /* force lookup of first TB */
@@ -419,30 +434,40 @@ int cpu_exec(CPUArchState *env)
                         cpu->exception_index = EXCP_DEBUG;
                         cpu_loop_exit(cpu);
                     }
-                    if (interrupt_request & CPU_INTERRUPT_HALT) {
+                    if (replay_mode == REPLAY_MODE_PLAY
+                        && !replay_has_interrupt()) {
+                        /* Do nothing */
+                    } else if (interrupt_request & CPU_INTERRUPT_HALT) {
+                        replay_interrupt();
                         cpu->interrupt_request &= ~CPU_INTERRUPT_HALT;
                         cpu->halted = 1;
                         cpu->exception_index = EXCP_HLT;
                         cpu_loop_exit(cpu);
                     }
 #if defined(TARGET_I386)
-                    if (interrupt_request & CPU_INTERRUPT_INIT) {
+                    else if (interrupt_request & CPU_INTERRUPT_INIT) {
+                        replay_interrupt();
                         cpu_svm_check_intercept_param(env, SVM_EXIT_INIT, 0);
                         do_cpu_init(x86_cpu);
                         cpu->exception_index = EXCP_HALTED;
                         cpu_loop_exit(cpu);
                     }
 #else
-                    if (interrupt_request & CPU_INTERRUPT_RESET) {
+                    else if (interrupt_request & CPU_INTERRUPT_RESET) {
+                        replay_interrupt();
                         cpu_reset(cpu);
+                        cpu_loop_exit(cpu);
                     }
 #endif
                     /* The target hook has 3 exit conditions:
                        False when the interrupt isn't processed,
                        True when it is, and we should restart on a new TB,
                        and via longjmp via cpu_loop_exit.  */
-                    if (cc->cpu_exec_interrupt(cpu, interrupt_request)) {
-                        next_tb = 0;
+                    else {
+                        replay_interrupt();
+                        if (cc->cpu_exec_interrupt(cpu, interrupt_request)) {
+                            next_tb = 0;
+                        }
                     }
                     /* Don't use the cached interrupt_request value,
                        do_interrupt may have updated the EXITTB flag. */
@@ -453,7 +478,8 @@ int cpu_exec(CPUArchState *env)
                         next_tb = 0;
                     }
                 }
-                if (unlikely(cpu->exit_request)) {
+                if (unlikely(cpu->exit_request
+                             || replay_has_interrupt())) {
                     cpu->exit_request = 0;
                     cpu->exception_index = EXCP_INTERRUPT;
                     cpu_loop_exit(cpu);
replay/replay-internal.h +5 -1
index 4d661a1..64cc3b2 100755
@@ -16,7 +16,11 @@
 
 enum ReplayEvents {
     /* for instruction event */
-    EVENT_INSTRUCTION
+    EVENT_INSTRUCTION,
+    /* for software interrupt */
+    EVENT_INTERRUPT,
+    /* for emulated exceptions */
+    EVENT_EXCEPTION
 };
 
 typedef struct ReplayState {
replay/replay.c +69 -0

72 lines not on this page · raw message

replay/replay.h +17 -0
index e425dea..eb3b0ff 100755
@@ -27,4 +27,21 @@ int replay_get_instructions(void);
 /*! Updates instructions counter in replay mode. */
 void replay_exec_instructions(void);
 
+/* Interrupts and exceptions */
+
+/*! Called by exception handler to write or read
+    exception processing events. */
+bool replay_exception(void);
+/*! Used to determine that exception is pending.
+    Does not proceed to the next event in the log. */
+bool replay_has_exception(void);
+/*! Called by interrupt handlers to write or read
+    interrupt processing events.
+    \return true if interrupt should be processed */
+bool replay_interrupt(void);
+/*! Tries to read interrupt event from the file.
+    Returns true, when interrupt request is pending */
+bool replay_has_interrupt(void);
+
+
 #endif
target-i386/seg_helper.c +4 -0
index fa374d0..9c00e6f 100644
@@ -22,6 +22,7 @@
 #include "qemu/log.h"
 #include "exec/helper-proto.h"
 #include "exec/cpu_ldst.h"
+#include "replay/replay.h"
 
 //#define DEBUG_PCALL
 
@@ -1294,6 +1295,9 @@ bool x86_cpu_exec_interrupt(CPUState *cs, int interrupt_request)
     if (interrupt_request & CPU_INTERRUPT_POLL) {
         cs->interrupt_request &= ~CPU_INTERRUPT_POLL;
         apic_poll_irq(cpu->apic_state);
+        if (replay_mode != REPLAY_MODE_NONE) {
+            return true;
+        }
     }
 #endif
     if (interrupt_request & CPU_INTERRUPT_SIPI) {
#38 lore raw

Re: [Qemu-devel] [RFC PATCH v8 09/21] replay: interrupts and exceptions

replay/replay.c void replay_exec_instructions(void) · line 91

Quoted 4 lines · Pavel Dovgalyuk
replay/replay.c +4 -0
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_save_instructions();
+        replay_put_event(EVENT_EXCEPTION);
+        return true;

Missing mutex lock/unlock.

replay/replay.c void replay_exec_instructions(void) · line 102

Quoted 11 lines
replay/replay.c +11 -0
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        bool res = false;
+        replay_exec_instructions();
+        replay_mutex_lock();
+        if (skip_async_events(EVENT_EXCEPTION)) {
+            replay_has_unread_data = 0;
+            res = true;
+        }
+        replay_mutex_unlock();
+        return res;
+    }

bool res;
replay_exec_instructions();
res = replay_has_exception();

if (res) {
    replay_has_unread_data = 0;
}

return res;

Same for replay_interrupt().

Perhaps worth factoring out two functions replay_cpu_event and replay_has_cpu_event? You choose.

target-i386/seg_helper.c bool x86_cpu_exec_interrupt(CPUState *cs, int interrupt_request) · line 1302

Quoted 10 lines
target-i386/seg_helper.c +3 -0
@@ -1294,6 +1295,9 @@ bool x86_cpu_exec_interrupt(CPUState *cs, int interrupt_request)
     if (interrupt_request & CPU_INTERRUPT_POLL) {
         cs->interrupt_request &= ~CPU_INTERRUPT_POLL;
         apic_poll_irq(cpu->apic_state);
+        if (replay_mode != REPLAY_MODE_NONE) {
+            return true;
+        }
     }
 #endif

Can you explain this? It probably needs a comment.

Paolo

#39 lore raw

Re: [Qemu-devel] [RFC PATCH v8 09/21] replay: interrupts and exceptions

Quoted 8 lines

From: Paolo Bonzini [mailto:[email protected]]

replay/replay.c void replay_exec_instructions(void) · line 91

Quoted 4 lines · Pavel Dovgalyuk
replay/replay.c +4 -0
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_save_instructions();
+        replay_put_event(EVENT_EXCEPTION);
+        return true;

Missing mutex lock/unlock.

I think not. We just have to write number of already executed instructions. This number is not linked to exception event. They could be read in replay while processing some other event.

Quoted 11 lines

target-i386/seg_helper.c bool x86_cpu_exec_interrupt(CPUState *cs, int interrupt_request) · line 1302

Quoted 9 lines
target-i386/seg_helper.c +3 -0
@@ -1294,6 +1295,9 @@ bool x86_cpu_exec_interrupt(CPUState *cs, int interrupt_request)
     if (interrupt_request & CPU_INTERRUPT_POLL) {
         cs->interrupt_request &= ~CPU_INTERRUPT_POLL;
         apic_poll_irq(cpu->apic_state);
+        if (replay_mode != REPLAY_MODE_NONE) {
+            return true;
+        }
     }
 #endif

Can you explain this? It probably needs a comment.

Each function call should process one interrupt at once, because it corresponds to single event in the log.

Pavel Dovgalyuk

A message that is not in the archive (sent off-list or not yet received): [email protected]

#41 lore raw

Re: [Qemu-devel] [RFC PATCH v8 09/21] replay: interrupts and exceptions

Quoted 4 lines · Pavel Dovgaluk

I think not. We just have to write number of already executed instructions. This number is not linked to exception event. They could be read in replay while processing some other event.

I was referring to replay_put_event(EVENT_EXCEPTION) only.

In principle you could run EVENT_EXCEPTION concurrently with other event writes, for example:

+        replay_mutex_lock();
+        replay_put_event(EVENT_CLOCK + kind);
+        replay_put_qword(clock);
+        replay_mutex_unlock();

and get

EVENT_CLOCK + kind (1 byte)
EVENT_EXCEPTION (1 byte)
clock (8 bytes)

in the replay stream.

I know it's really unlikely, perhaps even impossible in the current QEMU architecture that is dominated by the big QEMU lock. But the right thing to do is to lock the mutex around all event writes. Even if the write itself is atomic, it could happen in the middle of another write if you do not lock the mutex.

Paolo

#42 lore raw

RFC PATCH v8 10/21 replay: asynchronous events infrastructure

In reply to Pavel Dovgalyuk

This patch adds module for saving and replaying asynchronous events. These events include network packets, keyboard and mouse input, USB packets, thread pool and bottom halves callbacks.
All events are stored in the queue to be processed at synchronization points such as beginning of TB execution, or checkpoint in the iothread.

---
 replay/Makefile.objs     |    1 
 replay/replay-events.c   |  228 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   33 ++++++-
 replay/replay.h          |    4 +
 4 files changed, 265 insertions(+), 1 deletions(-)
 create mode 100755 replay/replay-events.c
replay/Makefile.objs +1 -0
index 1148f45..56da09c 100755
@@ -1,2 +1,3 @@
 obj-y += replay.o
 obj-y += replay-internal.o
+obj-y += replay-events.o
replay/replay-events.c new file +228 -0

228 lines not on this page · raw message

replay/replay-internal.h +32 -1
index 64cc3b2..1666d6e 100755
@@ -20,9 +20,19 @@ enum ReplayEvents {
     /* for software interrupt */
     EVENT_INTERRUPT,
     /* for emulated exceptions */
-    EVENT_EXCEPTION
+    EVENT_EXCEPTION,
+    /* for async events */
+    EVENT_ASYNC
 };
 
+/* Asynchronous events IDs */
+
+enum ReplayAsyncEventKind {
+    REPLAY_ASYNC_COUNT
+};
+
+typedef enum ReplayAsyncEventKind ReplayAsyncEventKind;
+
 typedef struct ReplayState {
     /*! Current step - number of processed instructions and timer events. */
     uint64_t current_step;
@@ -78,4 +88,25 @@ bool skip_async_events(int stop_event);
     reports an error and stops the execution. */
 void skip_async_events_until(unsigned int kind);
 
+/* Asynchronous events queue */
+
+/*! Initializes events' processing internals */
+void replay_init_events(void);
+/*! Clears internal data structures for events handling */
+void replay_finish_events(void);
+/*! Enables storing events in the queue */
+void replay_enable_events(void);
+/*! Flushes events queue */
+void replay_flush_events(void);
+/*! Clears events list before loading new VM state */
+void replay_clear_events(void);
+/*! Returns true if there are any unsaved events in the queue */
+bool replay_has_events(void);
+/*! Saves events from queue into the file */
+void replay_save_events(int opt);
+/*! Read events from the file into the input queue */
+void replay_read_events(int opt);
+/*! Adds specified async event to the queue */
+void replay_add_event(ReplayAsyncEventKind event_id, void *opaque);
+
 #endif
replay/replay.h +4 -0
index eb3b0ff..31ca3b9 100755
@@ -43,5 +43,9 @@ bool replay_interrupt(void);
     Returns true, when interrupt request is pending */
 bool replay_has_interrupt(void);
 
+/* Asynchronous events queue */
+
+/*! Disables storing events in the queue */
+void replay_disable_events(void);
 
 #endif
#43 lore raw

Re: [Qemu-devel] [RFC PATCH v8 10/21] replay: asynchronous events infrastructure

Quoted 46 lines · Pavel Dovgalyuk

This patch adds module for saving and replaying asynchronous events. These events include network packets, keyboard and mouse input, USB packets, thread pool and bottom halves callbacks.
All events are stored in the queue to be processed at synchronization points such as beginning of TB execution, or checkpoint in the iothread.

---
 replay/Makefile.objs     |    1 
 replay/replay-events.c   |  228 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   33 ++++++-
 replay/replay.h          |    4 +
 4 files changed, 265 insertions(+), 1 deletions(-)
 create mode 100755 replay/replay-events.c
replay/Makefile.objs +1 -0
index 1148f45..56da09c 100755
@@ -1,2 +1,3 @@
 obj-y += replay.o
 obj-y += replay-internal.o
+obj-y += replay-events.o
replay/replay-events.c new file +17 -0
new file mode 100755
index 0000000..dfd5efd
@@ -0,0 +1,228 @@
+/*
+ * replay-events.c
+ *
+ * Copyright (c) 2010-2015 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "replay.h"
+#include "replay-internal.h"
+
+typedef struct Event {
+    ReplayAsyncEventKind event_kind;

replay/replay-events.c line 27

Quoted 10 lines
replay/replay-events.c new file +10 -0
+    void *opaque;
+    void *opaque2;
+    uint64_t id;
+
+    QTAILQ_ENTRY(Event) events;
+} Event;
+
+static QTAILQ_HEAD(, Event) events_list = QTAILQ_HEAD_INITIALIZER(events_list);
+/* Mutex to protect events_list modifications */
+static QemuMutex lock;

Can you just reuse the replay_lock? Otherwise you have to document the lock hierarchy. But if a single coarser lock is enough, that would be nice.

replay/replay-events.c line 30

Quoted 3 lines
replay/replay-events.c new file +3 -0
+static unsigned int read_event_kind = -1;
+static uint64_t read_id = -1;
+static int read_opt = -1;

Please document what "opt" means.

replay/replay-events.c line 42

Quoted 12 lines
replay/replay-events.c new file +12 -0
+
+static bool replay_events_enabled = false;
+
+/* Functions */
+
+static void replay_run_event(Event *event)
+{
+    switch (event->event_kind) {
+    default:
+        fprintf(stderr, "Replay: invalid async event ID (%d) in the queue\n",
+                event->event_kind);
+        exit(1);

Please treat this the same as ferror(). fprintf(stderr) should be replaced by error_report(), which btw takes a string that is not terminated by \n.

Quoted 87 lines
replay/replay-events.c new file +87 -0
+        break;
+    }
+}
+
+void replay_enable_events(void)
+{
+    replay_events_enabled = true;
+}
+
+bool replay_has_events(void)
+{
+    return !QTAILQ_EMPTY(&events_list);
+}
+
+void replay_flush_events(void)
+{
+    qemu_mutex_lock(&lock);
+    while (!QTAILQ_EMPTY(&events_list)) {
+        Event *event = QTAILQ_FIRST(&events_list);
+        qemu_mutex_unlock(&lock);
+        replay_run_event(event);
+        qemu_mutex_lock(&lock);
+        QTAILQ_REMOVE(&events_list, event, events);
+        g_free(event);
+    }
+    qemu_mutex_unlock(&lock);
+}
+
+void replay_disable_events(void)
+{
+    replay_events_enabled = false;
+    /* Flush events queue before waiting of completion */
+    replay_flush_events();
+}
+
+void replay_clear_events(void)
+{
+    qemu_mutex_lock(&lock);
+    while (!QTAILQ_EMPTY(&events_list)) {
+        Event *event = QTAILQ_FIRST(&events_list);
+        QTAILQ_REMOVE(&events_list, event, events);
+
+        g_free(event);
+    }
+    qemu_mutex_unlock(&lock);
+}
+
+static void replay_add_event_internal(ReplayAsyncEventKind event_kind,
+                                      void *opaque,
+                                      void *opaque2, uint64_t id)
+{
+    if (event_kind >= REPLAY_ASYNC_COUNT) {
+        fprintf(stderr, "Replay: invalid async event ID (%d)\n", event_kind);
+        exit(1);
+    }
+    if (!replay_file || replay_mode == REPLAY_MODE_NONE
+        || !replay_events_enabled) {
+        Event e;
+        e.event_kind = event_kind;
+        e.opaque = opaque;
+        e.opaque2 = opaque2;
+        e.id = id;
+        replay_run_event(&e);
+        return;
+    }
+
+    Event *event = g_malloc0(sizeof(Event));
+    event->event_kind = event_kind;
+    event->opaque = opaque;
+    event->opaque2 = opaque2;
+    event->id = id;
+
+    qemu_mutex_lock(&lock);
+    QTAILQ_INSERT_TAIL(&events_list, event, events);
+    qemu_mutex_unlock(&lock);
+}
+
+void replay_add_event(ReplayAsyncEventKind event_kind, void *opaque)
+{
+    replay_add_event_internal(event_kind, opaque, NULL, 0);
+}
+
+/* Called with replay mutex locked */
+void replay_save_events(int opt)
+{
+    qemu_mutex_lock(&lock);
+    while (!QTAILQ_EMPTY(&events_list)) {

replay/replay-events.c line 139

Quoted 10 lines
replay/replay-events.c new file +10 -0
+        Event *event = QTAILQ_FIRST(&events_list);
+        if (replay_mode != REPLAY_MODE_PLAY) {
+            /* put the event into the file */
+            replay_put_event(EVENT_ASYNC);
+            replay_put_byte(opt);
+            replay_put_byte(event->event_kind);
+
+            /* save event-specific data */
+            switch (event->event_kind) {
+            }

Please extract body of "if" to a separate function replay_save_event.

Quoted 22 lines
replay/replay-events.c new file +22 -0
+        }
+
+        qemu_mutex_unlock(&lock);
+        replay_mutex_unlock();
+        replay_run_event(event);
+        replay_mutex_lock();
+        qemu_mutex_lock(&lock);
+        QTAILQ_REMOVE(&events_list, event, events);
+        g_free(event);
+    }
+    qemu_mutex_unlock(&lock);
+}
+
+/* Called with replay mutex locked */
+void replay_read_events(int opt)
+{
+    replay_fetch_data_kind();
+    while (replay_data_kind == EVENT_ASYNC) {
+        if (read_event_kind == -1) {
+            read_opt = replay_get_byte();
+            read_event_kind = replay_get_byte();
+            read_id = -1;

replay/replay-events.c line 171

Quoted 10 lines
replay/replay-events.c new file +10 -0
+            replay_check_error();
+        }
+
+        if (opt != read_opt) {
+            break;
+        }
+        /* Execute some events without searching them in the queue */
+        switch (read_event_kind) {
+        default:
+            fprintf(stderr, "Unknown ID %d of replay event\n", read_event_kind);

Again, please treat this the same as ferror().

Quoted 20 lines
replay/replay-events.c new file +20 -0
+            exit(1);
+            break;
+        }
+
+        qemu_mutex_lock(&lock);
+
+        Event *event = NULL;
+        Event *curr = NULL;
+        QTAILQ_FOREACH(curr, &events_list, events) {
+            if (curr->event_kind == read_event_kind
+                && (read_id == -1 || read_id == curr->id)) {
+                event = curr;
+                break;
+            }
+        }
+
+        if (event) {
+            /* read event-specific reading data */
+
+            QTAILQ_REMOVE(&events_list, event, events);

replay/replay-events.c line 201

Quoted 10 lines
replay/replay-events.c new file +10 -0
+
+            qemu_mutex_unlock(&lock);
+
+            /* reset unread data and other parameters to allow
+               reading other data from the log while
+               running the event */
+            replay_has_unread_data = 0;
+            read_event_kind = -1;
+            read_id = -1;
+            read_opt = -1;

Everything up to here can be extracted to a separate function that returns Event *. Then this function is:

   while (replay_data_kind == EVENT_ASYNC) {
       event = replay_read_event(opt);
       if (!event) {
           break;
       }
       replay_mutex_unlock();
       replay_run_event(event);
       replay_mutex_lock();
       g_free(event);
       replay_fetch_data_kind();

}

Quoted 31 lines
replay/replay-events.c new file +26 -0
+            replay_mutex_unlock();
+            replay_run_event(event);
+            replay_mutex_lock();
+            g_free(event);
+
+            replay_fetch_data_kind();
+        } else {
+            qemu_mutex_unlock(&lock);
+            /* No such event found in the queue */
+            break;
+        }
+    }
+}
+
+void replay_init_events(void)
+{
+    read_event_kind = -1;
+    qemu_mutex_init(&lock);
+}
+
+void replay_finish_events(void)
+{
+    replay_events_enabled = false;
+    replay_clear_events();
+    qemu_mutex_destroy(&lock);
+}
replay/replay-internal.h +0 -0
index 64cc3b2..1666d6e 100755

replay/replay-internal.h enum ReplayEvents { · line 23

Quoted 5 lines
replay/replay-internal.h +1 -1
@@ -20,9 +20,19 @@ enum ReplayEvents {
     /* for software interrupt */
     EVENT_INTERRUPT,
     /* for emulated exceptions */
-    EVENT_EXCEPTION
+    EVENT_EXCEPTION,

Leave a trailing comma, and the patches become nicer. :)

replay/replay-internal.h enum ReplayEvents { · line 25

Quoted 2 lines
replay/replay-internal.h +2 -0
+    /* for async events */
+    EVENT_ASYNC

Perhaps add EVENT_COUNT and add a sanity check on load?

Paolo

Quoted 54 lines
replay/replay-internal.h +29 -0
 };
 
+/* Asynchronous events IDs */
+
+enum ReplayAsyncEventKind {
+    REPLAY_ASYNC_COUNT
+};
+
+typedef enum ReplayAsyncEventKind ReplayAsyncEventKind;
+
 typedef struct ReplayState {
     /*! Current step - number of processed instructions and timer events. */
     uint64_t current_step;
@@ -78,4 +88,25 @@ bool skip_async_events(int stop_event);
     reports an error and stops the execution. */
 void skip_async_events_until(unsigned int kind);
 
+/* Asynchronous events queue */
+
+/*! Initializes events' processing internals */
+void replay_init_events(void);
+/*! Clears internal data structures for events handling */
+void replay_finish_events(void);
+/*! Enables storing events in the queue */
+void replay_enable_events(void);
+/*! Flushes events queue */
+void replay_flush_events(void);
+/*! Clears events list before loading new VM state */
+void replay_clear_events(void);
+/*! Returns true if there are any unsaved events in the queue */
+bool replay_has_events(void);
+/*! Saves events from queue into the file */
+void replay_save_events(int opt);
+/*! Read events from the file into the input queue */
+void replay_read_events(int opt);
+/*! Adds specified async event to the queue */
+void replay_add_event(ReplayAsyncEventKind event_id, void *opaque);
+
 #endif
replay/replay.h +4 -0
index eb3b0ff..31ca3b9 100755
@@ -43,5 +43,9 @@ bool replay_interrupt(void);
     Returns true, when interrupt request is pending */
 bool replay_has_interrupt(void);
 
+/* Asynchronous events queue */
+
+/*! Disables storing events in the queue */
+void replay_disable_events(void);
 
 #endif
#44 lore raw

RFC PATCH v8 11/21 replay: recording and replaying clock ticks

In reply to Pavel Dovgalyuk

Clock ticks are considered as the sources of non-deterministic data for virtual machine. This patch implements saving the clock values when they are acquired (virtual, host clock, rdtsc, and some other timers). When replaying the execution corresponding values are read from log and transfered to the module, which wants to read the values.
Such a design required the clock polling to be synchronized. Sometimes it is not true - e.g. when timeouts for timer lists are checked. In this case we use a cached value of the clock, passing it to the client code.

---
 cpus.c                   |    3 +-
 include/qemu/timer.h     |   10 +++++
 qemu-timer.c             |    7 ++--
 replay/Makefile.objs     |    1 +
 replay/replay-internal.h |   13 +++++++
 replay/replay-time.c     |   84 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   25 ++++++++++++++
 stubs/replay.c           |    9 +++++
 8 files changed, 147 insertions(+), 5 deletions(-)
 create mode 100755 replay/replay-time.c
cpus.c +2 -1
index 8787277..01d89aa 100644
@@ -353,7 +353,8 @@ static void icount_warp_rt(void *opaque)
 
     seqlock_write_lock(&timers_state.vm_clock_seqlock);
     if (runstate_is_running()) {
-        int64_t clock = cpu_get_clock_locked();
+        int64_t clock = REPLAY_CLOCK(REPLAY_CLOCK_VIRTUAL_RT,
+                                     cpu_get_clock_locked());
         int64_t warp_delta;
 
         warp_delta = clock - vm_clock_warp_start;
include/qemu/timer.h +10 -0
index 0666920..0c2472c 100644
@@ -4,6 +4,7 @@
 #include "qemu/typedefs.h"
 #include "qemu-common.h"
 #include "qemu/notify.h"
+#include "replay/replay.h"
 
 /* timers */
 
@@ -760,6 +761,8 @@ int64_t cpu_icount_to_ns(int64_t icount);
 /*******************************************/
 /* host CPU ticks (if available) */
 
+#define cpu_get_real_ticks cpu_get_real_ticks_impl
+
 #if defined(_ARCH_PPC)
 
 static inline int64_t cpu_get_real_ticks(void)
@@ -913,6 +916,13 @@ static inline int64_t cpu_get_real_ticks (void)
 }
 #endif
 
+#undef cpu_get_real_ticks
+
+static inline int64_t cpu_get_real_ticks(void)
+{
+    return REPLAY_CLOCK(REPLAY_CLOCK_REAL_TICKS, cpu_get_real_ticks_impl());
+}
+
 #ifdef CONFIG_PROFILER
 static inline int64_t profile_getclock(void)
 {
qemu-timer.c +4 -3
index 98d9d1b..bc981a2 100644
@@ -25,6 +25,7 @@
 #include "sysemu/sysemu.h"
 #include "monitor/monitor.h"
 #include "ui/console.h"
+#include "replay/replay.h"
 
 #include "hw/hw.h"
 
@@ -566,15 +567,15 @@ int64_t qemu_clock_get_ns(QEMUClockType type)
             return cpu_get_clock();
         }
     case QEMU_CLOCK_HOST:
-        now = get_clock_realtime();
+        now = REPLAY_CLOCK(REPLAY_CLOCK_HOST, get_clock_realtime());
         last = clock->last;
         clock->last = now;
-        if (now < last) {
+        if (now < last && replay_mode == REPLAY_MODE_NONE) {
             notifier_list_notify(&clock->reset_notifiers, &now);
         }
         return now;
     case QEMU_CLOCK_VIRTUAL_RT:
-        return cpu_get_clock();
+        return REPLAY_CLOCK(REPLAY_CLOCK_VIRTUAL_RT, cpu_get_clock());
     }
 }
 
replay/Makefile.objs +1 -0
index 56da09c..257c320 100755
@@ -1,3 +1,4 @@
 obj-y += replay.o
 obj-y += replay-internal.o
 obj-y += replay-events.o
+obj-y += replay-time.o
replay/replay-internal.h +12 -1
index 1666d6e..e906ec3 100755
@@ -22,7 +22,10 @@ enum ReplayEvents {
     /* for emulated exceptions */
     EVENT_EXCEPTION,
     /* for async events */
-    EVENT_ASYNC
+    EVENT_ASYNC,
+    /* for clock read/writes */
+    /* some of grteater codes are reserved for clocks */
+    EVENT_CLOCK
 };
 
 /* Asynchronous events IDs */
@@ -34,6 +37,8 @@ enum ReplayAsyncEventKind {
 typedef enum ReplayAsyncEventKind ReplayAsyncEventKind;
 
 typedef struct ReplayState {
+    /*! Cached clock values. */
+    int64_t cached_clock[REPLAY_CLOCK_COUNT];
     /*! Current step - number of processed instructions and timer events. */
     uint64_t current_step;
     /*! Number of instructions to be executed before other events happen. */
@@ -88,6 +93,12 @@ bool skip_async_events(int stop_event);
     reports an error and stops the execution. */
 void skip_async_events_until(unsigned int kind);
 
+/*! Reads next clock value from the file.
+    If clock kind read from the file is different from the parameter,
+    the value is not used.
+    If the parameter is -1, the clock value is read to the cache anyway. */
+void replay_read_next_clock(unsigned int kind);
+
 /* Asynchronous events queue */
 
 /*! Initializes events' processing internals */
replay/replay-time.c new file +84 -0

84 lines not on this page · raw message

replay/replay.h +25 -0
index 31ca3b9..8a8e7e0 100755
@@ -16,6 +16,18 @@
 #include <stdint.h>
 #include "qapi-types.h"
 
+/* replay clock kinds */
+enum ReplayClockKind {
+    /* rdtsc */
+    REPLAY_CLOCK_REAL_TICKS,
+    /* host_clock */
+    REPLAY_CLOCK_HOST,
+    /* virtual_rt_clock */
+    REPLAY_CLOCK_VIRTUAL_RT,
+    REPLAY_CLOCK_COUNT
+};
+typedef enum ReplayClockKind ReplayClockKind;
+
 extern ReplayMode replay_mode;
 
 /* Processing the instructions */
@@ -43,6 +55,19 @@ bool replay_interrupt(void);
     Returns true, when interrupt request is pending */
 bool replay_has_interrupt(void);
 
+/* Processing clocks and other time sources */
+
+/*! Save the specified clock */
+int64_t replay_save_clock(ReplayClockKind kind, int64_t clock);
+/*! Read the specified clock from the log or return cached data */
+int64_t replay_read_clock(ReplayClockKind kind);
+/*! Saves or reads the clock depending on the current replay mode. */
+#define REPLAY_CLOCK(clock, value)                                      \
+    (replay_mode == REPLAY_MODE_PLAY ? replay_read_clock((clock))       \
+        : replay_mode == REPLAY_MODE_RECORD                             \
+            ? replay_save_clock((clock), (value))                       \
+        : (value))
+
 /* Asynchronous events queue */
 
 /*! Disables storing events in the queue */
stubs/replay.c +9 -0
index 563c777..121bca6 100755
@@ -1,3 +1,12 @@
 #include "replay/replay.h"
 
 ReplayMode replay_mode;
+
+int64_t replay_save_clock(unsigned int kind, int64_t clock)
+{
+}
+
+int64_t replay_read_clock(unsigned int kind)
+{
+    return 0;
+}
#45 lore raw

Re: [Qemu-devel] [RFC PATCH v8 11/21] replay: recording and replaying clock ticks

Quoted 58 lines · Pavel Dovgalyuk

Clock ticks are considered as the sources of non-deterministic data for virtual machine. This patch implements saving the clock values when they are acquired (virtual, host clock, rdtsc, and some other timers). When replaying the execution corresponding values are read from log and transfered to the module, which wants to read the values.
Such a design required the clock polling to be synchronized. Sometimes it is not true - e.g. when timeouts for timer lists are checked. In this case we use a cached value of the clock, passing it to the client code.

---
 cpus.c                   |    3 +-
 include/qemu/timer.h     |   10 +++++
 qemu-timer.c             |    7 ++--
 replay/Makefile.objs     |    1 +
 replay/replay-internal.h |   13 +++++++
 replay/replay-time.c     |   84 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   25 ++++++++++++++
 stubs/replay.c           |    9 +++++
 8 files changed, 147 insertions(+), 5 deletions(-)
 create mode 100755 replay/replay-time.c
cpus.c +2 -1
index 8787277..01d89aa 100644
@@ -353,7 +353,8 @@ static void icount_warp_rt(void *opaque)
 
     seqlock_write_lock(&timers_state.vm_clock_seqlock);
     if (runstate_is_running()) {
-        int64_t clock = cpu_get_clock_locked();
+        int64_t clock = REPLAY_CLOCK(REPLAY_CLOCK_VIRTUAL_RT,
+                                     cpu_get_clock_locked());
         int64_t warp_delta;
 
         warp_delta = clock - vm_clock_warp_start;
include/qemu/timer.h +3 -0
index 0666920..0c2472c 100644
@@ -4,6 +4,7 @@
 #include "qemu/typedefs.h"
 #include "qemu-common.h"
 #include "qemu/notify.h"
+#include "replay/replay.h"
 
 /* timers */
 
@@ -760,6 +761,8 @@ int64_t cpu_icount_to_ns(int64_t icount);
 /*******************************************/
 /* host CPU ticks (if available) */
 
+#define cpu_get_real_ticks cpu_get_real_ticks_impl
+
 #if defined(_ARCH_PPC)
 
 static inline int64_t cpu_get_real_ticks(void)

include/qemu/timer.h static inline int64_t cpu_get_real_ticks (void) · line 921

Quoted 6 lines
include/qemu/timer.h +3 -0
@@ -913,6 +916,13 @@ static inline int64_t cpu_get_real_ticks (void)
 }
 #endif
 
+#undef cpu_get_real_ticks
+
+static inline int64_t cpu_get_real_ticks(void)

cpu_get_real_ticks should never be used. Please instead wrap cpu_get_ticks() with REPLAY_CLOCK.

Quoted 73 lines
include/qemu/timer.h +4 -0
+{
+    return REPLAY_CLOCK(REPLAY_CLOCK_REAL_TICKS, cpu_get_real_ticks_impl());
+}
+
 #ifdef CONFIG_PROFILER
 static inline int64_t profile_getclock(void)
 {
qemu-timer.c +4 -3
index 98d9d1b..bc981a2 100644
@@ -25,6 +25,7 @@
 #include "sysemu/sysemu.h"
 #include "monitor/monitor.h"
 #include "ui/console.h"
+#include "replay/replay.h"
 
 #include "hw/hw.h"
 
@@ -566,15 +567,15 @@ int64_t qemu_clock_get_ns(QEMUClockType type)
             return cpu_get_clock();
         }
     case QEMU_CLOCK_HOST:
-        now = get_clock_realtime();
+        now = REPLAY_CLOCK(REPLAY_CLOCK_HOST, get_clock_realtime());
         last = clock->last;
         clock->last = now;
-        if (now < last) {
+        if (now < last && replay_mode == REPLAY_MODE_NONE) {
             notifier_list_notify(&clock->reset_notifiers, &now);
         }
         return now;
     case QEMU_CLOCK_VIRTUAL_RT:
-        return cpu_get_clock();
+        return REPLAY_CLOCK(REPLAY_CLOCK_VIRTUAL_RT, cpu_get_clock());
     }
 }
 
replay/Makefile.objs +1 -0
index 56da09c..257c320 100755
@@ -1,3 +1,4 @@
 obj-y += replay.o
 obj-y += replay-internal.o
 obj-y += replay-events.o
+obj-y += replay-time.o
replay/replay-internal.h +6 -1
index 1666d6e..e906ec3 100755
@@ -22,7 +22,10 @@ enum ReplayEvents {
     /* for emulated exceptions */
     EVENT_EXCEPTION,
     /* for async events */
-    EVENT_ASYNC
+    EVENT_ASYNC,
+    /* for clock read/writes */
+    /* some of grteater codes are reserved for clocks */
+    EVENT_CLOCK
 };
 
 /* Asynchronous events IDs */
@@ -34,6 +37,8 @@ enum ReplayAsyncEventKind {
 typedef enum ReplayAsyncEventKind ReplayAsyncEventKind;
 
 typedef struct ReplayState {
+    /*! Cached clock values. */
+    int64_t cached_clock[REPLAY_CLOCK_COUNT];
     /*! Current step - number of processed instructions and timer events. */
     uint64_t current_step;
     /*! Number of instructions to be executed before other events happen. */

replay/replay-internal.h bool skip_async_events(int stop_event); · line 99

Quoted 7 lines
replay/replay-internal.h +4 -0
@@ -88,6 +93,12 @@ bool skip_async_events(int stop_event);
     reports an error and stops the execution. */
 void skip_async_events_until(unsigned int kind);
 
+/*! Reads next clock value from the file.
+    If clock kind read from the file is different from the parameter,
+    the value is not used.
+    If the parameter is -1, the clock value is read to the cache anyway. */

In what case could the clock kind not match?

Paolo

Quoted 156 lines
replay/replay-internal.h +2 -0
+void replay_read_next_clock(unsigned int kind);
+
 /* Asynchronous events queue */
 
 /*! Initializes events' processing internals */
replay/replay-time.c new file +84 -0
new file mode 100755
index 0000000..5f5bc6a
@@ -0,0 +1,84 @@
+/*
+ * replay-time.c
+ *
+ * Copyright (c) 2010-2015 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "replay.h"
+#include "replay-internal.h"
+
+
+int64_t replay_save_clock(ReplayClockKind kind, int64_t clock)
+{
+    replay_save_instructions();
+
+    if (kind >= REPLAY_CLOCK_COUNT) {
+        fprintf(stderr, "invalid clock ID %d for replay\n", kind);
+        exit(1);
+    }
+
+    if (replay_file) {
+        replay_mutex_lock();
+        replay_put_event(EVENT_CLOCK + kind);
+        replay_put_qword(clock);
+        replay_mutex_unlock();
+    }
+
+    return clock;
+}
+
+void replay_read_next_clock(ReplayClockKind kind)
+{
+    replay_fetch_data_kind();
+    if (replay_file) {
+        unsigned int read_kind = replay_data_kind - EVENT_CLOCK;
+
+        if (kind != -1 && read_kind != kind) {
+            return;
+        }
+        if (read_kind >= REPLAY_CLOCK_COUNT) {
+            fprintf(stderr,
+                    "invalid clock ID %d was read from replay\n", read_kind);
+            exit(1);
+        }
+
+        int64_t clock = replay_get_qword();
+
+        replay_check_error();
+        replay_has_unread_data = 0;
+
+        replay_state.cached_clock[read_kind] = clock;
+    }
+}
+
+/*! Reads next clock event from the input. */
+int64_t replay_read_clock(ReplayClockKind kind)
+{
+    if (kind >= REPLAY_CLOCK_COUNT) {
+        fprintf(stderr, "invalid clock ID %d for replay\n", kind);
+        exit(1);
+    }
+
+    replay_exec_instructions();
+
+    if (replay_file) {
+        int64_t ret;
+        replay_mutex_lock();
+        if (skip_async_events(EVENT_CLOCK + kind)) {
+            replay_read_next_clock(kind);
+        }
+        ret = replay_state.cached_clock[kind];
+        replay_mutex_unlock();
+
+        return ret;
+    }
+
+    fprintf(stderr, "REPLAY INTERNAL ERROR %d\n", __LINE__);
+    exit(1);
+}
replay/replay.h +25 -0
index 31ca3b9..8a8e7e0 100755
@@ -16,6 +16,18 @@
 #include <stdint.h>
 #include "qapi-types.h"
 
+/* replay clock kinds */
+enum ReplayClockKind {
+    /* rdtsc */
+    REPLAY_CLOCK_REAL_TICKS,
+    /* host_clock */
+    REPLAY_CLOCK_HOST,
+    /* virtual_rt_clock */
+    REPLAY_CLOCK_VIRTUAL_RT,
+    REPLAY_CLOCK_COUNT
+};
+typedef enum ReplayClockKind ReplayClockKind;
+
 extern ReplayMode replay_mode;
 
 /* Processing the instructions */
@@ -43,6 +55,19 @@ bool replay_interrupt(void);
     Returns true, when interrupt request is pending */
 bool replay_has_interrupt(void);
 
+/* Processing clocks and other time sources */
+
+/*! Save the specified clock */
+int64_t replay_save_clock(ReplayClockKind kind, int64_t clock);
+/*! Read the specified clock from the log or return cached data */
+int64_t replay_read_clock(ReplayClockKind kind);
+/*! Saves or reads the clock depending on the current replay mode. */
+#define REPLAY_CLOCK(clock, value)                                      \
+    (replay_mode == REPLAY_MODE_PLAY ? replay_read_clock((clock))       \
+        : replay_mode == REPLAY_MODE_RECORD                             \
+            ? replay_save_clock((clock), (value))                       \
+        : (value))
+
 /* Asynchronous events queue */
 
 /*! Disables storing events in the queue */
stubs/replay.c +9 -0
index 563c777..121bca6 100755
@@ -1,3 +1,12 @@
 #include "replay/replay.h"
 
 ReplayMode replay_mode;
+
+int64_t replay_save_clock(unsigned int kind, int64_t clock)
+{
+}
+
+int64_t replay_read_clock(unsigned int kind)
+{
+    return 0;
+}
#46 lore raw

Re: [Qemu-devel] [RFC PATCH v8 11/21] replay: recording and replaying clock ticks

Quoted 63 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 61 lines · Pavel Dovgalyuk

Clock ticks are considered as the sources of non-deterministic data for virtual machine. This patch implements saving the clock values when they are acquired (virtual, host clock, rdtsc, and some other timers). When replaying the execution corresponding values are read from log and transfered to the module, which wants to read the values.
Such a design required the clock polling to be synchronized. Sometimes it is not true - e.g. when timeouts for timer lists are checked. In this case we use a cached value of the clock, passing it to the client code.

---
 cpus.c                   |    3 +-
 include/qemu/timer.h     |   10 +++++
 qemu-timer.c             |    7 ++--
 replay/Makefile.objs     |    1 +
 replay/replay-internal.h |   13 +++++++
 replay/replay-time.c     |   84 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   25 ++++++++++++++
 stubs/replay.c           |    9 +++++
 8 files changed, 147 insertions(+), 5 deletions(-)
 create mode 100755 replay/replay-time.c
cpus.c +2 -1
index 8787277..01d89aa 100644
@@ -353,7 +353,8 @@ static void icount_warp_rt(void *opaque)
 
     seqlock_write_lock(&timers_state.vm_clock_seqlock);
     if (runstate_is_running()) {
-        int64_t clock = cpu_get_clock_locked();
+        int64_t clock = REPLAY_CLOCK(REPLAY_CLOCK_VIRTUAL_RT,
+                                     cpu_get_clock_locked());
         int64_t warp_delta;
 
         warp_delta = clock - vm_clock_warp_start;
include/qemu/timer.h +3 -0
index 0666920..0c2472c 100644
@@ -4,6 +4,7 @@
 #include "qemu/typedefs.h"
 #include "qemu-common.h"
 #include "qemu/notify.h"
+#include "replay/replay.h"
 
 /* timers */
 
@@ -760,6 +761,8 @@ int64_t cpu_icount_to_ns(int64_t icount);
 /*******************************************/
 /* host CPU ticks (if available) */
 
+#define cpu_get_real_ticks cpu_get_real_ticks_impl
+
 #if defined(_ARCH_PPC)
 
 static inline int64_t cpu_get_real_ticks(void)
@@ -913,6 +916,13 @@ static inline int64_t cpu_get_real_ticks (void)
 }
 #endif
 
Quoted 6 lines

include/qemu/timer.h static inline int64_t cpu_get_real_ticks (void) · line 921

Quoted 3 lines
include/qemu/timer.h +3 -0
+#undef cpu_get_real_ticks
+
+static inline int64_t cpu_get_real_ticks(void)

cpu_get_real_ticks should never be used. Please instead wrap cpu_get_ticks() with REPLAY_CLOCK.

I don't quite understand this comment.
Do you mean that I should move REPLAY_CLOCK to the cpu_get_real_ticks usages instead of it's implementation?

Quoted 77 lines
Quoted 77 lines
include/qemu/timer.h +4 -0
+{
+    return REPLAY_CLOCK(REPLAY_CLOCK_REAL_TICKS, cpu_get_real_ticks_impl());
+}
+
 #ifdef CONFIG_PROFILER
 static inline int64_t profile_getclock(void)
 {
qemu-timer.c +4 -3
index 98d9d1b..bc981a2 100644
@@ -25,6 +25,7 @@
 #include "sysemu/sysemu.h"
 #include "monitor/monitor.h"
 #include "ui/console.h"
+#include "replay/replay.h"
 
 #include "hw/hw.h"
 
@@ -566,15 +567,15 @@ int64_t qemu_clock_get_ns(QEMUClockType type)
             return cpu_get_clock();
         }
     case QEMU_CLOCK_HOST:
-        now = get_clock_realtime();
+        now = REPLAY_CLOCK(REPLAY_CLOCK_HOST, get_clock_realtime());
         last = clock->last;
         clock->last = now;
-        if (now < last) {
+        if (now < last && replay_mode == REPLAY_MODE_NONE) {
             notifier_list_notify(&clock->reset_notifiers, &now);
         }
         return now;
     case QEMU_CLOCK_VIRTUAL_RT:
-        return cpu_get_clock();
+        return REPLAY_CLOCK(REPLAY_CLOCK_VIRTUAL_RT, cpu_get_clock());
     }
 }
 
replay/Makefile.objs +1 -0
index 56da09c..257c320 100755
@@ -1,3 +1,4 @@
 obj-y += replay.o
 obj-y += replay-internal.o
 obj-y += replay-events.o
+obj-y += replay-time.o
replay/replay-internal.h +6 -1
index 1666d6e..e906ec3 100755
@@ -22,7 +22,10 @@ enum ReplayEvents {
     /* for emulated exceptions */
     EVENT_EXCEPTION,
     /* for async events */
-    EVENT_ASYNC
+    EVENT_ASYNC,
+    /* for clock read/writes */
+    /* some of grteater codes are reserved for clocks */
+    EVENT_CLOCK
 };
 
 /* Asynchronous events IDs */
@@ -34,6 +37,8 @@ enum ReplayAsyncEventKind {
 typedef enum ReplayAsyncEventKind ReplayAsyncEventKind;
 
 typedef struct ReplayState {
+    /*! Cached clock values. */
+    int64_t cached_clock[REPLAY_CLOCK_COUNT];
     /*! Current step - number of processed instructions and timer events. */
     uint64_t current_step;
     /*! Number of instructions to be executed before other events happen. */
@@ -88,6 +93,12 @@ bool skip_async_events(int stop_event);
     reports an error and stops the execution. */
 void skip_async_events_until(unsigned int kind);
 
Quoted 7 lines

replay/replay-internal.h bool skip_async_events(int stop_event); · line 99

Quoted 4 lines
replay/replay-internal.h +4 -0
+/*! Reads next clock value from the file.
+    If clock kind read from the file is different from the parameter,
+    the value is not used.
+    If the parameter is -1, the clock value is read to the cache anyway. */

In what case could the clock kind not match?

It was used in full version which had to skip clock from the log while loading the VM state.

Quoted 5 lines

replay/replay-internal.h bool skip_async_events(int stop_event); · line 103

Quoted 5 lines
replay/replay-internal.h +2 -0
+void replay_read_next_clock(unsigned int kind);
+
 /* Asynchronous events queue */
 

Pavel Dovgalyuk

#47 lore raw

Re: [Qemu-devel] [RFC PATCH v8 11/21] replay: recording and replaying clock ticks

Quoted 70 lines · Pavel Dovgaluk
Quoted 63 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 61 lines · Pavel Dovgalyuk

Clock ticks are considered as the sources of non-deterministic data for virtual machine. This patch implements saving the clock values when they are acquired (virtual, host clock, rdtsc, and some other timers). When replaying the execution corresponding values are read from log and transfered to the module, which wants to read the values.
Such a design required the clock polling to be synchronized. Sometimes it is not true - e.g. when timeouts for timer lists are checked. In this case we use a cached value of the clock, passing it to the client code.

---
 cpus.c                   |    3 +-
 include/qemu/timer.h     |   10 +++++
 qemu-timer.c             |    7 ++--
 replay/Makefile.objs     |    1 +
 replay/replay-internal.h |   13 +++++++
 replay/replay-time.c     |   84 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   25 ++++++++++++++
 stubs/replay.c           |    9 +++++
 8 files changed, 147 insertions(+), 5 deletions(-)
 create mode 100755 replay/replay-time.c
cpus.c +2 -1
index 8787277..01d89aa 100644
@@ -353,7 +353,8 @@ static void icount_warp_rt(void *opaque)
 
     seqlock_write_lock(&timers_state.vm_clock_seqlock);
     if (runstate_is_running()) {
-        int64_t clock = cpu_get_clock_locked();
+        int64_t clock = REPLAY_CLOCK(REPLAY_CLOCK_VIRTUAL_RT,
+                                     cpu_get_clock_locked());
         int64_t warp_delta;
 
         warp_delta = clock - vm_clock_warp_start;
include/qemu/timer.h +3 -0
index 0666920..0c2472c 100644
@@ -4,6 +4,7 @@
 #include "qemu/typedefs.h"
 #include "qemu-common.h"
 #include "qemu/notify.h"
+#include "replay/replay.h"
 
 /* timers */
 
@@ -760,6 +761,8 @@ int64_t cpu_icount_to_ns(int64_t icount);
 /*******************************************/
 /* host CPU ticks (if available) */
 
+#define cpu_get_real_ticks cpu_get_real_ticks_impl
+
 #if defined(_ARCH_PPC)
 
 static inline int64_t cpu_get_real_ticks(void)
@@ -913,6 +916,13 @@ static inline int64_t cpu_get_real_ticks (void)
 }
 #endif
 
Quoted 6 lines

include/qemu/timer.h static inline int64_t cpu_get_real_ticks (void) · line 921

Quoted 3 lines
include/qemu/timer.h +3 -0
+#undef cpu_get_real_ticks
+
+static inline int64_t cpu_get_real_ticks(void)

cpu_get_real_ticks should never be used. Please instead wrap cpu_get_ticks() with REPLAY_CLOCK.

Quoted 2 lines

I don't quite understand this comment.
Do you mean that I should move REPLAY_CLOCK to the cpu_get_real_ticks usages instead of it's implementation?

Only to the cpu_get_ticks usage. The others are okay.

Quoted 9 lines
Quoted 7 lines

include/qemu/timer.h static inline int64_t cpu_get_real_ticks (void) · line 925

Quoted 4 lines
include/qemu/timer.h +4 -0
+/*! Reads next clock value from the file.
+    If clock kind read from the file is different from the parameter,
+    the value is not used.
+    If the parameter is -1, the clock value is read to the cache anyway. */

In what case could the clock kind not match?

It was used in full version which had to skip clock from the log while loading the VM state.

So can it be removed for now?

Paolo

Quoted 9 lines
Quoted 5 lines

include/qemu/timer.h static inline int64_t cpu_get_real_ticks (void) · line 928

Quoted 5 lines
include/qemu/timer.h +2 -0
+void replay_read_next_clock(unsigned int kind);
+
 /* Asynchronous events queue */

Pavel Dovgalyuk

#48 lore raw

Re: [Qemu-devel] [RFC PATCH v8 11/21] replay: recording and replaying clock ticks

Quoted 72 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 70 lines · Pavel Dovgaluk
Quoted 69 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 58 lines · Pavel Dovgalyuk

Clock ticks are considered as the sources of non-deterministic data for virtual machine. This patch implements saving the clock values when they are acquired (virtual, host clock, rdtsc, and some other timers). When replaying the execution corresponding values are read from log and transfered to the module, which wants to read the values.
Such a design required the clock polling to be synchronized. Sometimes it is not true - e.g. when timeouts for timer lists are checked. In this case we use a cached value of the clock, passing it to the client code.

---
 cpus.c                   |    3 +-
 include/qemu/timer.h     |   10 +++++
 qemu-timer.c             |    7 ++--
 replay/Makefile.objs     |    1 +
 replay/replay-internal.h |   13 +++++++
 replay/replay-time.c     |   84 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   25 ++++++++++++++
 stubs/replay.c           |    9 +++++
 8 files changed, 147 insertions(+), 5 deletions(-)
 create mode 100755 replay/replay-time.c
cpus.c +2 -1
index 8787277..01d89aa 100644
@@ -353,7 +353,8 @@ static void icount_warp_rt(void *opaque)
 
     seqlock_write_lock(&timers_state.vm_clock_seqlock);
     if (runstate_is_running()) {
-        int64_t clock = cpu_get_clock_locked();
+        int64_t clock = REPLAY_CLOCK(REPLAY_CLOCK_VIRTUAL_RT,
+                                     cpu_get_clock_locked());
         int64_t warp_delta;
 
         warp_delta = clock - vm_clock_warp_start;
include/qemu/timer.h +3 -0
index 0666920..0c2472c 100644
@@ -4,6 +4,7 @@
 #include "qemu/typedefs.h"
 #include "qemu-common.h"
 #include "qemu/notify.h"
+#include "replay/replay.h"
 
 /* timers */
 
@@ -760,6 +761,8 @@ int64_t cpu_icount_to_ns(int64_t icount);
 /*******************************************/
 /* host CPU ticks (if available) */
 
+#define cpu_get_real_ticks cpu_get_real_ticks_impl
+
 #if defined(_ARCH_PPC)
 
 static inline int64_t cpu_get_real_ticks(void)

include/qemu/timer.h static inline int64_t cpu_get_real_ticks (void) · line 921

Quoted 6 lines
include/qemu/timer.h +3 -0
@@ -913,6 +916,13 @@ static inline int64_t cpu_get_real_ticks (void)
 }
 #endif
 
+#undef cpu_get_real_ticks
+
+static inline int64_t cpu_get_real_ticks(void)

cpu_get_real_ticks should never be used. Please instead wrap cpu_get_ticks() with REPLAY_CLOCK.

Quoted 5 lines
Quoted 2 lines

I don't quite understand this comment.
Do you mean that I should move REPLAY_CLOCK to the cpu_get_real_ticks usages instead of it's

implementation?

Only to the cpu_get_ticks usage. The others are okay.

cpu_get_ticks cannot call cpu_get_real_ticks in icount mode.
And other functions can. Then we should put REPLAY_CLOCK into those functions?

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
include/qemu/timer.h +4 -0
+/*! Reads next clock value from the file.
+    If clock kind read from the file is different from the parameter,
+    the value is not used.
+    If the parameter is -1, the clock value is read to the cache anyway. */
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

In what case could the clock kind not match?

It was used in full version which had to skip clock from the log while loading the VM state.

So can it be removed for now?

I think it can.

Pavel Dovgalyuk

#49 lore raw

Re: [Qemu-devel] [RFC PATCH v8 11/21] replay: recording and replaying clock ticks

Quoted 3 lines · Pavel Dovgaluk
Quoted 1 line

Only to the cpu_get_ticks usage. The others are okay.

cpu_get_ticks cannot call cpu_get_real_ticks in icount mode.

You're right, but...

Quoted 1 line

And other functions can.

... which functions? cpu_enable_ticks and cpu_disable_ticks call it, but the result is only used by cpu_get_ticks. As you said, this cannot be called in icount mode.

A couple of PPC device models call it, but they probably should call cpu_get_ticks instead. So it looks like handling REAL_TICKS is altogether unnecessary.

Paolo

#50 lore raw

RFC PATCH v8 12/21 replay: recording and replaying different timers

In reply to Pavel Dovgalyuk

This patch introduces functions for recording and replaying realtime sources, that do not use qemu-clock interface. These include return value of time() function in time_t and struct tm forms. Patch also adds warning to get_timedate function to prevent its usage in recording mode, because it may lead to non-determinism.

---
 hw/timer/mc146818rtc.c   |    3 +
 hw/timer/pl031.c         |    3 +
 include/qemu-common.h    |    1 
 replay/replay-internal.h |    4 +
 replay/replay-time.c     |  132 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |    8 +++
 vl.c                     |   17 +++++-
 7 files changed, 163 insertions(+), 5 deletions(-)
hw/timer/mc146818rtc.c +2 -1
index f18d128..92295fb 100644
@@ -28,6 +28,7 @@
 #include "qapi/visitor.h"
 #include "qapi-event.h"
 #include "qmp-commands.h"
+#include "replay/replay.h"
 
 #ifdef TARGET_I386
 #include "hw/i386/apic.h"
@@ -703,7 +704,7 @@ static void rtc_set_date_from_host(ISADevice *dev)
     RTCState *s = MC146818_RTC(dev);
     struct tm tm;
 
-    qemu_get_timedate(&tm, 0);
+    replay_get_timedate(&tm);
 
     s->base_rtc = mktimegm(&tm);
     s->last_update = qemu_clock_get_ns(rtc_clock);
hw/timer/pl031.c +2 -1
index 34d9b44..b02d288 100644
@@ -14,6 +14,7 @@
 #include "hw/sysbus.h"
 #include "qemu/timer.h"
 #include "sysemu/sysemu.h"
+#include "replay/replay.h"
 
 //#define DEBUG_PL031
 
@@ -200,7 +201,7 @@ static int pl031_init(SysBusDevice *dev)
     sysbus_init_mmio(dev, &s->iomem);
 
     sysbus_init_irq(dev, &s->irq);
-    qemu_get_timedate(&tm, 0);
+    replay_get_timedate(&tm);
     s->tick_offset = mktimegm(&tm) -
         qemu_clock_get_ns(rtc_clock) / get_ticks_per_sec();
 
include/qemu-common.h +1 -0
index 644b46d..8c9957e 100644
@@ -129,6 +129,7 @@ void dump_drift_info(FILE *f, fprintf_function cpu_fprintf);
 int qemu_main(int argc, char **argv, char **envp);
 #endif
 
+void qemu_get_timedate_no_warning(struct tm *tm, int offset);
 void qemu_get_timedate(struct tm *tm, int offset);
 int qemu_timedate_diff(struct tm *tm);
 
replay/replay-internal.h +4 -0
index e906ec3..be71e6f 100755
@@ -23,6 +23,10 @@ enum ReplayEvents {
     EVENT_EXCEPTION,
     /* for async events */
     EVENT_ASYNC,
+    /* for time_t event */
+    EVENT_TIME_T,
+    /* for tm event */
+    EVENT_TM,
     /* for clock read/writes */
     /* some of grteater codes are reserved for clocks */
     EVENT_CLOCK
replay/replay-time.c +132 -0

135 lines not on this page · raw message

replay/replay.h +8 -0
index 8a8e7e0..2f1402c 100755
@@ -14,6 +14,7 @@
 
 #include <stdbool.h>
 #include <stdint.h>
+#include <time.h>
 #include "qapi-types.h"
 
 /* replay clock kinds */
@@ -67,6 +68,13 @@ int64_t replay_read_clock(ReplayClockKind kind);
         : replay_mode == REPLAY_MODE_RECORD                             \
             ? replay_save_clock((clock), (value))                       \
         : (value))
+/*! Returns result of time() function execution in normal and record modes.
+    In play mode returns value read from the log. */
+time_t replay_time(void);
+/*! This function is used for obtaining current timedate.
+    In record mode it also saves in into the replay log.
+    In replay mode it just read from the log. */
+void replay_get_timedate(struct tm *tm);
 
 /* Asynchronous events queue */
 
vl.c +14 -3
index fbf4240..baff3b5 100644
@@ -118,6 +118,7 @@ int main(int argc, char **argv)
 #include "qapi/opts-visitor.h"
 #include "qom/object_interfaces.h"
 #include "qapi-event.h"
+#include "replay/replay.h"
 
 #define DEFAULT_RAM_SIZE 128
 
@@ -715,7 +716,7 @@ void vm_start(void)
 
 /***********************************************************/
 /* host time/date access */
-void qemu_get_timedate(struct tm *tm, int offset)
+void qemu_get_timedate_no_warning(struct tm *tm, int offset)
 {
     time_t ti;
 
@@ -732,6 +733,16 @@ void qemu_get_timedate(struct tm *tm, int offset)
     }
 }
 
+/* host time/date access with replay warning */
+void qemu_get_timedate(struct tm *tm, int offset)
+{
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        fprintf(stderr, "REPLAY WARNING! qemu_get_timedate "
+                        "function may lead to non-determinism\n");
+    }
+    qemu_get_timedate_no_warning(tm, offset);
+}
+
 int qemu_timedate_diff(struct tm *tm)
 {
     time_t seconds;
@@ -747,7 +758,7 @@ int qemu_timedate_diff(struct tm *tm)
     else
         seconds = mktimegm(tm) + rtc_date_offset;
 
-    return seconds - time(NULL);
+    return seconds - replay_time();
 }
 
 static void configure_rtc_date_offset(const char *startdate, int legacy)
@@ -785,7 +796,7 @@ static void configure_rtc_date_offset(const char *startdate, int legacy)
                             "'2006-06-17T16:01:21' or '2006-06-17'\n");
             exit(1);
         }
-        rtc_date_offset = time(NULL) - rtc_start_date;
+        rtc_date_offset = replay_time() - rtc_start_date;
     }
 }
 
#51 lore raw

Re: [Qemu-devel] [RFC PATCH v8 12/21] replay: recording and replaying different timers

Quoted 30 lines · Pavel Dovgalyuk

This patch introduces functions for recording and replaying realtime sources, that do not use qemu-clock interface. These include return value of time() function in time_t and struct tm forms. Patch also adds warning to get_timedate function to prevent its usage in recording mode, because it may lead to non-determinism.

---
 hw/timer/mc146818rtc.c   |    3 +
 hw/timer/pl031.c         |    3 +
 include/qemu-common.h    |    1 
 replay/replay-internal.h |    4 +
 replay/replay-time.c     |  132 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |    8 +++
 vl.c                     |   17 +++++-
 7 files changed, 163 insertions(+), 5 deletions(-)
hw/timer/mc146818rtc.c +1 -0
index f18d128..92295fb 100644
@@ -28,6 +28,7 @@
 #include "qapi/visitor.h"
 #include "qapi-event.h"
 #include "qmp-commands.h"
+#include "replay/replay.h"
 
 #ifdef TARGET_I386
 #include "hw/i386/apic.h"

hw/timer/mc146818rtc.c static void rtc_set_date_from_host(ISADevice *dev) · line 706

Quoted 4 lines
hw/timer/mc146818rtc.c +0 -1
@@ -703,7 +704,7 @@ static void rtc_set_date_from_host(ISADevice *dev)
     RTCState *s = MC146818_RTC(dev);
     struct tm tm;
 
-    qemu_get_timedate(&tm, 0);

What about just making qemu_get_timedate use qemu_clock_get_ns(QEMU_CLOCK_HOST) instead of time()? This would just work using the infrastructure of the previous patch.

Paolo

Quoted 275 lines
hw/timer/mc146818rtc.c +1 -0
+    replay_get_timedate(&tm);
 
     s->base_rtc = mktimegm(&tm);
     s->last_update = qemu_clock_get_ns(rtc_clock);
hw/timer/pl031.c +2 -1
index 34d9b44..b02d288 100644
@@ -14,6 +14,7 @@
 #include "hw/sysbus.h"
 #include "qemu/timer.h"
 #include "sysemu/sysemu.h"
+#include "replay/replay.h"
 
 //#define DEBUG_PL031
 
@@ -200,7 +201,7 @@ static int pl031_init(SysBusDevice *dev)
     sysbus_init_mmio(dev, &s->iomem);
 
     sysbus_init_irq(dev, &s->irq);
-    qemu_get_timedate(&tm, 0);
+    replay_get_timedate(&tm);
     s->tick_offset = mktimegm(&tm) -
         qemu_clock_get_ns(rtc_clock) / get_ticks_per_sec();
 
include/qemu-common.h +1 -0
index 644b46d..8c9957e 100644
@@ -129,6 +129,7 @@ void dump_drift_info(FILE *f, fprintf_function cpu_fprintf);
 int qemu_main(int argc, char **argv, char **envp);
 #endif
 
+void qemu_get_timedate_no_warning(struct tm *tm, int offset);
 void qemu_get_timedate(struct tm *tm, int offset);
 int qemu_timedate_diff(struct tm *tm);
 
replay/replay-internal.h +4 -0
index e906ec3..be71e6f 100755
@@ -23,6 +23,10 @@ enum ReplayEvents {
     EVENT_EXCEPTION,
     /* for async events */
     EVENT_ASYNC,
+    /* for time_t event */
+    EVENT_TIME_T,
+    /* for tm event */
+    EVENT_TM,
     /* for clock read/writes */
     /* some of grteater codes are reserved for clocks */
     EVENT_CLOCK
replay/replay-time.c +132 -0
index 5f5bc6a..282d00e 100755
@@ -82,3 +82,135 @@ int64_t replay_read_clock(ReplayClockKind kind)
     fprintf(stderr, "REPLAY INTERNAL ERROR %d\n", __LINE__);
     exit(1);
 }
+
+/*! Saves time_t value to the log */
+static void replay_save_time_t(time_t tm)
+{
+    replay_save_instructions();
+
+    if (replay_file) {
+        replay_mutex_lock();
+        replay_put_event(EVENT_TIME_T);
+        if (sizeof(tm) == 4) {
+            replay_put_dword(tm);
+        } else if (sizeof(tm) == 8) {
+            replay_put_qword(tm);
+        } else {
+            fprintf(stderr, "invalid time_t sizeof: %u\n",
+                    (unsigned)sizeof(tm));
+            exit(1);
+        }
+        replay_mutex_unlock();
+    }
+}
+
+/*! Reads time_t value from the log. Stops execution in case of error */
+static time_t replay_read_time_t(void)
+{
+    replay_exec_instructions();
+
+    if (replay_file) {
+        time_t tm;
+
+        replay_mutex_lock();
+        skip_async_events_until(EVENT_TIME_T);
+
+        if (sizeof(tm) == 4) {
+            tm = replay_get_dword();
+        } else if (sizeof(tm) == 8) {
+            tm = replay_get_qword();
+        } else {
+            fprintf(stderr, "invalid time_t sizeof: %u\n",
+                    (unsigned)sizeof(tm));
+            exit(1);
+        }
+
+        replay_check_error();
+
+        replay_has_unread_data = 0;
+        replay_mutex_unlock();
+
+        return tm;
+    }
+
+    fprintf(stderr, "REPLAY INTERNAL ERROR %d\n", __LINE__);
+    exit(1);
+}
+
+static void replay_save_tm(struct tm *tm)
+{
+    replay_save_instructions();
+
+    if (replay_file) {
+        replay_mutex_lock();
+        replay_put_event(EVENT_TM);
+
+        replay_put_dword(tm->tm_sec);
+        replay_put_dword(tm->tm_min);
+        replay_put_dword(tm->tm_hour);
+        replay_put_dword(tm->tm_mday);
+        replay_put_dword(tm->tm_mon);
+        replay_put_dword(tm->tm_year);
+        replay_put_dword(tm->tm_wday);
+        replay_put_dword(tm->tm_yday);
+        replay_put_dword(tm->tm_isdst);
+        replay_mutex_unlock();
+    }
+}
+
+static void replay_read_tm(struct tm *tm)
+{
+    replay_exec_instructions();
+
+    if (replay_file) {
+        replay_mutex_lock();
+        skip_async_events_until(EVENT_TM);
+
+        tm->tm_sec = replay_get_dword();
+        tm->tm_min = replay_get_dword();
+        tm->tm_hour = replay_get_dword();
+        tm->tm_mday = replay_get_dword();
+        tm->tm_mon = replay_get_dword();
+        tm->tm_year = replay_get_dword();
+        tm->tm_wday = replay_get_dword();
+        tm->tm_yday = replay_get_dword();
+        tm->tm_isdst = replay_get_dword();
+
+        replay_check_error();
+        replay_has_unread_data = 0;
+        replay_mutex_unlock();
+
+        return;
+    }
+
+    fprintf(stderr, "REPLAY INTERNAL ERROR %d\n", __LINE__);
+    exit(1);
+}
+
+void replay_get_timedate(struct tm *tm)
+{
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        qemu_get_timedate_no_warning(tm, 0);
+        replay_save_tm(tm);
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        replay_read_tm(tm);
+    } else {
+        qemu_get_timedate_no_warning(tm, 0);
+    }
+}
+
+time_t replay_time(void)
+{
+    time_t systime;
+
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        systime = time(NULL);
+        replay_save_time_t(systime);
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        systime = replay_read_time_t();
+    } else {
+        systime = time(NULL);
+    }
+
+    return systime;
+}
replay/replay.h +8 -0
index 8a8e7e0..2f1402c 100755
@@ -14,6 +14,7 @@
 
 #include <stdbool.h>
 #include <stdint.h>
+#include <time.h>
 #include "qapi-types.h"
 
 /* replay clock kinds */
@@ -67,6 +68,13 @@ int64_t replay_read_clock(ReplayClockKind kind);
         : replay_mode == REPLAY_MODE_RECORD                             \
             ? replay_save_clock((clock), (value))                       \
         : (value))
+/*! Returns result of time() function execution in normal and record modes.
+    In play mode returns value read from the log. */
+time_t replay_time(void);
+/*! This function is used for obtaining current timedate.
+    In record mode it also saves in into the replay log.
+    In replay mode it just read from the log. */
+void replay_get_timedate(struct tm *tm);
 
 /* Asynchronous events queue */
 
vl.c +14 -3
index fbf4240..baff3b5 100644
@@ -118,6 +118,7 @@ int main(int argc, char **argv)
 #include "qapi/opts-visitor.h"
 #include "qom/object_interfaces.h"
 #include "qapi-event.h"
+#include "replay/replay.h"
 
 #define DEFAULT_RAM_SIZE 128
 
@@ -715,7 +716,7 @@ void vm_start(void)
 
 /***********************************************************/
 /* host time/date access */
-void qemu_get_timedate(struct tm *tm, int offset)
+void qemu_get_timedate_no_warning(struct tm *tm, int offset)
 {
     time_t ti;
 
@@ -732,6 +733,16 @@ void qemu_get_timedate(struct tm *tm, int offset)
     }
 }
 
+/* host time/date access with replay warning */
+void qemu_get_timedate(struct tm *tm, int offset)
+{
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        fprintf(stderr, "REPLAY WARNING! qemu_get_timedate "
+                        "function may lead to non-determinism\n");
+    }
+    qemu_get_timedate_no_warning(tm, offset);
+}
+
 int qemu_timedate_diff(struct tm *tm)
 {
     time_t seconds;
@@ -747,7 +758,7 @@ int qemu_timedate_diff(struct tm *tm)
     else
         seconds = mktimegm(tm) + rtc_date_offset;
 
-    return seconds - time(NULL);
+    return seconds - replay_time();
 }
 
 static void configure_rtc_date_offset(const char *startdate, int legacy)
@@ -785,7 +796,7 @@ static void configure_rtc_date_offset(const char *startdate, int legacy)
                             "'2006-06-17T16:01:21' or '2006-06-17'\n");
             exit(1);
         }
-        rtc_date_offset = time(NULL) - rtc_start_date;
+        rtc_date_offset = replay_time() - rtc_start_date;
     }
 }
 
#52 lore raw

Re: [Qemu-devel] [RFC PATCH v8 12/21] replay: recording and replaying different timers

Quoted 35 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 33 lines · Pavel Dovgalyuk

This patch introduces functions for recording and replaying realtime sources, that do not use qemu-clock interface. These include return value of time() function in time_t and struct tm forms. Patch also adds warning to get_timedate function to prevent its usage in recording mode, because it may lead to non-determinism.

---
 hw/timer/mc146818rtc.c   |    3 +
 hw/timer/pl031.c         |    3 +
 include/qemu-common.h    |    1
 replay/replay-internal.h |    4 +
 replay/replay-time.c     |  132 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |    8 +++
 vl.c                     |   17 +++++-
 7 files changed, 163 insertions(+), 5 deletions(-)
hw/timer/mc146818rtc.c +1 -0
index f18d128..92295fb 100644
@@ -28,6 +28,7 @@
 #include "qapi/visitor.h"
 #include "qapi-event.h"
 #include "qmp-commands.h"
+#include "replay/replay.h"
 
 #ifdef TARGET_I386
 #include "hw/i386/apic.h"
@@ -703,7 +704,7 @@ static void rtc_set_date_from_host(ISADevice *dev)
     RTCState *s = MC146818_RTC(dev);
     struct tm tm;
 
Quoted 5 lines

hw/timer/mc146818rtc.c static void rtc_set_date_from_host(ISADevice *dev) · line 706

Quoted 1 line
hw/timer/mc146818rtc.c +0 -1
-    qemu_get_timedate(&tm, 0);

What about just making qemu_get_timedate use qemu_clock_get_ns(QEMU_CLOCK_HOST) instead of time()? This would just work using the infrastructure of the previous patch.

I can get rid of these calls, but localtime() function used in qemu_get_timedate() will not work deterministically. Should we save its' result then?

Pavel Dovgalyuk

#53 lore raw

Re: [Qemu-devel] [RFC PATCH v8 12/21] replay: recording and replaying different timers

Quoted 7 lines · Pavel Dovgaluk
Quoted 4 lines
Quoted 4 lines

What about just making qemu_get_timedate use qemu_clock_get_ns(QEMU_CLOCK_HOST) instead of time()? This would just work using the infrastructure of the previous patch.

I can get rid of these calls, but localtime() function used in qemu_get_timedate() will not work deterministically. Should we save its' result then?

It is deterministic if the user's timezone remains the same. You could just outlaw "-rtc base=localtime".

Also, if you choose to record the output of qemu_get_timedate(), you have to do the same for qemu_timedate_diff(). Switching qemu_timedate_diff to use QEMU_CLOCK_HOST would also work for qemu_timedate_diff().

Paolo

#54 lore raw

RFC PATCH v8 13/21 replay: shutdown event

In reply to Pavel Dovgalyuk

This patch records and replays simulator shutdown event.

---
 replay/replay-internal.h |    2 ++
 replay/replay.c          |   12 ++++++++++++
 replay/replay.h          |    5 +++++
 vl.c                     |    1 +
 4 files changed, 20 insertions(+), 0 deletions(-)
replay/replay-internal.h +2 -0
index be71e6f..68b2d45 100755
@@ -27,6 +27,8 @@ enum ReplayEvents {
     EVENT_TIME_T,
     /* for tm event */
     EVENT_TM,
+    /* for shutdown request */
+    EVENT_SHUTDOWN,
     /* for clock read/writes */
     /* some of grteater codes are reserved for clocks */
     EVENT_CLOCK
replay/replay.c +12 -0
index 307ac4b..cfa69fa 100755
@@ -13,6 +13,7 @@
 #include "replay.h"
 #include "replay-internal.h"
 #include "qemu/timer.h"
+#include "sysemu/sysemu.h"
 
 ReplayMode replay_mode = REPLAY_MODE_NONE;
 
@@ -34,6 +35,10 @@ bool skip_async_events(int stop_event)
             res = true;
         }
         switch (replay_data_kind) {
+        case EVENT_SHUTDOWN:
+            replay_has_unread_data = 0;
+            qemu_system_shutdown_request();
+            break;
         case EVENT_INSTRUCTION:
             replay_state.instructions_count = replay_get_dword();
             return res;
@@ -151,3 +156,10 @@ bool replay_has_interrupt(void)
     }
     return res;
 }
+
+void replay_shutdown_request(void)
+{
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_put_event(EVENT_SHUTDOWN);
+    }
+}
replay/replay.h +5 -0
index 2f1402c..e1c5fcf 100755
@@ -76,6 +76,11 @@ time_t replay_time(void);
     In replay mode it just read from the log. */
 void replay_get_timedate(struct tm *tm);
 
+/* Events */
+
+/*! Called when qemu shutdown is requested. */
+void replay_shutdown_request(void);
+
 /* Asynchronous events queue */
 
 /*! Disables storing events in the queue */
vl.c +1 -0
index baff3b5..905ea8a 100644
@@ -1720,6 +1720,7 @@ void qemu_system_killed(int signal, pid_t pid)
 void qemu_system_shutdown_request(void)
 {
     trace_qemu_system_shutdown_request();
+    replay_shutdown_request();
     shutdown_requested = 1;
     qemu_notify_event();
 }
Reviewed-by on v8 13/21
#55 lore raw

Re: [Qemu-devel] [RFC PATCH v8 13/21] replay: shutdown event

Quoted 81 lines · Pavel Dovgalyuk

This patch records and replays simulator shutdown event.

---
 replay/replay-internal.h |    2 ++
 replay/replay.c          |   12 ++++++++++++
 replay/replay.h          |    5 +++++
 vl.c                     |    1 +
 4 files changed, 20 insertions(+), 0 deletions(-)
replay/replay-internal.h +2 -0
index be71e6f..68b2d45 100755
@@ -27,6 +27,8 @@ enum ReplayEvents {
     EVENT_TIME_T,
     /* for tm event */
     EVENT_TM,
+    /* for shutdown request */
+    EVENT_SHUTDOWN,
     /* for clock read/writes */
     /* some of grteater codes are reserved for clocks */
     EVENT_CLOCK
replay/replay.c +12 -0
index 307ac4b..cfa69fa 100755
@@ -13,6 +13,7 @@
 #include "replay.h"
 #include "replay-internal.h"
 #include "qemu/timer.h"
+#include "sysemu/sysemu.h"
 
 ReplayMode replay_mode = REPLAY_MODE_NONE;
 
@@ -34,6 +35,10 @@ bool skip_async_events(int stop_event)
             res = true;
         }
         switch (replay_data_kind) {
+        case EVENT_SHUTDOWN:
+            replay_has_unread_data = 0;
+            qemu_system_shutdown_request();
+            break;
         case EVENT_INSTRUCTION:
             replay_state.instructions_count = replay_get_dword();
             return res;
@@ -151,3 +156,10 @@ bool replay_has_interrupt(void)
     }
     return res;
 }
+
+void replay_shutdown_request(void)
+{
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_put_event(EVENT_SHUTDOWN);
+    }
+}
replay/replay.h +5 -0
index 2f1402c..e1c5fcf 100755
@@ -76,6 +76,11 @@ time_t replay_time(void);
     In replay mode it just read from the log. */
 void replay_get_timedate(struct tm *tm);
 
+/* Events */
+
+/*! Called when qemu shutdown is requested. */
+void replay_shutdown_request(void);
+
 /* Asynchronous events queue */
 
 /*! Disables storing events in the queue */
vl.c +0 -0
index baff3b5..905ea8a 100644

vl.c void qemu_system_killed(int signal, pid_t pid) · line 1726

Quoted 7 lines
vl.c +1 -0
@@ -1720,6 +1720,7 @@ void qemu_system_killed(int signal, pid_t pid)
 void qemu_system_shutdown_request(void)
 {
     trace_qemu_system_shutdown_request();
+    replay_shutdown_request();
     shutdown_requested = 1;
     qemu_notify_event();
 }
#56 lore raw

RFC PATCH v8 14/21 replay: checkpoints

In reply to Pavel Dovgalyuk

This patch introduces checkpoints that synchronize cpu thread and iothread. When checkpoint is met in the code all asynchronous events from the queue are executed.

---
 block.c                  |   11 ++++++++++
 cpus.c                   |    7 ++++++-
 include/qemu/timer.h     |    6 ++++--
 main-loop.c              |    5 +++++
 qemu-timer.c             |   49 ++++++++++++++++++++++++++++++++++++++--------
 replay/replay-internal.h |    5 ++++-
 replay/replay.c          |   36 ++++++++++++++++++++++++++++++++++
 replay/replay.h          |   21 ++++++++++++++++++++
 stubs/replay.c           |   11 ++++++++++
 vl.c                     |    4 +++-
 10 files changed, 142 insertions(+), 13 deletions(-)
block.c +11 -0
index cbe4a32..a4f45c3 100644
@@ -1994,6 +1994,11 @@ void bdrv_drain_all(void)
     BlockDriverState *bs;
 
     while (busy) {
+        if (!replay_checkpoint(CHECKPOINT_BDRV_DRAIN)) {
+            /* Do not wait anymore, we stopped at some place in
+               the middle of execution during replay */
+            return;
+        }
         busy = false;
 
         QTAILQ_FOREACH(bs, &bdrv_states, device_list) {
@@ -2004,6 +2009,12 @@ void bdrv_drain_all(void)
             aio_context_release(aio_context);
         }
     }
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        /* Skip checkpoints from the log */
+        while (replay_checkpoint(CHECKPOINT_BDRV_DRAIN)) {
+            /* Nothing */
+        }
+    }
 }
 
 /* make a BlockDriverState anonymous by removing from bdrv_state and
cpus.c +6 -1
index 01d89aa..9c32491 100644
@@ -388,7 +388,7 @@ void qtest_clock_warp(int64_t dest)
         timers_state.qemu_icount_bias += warp;
         seqlock_write_unlock(&timers_state.vm_clock_seqlock);
 
-        qemu_clock_run_timers(QEMU_CLOCK_VIRTUAL);
+        qemu_clock_run_timers(QEMU_CLOCK_VIRTUAL, false);
         clock = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
     }
     qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
@@ -408,6 +408,11 @@ void qemu_clock_warp(QEMUClockType type)
         return;
     }
 
+    /* warp clock deterministically in record/replay mode */
+    if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP)) {
+        return;
+    }
+
     /*
      * If the CPUs have been sleeping, advance QEMU_CLOCK_VIRTUAL timer now.
      * This ensures that the deadline for the timer is computed correctly below.
include/qemu/timer.h +4 -2
index 0c2472c..26927b0 100644
@@ -240,13 +240,14 @@ void qemu_clock_unregister_reset_notifier(QEMUClockType type,
 /**
  * qemu_clock_run_timers:
  * @type: clock on which to operate
+ * @run_all: true, when called from qemu_clock_run_all_timers
  *
  * Run all the timers associated with the default timer list
  * of a clock.
  *
  * Returns: true if any timer ran.
  */
-bool qemu_clock_run_timers(QEMUClockType type);
+bool qemu_clock_run_timers(QEMUClockType type, bool run_all);
 
 /**
  * qemu_clock_run_all_timers:
@@ -337,12 +338,13 @@ QEMUClockType timerlist_get_clock(QEMUTimerList *timer_list);
 /**
  * timerlist_run_timers:
  * @timer_list: the timer list to use
+ * @run_all: true, when called from qemu_clock_run_all_timers
  *
  * Call all expired timers associated with the timer list.
  *
  * Returns: true if any timer expired
  */
-bool timerlist_run_timers(QEMUTimerList *timer_list);
+bool timerlist_run_timers(QEMUTimerList *timer_list, bool run_all);
 
 /**
  * timerlist_notify:
main-loop.c +5 -0
index 981bcb5..d6e93c3 100644
@@ -497,6 +497,11 @@ int main_loop_wait(int nonblocking)
     slirp_pollfds_poll(gpollfds, (ret < 0));
 #endif
 
+    /* CPU thread can infinitely wait for event after
+       missing the warp */
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
+    }
     qemu_clock_run_all_timers();
 
     return ret;
qemu-timer.c +41 -8

88 lines not on this page · raw message

replay/replay-internal.h +4 -1
index 68b2d45..c0a5800 100755
@@ -31,7 +31,10 @@ enum ReplayEvents {
     EVENT_SHUTDOWN,
     /* for clock read/writes */
     /* some of grteater codes are reserved for clocks */
-    EVENT_CLOCK
+    EVENT_CLOCK,
+    /* for checkpoint event */
+    /* some of grteater codes are reserved for checkpoints */
+    EVENT_CHECKPOINT = EVENT_CLOCK + REPLAY_CLOCK_COUNT
 };
 
 /* Asynchronous events IDs */
replay/replay.c +36 -0
index cfa69fa..7c4a801 100755
@@ -163,3 +163,39 @@ void replay_shutdown_request(void)
         replay_put_event(EVENT_SHUTDOWN);
     }
 }
+
+bool replay_checkpoint(ReplayCheckpoint checkpoint)
+{
+    bool res = false;
+    replay_save_instructions();
+
+    if (replay_file) {
+        if (replay_mode == REPLAY_MODE_PLAY) {
+            replay_mutex_lock();
+            if (!skip_async_events(EVENT_CHECKPOINT + checkpoint)) {
+                if (replay_data_kind == EVENT_ASYNC) {
+                    replay_read_events(checkpoint);
+                    replay_fetch_data_kind();
+                    res = replay_data_kind != EVENT_ASYNC;
+                    replay_mutex_unlock();
+                    return res;
+                }
+                replay_mutex_unlock();
+                return res;
+            }
+            replay_has_unread_data = 0;
+            replay_read_events(checkpoint);
+            replay_fetch_data_kind();
+            res = replay_data_kind != EVENT_ASYNC;
+            replay_mutex_unlock();
+            return res;
+        } else if (replay_mode == REPLAY_MODE_RECORD) {
+            replay_mutex_lock();
+            replay_put_event(EVENT_CHECKPOINT + checkpoint);
+            replay_save_events(checkpoint);
+            replay_mutex_unlock();
+        }
+    }
+
+    return true;
+}
replay/replay.h +21 -0
index e1c5fcf..39822b4 100755
@@ -29,6 +29,21 @@ enum ReplayClockKind {
 };
 typedef enum ReplayClockKind ReplayClockKind;
 
+/* IDs of the checkpoints */
+enum ReplayCheckpoint {
+    CHECKPOINT_BDRV_DRAIN,
+    CHECKPOINT_CLOCK_WARP,
+    CHECKPOINT_RESET_REQUESTED,
+    CHECKPOINT_CLOCK_VIRTUAL,
+    CHECKPOINT_CLOCK_VIRTUAL_ALL,
+    CHECKPOINT_CLOCK_HOST,
+    CHECKPOINT_CLOCK_HOST_ALL,
+    CHECKPOINT_CLOCK_VIRTUAL_RT,
+    CHECKPOINT_CLOCK_VIRTUAL_RT_ALL,
+    CHECKPOINT_COUNT
+};
+typedef enum ReplayCheckpoint ReplayCheckpoint;
+
 extern ReplayMode replay_mode;
 
 /* Processing the instructions */
@@ -80,6 +95,12 @@ void replay_get_timedate(struct tm *tm);
 
 /*! Called when qemu shutdown is requested. */
 void replay_shutdown_request(void);
+/*! Should be called at check points in the execution.
+    These check points are skipped, if they were not met.
+    Saves checkpoint in the SAVE mode and validates in the PLAY mode.
+    Returns 0 in PLAY mode if checkpoint was not found.
+    Returns 1 in all other cases. */
+bool replay_checkpoint(ReplayCheckpoint checkpoint);
 
 /* Asynchronous events queue */
 
stubs/replay.c +11 -0
index 121bca6..1be3575 100755
@@ -1,4 +1,5 @@
 #include "replay/replay.h"
+#include "sysemu/sysemu.h"
 
 ReplayMode replay_mode;
 
@@ -10,3 +11,13 @@ int64_t replay_read_clock(unsigned int kind)
 {
     return 0;
 }
+
+bool replay_checkpoint(ReplayCheckpoint checkpoint)
+{
+    return 0;
+}
+
+int runstate_is_running(void)
+{
+    return 0;
+}
vl.c +3 -1
index 905ea8a..86ba385 100644
@@ -1767,7 +1767,9 @@ static bool main_loop_should_exit(void)
             return true;
         }
     }
-    if (qemu_reset_requested()) {
+    if (qemu_reset_requested_get()
+        && replay_checkpoint(CHECKPOINT_RESET_REQUESTED)) {
+        qemu_reset_requested();
         pause_all_vcpus();
         cpu_synchronize_all_states();
         qemu_system_reset(VMRESET_REPORT);
#57 lore raw

Re: [Qemu-devel] [RFC PATCH v8 14/21] replay: checkpoints

Quoted 5 lines · Pavel Dovgalyuk

This patch introduces checkpoints that synchronize cpu thread and iothread. When checkpoint is met in the code all asynchronous events from the queue are executed.

I cannot understand this patch.

Please document the design in docs/ and explain why you have done the checkpoints this way.

In particular I cannot understand why you have these six checkpoints:

+    CHECKPOINT_CLOCK_VIRTUAL,
+    CHECKPOINT_CLOCK_VIRTUAL_ALL,
+    CHECKPOINT_CLOCK_HOST,
+    CHECKPOINT_CLOCK_HOST_ALL,
+    CHECKPOINT_CLOCK_VIRTUAL_RT,
+    CHECKPOINT_CLOCK_VIRTUAL_RT_ALL,

1) why do you need separate checkpoints for each clock?

2) why do you need separate checkpoints for _ALL and single-aio-context?

Paolo

Quoted 357 lines
---
 block.c                  |   11 ++++++++++
 cpus.c                   |    7 ++++++-
 include/qemu/timer.h     |    6 ++++--
 main-loop.c              |    5 +++++
 qemu-timer.c             |   49 ++++++++++++++++++++++++++++++++++++++--------
 replay/replay-internal.h |    5 ++++-
 replay/replay.c          |   36 ++++++++++++++++++++++++++++++++++
 replay/replay.h          |   21 ++++++++++++++++++++
 stubs/replay.c           |   11 ++++++++++
 vl.c                     |    4 +++-
 10 files changed, 142 insertions(+), 13 deletions(-)
block.c +11 -0
index cbe4a32..a4f45c3 100644
@@ -1994,6 +1994,11 @@ void bdrv_drain_all(void)
     BlockDriverState *bs;
 
     while (busy) {
+        if (!replay_checkpoint(CHECKPOINT_BDRV_DRAIN)) {
+            /* Do not wait anymore, we stopped at some place in
+               the middle of execution during replay */
+            return;
+        }
         busy = false;
 
         QTAILQ_FOREACH(bs, &bdrv_states, device_list) {
@@ -2004,6 +2009,12 @@ void bdrv_drain_all(void)
             aio_context_release(aio_context);
         }
     }
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        /* Skip checkpoints from the log */
+        while (replay_checkpoint(CHECKPOINT_BDRV_DRAIN)) {
+            /* Nothing */
+        }
+    }
 }
 
 /* make a BlockDriverState anonymous by removing from bdrv_state and
cpus.c +6 -1
index 01d89aa..9c32491 100644
@@ -388,7 +388,7 @@ void qtest_clock_warp(int64_t dest)
         timers_state.qemu_icount_bias += warp;
         seqlock_write_unlock(&timers_state.vm_clock_seqlock);
 
-        qemu_clock_run_timers(QEMU_CLOCK_VIRTUAL);
+        qemu_clock_run_timers(QEMU_CLOCK_VIRTUAL, false);
         clock = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
     }
     qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
@@ -408,6 +408,11 @@ void qemu_clock_warp(QEMUClockType type)
         return;
     }
 
+    /* warp clock deterministically in record/replay mode */
+    if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP)) {
+        return;
+    }
+
     /*
      * If the CPUs have been sleeping, advance QEMU_CLOCK_VIRTUAL timer now.
      * This ensures that the deadline for the timer is computed correctly below.
include/qemu/timer.h +4 -2
index 0c2472c..26927b0 100644
@@ -240,13 +240,14 @@ void qemu_clock_unregister_reset_notifier(QEMUClockType type,
 /**
  * qemu_clock_run_timers:
  * @type: clock on which to operate
+ * @run_all: true, when called from qemu_clock_run_all_timers
  *
  * Run all the timers associated with the default timer list
  * of a clock.
  *
  * Returns: true if any timer ran.
  */
-bool qemu_clock_run_timers(QEMUClockType type);
+bool qemu_clock_run_timers(QEMUClockType type, bool run_all);
 
 /**
  * qemu_clock_run_all_timers:
@@ -337,12 +338,13 @@ QEMUClockType timerlist_get_clock(QEMUTimerList *timer_list);
 /**
  * timerlist_run_timers:
  * @timer_list: the timer list to use
+ * @run_all: true, when called from qemu_clock_run_all_timers
  *
  * Call all expired timers associated with the timer list.
  *
  * Returns: true if any timer expired
  */
-bool timerlist_run_timers(QEMUTimerList *timer_list);
+bool timerlist_run_timers(QEMUTimerList *timer_list, bool run_all);
 
 /**
  * timerlist_notify:
main-loop.c +5 -0
index 981bcb5..d6e93c3 100644
@@ -497,6 +497,11 @@ int main_loop_wait(int nonblocking)
     slirp_pollfds_poll(gpollfds, (ret < 0));
 #endif
 
+    /* CPU thread can infinitely wait for event after
+       missing the warp */
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
+    }
     qemu_clock_run_all_timers();
 
     return ret;
qemu-timer.c +41 -8
index bc981a2..b6eb7b3 100644
@@ -465,7 +465,7 @@ bool timer_expired(QEMUTimer *timer_head, int64_t current_time)
     return timer_expired_ns(timer_head, current_time * timer_head->scale);
 }
 
-bool timerlist_run_timers(QEMUTimerList *timer_list)
+bool timerlist_run_timers(QEMUTimerList *timer_list, bool run_all)
 {
     QEMUTimer *ts;
     int64_t current_time;
@@ -473,6 +473,32 @@ bool timerlist_run_timers(QEMUTimerList *timer_list)
     QEMUTimerCB *cb;
     void *opaque;
 
+    switch (timer_list->clock->type) {
+    case QEMU_CLOCK_REALTIME:
+        break;
+    default:
+    case QEMU_CLOCK_VIRTUAL:
+        if ((replay_mode != REPLAY_MODE_NONE && !runstate_is_running())
+            || !replay_checkpoint(run_all ? CHECKPOINT_CLOCK_VIRTUAL_ALL
+                                          : CHECKPOINT_CLOCK_VIRTUAL)) {
+            return false;
+        }
+        break;
+    case QEMU_CLOCK_HOST:
+        if ((replay_mode != REPLAY_MODE_NONE && !runstate_is_running())
+            || !replay_checkpoint(run_all ? CHECKPOINT_CLOCK_HOST_ALL
+                                          : CHECKPOINT_CLOCK_HOST)) {
+            return false;
+        }
+    case QEMU_CLOCK_VIRTUAL_RT:
+        if ((replay_mode != REPLAY_MODE_NONE && !runstate_is_running())
+            || !replay_checkpoint(run_all ? CHECKPOINT_CLOCK_VIRTUAL_RT_ALL
+                                          : CHECKPOINT_CLOCK_VIRTUAL_RT)) {
+            return false;
+        }
+        break;
+    }
+
     qemu_event_reset(&timer_list->timers_done_ev);
     if (!timer_list->clock->enabled) {
         goto out;
@@ -505,9 +531,9 @@ out:
     return progress;
 }
 
-bool qemu_clock_run_timers(QEMUClockType type)
+bool qemu_clock_run_timers(QEMUClockType type, bool run_all)
 {
-    return timerlist_run_timers(main_loop_tlg.tl[type]);
+    return timerlist_run_timers(main_loop_tlg.tl[type], run_all);
 }
 
 void timerlistgroup_init(QEMUTimerListGroup *tlg,
@@ -532,7 +558,7 @@ bool timerlistgroup_run_timers(QEMUTimerListGroup *tlg)
     QEMUClockType type;
     bool progress = false;
     for (type = 0; type < QEMU_CLOCK_MAX; type++) {
-        progress |= timerlist_run_timers(tlg->tl[type]);
+        progress |= timerlist_run_timers(tlg->tl[type], false);
     }
     return progress;
 }
@@ -541,11 +567,18 @@ int64_t timerlistgroup_deadline_ns(QEMUTimerListGroup *tlg)
 {
     int64_t deadline = -1;
     QEMUClockType type;
+    bool play = replay_mode == REPLAY_MODE_PLAY;
     for (type = 0; type < QEMU_CLOCK_MAX; type++) {
         if (qemu_clock_use_for_deadline(tlg->tl[type]->clock->type)) {
-            deadline = qemu_soonest_timeout(deadline,
-                                            timerlist_deadline_ns(
-                                                tlg->tl[type]));
+            if (!play || tlg->tl[type]->clock->type == QEMU_CLOCK_REALTIME) {
+                deadline = qemu_soonest_timeout(deadline,
+                                                timerlist_deadline_ns(
+                                                    tlg->tl[type]));
+            } else {
+                /* Read clock from the replay file and
+                   do not calculate the deadline, based on virtual clock. */
+                qemu_clock_get_ns(tlg->tl[type]->clock->type);
+            }
         }
     }
     return deadline;
@@ -615,7 +648,7 @@ bool qemu_clock_run_all_timers(void)
     QEMUClockType type;
 
     for (type = 0; type < QEMU_CLOCK_MAX; type++) {
-        progress |= qemu_clock_run_timers(type);
+        progress |= qemu_clock_run_timers(type, true);
     }
 
     return progress;
replay/replay-internal.h +4 -1
index 68b2d45..c0a5800 100755
@@ -31,7 +31,10 @@ enum ReplayEvents {
     EVENT_SHUTDOWN,
     /* for clock read/writes */
     /* some of grteater codes are reserved for clocks */
-    EVENT_CLOCK
+    EVENT_CLOCK,
+    /* for checkpoint event */
+    /* some of grteater codes are reserved for checkpoints */
+    EVENT_CHECKPOINT = EVENT_CLOCK + REPLAY_CLOCK_COUNT
 };
 
 /* Asynchronous events IDs */
replay/replay.c +36 -0
index cfa69fa..7c4a801 100755
@@ -163,3 +163,39 @@ void replay_shutdown_request(void)
         replay_put_event(EVENT_SHUTDOWN);
     }
 }
+
+bool replay_checkpoint(ReplayCheckpoint checkpoint)
+{
+    bool res = false;
+    replay_save_instructions();
+
+    if (replay_file) {
+        if (replay_mode == REPLAY_MODE_PLAY) {
+            replay_mutex_lock();
+            if (!skip_async_events(EVENT_CHECKPOINT + checkpoint)) {
+                if (replay_data_kind == EVENT_ASYNC) {
+                    replay_read_events(checkpoint);
+                    replay_fetch_data_kind();
+                    res = replay_data_kind != EVENT_ASYNC;
+                    replay_mutex_unlock();
+                    return res;
+                }
+                replay_mutex_unlock();
+                return res;
+            }
+            replay_has_unread_data = 0;
+            replay_read_events(checkpoint);
+            replay_fetch_data_kind();
+            res = replay_data_kind != EVENT_ASYNC;
+            replay_mutex_unlock();
+            return res;
+        } else if (replay_mode == REPLAY_MODE_RECORD) {
+            replay_mutex_lock();
+            replay_put_event(EVENT_CHECKPOINT + checkpoint);
+            replay_save_events(checkpoint);
+            replay_mutex_unlock();
+        }
+    }
+
+    return true;
+}
replay/replay.h +21 -0
index e1c5fcf..39822b4 100755
@@ -29,6 +29,21 @@ enum ReplayClockKind {
 };
 typedef enum ReplayClockKind ReplayClockKind;
 
+/* IDs of the checkpoints */
+enum ReplayCheckpoint {
+    CHECKPOINT_BDRV_DRAIN,
+    CHECKPOINT_CLOCK_WARP,
+    CHECKPOINT_RESET_REQUESTED,
+    CHECKPOINT_CLOCK_VIRTUAL,
+    CHECKPOINT_CLOCK_VIRTUAL_ALL,
+    CHECKPOINT_CLOCK_HOST,
+    CHECKPOINT_CLOCK_HOST_ALL,
+    CHECKPOINT_CLOCK_VIRTUAL_RT,
+    CHECKPOINT_CLOCK_VIRTUAL_RT_ALL,
+    CHECKPOINT_COUNT
+};
+typedef enum ReplayCheckpoint ReplayCheckpoint;
+
 extern ReplayMode replay_mode;
 
 /* Processing the instructions */
@@ -80,6 +95,12 @@ void replay_get_timedate(struct tm *tm);
 
 /*! Called when qemu shutdown is requested. */
 void replay_shutdown_request(void);
+/*! Should be called at check points in the execution.
+    These check points are skipped, if they were not met.
+    Saves checkpoint in the SAVE mode and validates in the PLAY mode.
+    Returns 0 in PLAY mode if checkpoint was not found.
+    Returns 1 in all other cases. */
+bool replay_checkpoint(ReplayCheckpoint checkpoint);
 
 /* Asynchronous events queue */
 
stubs/replay.c +11 -0
index 121bca6..1be3575 100755
@@ -1,4 +1,5 @@
 #include "replay/replay.h"
+#include "sysemu/sysemu.h"
 
 ReplayMode replay_mode;
 
@@ -10,3 +11,13 @@ int64_t replay_read_clock(unsigned int kind)
 {
     return 0;
 }
+
+bool replay_checkpoint(ReplayCheckpoint checkpoint)
+{
+    return 0;
+}
+
+int runstate_is_running(void)
+{
+    return 0;
+}
vl.c +3 -1
index 905ea8a..86ba385 100644
@@ -1767,7 +1767,9 @@ static bool main_loop_should_exit(void)
             return true;
         }
     }
-    if (qemu_reset_requested()) {
+    if (qemu_reset_requested_get()
+        && replay_checkpoint(CHECKPOINT_RESET_REQUESTED)) {
+        qemu_reset_requested();
         pause_all_vcpus();
         cpu_synchronize_all_states();
         qemu_system_reset(VMRESET_REPORT);
#58 lore raw

RFC PATCH v8 15/21 aio: replace stack of bottom halves with queue

In reply to Pavel Dovgalyuk

Bottom halves in AIO context are stored and removes in LIFO order. It makes their execution non-deterministic. This patch replaces the stack with queue to preserve the order of bottom halves processing.

---
 async.c              |   25 +++++++++----------------
 include/block/aio.h  |    4 ++--
 include/qemu/queue.h |    7 +++++++
 3 files changed, 18 insertions(+), 18 deletions(-)
async.c +9 -16
index 2be88cc..bc6e83b 100644
@@ -35,7 +35,7 @@ struct QEMUBH {
     AioContext *ctx;
     QEMUBHFunc *cb;
     void *opaque;
-    QEMUBH *next;
+    QSIMPLEQ_ENTRY(QEMUBH) next;
     bool scheduled;
     bool idle;
     bool deleted;
@@ -51,10 +51,7 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque)
         .opaque = opaque,
     };
     qemu_mutex_lock(&ctx->bh_lock);
-    bh->next = ctx->first_bh;
-    /* Make sure that the members are ready before putting bh into list */
-    smp_wmb();
-    ctx->first_bh = bh;
+    QSIMPLEQ_INSERT_TAIL_RCU(&ctx->bh_queue, bh, next);
     qemu_mutex_unlock(&ctx->bh_lock);
     return bh;
 }
@@ -62,16 +59,15 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque)
 /* Multiple occurrences of aio_bh_poll cannot be called concurrently */
 int aio_bh_poll(AioContext *ctx)
 {
-    QEMUBH *bh, **bhp, *next;
+    QEMUBH *bh, *next;
     int ret;
 
     ctx->walking_bh++;
 
     ret = 0;
-    for (bh = ctx->first_bh; bh; bh = next) {
+    QSIMPLEQ_FOREACH_SAFE(bh, &ctx->bh_queue, next, next) {
         /* Make sure that fetching bh happens before accessing its members */
         smp_read_barrier_depends();
-        next = bh->next;
         if (!bh->deleted && bh->scheduled) {
             bh->scheduled = 0;
             /* Paired with write barrier in bh schedule to ensure reading for
@@ -90,14 +86,10 @@ int aio_bh_poll(AioContext *ctx)
     /* remove deleted bhs */
     if (!ctx->walking_bh) {
         qemu_mutex_lock(&ctx->bh_lock);
-        bhp = &ctx->first_bh;
-        while (*bhp) {
-            bh = *bhp;
+        QSIMPLEQ_FOREACH_SAFE(bh, &ctx->bh_queue, next, next) {
             if (bh->deleted) {
-                *bhp = bh->next;
+                QSIMPLEQ_REMOVE(&ctx->bh_queue, bh, QEMUBH, next);
                 g_free(bh);
-            } else {
-                bhp = &bh->next;
             }
         }
         qemu_mutex_unlock(&ctx->bh_lock);
@@ -161,7 +153,7 @@ aio_compute_timeout(AioContext *ctx)
     int timeout = -1;
     QEMUBH *bh;
 
-    for (bh = ctx->first_bh; bh; bh = bh->next) {
+    QSIMPLEQ_FOREACH(bh, &ctx->bh_queue, next) {
         if (!bh->deleted && bh->scheduled) {
             if (bh->idle) {
                 /* idle bottom halves will be polled at least
@@ -204,7 +196,7 @@ aio_ctx_check(GSource *source)
     AioContext *ctx = (AioContext *) source;
     QEMUBH *bh;
 
-    for (bh = ctx->first_bh; bh; bh = bh->next) {
+    QSIMPLEQ_FOREACH(bh, &ctx->bh_queue, next) {
         if (!bh->deleted && bh->scheduled) {
             return true;
 	}
@@ -311,6 +303,7 @@ AioContext *aio_context_new(Error **errp)
     qemu_mutex_init(&ctx->bh_lock);
     rfifolock_init(&ctx->lock, aio_rfifolock_cb, ctx);
     timerlistgroup_init(&ctx->tlg, aio_timerlist_notify, ctx);
+    QSIMPLEQ_INIT(&ctx->bh_queue);
 
     return ctx;
 }
include/block/aio.h +2 -2
index 7d1e26b..82cdf78 100644
@@ -71,8 +71,8 @@ struct AioContext {
     /* lock to protect between bh's adders and deleter */
     QemuMutex bh_lock;
 
-    /* Anchor of the list of Bottom Halves belonging to the context */
-    struct QEMUBH *first_bh;
+    /* List of Bottom Halves belonging to the context */
+    QSIMPLEQ_HEAD(, QEMUBH) bh_queue;
 
     /* A simple lock used to protect the first_bh list, and ensure that
      * no callbacks are removed while we're walking and dispatching callbacks.
include/qemu/queue.h +7 -0
index a98eb3a..b94c4d4 100644
@@ -268,6 +268,13 @@ struct {                                                                \
     (head)->sqh_last = &(elm)->field.sqe_next;                          \
 } while (/*CONSTCOND*/0)
 
+#define QSIMPLEQ_INSERT_TAIL_RCU(head, elm, field) do {                 \
+    (elm)->field.sqe_next = NULL;                                       \
+    *(head)->sqh_last = (elm);                                          \
+    (head)->sqh_last = &(elm)->field.sqe_next;                          \
+    smp_wmb();                                                          \
+} while (/*CONSTCOND*/0)
+
 #define QSIMPLEQ_INSERT_AFTER(head, listelm, elm, field) do {           \
     if (((elm)->field.sqe_next = (listelm)->field.sqe_next) == NULL)    \
         (head)->sqh_last = &(elm)->field.sqe_next;                      \
#59 lore raw

Re: [Qemu-devel] [RFC PATCH v8 15/21] aio: replace stack of bottom halves with queue

Quoted 43 lines · Pavel Dovgalyuk

Bottom halves in AIO context are stored and removes in LIFO order. It makes their execution non-deterministic. This patch replaces the stack with queue to preserve the order of bottom halves processing.

---
 async.c              |   25 +++++++++----------------
 include/block/aio.h  |    4 ++--
 include/qemu/queue.h |    7 +++++++
 3 files changed, 18 insertions(+), 18 deletions(-)
async.c +3 -6
index 2be88cc..bc6e83b 100644
@@ -35,7 +35,7 @@ struct QEMUBH {
     AioContext *ctx;
     QEMUBHFunc *cb;
     void *opaque;
-    QEMUBH *next;
+    QSIMPLEQ_ENTRY(QEMUBH) next;
     bool scheduled;
     bool idle;
     bool deleted;
@@ -51,10 +51,7 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque)
         .opaque = opaque,
     };
     qemu_mutex_lock(&ctx->bh_lock);
-    bh->next = ctx->first_bh;
-    /* Make sure that the members are ready before putting bh into list */
-    smp_wmb();
-    ctx->first_bh = bh;
+    QSIMPLEQ_INSERT_TAIL_RCU(&ctx->bh_queue, bh, next);
     qemu_mutex_unlock(&ctx->bh_lock);
     return bh;
 }
@@ -62,16 +59,15 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque)
 /* Multiple occurrences of aio_bh_poll cannot be called concurrently */
 int aio_bh_poll(AioContext *ctx)
 {
-    QEMUBH *bh, **bhp, *next;
+    QEMUBH *bh, *next;

async.c QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque) · line 74

Quoted 10 lines
async.c +1 -2
     int ret;
 
     ctx->walking_bh++;
 
     ret = 0;
-    for (bh = ctx->first_bh; bh; bh = next) {
+    QSIMPLEQ_FOREACH_SAFE(bh, &ctx->bh_queue, next, next) {
         /* Make sure that fetching bh happens before accessing its members */
         smp_read_barrier_depends();
-        next = bh->next;

Must use QSIMPLEQ_FOREACH. Otherwise, the access of next is before the smp_read_barrier_depends().

Quoted 66 lines
async.c +5 -8
         if (!bh->deleted && bh->scheduled) {
             bh->scheduled = 0;
             /* Paired with write barrier in bh schedule to ensure reading for
@@ -90,14 +86,10 @@ int aio_bh_poll(AioContext *ctx)
     /* remove deleted bhs */
     if (!ctx->walking_bh) {
         qemu_mutex_lock(&ctx->bh_lock);
-        bhp = &ctx->first_bh;
-        while (*bhp) {
-            bh = *bhp;
+        QSIMPLEQ_FOREACH_SAFE(bh, &ctx->bh_queue, next, next) {
             if (bh->deleted) {
-                *bhp = bh->next;
+                QSIMPLEQ_REMOVE(&ctx->bh_queue, bh, QEMUBH, next);
                 g_free(bh);
-            } else {
-                bhp = &bh->next;
             }
         }
         qemu_mutex_unlock(&ctx->bh_lock);
@@ -161,7 +153,7 @@ aio_compute_timeout(AioContext *ctx)
     int timeout = -1;
     QEMUBH *bh;
 
-    for (bh = ctx->first_bh; bh; bh = bh->next) {
+    QSIMPLEQ_FOREACH(bh, &ctx->bh_queue, next) {
         if (!bh->deleted && bh->scheduled) {
             if (bh->idle) {
                 /* idle bottom halves will be polled at least
@@ -204,7 +196,7 @@ aio_ctx_check(GSource *source)
     AioContext *ctx = (AioContext *) source;
     QEMUBH *bh;
 
-    for (bh = ctx->first_bh; bh; bh = bh->next) {
+    QSIMPLEQ_FOREACH(bh, &ctx->bh_queue, next) {
         if (!bh->deleted && bh->scheduled) {
             return true;
 	}
@@ -311,6 +303,7 @@ AioContext *aio_context_new(Error **errp)
     qemu_mutex_init(&ctx->bh_lock);
     rfifolock_init(&ctx->lock, aio_rfifolock_cb, ctx);
     timerlistgroup_init(&ctx->tlg, aio_timerlist_notify, ctx);
+    QSIMPLEQ_INIT(&ctx->bh_queue);
 
     return ctx;
 }
include/block/aio.h +2 -2
index 7d1e26b..82cdf78 100644
@@ -71,8 +71,8 @@ struct AioContext {
     /* lock to protect between bh's adders and deleter */
     QemuMutex bh_lock;
 
-    /* Anchor of the list of Bottom Halves belonging to the context */
-    struct QEMUBH *first_bh;
+    /* List of Bottom Halves belonging to the context */
+    QSIMPLEQ_HEAD(, QEMUBH) bh_queue;
 
     /* A simple lock used to protect the first_bh list, and ensure that
      * no callbacks are removed while we're walking and dispatching callbacks.
include/qemu/queue.h +0 -0
index a98eb3a..b94c4d4 100644

include/qemu/queue.h struct { \ · line 275

Quoted 8 lines
include/qemu/queue.h +5 -0
@@ -268,6 +268,13 @@ struct {                                                                \
     (head)->sqh_last = &(elm)->field.sqe_next;                          \
 } while (/*CONSTCOND*/0)
 
+#define QSIMPLEQ_INSERT_TAIL_RCU(head, elm, field) do {                 \
+    (elm)->field.sqe_next = NULL;                                       \
+    *(head)->sqh_last = (elm);                                          \
+    (head)->sqh_last = &(elm)->field.sqe_next;                          \
+    smp_wmb();                                                          \

smp_wmb() must be after the (elm) assignment.

Paolo

Quoted 6 lines
include/qemu/queue.h +2 -0
+} while (/*CONSTCOND*/0)
+
 #define QSIMPLEQ_INSERT_AFTER(head, listelm, elm, field) do {           \
     if (((elm)->field.sqe_next = (listelm)->field.sqe_next) == NULL)    \
         (head)->sqh_last = &(elm)->field.sqe_next;                      \
#60 lore raw

RFC PATCH v8 16/21 replay: bottom halves

In reply to Pavel Dovgalyuk

This patch introduces bottom half event for replay queue. It saves the events into the queue and process them at the checkpoints and instructions execution.

---
 async.c                  |   24 +++++++++++++++++++++++-
 dma-helpers.c            |    4 +++-
 hw/ide/ahci.c            |    4 +++-
 hw/ide/core.c            |    4 +++-
 hw/timer/arm_timer.c     |    2 +-
 hw/usb/hcd-uhci.c        |    2 +-
 include/block/aio.h      |   18 ++++++++++++++++++
 include/qemu/main-loop.h |    1 +
 main-loop.c              |    5 +++++
 replay/replay-events.c   |   16 ++++++++++++++++
 replay/replay-internal.h |    1 +
 replay/replay.h          |    2 ++
 stubs/replay.c           |    4 ++++
 13 files changed, 81 insertions(+), 6 deletions(-)
async.c +23 -1
index bc6e83b..0c02ae2 100644
@@ -27,6 +27,7 @@
 #include "block/thread-pool.h"
 #include "qemu/main-loop.h"
 #include "qemu/atomic.h"
+#include "replay/replay.h"
 
 /***********************************************************/
 /* bottom halves (can be seen as timers which expire ASAP) */
@@ -39,6 +40,8 @@ struct QEMUBH {
     bool scheduled;
     bool idle;
     bool deleted;
+    bool replay;
+    uint64_t id;
 };
 
 QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque)
@@ -56,6 +59,21 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque)
     return bh;
 }
 
+QEMUBH *aio_bh_new_replay(AioContext *ctx, QEMUBHFunc *cb, void *opaque,
+                          uint64_t id)
+{
+    QEMUBH *bh = aio_bh_new(ctx, cb, opaque);
+    bh->replay = true;
+    bh->id = id;
+    return bh;
+}
+
+void aio_bh_call(void *opaque)
+{
+    QEMUBH *bh = (QEMUBH *)opaque;
+    bh->cb(bh->opaque);
+}
+
 /* Multiple occurrences of aio_bh_poll cannot be called concurrently */
 int aio_bh_poll(AioContext *ctx)
 {
@@ -77,7 +95,11 @@ int aio_bh_poll(AioContext *ctx)
             if (!bh->idle)
                 ret = 1;
             bh->idle = 0;
-            bh->cb(bh->opaque);
+            if (!bh->replay) {
+                aio_bh_call(bh);
+            } else {
+                replay_add_bh_event(bh, bh->id);
+            }
         }
     }
 
dma-helpers.c +3 -1
index 6918572..357d7e9 100644
@@ -13,6 +13,7 @@
 #include "qemu/range.h"
 #include "qemu/thread.h"
 #include "qemu/main-loop.h"
+#include "replay/replay.h"
 
 /* #define DEBUG_IOMMU */
 
@@ -96,7 +97,8 @@ static void continue_after_map_failure(void *opaque)
 {
     DMAAIOCB *dbs = (DMAAIOCB *)opaque;
 
-    dbs->bh = qemu_bh_new(reschedule_dma, dbs);
+    dbs->bh = qemu_bh_new_replay(reschedule_dma, dbs,
+                                 replay_get_current_step());
     qemu_bh_schedule(dbs->bh);
 }
 
hw/ide/ahci.c +3 -1
index 5651372..13d7f84 100644
@@ -33,6 +33,7 @@
 #include "internal.h"
 #include <hw/ide/pci.h>
 #include <hw/ide/ahci.h>
+#include "replay/replay.h"
 
 #define DEBUG_AHCI 0
 
@@ -1243,7 +1244,8 @@ static void ahci_cmd_done(IDEDMA *dma)
 
     if (!ad->check_bh) {
         /* maybe we still have something to process, check later */
-        ad->check_bh = qemu_bh_new(ahci_check_cmd_bh, ad);
+        ad->check_bh = qemu_bh_new_replay(ahci_check_cmd_bh, ad,
+                                          replay_get_current_step());
         qemu_bh_schedule(ad->check_bh);
     }
 }
hw/ide/core.c +3 -1
index d4af5e2..d76244a 100644
@@ -32,6 +32,7 @@
 #include "sysemu/dma.h"
 #include "hw/block/block.h"
 #include "sysemu/block-backend.h"
+#include "replay/replay.h"
 
 #include <hw/ide/internal.h>
 
@@ -448,7 +449,8 @@ BlockAIOCB *ide_issue_trim(BlockBackend *blk,
 
     iocb = blk_aio_get(&trim_aiocb_info, blk, cb, opaque);
     iocb->blk = blk;
-    iocb->bh = qemu_bh_new(ide_trim_bh_cb, iocb);
+    iocb->bh = qemu_bh_new_replay(ide_trim_bh_cb, iocb,
+                                  replay_get_current_step());
     iocb->ret = 0;
     iocb->qiov = qiov;
     iocb->i = -1;
hw/timer/arm_timer.c +1 -1
index 1452910..97784a0 100644
@@ -168,7 +168,7 @@ static arm_timer_state *arm_timer_init(uint32_t freq)
     s->freq = freq;
     s->control = TIMER_CTRL_IE;
 
-    bh = qemu_bh_new(arm_timer_tick, s);
+    bh = qemu_bh_new_replay(arm_timer_tick, s, 0);
     s->timer = ptimer_init(bh);
     vmstate_register(NULL, -1, &vmstate_arm_timer, s);
     return s;
hw/usb/hcd-uhci.c +1 -1
index 4a4215d..0d94143 100644
@@ -1221,7 +1221,7 @@ static int usb_uhci_common_initfn(PCIDevice *dev)
                               USB_SPEED_MASK_LOW | USB_SPEED_MASK_FULL);
         }
     }
-    s->bh = qemu_bh_new(uhci_bh, s);
+    s->bh = qemu_bh_new_replay(uhci_bh, s, 0);
     s->frame_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, uhci_frame_timer, s);
     s->num_ports_vmstate = NB_PORTS;
     QTAILQ_INIT(&s->queues);
include/block/aio.h +18 -0
index 82cdf78..ed76b43 100644
@@ -35,6 +35,8 @@ struct BlockAIOCB {
     const AIOCBInfo *aiocb_info;
     BlockDriverState *bs;
     BlockCompletionFunc *cb;
+    bool replay;
+    uint64_t replay_step;
     void *opaque;
     int refcnt;
 };
@@ -144,6 +146,17 @@ void aio_context_release(AioContext *ctx);
 QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque);
 
 /**
+ * aio_bh_new_replay: Allocate a new bottom half structure for replay.
+ *
+ * This function calls aio_bh_new function and also fills replay parameters
+ * of the BH structure. BH created with this function in record/replay mode
+ * are executed through the replay queue only at checkpoints and instructions
+ * executions.
+ */
+QEMUBH *aio_bh_new_replay(AioContext *ctx, QEMUBHFunc *cb, void *opaque,
+                          uint64_t id);
+
+/**
  * aio_notify: Force processing of pending events.
  *
  * Similar to signaling a condition variable, aio_notify forces
@@ -159,6 +172,11 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque);
 void aio_notify(AioContext *ctx);
 
 /**
+ * aio_bh_call: Executes callback function of the specified BH.
+ */
+void aio_bh_call(void *opaque);
+
+/**
  * aio_bh_poll: Poll bottom halves for an AioContext.
  *
  * These are internal functions used by the QEMU main loop.
include/qemu/main-loop.h +1 -0
index 62c68c0..f5a98fe 100644
@@ -306,6 +306,7 @@ void qemu_iohandler_fill(GArray *pollfds);
 void qemu_iohandler_poll(GArray *pollfds, int rc);
 
 QEMUBH *qemu_bh_new(QEMUBHFunc *cb, void *opaque);
+QEMUBH *qemu_bh_new_replay(QEMUBHFunc *cb, void *opaque, uint64_t id);
 void qemu_bh_schedule_idle(QEMUBH *bh);
 
 #endif
main-loop.c +5 -0
index d6e93c3..df99bcf 100644
@@ -513,3 +513,8 @@ QEMUBH *qemu_bh_new(QEMUBHFunc *cb, void *opaque)
 {
     return aio_bh_new(qemu_aio_context, cb, opaque);
 }
+
+QEMUBH *qemu_bh_new_replay(QEMUBHFunc *cb, void *opaque, uint64_t id)
+{
+    return aio_bh_new_replay(qemu_aio_context, cb, opaque, id);
+}
replay/replay-events.c +16 -0
index dfd5efd..7e82b61 100755
@@ -36,6 +36,9 @@ static bool replay_events_enabled = false;
 static void replay_run_event(Event *event)
 {
     switch (event->event_kind) {
+    case REPLAY_ASYNC_EVENT_BH:
+        aio_bh_call(event->opaque);
+        break;
     default:
         fprintf(stderr, "Replay: invalid async event ID (%d) in the queue\n",
                 event->event_kind);
@@ -122,6 +125,11 @@ void replay_add_event(ReplayAsyncEventKind event_kind, void *opaque)
     replay_add_event_internal(event_kind, opaque, NULL, 0);
 }
 
+void replay_add_bh_event(void *bh, uint64_t id)
+{
+    replay_add_event_internal(REPLAY_ASYNC_EVENT_BH, bh, NULL, id);
+}
+
 /* Called with replay mutex locked */
 void replay_save_events(int opt)
 {
@@ -136,6 +144,9 @@ void replay_save_events(int opt)
 
             /* save event-specific data */
             switch (event->event_kind) {
+            case REPLAY_ASYNC_EVENT_BH:
+                replay_put_qword(event->id);
+                break;
             }
         }
 
@@ -167,6 +178,11 @@ void replay_read_events(int opt)
         }
         /* Execute some events without searching them in the queue */
         switch (read_event_kind) {
+        case REPLAY_ASYNC_EVENT_BH:
+            if (read_id == -1) {
+                read_id = replay_get_qword();
+            }
+            break;
         default:
             fprintf(stderr, "Unknown ID %d of replay event\n", read_event_kind);
             exit(1);
replay/replay-internal.h +1 -0
index c0a5800..0e6ba2a 100755
@@ -40,6 +40,7 @@ enum ReplayEvents {
 /* Asynchronous events IDs */
 
 enum ReplayAsyncEventKind {
+    REPLAY_ASYNC_EVENT_BH,
     REPLAY_ASYNC_COUNT
 };
 
replay/replay.h +2 -0
index 39822b4..073e105 100755
@@ -106,5 +106,7 @@ bool replay_checkpoint(ReplayCheckpoint checkpoint);
 
 /*! Disables storing events in the queue */
 void replay_disable_events(void);
+/*! Adds BH event to the queue */
+void replay_add_bh_event(void *bh, uint64_t id);
 
 #endif
stubs/replay.c +4 -0
index 1be3575..268f3e0 100755
@@ -21,3 +21,7 @@ int runstate_is_running(void)
 {
     return 0;
 }
+
+void replay_add_bh_event(void *bh, uint64_t id)
+{
+}
#61 lore raw

Re: [Qemu-devel] [RFC PATCH v8 16/21] replay: bottom halves

Quoted 2 lines · Pavel Dovgalyuk

This patch introduces bottom half event for replay queue. It saves the events into the queue and process them at the checkpoints and instructions execution.

Which bottom halves must _not_ go through aio/qemu_bh_new_replay?

async.c QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque) · line 63

Quoted 2 lines
async.c +2 -0
+QEMUBH *aio_bh_new_replay(AioContext *ctx, QEMUBHFunc *cb, void *opaque,
+                          uint64_t id)

id is superfluous, it can always be replay_get_current_step().

It seems to me that the device models can always use qemu_bh_new_replay.
There are few if any uses of qemu_bh_new outside device models.
Perhaps convert those to aio_bh_new, and make qemu_bh_new always do special replay treatment?

async.c QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque) · line 69

Quoted 6 lines
async.c +6 -0
+{
+    QEMUBH *bh = aio_bh_new(ctx, cb, opaque);
+    bh->replay = true;
+    bh->id = id;
+    return bh;
+}

Paolo

#62 lore raw

Re: [Qemu-devel] [RFC PATCH v8 16/21] replay: bottom halves

Quoted 6 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 2 lines · Pavel Dovgalyuk

This patch introduces bottom half event for replay queue. It saves the events into the queue and process them at the checkpoints and instructions execution.

Which bottom halves must _not_ go through aio/qemu_bh_new_replay?

The block layer and migration internal ones.
They should not be replayed because are used for simulator purposes and not for VM execution.

Quoted 10 lines

async.c QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque) · line 63

Quoted 3 lines
async.c +2 -0
+QEMUBH *aio_bh_new_replay(AioContext *ctx, QEMUBHFunc *cb, void *opaque,
+                          uint64_t id)

id is superfluous, it can always be replay_get_current_step().

It seems to me that the device models can always use qemu_bh_new_replay.
There are few if any uses of qemu_bh_new outside device models.
Perhaps convert those to aio_bh_new, and make qemu_bh_new always do special replay treatment?

I've created some kind of design document.
Then I'll probably leave these changes until that document review.

Pavel Dovgalyuk

#63 lore raw

RFC PATCH v8 17/21 replay: replay aio requests

In reply to Pavel Dovgalyuk

This patch adds identifier to aio requests. ID is used for creating bottom halves and identifying them while replaying.
The patch also introduces several functions that make possible replaying of the aio requests.

---
 block.c                        |   81 ++++++++++++++++++++++++++++++++++++----
 block/block-backend.c          |   30 ++++++++++++++-
 block/qcow2.c                  |    4 ++
 dma-helpers.c                  |    6 ++-
 hw/block/virtio-blk.c          |   10 ++---
 hw/ide/atapi.c                 |   10 +++--
 hw/ide/core.c                  |   14 ++++---
 include/block/block.h          |   15 +++++++
 include/qemu-common.h          |    2 +
 include/sysemu/block-backend.h |   10 +++++
 qemu-io-cmds.c                 |    2 -
 stubs/replay.c                 |    5 ++
 trace-events                   |    2 +
 util/iov.c                     |    4 ++
 14 files changed, 167 insertions(+), 28 deletions(-)
block.c +72 -9
index a4f45c3..7f6fa8b 100644
@@ -83,7 +83,8 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
                                          BdrvRequestFlags flags,
                                          BlockCompletionFunc *cb,
                                          void *opaque,
-                                         bool is_write);
+                                         bool is_write,
+                                         bool aio_replay);
 static void coroutine_fn bdrv_co_do_rw(void *opaque);
 static int coroutine_fn bdrv_co_do_write_zeroes(BlockDriverState *bs,
     int64_t sector_num, int nb_sectors, BdrvRequestFlags flags);
@@ -4425,7 +4426,19 @@ BlockAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,
     trace_bdrv_aio_readv(bs, sector_num, nb_sectors, opaque);
 
     return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
-                                 cb, opaque, false);
+                                 cb, opaque, false, false);
+}
+
+BlockAIOCB *bdrv_aio_readv_replay(BlockDriverState *bs,
+                                  int64_t sector_num,
+                                  QEMUIOVector *qiov, int nb_sectors,
+                                  BlockCompletionFunc *cb,
+                                  void *opaque)
+{
+    trace_bdrv_aio_readv_replay(bs, sector_num, nb_sectors, opaque);
+
+    return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
+                                 cb, opaque, false, true);
 }
 
 BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
@@ -4435,7 +4448,19 @@ BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
     trace_bdrv_aio_writev(bs, sector_num, nb_sectors, opaque);
 
     return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
-                                 cb, opaque, true);
+                                 cb, opaque, true, false);
+}
+
+BlockAIOCB *bdrv_aio_writev_replay(BlockDriverState *bs,
+                                   int64_t sector_num,
+                                   QEMUIOVector *qiov, int nb_sectors,
+                                   BlockCompletionFunc *cb,
+                                   void *opaque)
+{
+    trace_bdrv_aio_writev_replay(bs, sector_num, nb_sectors, opaque);
+
+    return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
+                                 cb, opaque, true, true);
 }
 
 BlockAIOCB *bdrv_aio_write_zeroes(BlockDriverState *bs,
@@ -4446,7 +4471,7 @@ BlockAIOCB *bdrv_aio_write_zeroes(BlockDriverState *bs,
 
     return bdrv_co_aio_rw_vector(bs, sector_num, NULL, nb_sectors,
                                  BDRV_REQ_ZERO_WRITE | flags,
-                                 cb, opaque, true);
+                                 cb, opaque, true, true);
 }
 
 
@@ -4593,7 +4618,8 @@ static int multiwrite_merge(BlockDriverState *bs, BlockRequest *reqs,
  * requests. However, the fields opaque and error are left unmodified as they
  * are used to signal failure for a single request to the caller.
  */
-int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs)
+int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs,
+                        bool replay)
 {
     MultiwriteCB *mcb;
     int i;
@@ -4631,7 +4657,7 @@ int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs)
         bdrv_co_aio_rw_vector(bs, reqs[i].sector, reqs[i].qiov,
                               reqs[i].nb_sectors, reqs[i].flags,
                               multiwrite_cb, mcb,
-                              true);
+                              true, replay);
     }
 
     return 0;
@@ -4776,7 +4802,12 @@ static void coroutine_fn bdrv_co_do_rw(void *opaque)
             acb->req.nb_sectors, acb->req.qiov, acb->req.flags);
     }
 
-    acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    if (acb->common.replay) {
+        acb->bh = aio_bh_new_replay(bdrv_get_aio_context(bs), bdrv_co_em_bh,
+                                    acb, acb->common.replay_step);
+    } else {
+        acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    }
     qemu_bh_schedule(acb->bh);
 }
 
@@ -4787,7 +4818,8 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
                                          BdrvRequestFlags flags,
                                          BlockCompletionFunc *cb,
                                          void *opaque,
-                                         bool is_write)
+                                         bool is_write,
+                                         bool aio_replay)
 {
     Coroutine *co;
     BlockAIOCBCoroutine *acb;
@@ -4798,6 +4830,11 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
     acb->req.qiov = qiov;
     acb->req.flags = flags;
     acb->is_write = is_write;
+    acb->done = NULL;
+    acb->common.replay = aio_replay;
+    if (aio_replay) {
+        acb->common.replay_step = replay_get_current_step();
+    }
 
     co = qemu_coroutine_create(bdrv_co_do_rw);
     qemu_coroutine_enter(co, acb);
@@ -4811,7 +4848,12 @@ static void coroutine_fn bdrv_aio_flush_co_entry(void *opaque)
     BlockDriverState *bs = acb->common.bs;
 
     acb->req.error = bdrv_co_flush(bs);
-    acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    if (acb->common.replay) {
+        acb->bh = aio_bh_new_replay(bdrv_get_aio_context(bs), bdrv_co_em_bh,
+                                    acb, acb->common.replay_step);
+    } else {
+        acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    }
     qemu_bh_schedule(acb->bh);
 }
 
@@ -4831,6 +4873,25 @@ BlockAIOCB *bdrv_aio_flush(BlockDriverState *bs,
     return &acb->common;
 }
 
+BlockAIOCB *bdrv_aio_flush_replay(BlockDriverState *bs,
+        BlockCompletionFunc *cb, void *opaque)
+{
+    trace_bdrv_aio_flush(bs, opaque);
+
+    Coroutine *co;
+    BlockAIOCBCoroutine *acb;
+
+    acb = qemu_aio_get(&bdrv_em_co_aiocb_info, bs, cb, opaque);
+    acb->done = NULL;
+    acb->common.replay = true;
+    acb->common.replay_step = replay_get_current_step();
+
+    co = qemu_coroutine_create(bdrv_aio_flush_co_entry);
+    qemu_coroutine_enter(co, acb);
+
+    return &acb->common;
+}
+
 static void coroutine_fn bdrv_aio_discard_co_entry(void *opaque)
 {
     BlockAIOCBCoroutine *acb = opaque;
@@ -4881,6 +4942,8 @@ void *qemu_aio_get(const AIOCBInfo *aiocb_info, BlockDriverState *bs,
     acb->cb = cb;
     acb->opaque = opaque;
     acb->refcnt = 1;
+    acb->replay_step = 0;
+    acb->replay = false;
     return acb;
 }
 
block/block-backend.c +29 -1
index d00c129..451041a 100644
@@ -447,6 +447,14 @@ BlockAIOCB *blk_aio_readv(BlockBackend *blk, int64_t sector_num,
     return bdrv_aio_readv(blk->bs, sector_num, iov, nb_sectors, cb, opaque);
 }
 
+BlockAIOCB *blk_aio_readv_replay(BlockBackend *blk, int64_t sector_num,
+                                 QEMUIOVector *iov, int nb_sectors,
+                                 BlockCompletionFunc *cb, void *opaque)
+{
+    return bdrv_aio_readv_replay(blk->bs, sector_num, iov, nb_sectors,
+                                 cb, opaque);
+}
+
 BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,
                            QEMUIOVector *iov, int nb_sectors,
                            BlockCompletionFunc *cb, void *opaque)
@@ -454,12 +462,26 @@ BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,
     return bdrv_aio_writev(blk->bs, sector_num, iov, nb_sectors, cb, opaque);
 }
 
+BlockAIOCB *blk_aio_writev_replay(BlockBackend *blk, int64_t sector_num,
+                                  QEMUIOVector *iov, int nb_sectors,
+                                  BlockCompletionFunc *cb, void *opaque)
+{
+    return bdrv_aio_writev_replay(blk->bs, sector_num, iov, nb_sectors,
+                                  cb, opaque);
+}
+
 BlockAIOCB *blk_aio_flush(BlockBackend *blk,
                           BlockCompletionFunc *cb, void *opaque)
 {
     return bdrv_aio_flush(blk->bs, cb, opaque);
 }
 
+BlockAIOCB *blk_aio_flush_replay(BlockBackend *blk,
+                                 BlockCompletionFunc *cb, void *opaque)
+{
+    return bdrv_aio_flush_replay(blk->bs, cb, opaque);
+}
+
 BlockAIOCB *blk_aio_discard(BlockBackend *blk,
                             int64_t sector_num, int nb_sectors,
                             BlockCompletionFunc *cb, void *opaque)
@@ -479,7 +501,13 @@ void blk_aio_cancel_async(BlockAIOCB *acb)
 
 int blk_aio_multiwrite(BlockBackend *blk, BlockRequest *reqs, int num_reqs)
 {
-    return bdrv_aio_multiwrite(blk->bs, reqs, num_reqs);
+    return bdrv_aio_multiwrite(blk->bs, reqs, num_reqs, false);
+}
+
+int blk_aio_multiwrite_replay(BlockBackend *blk, BlockRequest *reqs,
+                              int num_reqs)
+{
+    return bdrv_aio_multiwrite(blk->bs, reqs, num_reqs, true);
 }
 
 int blk_ioctl(BlockBackend *blk, unsigned long int req, void *buf)
block/qcow2.c +4 -0
index e4e690a..4a2f6ae 100644
@@ -1140,6 +1140,8 @@ static coroutine_fn int qcow2_co_readv(BlockDriverState *bs, int64_t sector_num,
     uint8_t *cluster_data = NULL;
 
     qemu_iovec_init(&hd_qiov, qiov->niov);
+    hd_qiov.replay = qiov->replay;
+    hd_qiov.replay_step = qiov->replay_step;
 
     qemu_co_mutex_lock(&s->lock);
 
@@ -1297,6 +1299,8 @@ static coroutine_fn int qcow2_co_writev(BlockDriverState *bs,
                                  remaining_sectors);
 
     qemu_iovec_init(&hd_qiov, qiov->niov);
+    hd_qiov.replay = qiov->replay;
+    hd_qiov.replay_step = qiov->replay_step;
 
     s->cluster_cache_offset = -1; /* disable compressed cache */
 
dma-helpers.c +4 -2
index 357d7e9..4faf6d3 100644
@@ -212,6 +212,8 @@ BlockAIOCB *dma_blk_io(
     dbs->io_func = io_func;
     dbs->bh = NULL;
     qemu_iovec_init(&dbs->iov, sg->nsg);
+    dbs->iov.replay = true;
+    dbs->iov.replay_step = replay_get_current_step();
     dma_blk_cb(dbs, 0);
     return &dbs->common;
 }
@@ -221,7 +223,7 @@ BlockAIOCB *dma_blk_read(BlockBackend *blk,
                          QEMUSGList *sg, uint64_t sector,
                          void (*cb)(void *opaque, int ret), void *opaque)
 {
-    return dma_blk_io(blk, sg, sector, blk_aio_readv, cb, opaque,
+    return dma_blk_io(blk, sg, sector, blk_aio_readv_replay, cb, opaque,
                       DMA_DIRECTION_FROM_DEVICE);
 }
 
@@ -229,7 +231,7 @@ BlockAIOCB *dma_blk_write(BlockBackend *blk,
                           QEMUSGList *sg, uint64_t sector,
                           void (*cb)(void *opaque, int ret), void *opaque)
 {
-    return dma_blk_io(blk, sg, sector, blk_aio_writev, cb, opaque,
+    return dma_blk_io(blk, sg, sector, blk_aio_writev_replay, cb, opaque,
                       DMA_DIRECTION_TO_DEVICE);
 }
 
hw/block/virtio-blk.c +5 -5
index b19b102..2413cbe 100644
@@ -265,7 +265,7 @@ void virtio_submit_multiwrite(BlockBackend *blk, MultiReqBuffer *mrb)
         return;
     }
 
-    ret = blk_aio_multiwrite(blk, mrb->blkreq, mrb->num_writes);
+    ret = blk_aio_multiwrite_replay(blk, mrb->blkreq, mrb->num_writes);
     if (ret != 0) {
         for (i = 0; i < mrb->num_writes; i++) {
             if (mrb->blkreq[i].error) {
@@ -286,7 +286,7 @@ static void virtio_blk_handle_flush(VirtIOBlockReq *req, MultiReqBuffer *mrb)
      * Make sure all outstanding writes are posted to the backing device.
      */
     virtio_submit_multiwrite(req->dev->blk, mrb);
-    blk_aio_flush(req->dev->blk, virtio_blk_flush_complete, req);
+    blk_aio_flush_replay(req->dev->blk, virtio_blk_flush_complete, req);
 }
 
 static bool virtio_blk_sect_range_ok(VirtIOBlock *dev,
@@ -357,9 +357,9 @@ static void virtio_blk_handle_read(VirtIOBlockReq *req)
 
     block_acct_start(blk_get_stats(req->dev->blk), &req->acct, req->qiov.size,
                      BLOCK_ACCT_READ);
-    blk_aio_readv(req->dev->blk, sector, &req->qiov,
-                  req->qiov.size / BDRV_SECTOR_SIZE,
-                  virtio_blk_rw_complete, req);
+    blk_aio_readv_replay(req->dev->blk, sector, &req->qiov,
+                          req->qiov.size / BDRV_SECTOR_SIZE,
+                          virtio_blk_rw_complete, req);
 }
 
 void virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb)
hw/ide/atapi.c +6 -4
index a71e6e0..a69b620 100644
@@ -25,7 +25,7 @@
 
 #include "hw/ide/internal.h"
 #include "hw/scsi/scsi.h"
-#include "sysemu/block-backend.h"
+#include "replay/replay.h"
 
 static void ide_atapi_cmd_read_dma_cb(void *opaque, int ret);
 
@@ -350,10 +350,12 @@ static void ide_atapi_cmd_read_dma_cb(void *opaque, int ret)
     s->bus->dma->iov.iov_base = (void *)(s->io_buffer + data_offset);
     s->bus->dma->iov.iov_len = n * 4 * 512;
     qemu_iovec_init_external(&s->bus->dma->qiov, &s->bus->dma->iov, 1);
+    s->bus->dma->qiov.replay = true;
+    s->bus->dma->qiov.replay_step = replay_get_current_step();
 
-    s->bus->dma->aiocb = blk_aio_readv(s->blk, (int64_t)s->lba << 2,
-                                       &s->bus->dma->qiov, n * 4,
-                                       ide_atapi_cmd_read_dma_cb, s);
+    s->bus->dma->aiocb = blk_aio_readv_replay(s->blk, (int64_t)s->lba << 2,
+                                              &s->bus->dma->qiov, n * 4,
+                                              ide_atapi_cmd_read_dma_cb, s);
     return;
 
 eot:
hw/ide/core.c +9 -5
index d76244a..1fff193 100644
@@ -630,11 +630,13 @@ void ide_sector_read(IDEState *s)
     s->iov.iov_base = s->io_buffer;
     s->iov.iov_len  = n * BDRV_SECTOR_SIZE;
     qemu_iovec_init_external(&s->qiov, &s->iov, 1);
+    s->qiov.replay = true;
+    s->qiov.replay_step = replay_get_current_step();
 
     block_acct_start(blk_get_stats(s->blk), &s->acct,
                      n * BDRV_SECTOR_SIZE, BLOCK_ACCT_READ);
-    s->pio_aiocb = blk_aio_readv(s->blk, sector_num, &s->qiov, n,
-                                 ide_sector_read_cb, s);
+    s->pio_aiocb = blk_aio_readv_replay(s->blk, sector_num, &s->qiov, n,
+                                        ide_sector_read_cb, s);
 }
 
 static void dma_buf_commit(IDEState *s, uint32_t tx_bytes)
@@ -888,11 +890,13 @@ void ide_sector_write(IDEState *s)
     s->iov.iov_base = s->io_buffer;
     s->iov.iov_len  = n * BDRV_SECTOR_SIZE;
     qemu_iovec_init_external(&s->qiov, &s->iov, 1);
+    s->qiov.replay = true;
+    s->qiov.replay_step = replay_get_current_step();
 
     block_acct_start(blk_get_stats(s->blk), &s->acct,
                      n * BDRV_SECTOR_SIZE, BLOCK_ACCT_READ);
-    s->pio_aiocb = blk_aio_writev(s->blk, sector_num, &s->qiov, n,
-                                  ide_sector_write_cb, s);
+    s->pio_aiocb = blk_aio_writev_replay(s->blk, sector_num, &s->qiov, n,
+                                         ide_sector_write_cb, s);
 }
 
 static void ide_flush_cb(void *opaque, int ret)
@@ -928,7 +932,7 @@ void ide_flush_cache(IDEState *s)
 
     s->status |= BUSY_STAT;
     block_acct_start(blk_get_stats(s->blk), &s->acct, 0, BLOCK_ACCT_FLUSH);
-    s->pio_aiocb = blk_aio_flush(s->blk, ide_flush_cb, s);
+    s->pio_aiocb = blk_aio_flush_replay(s->blk, ide_flush_cb, s);
 }
 
 static void ide_cfata_metadata_inquiry(IDEState *s)
include/block/block.h +14 -1
index 3082d2b..06ce097 100644
@@ -293,11 +293,24 @@ typedef void BlockDriverDirtyHandler(BlockDriverState *bs, int64_t sector,
 BlockAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,
                            QEMUIOVector *iov, int nb_sectors,
                            BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *bdrv_aio_readv_replay(BlockDriverState *bs,
+                                  int64_t sector_num,
+                                  QEMUIOVector *iov, int nb_sectors,
+                                  BlockCompletionFunc *cb,
+                                  void *opaque);
 BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
                             QEMUIOVector *iov, int nb_sectors,
                             BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *bdrv_aio_writev_replay(BlockDriverState *bs,
+                                   int64_t sector_num,
+                                   QEMUIOVector *iov, int nb_sectors,
+                                   BlockCompletionFunc *cb,
+                                   void *opaque);
 BlockAIOCB *bdrv_aio_flush(BlockDriverState *bs,
                            BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *bdrv_aio_flush_replay(BlockDriverState *bs,
+                                  BlockCompletionFunc *cb,
+                                  void *opaque);
 BlockAIOCB *bdrv_aio_discard(BlockDriverState *bs,
                              int64_t sector_num, int nb_sectors,
                              BlockCompletionFunc *cb, void *opaque);
@@ -318,7 +331,7 @@ typedef struct BlockRequest {
 } BlockRequest;
 
 int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs,
-    int num_reqs);
+                        int num_reqs, bool replay);
 
 /* sg packet commands */
 int bdrv_ioctl(BlockDriverState *bs, unsigned long int req, void *buf);
include/qemu-common.h +2 -0
index 8c9957e..016556a 100644
@@ -320,6 +320,8 @@ typedef struct QEMUIOVector {
     int niov;
     int nalloc;
     size_t size;
+    bool replay;
+    uint64_t replay_step;
 } QEMUIOVector;
 
 void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint);
include/sysemu/block-backend.h +10 -0
index 8871a02..c34855e 100644
@@ -94,17 +94,27 @@ void blk_get_geometry(BlockBackend *blk, uint64_t *nb_sectors_ptr);
 BlockAIOCB *blk_aio_readv(BlockBackend *blk, int64_t sector_num,
                           QEMUIOVector *iov, int nb_sectors,
                           BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *blk_aio_readv_replay(BlockBackend *blk, int64_t sector_num,
+                                 QEMUIOVector *iov, int nb_sectors,
+                                 BlockCompletionFunc *cb, void *opaque);
 BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,
                            QEMUIOVector *iov, int nb_sectors,
                            BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *blk_aio_writev_replay(BlockBackend *blk, int64_t sector_num,
+                                  QEMUIOVector *iov, int nb_sectors,
+                                  BlockCompletionFunc *cb, void *opaque);
 BlockAIOCB *blk_aio_flush(BlockBackend *blk,
                           BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *blk_aio_flush_replay(BlockBackend *blk,
+                                 BlockCompletionFunc *cb, void *opaque);
 BlockAIOCB *blk_aio_discard(BlockBackend *blk,
                             int64_t sector_num, int nb_sectors,
                             BlockCompletionFunc *cb, void *opaque);
 void blk_aio_cancel(BlockAIOCB *acb);
 void blk_aio_cancel_async(BlockAIOCB *acb);
 int blk_aio_multiwrite(BlockBackend *blk, BlockRequest *reqs, int num_reqs);
+int blk_aio_multiwrite_replay(BlockBackend *blk, BlockRequest *reqs,
+                              int num_reqs);
 int blk_ioctl(BlockBackend *blk, unsigned long int req, void *buf);
 BlockAIOCB *blk_aio_ioctl(BlockBackend *blk, unsigned long int req, void *buf,
                           BlockCompletionFunc *cb, void *opaque);
qemu-io-cmds.c +1 -1
index e708552..a7d8d4e 100644
@@ -583,7 +583,7 @@ static int do_aio_multiwrite(BlockDriverState *bs, BlockRequest* reqs,
         *total += reqs[i].qiov->size;
     }
 
-    ret = bdrv_aio_multiwrite(bs, reqs, num_reqs);
+    ret = bdrv_aio_multiwrite(bs, reqs, num_reqs, false);
     if (ret < 0) {
         return ret;
     }
stubs/replay.c +5 -0
index 268f3e0..95b43f3 100755
@@ -25,3 +25,8 @@ int runstate_is_running(void)
 void replay_add_bh_event(void *bh, uint64_t id)
 {
 }
+
+uint64_t replay_get_current_step(void)
+{
+    return 0;
+}
trace-events +2 -0
index 4ec81eb..2dd21ef 100644
@@ -64,7 +64,9 @@ bdrv_aio_multiwrite(void *mcb, int num_callbacks, int num_reqs) "mcb %p num_call
 bdrv_aio_discard(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_flush(void *bs, void *opaque) "bs %p opaque %p"
 bdrv_aio_readv(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
+bdrv_aio_readv_replay(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_writev(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
+bdrv_aio_writev_replay(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_write_zeroes(void *bs, int64_t sector_num, int nb_sectors, int flags, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d flags %#x opaque %p"
 bdrv_lock_medium(void *bs, bool locked) "bs %p locked %d"
 bdrv_co_readv(void *bs, int64_t sector_num, int nb_sector) "bs %p sector_num %"PRId64" nb_sectors %d"
util/iov.c +4 -0
index 2fb18e6..96517e7 100644
@@ -257,6 +257,8 @@ void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint)
     qiov->niov = 0;
     qiov->nalloc = alloc_hint;
     qiov->size = 0;
+    qiov->replay = false;
+    qiov->replay_step = 0;
 }
 
 void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov)
@@ -267,6 +269,8 @@ void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov)
     qiov->niov = niov;
     qiov->nalloc = -1;
     qiov->size = 0;
+    qiov->replay = false;
+    qiov->replay_step = 0;
     for (i = 0; i < niov; i++)
         qiov->size += iov[i].iov_len;
 }
#64 lore raw

Re: [Qemu-devel] [RFC PATCH v8 17/21] replay: replay aio requests

Quoted 6 lines · Pavel Dovgalyuk

This patch adds identifier to aio requests. ID is used for creating bottom halves and identifying them while replaying.
The patch also introduces several functions that make possible replaying of the aio requests.

Ah, some obscure parts of patch 16 now get clearer. :)

But it's still not clear what is the design. You have to document this too, then I (or others) can start reviewing the patches.

Paolo

Quoted 557 lines
---
 block.c                        |   81 ++++++++++++++++++++++++++++++++++++----
 block/block-backend.c          |   30 ++++++++++++++-
 block/qcow2.c                  |    4 ++
 dma-helpers.c                  |    6 ++-
 hw/block/virtio-blk.c          |   10 ++---
 hw/ide/atapi.c                 |   10 +++--
 hw/ide/core.c                  |   14 ++++---
 include/block/block.h          |   15 +++++++
 include/qemu-common.h          |    2 +
 include/sysemu/block-backend.h |   10 +++++
 qemu-io-cmds.c                 |    2 -
 stubs/replay.c                 |    5 ++
 trace-events                   |    2 +
 util/iov.c                     |    4 ++
 14 files changed, 167 insertions(+), 28 deletions(-)
block.c +72 -9
index a4f45c3..7f6fa8b 100644
@@ -83,7 +83,8 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
                                          BdrvRequestFlags flags,
                                          BlockCompletionFunc *cb,
                                          void *opaque,
-                                         bool is_write);
+                                         bool is_write,
+                                         bool aio_replay);
 static void coroutine_fn bdrv_co_do_rw(void *opaque);
 static int coroutine_fn bdrv_co_do_write_zeroes(BlockDriverState *bs,
     int64_t sector_num, int nb_sectors, BdrvRequestFlags flags);
@@ -4425,7 +4426,19 @@ BlockAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,
     trace_bdrv_aio_readv(bs, sector_num, nb_sectors, opaque);
 
     return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
-                                 cb, opaque, false);
+                                 cb, opaque, false, false);
+}
+
+BlockAIOCB *bdrv_aio_readv_replay(BlockDriverState *bs,
+                                  int64_t sector_num,
+                                  QEMUIOVector *qiov, int nb_sectors,
+                                  BlockCompletionFunc *cb,
+                                  void *opaque)
+{
+    trace_bdrv_aio_readv_replay(bs, sector_num, nb_sectors, opaque);
+
+    return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
+                                 cb, opaque, false, true);
 }
 
 BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
@@ -4435,7 +4448,19 @@ BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
     trace_bdrv_aio_writev(bs, sector_num, nb_sectors, opaque);
 
     return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
-                                 cb, opaque, true);
+                                 cb, opaque, true, false);
+}
+
+BlockAIOCB *bdrv_aio_writev_replay(BlockDriverState *bs,
+                                   int64_t sector_num,
+                                   QEMUIOVector *qiov, int nb_sectors,
+                                   BlockCompletionFunc *cb,
+                                   void *opaque)
+{
+    trace_bdrv_aio_writev_replay(bs, sector_num, nb_sectors, opaque);
+
+    return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
+                                 cb, opaque, true, true);
 }
 
 BlockAIOCB *bdrv_aio_write_zeroes(BlockDriverState *bs,
@@ -4446,7 +4471,7 @@ BlockAIOCB *bdrv_aio_write_zeroes(BlockDriverState *bs,
 
     return bdrv_co_aio_rw_vector(bs, sector_num, NULL, nb_sectors,
                                  BDRV_REQ_ZERO_WRITE | flags,
-                                 cb, opaque, true);
+                                 cb, opaque, true, true);
 }
 
 
@@ -4593,7 +4618,8 @@ static int multiwrite_merge(BlockDriverState *bs, BlockRequest *reqs,
  * requests. However, the fields opaque and error are left unmodified as they
  * are used to signal failure for a single request to the caller.
  */
-int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs)
+int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs,
+                        bool replay)
 {
     MultiwriteCB *mcb;
     int i;
@@ -4631,7 +4657,7 @@ int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs)
         bdrv_co_aio_rw_vector(bs, reqs[i].sector, reqs[i].qiov,
                               reqs[i].nb_sectors, reqs[i].flags,
                               multiwrite_cb, mcb,
-                              true);
+                              true, replay);
     }
 
     return 0;
@@ -4776,7 +4802,12 @@ static void coroutine_fn bdrv_co_do_rw(void *opaque)
             acb->req.nb_sectors, acb->req.qiov, acb->req.flags);
     }
 
-    acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    if (acb->common.replay) {
+        acb->bh = aio_bh_new_replay(bdrv_get_aio_context(bs), bdrv_co_em_bh,
+                                    acb, acb->common.replay_step);
+    } else {
+        acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    }
     qemu_bh_schedule(acb->bh);
 }
 
@@ -4787,7 +4818,8 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
                                          BdrvRequestFlags flags,
                                          BlockCompletionFunc *cb,
                                          void *opaque,
-                                         bool is_write)
+                                         bool is_write,
+                                         bool aio_replay)
 {
     Coroutine *co;
     BlockAIOCBCoroutine *acb;
@@ -4798,6 +4830,11 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
     acb->req.qiov = qiov;
     acb->req.flags = flags;
     acb->is_write = is_write;
+    acb->done = NULL;
+    acb->common.replay = aio_replay;
+    if (aio_replay) {
+        acb->common.replay_step = replay_get_current_step();
+    }
 
     co = qemu_coroutine_create(bdrv_co_do_rw);
     qemu_coroutine_enter(co, acb);
@@ -4811,7 +4848,12 @@ static void coroutine_fn bdrv_aio_flush_co_entry(void *opaque)
     BlockDriverState *bs = acb->common.bs;
 
     acb->req.error = bdrv_co_flush(bs);
-    acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    if (acb->common.replay) {
+        acb->bh = aio_bh_new_replay(bdrv_get_aio_context(bs), bdrv_co_em_bh,
+                                    acb, acb->common.replay_step);
+    } else {
+        acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    }
     qemu_bh_schedule(acb->bh);
 }
 
@@ -4831,6 +4873,25 @@ BlockAIOCB *bdrv_aio_flush(BlockDriverState *bs,
     return &acb->common;
 }
 
+BlockAIOCB *bdrv_aio_flush_replay(BlockDriverState *bs,
+        BlockCompletionFunc *cb, void *opaque)
+{
+    trace_bdrv_aio_flush(bs, opaque);
+
+    Coroutine *co;
+    BlockAIOCBCoroutine *acb;
+
+    acb = qemu_aio_get(&bdrv_em_co_aiocb_info, bs, cb, opaque);
+    acb->done = NULL;
+    acb->common.replay = true;
+    acb->common.replay_step = replay_get_current_step();
+
+    co = qemu_coroutine_create(bdrv_aio_flush_co_entry);
+    qemu_coroutine_enter(co, acb);
+
+    return &acb->common;
+}
+
 static void coroutine_fn bdrv_aio_discard_co_entry(void *opaque)
 {
     BlockAIOCBCoroutine *acb = opaque;
@@ -4881,6 +4942,8 @@ void *qemu_aio_get(const AIOCBInfo *aiocb_info, BlockDriverState *bs,
     acb->cb = cb;
     acb->opaque = opaque;
     acb->refcnt = 1;
+    acb->replay_step = 0;
+    acb->replay = false;
     return acb;
 }
 
block/block-backend.c +29 -1
index d00c129..451041a 100644
@@ -447,6 +447,14 @@ BlockAIOCB *blk_aio_readv(BlockBackend *blk, int64_t sector_num,
     return bdrv_aio_readv(blk->bs, sector_num, iov, nb_sectors, cb, opaque);
 }
 
+BlockAIOCB *blk_aio_readv_replay(BlockBackend *blk, int64_t sector_num,
+                                 QEMUIOVector *iov, int nb_sectors,
+                                 BlockCompletionFunc *cb, void *opaque)
+{
+    return bdrv_aio_readv_replay(blk->bs, sector_num, iov, nb_sectors,
+                                 cb, opaque);
+}
+
 BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,
                            QEMUIOVector *iov, int nb_sectors,
                            BlockCompletionFunc *cb, void *opaque)
@@ -454,12 +462,26 @@ BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,
     return bdrv_aio_writev(blk->bs, sector_num, iov, nb_sectors, cb, opaque);
 }
 
+BlockAIOCB *blk_aio_writev_replay(BlockBackend *blk, int64_t sector_num,
+                                  QEMUIOVector *iov, int nb_sectors,
+                                  BlockCompletionFunc *cb, void *opaque)
+{
+    return bdrv_aio_writev_replay(blk->bs, sector_num, iov, nb_sectors,
+                                  cb, opaque);
+}
+
 BlockAIOCB *blk_aio_flush(BlockBackend *blk,
                           BlockCompletionFunc *cb, void *opaque)
 {
     return bdrv_aio_flush(blk->bs, cb, opaque);
 }
 
+BlockAIOCB *blk_aio_flush_replay(BlockBackend *blk,
+                                 BlockCompletionFunc *cb, void *opaque)
+{
+    return bdrv_aio_flush_replay(blk->bs, cb, opaque);
+}
+
 BlockAIOCB *blk_aio_discard(BlockBackend *blk,
                             int64_t sector_num, int nb_sectors,
                             BlockCompletionFunc *cb, void *opaque)
@@ -479,7 +501,13 @@ void blk_aio_cancel_async(BlockAIOCB *acb)
 
 int blk_aio_multiwrite(BlockBackend *blk, BlockRequest *reqs, int num_reqs)
 {
-    return bdrv_aio_multiwrite(blk->bs, reqs, num_reqs);
+    return bdrv_aio_multiwrite(blk->bs, reqs, num_reqs, false);
+}
+
+int blk_aio_multiwrite_replay(BlockBackend *blk, BlockRequest *reqs,
+                              int num_reqs)
+{
+    return bdrv_aio_multiwrite(blk->bs, reqs, num_reqs, true);
 }
 
 int blk_ioctl(BlockBackend *blk, unsigned long int req, void *buf)
block/qcow2.c +4 -0
index e4e690a..4a2f6ae 100644
@@ -1140,6 +1140,8 @@ static coroutine_fn int qcow2_co_readv(BlockDriverState *bs, int64_t sector_num,
     uint8_t *cluster_data = NULL;
 
     qemu_iovec_init(&hd_qiov, qiov->niov);
+    hd_qiov.replay = qiov->replay;
+    hd_qiov.replay_step = qiov->replay_step;
 
     qemu_co_mutex_lock(&s->lock);
 
@@ -1297,6 +1299,8 @@ static coroutine_fn int qcow2_co_writev(BlockDriverState *bs,
                                  remaining_sectors);
 
     qemu_iovec_init(&hd_qiov, qiov->niov);
+    hd_qiov.replay = qiov->replay;
+    hd_qiov.replay_step = qiov->replay_step;
 
     s->cluster_cache_offset = -1; /* disable compressed cache */
 
dma-helpers.c +4 -2
index 357d7e9..4faf6d3 100644
@@ -212,6 +212,8 @@ BlockAIOCB *dma_blk_io(
     dbs->io_func = io_func;
     dbs->bh = NULL;
     qemu_iovec_init(&dbs->iov, sg->nsg);
+    dbs->iov.replay = true;
+    dbs->iov.replay_step = replay_get_current_step();
     dma_blk_cb(dbs, 0);
     return &dbs->common;
 }
@@ -221,7 +223,7 @@ BlockAIOCB *dma_blk_read(BlockBackend *blk,
                          QEMUSGList *sg, uint64_t sector,
                          void (*cb)(void *opaque, int ret), void *opaque)
 {
-    return dma_blk_io(blk, sg, sector, blk_aio_readv, cb, opaque,
+    return dma_blk_io(blk, sg, sector, blk_aio_readv_replay, cb, opaque,
                       DMA_DIRECTION_FROM_DEVICE);
 }
 
@@ -229,7 +231,7 @@ BlockAIOCB *dma_blk_write(BlockBackend *blk,
                           QEMUSGList *sg, uint64_t sector,
                           void (*cb)(void *opaque, int ret), void *opaque)
 {
-    return dma_blk_io(blk, sg, sector, blk_aio_writev, cb, opaque,
+    return dma_blk_io(blk, sg, sector, blk_aio_writev_replay, cb, opaque,
                       DMA_DIRECTION_TO_DEVICE);
 }
 
hw/block/virtio-blk.c +5 -5
index b19b102..2413cbe 100644
@@ -265,7 +265,7 @@ void virtio_submit_multiwrite(BlockBackend *blk, MultiReqBuffer *mrb)
         return;
     }
 
-    ret = blk_aio_multiwrite(blk, mrb->blkreq, mrb->num_writes);
+    ret = blk_aio_multiwrite_replay(blk, mrb->blkreq, mrb->num_writes);
     if (ret != 0) {
         for (i = 0; i < mrb->num_writes; i++) {
             if (mrb->blkreq[i].error) {
@@ -286,7 +286,7 @@ static void virtio_blk_handle_flush(VirtIOBlockReq *req, MultiReqBuffer *mrb)
      * Make sure all outstanding writes are posted to the backing device.
      */
     virtio_submit_multiwrite(req->dev->blk, mrb);
-    blk_aio_flush(req->dev->blk, virtio_blk_flush_complete, req);
+    blk_aio_flush_replay(req->dev->blk, virtio_blk_flush_complete, req);
 }
 
 static bool virtio_blk_sect_range_ok(VirtIOBlock *dev,
@@ -357,9 +357,9 @@ static void virtio_blk_handle_read(VirtIOBlockReq *req)
 
     block_acct_start(blk_get_stats(req->dev->blk), &req->acct, req->qiov.size,
                      BLOCK_ACCT_READ);
-    blk_aio_readv(req->dev->blk, sector, &req->qiov,
-                  req->qiov.size / BDRV_SECTOR_SIZE,
-                  virtio_blk_rw_complete, req);
+    blk_aio_readv_replay(req->dev->blk, sector, &req->qiov,
+                          req->qiov.size / BDRV_SECTOR_SIZE,
+                          virtio_blk_rw_complete, req);
 }
 
 void virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb)
hw/ide/atapi.c +6 -4
index a71e6e0..a69b620 100644
@@ -25,7 +25,7 @@
 
 #include "hw/ide/internal.h"
 #include "hw/scsi/scsi.h"
-#include "sysemu/block-backend.h"
+#include "replay/replay.h"
 
 static void ide_atapi_cmd_read_dma_cb(void *opaque, int ret);
 
@@ -350,10 +350,12 @@ static void ide_atapi_cmd_read_dma_cb(void *opaque, int ret)
     s->bus->dma->iov.iov_base = (void *)(s->io_buffer + data_offset);
     s->bus->dma->iov.iov_len = n * 4 * 512;
     qemu_iovec_init_external(&s->bus->dma->qiov, &s->bus->dma->iov, 1);
+    s->bus->dma->qiov.replay = true;
+    s->bus->dma->qiov.replay_step = replay_get_current_step();
 
-    s->bus->dma->aiocb = blk_aio_readv(s->blk, (int64_t)s->lba << 2,
-                                       &s->bus->dma->qiov, n * 4,
-                                       ide_atapi_cmd_read_dma_cb, s);
+    s->bus->dma->aiocb = blk_aio_readv_replay(s->blk, (int64_t)s->lba << 2,
+                                              &s->bus->dma->qiov, n * 4,
+                                              ide_atapi_cmd_read_dma_cb, s);
     return;
 
 eot:
hw/ide/core.c +9 -5
index d76244a..1fff193 100644
@@ -630,11 +630,13 @@ void ide_sector_read(IDEState *s)
     s->iov.iov_base = s->io_buffer;
     s->iov.iov_len  = n * BDRV_SECTOR_SIZE;
     qemu_iovec_init_external(&s->qiov, &s->iov, 1);
+    s->qiov.replay = true;
+    s->qiov.replay_step = replay_get_current_step();
 
     block_acct_start(blk_get_stats(s->blk), &s->acct,
                      n * BDRV_SECTOR_SIZE, BLOCK_ACCT_READ);
-    s->pio_aiocb = blk_aio_readv(s->blk, sector_num, &s->qiov, n,
-                                 ide_sector_read_cb, s);
+    s->pio_aiocb = blk_aio_readv_replay(s->blk, sector_num, &s->qiov, n,
+                                        ide_sector_read_cb, s);
 }
 
 static void dma_buf_commit(IDEState *s, uint32_t tx_bytes)
@@ -888,11 +890,13 @@ void ide_sector_write(IDEState *s)
     s->iov.iov_base = s->io_buffer;
     s->iov.iov_len  = n * BDRV_SECTOR_SIZE;
     qemu_iovec_init_external(&s->qiov, &s->iov, 1);
+    s->qiov.replay = true;
+    s->qiov.replay_step = replay_get_current_step();
 
     block_acct_start(blk_get_stats(s->blk), &s->acct,
                      n * BDRV_SECTOR_SIZE, BLOCK_ACCT_READ);
-    s->pio_aiocb = blk_aio_writev(s->blk, sector_num, &s->qiov, n,
-                                  ide_sector_write_cb, s);
+    s->pio_aiocb = blk_aio_writev_replay(s->blk, sector_num, &s->qiov, n,
+                                         ide_sector_write_cb, s);
 }
 
 static void ide_flush_cb(void *opaque, int ret)
@@ -928,7 +932,7 @@ void ide_flush_cache(IDEState *s)
 
     s->status |= BUSY_STAT;
     block_acct_start(blk_get_stats(s->blk), &s->acct, 0, BLOCK_ACCT_FLUSH);
-    s->pio_aiocb = blk_aio_flush(s->blk, ide_flush_cb, s);
+    s->pio_aiocb = blk_aio_flush_replay(s->blk, ide_flush_cb, s);
 }
 
 static void ide_cfata_metadata_inquiry(IDEState *s)
include/block/block.h +14 -1
index 3082d2b..06ce097 100644
@@ -293,11 +293,24 @@ typedef void BlockDriverDirtyHandler(BlockDriverState *bs, int64_t sector,
 BlockAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,
                            QEMUIOVector *iov, int nb_sectors,
                            BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *bdrv_aio_readv_replay(BlockDriverState *bs,
+                                  int64_t sector_num,
+                                  QEMUIOVector *iov, int nb_sectors,
+                                  BlockCompletionFunc *cb,
+                                  void *opaque);
 BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
                             QEMUIOVector *iov, int nb_sectors,
                             BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *bdrv_aio_writev_replay(BlockDriverState *bs,
+                                   int64_t sector_num,
+                                   QEMUIOVector *iov, int nb_sectors,
+                                   BlockCompletionFunc *cb,
+                                   void *opaque);
 BlockAIOCB *bdrv_aio_flush(BlockDriverState *bs,
                            BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *bdrv_aio_flush_replay(BlockDriverState *bs,
+                                  BlockCompletionFunc *cb,
+                                  void *opaque);
 BlockAIOCB *bdrv_aio_discard(BlockDriverState *bs,
                              int64_t sector_num, int nb_sectors,
                              BlockCompletionFunc *cb, void *opaque);
@@ -318,7 +331,7 @@ typedef struct BlockRequest {
 } BlockRequest;
 
 int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs,
-    int num_reqs);
+                        int num_reqs, bool replay);
 
 /* sg packet commands */
 int bdrv_ioctl(BlockDriverState *bs, unsigned long int req, void *buf);
include/qemu-common.h +2 -0
index 8c9957e..016556a 100644
@@ -320,6 +320,8 @@ typedef struct QEMUIOVector {
     int niov;
     int nalloc;
     size_t size;
+    bool replay;
+    uint64_t replay_step;
 } QEMUIOVector;
 
 void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint);
include/sysemu/block-backend.h +10 -0
index 8871a02..c34855e 100644
@@ -94,17 +94,27 @@ void blk_get_geometry(BlockBackend *blk, uint64_t *nb_sectors_ptr);
 BlockAIOCB *blk_aio_readv(BlockBackend *blk, int64_t sector_num,
                           QEMUIOVector *iov, int nb_sectors,
                           BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *blk_aio_readv_replay(BlockBackend *blk, int64_t sector_num,
+                                 QEMUIOVector *iov, int nb_sectors,
+                                 BlockCompletionFunc *cb, void *opaque);
 BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,
                            QEMUIOVector *iov, int nb_sectors,
                            BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *blk_aio_writev_replay(BlockBackend *blk, int64_t sector_num,
+                                  QEMUIOVector *iov, int nb_sectors,
+                                  BlockCompletionFunc *cb, void *opaque);
 BlockAIOCB *blk_aio_flush(BlockBackend *blk,
                           BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *blk_aio_flush_replay(BlockBackend *blk,
+                                 BlockCompletionFunc *cb, void *opaque);
 BlockAIOCB *blk_aio_discard(BlockBackend *blk,
                             int64_t sector_num, int nb_sectors,
                             BlockCompletionFunc *cb, void *opaque);
 void blk_aio_cancel(BlockAIOCB *acb);
 void blk_aio_cancel_async(BlockAIOCB *acb);
 int blk_aio_multiwrite(BlockBackend *blk, BlockRequest *reqs, int num_reqs);
+int blk_aio_multiwrite_replay(BlockBackend *blk, BlockRequest *reqs,
+                              int num_reqs);
 int blk_ioctl(BlockBackend *blk, unsigned long int req, void *buf);
 BlockAIOCB *blk_aio_ioctl(BlockBackend *blk, unsigned long int req, void *buf,
                           BlockCompletionFunc *cb, void *opaque);
qemu-io-cmds.c +1 -1
index e708552..a7d8d4e 100644
@@ -583,7 +583,7 @@ static int do_aio_multiwrite(BlockDriverState *bs, BlockRequest* reqs,
         *total += reqs[i].qiov->size;
     }
 
-    ret = bdrv_aio_multiwrite(bs, reqs, num_reqs);
+    ret = bdrv_aio_multiwrite(bs, reqs, num_reqs, false);
     if (ret < 0) {
         return ret;
     }
stubs/replay.c +5 -0
index 268f3e0..95b43f3 100755
@@ -25,3 +25,8 @@ int runstate_is_running(void)
 void replay_add_bh_event(void *bh, uint64_t id)
 {
 }
+
+uint64_t replay_get_current_step(void)
+{
+    return 0;
+}
trace-events +2 -0
index 4ec81eb..2dd21ef 100644
@@ -64,7 +64,9 @@ bdrv_aio_multiwrite(void *mcb, int num_callbacks, int num_reqs) "mcb %p num_call
 bdrv_aio_discard(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_flush(void *bs, void *opaque) "bs %p opaque %p"
 bdrv_aio_readv(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
+bdrv_aio_readv_replay(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_writev(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
+bdrv_aio_writev_replay(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_write_zeroes(void *bs, int64_t sector_num, int nb_sectors, int flags, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d flags %#x opaque %p"
 bdrv_lock_medium(void *bs, bool locked) "bs %p locked %d"
 bdrv_co_readv(void *bs, int64_t sector_num, int nb_sector) "bs %p sector_num %"PRId64" nb_sectors %d"
util/iov.c +4 -0
index 2fb18e6..96517e7 100644
@@ -257,6 +257,8 @@ void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint)
     qiov->niov = 0;
     qiov->nalloc = alloc_hint;
     qiov->size = 0;
+    qiov->replay = false;
+    qiov->replay_step = 0;
 }
 
 void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov)
@@ -267,6 +269,8 @@ void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov)
     qiov->niov = niov;
     qiov->nalloc = -1;
     qiov->size = 0;
+    qiov->replay = false;
+    qiov->replay_step = 0;
     for (i = 0; i < niov; i++)
         qiov->size += iov[i].iov_len;
 }
#65 lore raw

RFC PATCH v8 18/21 replay: thread pool

In reply to Pavel Dovgalyuk

This patch modifies thread pool to allow replaying asynchronous thread tasks synchronously in replay mode.

---
 block/raw-posix.c           |    6 ++++-
 block/raw-win32.c           |    4 +++-
 include/block/thread-pool.h |    4 +++-
 replay/replay-events.c      |   11 ++++++++++
 replay/replay-internal.h    |    1 +
 replay/replay.h             |    2 ++
 stubs/replay.c              |    4 ++++
 tests/test-thread-pool.c    |    7 ++++--
 thread-pool.c               |   49 ++++++++++++++++++++++++++++++-------------
 9 files changed, 66 insertions(+), 22 deletions(-)
block/raw-posix.c +4 -2
index e51293a..f878e06 100644
@@ -1073,7 +1073,9 @@ static BlockAIOCB *paio_submit(BlockDriverState *bs, int fd,
 
     trace_paio_submit(acb, opaque, sector_num, nb_sectors, type);
     pool = aio_get_thread_pool(bdrv_get_aio_context(bs));
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque,
+                                  qiov ? qiov->replay : false,
+                                  qiov ? qiov->replay_step : 0);
 }
 
 static BlockAIOCB *raw_aio_submit(BlockDriverState *bs,
@@ -1986,7 +1988,7 @@ static BlockAIOCB *hdev_aio_ioctl(BlockDriverState *bs,
     acb->aio_ioctl_buf = buf;
     acb->aio_ioctl_cmd = req;
     pool = aio_get_thread_pool(bdrv_get_aio_context(bs));
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque, false, 0);
 }
 
 #elif defined(__FreeBSD__) || defined(__FreeBSD_kernel__)
block/raw-win32.c +3 -1
index 06243d7..7ac693b 100644
@@ -158,7 +158,9 @@ static BlockAIOCB *paio_submit(BlockDriverState *bs, HANDLE hfile,
 
     trace_paio_submit(acb, opaque, sector_num, nb_sectors, type);
     pool = aio_get_thread_pool(bdrv_get_aio_context(bs));
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque,
+                                  qiov ? qiov->replay : false,
+                                  qiov ? qiov->replay_step : 0);
 }
 
 int qemu_ftruncate64(int fd, int64_t length)
include/block/thread-pool.h +3 -1
index 42eb5e8..801ac00 100644
@@ -29,9 +29,11 @@ void thread_pool_free(ThreadPool *pool);
 
 BlockAIOCB *thread_pool_submit_aio(ThreadPool *pool,
         ThreadPoolFunc *func, void *arg,
-        BlockCompletionFunc *cb, void *opaque);
+        BlockCompletionFunc *cb, void *opaque,
+        bool replay, uint64_t replay_step);
 int coroutine_fn thread_pool_submit_co(ThreadPool *pool,
         ThreadPoolFunc *func, void *arg);
 void thread_pool_submit(ThreadPool *pool, ThreadPoolFunc *func, void *arg);
+void thread_pool_work(ThreadPool *pool, void *r);
 
 #endif
replay/replay-events.c +11 -0
index 7e82b61..f4ec702 100755
@@ -12,6 +12,7 @@
 #include "qemu-common.h"
 #include "replay.h"
 #include "replay-internal.h"
+#include "block/thread-pool.h"
 
 typedef struct Event {
     ReplayAsyncEventKind event_kind;
@@ -39,6 +40,9 @@ static void replay_run_event(Event *event)
     case REPLAY_ASYNC_EVENT_BH:
         aio_bh_call(event->opaque);
         break;
+    case REPLAY_ASYNC_EVENT_THREAD:
+        thread_pool_work((ThreadPool *)event->opaque, event->opaque2);
+        break;
     default:
         fprintf(stderr, "Replay: invalid async event ID (%d) in the queue\n",
                 event->event_kind);
@@ -130,6 +134,11 @@ void replay_add_bh_event(void *bh, uint64_t id)
     replay_add_event_internal(REPLAY_ASYNC_EVENT_BH, bh, NULL, id);
 }
 
+void replay_add_thread_event(void *opaque, void *opaque2, uint64_t id)
+{
+    replay_add_event_internal(REPLAY_ASYNC_EVENT_THREAD, opaque, opaque2, id);
+}
+
 /* Called with replay mutex locked */
 void replay_save_events(int opt)
 {
@@ -145,6 +154,7 @@ void replay_save_events(int opt)
             /* save event-specific data */
             switch (event->event_kind) {
             case REPLAY_ASYNC_EVENT_BH:
+            case REPLAY_ASYNC_EVENT_THREAD:
                 replay_put_qword(event->id);
                 break;
             }
@@ -179,6 +189,7 @@ void replay_read_events(int opt)
         /* Execute some events without searching them in the queue */
         switch (read_event_kind) {
         case REPLAY_ASYNC_EVENT_BH:
+        case REPLAY_ASYNC_EVENT_THREAD:
             if (read_id == -1) {
                 read_id = replay_get_qword();
             }
replay/replay-internal.h +1 -0
index 0e6ba2a..4d242aa 100755
@@ -41,6 +41,7 @@ enum ReplayEvents {
 
 enum ReplayAsyncEventKind {
     REPLAY_ASYNC_EVENT_BH,
+    REPLAY_ASYNC_EVENT_THREAD,
     REPLAY_ASYNC_COUNT
 };
 
replay/replay.h +2 -0
index 073e105..3110d46 100755
@@ -108,5 +108,7 @@ bool replay_checkpoint(ReplayCheckpoint checkpoint);
 void replay_disable_events(void);
 /*! Adds BH event to the queue */
 void replay_add_bh_event(void *bh, uint64_t id);
+/*! Adds thread event to the queue */
+void replay_add_thread_event(void *pool, void *req, uint64_t id);
 
 #endif
stubs/replay.c +4 -0
index 95b43f3..81eddae 100755
@@ -30,3 +30,7 @@ uint64_t replay_get_current_step(void)
 {
     return 0;
 }
+
+void replay_add_thread_event(void *opaque, void *opaque2, uint64_t id)
+{
+}
tests/test-thread-pool.c +4 -3
index 6a0b981..f32594c 100644
@@ -56,7 +56,7 @@ static void test_submit_aio(void)
 {
     WorkerTestData data = { .n = 0, .ret = -EINPROGRESS };
     data.aiocb = thread_pool_submit_aio(pool, worker_cb, &data,
-                                        done_cb, &data);
+                                        done_cb, &data, false, 0);
 
     /* The callbacks are not called until after the first wait.  */
     active = 1;
@@ -120,7 +120,8 @@ static void test_submit_many(void)
     for (i = 0; i < 100; i++) {
         data[i].n = 0;
         data[i].ret = -EINPROGRESS;
-        thread_pool_submit_aio(pool, worker_cb, &data[i], done_cb, &data[i]);
+        thread_pool_submit_aio(pool, worker_cb, &data[i], done_cb, &data[i],
+                               false, 0);
     }
 
     active = 100;
@@ -149,7 +150,7 @@ static void do_test_cancel(bool sync)
         data[i].n = 0;
         data[i].ret = -EINPROGRESS;
         data[i].aiocb = thread_pool_submit_aio(pool, long_cb, &data[i],
-                                               done_cb, &data[i]);
+                                               done_cb, &data[i], false, 0);
     }
 
     /* Starting the threads may be left to a bottom half.  Let it
thread-pool.c +34 -15

99 lines not on this page · raw message

#66 lore raw

Re: [Qemu-devel] [RFC PATCH v8 18/21] replay: thread pool

block/raw-posix.c static BlockAIOCB *paio_submit(BlockDriverState *bs, int fd, · line 1078

Quoted 4 lines · Pavel Dovgalyuk
block/raw-posix.c +3 -1
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque,
+                                  qiov ? qiov->replay : false,
+                                  qiov ? qiov->replay_step : 0);

Adding the replay/replay_step to the QEMUIOVector is... weird. You're paying for the fact that we do not have a BlockRequest struct that is used throughout.

I guess we can live with this until other problems are solved, then we can discuss this with the block folks and go for a more complete solution.

Paolo

#67 lore raw

RFC PATCH v8 19/21 replay: initialization and deinitialization

In reply to Pavel Dovgalyuk

This patch introduces the functions for enabling the record/replay and for freeing the resources when simulator closes.

---
 block.c                  |    2 -
 exec.c                   |    1 
 replay/replay-internal.c |    1 
 replay/replay-internal.h |    4 +
 replay/replay.c          |  140 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   13 ++++
 stubs/replay.c           |    1 
 vl.c                     |    5 ++
 8 files changed, 165 insertions(+), 2 deletions(-)
block.c +1 -1
index 7f6fa8b..9923a80 100644
@@ -2010,7 +2010,7 @@ void bdrv_drain_all(void)
             aio_context_release(aio_context);
         }
     }
-    if (replay_mode == REPLAY_MODE_PLAY) {
+    if (replay_mode == REPLAY_MODE_PLAY && replay_checkpoints) {
         /* Skip checkpoints from the log */
         while (replay_checkpoint(CHECKPOINT_BDRV_DRAIN)) {
             /* Nothing */
exec.c +1 -0
index 410371d..f70f03d 100644
@@ -783,6 +783,7 @@ void cpu_abort(CPUState *cpu, const char *fmt, ...)
     }
     va_end(ap2);
     va_end(ap);
+    replay_finish();
 #if defined(CONFIG_USER_ONLY)
     {
         struct sigaction act;
replay/replay-internal.c +1 -0
index 49b37a6..d608b71 100755
@@ -33,6 +33,7 @@ void replay_put_byte(uint8_t byte)
 
 void replay_put_event(uint8_t event)
 {
+    assert(event <= EVENT_END);
     replay_put_byte(event);
 }
 
replay/replay-internal.h +3 -1
index 4d242aa..1e5d037 100755
@@ -34,7 +34,9 @@ enum ReplayEvents {
     EVENT_CLOCK,
     /* for checkpoint event */
     /* some of grteater codes are reserved for checkpoints */
-    EVENT_CHECKPOINT = EVENT_CLOCK + REPLAY_CLOCK_COUNT
+    EVENT_CHECKPOINT = EVENT_CLOCK + REPLAY_CLOCK_COUNT,
+    /* end of log event */
+    EVENT_END = EVENT_CHECKPOINT + CHECKPOINT_COUNT
 };
 
 /* Asynchronous events IDs */
replay/replay.c +140 -0

158 lines not on this page · raw message

replay/replay.h +13 -0
index 3110d46..231b8ec 100755
@@ -17,6 +17,8 @@
 #include <time.h>
 #include "qapi-types.h"
 
+struct QemuOpts;
+
 /* replay clock kinds */
 enum ReplayClockKind {
     /* rdtsc */
@@ -45,6 +47,17 @@ enum ReplayCheckpoint {
 typedef enum ReplayCheckpoint ReplayCheckpoint;
 
 extern ReplayMode replay_mode;
+/*! Is set to true after finishing initialization */
+extern bool replay_checkpoints;
+
+/* Replay process control functions */
+
+/*! Enables recording or saving event log with specified parameters */
+void replay_configure(struct QemuOpts *opts);
+/*! Initializes timers used for snapshotting and enables events recording */
+void replay_init_timer(void);
+/*! Closes replay log file and frees other resources. */
+void replay_finish(void);
 
 /* Processing the instructions */
 
stubs/replay.c +1 -0
index 81eddae..7d87964 100755
@@ -2,6 +2,7 @@
 #include "sysemu/sysemu.h"
 
 ReplayMode replay_mode;
+bool replay_checkpoints;
 
 int64_t replay_save_clock(unsigned int kind, int64_t clock)
 {
vl.c +5 -0
index 86ba385..ae3e97e 100644
@@ -4376,7 +4376,12 @@ int main(int argc, char **argv, char **envp)
         }
     }
 
+    replay_init_timer();
+
     main_loop();
+    if (replay_mode != REPLAY_MODE_NONE) {
+        replay_disable_events();
+    }
     bdrv_close_all();
     pause_all_vcpus();
     res_free();
#68 lore raw

Re: [Qemu-devel] [RFC PATCH v8 19/21] replay: initialization and deinitialization

Quoted 45 lines · Pavel Dovgalyuk

This patch introduces the functions for enabling the record/replay and for freeing the resources when simulator closes.

---
 block.c                  |    2 -
 exec.c                   |    1 
 replay/replay-internal.c |    1 
 replay/replay-internal.h |    4 +
 replay/replay.c          |  140 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   13 ++++
 stubs/replay.c           |    1 
 vl.c                     |    5 ++
 8 files changed, 165 insertions(+), 2 deletions(-)
block.c +1 -1
index 7f6fa8b..9923a80 100644
@@ -2010,7 +2010,7 @@ void bdrv_drain_all(void)
             aio_context_release(aio_context);
         }
     }
-    if (replay_mode == REPLAY_MODE_PLAY) {
+    if (replay_mode == REPLAY_MODE_PLAY && replay_checkpoints) {
         /* Skip checkpoints from the log */
         while (replay_checkpoint(CHECKPOINT_BDRV_DRAIN)) {
             /* Nothing */
exec.c +1 -0
index 410371d..f70f03d 100644
@@ -783,6 +783,7 @@ void cpu_abort(CPUState *cpu, const char *fmt, ...)
     }
     va_end(ap2);
     va_end(ap);
+    replay_finish();
 #if defined(CONFIG_USER_ONLY)
     {
         struct sigaction act;
replay/replay-internal.c +0 -0
index 49b37a6..d608b71 100755

replay/replay-internal.c void replay_put_byte(uint8_t byte) · line 36

Quoted 4 lines
replay/replay-internal.c +1 -0
@@ -33,6 +33,7 @@ void replay_put_byte(uint8_t byte)
 
 void replay_put_event(uint8_t event)
 {
+    assert(event <= EVENT_END);

Please move (way) earlier in the series.

Quoted 8 lines
replay/replay-internal.c +0 -0
     replay_put_byte(event);
 }
 
replay/replay-internal.h +0 -0
index 4d242aa..1e5d037 100755

replay/replay-internal.h enum ReplayEvents { · line 39

Quoted 7 lines
replay/replay-internal.h +3 -1
@@ -34,7 +34,9 @@ enum ReplayEvents {
     EVENT_CLOCK,
     /* for checkpoint event */
     /* some of grteater codes are reserved for checkpoints */
-    EVENT_CHECKPOINT = EVENT_CLOCK + REPLAY_CLOCK_COUNT
+    EVENT_CHECKPOINT = EVENT_CLOCK + REPLAY_CLOCK_COUNT,
+    /* end of log event */
+    EVENT_END = EVENT_CHECKPOINT + CHECKPOINT_COUNT

Please add EVENT_END in the beginning of the series. If you do:

EVENT_CLOCK,
EVENT_CLOCK_LAST = EVENT_CLOCK + REPLAY_CLOCK_COUNT - 1, EVENT_CHECKPOINT,
EVENT_CHECKPOINT_LAST = EVENT_CHECKPOINT + CHECKPOINT_COUNT - 1, EVENT_END,

the patches look quite nice.

Quoted 8 lines
replay/replay-internal.h +0 -0
 };
 
 /* Asynchronous events IDs */
replay/replay.c +0 -0
index 7c4a801..fa738c2 100755

replay/replay.c line 20

Quoted 6 lines
replay/replay.c +3 -0
@@ -15,9 +15,18 @@
 #include "qemu/timer.h"
 #include "sysemu/sysemu.h"
 
+/* Current version of the replay mechanism.
+   Increase it when file format changes. */
+#define REPLAY_VERSION              0xe02002

Any meaning? Perhaps make it 'QRR\0' or something identifiable?

replay/replay.c line 29

Quoted 9 lines
replay/replay.c +6 -0
+/* Size of replay log header */
+#define HEADER_SIZE                 (sizeof(uint32_t) + sizeof(uint64_t))
+
 ReplayMode replay_mode = REPLAY_MODE_NONE;
 
+/* Name of replay file  */
+static char *replay_filename;
 ReplayState replay_state;
+bool replay_checkpoints;

Why is replay_checkpoints only defined and used in this patch? Is it a duplicate of replay_events_enabled?

Quoted 14 lines
replay/replay.c +4 -0
 bool skip_async_events(int stop_event)
 {
@@ -167,6 +176,10 @@ void replay_shutdown_request(void)
 bool replay_checkpoint(ReplayCheckpoint checkpoint)
 {
     bool res = false;
+    if (!replay_checkpoints) {
+        return true;
+    }
+
     replay_save_instructions();
 
     if (replay_file) {

replay/replay.c bool replay_checkpoint(ReplayCheckpoint checkpoint) · line 221

Quoted 10 lines
replay/replay.c +7 -0
@@ -199,3 +212,130 @@ bool replay_checkpoint(ReplayCheckpoint checkpoint)
 
     return true;
 }
+
+static void replay_enable(const char *fname, int mode)
+{
+    const char *fmode = NULL;
+    if (replay_file) {
+        fprintf(stderr,
+                "Replay: some record/replay operation is already started\n");

Would this have to be an assertion?

Quoted 26 lines
replay/replay.c +26 -0
+        return;
+    }
+
+    switch (mode) {
+    case REPLAY_MODE_RECORD:
+        fmode = "wb";
+        break;
+    case REPLAY_MODE_PLAY:
+        fmode = "rb";
+        break;
+    default:
+        fprintf(stderr, "Replay: internal error: invalid replay mode\n");
+        exit(1);
+    }
+
+    atexit(replay_finish);
+
+    replay_mutex_init();
+
+    replay_file = fopen(fname, fmode);
+    if (replay_file == NULL) {
+        fprintf(stderr, "Replay: open %s: %s\n", fname, strerror(errno));
+        exit(1);
+    }
+
+    replay_filename = g_strdup(fname);

replay/replay.c bool replay_checkpoint(ReplayCheckpoint checkpoint) · line 257

Quoted 10 lines
replay/replay.c +10 -0
+
+    replay_mode = mode;
+    replay_has_unread_data = 0;
+    replay_data_kind = -1;
+    replay_state.instructions_count = 0;
+    replay_state.current_step = 0;
+
+    /* skip file header for RECORD and check it for PLAY */
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        fseek(replay_file, HEADER_SIZE, SEEK_SET);

Why can't you write the header here?

Quoted 40 lines
replay/replay.c +40 -0
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        unsigned int version = replay_get_dword();
+        uint64_t offset = replay_get_qword();
+        if (version != REPLAY_VERSION) {
+            fprintf(stderr, "Replay: invalid input log file version\n");
+            exit(1);
+        }
+        /* go to the beginning */
+        fseek(replay_file, 12, SEEK_SET);
+    }
+
+    replay_init_events();
+}
+
+void replay_configure(QemuOpts *opts)
+{
+    const char *fname;
+    const char *rr;
+    ReplayMode mode = REPLAY_MODE_NONE;
+
+    rr = qemu_opt_get(opts, "rr");
+    if (!rr) {
+        /* Just enabling icount */
+        return;
+    } else if (!strcmp(rr, "record")) {
+        mode = REPLAY_MODE_RECORD;
+    } else if (!strcmp(rr, "replay")) {
+        mode = REPLAY_MODE_PLAY;
+    } else {
+        fprintf(stderr, "Invalid icount rr option: %s\n", rr);
+        exit(1);
+    }
+
+    fname = qemu_opt_get(opts, "rrfname");
+    if (!fname) {
+        fprintf(stderr, "File name not specified for replay\n");
+        exit(1);
+    }
+
+    replay_enable(fname, mode);

replay/replay.c bool replay_checkpoint(ReplayCheckpoint checkpoint) · line 307

Quoted 10 lines
replay/replay.c +10 -0
+}
+
+void replay_init_timer(void)
+{
+    if (replay_mode == REPLAY_MODE_NONE) {
+        return;
+    }
+
+    replay_checkpoints = true;
+    replay_enable_events();

Move if to replay_enable_events() and drop replay_init_timer() altogether?

Quoted 11 lines
replay/replay.c +11 -0
+}
+
+void replay_finish(void)
+{
+    if (replay_mode == REPLAY_MODE_NONE) {
+        return;
+    }
+
+    replay_save_instructions();
+
+    /* finalize the file */

replay/replay.c bool replay_checkpoint(ReplayCheckpoint checkpoint) · line 328

Quoted 10 lines
replay/replay.c +10 -0
+    if (replay_file) {
+        if (replay_mode == REPLAY_MODE_RECORD) {
+            uint64_t offset = 0;
+            /* write end event */
+            replay_put_event(EVENT_END);
+
+            /* write header */
+            fseek(replay_file, 0, SEEK_SET);
+            replay_put_dword(REPLAY_VERSION);
+            replay_put_qword(offset);

Why is this done here? You're writing a constant zero.

Quoted 18 lines
replay/replay.c +13 -0
+        }
+
+        fclose(replay_file);
+        replay_file = NULL;
+    }
+    if (replay_filename) {
+        g_free(replay_filename);
+        replay_filename = NULL;
+    }
+
+    replay_mutex_destroy();
+    replay_finish_events();
+}
replay/replay.h +0 -0
index 3110d46..231b8ec 100755

replay/replay.h line 20

Quoted 4 lines
replay/replay.h +1 -0
@@ -17,6 +17,8 @@
 #include <time.h>
 #include "qapi-types.h"
 
+struct QemuOpts;

#include instead?

Quoted 38 lines
replay/replay.h +11 -0
 /* replay clock kinds */
 enum ReplayClockKind {
     /* rdtsc */
@@ -45,6 +47,17 @@ enum ReplayCheckpoint {
 typedef enum ReplayCheckpoint ReplayCheckpoint;
 
 extern ReplayMode replay_mode;
+/*! Is set to true after finishing initialization */
+extern bool replay_checkpoints;
+
+/* Replay process control functions */
+
+/*! Enables recording or saving event log with specified parameters */
+void replay_configure(struct QemuOpts *opts);
+/*! Initializes timers used for snapshotting and enables events recording */
+void replay_init_timer(void);
+/*! Closes replay log file and frees other resources. */
+void replay_finish(void);
 
 /* Processing the instructions */
 
stubs/replay.c +1 -0
index 81eddae..7d87964 100755
@@ -2,6 +2,7 @@
 #include "sysemu/sysemu.h"
 
 ReplayMode replay_mode;
+bool replay_checkpoints;
 
 int64_t replay_save_clock(unsigned int kind, int64_t clock)
 {
vl.c +0 -0
index 86ba385..ae3e97e 100644

vl.c int main(int argc, char **argv, char **envp) · line 4379

Quoted 4 lines
vl.c +1 -0
@@ -4376,7 +4376,12 @@ int main(int argc, char **argv, char **envp)
         }
     }
 
+    replay_init_timer();

replay_enable_events()

vl.c int main(int argc, char **argv, char **envp) · line 4383

Quoted 3 lines
vl.c +2 -0
     main_loop();
+    if (replay_mode != REPLAY_MODE_NONE) {
+        replay_disable_events();

Move "if" to replay_disable_events.

Paolo

Quoted 5 lines
vl.c +1 -0
+    }
     bdrv_close_all();
     pause_all_vcpus();
     res_free();
#69 lore raw

Re: [Qemu-devel] [RFC PATCH v8 19/21] replay: initialization and deinitialization

Quoted 18 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 16 lines · Pavel Dovgalyuk

This patch introduces the functions for enabling the record/replay and for freeing the resources when simulator closes.

};

/* Asynchronous events IDs */

replay/replay.c +0 -0
index 7c4a801..fa738c2 100755
@@ -15,9 +15,18 @@
 #include "qemu/timer.h"
 #include "sysemu/sysemu.h"
 
Quoted 5 lines

replay/replay.c line 20

Quoted 3 lines
replay/replay.c +3 -0
+/* Current version of the replay mechanism.
+   Increase it when file format changes. */
+#define REPLAY_VERSION              0xe02002

Any meaning? Perhaps make it 'QRR\0' or something identifiable?

Version id has no special meaning, but has to be distinct from other RR versions. When format of the log file changes we have to update version id to prevent reading incompatible files.

Quoted 11 lines
Quoted 11 lines
replay/replay.c +11 -0
+    replay_file = fopen(fname, fmode);
+    if (replay_file == NULL) {
+        fprintf(stderr, "Replay: open %s: %s\n", fname, strerror(errno));
+        exit(1);
+    }
+
+    replay_filename = g_strdup(fname);
+
+    replay_mode = mode;
+    replay_has_unread_data = 0;
+    replay_data_kind = -1;
Quoted 8 lines

replay/replay.c bool replay_checkpoint(ReplayCheckpoint checkpoint) · line 257

Quoted 6 lines
replay/replay.c +6 -0
+    replay_state.instructions_count = 0;
+    replay_state.current_step = 0;
+
+    /* skip file header for RECORD and check it for PLAY */
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        fseek(replay_file, HEADER_SIZE, SEEK_SET);

Why can't you write the header here?

We don't write header here to detect broken log files at the replay stage.

Quoted 7 lines
Quoted 7 lines
replay/replay.c +7 -0
+    replay_save_instructions();
+
+    /* finalize the file */
+    if (replay_file) {
+        if (replay_mode == REPLAY_MODE_RECORD) {
+            uint64_t offset = 0;
+            /* write end event */
Quoted 8 lines

replay/replay.c bool replay_checkpoint(ReplayCheckpoint checkpoint) · line 328

Quoted 6 lines
replay/replay.c +6 -0
+            replay_put_event(EVENT_END);
+
+            /* write header */
+            fseek(replay_file, 0, SEEK_SET);
+            replay_put_dword(REPLAY_VERSION);
+            replay_put_qword(offset);

Why is this done here? You're writing a constant zero.

Right, this offset is the field reserved for snapshots table. It will be introduced in other patches.

Pavel Dovgalyuk

#70 lore raw

Re: [Qemu-devel] [RFC PATCH v8 19/21] replay: initialization and deinitialization

Quoted 48 lines · Pavel Dovgaluk
Quoted 18 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 16 lines · Pavel Dovgalyuk

This patch introduces the functions for enabling the record/replay and for freeing the resources when simulator closes.

};

/* Asynchronous events IDs */

replay/replay.c +0 -0
index 7c4a801..fa738c2 100755
@@ -15,9 +15,18 @@
 #include "qemu/timer.h"
 #include "sysemu/sysemu.h"
 
Quoted 5 lines

replay/replay.c line 20

Quoted 3 lines
replay/replay.c +3 -0
+/* Current version of the replay mechanism.
+   Increase it when file format changes. */
+#define REPLAY_VERSION              0xe02002

Any meaning? Perhaps make it 'QRR\0' or something identifiable?

Version id has no special meaning, but has to be distinct from other RR versions. When format of the log file changes we have to update version id to prevent reading incompatible files.

Quoted 11 lines
Quoted 11 lines
replay/replay.c +11 -0
+    replay_file = fopen(fname, fmode);
+    if (replay_file == NULL) {
+        fprintf(stderr, "Replay: open %s: %s\n", fname, strerror(errno));
+        exit(1);
+    }
+
+    replay_filename = g_strdup(fname);
+
+    replay_mode = mode;
+    replay_has_unread_data = 0;
+    replay_data_kind = -1;
Quoted 8 lines

replay/replay.c bool replay_checkpoint(ReplayCheckpoint checkpoint) · line 257

Quoted 6 lines
replay/replay.c +6 -0
+    replay_state.instructions_count = 0;
+    replay_state.current_step = 0;
+
+    /* skip file header for RECORD and check it for PLAY */
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        fseek(replay_file, HEADER_SIZE, SEEK_SET);

Why can't you write the header here?

Quoted 1 line

We don't write header here to detect broken log files at the replay stage.

I guess this is also a provision for something that you'll do in the future.

Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
replay/replay.c +7 -0
+    replay_save_instructions();
+
+    /* finalize the file */
+    if (replay_file) {
+        if (replay_mode == REPLAY_MODE_RECORD) {
+            uint64_t offset = 0;
+            /* write end event */
Quoted 8 lines

replay/replay.c bool replay_checkpoint(ReplayCheckpoint checkpoint) · line 328

Quoted 6 lines
replay/replay.c +6 -0
+            replay_put_event(EVENT_END);
+
+            /* write header */
+            fseek(replay_file, 0, SEEK_SET);
+            replay_put_dword(REPLAY_VERSION);
+            replay_put_qword(offset);

Why is this done here? You're writing a constant zero.

Quoted 2 lines

Right, this offset is the field reserved for snapshots table. It will be introduced in other patches.

Okay, then please add comments.

Paolo

#71 lore raw

RFC PATCH v8 20/21 replay: command line options

In reply to Pavel Dovgalyuk

This patch introduces command line options for enabling recording or replaying virtual machine behavior. "-record" option starts recording of the execution and saves it into the log, specified with "fname" parameter. "-replay" option is intended for replaying previously saved log.

---
 cpus.c          |    3 ++-
 qemu-options.hx |    8 ++++++--
 vl.c            |   27 ++++++++++++++++++++++++++-
 3 files changed, 34 insertions(+), 4 deletions(-)
cpus.c +2 -1
index 9c32491..7689cec 100644
@@ -912,9 +912,10 @@ static void qemu_wait_io_event_common(CPUState *cpu)
 static void qemu_tcg_wait_io_event(void)
 {
     CPUState *cpu;
+    GMainContext *context = g_main_context_default();
 
     while (all_cpu_threads_idle()) {
-       /* Start accounting real time to the virtual clock if the CPUs
+        /* Start accounting real time to the virtual clock if the CPUs
           are idle.  */
         qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
         qemu_cond_wait(tcg_halt_cond, &qemu_global_mutex);
qemu-options.hx +6 -2
index 10b9568..cb4b577 100644
@@ -3037,11 +3037,11 @@ re-inject them.
 ETEXI
 
 DEF("icount", HAS_ARG, QEMU_OPTION_icount, \
-    "-icount [shift=N|auto][,align=on|off]\n" \
+    "-icount [shift=N|auto][,align=on|off][,rr=record|replay,rrfname=<filename>]\n" \
     "                enable virtual instruction counter with 2^N clock ticks per\n" \
     "                instruction and enable aligning the host and virtual clocks\n", QEMU_ARCH_ALL)
 STEXI
-@item -icount [shift=@var{N}|auto]
+@item -icount [shift=@var{N}|auto][,rr=record|replay,rrfname=@var{filename}]
 @findex -icount
 Enable virtual instruction counter.  The virtual cpu will execute one
 instruction every 2^@var{N} ns of virtual time.  If @code{auto} is specified
@@ -3063,6 +3063,10 @@ Currently this option does not work when @option{shift} is @code{auto}.
 Note: The sync algorithm will work for those shift values for which
 the guest clock runs ahead of the host clock. Typically this happens
 when the shift value is high (how high depends on the host machine).
+
+When @option{rr} option is specified deterministic record/replay is enabled.
+Replay log is written into @var{filename} file in record mode and
+read from this file in replay mode.
 ETEXI
 
 DEF("watchdog", HAS_ARG, QEMU_OPTION_watchdog, \
vl.c +26 -1
index ae3e97e..39d9024 100644
@@ -475,6 +475,12 @@ static QemuOptsList qemu_icount_opts = {
         }, {
             .name = "align",
             .type = QEMU_OPT_BOOL,
+        }, {
+            .name = "rr",
+            .type = QEMU_OPT_STRING,
+        }, {
+            .name = "rrfname",
+            .type = QEMU_OPT_STRING,
         },
         { /* end of list */ }
     },
@@ -2752,6 +2758,8 @@ int main(int argc, char **argv, char **envp)
 {
     int i;
     int snapshot, linux_boot;
+    int not_compatible_replay_param = 0;
+    const char *icount_option = NULL;
     const char *initrd_filename;
     const char *kernel_filename, *kernel_cmdline;
     const char *boot_order;
@@ -2949,6 +2957,7 @@ int main(int argc, char **argv, char **envp)
                 break;
             case QEMU_OPTION_pflash:
                 drive_add(IF_PFLASH, -1, optarg, PFLASH_OPTS);
+                not_compatible_replay_param++;
                 break;
             case QEMU_OPTION_snapshot:
                 snapshot = 1;
@@ -3105,6 +3114,7 @@ int main(int argc, char **argv, char **envp)
 #endif
             case QEMU_OPTION_bt:
                 add_device_config(DEV_BT, optarg);
+                not_compatible_replay_param++;
                 break;
             case QEMU_OPTION_audio_help:
                 AUD_help ();
@@ -3244,6 +3254,7 @@ int main(int argc, char **argv, char **envp)
                 if (!opts) {
                     exit(1);
                 }
+                not_compatible_replay_param++;
                 break;
             case QEMU_OPTION_fsdev:
                 olist = qemu_find_opts("fsdev");
@@ -3372,6 +3383,7 @@ int main(int argc, char **argv, char **envp)
                 if (strncmp(optarg, "mon:", 4) == 0) {
                     default_monitor = 0;
                 }
+                not_compatible_replay_param++;
                 break;
             case QEMU_OPTION_debugcon:
                 add_device_config(DEV_DEBUGCON, optarg);
@@ -3489,6 +3501,7 @@ int main(int argc, char **argv, char **envp)
                 if (!qemu_opts_parse(qemu_find_opts("smp-opts"), optarg, 1)) {
                     exit(1);
                 }
+                not_compatible_replay_param++;
                 break;
             case QEMU_OPTION_vnc:
 #ifdef CONFIG_VNC
@@ -3628,6 +3641,7 @@ int main(int argc, char **argv, char **envp)
                 if (!icount_opts) {
                     exit(1);
                 }
+                replay_configure(icount_opts);
                 break;
             case QEMU_OPTION_incoming:
                 incoming = optarg;
@@ -3786,6 +3800,12 @@ int main(int argc, char **argv, char **envp)
         exit(1);
     }
 
+    if (not_compatible_replay_param && (replay_mode != REPLAY_MODE_NONE)) {
+        fprintf(stderr, "options -smp, -pflash, -chardev, -bt, -parallel "
+                        "are not compatible with record/replay\n");
+        exit(1);
+    }
+
     if (qemu_opts_foreach(qemu_find_opts("sandbox"), parse_sandbox, NULL, 0)) {
         exit(1);
     }
@@ -4136,6 +4156,11 @@ int main(int argc, char **argv, char **envp)
         qemu_opts_del(icount_opts);
     }
 
+    if (replay_mode != REPLAY_MODE_NONE && !use_icount) {
+        fprintf(stderr, "Please enable icount to use record/replay\n");
+        exit(1);
+    }
+
     /* clean up network at qemu process termination */
     atexit(&net_cleanup);
 
@@ -4172,7 +4197,7 @@ int main(int argc, char **argv, char **envp)
     }
 
     /* open the virtual block devices */
-    if (snapshot)
+    if (snapshot || replay_mode != REPLAY_MODE_NONE)
         qemu_opts_foreach(qemu_find_opts("drive"), drive_enable_snapshot, NULL, 0);
     if (qemu_opts_foreach(qemu_find_opts("drive"), drive_init_func,
                           &machine_class->block_default_type, 1) != 0) {
#72 lore raw

Re: [Qemu-devel] [RFC PATCH v8 20/21] replay: command line options

Quoted 18 lines · Pavel Dovgalyuk

This patch introduces command line options for enabling recording or replaying virtual machine behavior. "-record" option starts recording of the execution and saves it into the log, specified with "fname" parameter. "-replay" option is intended for replaying previously saved log.

---
 cpus.c          |    3 ++-
 qemu-options.hx |    8 ++++++--
 vl.c            |   27 ++++++++++++++++++++++++++-
 3 files changed, 34 insertions(+), 4 deletions(-)
cpus.c +0 -0
index 9c32491..7689cec 100644
@@ -912,9 +912,10 @@ static void qemu_wait_io_event_common(CPUState *cpu)
 static void qemu_tcg_wait_io_event(void)

cpus.c static void qemu_wait_io_event_common(CPUState *cpu) · line 921

Quoted 10 lines
cpus.c +2 -1
 {
     CPUState *cpu;
+    GMainContext *context = g_main_context_default();
 
     while (all_cpu_threads_idle()) {
-       /* Start accounting real time to the virtual clock if the CPUs
+        /* Start accounting real time to the virtual clock if the CPUs
           are idle.  */
         qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
         qemu_cond_wait(tcg_halt_cond, &qemu_global_mutex);

Extraneous change.

qemu-options.hx re-inject them. · line 3040

Quoted 10 lines
qemu-options.hx +1 -1
index 10b9568..cb4b577 100644
@@ -3037,11 +3037,11 @@ re-inject them.
 ETEXI
 
 DEF("icount", HAS_ARG, QEMU_OPTION_icount, \
-    "-icount [shift=N|auto][,align=on|off]\n" \
+    "-icount [shift=N|auto][,align=on|off][,rr=record|replay,rrfname=<filename>]\n" \

rrfile?

Quoted 46 lines
qemu-options.hx +5 -1
     "                enable virtual instruction counter with 2^N clock ticks per\n" \
     "                instruction and enable aligning the host and virtual clocks\n", QEMU_ARCH_ALL)
 STEXI
-@item -icount [shift=@var{N}|auto]
+@item -icount [shift=@var{N}|auto][,rr=record|replay,rrfname=@var{filename}]
 @findex -icount
 Enable virtual instruction counter.  The virtual cpu will execute one
 instruction every 2^@var{N} ns of virtual time.  If @code{auto} is specified
@@ -3063,6 +3063,10 @@ Currently this option does not work when @option{shift} is @code{auto}.
 Note: The sync algorithm will work for those shift values for which
 the guest clock runs ahead of the host clock. Typically this happens
 when the shift value is high (how high depends on the host machine).
+
+When @option{rr} option is specified deterministic record/replay is enabled.
+Replay log is written into @var{filename} file in record mode and
+read from this file in replay mode.
 ETEXI
 
 DEF("watchdog", HAS_ARG, QEMU_OPTION_watchdog, \
vl.c +8 -0
index ae3e97e..39d9024 100644
@@ -475,6 +475,12 @@ static QemuOptsList qemu_icount_opts = {
         }, {
             .name = "align",
             .type = QEMU_OPT_BOOL,
+        }, {
+            .name = "rr",
+            .type = QEMU_OPT_STRING,
+        }, {
+            .name = "rrfname",
+            .type = QEMU_OPT_STRING,
         },
         { /* end of list */ }
     },
@@ -2752,6 +2758,8 @@ int main(int argc, char **argv, char **envp)
 {
     int i;
     int snapshot, linux_boot;
+    int not_compatible_replay_param = 0;
+    const char *icount_option = NULL;
     const char *initrd_filename;
     const char *kernel_filename, *kernel_cmdline;
     const char *boot_order;

vl.c int main(int argc, char **argv, char **envp) · line 2960

Quoted 4 lines
vl.c +1 -0
@@ -2949,6 +2957,7 @@ int main(int argc, char **argv, char **envp)
                 break;
             case QEMU_OPTION_pflash:
                 drive_add(IF_PFLASH, -1, optarg, PFLASH_OPTS);
+                not_compatible_replay_param++;

Why not compatible?

vl.c int main(int argc, char **argv, char **envp) · line 3117

Quoted 8 lines
vl.c +1 -0
                 break;
             case QEMU_OPTION_snapshot:
                 snapshot = 1;
@@ -3105,6 +3114,7 @@ int main(int argc, char **argv, char **envp)
 #endif
             case QEMU_OPTION_bt:
                 add_device_config(DEV_BT, optarg);
+                not_compatible_replay_param++;

Could it be enough to add a migration blocker?

vl.c int main(int argc, char **argv, char **envp) · line 3257

Quoted 8 lines
vl.c +1 -0
                 break;
             case QEMU_OPTION_audio_help:
                 AUD_help ();
@@ -3244,6 +3254,7 @@ int main(int argc, char **argv, char **envp)
                 if (!opts) {
                     exit(1);
                 }
+                not_compatible_replay_param++;

Why not compatible?

vl.c int main(int argc, char **argv, char **envp) · line 3386

Quoted 8 lines
vl.c +1 -0
                 break;
             case QEMU_OPTION_fsdev:
                 olist = qemu_find_opts("fsdev");
@@ -3372,6 +3383,7 @@ int main(int argc, char **argv, char **envp)
                 if (strncmp(optarg, "mon:", 4) == 0) {
                     default_monitor = 0;
                 }
+                not_compatible_replay_param++;

Add a migration blocker?

vl.c int main(int argc, char **argv, char **envp) · line 3504

Quoted 8 lines
vl.c +1 -0
                 break;
             case QEMU_OPTION_debugcon:
                 add_device_config(DEV_DEBUGCON, optarg);
@@ -3489,6 +3501,7 @@ int main(int argc, char **argv, char **envp)
                 if (!qemu_opts_parse(qemu_find_opts("smp-opts"), optarg, 1)) {
                     exit(1);
                 }
+                not_compatible_replay_param++;

Check smp_cpus instead in replay_configure?

vl.c int main(int argc, char **argv, char **envp) · line 3644

Quoted 8 lines
vl.c +1 -0
                 break;
             case QEMU_OPTION_vnc:
 #ifdef CONFIG_VNC
@@ -3628,6 +3641,7 @@ int main(int argc, char **argv, char **envp)
                 if (!icount_opts) {
                     exit(1);
                 }
+                replay_configure(icount_opts);

Perhaps move this outside the switch so that you have a fuller view of the command line?

vl.c int main(int argc, char **argv, char **envp) · line 3807

Quoted 12 lines
vl.c +5 -0
                 break;
             case QEMU_OPTION_incoming:
                 incoming = optarg;
@@ -3786,6 +3800,12 @@ int main(int argc, char **argv, char **envp)
         exit(1);
     }
 
+    if (not_compatible_replay_param && (replay_mode != REPLAY_MODE_NONE)) {
+        fprintf(stderr, "options -smp, -pflash, -chardev, -bt, -parallel "
+                        "are not compatible with record/replay\n");
+        exit(1);
+    }

... so this (or some replacement) can be done in replay_configure.

vl.c int main(int argc, char **argv, char **envp) · line 4162

Quoted 11 lines
vl.c +4 -0
     if (qemu_opts_foreach(qemu_find_opts("sandbox"), parse_sandbox, NULL, 0)) {
         exit(1);
     }
@@ -4136,6 +4156,11 @@ int main(int argc, char **argv, char **envp)
         qemu_opts_del(icount_opts);
     }
 
+    if (replay_mode != REPLAY_MODE_NONE && !use_icount) {
+        fprintf(stderr, "Please enable icount to use record/replay\n");
+        exit(1);
+    }

Same here.

Paolo

Quoted 13 lines
vl.c +1 -1
     /* clean up network at qemu process termination */
     atexit(&net_cleanup);
 
@@ -4172,7 +4197,7 @@ int main(int argc, char **argv, char **envp)
     }
 
     /* open the virtual block devices */
-    if (snapshot)
+    if (snapshot || replay_mode != REPLAY_MODE_NONE)
         qemu_opts_foreach(qemu_find_opts("drive"), drive_enable_snapshot, NULL, 0);
     if (qemu_opts_foreach(qemu_find_opts("drive"), drive_init_func,
                           &machine_class->block_default_type, 1) != 0) {
#73 lore raw

Re: [Qemu-devel] [RFC PATCH v8 20/21] replay: command line options

Quoted 7 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 5 lines · Pavel Dovgalyuk

This patch introduces command line options for enabling recording or replaying virtual machine behavior. "-record" option starts recording of the execution and saves it into the log, specified with "fname" parameter. "-replay" option is intended for replaying previously saved log.

Quoted 11 lines

vl.c int main(int argc, char **argv, char **envp) · line 3117

Quoted 9 lines
                 break;
             case QEMU_OPTION_snapshot:
                 snapshot = 1;
vl.c +1 -0
@@ -3105,6 +3114,7 @@ int main(int argc, char **argv, char **envp)
 #endif
             case QEMU_OPTION_bt:
                 add_device_config(DEV_BT, optarg);
+                not_compatible_replay_param++;

Could it be enough to add a migration blocker?

Record/replay core does not use migration subsystem. That is why it should check the hardware by itself.

Quoted 11 lines

vl.c int main(int argc, char **argv, char **envp) · line 3257

Quoted 9 lines
vl.c +1 -0
                 break;
             case QEMU_OPTION_audio_help:
                 AUD_help ();
@@ -3244,6 +3254,7 @@ int main(int argc, char **argv, char **envp)
                 if (!opts) {
                     exit(1);
                 }
+                not_compatible_replay_param++;

Why not compatible?

Replay for audio adapter will be added in latter patches. Trying to record/replay machine with audio using current set of patches will break the replay.

Pavel Dovgalyuk

#74 lore raw

Re: [Qemu-devel] [RFC PATCH v8 20/21] replay: command line options

Quoted 19 lines · Pavel Dovgaluk
Quoted 7 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 5 lines · Pavel Dovgalyuk

This patch introduces command line options for enabling recording or replaying virtual machine behavior. "-record" option starts recording of the execution and saves it into the log, specified with "fname" parameter. "-replay" option is intended for replaying previously saved log.

Quoted 11 lines

vl.c int main(int argc, char **argv, char **envp) · line 3117

Quoted 9 lines
                 break;
             case QEMU_OPTION_snapshot:
                 snapshot = 1;
vl.c +1 -0
@@ -3105,6 +3114,7 @@ int main(int argc, char **argv, char **envp)
 #endif
             case QEMU_OPTION_bt:
                 add_device_config(DEV_BT, optarg);
+                not_compatible_replay_param++;

Could it be enough to add a migration blocker?

Quoted 2 lines

Record/replay core does not use migration subsystem. That is why it should check the hardware by itself.

Right. But is there anything that supports migration but not replay? If the two overlap, you can just check migration blockers to see if record/replay can be allowed.

Quoted 12 lines
Quoted 11 lines

vl.c int main(int argc, char **argv, char **envp) · line 3257

Quoted 9 lines
vl.c +1 -0
                 break;
             case QEMU_OPTION_audio_help:
                 AUD_help ();
@@ -3244,6 +3254,7 @@ int main(int argc, char **argv, char **envp)
                 if (!opts) {
                     exit(1);
                 }
+                not_compatible_replay_param++;

Why not compatible?

Quoted 3 lines

Replay for audio adapter will be added in latter patches. Trying to record/replay machine with audio using current set of patches will break the replay.

For this case you can try adding a mechanism similar to migration blockers (replay blockers).

Paolo

#75 lore raw

Re: [Qemu-devel] [RFC PATCH v8 20/21] replay: command line options

Quoted 14 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 12 lines · Pavel Dovgaluk
Quoted 11 lines

From: Paolo Bonzini [mailto:[email protected]]

vl.c int main(int argc, char **argv, char **envp) · line 3257

Quoted 8 lines
                 break;
             case QEMU_OPTION_audio_help:
                 AUD_help ();
vl.c +1 -0
@@ -3244,6 +3254,7 @@ int main(int argc, char **argv, char **envp)
                 if (!opts) {
                     exit(1);
                 }
+                not_compatible_replay_param++;

Why not compatible?

Quoted 6 lines
Quoted 3 lines

Replay for audio adapter will be added in latter patches. Trying to record/replay machine with audio using current set of patches will break the replay.

For this case you can try adding a mechanism similar to migration blockers (replay blockers).

Do you mean adding flag to vmsd similar to unmigratable?

Pavel Dovgalyuk

#76 lore raw

Re: [Qemu-devel] [RFC PATCH v8 20/21] replay: command line options

Quoted 8 lines · Pavel Dovgaluk
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

Replay for audio adapter will be added in latter patches. Trying to record/replay machine with audio using current set of patches will break the replay.

For this case you can try adding a mechanism similar to migration blockers (replay blockers).

Do you mean adding flag to vmsd similar to unmigratable?

No, see this in migration/migration.c:

static GSList *migration_blockers;

void migrate_add_blocker(Error *reason)

{
    migration_blockers = g_slist_prepend(migration_blockers, reason);
}

void migrate_del_blocker(Error *reason)

{
    migration_blockers = g_slist_remove(migration_blockers, reason);
}

...

    if (migration_blockers) {
        *errp = error_copy(migration_blockers->data);
        return;
    }

Paolo

#77 lore raw

RFC PATCH v8 21/21 replay: recording of the user input

In reply to Pavel Dovgalyuk

This records user input (keyboard and mouse events) in record mode and replays these input events in replay mode.

---
 include/ui/input.h       |    2 +
 replay/Makefile.objs     |    1 
 replay/replay-events.c   |   55 +++++++++++++++++++
 replay/replay-input.c    |  130 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   13 +++++
 replay/replay.h          |    5 ++
 ui/input.c               |   68 +++++++++++++++++-------
 7 files changed, 255 insertions(+), 19 deletions(-)
 create mode 100755 replay/replay-input.c
include/ui/input.h +2 -0
index 5d5ac00..d06a12d 100644
@@ -33,7 +33,9 @@ void qemu_input_handler_bind(QemuInputHandlerState *s,
                              const char *device_id, int head,
                              Error **errp);
 void qemu_input_event_send(QemuConsole *src, InputEvent *evt);
+void qemu_input_event_send_impl(QemuConsole *src, InputEvent *evt);
 void qemu_input_event_sync(void);
+void qemu_input_event_sync_impl(void);
 
 InputEvent *qemu_input_event_new_key(KeyValue *key, bool down);
 void qemu_input_event_send_key(QemuConsole *src, KeyValue *key, bool down);
replay/Makefile.objs +1 -0
index 257c320..3936296 100755
@@ -2,3 +2,4 @@ obj-y += replay.o
 obj-y += replay-internal.o
 obj-y += replay-events.o
 obj-y += replay-time.o
+obj-y += replay-input.o
replay/replay-events.c +55 -0

91 lines not on this page · raw message

replay/replay-input.c new file +130 -0

130 lines not on this page · raw message

replay/replay-internal.h +13 -0
index 1e5d037..a32957c 100755
@@ -44,6 +44,8 @@ enum ReplayEvents {
 enum ReplayAsyncEventKind {
     REPLAY_ASYNC_EVENT_BH,
     REPLAY_ASYNC_EVENT_THREAD,
+    REPLAY_ASYNC_EVENT_INPUT,
+    REPLAY_ASYNC_EVENT_INPUT_SYNC,
     REPLAY_ASYNC_COUNT
 };
 
@@ -133,4 +135,15 @@ void replay_read_events(int opt);
 /*! Adds specified async event to the queue */
 void replay_add_event(ReplayAsyncEventKind event_id, void *opaque);
 
+/* Input events */
+
+/*! Saves input event to the log */
+void replay_save_input_event(InputEvent *evt);
+/*! Reads input event from the log */
+InputEvent *replay_read_input_event(void);
+/*! Adds input event to the queue */
+void replay_add_input_event(struct InputEvent *event);
+/*! Adds input sync event to the queue */
+void replay_add_input_sync_event(void);
+
 #endif
replay/replay.h +5 -0
index 231b8ec..503500a 100755
@@ -16,6 +16,7 @@
 #include <stdint.h>
 #include <time.h>
 #include "qapi-types.h"
+#include "qemu/typedefs.h"
 
 struct QemuOpts;
 
@@ -123,5 +124,9 @@ void replay_disable_events(void);
 void replay_add_bh_event(void *bh, uint64_t id);
 /*! Adds thread event to the queue */
 void replay_add_thread_event(void *pool, void *req, uint64_t id);
+/*! Adds input event to the queue */
+void replay_input_event(QemuConsole *src, InputEvent *evt);
+/*! Adds input sync event to the queue */
+void replay_input_sync_event(void);
 
 #endif
ui/input.c +49 -19

130 lines not on this page · raw message

#78 lore raw

Re: [Qemu-devel] [RFC PATCH v8 21/21] replay: recording of the user input

replay/replay-input.c line 115

Quoted 6 lines · Pavel Dovgalyuk
replay/replay-input.c new file +6 -0
+void replay_input_event(QemuConsole *src, InputEvent *evt)
+{
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        /* Nothing */
+    } else if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_add_input_event(evt);

Does replay_add_input_event ultimately call qemu_input_event_send_impl?

replay/replay-input.c line 120

Quoted 5 lines
replay/replay-input.c new file +5 -0
+    } else {
+        qemu_input_event_send_impl(src, evt);
+    }
+}
+

Perhaps make this and replay_input_sync_event return a bool and in the caller do:

    if (replay_input_event(src, evt)) {
        qemu_input_event_send_impl(src, evt):
    }

ui/input.c InputEvent *qemu_input_event_new_key(KeyValue *key, bool down) · line 377

Quoted 8 lines
ui/input.c +8 -0
+    if (replay_mode != REPLAY_MODE_PLAY) {
+        evt = qemu_input_event_new_key(key, down);
+        if (QTAILQ_EMPTY(&kbd_queue)) {
+            qemu_input_event_send(src, evt);
+            qemu_input_event_sync();
+            if (replay_mode != REPLAY_MODE_RECORD) {
+                qapi_free_InputEvent(evt);
+            }

This is wrong. You have different lifetimes for different modes. Please make a copy of the event in the implementation of record mode.

Also, you do not need the "if" for replay mode. The functions would just do nothing.

ui/input.c InputEvent *qemu_input_event_new_key(KeyValue *key, bool down) · line 382

Quoted 5 lines
ui/input.c +5 -0
+        } else {
+            if (replay_mode != REPLAY_MODE_NONE) {
+                fprintf(stderr, "Input queue is not supported "
+                                "in record/replay mode\n");
+                exit(1);

Why? For record mode should just work since qemu_input_event_send is called in qemu_input_queue_process.

Replay mode can just do nothing, by returning early from qemu_input_queue_event/qemu_input_queue_sync.

Paolo

Quoted 5 lines
ui/input.c +4 -0
+            }
+            qemu_input_queue_event(&kbd_queue, src, evt);
+            qemu_input_queue_sync(&kbd_queue);
+        }
     }
#79 lore raw

Re: [Qemu-devel] [RFC PATCH v8 21/21] replay: recording of the user input

Quoted 10 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

replay/replay-input.c line 115

Quoted 6 lines · Pavel Dovgalyuk
replay/replay-input.c new file +6 -0
+void replay_input_event(QemuConsole *src, InputEvent *evt)
+{
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        /* Nothing */
+    } else if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_add_input_event(evt);

Does replay_add_input_event ultimately call qemu_input_event_send_impl?

No, it just adds event to the queue.

Quoted 9 lines

replay/replay-input.c line 120

Quoted 5 lines
replay/replay-input.c new file +5 -0
+    } else {
+        qemu_input_event_send_impl(src, evt);
+    }
+}
+

Perhaps make this and replay_input_sync_event return a bool and in the caller do:

Quoted 3 lines
    if (replay_input_event(src, evt)) {
        qemu_input_event_send_impl(src, evt):
    }

No, we can't. qemu_input_event_send_impl is called when the queue is saved to the log.

Pavel Dovgalyuk

#80 lore raw

Re: [Qemu-devel] [RFC PATCH v8 21/21] replay: recording of the user input

In reply to Paolo Bonzini
Quoted 11 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

ui/input.c InputEvent *qemu_input_event_new_key(KeyValue *key, bool down) · line 377

Quoted 8 lines · Pavel Dovgalyuk
ui/input.c +8 -0
+    if (replay_mode != REPLAY_MODE_PLAY) {
+        evt = qemu_input_event_new_key(key, down);
+        if (QTAILQ_EMPTY(&kbd_queue)) {
+            qemu_input_event_send(src, evt);
+            qemu_input_event_sync();
+            if (replay_mode != REPLAY_MODE_RECORD) {
+                qapi_free_InputEvent(evt);
+            }
Quoted 2 lines

This is wrong. You have different lifetimes for different modes. Please make a copy of the event in the implementation of record mode.

What is the correct way for cloning the QAPI type?
I should invent the cloning visitor or just create a switch for correct cloning of the InputEvent union?

Pavel Dovgalyuk

#81 lore raw

Re: [Qemu-devel] [RFC PATCH v8 21/21] replay: recording of the user input

Quoted 5 lines · Pavel Dovgaluk
Quoted 3 lines
Quoted 3 lines

This is wrong. You have different lifetimes for different modes. Please make a copy of the event in the implementation of record mode.

What is the correct way for cloning the QAPI type?
I should invent the cloning visitor or just create a switch for correct cloning of the InputEvent union?

You can use the existing visitors to clone objects, see qapi_copy_SocketAddress in qemu-char.c.

Paolo

#82 lore raw
In reply to Pavel Dovgalyuk

Ping?

Pavel Dovgalyuk

Quoted 225 lines

-----Original Message-----
From: Pavel Dovgalyuk [mailto:[email protected]]
Sent: Thursday, January 22, 2015 11:52 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [RFC PATCH v8 00/21] Deterministic replay core

This set of patches is related to the reverse execution and deterministic replay of qemu execution This implementation of deterministic replay can be used for deterministic debugging of guest code through gdb remote interface.

These patches include only core function of the replay, excluding the support for replaying serial, audio, network, and USB devices' operations. Reverse debugging and monitor commands were also excluded to be submitted later as separate patches.

Execution recording writes non-deterministic events log, which can be later used for replaying the execution anywhere and for unlimited number of times. It also supports checkpointing for faster rewinding during reverse debugging. Execution replaying reads the log and replays all non-deterministic events including external input, hardware clocks, and interrupts.

Deterministic replay has the following features:
* Deterministically replays whole system execution and all contents of the memory, state of the hadrware devices, clocks, and screen of the VM.
* Writes execution log into the file for latter replaying for multiple times on different machines.
* Supports i386, x86_64, and ARM hardware platforms.
* Performs deterministic replay of all operations with keyboard and mouse input devices.
* Supports auto-checkpointing for convenient reverse debugging.

Usage of the record/replay:
* First, record the execution, by adding the following string to the command line:
'-record fname=replay.bin -icount 7 -net none'. Block devices' images are not actually changed in the recording mode, because all of the changes are written to the temporary overlay file.
* Then you can replay it for the multiple times by using another command line option: '-replay fname=replay.bin -icount 7 -net none'
* '-net none' option should also be specified if network replay patches are not applied.

Paper with short description of deterministic replay implementation:
http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html

Modifications of qemu include:
* wrappers for clock and time functions to save their return values in the log
* saving different asynchronous events (e.g. system shutdown) into the log
* synchronization of the bottom halves execution
* synchronization of the threads from thread pool
* recording/replaying user input (mouse and keyboard)
* adding internal events for cpu and io synchronization

v8 changes:
* Simplified processing of the shutdown event (as suggested by Paolo Bonzini)
* Replaced stack of bottom halves in AIO context with QSIMPLEQ (as suggested by Paolo
Bonzini)
* Moved replay_submode out of the series (as suggested by Paolo Bonzini)
* Moved suffix option out of the series
* Converted some of the defines into enums (as suggested by Paolo Bonzini)
* Encapsulated save_tm/read_tm calls into the single function (as suggested by Paolo Bonzini)
* Moved record/replay options to icount group (as suggested by Paolo Bonzini)
* Updated mutex protection for the events queue (as suggested by Paolo Bonzini)
* Added mutex to protect replay log file (as suggested by Paolo Bonzini)
* Minor cleanups

v7 changes:
* Removed patches that were applied to upstream.

v6 changes:
* Fixed replay stub return value (as suggested by Eric Blake)
* Fixed icount warping.
* Virtual rt clock now uses cpu_get_clock() (as suggested by Paolo Bonzini)
* Replated get_clock_realtime and get_clock calls with qemu clock requests (as suggested by
Paolo Bonzini)
* Modified can_do_io logic to allow requesting icount from cpu_exec function (as suggested by
Paolo Bonzini)
* Removed applied patches.

v5 changes:
* Minor changes.
* Used fixed-width integer types for read/write functions (as suggested by Alex Bennee)
* Moved savevm-related code out of the core.
* Added new traced clock for deterministic virtual clock warping (as suggested by Paolo
Bonzini)
* Fixed exception_index reset for user mode (as suggested by Paolo Bonzini)
* Adopted Paolo's icount patches
* Fixed hardware interrupts replaying

v4 changes:
* Updated block drivers to support new bdrv_open interface.
* Moved migration patches into separate series (as suggested by Paolo Bonzini)
* Fixed a bug in replay_break operation.
* Fixed rtl8139 migration for replay.
* Fixed 'period' parameter processing for record mode.
* Fixed bug in 'reverse-stepi' implementation.
* Fixed replay without making any snapshots (even the starting one).
* Moved core replay patches into the separate series.
* Fixed reverse step and reverse continue support.
* Fixed several bugs in icount subsystem.
* Reusing native qemu icount for replay instructions counting.
* Separated core patches into their own series.

v3 changes:
* Fixed bug with replay of the aio write operations.
* Added virtual clock based on replay icount.
* Removed duplicated saving of interrupt_request CPU field.
* Fixed some coding style issues.
* Renamed QMP commands for controlling reverse execution (as suggested by Eric Blake)
* Replay mode and submode implemented as QAPI enumerations (as suggested by Eric Blake)
* Added description and example for replay-info command (as suggested by Eric Blake)
* Added information about the current breakpoint to the output of replay-info (as suggested
by Eric Blake)
* Updated version id for HPET vmstate (as suggested by Paolo Bonzini)
* Removed static fields from parallel vmstate (as suggested by Paolo Bonzini)
* New vmstate fields for mc146818rtc, pckbd, kvmapic, serial, fdc, rtl8139 moved to
subsection (as suggested by Paolo Bonzini)
* Disabled textmode cursor blinking, when virtual machine is stopped (as suggested by Paolo
Bonzini)
* Extracted saving of exception_index to separate patch (as suggested by Paolo Bonzini)

v2 changes:
* Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
* Added QMP versions of replay commands (as suggested by Eric Blake)
* Removed some optional features of replay to make patches cleaner
* Minor changes and code cleanup were made

---

Pavel Dovgalyuk (21):
i386: partial revert of interrupt poll fix
replay: global variables and function stubs
sysemu: system functions for replay
replay: internal functions for replay log
replay: introduce mutex to protect the replay log
replay: introduce icount event
cpu-exec: allow temporary disabling icount
cpu: replay instructions sequence
replay: interrupts and exceptions
replay: asynchronous events infrastructure
replay: recording and replaying clock ticks
replay: recording and replaying different timers
replay: shutdown event
replay: checkpoints
aio: replace stack of bottom halves with queue
replay: bottom halves
replay: replay aio requests
replay: thread pool
replay: initialization and deinitialization
replay: command line options
replay: recording of the user input

 Makefile.target                |    1
 async.c                        |   49 ++++--
 block.c                        |   92 ++++++++++-
 block/block-backend.c          |   30 +++-
 block/qcow2.c                  |    4
 block/raw-posix.c              |    6 -
 block/raw-win32.c              |    4
 cpu-exec.c                     |   60 ++++++-
 cpus.c                         |   45 ++++-
 dma-helpers.c                  |   10 +
 exec.c                         |    1
 hw/block/virtio-blk.c          |   10 +
 hw/ide/ahci.c                  |    4
 hw/ide/atapi.c                 |   10 +
 hw/ide/core.c                  |   18 +-
 hw/timer/arm_timer.c           |    2
 hw/timer/mc146818rtc.c         |    3
 hw/timer/pl031.c               |    3
 hw/usb/hcd-uhci.c              |    2
 include/block/aio.h            |   22 ++-
 include/block/block.h          |   15 ++
 include/block/thread-pool.h    |    4
 include/exec/exec-all.h        |    2
 include/qemu-common.h          |    3
 include/qemu/main-loop.h       |    1
 include/qemu/queue.h           |    7 +
 include/qemu/timer.h           |   16 ++
 include/qom/cpu.h              |   10 +
 include/sysemu/block-backend.h |   10 +
 include/sysemu/cpus.h          |    1
 include/ui/input.h             |    2
 main-loop.c                    |   10 +
 qapi-schema.json               |   18 ++
 qemu-io-cmds.c                 |    2
 qemu-options.hx                |    8 +
 qemu-timer.c                   |   56 +++++--
 replay/Makefile.objs           |    5 +
 replay/replay-events.c         |  310 ++++++++++++++++++++++++++++++++++++
 replay/replay-input.c          |  130 +++++++++++++++
 replay/replay-internal.c       |  183 +++++++++++++++++++++
 replay/replay-internal.h       |  149 +++++++++++++++++
 replay/replay-time.c           |  216 +++++++++++++++++++++++++
 replay/replay.c                |  341 ++++++++++++++++++++++++++++++++++++++++
 replay/replay.h                |  132 +++++++++++++++
 stubs/Makefile.objs            |    1
 stubs/replay.c                 |   37 ++++
 target-i386/cpu.c              |   10 -
 target-i386/seg_helper.c       |    4
 tests/test-thread-pool.c       |    7 -
 thread-pool.c                  |   49 ++++--
 trace-events                   |    2
 translate-all.c                |   10 +
 ui/input.c                     |   68 ++++++--
 util/iov.c                     |    4
 vl.c                           |   54 ++++++
 55 files changed, 2101 insertions(+), 152 deletions(-)
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay-events.c
 create mode 100755 replay/replay-input.c
 create mode 100755 replay/replay-internal.c
 create mode 100755 replay/replay-internal.h
 create mode 100755 replay/replay-time.c
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c

--
Pavel Dovgalyuk

A message that is not in the archive (sent off-list or not yet received): [email protected]

#84 lore raw
Quoted 1 line · Pavel Dovgaluk

Ping?

Reviewed 13 patches out of 21. Made some comments, but overall I'm really pleased. Thanks for persisting!

Will continue tomorrow.

Paolo

Quoted 230 lines

Pavel Dovgalyuk

Quoted 225 lines

-----Original Message-----
From: Pavel Dovgalyuk [mailto:[email protected]]
Sent: Thursday, January 22, 2015 11:52 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [RFC PATCH v8 00/21] Deterministic replay core

This set of patches is related to the reverse execution and deterministic replay of qemu execution This implementation of deterministic replay can be used for deterministic debugging of guest code through gdb remote interface.

These patches include only core function of the replay, excluding the support for replaying serial, audio, network, and USB devices' operations. Reverse debugging and monitor commands were also excluded to be submitted later as separate patches.

Execution recording writes non-deterministic events log, which can be later used for replaying the execution anywhere and for unlimited number of times. It also supports checkpointing for faster rewinding during reverse debugging. Execution replaying reads the log and replays all non-deterministic events including external input, hardware clocks, and interrupts.

Deterministic replay has the following features:
* Deterministically replays whole system execution and all contents of the memory, state of the hadrware devices, clocks, and screen of the VM.
* Writes execution log into the file for latter replaying for multiple times on different machines.
* Supports i386, x86_64, and ARM hardware platforms.
* Performs deterministic replay of all operations with keyboard and mouse input devices.
* Supports auto-checkpointing for convenient reverse debugging.

Usage of the record/replay:
* First, record the execution, by adding the following string to the command line:
'-record fname=replay.bin -icount 7 -net none'. Block devices' images are not actually changed in the recording mode, because all of the changes are written to the temporary overlay file.
* Then you can replay it for the multiple times by using another command line option: '-replay fname=replay.bin -icount 7 -net none'
* '-net none' option should also be specified if network replay patches are not applied.

Paper with short description of deterministic replay implementation:
http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html

Modifications of qemu include:
* wrappers for clock and time functions to save their return values in the log
* saving different asynchronous events (e.g. system shutdown) into the log
* synchronization of the bottom halves execution
* synchronization of the threads from thread pool
* recording/replaying user input (mouse and keyboard)
* adding internal events for cpu and io synchronization

v8 changes:
* Simplified processing of the shutdown event (as suggested by Paolo Bonzini)
* Replaced stack of bottom halves in AIO context with QSIMPLEQ (as suggested by Paolo
Bonzini)
* Moved replay_submode out of the series (as suggested by Paolo Bonzini)
* Moved suffix option out of the series
* Converted some of the defines into enums (as suggested by Paolo Bonzini)
* Encapsulated save_tm/read_tm calls into the single function (as suggested by Paolo Bonzini)
* Moved record/replay options to icount group (as suggested by Paolo Bonzini)
* Updated mutex protection for the events queue (as suggested by Paolo Bonzini)
* Added mutex to protect replay log file (as suggested by Paolo Bonzini)
* Minor cleanups

v7 changes:
* Removed patches that were applied to upstream.

v6 changes:
* Fixed replay stub return value (as suggested by Eric Blake)
* Fixed icount warping.
* Virtual rt clock now uses cpu_get_clock() (as suggested by Paolo Bonzini)
* Replated get_clock_realtime and get_clock calls with qemu clock requests (as suggested by
Paolo Bonzini)
* Modified can_do_io logic to allow requesting icount from cpu_exec function (as suggested by
Paolo Bonzini)
* Removed applied patches.

v5 changes:
* Minor changes.
* Used fixed-width integer types for read/write functions (as suggested by Alex Bennee)
* Moved savevm-related code out of the core.
* Added new traced clock for deterministic virtual clock warping (as suggested by Paolo
Bonzini)
* Fixed exception_index reset for user mode (as suggested by Paolo Bonzini)
* Adopted Paolo's icount patches
* Fixed hardware interrupts replaying

v4 changes:
* Updated block drivers to support new bdrv_open interface.
* Moved migration patches into separate series (as suggested by Paolo Bonzini)
* Fixed a bug in replay_break operation.
* Fixed rtl8139 migration for replay.
* Fixed 'period' parameter processing for record mode.
* Fixed bug in 'reverse-stepi' implementation.
* Fixed replay without making any snapshots (even the starting one).
* Moved core replay patches into the separate series.
* Fixed reverse step and reverse continue support.
* Fixed several bugs in icount subsystem.
* Reusing native qemu icount for replay instructions counting.
* Separated core patches into their own series.

v3 changes:
* Fixed bug with replay of the aio write operations.
* Added virtual clock based on replay icount.
* Removed duplicated saving of interrupt_request CPU field.
* Fixed some coding style issues.
* Renamed QMP commands for controlling reverse execution (as suggested by Eric Blake)
* Replay mode and submode implemented as QAPI enumerations (as suggested by Eric Blake)
* Added description and example for replay-info command (as suggested by Eric Blake)
* Added information about the current breakpoint to the output of replay-info (as suggested
by Eric Blake)
* Updated version id for HPET vmstate (as suggested by Paolo Bonzini)
* Removed static fields from parallel vmstate (as suggested by Paolo Bonzini)
* New vmstate fields for mc146818rtc, pckbd, kvmapic, serial, fdc, rtl8139 moved to
subsection (as suggested by Paolo Bonzini)
* Disabled textmode cursor blinking, when virtual machine is stopped (as suggested by Paolo
Bonzini)
* Extracted saving of exception_index to separate patch (as suggested by Paolo Bonzini)

v2 changes:
* Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
* Added QMP versions of replay commands (as suggested by Eric Blake)
* Removed some optional features of replay to make patches cleaner
* Minor changes and code cleanup were made

---

Pavel Dovgalyuk (21):
i386: partial revert of interrupt poll fix
replay: global variables and function stubs
sysemu: system functions for replay
replay: internal functions for replay log
replay: introduce mutex to protect the replay log
replay: introduce icount event
cpu-exec: allow temporary disabling icount
cpu: replay instructions sequence
replay: interrupts and exceptions
replay: asynchronous events infrastructure
replay: recording and replaying clock ticks
replay: recording and replaying different timers
replay: shutdown event
replay: checkpoints
aio: replace stack of bottom halves with queue
replay: bottom halves
replay: replay aio requests
replay: thread pool
replay: initialization and deinitialization
replay: command line options
replay: recording of the user input

 Makefile.target                |    1
 async.c                        |   49 ++++--
 block.c                        |   92 ++++++++++-
 block/block-backend.c          |   30 +++-
 block/qcow2.c                  |    4
 block/raw-posix.c              |    6 -
 block/raw-win32.c              |    4
 cpu-exec.c                     |   60 ++++++-
 cpus.c                         |   45 ++++-
 dma-helpers.c                  |   10 +
 exec.c                         |    1
 hw/block/virtio-blk.c          |   10 +
 hw/ide/ahci.c                  |    4
 hw/ide/atapi.c                 |   10 +
 hw/ide/core.c                  |   18 +-
 hw/timer/arm_timer.c           |    2
 hw/timer/mc146818rtc.c         |    3
 hw/timer/pl031.c               |    3
 hw/usb/hcd-uhci.c              |    2
 include/block/aio.h            |   22 ++-
 include/block/block.h          |   15 ++
 include/block/thread-pool.h    |    4
 include/exec/exec-all.h        |    2
 include/qemu-common.h          |    3
 include/qemu/main-loop.h       |    1
 include/qemu/queue.h           |    7 +
 include/qemu/timer.h           |   16 ++
 include/qom/cpu.h              |   10 +
 include/sysemu/block-backend.h |   10 +
 include/sysemu/cpus.h          |    1
 include/ui/input.h             |    2
 main-loop.c                    |   10 +
 qapi-schema.json               |   18 ++
 qemu-io-cmds.c                 |    2
 qemu-options.hx                |    8 +
 qemu-timer.c                   |   56 +++++--
 replay/Makefile.objs           |    5 +
 replay/replay-events.c         |  310 ++++++++++++++++++++++++++++++++++++
 replay/replay-input.c          |  130 +++++++++++++++
 replay/replay-internal.c       |  183 +++++++++++++++++++++
 replay/replay-internal.h       |  149 +++++++++++++++++
 replay/replay-time.c           |  216 +++++++++++++++++++++++++
 replay/replay.c                |  341 ++++++++++++++++++++++++++++++++++++++++
 replay/replay.h                |  132 +++++++++++++++
 stubs/Makefile.objs            |    1
 stubs/replay.c                 |   37 ++++
 target-i386/cpu.c              |   10 -
 target-i386/seg_helper.c       |    4
 tests/test-thread-pool.c       |    7 -
 thread-pool.c                  |   49 ++++--
 trace-events                   |    2
 translate-all.c                |   10 +
 ui/input.c                     |   68 ++++++--
 util/iov.c                     |    4
 vl.c                           |   54 ++++++
 55 files changed, 2101 insertions(+), 152 deletions(-)
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay-events.c
 create mode 100755 replay/replay-input.c
 create mode 100755 replay/replay-internal.c
 create mode 100755 replay/replay-internal.h
 create mode 100755 replay/replay-time.c
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c

--
Pavel Dovgalyuk

#85 lore raw
Quoted 9 lines · Paolo Bonzini
Quoted 1 line · Pavel Dovgaluk

Ping?

Reviewed 13 patches out of 21. Made some comments, but overall I'm really pleased. Thanks for persisting!

Will continue tomorrow.

The second part has a few more problems:

1) some simple and stylistic (patch 15, 19-21)

2) the invasive changes in patches 16-18 are, well, invasive. Lacking a design and example document it is hard to review them. The block maintainers, also, probably have better ideas of what to do.

3) A design document is also needed for checkpoints (patch 14)

Paolo

#86 lore raw
In reply to Pavel Dovgalyuk
Quoted 4 lines · Pavel Dovgalyuk

These patches include only core function of the replay, excluding the support for replaying serial, audio, network, and USB devices' operations. Reverse debugging and monitor commands were also excluded to be submitted later as separate patches.

BTW, I am interested in the serial part of the RR functionality, since it is probably simpler than block.

Paolo

#87 lore raw
Quoted 9 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

Quoted 4 lines · Pavel Dovgalyuk

These patches include only core function of the replay, excluding the support for replaying serial, audio, network, and USB devices' operations. Reverse debugging and monitor commands were also excluded to be submitted later as separate patches.

BTW, I am interested in the serial part of the RR functionality, since it is probably simpler than block.

Right, it's quite simple.
But replaying block layer gives us a usable version of record/replay. That is why serial devices replay is not included in the very first series.

Pavel Dovgalyuk