Skip to contentSkip to conversation

Deterministic replay core

RFC PATCH Superseded Pavel Dovgalyuk 3 participants 72 messages Jan 12, 2015 – Jan 22, 2015 4 messages not in the archive

Subsystems (2)

v7 of 18 Superseded Latest is v18 →

lore
Share & tools
lore
#0 lore raw

v7 cover Deterministic replay core

This set of patches is related to the reverse execution and deterministic replay of qemu execution This implementation of deterministic replay can be used for deterministic debugging of guest code through gdb remote interface.

These patches include only core function of the replay, excluding the support for replaying serial, audio, network, and USB devices' operations. Reverse debugging and monitor commands were also excluded to be submitted later as separate patches.

Execution recording writes non-deterministic events log, which can be later used for replaying the execution anywhere and for unlimited number of times. It also supports checkpointing for faster rewinding during reverse debugging. Execution replaying reads the log and replays all non-deterministic events including external input, hardware clocks, and interrupts.

Deterministic replay has the following features:
* Deterministically replays whole system execution and all contents of the memory, state of the hadrware devices, clocks, and screen of the VM.
* Writes execution log into the file for latter replaying for multiple times on different machines.
* Supports i386, x86_64, and ARM hardware platforms.
* Performs deterministic replay of all operations with keyboard and mouse input devices.
* Supports auto-checkpointing for convenient reverse debugging.

Usage of the record/replay:
* First, record the execution, by adding the following string to the command line:
'-record fname=replay.bin -icount 7 -net none'. Block devices' images are not actually changed in the recording mode, because all of the changes are written to the temporary overlay file.
* Then you can replay it for the multiple times by using another command line option: '-replay fname=replay.bin -icount 7 -net none'
* '-net none' option should also be specified if network replay patches are not applied.

Paper with short description of deterministic replay implementation:
http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html

Modifications of qemu include:
* wrappers for clock and time functions to save their return values in the log
* saving different asynchronous events (e.g. system shutdown) into the log
* synchronization of the bottom halves execution
* synchronization of the threads from thread pool
* recording/replaying user input (mouse and keyboard)
* adding internal events for cpu and io synchronization

v7 changes:
* Removed patches that were applied to upstream.

v6 changes:
* Fixed replay stub return value (as suggested by Eric Blake)
* Fixed icount warping.
* Virtual rt clock now uses cpu_get_clock() (as suggested by Paolo Bonzini)
* Replated get_clock_realtime and get_clock calls with qemu clock requests (as suggested by Paolo Bonzini)
* Modified can_do_io logic to allow requesting icount from cpu_exec function (as suggested by Paolo Bonzini)
* Removed applied patches.

v5 changes:
* Minor changes.
* Used fixed-width integer types for read/write functions (as suggested by Alex Bennee)
* Moved savevm-related code out of the core.
* Added new traced clock for deterministic virtual clock warping (as suggested by Paolo Bonzini)
* Fixed exception_index reset for user mode (as suggested by Paolo Bonzini)
* Adopted Paolo's icount patches
* Fixed hardware interrupts replaying

v4 changes:
* Updated block drivers to support new bdrv_open interface.
* Moved migration patches into separate series (as suggested by Paolo Bonzini)
* Fixed a bug in replay_break operation.
* Fixed rtl8139 migration for replay.
* Fixed 'period' parameter processing for record mode.
* Fixed bug in 'reverse-stepi' implementation.
* Fixed replay without making any snapshots (even the starting one).
* Moved core replay patches into the separate series.
* Fixed reverse step and reverse continue support.
* Fixed several bugs in icount subsystem.
* Reusing native qemu icount for replay instructions counting.
* Separated core patches into their own series.

v3 changes:
* Fixed bug with replay of the aio write operations.
* Added virtual clock based on replay icount.
* Removed duplicated saving of interrupt_request CPU field.
* Fixed some coding style issues.
* Renamed QMP commands for controlling reverse execution (as suggested by Eric Blake)
* Replay mode and submode implemented as QAPI enumerations (as suggested by Eric Blake)
* Added description and example for replay-info command (as suggested by Eric Blake)
* Added information about the current breakpoint to the output of replay-info (as suggested by Eric Blake)
* Updated version id for HPET vmstate (as suggested by Paolo Bonzini)
* Removed static fields from parallel vmstate (as suggested by Paolo Bonzini)
* New vmstate fields for mc146818rtc, pckbd, kvmapic, serial, fdc, rtl8139 moved to subsection (as suggested by Paolo Bonzini)
* Disabled textmode cursor blinking, when virtual machine is stopped (as suggested by Paolo Bonzini)
* Extracted saving of exception_index to separate patch (as suggested by Paolo Bonzini)

v2 changes:
* Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
* Added QMP versions of replay commands (as suggested by Eric Blake)
* Removed some optional features of replay to make patches cleaner
* Minor changes and code cleanup were made

---

Pavel Dovgalyuk (21):
i386: partial revert of interrupt poll fix
replay: global variables and function stubs
sysemu: system functions for replay
replay: internal functions for replay log
replay: introduce icount event
cpu-exec: allow temporary disabling icount
cpu: replay instructions sequence
replay: interrupts and exceptions
replay: asynchronous events infrastructure
cpus: make icount warp deterministic in replay mode
timer: fix usage of clock functions
replay: recording and replaying clock ticks
replay: recording and replaying different timers
replay: shutdown event
replay: checkpoints
replay: bottom halves
replay: replay aio requests
replay: thread pool
replay: initialization and deinitialization
replay: command line options
replay: recording of the user input

 Makefile.target                |    1 
 async.c                        |   46 +++++-
 block.c                        |   92 ++++++++++-
 block/block-backend.c          |   30 ++++
 block/qcow2.c                  |    4 
 block/raw-posix.c              |    6 -
 block/raw-win32.c              |    4 
 cpu-exec.c                     |   54 +++++--
 cpus.c                         |   47 ++++--
 dma-helpers.c                  |   10 +
 exec.c                         |    1 
 hw/block/virtio-blk.c          |   10 +
 hw/ide/ahci.c                  |    4 
 hw/ide/atapi.c                 |   10 +
 hw/ide/core.c                  |   18 +-
 hw/timer/arm_timer.c           |    2 
 hw/timer/mc146818rtc.c         |   10 +
 hw/timer/pl031.c               |   10 +
 hw/usb/hcd-uhci.c              |    2 
 include/block/aio.h            |   18 ++
 include/block/block.h          |   15 ++
 include/block/thread-pool.h    |    4 
 include/exec/exec-all.h        |    2 
 include/qemu-common.h          |    3 
 include/qemu/main-loop.h       |    1 
 include/qemu/timer.h           |   16 ++
 include/qom/cpu.h              |   10 +
 include/sysemu/block-backend.h |   10 +
 include/sysemu/cpus.h          |    1 
 include/sysemu/sysemu.h        |    1 
 include/ui/input.h             |    2 
 main-loop.c                    |   10 +
 pc-bios/s390-ccw/virtio.c      |    2 
 qapi-schema.json               |   32 ++++
 qemu-io-cmds.c                 |    2 
 qemu-options.hx                |   27 +++
 qemu-timer.c                   |   53 +++++--
 replay/Makefile.objs           |    5 +
 replay/replay-events.c         |  292 ++++++++++++++++++++++++++++++++++++
 replay/replay-input.c          |  108 +++++++++++++
 replay/replay-internal.c       |  155 +++++++++++++++++++
 replay/replay-internal.h       |  132 ++++++++++++++++
 replay/replay-time.c           |  191 ++++++++++++++++++++++++
 replay/replay.c                |  322 ++++++++++++++++++++++++++++++++++++++++
 replay/replay.h                |  119 +++++++++++++++
 stubs/Makefile.objs            |    1 
 stubs/replay.c                 |   42 +++++
 target-i386/cpu.c              |   10 -
 target-i386/seg_helper.c       |    4 
 tests/test-thread-pool.c       |    7 -
 thread-pool.c                  |   49 ++++--
 trace-events                   |    2 
 translate-all.c                |   10 +
 ui/input.c                     |   80 ++++++++--
 util/iov.c                     |    4 
 vl.c                           |  112 +++++++++++++-
 56 files changed, 2075 insertions(+), 140 deletions(-)
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay-events.c
 create mode 100755 replay/replay-input.c
 create mode 100755 replay/replay-internal.c
 create mode 100755 replay/replay-internal.h
 create mode 100755 replay/replay-time.c
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c
Signature · 1 line
Pavel Dovgalyuk
#1 lore raw

RFC PATCH v7 1/21 i386: partial revert of interrupt poll fix

Processing CPU_INTERRUPT_POLL requests in cpu_has_work functions break the determinism of cpu_exec. This patch is required to make interrupts processing deterministic.

---
 cpu-exec.c        |    6 ++++++
 target-i386/cpu.c |   10 ++--------
 2 files changed, 8 insertions(+), 8 deletions(-)
cpu-exec.c +6 -0
index a4f0eff..0a485bb 100644
@@ -338,6 +338,12 @@ int cpu_exec(CPUArchState *env)
     volatile bool have_tb_lock = false;
 
     if (cpu->halted) {
+#ifdef TARGET_I386
+        if (cpu->interrupt_request & CPU_INTERRUPT_POLL) {
+            apic_poll_irq(x86_cpu->apic_state);
+            cpu_reset_interrupt(cpu, CPU_INTERRUPT_POLL);
+        }
+#endif
         if (!cpu_has_work(cpu)) {
             return EXCP_HALTED;
         }
target-i386/cpu.c +2 -8
index b81ac5c..8f1f7aa 100644
@@ -2970,14 +2970,8 @@ static bool x86_cpu_has_work(CPUState *cs)
     X86CPU *cpu = X86_CPU(cs);
     CPUX86State *env = &cpu->env;
 
-#if !defined(CONFIG_USER_ONLY)
-    if (cs->interrupt_request & CPU_INTERRUPT_POLL) {
-        apic_poll_irq(cpu->apic_state);
-        cpu_reset_interrupt(cs, CPU_INTERRUPT_POLL);
-    }
-#endif
-
-    return ((cs->interrupt_request & CPU_INTERRUPT_HARD) &&
+    return ((cs->interrupt_request & (CPU_INTERRUPT_HARD |
+                                      CPU_INTERRUPT_POLL)) &&
             (env->eflags & IF_MASK)) ||
            (cs->interrupt_request & (CPU_INTERRUPT_NMI |
                                      CPU_INTERRUPT_INIT |
#2 lore raw

RFC PATCH v7 2/21 replay: global variables and function stubs

In reply to Pavel Dovgalyuk

This patch adds global variables, defines, functions declarations, and function stubs for deterministic VM replay used by external modules.

---
 Makefile.target      |    1 +
 qapi-schema.json     |   32 ++++++++++++++++++++++++++++++++
 replay/Makefile.objs |    1 +
 replay/replay.c      |   25 +++++++++++++++++++++++++
 replay/replay.h      |   23 +++++++++++++++++++++++
 stubs/Makefile.objs  |    1 +
 stubs/replay.c       |    8 ++++++++
 7 files changed, 91 insertions(+), 0 deletions(-)
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c
Makefile.target +1 -0
index e9ff1ee..a45378f 100644
@@ -83,6 +83,7 @@ all: $(PROGS) stap
 #########################################################
 # cpu emulator library
 obj-y = exec.o translate-all.o cpu-exec.o
+obj-y += replay/
 obj-y += tcg/tcg.o tcg/optimize.o
 obj-$(CONFIG_TCG_INTERPRETER) += tci.o
 obj-$(CONFIG_TCG_INTERPRETER) += disas/tci.o
qapi-schema.json +32 -0
index 563b4ad..a2a6636 100644
@@ -3515,3 +3515,35 @@
 # Since: 2.1
 ##
 { 'command': 'rtc-reset-reinjection' }
+
+##
+# ReplayMode:
+#
+# Mode of the replay subsystem.
+#
+# @none: normal execution mode. Replay or record are not enabled.
+#
+# @record: record mode. All non-deterministic data is written into the
+#          replay log.
+#
+# @play: replay mode. Non-deterministic data required for system execution
+#        is read from the log.
+#
+# Since: 2.3
+##
+{ 'enum': 'ReplayMode',
+  'data': [ 'none', 'record', 'play' ] }
+
+##
+# ReplaySubmode:
+#
+# Submode of the replay subsystem.
+#
+# @unknown: used for modes different from play.
+#
+# @normal: normal replay mode.
+#
+# Since: 2.3
+##
+{ 'enum': 'ReplaySubmode',
+  'data': [ 'unknown', 'normal' ] }
replay/Makefile.objs new file +1 -0
new file mode 100755
index 0000000..7ea860f
@@ -0,0 +1 @@
+obj-y += replay.o
replay/replay.c new file +25 -0
new file mode 100755
index 0000000..ac976b2
@@ -0,0 +1,25 @@
+/*
+ * replay.c
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "replay.h"
+
+ReplayMode replay_mode = REPLAY_MODE_NONE;
+/*! Stores current submode for PLAY mode */
+ReplaySubmode play_submode = REPLAY_SUBMODE_UNKNOWN;
+
+/* Suffix for the disk images filenames */
+char *replay_image_suffix;
+
+
+ReplaySubmode replay_get_play_submode(void)
+{
+    return play_submode;
+}
replay/replay.h new file +23 -0
new file mode 100755
index 0000000..51a18fe
@@ -0,0 +1,23 @@
+#ifndef REPLAY_H
+#define REPLAY_H
+
+/*
+ * replay.h
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qapi-types.h"
+
+extern ReplayMode replay_mode;
+extern char *replay_image_suffix;
+
+/*! Returns replay play submode */
+ReplaySubmode replay_get_play_submode(void);
+
+#endif
stubs/Makefile.objs +1 -0
index 5e347d0..45a6c71 100644
@@ -27,6 +27,7 @@ stub-obj-y += notify-event.o
 stub-obj-y += pci-drive-hot-add.o
 stub-obj-$(CONFIG_SPICE) += qemu-chr-open-spice.o
 stub-obj-y += qtest.o
+stub-obj-y += replay.o
 stub-obj-y += reset.o
 stub-obj-y += runstate-check.o
 stub-obj-y += set-fd-handler.o
stubs/replay.c new file +8 -0
new file mode 100755
index 0000000..f923d7d
@@ -0,0 +1,8 @@
+#include "replay/replay.h"
+
+ReplayMode replay_mode;
+
+ReplaySubmode replay_get_play_submode(void)
+{
+    return REPLAY_SUBMODE_UNKNOWN;
+}
#3 lore raw

Re: [Qemu-devel] [RFC PATCH v7 02/21] replay: global variables and function stubs

qapi-schema.json line 3549

Quoted 13 lines · Pavel Dovgalyuk
qapi-schema.json +13 -0
+##
+# ReplaySubmode:
+#
+# Submode of the replay subsystem.
+#
+# @unknown: used for modes different from play.
+#
+# @normal: normal replay mode.
+#
+# Since: 2.3
+##
+{ 'enum': 'ReplaySubmode',
+  'data': [ 'unknown', 'normal' ] }

What is a submode? Can you just check if replay_file is NULL?

Paolo

#4 lore raw

Re: [Qemu-devel] [RFC PATCH v7 02/21] replay: global variables and function stubs

Quoted 17 lines

From: Paolo Bonzini [mailto:[email protected]]

qapi-schema.json line 3549

Quoted 13 lines · Pavel Dovgalyuk
qapi-schema.json +13 -0
+##
+# ReplaySubmode:
+#
+# Submode of the replay subsystem.
+#
+# @unknown: used for modes different from play.
+#
+# @normal: normal replay mode.
+#
+# Since: 2.3
+##
+{ 'enum': 'ReplaySubmode',
+  'data': [ 'unknown', 'normal' ] }

What is a submode? Can you just check if replay_file is NULL?

Submode is not very useful for these core patches. More submodes will be introduces in reverse debugging patches.

Pavel Dovgalyuk

#5 lore raw

Re: [Qemu-devel] [RFC PATCH v7 02/21] replay: global variables and function stubs

Quoted 6 lines · Pavel Dovgaluk
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

qapi-schema.json line 3549

Quoted 2 lines
qapi-schema.json +2 -0
+{ 'enum': 'ReplaySubmode',
+  'data': [ 'unknown', 'normal' ] }

What is a submode? Can you just check if replay_file is NULL?

Submode is not very useful for these core patches. More submodes will be introduces in reverse debugging patches.

I see. For now, can you replace it with just a replay_file check, and move this definition to the reverse debugging patches?

Paolo

#6 lore raw

Re: [Qemu-devel] [RFC PATCH v7 02/21] replay: global variables and function stubs

In reply to Pavel Dovgalyuk
Quoted 33 lines · Pavel Dovgalyuk

This patch adds global variables, defines, functions declarations, and function stubs for deterministic VM replay used by external modules.

---

+++ b/qapi-schema.json

qapi-schema.json +22 -0
@@ -3515,3 +3515,35 @@
 # Since: 2.1
 ##
 { 'command': 'rtc-reset-reinjection' }
+
+##
+# ReplayMode:
+#
+# Mode of the replay subsystem.
+#
+# @none: normal execution mode. Replay or record are not enabled.
+#
+# @record: record mode. All non-deterministic data is written into the
+#          replay log.
+#
+# @play: replay mode. Non-deterministic data required for system execution
+#        is read from the log.
+#
+# Since: 2.3
+##
+{ 'enum': 'ReplayMode',
+  'data': [ 'none', 'record', 'play' ] }
+
+##
+# ReplaySubmode:
+#

qapi-schema.json line 3549

Quoted 10 lines
qapi-schema.json +10 -0
+# Submode of the replay subsystem.
+#
+# @unknown: used for modes different from play.
+#
+# @normal: normal replay mode.
+#
+# Since: 2.3
+##
+{ 'enum': 'ReplaySubmode',
+  'data': [ 'unknown', 'normal' ] }

I'm still not sure how ReplaySubmode fits in, as nothing in this patch is using either new type in the QAPI.

replay/replay.c line 5

Quoted 7 lines

+++ b/replay/replay.c

replay/replay.c new file +5 -0
@@ -0,0 +1,25 @@
+/*
+ * replay.c
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming
+ *                         of the Russian Academy of Sciences.

Now that your series revision work is continuing in 2015, you may want to update the years.

replay/replay.c line 16

Quoted 11 lines
replay/replay.c new file +11 -0
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "replay.h"
+
+ReplayMode replay_mode = REPLAY_MODE_NONE;
+/*! Stores current submode for PLAY mode */
+ReplaySubmode play_submode = REPLAY_SUBMODE_UNKNOWN;

It sounds like the submode will always be 'unknown' when not in play mode, and only 'normal' when in play mode? If so, there's no point in having it. If a later patch introduces separate play modes, then that would be the point where it makes more sense to introduce an optional submode designator.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#7 lore raw

RFC PATCH v7 3/21 sysemu: system functions for replay

In reply to Pavel Dovgalyuk

This patch removes "static" specifier from several qemu function to make them visible to the replay module. It also invents several system functions that will be used by replay.

---
 cpus.c                  |    4 ++--
 include/exec/exec-all.h |    1 +
 include/qom/cpu.h       |   10 ++++++++++
 include/sysemu/cpus.h   |    1 +
 translate-all.c         |    8 ++++++++
 5 files changed, 22 insertions(+), 2 deletions(-)
cpus.c +2 -2
index 2edb5cd..96580ad 100644
@@ -88,7 +88,7 @@ static bool cpu_thread_is_idle(CPUState *cpu)
     return true;
 }
 
-static bool all_cpu_threads_idle(void)
+bool all_cpu_threads_idle(void)
 {
     CPUState *cpu;
 
@@ -1121,7 +1121,7 @@ bool qemu_cpu_is_self(CPUState *cpu)
     return qemu_thread_is_self(cpu->thread);
 }
 
-static bool qemu_in_vcpu_thread(void)
+bool qemu_in_vcpu_thread(void)
 {
     return current_cpu && qemu_cpu_is_self(current_cpu);
 }
include/exec/exec-all.h +1 -0
index 4a6237f..63fb1a3 100644
@@ -214,6 +214,7 @@ static inline unsigned int tb_phys_hash_func(tb_page_addr_t pc)
 
 void tb_free(TranslationBlock *tb);
 void tb_flush(CPUArchState *env);
+void tb_flush_all(void);
 void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr);
 
 #if defined(USE_DIRECT_JUMP)
include/qom/cpu.h +10 -0
index 2098f1c..5afb44c 100644
@@ -480,6 +480,16 @@ static inline bool cpu_has_work(CPUState *cpu)
 bool qemu_cpu_is_self(CPUState *cpu);
 
 /**
+ * qemu_in_vcpu_thread:
+ *
+ * Checks whether the caller is executing on the vCPU thread
+ * of the current vCPU.
+ *
+ * Returns: %true if called from vCPU's thread, %false otherwise.
+ */
+bool qemu_in_vcpu_thread(void);
+
+/**
  * qemu_cpu_kick:
  * @cpu: The vCPU to kick.
  *
include/sysemu/cpus.h +1 -0
index 3f162a9..86ae556 100644
@@ -6,6 +6,7 @@ void qemu_init_cpu_loop(void);
 void resume_all_vcpus(void);
 void pause_all_vcpus(void);
 void cpu_stop_current(void);
+bool all_cpu_threads_idle(void);
 
 void cpu_synchronize_all_states(void);
 void cpu_synchronize_all_post_reset(void);
translate-all.c +8 -0
index 8fa4378..b04ada1 100644
@@ -812,6 +812,14 @@ void tb_flush(CPUArchState *env1)
     tcg_ctx.tb_ctx.tb_flush_count++;
 }
 
+void tb_flush_all(void)
+{
+    CPUState *cpu;
+    for (cpu = first_cpu ; cpu != NULL ; cpu = CPU_NEXT(cpu)) {
+        tb_flush(cpu->env_ptr);
+    }
+}
+
 #ifdef DEBUG_TB_CHECK
 
 static void tb_invalidate_check(target_ulong address)
#8 lore raw

RFC PATCH v7 4/21 replay: internal functions for replay log

In reply to Pavel Dovgalyuk

This patch adds functions to perform read and write operations with replay log.

---
 replay/Makefile.objs     |    1 
 replay/replay-internal.c |  141 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   50 ++++++++++++++++
 3 files changed, 192 insertions(+), 0 deletions(-)
 create mode 100755 replay/replay-internal.c
 create mode 100755 replay/replay-internal.h
replay/Makefile.objs +1 -0
index 7ea860f..1148f45 100755
@@ -1 +1,2 @@
 obj-y += replay.o
+obj-y += replay-internal.o
replay/replay-internal.c new file +141 -0

141 lines not on this page · raw message

replay/replay-internal.h new file +50 -0

50 lines not on this page · raw message

#9 lore raw

Re: [Qemu-devel] [RFC PATCH v7 04/21] replay: internal functions for replay log

replay/replay-internal.c line 17

Quoted 4 lines · Pavel Dovgalyuk
replay/replay-internal.c new file +4 -0
+
+volatile unsigned int replay_data_kind = -1;
+volatile unsigned int replay_has_unread_data;
+

Why does this have to be volatile?

Paolo

#10 lore raw

Re: [Qemu-devel] [RFC PATCH v7 04/21] replay: internal functions for replay log

Quoted 8 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

replay/replay-internal.c line 17

Quoted 4 lines · Pavel Dovgalyuk
replay/replay-internal.c new file +4 -0
+
+volatile unsigned int replay_data_kind = -1;
+volatile unsigned int replay_has_unread_data;
+

Why does this have to be volatile?

I think 'volatile' is an artifact from the earlier version. Could be removed then.

Pavel Dovgalyuk

#11 lore raw

RFC PATCH v7 5/21 replay: introduce icount event

In reply to Pavel Dovgalyuk

This patch adds icount event to the replay subsystem. This event corresponds to execution of several instructions and used to synchronize input events in the replay phase.

---
 replay/replay-internal.c |   14 ++++++++++++++
 replay/replay-internal.h |   18 ++++++++++++++++++
 replay/replay.c          |   46 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |    7 +++++++
 4 files changed, 85 insertions(+), 0 deletions(-)
replay/replay-internal.c +14 -0
index 429b13c..83a53bd 100755
@@ -10,6 +10,7 @@
  */
 
 #include "qemu-common.h"
+#include "replay.h"
 #include "replay-internal.h"
 
 volatile unsigned int replay_data_kind = -1;
@@ -139,3 +140,16 @@ void replay_fetch_data_kind(void)
         }
     }
 }
+
+/*! Saves cached instructions. */
+void replay_save_instructions(void)
+{
+    if (replay_file && replay_mode == REPLAY_MODE_RECORD) {
+        int diff = (int)(replay_get_current_step() - replay_state.current_step);
+        if (first_cpu != NULL && diff > 0) {
+            replay_put_event(EVENT_INSTRUCTION);
+            replay_put_dword(diff);
+            replay_state.current_step += diff;
+        }
+    }
+}
replay/replay-internal.h +18 -0
index 927f7c7..582b44c 100755
@@ -14,6 +14,17 @@
 
 #include <stdio.h>
 
+/* for instruction event */
+#define EVENT_INSTRUCTION           32
+
+typedef struct ReplayState {
+    /*! Current step - number of processed instructions and timer events. */
+    uint64_t current_step;
+    /*! Number of instructions to be executed before other events happen. */
+    int instructions_count;
+} ReplayState;
+extern ReplayState replay_state;
+
 extern volatile unsigned int replay_data_kind;
 extern volatile unsigned int replay_has_unread_data;
 
@@ -47,4 +58,11 @@ void replay_save_instructions(void);
     Terminates the program in case of error. */
 void validate_data_kind(int kind);
 
+/*! Skips async events until some sync event will be found. */
+bool skip_async_events(int stop_event);
+/*! Skips async events invocations from the input,
+    until required data kind is found. If the requested data is not found
+    reports an error and stops the execution. */
+void skip_async_events_until(unsigned int kind);
+
 #endif
replay/replay.c +46 -0

61 lines not on this page · raw message

replay/replay.h +7 -0
index 51a18fe..e40daf5 100755
@@ -12,6 +12,8 @@
  *
  */
 
+#include <stdbool.h>
+#include <stdint.h>
 #include "qapi-types.h"
 
 extern ReplayMode replay_mode;
@@ -20,4 +22,9 @@ extern char *replay_image_suffix;
 /*! Returns replay play submode */
 ReplaySubmode replay_get_play_submode(void);
 
+/* Processing the instructions */
+
+/*! Returns number of executed instructions. */
+uint64_t replay_get_current_step(void);
+
 #endif
#12 lore raw

RFC PATCH v7 6/21 cpu-exec: allow temporary disabling icount

In reply to Pavel Dovgalyuk

This patch is required for deterministic replay to generate an exception by trying executing an instruction without changing icount. It adds new flag to TB for disabling icount while translating it.

---
 cpu-exec.c              |    7 ++++---
 include/exec/exec-all.h |    1 +
 translate-all.c         |    2 +-
 3 files changed, 6 insertions(+), 4 deletions(-)
cpu-exec.c +4 -3
index 0a485bb..49f01f5 100644
@@ -200,7 +200,7 @@ static inline tcg_target_ulong cpu_tb_exec(CPUState *cpu, uint8_t *tb_ptr)
 /* Execute the code without caching the generated code. An interpreter
    could be used if available. */
 static void cpu_exec_nocache(CPUArchState *env, int max_cycles,
-                             TranslationBlock *orig_tb)
+                             TranslationBlock *orig_tb, bool ignore_icount)
 {
     CPUState *cpu = ENV_GET_CPU(env);
     TranslationBlock *tb;
@@ -216,7 +216,8 @@ static void cpu_exec_nocache(CPUArchState *env, int max_cycles,
     /* tb_gen_code can flush our orig_tb, invalidate it now */
     tb_phys_invalidate(orig_tb, -1);
     tb = tb_gen_code(cpu, pc, cs_base, flags,
-                     max_cycles | CF_NOCACHE);
+                     max_cycles | CF_NOCACHE
+                         | (ignore_icount ? CF_IGNORE_ICOUNT : 0));
     cpu->current_tb = tb;
     /* execute the generated code */
     trace_exec_tb_nocache(tb, tb->pc);
@@ -525,7 +526,7 @@ int cpu_exec(CPUArchState *env)
                         } else {
                             if (insns_left > 0) {
                                 /* Execute remaining instructions.  */
-                                cpu_exec_nocache(env, insns_left, tb);
+                                cpu_exec_nocache(env, insns_left, tb, false);
                                 align_clocks(&sc, cpu);
                             }
                             cpu->exception_index = EXCP_INTERRUPT;
include/exec/exec-all.h +1 -0
index 63fb1a3..48a7541 100644
@@ -148,6 +148,7 @@ struct TranslationBlock {
 #define CF_LAST_IO     0x8000 /* Last insn may be an IO access.  */
 #define CF_NOCACHE     0x10000 /* To be freed after execution */
 #define CF_USE_ICOUNT  0x20000
+#define CF_IGNORE_ICOUNT 0x40000 /* Do not generate icount code */
 
     void *tc_ptr;    /* pointer to the translated code */
     /* next matching tb for physical address. */
translate-all.c +1 -1
index b04ada1..c9abb55 100644
@@ -1053,7 +1053,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
     int code_gen_size;
 
     phys_pc = get_page_addr_code(env, pc);
-    if (use_icount) {
+    if (use_icount && !(cflags & CF_IGNORE_ICOUNT)) {
         cflags |= CF_USE_ICOUNT;
     }
     tb = tb_alloc(pc);
#13 lore raw

RFC PATCH v7 7/21 cpu: replay instructions sequence

In reply to Pavel Dovgalyuk

This patch adds calls to replay functions into the icount setup block. In record mode number of executed instructions is written to the log. In replay mode number of istructions to execute is taken from the replay log.

---
 cpu-exec.c      |    1 +
 cpus.c          |   28 ++++++++++++++++++----------
 replay/replay.c |   21 +++++++++++++++++++++
 replay/replay.h |    4 ++++
 4 files changed, 44 insertions(+), 10 deletions(-)
cpu-exec.c +1 -0
index 49f01f5..99a0993 100644
@@ -531,6 +531,7 @@ int cpu_exec(CPUArchState *env)
                             }
                             cpu->exception_index = EXCP_INTERRUPT;
                             next_tb = 0;
+                            qemu_notify_event();
                             cpu_loop_exit(cpu);
                         }
                         break;
cpus.c +18 -10
index 96580ad..0c368f2 100644
@@ -41,6 +41,7 @@
 #include "qemu/seqlock.h"
 #include "qapi-event.h"
 #include "hw/nmi.h"
+#include "replay/replay.h"
 
 #ifndef _WIN32
 #include "qemu/compatfd.h"
@@ -1342,18 +1343,22 @@ static int tcg_cpu_exec(CPUArchState *env)
                                     + cpu->icount_extra);
         cpu->icount_decr.u16.low = 0;
         cpu->icount_extra = 0;
-        deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
+        if (replay_mode != REPLAY_MODE_PLAY) {
+            deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
 
-        /* Maintain prior (possibly buggy) behaviour where if no deadline
-         * was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than
-         * INT32_MAX nanoseconds ahead, we still use INT32_MAX
-         * nanoseconds.
-         */
-        if ((deadline < 0) || (deadline > INT32_MAX)) {
-            deadline = INT32_MAX;
-        }
+            /* Maintain prior (possibly buggy) behaviour where if no deadline
+             * was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than
+             * INT32_MAX nanoseconds ahead, we still use INT32_MAX
+             * nanoseconds.
+             */
+            if ((deadline < 0) || (deadline > INT32_MAX)) {
+                deadline = INT32_MAX;
+            }
 
-        count = qemu_icount_round(deadline);
+            count = qemu_icount_round(deadline);
+        } else {
+            count = replay_get_instructions();
+        }
         timers_state.qemu_icount += count;
         decr = (count > 0xffff) ? 0xffff : count;
         count -= decr;
@@ -1371,6 +1376,9 @@ static int tcg_cpu_exec(CPUArchState *env)
                         + cpu->icount_extra);
         cpu->icount_decr.u32 = 0;
         cpu->icount_extra = 0;
+        if (replay_mode == REPLAY_MODE_PLAY) {
+            replay_exec_instructions();
+        }
     }
     return ret;
 }
replay/replay.c +21 -0
index 4b09685..8e391da 100755
@@ -69,3 +69,24 @@ uint64_t replay_get_current_step(void)
 {
     return cpu_get_icount_raw();
 }
+
+int replay_get_instructions(void)
+{
+    if (skip_async_events(EVENT_INSTRUCTION)) {
+        return replay_state.instructions_count;
+    }
+    return 0;
+}
+
+void replay_exec_instructions(void)
+{
+    if (replay_state.instructions_count > 0) {
+        int count = (int)(replay_get_current_step()
+                          - replay_state.current_step);
+        replay_state.instructions_count -= count;
+        replay_state.current_step += count;
+        if (replay_state.instructions_count == 0 && count != 0) {
+            replay_has_unread_data = 0;
+        }
+    }
+}
replay/replay.h +4 -0
index e40daf5..382dc7b 100755
@@ -26,5 +26,9 @@ ReplaySubmode replay_get_play_submode(void);
 
 /*! Returns number of executed instructions. */
 uint64_t replay_get_current_step(void);
+/*! Returns number of instructions to execute in replay mode. */
+int replay_get_instructions(void);
+/*! Updates instructions counter in replay mode. */
+void replay_exec_instructions(void);
 
 #endif
#14 lore raw

RFC PATCH v7 8/21 replay: interrupts and exceptions

In reply to Pavel Dovgalyuk

This patch includes modifications of common cpu files. All interrupts and exceptions occured during recording are written into the replay log. These events allow correct replaying the execution by kicking cpu thread when one of these events is found in the log.

---
 cpu-exec.c               |   42 ++++++++++++++++++++++++++--------
 replay/replay-internal.h |    4 +++
 replay/replay.c          |   57 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   17 ++++++++++++++
 target-i386/seg_helper.c |    4 +++
 5 files changed, 114 insertions(+), 10 deletions(-)
cpu-exec.c +32 -10
index 99a0993..bb20cf4 100644
@@ -24,6 +24,7 @@
 #include "qemu/atomic.h"
 #include "sysemu/qtest.h"
 #include "qemu/timer.h"
+#include "replay/replay.h"
 
 /* -icount align implementation. */
 
@@ -338,22 +339,25 @@ int cpu_exec(CPUArchState *env)
     /* This must be volatile so it is not trashed by longjmp() */
     volatile bool have_tb_lock = false;
 
+    /* replay_interrupt may need current_cpu */
+    current_cpu = cpu;
+
     if (cpu->halted) {
 #ifdef TARGET_I386
-        if (cpu->interrupt_request & CPU_INTERRUPT_POLL) {
+        if ((cpu->interrupt_request & CPU_INTERRUPT_POLL)
+            && replay_interrupt()) {
             apic_poll_irq(x86_cpu->apic_state);
             cpu_reset_interrupt(cpu, CPU_INTERRUPT_POLL);
         }
 #endif
         if (!cpu_has_work(cpu)) {
+            current_cpu = NULL;
             return EXCP_HALTED;
         }
 
         cpu->halted = 0;
     }
 
-    current_cpu = cpu;
-
     /* As long as current_cpu is null, up to the assignment just above,
      * requests by other threads to exit the execution loop are expected to
      * be issued using the exit_request global. We must make sure that our
@@ -400,10 +404,21 @@ int cpu_exec(CPUArchState *env)
                     cpu->exception_index = -1;
                     break;
 #else
-                    cc->do_interrupt(cpu);
-                    cpu->exception_index = -1;
+                    if (replay_exception()) {
+                        cc->do_interrupt(cpu);
+                        cpu->exception_index = -1;
+                    } else if (!replay_has_interrupt()) {
+                        /* give a chance to iothread in replay mode */
+                        ret = EXCP_INTERRUPT;
+                        break;
+                    }
 #endif
                 }
+            } else if (replay_has_exception()
+                       && cpu->icount_decr.u16.low + cpu->icount_extra == 0) {
+                /* try to cause an exception pending in the log */
+                cpu_exec_nocache(env, 1, tb_find_fast(env), true);
+                break;
             }
 
             next_tb = 0; /* force lookup of first TB */
@@ -419,21 +434,24 @@ int cpu_exec(CPUArchState *env)
                         cpu->exception_index = EXCP_DEBUG;
                         cpu_loop_exit(cpu);
                     }
-                    if (interrupt_request & CPU_INTERRUPT_HALT) {
+                    if ((interrupt_request & CPU_INTERRUPT_HALT)
+                        && replay_interrupt()) {
                         cpu->interrupt_request &= ~CPU_INTERRUPT_HALT;
                         cpu->halted = 1;
                         cpu->exception_index = EXCP_HLT;
                         cpu_loop_exit(cpu);
                     }
 #if defined(TARGET_I386)
-                    if (interrupt_request & CPU_INTERRUPT_INIT) {
+                    if ((interrupt_request & CPU_INTERRUPT_INIT)
+                        && replay_interrupt()) {
                         cpu_svm_check_intercept_param(env, SVM_EXIT_INIT, 0);
                         do_cpu_init(x86_cpu);
                         cpu->exception_index = EXCP_HALTED;
                         cpu_loop_exit(cpu);
                     }
 #else
-                    if (interrupt_request & CPU_INTERRUPT_RESET) {
+                    if ((interrupt_request & CPU_INTERRUPT_RESET)
+                        && replay_interrupt()) {
                         cpu_reset(cpu);
                     }
 #endif
@@ -441,7 +459,10 @@ int cpu_exec(CPUArchState *env)
                        False when the interrupt isn't processed,
                        True when it is, and we should restart on a new TB,
                        and via longjmp via cpu_loop_exit.  */
-                    if (cc->cpu_exec_interrupt(cpu, interrupt_request)) {
+                    if ((replay_mode != REPLAY_MODE_PLAY
+                            || replay_has_interrupt())
+                        && cc->cpu_exec_interrupt(cpu, interrupt_request)) {
+                        replay_interrupt();
                         next_tb = 0;
                     }
                     /* Don't use the cached interrupt_request value,
@@ -453,7 +474,8 @@ int cpu_exec(CPUArchState *env)
                         next_tb = 0;
                     }
                 }
-                if (unlikely(cpu->exit_request)) {
+                if (unlikely(cpu->exit_request
+                             || replay_has_interrupt())) {
                     cpu->exit_request = 0;
                     cpu->exception_index = EXCP_INTERRUPT;
                     cpu_loop_exit(cpu);
replay/replay-internal.h +4 -0
index 582b44c..fd5c230 100755
@@ -14,6 +14,10 @@
 
 #include <stdio.h>
 
+/* for software interrupt */
+#define EVENT_INTERRUPT             15
+/* for emulated exceptions */
+#define EVENT_EXCEPTION             23
 /* for instruction event */
 #define EVENT_INSTRUCTION           32
 
replay/replay.c +57 -0

60 lines not on this page · raw message

replay/replay.h +17 -0
index 382dc7b..60dfa4a 100755
@@ -31,4 +31,21 @@ int replay_get_instructions(void);
 /*! Updates instructions counter in replay mode. */
 void replay_exec_instructions(void);
 
+/* Interrupts and exceptions */
+
+/*! Called by exception handler to write or read
+    exception processing events. */
+bool replay_exception(void);
+/*! Used to determine that exception is pending.
+    Does not proceed to the next event in the log. */
+bool replay_has_exception(void);
+/*! Called by interrupt handlers to write or read
+    interrupt processing events.
+    \return true if interrupt should be processed */
+bool replay_interrupt(void);
+/*! Tries to read interrupt event from the file.
+    Returns true, when interrupt request is pending */
+bool replay_has_interrupt(void);
+
+
 #endif
target-i386/seg_helper.c +4 -0
index c98eeb4..945e3c2 100644
@@ -22,6 +22,7 @@
 #include "qemu/log.h"
 #include "exec/helper-proto.h"
 #include "exec/cpu_ldst.h"
+#include "replay/replay.h"
 
 //#define DEBUG_PCALL
 
@@ -1280,6 +1281,9 @@ bool x86_cpu_exec_interrupt(CPUState *cs, int interrupt_request)
     if (interrupt_request & CPU_INTERRUPT_POLL) {
         cs->interrupt_request &= ~CPU_INTERRUPT_POLL;
         apic_poll_irq(cpu->apic_state);
+        if (replay_mode != REPLAY_MODE_NONE) {
+            return true;
+        }
     }
 #endif
     if (interrupt_request & CPU_INTERRUPT_SIPI) {
#15 lore raw

Re: [Qemu-devel] [RFC PATCH v7 08/21] replay: interrupts and exceptions

cpu-exec.c int cpu_exec(CPUArchState *env) · line 409

Quoted 4 lines · Pavel Dovgalyuk
cpu-exec.c +3 -0
+                    if (replay_exception()) {
+                        cc->do_interrupt(cpu);
+                        cpu->exception_index = -1;

I cannot see replay_exception() in the series?

Review int cpu_exec(CPUArchState *env) · line 435

Quoted 3 lines
(quoted diff) +0 -0
@@ -419,21 +434,24 @@ int cpu_exec(CPUArchState *env)
                         cpu->exception_index = EXCP_DEBUG;
                         cpu_loop_exit(cpu);

Why not for EXCP_DEBUG?

Quoted 15 lines
cpu-exec.c +4 -2
                     }
-                    if (interrupt_request & CPU_INTERRUPT_HALT) {
+                    if ((interrupt_request & CPU_INTERRUPT_HALT)
+                        && replay_interrupt()) {
                         cpu->interrupt_request &= ~CPU_INTERRUPT_HALT;
                         cpu->halted = 1;
                         cpu->exception_index = EXCP_HLT;
                         cpu_loop_exit(cpu);
                     }
 #if defined(TARGET_I386)
-                    if (interrupt_request & CPU_INTERRUPT_INIT) {
+                    if ((interrupt_request & CPU_INTERRUPT_INIT)
+                        && replay_interrupt()) {
                         cpu_svm_check_intercept_param(env, SVM_EXIT_INIT, 0);
                         do_cpu_init(x86_cpu);

cpu-exec.c int cpu_exec(CPUArchState *env) · line 457

Quoted 10 lines
cpu-exec.c +2 -1
                         cpu->exception_index = EXCP_HALTED;
                         cpu_loop_exit(cpu);
                     }
 #else
-                    if (interrupt_request & CPU_INTERRUPT_RESET) {
+                    if ((interrupt_request & CPU_INTERRUPT_RESET)
+                        && replay_interrupt()) {
                         cpu_reset(cpu);
                     }
 #endif

Perhaps check the replay_interrupt() outside, in an && with "if (unlikely(interrupt_request))"?

cpu-exec.c int cpu_exec(CPUArchState *env) · line 465

Quoted 9 lines
cpu-exec.c +4 -1
@@ -441,7 +459,10 @@ int cpu_exec(CPUArchState *env)
                        False when the interrupt isn't processed,
                        True when it is, and we should restart on a new TB,
                        and via longjmp via cpu_loop_exit.  */
-                    if (cc->cpu_exec_interrupt(cpu, interrupt_request)) {
+                    if ((replay_mode != REPLAY_MODE_PLAY
+                            || replay_has_interrupt())
+                        && cc->cpu_exec_interrupt(cpu, interrupt_request)) {
+                        replay_interrupt();

Please put this in a separate function like:

    if (replay_mode == REPLAY_MODE_PLAY && !replay_has_interrupt()) {
        return false;
    }
    ret = cc->cpu_exec_interrupt(cpu, interrupt_request);
    if (ret) {
        replay_interrupt();
    }
    return ret;

Paolo

Quoted 4 lines
cpu-exec.c +0 -0
                         next_tb = 0;
                     }
                     /* Don't use the cached interrupt_request value,
@@ -453,7 +474,8 @@ int cpu_exec(CPUArchState *env)
#16 lore raw

Re: [Qemu-devel] [RFC PATCH v7 08/21] replay: interrupts and exceptions

Quoted 8 lines

From: Paolo Bonzini [mailto:[email protected]]

cpu-exec.c int cpu_exec(CPUArchState *env) · line 409

Quoted 4 lines · Pavel Dovgalyuk
cpu-exec.c +3 -0
+                    if (replay_exception()) {
+                        cc->do_interrupt(cpu);
+                        cpu->exception_index = -1;

I cannot see replay_exception() in the series?

It is in the patch number 8.

Quoted 6 lines

Review int cpu_exec(CPUArchState *env) · line 435

Quoted 4 lines
(quoted diff) +0 -0
@@ -419,21 +434,24 @@ int cpu_exec(CPUArchState *env)
                         cpu->exception_index = EXCP_DEBUG;
                         cpu_loop_exit(cpu);

Why not for EXCP_DEBUG?

As far as I understand, EXCP_DEBUG is used for external debugger (like gdb). Debugger is usually connected to VM during replay.
That is why we do not need to replay EXCP_DEBUG - it may appear in different places, and it does not affect on behavior of the virtual machine.

Quoted 25 lines
Quoted 14 lines
cpu-exec.c +4 -2
-                    if (interrupt_request & CPU_INTERRUPT_HALT) {
+                    if ((interrupt_request & CPU_INTERRUPT_HALT)
+                        && replay_interrupt()) {
                         cpu->interrupt_request &= ~CPU_INTERRUPT_HALT;
                         cpu->halted = 1;
                         cpu->exception_index = EXCP_HLT;
                         cpu_loop_exit(cpu);
                     }
 #if defined(TARGET_I386)
-                    if (interrupt_request & CPU_INTERRUPT_INIT) {
+                    if ((interrupt_request & CPU_INTERRUPT_INIT)
+                        && replay_interrupt()) {
                         cpu_svm_check_intercept_param(env, SVM_EXIT_INIT, 0);
                         do_cpu_init(x86_cpu);

cpu-exec.c int cpu_exec(CPUArchState *env) · line 457

Quoted 10 lines
cpu-exec.c +2 -1
                         cpu->exception_index = EXCP_HALTED;
                         cpu_loop_exit(cpu);
                     }
 #else
-                    if (interrupt_request & CPU_INTERRUPT_RESET) {
+                    if ((interrupt_request & CPU_INTERRUPT_RESET)
+                        && replay_interrupt()) {
                         cpu_reset(cpu);
                     }
 #endif
Quoted 2 lines

Perhaps check the replay_interrupt() outside, in an && with "if (unlikely(interrupt_request))"?

You mean that I should wrap whole condition into "unlikely"?

Quoted 12 lines

cpu-exec.c int cpu_exec(CPUArchState *env) · line 465

Quoted 9 lines
cpu-exec.c +4 -1
@@ -441,7 +459,10 @@ int cpu_exec(CPUArchState *env)
                        False when the interrupt isn't processed,
                        True when it is, and we should restart on a new TB,
                        and via longjmp via cpu_loop_exit.  */
-                    if (cc->cpu_exec_interrupt(cpu, interrupt_request)) {
+                    if ((replay_mode != REPLAY_MODE_PLAY
+                            || replay_has_interrupt())
+                        && cc->cpu_exec_interrupt(cpu, interrupt_request)) {
+                        replay_interrupt();

Please put this in a separate function like:

Quoted 8 lines
    if (replay_mode == REPLAY_MODE_PLAY && !replay_has_interrupt()) {
        return false;
    }
    ret = cc->cpu_exec_interrupt(cpu, interrupt_request);
    if (ret) {
        replay_interrupt();
    }
    return ret;

Ok.

Pavel Dovgalyuk

#17 lore raw

Re: [Qemu-devel] [RFC PATCH v7 08/21] replay: interrupts and exceptions

Quoted 8 lines · Pavel Dovgaluk
Quoted 6 lines

cpu-exec.c int cpu_exec(CPUArchState *env) · line 409

Quoted 4 lines
cpu-exec.c +3 -0
+                    if (replay_exception()) {
+                        cc->do_interrupt(cpu);
+                        cpu->exception_index = -1;

I cannot see replay_exception() in the series?

It is in the patch number 8.

Oops, it is in this patch in fact.

Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

Review int cpu_exec(CPUArchState *env) · line 435

Quoted 3 lines
(quoted diff) +0 -0
@@ -419,21 +434,24 @@ int cpu_exec(CPUArchState *env)
                         cpu->exception_index = EXCP_DEBUG;
                         cpu_loop_exit(cpu);

Why not for EXCP_DEBUG?

As far as I understand, EXCP_DEBUG is used for external debugger (like gdb). Debugger is usually connected to VM during replay.
That is why we do not need to replay EXCP_DEBUG - it may appear in different places, and it does not affect on behavior of the virtual machine.

Ok.

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 24 lines
cpu-exec.c +6 -3
-                    if (interrupt_request & CPU_INTERRUPT_HALT) {
+                    if ((interrupt_request & CPU_INTERRUPT_HALT)
+                        && replay_interrupt()) {
                         cpu->interrupt_request &= ~CPU_INTERRUPT_HALT;
                         cpu->halted = 1;
                         cpu->exception_index = EXCP_HLT;
                         cpu_loop_exit(cpu);
                     }
 #if defined(TARGET_I386)
-                    if (interrupt_request & CPU_INTERRUPT_INIT) {
+                    if ((interrupt_request & CPU_INTERRUPT_INIT)
+                        && replay_interrupt()) {
                         cpu_svm_check_intercept_param(env, SVM_EXIT_INIT, 0);
                         do_cpu_init(x86_cpu);
                         cpu->exception_index = EXCP_HALTED;
                         cpu_loop_exit(cpu);
                     }
 #else
-                    if (interrupt_request & CPU_INTERRUPT_RESET) {
+                    if ((interrupt_request & CPU_INTERRUPT_RESET)
+                        && replay_interrupt()) {
                         cpu_reset(cpu);
                     }
 #endif
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

Perhaps check the replay_interrupt() outside, in an && with "if (unlikely(interrupt_request))"?

You mean that I should wrap whole condition into "unlikely"?

No, I wanted to have a single check of "replay_interrupt()" and/or "replay_has_interrupt()".

BTW, I think this is incorrect:

cpu-exec.c int cpu_exec(CPUArchState *env) · line 465

Quoted 4 lines
cpu-exec.c +4 -0
+                    if ((replay_mode != REPLAY_MODE_PLAY
+                            || replay_has_interrupt())
+                        && cc->cpu_exec_interrupt(cpu, interrupt_request)) {
+                        replay_interrupt();

because cc->cpu_exec_interrupt() can exit with cpu_loop_exit(cpu).

I guess it could be something like this:

    if (replay_mode == REPLAY_MODE_PLAY && !replay_has_interrupt()) {
        /* do nothing */
    } else if (interrupt_request & CPU_INTERRUPT_HALT) {
        replay_interrupt();
        ...
        cpu_loop_exit(cpu);
    } else if (interrupt_request & CPU_INTERRUPT_INIT) {
        replay_interrupt();
        ...
        cpu_loop_exit(cpu);
    } else {
        replay_interrupt();
        if (cc->cpu_exec_interrupt(cpu, interrupt_request)) {
            next_tb = 0;
        }
    }

Paolo

#18 lore raw

Re: [Qemu-devel] [RFC PATCH v7 08/21] replay: interrupts and exceptions

Quoted 11 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 5 lines · Pavel Dovgaluk
Quoted 3 lines
Quoted 3 lines

Perhaps check the replay_interrupt() outside, in an && with "if (unlikely(interrupt_request))"?

You mean that I should wrap whole condition into "unlikely"?

No, I wanted to have a single check of "replay_interrupt()" and/or "replay_has_interrupt()".

Quoted 8 lines

BTW, I think this is incorrect:

cpu-exec.c int cpu_exec(CPUArchState *env) · line 465

Quoted 4 lines
cpu-exec.c +4 -0
+                    if ((replay_mode != REPLAY_MODE_PLAY
+                            || replay_has_interrupt())
+                        && cc->cpu_exec_interrupt(cpu, interrupt_request)) {
+                        replay_interrupt();

because cc->cpu_exec_interrupt() can exit with cpu_loop_exit(cpu).

Haven't found any. Do you have an example?

Quoted 19 lines

I guess it could be something like this:

    if (replay_mode == REPLAY_MODE_PLAY && !replay_has_interrupt()) {
        /* do nothing */
    } else if (interrupt_request & CPU_INTERRUPT_HALT) {
        replay_interrupt();
        ...
        cpu_loop_exit(cpu);
    } else if (interrupt_request & CPU_INTERRUPT_INIT) {
        replay_interrupt();
        ...
        cpu_loop_exit(cpu);
    } else {
        replay_interrupt();
        if (cc->cpu_exec_interrupt(cpu, interrupt_request)) {
            next_tb = 0;
        }
    }

Is it normal that processing of the reset request does not execute cpu_loop_exit(cpu)?

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

cpu-exec.c int cpu_exec(CPUArchState *env) · line 457

Quoted 7 lines

#else

cpu-exec.c +2 -1
-                    if (interrupt_request & CPU_INTERRUPT_RESET) {
+                    if ((interrupt_request & CPU_INTERRUPT_RESET)
+                        && replay_interrupt()) {
                         cpu_reset(cpu);
                     }
 #endif

Pavel Dovgalyuk

A message that is not in the archive (sent off-list or not yet received): [email protected]

#20 lore raw

Re: [Qemu-devel] [RFC PATCH v7 08/21] replay: interrupts and exceptions

Quoted 13 lines · Pavel Dovgaluk
Quoted 13 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 5 lines · Pavel Dovgaluk
Quoted 3 lines
Quoted 3 lines

Perhaps check the replay_interrupt() outside, in an && with "if (unlikely(interrupt_request))"?

You mean that I should wrap whole condition into "unlikely"?

No, I wanted to have a single check of "replay_interrupt()" and/or "replay_has_interrupt()".

BTW, I think this is incorrect:

Quoted 8 lines
Quoted 6 lines

cpu-exec.c int cpu_exec(CPUArchState *env) · line 465

Quoted 4 lines
cpu-exec.c +4 -0
+                    if ((replay_mode != REPLAY_MODE_PLAY
+                            || replay_has_interrupt())
+                        && cc->cpu_exec_interrupt(cpu, interrupt_request)) {
+                        replay_interrupt();

because cc->cpu_exec_interrupt() can exit with cpu_loop_exit(cpu).

Haven't found any. Do you have an example?

Yes:

    cpu_svm_check_intercept_param ->
    helper_svm_check_intercept_param ->
    helper_vmexit -> cpu_loop_exit
Quoted 17 lines
Quoted 8 lines
    if (replay_mode == REPLAY_MODE_PLAY && !replay_has_interrupt()) {
        /* do nothing */
    } else if (interrupt_request & CPU_INTERRUPT_HALT) {
        replay_interrupt();
        ...
        cpu_loop_exit(cpu);
    } else if (interrupt_request & CPU_INTERRUPT_INIT) {
        replay_interrupt();
Quoted 8 lines
        ...
        cpu_loop_exit(cpu);
    } else {
        replay_interrupt();
        if (cc->cpu_exec_interrupt(cpu, interrupt_request)) {
            next_tb = 0;
        }
    }
Quoted 1 line

Is it normal that processing of the reset request does not execute cpu_loop_exit(cpu)?

I think it is okay. INIT executes cpu_loop_exit() on x86 because processors other than the boot processor are halted after they receive INIT.

Paolo

#21 lore raw

Re: [Qemu-devel] [RFC PATCH v7 08/21] replay: interrupts and exceptions

Quoted 22 lines

From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini

Quoted 20 lines · Pavel Dovgaluk
Quoted 19 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 5 lines · Pavel Dovgaluk
Quoted 3 lines
Quoted 3 lines

Perhaps check the replay_interrupt() outside, in an && with "if (unlikely(interrupt_request))"?

You mean that I should wrap whole condition into "unlikely"?

No, I wanted to have a single check of "replay_interrupt()" and/or "replay_has_interrupt()".

BTW, I think this is incorrect:

cpu-exec.c int cpu_exec(CPUArchState *env) · line 465

Quoted 4 lines
cpu-exec.c +4 -0
+                    if ((replay_mode != REPLAY_MODE_PLAY
+                            || replay_has_interrupt())
+                        && cc->cpu_exec_interrupt(cpu, interrupt_request)) {
+                        replay_interrupt();

because cc->cpu_exec_interrupt() can exit with cpu_loop_exit(cpu).

Quoted 7 lines
Quoted 1 line

Haven't found any. Do you have an example?

Yes:

    cpu_svm_check_intercept_param ->
    helper_svm_check_intercept_param ->
    helper_vmexit -> cpu_loop_exit

Thanks.

Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
    if (replay_mode == REPLAY_MODE_PLAY && !replay_has_interrupt()) {
        /* do nothing */
    } else if (interrupt_request & CPU_INTERRUPT_HALT) {
        replay_interrupt();
        ...
        cpu_loop_exit(cpu);
    } else if (interrupt_request & CPU_INTERRUPT_INIT) {
        replay_interrupt();
        ...
        cpu_loop_exit(cpu);
    } else {
        replay_interrupt();
        if (cc->cpu_exec_interrupt(cpu, interrupt_request)) {
            next_tb = 0;
        }
    }
Quoted 4 lines
Quoted 1 line

Is it normal that processing of the reset request does not execute cpu_loop_exit(cpu)?

I think it is okay. INIT executes cpu_loop_exit() on x86 because processors other than the boot processor are halted after they receive INIT.

Then I cannot put everything in one if-else chain because it will change the behavior of the code. After processing RESET branch we can also process hardware interrupts (in unmodified code).

Pavel Dovgalyuk

A message that is not in the archive (sent off-list or not yet received): [email protected]

#23 lore raw

Re: [Qemu-devel] [RFC PATCH v7 08/21] replay: interrupts and exceptions

Quoted 4 lines · Pavel Dovgaluk
Quoted 2 lines
Quoted 2 lines

I think it is okay. INIT executes cpu_loop_exit() on x86 because processors other than the boot processor are halted after they receive INIT.

Then I cannot put everything in one if-else chain because it will change the behavior of the code. After processing RESET branch we can also process hardware interrupts (in unmodified code).

Then if it's simpler you can add cpu_loop_exit(). It's not required, but it's also definitely not a fast path.

Paolo

#24 lore raw

RFC PATCH v7 9/21 replay: asynchronous events infrastructure

In reply to Pavel Dovgalyuk

This patch adds module for saving and replaying asynchronous events. These events include network packets, keyboard and mouse input, USB packets, thread pool and bottom halves callbacks.
All events are stored in the queue to be processed at synchronization points such as beginning of TB execution, or checkpoint in the iothread.

---
 replay/Makefile.objs     |    1 
 replay/replay-events.c   |  217 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   27 ++++++
 replay/replay.h          |    4 +
 4 files changed, 249 insertions(+), 0 deletions(-)
 create mode 100755 replay/replay-events.c
replay/Makefile.objs +1 -0
index 1148f45..56da09c 100755
@@ -1,2 +1,3 @@
 obj-y += replay.o
 obj-y += replay-internal.o
+obj-y += replay-events.o
replay/replay-events.c new file +217 -0

217 lines not on this page · raw message

replay/replay-internal.h +27 -0
index fd5c230..fcba977 100755
@@ -18,9 +18,15 @@
 #define EVENT_INTERRUPT             15
 /* for emulated exceptions */
 #define EVENT_EXCEPTION             23
+/* for async events */
+#define EVENT_ASYNC_OPT             25
 /* for instruction event */
 #define EVENT_INSTRUCTION           32
 
+/* Asynchronous events IDs */
+
+#define REPLAY_ASYNC_COUNT             0
+
 typedef struct ReplayState {
     /*! Current step - number of processed instructions and timer events. */
     uint64_t current_step;
@@ -69,4 +75,25 @@ bool skip_async_events(int stop_event);
     reports an error and stops the execution. */
 void skip_async_events_until(unsigned int kind);
 
+/* Asynchronous events queue */
+
+/*! Initializes events' processing internals */
+void replay_init_events(void);
+/*! Clears internal data structures for events handling */
+void replay_finish_events(void);
+/*! Enables storing events in the queue */
+void replay_enable_events(void);
+/*! Flushes events queue */
+void replay_flush_events(void);
+/*! Clears events list before loading new VM state */
+void replay_clear_events(void);
+/*! Returns true if there are any unsaved events in the queue */
+bool replay_has_events(void);
+/*! Saves events from queue into the file */
+void replay_save_events(int opt);
+/*! Read events from the file into the input queue */
+void replay_read_events(int opt);
+/*! Adds specified async event to the queue */
+void replay_add_event(int event_id, void *opaque);
+
 #endif
replay/replay.h +4 -0
index 60dfa4a..90a949b 100755
@@ -47,5 +47,9 @@ bool replay_interrupt(void);
     Returns true, when interrupt request is pending */
 bool replay_has_interrupt(void);
 
+/* Asynchronous events queue */
+
+/*! Disables storing events in the queue */
+void replay_disable_events(void);
 
 #endif
#25 lore raw

Re: [Qemu-devel] [RFC PATCH v7 09/21] replay: asynchronous events infrastructure

replay/replay-events.c line 66

Quoted 10 lines · Pavel Dovgalyuk
replay/replay-events.c new file +10 -0
+void replay_flush_events(void)
+{
+    qemu_mutex_lock(&lock);
+    while (!QTAILQ_EMPTY(&events_list)) {
+        Event *event = QTAILQ_FIRST(&events_list);
+        replay_run_event(event);
+        QTAILQ_REMOVE(&events_list, event, events);
+        g_free(event);
+    }
+    qemu_mutex_unlock(&lock);

replay_run_event must run without the replay mutex, otherwise you risk ABBA deadlocks.

Paolo

#26 lore raw

RFC PATCH v7 10/21 cpus: make icount warp deterministic in replay mode

In reply to Pavel Dovgalyuk

This patch fixes icount warp rt timer. It should be based on virtual_rt clock to be deterministic.

---
 cpus.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
cpus.c +1 -1
index 0c368f2..8787277 100644
@@ -325,7 +325,7 @@ static void icount_adjust(void)
 static void icount_adjust_rt(void *opaque)
 {
     timer_mod(icount_rt_timer,
-                   qemu_clock_get_ms(QEMU_CLOCK_REALTIME) + 1000);
+              qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL_RT) + 1000);
     icount_adjust();
 }
 
#27 lore raw

Re: [Qemu-devel] [RFC PATCH v7 10/21] cpus: make icount warp deterministic in replay mode

Quoted 14 lines · Pavel Dovgalyuk

This patch fixes icount warp rt timer. It should be based on virtual_rt clock to be deterministic.

---
 cpus.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
cpus.c +0 -0
index 0c368f2..8787277 100644

cpus.c static void icount_adjust(void) · line 331

Quoted 8 lines
cpus.c +1 -1
@@ -325,7 +325,7 @@ static void icount_adjust(void)
 static void icount_adjust_rt(void *opaque)
 {
     timer_mod(icount_rt_timer,
-                   qemu_clock_get_ms(QEMU_CLOCK_REALTIME) + 1000);
+              qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL_RT) + 1000);
     icount_adjust();
 }
 

Oops, this is indeed a rebase conflict in the VIRTUAL_RT patches. I'll include this now.

Paolo

#28 lore raw

RFC PATCH v7 11/21 timer: fix usage of clock functions

In reply to Pavel Dovgalyuk

This patch replaces calling of get_clock_realtime() everywhere except the timer module. All calls are replaced with host clock requests. Patch also replaces get_clock() calls with realtime clock requests. Usage of this interface simplifies implementation of record/replay.

---
 pc-bios/s390-ccw/virtio.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
pc-bios/s390-ccw/virtio.c +1 -1
index c0540d1..f1867ba 100644
@@ -164,7 +164,7 @@ static u64 get_clock(void)
 
 static ulong get_second(void)
 {
-    return (get_clock() >> 12) / 1000000;
+    return (qemu_clock_get_ns(QEMU_CLOCK_REALTIME) >> 12) / 1000000;
 }
 
 /*
#29 lore raw

Re: [Qemu-devel] [RFC PATCH v7 11/21] timer: fix usage of clock functions

Quoted 6 lines · Pavel Dovgalyuk

This patch replaces calling of get_clock_realtime() everywhere except the timer module. All calls are replaced with host clock requests. Patch also replaces get_clock() calls with realtime clock requests. Usage of this interface simplifies implementation of record/replay.

This is firmware, not part of QEMU. Clock functions are not used directly anymore.

Paolo

#30 lore raw

RFC PATCH v7 12/21 replay: recording and replaying clock ticks

In reply to Pavel Dovgalyuk

Clock ticks are considered as the sources of non-deterministic data for virtual machine. This patch implements saving the clock values when they are acquired (virtual, host clock, rdtsc, and some other timers). When replaying the execution corresponding values are read from log and transfered to the module, which wants to read the values.
Such a design required the clock polling to be synchronized. Sometimes it is not true - e.g. when timeouts for timer lists are checked. In this case we use a cached value of the clock, passing it to the client code.

---
 cpus.c                   |    3 +-
 include/qemu/timer.h     |   10 ++++++
 qemu-timer.c             |    7 ++--
 replay/Makefile.objs     |    1 +
 replay/replay-internal.h |   11 ++++++
 replay/replay-time.c     |   79 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   23 +++++++++++++
 stubs/replay.c           |    9 +++++
 8 files changed, 139 insertions(+), 4 deletions(-)
 create mode 100755 replay/replay-time.c
cpus.c +2 -1
index 8787277..01d89aa 100644
@@ -353,7 +353,8 @@ static void icount_warp_rt(void *opaque)
 
     seqlock_write_lock(&timers_state.vm_clock_seqlock);
     if (runstate_is_running()) {
-        int64_t clock = cpu_get_clock_locked();
+        int64_t clock = REPLAY_CLOCK(REPLAY_CLOCK_VIRTUAL_RT,
+                                     cpu_get_clock_locked());
         int64_t warp_delta;
 
         warp_delta = clock - vm_clock_warp_start;
include/qemu/timer.h +10 -0
index d9df094..a8bc9eb 100644
@@ -4,6 +4,7 @@
 #include "qemu/typedefs.h"
 #include "qemu-common.h"
 #include "qemu/notify.h"
+#include "replay/replay.h"
 
 /* timers */
 
@@ -760,6 +761,8 @@ int64_t cpu_icount_to_ns(int64_t icount);
 /*******************************************/
 /* host CPU ticks (if available) */
 
+#define cpu_get_real_ticks cpu_get_real_ticks_impl
+
 #if defined(_ARCH_PPC)
 
 static inline int64_t cpu_get_real_ticks(void)
@@ -913,6 +916,13 @@ static inline int64_t cpu_get_real_ticks (void)
 }
 #endif
 
+#undef cpu_get_real_ticks
+
+static inline int64_t cpu_get_real_ticks(void)
+{
+    return REPLAY_CLOCK(REPLAY_CLOCK_REAL_TICKS, cpu_get_real_ticks_impl());
+}
+
 #ifdef CONFIG_PROFILER
 static inline int64_t profile_getclock(void)
 {
qemu-timer.c +4 -3
index cb7d988..19b82f6 100644
@@ -25,6 +25,7 @@
 #include "sysemu/sysemu.h"
 #include "monitor/monitor.h"
 #include "ui/console.h"
+#include "replay/replay.h"
 
 #include "hw/hw.h"
 
@@ -566,15 +567,15 @@ int64_t qemu_clock_get_ns(QEMUClockType type)
             return cpu_get_clock();
         }
     case QEMU_CLOCK_HOST:
-        now = get_clock_realtime();
+        now = REPLAY_CLOCK(REPLAY_CLOCK_HOST, get_clock_realtime());
         last = clock->last;
         clock->last = now;
-        if (now < last) {
+        if (now < last && replay_mode == REPLAY_MODE_NONE) {
             notifier_list_notify(&clock->reset_notifiers, &now);
         }
         return now;
     case QEMU_CLOCK_VIRTUAL_RT:
-        return cpu_get_clock();
+        return REPLAY_CLOCK(REPLAY_CLOCK_VIRTUAL_RT, cpu_get_clock());
     }
 }
 
replay/Makefile.objs +1 -0
index 56da09c..257c320 100755
@@ -1,3 +1,4 @@
 obj-y += replay.o
 obj-y += replay-internal.o
 obj-y += replay-events.o
+obj-y += replay-time.o
replay/replay-internal.h +11 -0
index fcba977..c36d7de 100755
@@ -22,12 +22,17 @@
 #define EVENT_ASYNC_OPT             25
 /* for instruction event */
 #define EVENT_INSTRUCTION           32
+/* for clock read/writes */
+#define EVENT_CLOCK                 64
+/* some of grteater codes are reserved for clocks */
 
 /* Asynchronous events IDs */
 
 #define REPLAY_ASYNC_COUNT             0
 
 typedef struct ReplayState {
+    /*! Cached clock values. */
+    int64_t cached_clock[REPLAY_CLOCK_COUNT];
     /*! Current step - number of processed instructions and timer events. */
     uint64_t current_step;
     /*! Number of instructions to be executed before other events happen. */
@@ -75,6 +80,12 @@ bool skip_async_events(int stop_event);
     reports an error and stops the execution. */
 void skip_async_events_until(unsigned int kind);
 
+/*! Reads next clock value from the file.
+    If clock kind read from the file is different from the parameter,
+    the value is not used.
+    If the parameter is -1, the clock value is read to the cache anyway. */
+void replay_read_next_clock(unsigned int kind);
+
 /* Asynchronous events queue */
 
 /*! Initializes events' processing internals */
replay/replay-time.c new file +79 -0

79 lines not on this page · raw message

replay/replay.h +23 -0
index 90a949b..1c18c0e 100755
@@ -16,6 +16,16 @@
 #include <stdint.h>
 #include "qapi-types.h"
 
+/* replay clock kinds */
+/* rdtsc */
+#define REPLAY_CLOCK_REAL_TICKS 0
+/* host_clock */
+#define REPLAY_CLOCK_HOST       1
+/* virtual_rt_clock */
+#define REPLAY_CLOCK_VIRTUAL_RT 2
+
+#define REPLAY_CLOCK_COUNT      3
+
 extern ReplayMode replay_mode;
 extern char *replay_image_suffix;
 
@@ -47,6 +57,19 @@ bool replay_interrupt(void);
     Returns true, when interrupt request is pending */
 bool replay_has_interrupt(void);
 
+/* Processing clocks and other time sources */
+
+/*! Save the specified clock */
+int64_t replay_save_clock(unsigned int kind, int64_t clock);
+/*! Read the specified clock from the log or return cached data */
+int64_t replay_read_clock(unsigned int kind);
+/*! Saves or reads the clock depending on the current replay mode. */
+#define REPLAY_CLOCK(clock, value)                                      \
+    (replay_mode == REPLAY_MODE_PLAY ? replay_read_clock((clock))       \
+        : replay_mode == REPLAY_MODE_RECORD                             \
+            ? replay_save_clock((clock), (value))                       \
+        : (value))
+
 /* Asynchronous events queue */
 
 /*! Disables storing events in the queue */
stubs/replay.c +9 -0
index f923d7d..397d4f2 100755
@@ -6,3 +6,12 @@ ReplaySubmode replay_get_play_submode(void)
 {
     return REPLAY_SUBMODE_UNKNOWN;
 }
+
+int64_t replay_save_clock(unsigned int kind, int64_t clock)
+{
+}
+
+int64_t replay_read_clock(unsigned int kind)
+{
+    return 0;
+}
#31 lore raw

Re: [Qemu-devel] [RFC PATCH v7 12/21] replay: recording and replaying clock ticks

Quoted 12 lines · Pavel Dovgalyuk
replay/replay-time.c new file +12 -0
+/*! Reads next clock event from the input. */
+int64_t replay_read_clock(unsigned int kind)
+{
+    if (kind >= REPLAY_CLOCK_COUNT) {
+        fprintf(stderr, "invalid clock ID %d for replay\n", kind);
+        exit(1);
+    }
+
+    replay_exec_instructions();
+
+    if (replay_file) {
+        if (skip_async_events(EVENT_CLOCK + kind)) {

replay/replay-time.c line 79

Quoted 10 lines
replay/replay-time.c new file +10 -0
+            replay_read_next_clock(kind);
+        }
+        int64_t ret = replay_state.cached_clock[kind];
+
+        return ret;
+    }
+
+    fprintf(stderr, "REPLAY INTERNAL ERROR %d\n", __LINE__);
+    exit(1);
+}

Is this thread safe? Please introduce the record/replay QemuMutex in patch 4, and all along the series document which variables it protects.

Paolo

Quoted 28 lines
replay/replay.h +13 -0
index 90a949b..1c18c0e 100755
@@ -16,6 +16,16 @@
 #include <stdint.h>
 #include "qapi-types.h"
 
+/* replay clock kinds */
+/* rdtsc */
+#define REPLAY_CLOCK_REAL_TICKS 0
+/* host_clock */
+#define REPLAY_CLOCK_HOST       1
+/* virtual_rt_clock */
+#define REPLAY_CLOCK_VIRTUAL_RT 2
+
+#define REPLAY_CLOCK_COUNT      3
+
 extern ReplayMode replay_mode;
 extern char *replay_image_suffix;
 
@@ -47,6 +57,19 @@ bool replay_interrupt(void);
     Returns true, when interrupt request is pending */
 bool replay_has_interrupt(void);
 
+/* Processing clocks and other time sources */
+
+/*! Save the specified clock */

replay/replay.h bool replay_interrupt(void); · line 72

Quoted 10 lines
replay/replay.h +10 -0
+int64_t replay_save_clock(unsigned int kind, int64_t clock);
+/*! Read the specified clock from the log or return cached data */
+int64_t replay_read_clock(unsigned int kind);
+/*! Saves or reads the clock depending on the current replay mode. */
+#define REPLAY_CLOCK(clock, value)                                      \
+    (replay_mode == REPLAY_MODE_PLAY ? replay_read_clock((clock))       \
+        : replay_mode == REPLAY_MODE_RECORD                             \
+            ? replay_save_clock((clock), (value))                       \
+        : (value))
+

Inline functions please, not macros.

#32 lore raw

Re: [Qemu-devel] [RFC PATCH v7 12/21] replay: recording and replaying clock ticks

Quoted 27 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 25 lines · Pavel Dovgalyuk
replay/replay.h +10 -0
index 90a949b..1c18c0e 100755
@@ -16,6 +16,16 @@
 #include <stdint.h>
 #include "qapi-types.h"
 
+/* replay clock kinds */
+/* rdtsc */
+#define REPLAY_CLOCK_REAL_TICKS 0
+/* host_clock */
+#define REPLAY_CLOCK_HOST       1
+/* virtual_rt_clock */
+#define REPLAY_CLOCK_VIRTUAL_RT 2
+
+#define REPLAY_CLOCK_COUNT      3
+
 extern ReplayMode replay_mode;
 extern char *replay_image_suffix;
 
@@ -47,6 +57,19 @@ bool replay_interrupt(void);
     Returns true, when interrupt request is pending */
 bool replay_has_interrupt(void);
 
Quoted 15 lines

replay/replay.h bool replay_interrupt(void); · line 72

Quoted 13 lines
replay/replay.h +13 -0
+/* Processing clocks and other time sources */
+
+/*! Save the specified clock */
+int64_t replay_save_clock(unsigned int kind, int64_t clock);
+/*! Read the specified clock from the log or return cached data */
+int64_t replay_read_clock(unsigned int kind);
+/*! Saves or reads the clock depending on the current replay mode. */
+#define REPLAY_CLOCK(clock, value)                                      \
+    (replay_mode == REPLAY_MODE_PLAY ? replay_read_clock((clock))       \
+        : replay_mode == REPLAY_MODE_RECORD                             \
+            ? replay_save_clock((clock), (value))                       \
+        : (value))
+

Inline functions please, not macros.

Macro is required here, because I do not want the "value" to be computed in replay mode at all.

Pavel Dovgalyuk

#33 lore raw

Re: [Qemu-devel] [RFC PATCH v7 12/21] replay: recording and replaying clock ticks

Quoted 10 lines · Pavel Dovgaluk
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

replay/replay.h bool replay_interrupt(void); · line 72

Quoted 6 lines
replay/replay.h +6 -0
+#define REPLAY_CLOCK(clock, value)                                      \
+    (replay_mode == REPLAY_MODE_PLAY ? replay_read_clock((clock))       \
+        : replay_mode == REPLAY_MODE_RECORD                             \
+            ? replay_save_clock((clock), (value))                       \
+        : (value))
+

Inline functions please, not macros.

Macro is required here, because I do not want the "value" to be computed in replay mode at all.

Indeed you're right.

Paolo

#34 lore raw

Re: [Qemu-devel] [RFC PATCH v7 12/21] replay: recording and replaying clock ticks

In reply to Paolo Bonzini
Quoted 18 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 16 lines · Pavel Dovgalyuk
replay/replay-time.c new file +16 -0
+/*! Reads next clock event from the input. */
+int64_t replay_read_clock(unsigned int kind)
+{
+    if (kind >= REPLAY_CLOCK_COUNT) {
+        fprintf(stderr, "invalid clock ID %d for replay\n", kind);
+        exit(1);
+    }
+
+    replay_exec_instructions();
+
+    if (replay_file) {
+        if (skip_async_events(EVENT_CLOCK + kind)) {
+            replay_read_next_clock(kind);
+        }
+        int64_t ret = replay_state.cached_clock[kind];
+
Quoted 8 lines

replay/replay-time.c line 79

Quoted 6 lines
replay/replay-time.c new file +6 -0
+        return ret;
+    }
+
+    fprintf(stderr, "REPLAY INTERNAL ERROR %d\n", __LINE__);
+    exit(1);
+}

Is this thread safe?

It is, because order of main_loop and cpu_exec executions is protected by global mutex.

Quoted 2 lines

Please introduce the record/replay QemuMutex in
patch 4, and all along the series document which variables it protects.

record/replay only uses mutex for events queue, because events may be added to the queue by the different threads.

Pavel Dovgalyuk

#35 lore raw

Re: [Qemu-devel] [RFC PATCH v7 12/21] replay: recording and replaying clock ticks

Quoted 24 lines · Pavel Dovgaluk
Quoted 16 lines
Quoted 16 lines

+/*! Reads next clock event from the input. */

Quoted 15 lines
Quoted 15 lines
replay/replay-time.c new file +15 -0
+int64_t replay_read_clock(unsigned int kind)
+{
+    if (kind >= REPLAY_CLOCK_COUNT) {
+        fprintf(stderr, "invalid clock ID %d for replay\n", kind);
+        exit(1);
+    }
+
+    replay_exec_instructions();
+
+    if (replay_file) {
+        if (skip_async_events(EVENT_CLOCK + kind)) {
+            replay_read_next_clock(kind);
+        }
+        int64_t ret = replay_state.cached_clock[kind];
+
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

replay/replay-time.c line 79

Quoted 6 lines
replay/replay-time.c new file +6 -0
+        return ret;
+    }
+
+    fprintf(stderr, "REPLAY INTERNAL ERROR %d\n", __LINE__);
+    exit(1);
+}

Is this thread safe?

Quoted 3 lines

It is, because order of main_loop and cpu_exec executions is protected by global mutex.

Please document exactly which globals are protected by the rr QemuMutex, and which by the global mutex. But I think as many variables as possible should be protected by the rr QemuMutex, for two reasons:

1) people are working on threaded TCG. While SMP record/replay is a whole different story, even UP TCG will be multithreaded.

2) in the end it makes reviewing the code simpler.

Paolo

#36 lore raw

Re: [Qemu-devel] [RFC PATCH v7 12/21] replay: recording and replaying clock ticks

Quoted 30 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 24 lines · Pavel Dovgaluk
Quoted 16 lines
Quoted 16 lines

+/*! Reads next clock event from the input. */

Quoted 15 lines
Quoted 15 lines
replay/replay-time.c new file +15 -0
+int64_t replay_read_clock(unsigned int kind)
+{
+    if (kind >= REPLAY_CLOCK_COUNT) {
+        fprintf(stderr, "invalid clock ID %d for replay\n", kind);
+        exit(1);
+    }
+
+    replay_exec_instructions();
+
+    if (replay_file) {
+        if (skip_async_events(EVENT_CLOCK + kind)) {
+            replay_read_next_clock(kind);
+        }
+        int64_t ret = replay_state.cached_clock[kind];
+
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

replay/replay-time.c line 79

Quoted 6 lines
replay/replay-time.c new file +6 -0
+        return ret;
+    }
+
+    fprintf(stderr, "REPLAY INTERNAL ERROR %d\n", __LINE__);
+    exit(1);
+}

Is this thread safe?

Quoted 3 lines

It is, because order of main_loop and cpu_exec executions is protected by global mutex.

Quoted 8 lines

Please document exactly which globals are protected by the rr QemuMutex, and which by the global mutex. But I think as many variables as possible should be protected by the rr QemuMutex, for two reasons:

1) people are working on threaded TCG. While SMP record/replay is a whole different story, even UP TCG will be multithreaded.

2) in the end it makes reviewing the code simpler.

There is one problem with protecting log file inside the replay code. We probably should have the following sequence of calls:

lock_replay_mutex
replay_save_events
replay_run_event
unlock_replay_mutex

But replay_run_event can also generate some log events and we will have deadlock here. That is why we rely on global mutex.

Pavel Dovgalyuk

#37 lore raw

Re: [Qemu-devel] [RFC PATCH v7 12/21] replay: recording and replaying clock ticks

Quoted 10 lines · Pavel Dovgaluk

There is one problem with protecting log file inside the replay code. We probably should have the following sequence of calls:

lock_replay_mutex
replay_save_events
replay_run_event
unlock_replay_mutex

But replay_run_event can also generate some log events and we will have deadlock here. That is why we rely on global mutex.

I think replay_run_event should run with the lock _not_ taken, that is:

    qemu_mutex_lock(&lock);
    while (!QTAILQ_EMPTY(&events_list)) {
        Event *event = QTAILQ_FIRST(&events_list);
        QTAILQ_REMOVE(&events_list, event, events);
        qemu_mutex_unlock(&lock);
        replay_run_event(event);
        g_free(event);
        qemu_mutex_lock(&lock);
    }
    qemu_mutex_unlock(&lock);

Paolo

#38 lore raw

Re: [Qemu-devel] [RFC PATCH v7 12/21] replay: recording and replaying clock ticks

Quoted 15 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 10 lines · Pavel Dovgaluk

There is one problem with protecting log file inside the replay code. We probably should have the following sequence of calls:

lock_replay_mutex
replay_save_events
replay_run_event
unlock_replay_mutex

But replay_run_event can also generate some log events and we will have deadlock here. That is why we rely on global mutex.

I think replay_run_event should run with the lock _not_ taken, that is:

Quoted 10 lines
    qemu_mutex_lock(&lock);
    while (!QTAILQ_EMPTY(&events_list)) {
        Event *event = QTAILQ_FIRST(&events_list);
        QTAILQ_REMOVE(&events_list, event, events);
        qemu_mutex_unlock(&lock);
        replay_run_event(event);
        g_free(event);
        qemu_mutex_lock(&lock);
    }
    qemu_mutex_unlock(&lock);

It will work for protecting the events list (I've already did this).
But that will not work for protecting the log file.
replay_run_event can write some data to the log. And also some other function like replay_checkpoint can also write to the log simultaneously (if we will remove the global mutex). That's why we cannot simply replace the global mutex with some kind of local one.

Pavel Dovgalyuk

#39 lore raw

Re: [Qemu-devel] [RFC PATCH v7 12/21] replay: recording and replaying clock ticks

Quoted 5 lines · Pavel Dovgaluk

It will work for protecting the events list (I've already did this).
But that will not work for protecting the log file.
replay_run_event can write some data to the log. And also some other function like replay_checkpoint can also write to the log simultaneously (if we will remove the global mutex). That's why we cannot simply replace the global mutex with some kind of local one.

That's why you have to document very well the architecture and the locking policy. For example, why can't replay_run_event (or something that it calls) take the replay lock locally, when it writes to the log?

If you do not document the architecture, we run in circles with me asking possibly stupid questions and you thinking that my questions are stupid. :)

Paolo

#40 lore raw

Re: [Qemu-devel] [RFC PATCH v7 12/21] replay: recording and replaying clock ticks

Quoted 11 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 3 lines · Pavel Dovgaluk

It will work for protecting the events list (I've already did this). But that will not work for protecting the log file.
replay_run_event can write some data to the log. And also some other function like

replay_checkpoint

Quoted 2 lines

can also write to the log simultaneously (if we will remove the global mutex). That's why we cannot simply replace the global mutex with some kind of local one.

That's why you have to document very well the architecture and the locking policy.

Ok.

Quoted 2 lines

For example, why can't replay_run_event (or something
that it calls) take the replay lock locally, when it writes to the log?

replay_run_event can take the lock. Suppose that it writes data 'A'. replay_run_event itself corresponds to some event 'E'.
We expect that the following sequence of the events should occur: 'E', 'A'. But if something will be written to the log between 'E' and 'A' then replay_run_event in replay mode will stuck, because it will not see its data 'A'.

Pavel Dovgalyuk

#41 lore raw

Re: [Qemu-devel] [RFC PATCH v7 12/21] replay: recording and replaying clock ticks

Quoted 8 lines · Pavel Dovgaluk
Quoted 2 lines

For example, why can't replay_run_event (or something
that it calls) take the replay lock locally, when it writes to the log?

replay_run_event can take the lock. Suppose that it writes data 'A'. replay_run_event itself corresponds to some event 'E'.
We expect that the following sequence of the events should occur: 'E', 'A'. But if something will be written to the log between 'E' and 'A' then replay_run_event in replay mode will stuck, because it will not see its data 'A'.

It would be easier if you pointed me to actual code in the series. But this doesn't seem impossible to fix by atomically writing the 'E' and 'A' in the same critical section.

This is for example how QMP events are thread-safe: it just adds the JSON string corresponding to the QMP event atomically in monitor_puts.

Paolo

#42 lore raw

Re: [Qemu-devel] [RFC PATCH v7 12/21] replay: recording and replaying clock ticks

Quoted 11 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 8 lines · Pavel Dovgaluk
Quoted 2 lines

For example, why can't replay_run_event (or something
that it calls) take the replay lock locally, when it writes to the log?

replay_run_event can take the lock. Suppose that it writes data 'A'. replay_run_event itself corresponds to some event 'E'.
We expect that the following sequence of the events should occur: 'E', 'A'. But if something will be written to the log between 'E' and 'A' then replay_run_event in replay mode will stuck, because it will not see its data 'A'.

Quoted 3 lines

It would be easier if you pointed me to actual code in the series. But this doesn't seem impossible to fix by atomically writing the 'E' and 'A' in the same critical section.

Because 'A' is written only inside some of the replay_run_event callbacks. It depends on type of the event and it's processing function inside the QEMU core. There could be no 'A' at all.

Pavel Dovgalyuk

#43 lore raw

Re: [Qemu-devel] [RFC PATCH v7 12/21] replay: recording and replaying clock ticks

Quoted 7 lines · Pavel Dovgaluk
Quoted 3 lines

It would be easier if you pointed me to actual code in the series. But this doesn't seem impossible to fix by atomically writing the 'E' and 'A' in the same critical section.

Because 'A' is written only inside some of the replay_run_event callbacks. It depends on type of the event and it's processing function inside the QEMU core. There could be no 'A' at all.

Why can't that code write the 'E' as well?

Paolo

#44 lore raw

Re: [Qemu-devel] [RFC PATCH v7 12/21] replay: recording and replaying clock ticks

Quoted 11 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 7 lines · Pavel Dovgaluk
Quoted 3 lines

It would be easier if you pointed me to actual code in the series. But this doesn't seem impossible to fix by atomically writing the 'E' and 'A' in the same critical section.

Because 'A' is written only inside some of the replay_run_event callbacks. It depends on type of the event and it's processing function inside the QEMU core. There could be no 'A' at all.

Why can't that code write the 'E' as well?

Because such callbacks do not know that they are called from record/replay event. They may be called from record/replay code and from other parts of QEMU. And they may write save something low-level like timer request.

Pavel Dovgalyuk

#45 lore raw

Re: [Qemu-devel] [RFC PATCH v7 12/21] replay: recording and replaying clock ticks

Quoted 9 lines · Pavel Dovgaluk
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

Because 'A' is written only inside some of the replay_run_event callbacks. It depends on type of the event and it's processing function inside the QEMU core. There could be no 'A' at all.

Why can't that code write the 'E' as well?

Because such callbacks do not know that they are called from record/replay event. They may be called from record/replay code and from other parts of QEMU. And they may write save something low-level like timer request.

I do not understand. Can you make an example with actual function names and data that is written?

Paolo

#46 lore raw

Re: [Qemu-devel] [RFC PATCH v7 12/21] replay: recording and replaying clock ticks

Quoted 7 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 5 lines · Pavel Dovgaluk
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

Because 'A' is written only inside some of the replay_run_event callbacks. It depends on type of the event and it's processing function inside the QEMU core. There could be no 'A' at all.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

Why can't that code write the 'E' as well?

Because such callbacks do not know that they are called from record/replay event. They may be called from record/replay code and from other parts of QEMU. And they may write save something low-level like timer request.

I do not understand. Can you make an example with actual function names and data that is written?

I haven't found any actual examples. Probably they occurred in previous version which wrote virtual timers into the log.
Anyway I will add the mutex then.

Pavel Dovgalyuk

#47 lore raw

RFC PATCH v7 13/21 replay: recording and replaying different timers

In reply to Pavel Dovgalyuk

This patch introduces functions for recording and replaying realtime sources, that do not use qemu-clock interface. These include return value of time() function in time_t and struct tm forms. Patch also adds warning to get_timedate function to prevent its usage in recording mode, because it may lead to non-determinism.

---
 hw/timer/mc146818rtc.c   |   10 ++++
 hw/timer/pl031.c         |   10 ++++
 include/qemu-common.h    |    1 
 replay/replay-internal.h |    4 ++
 replay/replay-time.c     |  112 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |    8 +++
 vl.c                     |   17 ++++++-
 7 files changed, 157 insertions(+), 5 deletions(-)
hw/timer/mc146818rtc.c +9 -1
index f18d128..7e27931 100644
@@ -28,6 +28,7 @@
 #include "qapi/visitor.h"
 #include "qapi-event.h"
 #include "qmp-commands.h"
+#include "replay/replay.h"
 
 #ifdef TARGET_I386
 #include "hw/i386/apic.h"
@@ -703,7 +704,14 @@ static void rtc_set_date_from_host(ISADevice *dev)
     RTCState *s = MC146818_RTC(dev);
     struct tm tm;
 
-    qemu_get_timedate(&tm, 0);
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        qemu_get_timedate_no_warning(&tm, 0);
+        replay_save_tm(&tm);
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        replay_read_tm(&tm);
+    } else {
+        qemu_get_timedate_no_warning(&tm, 0);
+    }
 
     s->base_rtc = mktimegm(&tm);
     s->last_update = qemu_clock_get_ns(rtc_clock);
hw/timer/pl031.c +9 -1
index 34d9b44..19264f2 100644
@@ -14,6 +14,7 @@
 #include "hw/sysbus.h"
 #include "qemu/timer.h"
 #include "sysemu/sysemu.h"
+#include "replay/replay.h"
 
 //#define DEBUG_PL031
 
@@ -200,7 +201,14 @@ static int pl031_init(SysBusDevice *dev)
     sysbus_init_mmio(dev, &s->iomem);
 
     sysbus_init_irq(dev, &s->irq);
-    qemu_get_timedate(&tm, 0);
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        qemu_get_timedate_no_warning(&tm, 0);
+        replay_save_tm(&tm);
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        replay_read_tm(&tm);
+    } else {
+        qemu_get_timedate_no_warning(&tm, 0);
+    }
     s->tick_offset = mktimegm(&tm) -
         qemu_clock_get_ns(rtc_clock) / get_ticks_per_sec();
 
include/qemu-common.h +1 -0
index f862214..e30adc5 100644
@@ -129,6 +129,7 @@ void dump_drift_info(FILE *f, fprintf_function cpu_fprintf);
 int qemu_main(int argc, char **argv, char **envp);
 #endif
 
+void qemu_get_timedate_no_warning(struct tm *tm, int offset);
 void qemu_get_timedate(struct tm *tm, int offset);
 int qemu_timedate_diff(struct tm *tm);
 
replay/replay-internal.h +4 -0
index c36d7de..009029d 100755
@@ -14,6 +14,10 @@
 
 #include <stdio.h>
 
+/* for time_t event */
+#define EVENT_TIME_T                1
+/* for tm event */
+#define EVENT_TM                    2
 /* for software interrupt */
 #define EVENT_INTERRUPT             15
 /* for emulated exceptions */
replay/replay-time.c +112 -0

115 lines not on this page · raw message

replay/replay.h +8 -0
index 1c18c0e..0c02e03 100755
@@ -14,6 +14,7 @@
 
 #include <stdbool.h>
 #include <stdint.h>
+#include <time.h>
 #include "qapi-types.h"
 
 /* replay clock kinds */
@@ -69,6 +70,13 @@ int64_t replay_read_clock(unsigned int kind);
         : replay_mode == REPLAY_MODE_RECORD                             \
             ? replay_save_clock((clock), (value))                       \
         : (value))
+/*! Returns result of time() function execution in normal and record modes.
+    In play mode returns value read from the log. */
+time_t replay_time(void);
+/*! Saves struct tm value to the log */
+void replay_save_tm(struct tm *tm);
+/*! Reads struct tm value from the log. Stops execution in case of error */
+void replay_read_tm(struct tm *tm);
 
 /* Asynchronous events queue */
 
vl.c +14 -3
index 7786b2f..f4be08c 100644
@@ -118,6 +118,7 @@ int main(int argc, char **argv)
 #include "qapi/opts-visitor.h"
 #include "qom/object_interfaces.h"
 #include "qapi-event.h"
+#include "replay/replay.h"
 
 #define DEFAULT_RAM_SIZE 128
 
@@ -715,7 +716,7 @@ void vm_start(void)
 
 /***********************************************************/
 /* host time/date access */
-void qemu_get_timedate(struct tm *tm, int offset)
+void qemu_get_timedate_no_warning(struct tm *tm, int offset)
 {
     time_t ti;
 
@@ -732,6 +733,16 @@ void qemu_get_timedate(struct tm *tm, int offset)
     }
 }
 
+/* host time/date access with replay warning */
+void qemu_get_timedate(struct tm *tm, int offset)
+{
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        fprintf(stderr, "REPLAY WARNING! qemu_get_timedate "
+                        "function may lead to non-determinism\n");
+    }
+    qemu_get_timedate_no_warning(tm, offset);
+}
+
 int qemu_timedate_diff(struct tm *tm)
 {
     time_t seconds;
@@ -747,7 +758,7 @@ int qemu_timedate_diff(struct tm *tm)
     else
         seconds = mktimegm(tm) + rtc_date_offset;
 
-    return seconds - time(NULL);
+    return seconds - replay_time();
 }
 
 static void configure_rtc_date_offset(const char *startdate, int legacy)
@@ -785,7 +796,7 @@ static void configure_rtc_date_offset(const char *startdate, int legacy)
                             "'2006-06-17T16:01:21' or '2006-06-17'\n");
             exit(1);
         }
-        rtc_date_offset = time(NULL) - rtc_start_date;
+        rtc_date_offset = replay_time() - rtc_start_date;
     }
 }
 
#48 lore raw

Re: [Qemu-devel] [RFC PATCH v7 13/21] replay: recording and replaying different timers

hw/timer/mc146818rtc.c static void rtc_set_date_from_host(ISADevice *dev) · line 714

Quoted 9 lines · Pavel Dovgalyuk
hw/timer/mc146818rtc.c +8 -0
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        qemu_get_timedate_no_warning(&tm, 0);
+        replay_save_tm(&tm);
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        replay_read_tm(&tm);
+    } else {
+        qemu_get_timedate_no_warning(&tm, 0);
+    }

Can this be encapsulated in a simple function replay_get_timedate? I think the fewest replay_mode checks you have outside replay/, the better.

Paolo

#49 lore raw

RFC PATCH v7 14/21 replay: shutdown event

In reply to Pavel Dovgalyuk

This patch records and replays simulator shutdown event.

---
 include/sysemu/sysemu.h  |    1 +
 replay/replay-internal.h |    2 ++
 replay/replay.c          |   11 +++++++++++
 replay/replay.h          |    5 +++++
 vl.c                     |    8 +++++++-
 5 files changed, 26 insertions(+), 1 deletions(-)
include/sysemu/sysemu.h +1 -0
index 748d059..3e166e4 100644
@@ -57,6 +57,7 @@ void qemu_register_suspend_notifier(Notifier *notifier);
 void qemu_system_wakeup_request(WakeupReason reason);
 void qemu_system_wakeup_enable(WakeupReason reason, bool enabled);
 void qemu_register_wakeup_notifier(Notifier *notifier);
+void qemu_system_shutdown_request_impl(void);
 void qemu_system_shutdown_request(void);
 void qemu_system_powerdown_request(void);
 void qemu_register_powerdown_notifier(Notifier *notifier);
replay/replay-internal.h +2 -0
index 009029d..ec6973d 100755
@@ -20,6 +20,8 @@
 #define EVENT_TM                    2
 /* for software interrupt */
 #define EVENT_INTERRUPT             15
+/* for shutdown request */
+#define EVENT_SHUTDOWN              20
 /* for emulated exceptions */
 #define EVENT_EXCEPTION             23
 /* for async events */
replay/replay.c +11 -0
index ec1fabc..7a0b7b1 100755
@@ -44,6 +44,10 @@ bool skip_async_events(int stop_event)
             res = true;
         }
         switch (replay_data_kind) {
+        case EVENT_SHUTDOWN:
+            replay_has_unread_data = 0;
+            qemu_system_shutdown_request_impl();
+            break;
         case EVENT_INSTRUCTION:
             replay_state.instructions_count = replay_get_dword();
             return res;
@@ -147,3 +151,10 @@ bool replay_has_interrupt(void)
     }
     return false;
 }
+
+void replay_shutdown_request(void)
+{
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_put_event(EVENT_SHUTDOWN);
+    }
+}
replay/replay.h +5 -0
index 0c02e03..00c9906 100755
@@ -78,6 +78,11 @@ void replay_save_tm(struct tm *tm);
 /*! Reads struct tm value from the log. Stops execution in case of error */
 void replay_read_tm(struct tm *tm);
 
+/* Events */
+
+/*! Called when qemu shutdown is requested. */
+void replay_shutdown_request(void);
+
 /* Asynchronous events queue */
 
 /*! Disables storing events in the queue */
vl.c +7 -1
index f4be08c..e37a016 100644
@@ -1713,13 +1713,19 @@ void qemu_system_killed(int signal, pid_t pid)
     qemu_system_shutdown_request();
 }
 
-void qemu_system_shutdown_request(void)
+void qemu_system_shutdown_request_impl(void)
 {
     trace_qemu_system_shutdown_request();
     shutdown_requested = 1;
     qemu_notify_event();
 }
 
+void qemu_system_shutdown_request(void)
+{
+    replay_shutdown_request();
+    qemu_system_shutdown_request_impl();
+}
+
 static void qemu_system_powerdown(void)
 {
     qapi_event_send_powerdown(&error_abort);
#50 lore raw

Re: [Qemu-devel] [RFC PATCH v7 14/21] replay: shutdown event

vl.c void qemu_system_killed(int signal, pid_t pid) · line 1728

Quoted 13 lines · Pavel Dovgalyuk
vl.c +7 -0
+void qemu_system_shutdown_request_impl(void)
 {
     trace_qemu_system_shutdown_request();
     shutdown_requested = 1;
     qemu_notify_event();
 }
 
+void qemu_system_shutdown_request(void)
+{
+    replay_shutdown_request();
+    qemu_system_shutdown_request_impl();
+}
+

No need to have a separate function.

Paolo

#51 lore raw

RFC PATCH v7 15/21 replay: checkpoints

In reply to Pavel Dovgalyuk

This patch introduces checkpoints that synchronize cpu thread and iothread. When checkpoint is met in the code all asynchronous events from the queue are executed.

---
 block.c                  |   11 +++++++++++
 cpus.c                   |    7 ++++++-
 include/qemu/timer.h     |    6 ++++--
 main-loop.c              |    5 +++++
 qemu-timer.c             |   46 ++++++++++++++++++++++++++++++++++++++--------
 replay/replay-internal.h |    3 +++
 replay/replay.c          |   28 ++++++++++++++++++++++++++++
 replay/replay.h          |    6 ++++++
 stubs/replay.c           |   11 +++++++++++
 vl.c                     |    3 ++-
 10 files changed, 114 insertions(+), 12 deletions(-)
block.c +11 -0
index 4165d42..35f389d 100644
@@ -1966,6 +1966,11 @@ void bdrv_drain_all(void)
     BlockDriverState *bs;
 
     while (busy) {
+        if (!replay_checkpoint(8)) {
+            /* Do not wait anymore, we stopped at some place in
+               the middle of execution during replay */
+            return;
+        }
         busy = false;
 
         QTAILQ_FOREACH(bs, &bdrv_states, device_list) {
@@ -1976,6 +1981,12 @@ void bdrv_drain_all(void)
             aio_context_release(aio_context);
         }
     }
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        /* Skip checkpoints from the log */
+        while (replay_checkpoint(8)) {
+            /* Nothing */
+        }
+    }
 }
 
 /* make a BlockDriverState anonymous by removing from bdrv_state and
cpus.c +6 -1
index 01d89aa..907a4bb 100644
@@ -388,7 +388,7 @@ void qtest_clock_warp(int64_t dest)
         timers_state.qemu_icount_bias += warp;
         seqlock_write_unlock(&timers_state.vm_clock_seqlock);
 
-        qemu_clock_run_timers(QEMU_CLOCK_VIRTUAL);
+        qemu_clock_run_timers(QEMU_CLOCK_VIRTUAL, false);
         clock = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
     }
     qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
@@ -408,6 +408,11 @@ void qemu_clock_warp(QEMUClockType type)
         return;
     }
 
+    /* warp clock deterministically in record/replay mode */
+    if (!replay_checkpoint(4)) {
+        return;
+    }
+
     /*
      * If the CPUs have been sleeping, advance QEMU_CLOCK_VIRTUAL timer now.
      * This ensures that the deadline for the timer is computed correctly below.
include/qemu/timer.h +4 -2
index a8bc9eb..19a79b8 100644
@@ -240,13 +240,14 @@ void qemu_clock_unregister_reset_notifier(QEMUClockType type,
 /**
  * qemu_clock_run_timers:
  * @type: clock on which to operate
+ * @run_all: true, when called from qemu_clock_run_all_timers
  *
  * Run all the timers associated with the default timer list
  * of a clock.
  *
  * Returns: true if any timer ran.
  */
-bool qemu_clock_run_timers(QEMUClockType type);
+bool qemu_clock_run_timers(QEMUClockType type, bool run_all);
 
 /**
  * qemu_clock_run_all_timers:
@@ -337,12 +338,13 @@ QEMUClockType timerlist_get_clock(QEMUTimerList *timer_list);
 /**
  * timerlist_run_timers:
  * @timer_list: the timer list to use
+ * @run_all: true, when called from qemu_clock_run_all_timers
  *
  * Call all expired timers associated with the timer list.
  *
  * Returns: true if any timer expired
  */
-bool timerlist_run_timers(QEMUTimerList *timer_list);
+bool timerlist_run_timers(QEMUTimerList *timer_list, bool run_all);
 
 /**
  * timerlist_notify:
main-loop.c +5 -0
index 981bcb5..d6e93c3 100644
@@ -497,6 +497,11 @@ int main_loop_wait(int nonblocking)
     slirp_pollfds_poll(gpollfds, (ret < 0));
 #endif
 
+    /* CPU thread can infinitely wait for event after
+       missing the warp */
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
+    }
     qemu_clock_run_all_timers();
 
     return ret;
qemu-timer.c +38 -8

85 lines not on this page · raw message

replay/replay-internal.h +3 -0
index ec6973d..5dad566 100755
@@ -32,6 +32,9 @@
 #define EVENT_CLOCK                 64
 /* some of grteater codes are reserved for clocks */
 
+/* for checkpoint event */
+#define EVENT_CHECKPOINT            96
+
 /* Asynchronous events IDs */
 
 #define REPLAY_ASYNC_COUNT             0
replay/replay.c +28 -0
index 7a0b7b1..27176c3 100755
@@ -158,3 +158,31 @@ void replay_shutdown_request(void)
         replay_put_event(EVENT_SHUTDOWN);
     }
 }
+
+/* Used checkpoints: 2 3 4 5 6 7 8 9 10 11 */
+int replay_checkpoint(unsigned int checkpoint)
+{
+    replay_save_instructions();
+
+    if (replay_file) {
+        if (replay_mode == REPLAY_MODE_PLAY) {
+            if (!skip_async_events(EVENT_CHECKPOINT + checkpoint)) {
+                if (replay_data_kind == EVENT_ASYNC_OPT) {
+                    replay_read_events(checkpoint);
+                    replay_fetch_data_kind();
+                    return replay_data_kind != EVENT_ASYNC_OPT;
+                }
+                return 0;
+            }
+            replay_has_unread_data = 0;
+            replay_read_events(checkpoint);
+            replay_fetch_data_kind();
+            return replay_data_kind != EVENT_ASYNC_OPT;
+        } else if (replay_mode == REPLAY_MODE_RECORD) {
+            replay_put_event(EVENT_CHECKPOINT + checkpoint);
+            replay_save_events(checkpoint);
+        }
+    }
+
+    return 1;
+}
replay/replay.h +6 -0
index 00c9906..6961751 100755
@@ -82,6 +82,12 @@ void replay_read_tm(struct tm *tm);
 
 /*! Called when qemu shutdown is requested. */
 void replay_shutdown_request(void);
+/*! Should be called at check points in the execution.
+    These check points are skipped, if they were not met.
+    Saves checkpoint in the SAVE mode and validates in the PLAY mode.
+    Returns 0 in PLAY mode if checkpoint was not found.
+    Returns 1 in all other cases. */
+int replay_checkpoint(unsigned int checkpoint);
 
 /* Asynchronous events queue */
 
stubs/replay.c +11 -0
index 397d4f2..3bb2d90 100755
@@ -1,4 +1,5 @@
 #include "replay/replay.h"
+#include "sysemu/sysemu.h"
 
 ReplayMode replay_mode;
 
@@ -15,3 +16,13 @@ int64_t replay_read_clock(unsigned int kind)
 {
     return 0;
 }
+
+int replay_checkpoint(unsigned int checkpoint)
+{
+    return 0;
+}
+
+int runstate_is_running(void)
+{
+    return 0;
+}
vl.c +2 -1
index e37a016..663e4db 100644
@@ -1768,7 +1768,8 @@ static bool main_loop_should_exit(void)
             return true;
         }
     }
-    if (qemu_reset_requested()) {
+    if (qemu_reset_requested_get() && replay_checkpoint(7)) {
+        qemu_reset_requested();
         pause_all_vcpus();
         cpu_synchronize_all_states();
         qemu_system_reset(VMRESET_REPORT);
#52 lore raw

Re: [Qemu-devel] [RFC PATCH v7 15/21] replay: checkpoints

qemu-timer.c bool timerlist_run_timers(QEMUTimerList *timer_list) · line 485

Quoted 7 lines · Pavel Dovgalyuk
qemu-timer.c +7 -0
+    default:
+    case QEMU_CLOCK_VIRTUAL:
+        if ((replay_mode != REPLAY_MODE_NONE && !runstate_is_running())
+            || !replay_checkpoint(run_all ? 2 : 3)) {
+            return false;
+        }
+        break;

Please document the meaning of the numbers by making an enum. Why do you have to distinguish run_all?

Paolo

#53 lore raw

Re: [Qemu-devel] [RFC PATCH v7 15/21] replay: checkpoints

Quoted 11 lines

From: Paolo Bonzini [mailto:[email protected]]

qemu-timer.c bool timerlist_run_timers(QEMUTimerList *timer_list) · line 485

Quoted 7 lines · Pavel Dovgalyuk
qemu-timer.c +7 -0
+    default:
+    case QEMU_CLOCK_VIRTUAL:
+        if ((replay_mode != REPLAY_MODE_NONE && !runstate_is_running())
+            || !replay_checkpoint(run_all ? 2 : 3)) {
+            return false;
+        }
+        break;

Please document the meaning of the numbers by making an enum.

The numbers have no meaning. They just have to be distinct in different places.

Quoted 1 line

Why do you have to distinguish run_all?

It seems to be early versions artifact. I'll remove it.

Pavel Dovgalyuk

#54 lore raw

Re: [Qemu-devel] [RFC PATCH v7 15/21] replay: checkpoints

In reply to Paolo Bonzini
Quoted 11 lines

From: Pavel Dovgaluk [mailto:[email protected]]

Quoted 10 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 7 lines · Pavel Dovgalyuk
qemu-timer.c +7 -0
+    default:
+    case QEMU_CLOCK_VIRTUAL:
+        if ((replay_mode != REPLAY_MODE_NONE && !runstate_is_running())
+            || !replay_checkpoint(run_all ? 2 : 3)) {
+            return false;
+        }
+        break;
Quoted 7 lines
Quoted 1 line

Please document the meaning of the numbers by making an enum.

The numbers have no meaning. They just have to be distinct in different places.

Quoted 1 line

Why do you have to distinguish run_all?

It seems to be early versions artifact. I'll remove it.

Sorry, missed one thing.
run_all is used to distinguish timers processed in AIO by calling of timerlistgroup_run_timers function and in main loop by calling qemu_clock_run_all_timers.
We need to distinguish that to secure the sequence of the events.
It makes sense when we use checkpointing while recording the execution.

Pavel Dovgalyuk

#55 lore raw

Re: [Qemu-devel] [RFC PATCH v7 15/21] replay: checkpoints

Quoted 1 line · Pavel Dovgaluk

The numbers have no meaning. They just have to be distinct in different places.

This is easier to achieve if you give a name to each place.

Quoted 3 lines

Sorry, missed one thing.
run_all is used to distinguish timers processed in AIO by calling of timerlistgroup_run_timers function and in main loop by calling qemu_clock_run_all_timers.

Should you instead distinguish which TimerListGroup is being run? Then main loop would checkpoint once for every TimerListGroup (which means twice).

Quoted 2 lines

We need to distinguish that to secure the sequence of the events. It makes sense when we use checkpointing while recording the execution.

I see.

Paolo

#56 lore raw

Re: [Qemu-devel] [RFC PATCH v7 15/21] replay: checkpoints

Quoted 6 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 1 line · Pavel Dovgaluk

The numbers have no meaning. They just have to be distinct in different places.

This is easier to achieve if you give a name to each place.

Quoted 8 lines
Quoted 2 lines

Sorry, missed one thing.
run_all is used to distinguish timers processed in AIO by calling of

timerlistgroup_run_timers function

Quoted 1 line

and in main loop by calling qemu_clock_run_all_timers.

Should you instead distinguish which TimerListGroup is being run? Then main loop would checkpoint once for every TimerListGroup (which means twice).

Then I'll have to introduce some kind of deterministic ID for them and new asynchronous event to be saved instead of common checkpoints. Because these calls can be invoked in any order.
But I have no idea of how to make such a deterministic ID.

Pavel Dovgalyuk

A message that is not in the archive (sent off-list or not yet received): [email protected]

#58 lore raw

Re: [Qemu-devel] [RFC PATCH v7 15/21] replay: checkpoints

Quoted 7 lines · Pavel Dovgaluk
Quoted 3 lines
Quoted 3 lines

Should you instead distinguish which TimerListGroup is being run? Then main loop would checkpoint once for every TimerListGroup (which means twice).

Then I'll have to introduce some kind of deterministic ID for them and new asynchronous event to be saved instead of common checkpoints. Because these calls can be invoked in any order.
But I have no idea of how to make such a deterministic ID.

There is currently one TimerListGroup per iothread object, and two more.
You can start by not supporting iothreads, or you can give a
deterministic ID to iothreads.

Paolo

#59 lore raw

Re: [Qemu-devel] [RFC PATCH v7 15/21] replay: checkpoints

In reply to a message not in the archive
Quoted 7 lines · Pavel Dovgaluk
Quoted 3 lines
Quoted 3 lines

Should you instead distinguish which TimerListGroup is being run? Then main loop would checkpoint once for every TimerListGroup (which means twice).

Then I'll have to introduce some kind of deterministic ID for them and new asynchronous event to be saved instead of common checkpoints. Because these calls can be invoked in any order.
But I have no idea of how to make such a deterministic ID.

There is currently one TimerListGroup per iothread object, and two more.
You can start by not supporting iothreads, or you can give a
deterministic ID to iothreads. Thread pools are similar to TimerListGroups, except there is only one global pool instead of two.

Paolo

#60 lore raw

Re: [Qemu-devel] [RFC PATCH v7 15/21] replay: checkpoints

In reply to Paolo Bonzini
Quoted 6 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 1 line · Pavel Dovgaluk

The numbers have no meaning. They just have to be distinct in different places.

This is easier to achieve if you give a name to each place.

Quoted 8 lines
Quoted 2 lines

Sorry, missed one thing.
run_all is used to distinguish timers processed in AIO by calling of

timerlistgroup_run_timers function

Quoted 1 line

and in main loop by calling qemu_clock_run_all_timers.

Should you instead distinguish which TimerListGroup is being run? Then main loop would checkpoint once for every TimerListGroup (which means twice).

I checked that. We should distinguish only run_all_timers and all other calls. Distinguishing the TimerListGroups does not work - replay hangs.

Pavel Dovgalyuk

#61 lore raw

RFC PATCH v7 16/21 replay: bottom halves

In reply to Pavel Dovgalyuk

This patch introduces bottom half event for replay queue. It saves the events into the queue and process them at the checkpoints and instructions execution.

---
 async.c                  |   46 ++++++++++++++++++++++++++++++++++++++++------
 dma-helpers.c            |    4 +++-
 hw/ide/ahci.c            |    4 +++-
 hw/ide/core.c            |    4 +++-
 hw/timer/arm_timer.c     |    2 +-
 hw/usb/hcd-uhci.c        |    2 +-
 include/block/aio.h      |   18 ++++++++++++++++++
 include/qemu/main-loop.h |    1 +
 main-loop.c              |    5 +++++
 replay/replay-events.c   |   16 ++++++++++++++++
 replay/replay-internal.h |    3 ++-
 replay/replay.h          |    2 ++
 stubs/replay.c           |    4 ++++
 13 files changed, 99 insertions(+), 12 deletions(-)
async.c +40 -6
index 3939b79..bd618c6 100644
@@ -27,6 +27,7 @@
 #include "block/thread-pool.h"
 #include "qemu/main-loop.h"
 #include "qemu/atomic.h"
+#include "replay/replay.h"
 
 /***********************************************************/
 /* bottom halves (can be seen as timers which expire ASAP) */
@@ -39,24 +40,53 @@ struct QEMUBH {
     bool scheduled;
     bool idle;
     bool deleted;
+    bool replay;
+    uint64_t id;
 };
 
 QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque)
 {
-    QEMUBH *bh;
+    QEMUBH *bh, **last;
     bh = g_new0(QEMUBH, 1);
     bh->ctx = ctx;
     bh->cb = cb;
     bh->opaque = opaque;
     qemu_mutex_lock(&ctx->bh_lock);
-    bh->next = ctx->first_bh;
-    /* Make sure that the members are ready before putting bh into list */
-    smp_wmb();
-    ctx->first_bh = bh;
+    if (replay_mode != REPLAY_MODE_NONE) {
+        /* Slower way, but this is a queue and not a stack.
+           Replay will process the BH in the same order they
+           came into the queue. */
+        last = &ctx->first_bh;
+        while (*last) {
+            last = &(*last)->next;
+        }
+        smp_wmb();
+        *last = bh;
+    } else {
+        bh->next = ctx->first_bh;
+        /* Make sure that the members are ready before putting bh into list */
+        smp_wmb();
+        ctx->first_bh = bh;
+    }
     qemu_mutex_unlock(&ctx->bh_lock);
     return bh;
 }
 
+QEMUBH *aio_bh_new_replay(AioContext *ctx, QEMUBHFunc *cb, void *opaque,
+                          uint64_t id)
+{
+    QEMUBH *bh = aio_bh_new(ctx, cb, opaque);
+    bh->replay = true;
+    bh->id = id;
+    return bh;
+}
+
+void aio_bh_call(void *opaque)
+{
+    QEMUBH *bh = (QEMUBH *)opaque;
+    bh->cb(bh->opaque);
+}
+
 /* Multiple occurrences of aio_bh_poll cannot be called concurrently */
 int aio_bh_poll(AioContext *ctx)
 {
@@ -79,7 +109,11 @@ int aio_bh_poll(AioContext *ctx)
             if (!bh->idle)
                 ret = 1;
             bh->idle = 0;
-            bh->cb(bh->opaque);
+            if (!bh->replay) {
+                aio_bh_call(bh);
+            } else {
+                replay_add_bh_event(bh, bh->id);
+            }
         }
     }
 
dma-helpers.c +3 -1
index 6918572..357d7e9 100644
@@ -13,6 +13,7 @@
 #include "qemu/range.h"
 #include "qemu/thread.h"
 #include "qemu/main-loop.h"
+#include "replay/replay.h"
 
 /* #define DEBUG_IOMMU */
 
@@ -96,7 +97,8 @@ static void continue_after_map_failure(void *opaque)
 {
     DMAAIOCB *dbs = (DMAAIOCB *)opaque;
 
-    dbs->bh = qemu_bh_new(reschedule_dma, dbs);
+    dbs->bh = qemu_bh_new_replay(reschedule_dma, dbs,
+                                 replay_get_current_step());
     qemu_bh_schedule(dbs->bh);
 }
 
hw/ide/ahci.c +3 -1
index 5651372..13d7f84 100644
@@ -33,6 +33,7 @@
 #include "internal.h"
 #include <hw/ide/pci.h>
 #include <hw/ide/ahci.h>
+#include "replay/replay.h"
 
 #define DEBUG_AHCI 0
 
@@ -1243,7 +1244,8 @@ static void ahci_cmd_done(IDEDMA *dma)
 
     if (!ad->check_bh) {
         /* maybe we still have something to process, check later */
-        ad->check_bh = qemu_bh_new(ahci_check_cmd_bh, ad);
+        ad->check_bh = qemu_bh_new_replay(ahci_check_cmd_bh, ad,
+                                          replay_get_current_step());
         qemu_bh_schedule(ad->check_bh);
     }
 }
hw/ide/core.c +3 -1
index d4af5e2..d76244a 100644
@@ -32,6 +32,7 @@
 #include "sysemu/dma.h"
 #include "hw/block/block.h"
 #include "sysemu/block-backend.h"
+#include "replay/replay.h"
 
 #include <hw/ide/internal.h>
 
@@ -448,7 +449,8 @@ BlockAIOCB *ide_issue_trim(BlockBackend *blk,
 
     iocb = blk_aio_get(&trim_aiocb_info, blk, cb, opaque);
     iocb->blk = blk;
-    iocb->bh = qemu_bh_new(ide_trim_bh_cb, iocb);
+    iocb->bh = qemu_bh_new_replay(ide_trim_bh_cb, iocb,
+                                  replay_get_current_step());
     iocb->ret = 0;
     iocb->qiov = qiov;
     iocb->i = -1;
hw/timer/arm_timer.c +1 -1
index 1452910..97784a0 100644
@@ -168,7 +168,7 @@ static arm_timer_state *arm_timer_init(uint32_t freq)
     s->freq = freq;
     s->control = TIMER_CTRL_IE;
 
-    bh = qemu_bh_new(arm_timer_tick, s);
+    bh = qemu_bh_new_replay(arm_timer_tick, s, 0);
     s->timer = ptimer_init(bh);
     vmstate_register(NULL, -1, &vmstate_arm_timer, s);
     return s;
hw/usb/hcd-uhci.c +1 -1
index 4a4215d..0d94143 100644
@@ -1221,7 +1221,7 @@ static int usb_uhci_common_initfn(PCIDevice *dev)
                               USB_SPEED_MASK_LOW | USB_SPEED_MASK_FULL);
         }
     }
-    s->bh = qemu_bh_new(uhci_bh, s);
+    s->bh = qemu_bh_new_replay(uhci_bh, s, 0);
     s->frame_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, uhci_frame_timer, s);
     s->num_ports_vmstate = NB_PORTS;
     QTAILQ_INIT(&s->queues);
include/block/aio.h +18 -0
index 6bf0e04..5a77431 100644
@@ -35,6 +35,8 @@ struct BlockAIOCB {
     const AIOCBInfo *aiocb_info;
     BlockDriverState *bs;
     BlockCompletionFunc *cb;
+    bool replay;
+    uint64_t replay_step;
     void *opaque;
     int refcnt;
 };
@@ -144,6 +146,17 @@ void aio_context_release(AioContext *ctx);
 QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque);
 
 /**
+ * aio_bh_new_replay: Allocate a new bottom half structure for replay.
+ *
+ * This function calls aio_bh_new function and also fills replay parameters
+ * of the BH structure. BH created with this function in record/replay mode
+ * are executed through the replay queue only at checkpoints and instructions
+ * executions.
+ */
+QEMUBH *aio_bh_new_replay(AioContext *ctx, QEMUBHFunc *cb, void *opaque,
+                          uint64_t id);
+
+/**
  * aio_notify: Force processing of pending events.
  *
  * Similar to signaling a condition variable, aio_notify forces
@@ -159,6 +172,11 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque);
 void aio_notify(AioContext *ctx);
 
 /**
+ * aio_bh_call: Executes callback function of the specified BH.
+ */
+void aio_bh_call(void *opaque);
+
+/**
  * aio_bh_poll: Poll bottom halves for an AioContext.
  *
  * These are internal functions used by the QEMU main loop.
include/qemu/main-loop.h +1 -0
index 62c68c0..f5a98fe 100644
@@ -306,6 +306,7 @@ void qemu_iohandler_fill(GArray *pollfds);
 void qemu_iohandler_poll(GArray *pollfds, int rc);
 
 QEMUBH *qemu_bh_new(QEMUBHFunc *cb, void *opaque);
+QEMUBH *qemu_bh_new_replay(QEMUBHFunc *cb, void *opaque, uint64_t id);
 void qemu_bh_schedule_idle(QEMUBH *bh);
 
 #endif
main-loop.c +5 -0
index d6e93c3..df99bcf 100644
@@ -513,3 +513,8 @@ QEMUBH *qemu_bh_new(QEMUBHFunc *cb, void *opaque)
 {
     return aio_bh_new(qemu_aio_context, cb, opaque);
 }
+
+QEMUBH *qemu_bh_new_replay(QEMUBHFunc *cb, void *opaque, uint64_t id)
+{
+    return aio_bh_new_replay(qemu_aio_context, cb, opaque, id);
+}
replay/replay-events.c +16 -0
index f3c9b16..1aee0a4 100755
@@ -36,6 +36,9 @@ static bool replay_events_enabled = false;
 static void replay_run_event(Event *event)
 {
     switch (event->event_kind) {
+    case REPLAY_ASYNC_EVENT_BH:
+        aio_bh_call(event->opaque);
+        break;
     default:
         fprintf(stderr, "Replay: invalid async event ID (%d) in the queue\n",
                 event->event_kind);
@@ -119,6 +122,11 @@ void replay_add_event(int event_kind, void *opaque)
     replay_add_event_internal(event_kind, opaque, NULL, 0);
 }
 
+void replay_add_bh_event(void *bh, uint64_t id)
+{
+    replay_add_event_internal(REPLAY_ASYNC_EVENT_BH, bh, NULL, id);
+}
+
 void replay_save_events(int opt)
 {
     qemu_mutex_lock(&lock);
@@ -132,6 +140,9 @@ void replay_save_events(int opt)
 
             /* save event-specific data */
             switch (event->event_kind) {
+            case REPLAY_ASYNC_EVENT_BH:
+                replay_put_qword(event->id);
+                break;
             }
         }
 
@@ -158,6 +169,11 @@ void replay_read_events(int opt)
         }
         /* Execute some events without searching them in the queue */
         switch (read_event_kind) {
+        case REPLAY_ASYNC_EVENT_BH:
+            if (read_id == -1) {
+                read_id = replay_get_qword();
+            }
+            break;
         default:
             fprintf(stderr, "Unknown ID %d of replay event\n", read_event_kind);
             exit(1);
replay/replay-internal.h +2 -1
index 5dad566..6e0c2e9 100755
@@ -37,7 +37,8 @@
 
 /* Asynchronous events IDs */
 
-#define REPLAY_ASYNC_COUNT             0
+#define REPLAY_ASYNC_EVENT_BH          0
+#define REPLAY_ASYNC_COUNT             1
 
 typedef struct ReplayState {
     /*! Cached clock values. */
replay/replay.h +2 -0
index 6961751..d3f168d 100755
@@ -93,5 +93,7 @@ int replay_checkpoint(unsigned int checkpoint);
 
 /*! Disables storing events in the queue */
 void replay_disable_events(void);
+/*! Adds BH event to the queue */
+void replay_add_bh_event(void *bh, uint64_t id);
 
 #endif
stubs/replay.c +4 -0
index 3bb2d90..448cab0 100755
@@ -26,3 +26,7 @@ int runstate_is_running(void)
 {
     return 0;
 }
+
+void replay_add_bh_event(void *bh, uint64_t id)
+{
+}
#62 lore raw

Re: [Qemu-devel] [RFC PATCH v7 16/21] replay: bottom halves

Quoted 10 lines · Pavel Dovgalyuk
async.c +6 -4
-    bh->next = ctx->first_bh;
-    /* Make sure that the members are ready before putting bh into list */
-    smp_wmb();
-    ctx->first_bh = bh;
+    if (replay_mode != REPLAY_MODE_NONE) {
+        /* Slower way, but this is a queue and not a stack.
+           Replay will process the BH in the same order they
+           came into the queue. */
+        last = &ctx->first_bh;
+        while (*last) {

async.c struct QEMUBH { · line 70

Quoted 10 lines
async.c +10 -0
+            last = &(*last)->next;
+        }
+        smp_wmb();
+        *last = bh;
+    } else {
+        bh->next = ctx->first_bh;
+        /* Make sure that the members are ready before putting bh into list */
+        smp_wmb();
+        ctx->first_bh = bh;
+    }

I am not sure I understand the reason for this, but feel free to change the list to a QSIMPLEQ, and introduce QSIMPLEQ_INSERT_TAIL_RCU that has the appropriate barriers.

Paolo

#63 lore raw

RFC PATCH v7 17/21 replay: replay aio requests

In reply to Pavel Dovgalyuk

This patch adds identifier to aio requests. ID is used for creating bottom halves and identifying them while replaying.
The patch also introduces several functions that make possible replaying of the aio requests.

---
 block.c                        |   81 ++++++++++++++++++++++++++++++++++++----
 block/block-backend.c          |   30 ++++++++++++++-
 block/qcow2.c                  |    4 ++
 dma-helpers.c                  |    6 ++-
 hw/block/virtio-blk.c          |   10 ++---
 hw/ide/atapi.c                 |   10 +++--
 hw/ide/core.c                  |   14 ++++---
 include/block/block.h          |   15 +++++++
 include/qemu-common.h          |    2 +
 include/sysemu/block-backend.h |   10 +++++
 qemu-io-cmds.c                 |    2 -
 stubs/replay.c                 |    5 ++
 trace-events                   |    2 +
 util/iov.c                     |    4 ++
 14 files changed, 167 insertions(+), 28 deletions(-)
block.c +72 -9
index 35f389d..e4cad41 100644
@@ -83,7 +83,8 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
                                          BdrvRequestFlags flags,
                                          BlockCompletionFunc *cb,
                                          void *opaque,
-                                         bool is_write);
+                                         bool is_write,
+                                         bool aio_replay);
 static void coroutine_fn bdrv_co_do_rw(void *opaque);
 static int coroutine_fn bdrv_co_do_write_zeroes(BlockDriverState *bs,
     int64_t sector_num, int nb_sectors, BdrvRequestFlags flags);
@@ -4396,7 +4397,19 @@ BlockAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,
     trace_bdrv_aio_readv(bs, sector_num, nb_sectors, opaque);
 
     return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
-                                 cb, opaque, false);
+                                 cb, opaque, false, false);
+}
+
+BlockAIOCB *bdrv_aio_readv_replay(BlockDriverState *bs,
+                                  int64_t sector_num,
+                                  QEMUIOVector *qiov, int nb_sectors,
+                                  BlockCompletionFunc *cb,
+                                  void *opaque)
+{
+    trace_bdrv_aio_readv_replay(bs, sector_num, nb_sectors, opaque);
+
+    return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
+                                 cb, opaque, false, true);
 }
 
 BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
@@ -4406,7 +4419,19 @@ BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
     trace_bdrv_aio_writev(bs, sector_num, nb_sectors, opaque);
 
     return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
-                                 cb, opaque, true);
+                                 cb, opaque, true, false);
+}
+
+BlockAIOCB *bdrv_aio_writev_replay(BlockDriverState *bs,
+                                   int64_t sector_num,
+                                   QEMUIOVector *qiov, int nb_sectors,
+                                   BlockCompletionFunc *cb,
+                                   void *opaque)
+{
+    trace_bdrv_aio_writev_replay(bs, sector_num, nb_sectors, opaque);
+
+    return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
+                                 cb, opaque, true, true);
 }
 
 BlockAIOCB *bdrv_aio_write_zeroes(BlockDriverState *bs,
@@ -4417,7 +4442,7 @@ BlockAIOCB *bdrv_aio_write_zeroes(BlockDriverState *bs,
 
     return bdrv_co_aio_rw_vector(bs, sector_num, NULL, nb_sectors,
                                  BDRV_REQ_ZERO_WRITE | flags,
-                                 cb, opaque, true);
+                                 cb, opaque, true, true);
 }
 
 
@@ -4564,7 +4589,8 @@ static int multiwrite_merge(BlockDriverState *bs, BlockRequest *reqs,
  * requests. However, the fields opaque and error are left unmodified as they
  * are used to signal failure for a single request to the caller.
  */
-int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs)
+int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs,
+                        bool replay)
 {
     MultiwriteCB *mcb;
     int i;
@@ -4602,7 +4628,7 @@ int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs)
         bdrv_co_aio_rw_vector(bs, reqs[i].sector, reqs[i].qiov,
                               reqs[i].nb_sectors, reqs[i].flags,
                               multiwrite_cb, mcb,
-                              true);
+                              true, replay);
     }
 
     return 0;
@@ -4747,7 +4773,12 @@ static void coroutine_fn bdrv_co_do_rw(void *opaque)
             acb->req.nb_sectors, acb->req.qiov, acb->req.flags);
     }
 
-    acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    if (acb->common.replay) {
+        acb->bh = aio_bh_new_replay(bdrv_get_aio_context(bs), bdrv_co_em_bh,
+                                    acb, acb->common.replay_step);
+    } else {
+        acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    }
     qemu_bh_schedule(acb->bh);
 }
 
@@ -4758,7 +4789,8 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
                                          BdrvRequestFlags flags,
                                          BlockCompletionFunc *cb,
                                          void *opaque,
-                                         bool is_write)
+                                         bool is_write,
+                                         bool aio_replay)
 {
     Coroutine *co;
     BlockAIOCBCoroutine *acb;
@@ -4769,6 +4801,11 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
     acb->req.qiov = qiov;
     acb->req.flags = flags;
     acb->is_write = is_write;
+    acb->done = NULL;
+    acb->common.replay = aio_replay;
+    if (aio_replay) {
+        acb->common.replay_step = replay_get_current_step();
+    }
 
     co = qemu_coroutine_create(bdrv_co_do_rw);
     qemu_coroutine_enter(co, acb);
@@ -4782,7 +4819,12 @@ static void coroutine_fn bdrv_aio_flush_co_entry(void *opaque)
     BlockDriverState *bs = acb->common.bs;
 
     acb->req.error = bdrv_co_flush(bs);
-    acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    if (acb->common.replay) {
+        acb->bh = aio_bh_new_replay(bdrv_get_aio_context(bs), bdrv_co_em_bh,
+                                    acb, acb->common.replay_step);
+    } else {
+        acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    }
     qemu_bh_schedule(acb->bh);
 }
 
@@ -4802,6 +4844,25 @@ BlockAIOCB *bdrv_aio_flush(BlockDriverState *bs,
     return &acb->common;
 }
 
+BlockAIOCB *bdrv_aio_flush_replay(BlockDriverState *bs,
+        BlockCompletionFunc *cb, void *opaque)
+{
+    trace_bdrv_aio_flush(bs, opaque);
+
+    Coroutine *co;
+    BlockAIOCBCoroutine *acb;
+
+    acb = qemu_aio_get(&bdrv_em_co_aiocb_info, bs, cb, opaque);
+    acb->done = NULL;
+    acb->common.replay = true;
+    acb->common.replay_step = replay_get_current_step();
+
+    co = qemu_coroutine_create(bdrv_aio_flush_co_entry);
+    qemu_coroutine_enter(co, acb);
+
+    return &acb->common;
+}
+
 static void coroutine_fn bdrv_aio_discard_co_entry(void *opaque)
 {
     BlockAIOCBCoroutine *acb = opaque;
@@ -4852,6 +4913,8 @@ void *qemu_aio_get(const AIOCBInfo *aiocb_info, BlockDriverState *bs,
     acb->cb = cb;
     acb->opaque = opaque;
     acb->refcnt = 1;
+    acb->replay_step = 0;
+    acb->replay = false;
     return acb;
 }
 
block/block-backend.c +29 -1
index ef16d73..6e3ef46 100644
@@ -451,6 +451,14 @@ BlockAIOCB *blk_aio_readv(BlockBackend *blk, int64_t sector_num,
     return bdrv_aio_readv(blk->bs, sector_num, iov, nb_sectors, cb, opaque);
 }
 
+BlockAIOCB *blk_aio_readv_replay(BlockBackend *blk, int64_t sector_num,
+                                 QEMUIOVector *iov, int nb_sectors,
+                                 BlockCompletionFunc *cb, void *opaque)
+{
+    return bdrv_aio_readv_replay(blk->bs, sector_num, iov, nb_sectors,
+                                 cb, opaque);
+}
+
 BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,
                            QEMUIOVector *iov, int nb_sectors,
                            BlockCompletionFunc *cb, void *opaque)
@@ -458,12 +466,26 @@ BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,
     return bdrv_aio_writev(blk->bs, sector_num, iov, nb_sectors, cb, opaque);
 }
 
+BlockAIOCB *blk_aio_writev_replay(BlockBackend *blk, int64_t sector_num,
+                                  QEMUIOVector *iov, int nb_sectors,
+                                  BlockCompletionFunc *cb, void *opaque)
+{
+    return bdrv_aio_writev_replay(blk->bs, sector_num, iov, nb_sectors,
+                                  cb, opaque);
+}
+
 BlockAIOCB *blk_aio_flush(BlockBackend *blk,
                           BlockCompletionFunc *cb, void *opaque)
 {
     return bdrv_aio_flush(blk->bs, cb, opaque);
 }
 
+BlockAIOCB *blk_aio_flush_replay(BlockBackend *blk,
+                                 BlockCompletionFunc *cb, void *opaque)
+{
+    return bdrv_aio_flush_replay(blk->bs, cb, opaque);
+}
+
 BlockAIOCB *blk_aio_discard(BlockBackend *blk,
                             int64_t sector_num, int nb_sectors,
                             BlockCompletionFunc *cb, void *opaque)
@@ -483,7 +505,13 @@ void blk_aio_cancel_async(BlockAIOCB *acb)
 
 int blk_aio_multiwrite(BlockBackend *blk, BlockRequest *reqs, int num_reqs)
 {
-    return bdrv_aio_multiwrite(blk->bs, reqs, num_reqs);
+    return bdrv_aio_multiwrite(blk->bs, reqs, num_reqs, false);
+}
+
+int blk_aio_multiwrite_replay(BlockBackend *blk, BlockRequest *reqs,
+                              int num_reqs)
+{
+    return bdrv_aio_multiwrite(blk->bs, reqs, num_reqs, true);
 }
 
 int blk_ioctl(BlockBackend *blk, unsigned long int req, void *buf)
block/qcow2.c +4 -0
index e4e690a..4a2f6ae 100644
@@ -1140,6 +1140,8 @@ static coroutine_fn int qcow2_co_readv(BlockDriverState *bs, int64_t sector_num,
     uint8_t *cluster_data = NULL;
 
     qemu_iovec_init(&hd_qiov, qiov->niov);
+    hd_qiov.replay = qiov->replay;
+    hd_qiov.replay_step = qiov->replay_step;
 
     qemu_co_mutex_lock(&s->lock);
 
@@ -1297,6 +1299,8 @@ static coroutine_fn int qcow2_co_writev(BlockDriverState *bs,
                                  remaining_sectors);
 
     qemu_iovec_init(&hd_qiov, qiov->niov);
+    hd_qiov.replay = qiov->replay;
+    hd_qiov.replay_step = qiov->replay_step;
 
     s->cluster_cache_offset = -1; /* disable compressed cache */
 
dma-helpers.c +4 -2
index 357d7e9..4faf6d3 100644
@@ -212,6 +212,8 @@ BlockAIOCB *dma_blk_io(
     dbs->io_func = io_func;
     dbs->bh = NULL;
     qemu_iovec_init(&dbs->iov, sg->nsg);
+    dbs->iov.replay = true;
+    dbs->iov.replay_step = replay_get_current_step();
     dma_blk_cb(dbs, 0);
     return &dbs->common;
 }
@@ -221,7 +223,7 @@ BlockAIOCB *dma_blk_read(BlockBackend *blk,
                          QEMUSGList *sg, uint64_t sector,
                          void (*cb)(void *opaque, int ret), void *opaque)
 {
-    return dma_blk_io(blk, sg, sector, blk_aio_readv, cb, opaque,
+    return dma_blk_io(blk, sg, sector, blk_aio_readv_replay, cb, opaque,
                       DMA_DIRECTION_FROM_DEVICE);
 }
 
@@ -229,7 +231,7 @@ BlockAIOCB *dma_blk_write(BlockBackend *blk,
                           QEMUSGList *sg, uint64_t sector,
                           void (*cb)(void *opaque, int ret), void *opaque)
 {
-    return dma_blk_io(blk, sg, sector, blk_aio_writev, cb, opaque,
+    return dma_blk_io(blk, sg, sector, blk_aio_writev_replay, cb, opaque,
                       DMA_DIRECTION_TO_DEVICE);
 }
 
hw/block/virtio-blk.c +5 -5
index b19b102..2413cbe 100644
@@ -265,7 +265,7 @@ void virtio_submit_multiwrite(BlockBackend *blk, MultiReqBuffer *mrb)
         return;
     }
 
-    ret = blk_aio_multiwrite(blk, mrb->blkreq, mrb->num_writes);
+    ret = blk_aio_multiwrite_replay(blk, mrb->blkreq, mrb->num_writes);
     if (ret != 0) {
         for (i = 0; i < mrb->num_writes; i++) {
             if (mrb->blkreq[i].error) {
@@ -286,7 +286,7 @@ static void virtio_blk_handle_flush(VirtIOBlockReq *req, MultiReqBuffer *mrb)
      * Make sure all outstanding writes are posted to the backing device.
      */
     virtio_submit_multiwrite(req->dev->blk, mrb);
-    blk_aio_flush(req->dev->blk, virtio_blk_flush_complete, req);
+    blk_aio_flush_replay(req->dev->blk, virtio_blk_flush_complete, req);
 }
 
 static bool virtio_blk_sect_range_ok(VirtIOBlock *dev,
@@ -357,9 +357,9 @@ static void virtio_blk_handle_read(VirtIOBlockReq *req)
 
     block_acct_start(blk_get_stats(req->dev->blk), &req->acct, req->qiov.size,
                      BLOCK_ACCT_READ);
-    blk_aio_readv(req->dev->blk, sector, &req->qiov,
-                  req->qiov.size / BDRV_SECTOR_SIZE,
-                  virtio_blk_rw_complete, req);
+    blk_aio_readv_replay(req->dev->blk, sector, &req->qiov,
+                          req->qiov.size / BDRV_SECTOR_SIZE,
+                          virtio_blk_rw_complete, req);
 }
 
 void virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb)
hw/ide/atapi.c +6 -4
index c63b7e5..02b11df 100644
@@ -25,7 +25,7 @@
 
 #include "hw/ide/internal.h"
 #include "hw/scsi/scsi.h"
-#include "sysemu/block-backend.h"
+#include "replay/replay.h"
 
 static void ide_atapi_cmd_read_dma_cb(void *opaque, int ret);
 
@@ -350,10 +350,12 @@ static void ide_atapi_cmd_read_dma_cb(void *opaque, int ret)
     s->bus->dma->iov.iov_base = (void *)(s->io_buffer + data_offset);
     s->bus->dma->iov.iov_len = n * 4 * 512;
     qemu_iovec_init_external(&s->bus->dma->qiov, &s->bus->dma->iov, 1);
+    s->bus->dma->qiov.replay = true;
+    s->bus->dma->qiov.replay_step = replay_get_current_step();
 
-    s->bus->dma->aiocb = blk_aio_readv(s->blk, (int64_t)s->lba << 2,
-                                       &s->bus->dma->qiov, n * 4,
-                                       ide_atapi_cmd_read_dma_cb, s);
+    s->bus->dma->aiocb = blk_aio_readv_replay(s->blk, (int64_t)s->lba << 2,
+                                              &s->bus->dma->qiov, n * 4,
+                                              ide_atapi_cmd_read_dma_cb, s);
     return;
 
 eot:
hw/ide/core.c +9 -5
index d76244a..1fff193 100644
@@ -630,11 +630,13 @@ void ide_sector_read(IDEState *s)
     s->iov.iov_base = s->io_buffer;
     s->iov.iov_len  = n * BDRV_SECTOR_SIZE;
     qemu_iovec_init_external(&s->qiov, &s->iov, 1);
+    s->qiov.replay = true;
+    s->qiov.replay_step = replay_get_current_step();
 
     block_acct_start(blk_get_stats(s->blk), &s->acct,
                      n * BDRV_SECTOR_SIZE, BLOCK_ACCT_READ);
-    s->pio_aiocb = blk_aio_readv(s->blk, sector_num, &s->qiov, n,
-                                 ide_sector_read_cb, s);
+    s->pio_aiocb = blk_aio_readv_replay(s->blk, sector_num, &s->qiov, n,
+                                        ide_sector_read_cb, s);
 }
 
 static void dma_buf_commit(IDEState *s, uint32_t tx_bytes)
@@ -888,11 +890,13 @@ void ide_sector_write(IDEState *s)
     s->iov.iov_base = s->io_buffer;
     s->iov.iov_len  = n * BDRV_SECTOR_SIZE;
     qemu_iovec_init_external(&s->qiov, &s->iov, 1);
+    s->qiov.replay = true;
+    s->qiov.replay_step = replay_get_current_step();
 
     block_acct_start(blk_get_stats(s->blk), &s->acct,
                      n * BDRV_SECTOR_SIZE, BLOCK_ACCT_READ);
-    s->pio_aiocb = blk_aio_writev(s->blk, sector_num, &s->qiov, n,
-                                  ide_sector_write_cb, s);
+    s->pio_aiocb = blk_aio_writev_replay(s->blk, sector_num, &s->qiov, n,
+                                         ide_sector_write_cb, s);
 }
 
 static void ide_flush_cb(void *opaque, int ret)
@@ -928,7 +932,7 @@ void ide_flush_cache(IDEState *s)
 
     s->status |= BUSY_STAT;
     block_acct_start(blk_get_stats(s->blk), &s->acct, 0, BLOCK_ACCT_FLUSH);
-    s->pio_aiocb = blk_aio_flush(s->blk, ide_flush_cb, s);
+    s->pio_aiocb = blk_aio_flush_replay(s->blk, ide_flush_cb, s);
 }
 
 static void ide_cfata_metadata_inquiry(IDEState *s)
include/block/block.h +14 -1
index 6e7275d..2a75fc3 100644
@@ -292,11 +292,24 @@ typedef void BlockDriverDirtyHandler(BlockDriverState *bs, int64_t sector,
 BlockAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,
                            QEMUIOVector *iov, int nb_sectors,
                            BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *bdrv_aio_readv_replay(BlockDriverState *bs,
+                                  int64_t sector_num,
+                                  QEMUIOVector *iov, int nb_sectors,
+                                  BlockCompletionFunc *cb,
+                                  void *opaque);
 BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
                             QEMUIOVector *iov, int nb_sectors,
                             BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *bdrv_aio_writev_replay(BlockDriverState *bs,
+                                   int64_t sector_num,
+                                   QEMUIOVector *iov, int nb_sectors,
+                                   BlockCompletionFunc *cb,
+                                   void *opaque);
 BlockAIOCB *bdrv_aio_flush(BlockDriverState *bs,
                            BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *bdrv_aio_flush_replay(BlockDriverState *bs,
+                                  BlockCompletionFunc *cb,
+                                  void *opaque);
 BlockAIOCB *bdrv_aio_discard(BlockDriverState *bs,
                              int64_t sector_num, int nb_sectors,
                              BlockCompletionFunc *cb, void *opaque);
@@ -317,7 +330,7 @@ typedef struct BlockRequest {
 } BlockRequest;
 
 int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs,
-    int num_reqs);
+                        int num_reqs, bool replay);
 
 /* sg packet commands */
 int bdrv_ioctl(BlockDriverState *bs, unsigned long int req, void *buf);
include/qemu-common.h +2 -0
index e30adc5..01f7c6b 100644
@@ -320,6 +320,8 @@ typedef struct QEMUIOVector {
     int niov;
     int nalloc;
     size_t size;
+    bool replay;
+    uint64_t replay_step;
 } QEMUIOVector;
 
 void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint);
include/sysemu/block-backend.h +10 -0
index 8871a02..c34855e 100644
@@ -94,17 +94,27 @@ void blk_get_geometry(BlockBackend *blk, uint64_t *nb_sectors_ptr);
 BlockAIOCB *blk_aio_readv(BlockBackend *blk, int64_t sector_num,
                           QEMUIOVector *iov, int nb_sectors,
                           BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *blk_aio_readv_replay(BlockBackend *blk, int64_t sector_num,
+                                 QEMUIOVector *iov, int nb_sectors,
+                                 BlockCompletionFunc *cb, void *opaque);
 BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,
                            QEMUIOVector *iov, int nb_sectors,
                            BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *blk_aio_writev_replay(BlockBackend *blk, int64_t sector_num,
+                                  QEMUIOVector *iov, int nb_sectors,
+                                  BlockCompletionFunc *cb, void *opaque);
 BlockAIOCB *blk_aio_flush(BlockBackend *blk,
                           BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *blk_aio_flush_replay(BlockBackend *blk,
+                                 BlockCompletionFunc *cb, void *opaque);
 BlockAIOCB *blk_aio_discard(BlockBackend *blk,
                             int64_t sector_num, int nb_sectors,
                             BlockCompletionFunc *cb, void *opaque);
 void blk_aio_cancel(BlockAIOCB *acb);
 void blk_aio_cancel_async(BlockAIOCB *acb);
 int blk_aio_multiwrite(BlockBackend *blk, BlockRequest *reqs, int num_reqs);
+int blk_aio_multiwrite_replay(BlockBackend *blk, BlockRequest *reqs,
+                              int num_reqs);
 int blk_ioctl(BlockBackend *blk, unsigned long int req, void *buf);
 BlockAIOCB *blk_aio_ioctl(BlockBackend *blk, unsigned long int req, void *buf,
                           BlockCompletionFunc *cb, void *opaque);
qemu-io-cmds.c +1 -1
index e708552..a7d8d4e 100644
@@ -583,7 +583,7 @@ static int do_aio_multiwrite(BlockDriverState *bs, BlockRequest* reqs,
         *total += reqs[i].qiov->size;
     }
 
-    ret = bdrv_aio_multiwrite(bs, reqs, num_reqs);
+    ret = bdrv_aio_multiwrite(bs, reqs, num_reqs, false);
     if (ret < 0) {
         return ret;
     }
stubs/replay.c +5 -0
index 448cab0..9beecb5 100755
@@ -30,3 +30,8 @@ int runstate_is_running(void)
 void replay_add_bh_event(void *bh, uint64_t id)
 {
 }
+
+uint64_t replay_get_current_step(void)
+{
+    return 0;
+}
trace-events +2 -0
index 8acbcce..7238c8c 100644
@@ -64,7 +64,9 @@ bdrv_aio_multiwrite(void *mcb, int num_callbacks, int num_reqs) "mcb %p num_call
 bdrv_aio_discard(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_flush(void *bs, void *opaque) "bs %p opaque %p"
 bdrv_aio_readv(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
+bdrv_aio_readv_replay(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_writev(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
+bdrv_aio_writev_replay(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_write_zeroes(void *bs, int64_t sector_num, int nb_sectors, int flags, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d flags %#x opaque %p"
 bdrv_lock_medium(void *bs, bool locked) "bs %p locked %d"
 bdrv_co_readv(void *bs, int64_t sector_num, int nb_sector) "bs %p sector_num %"PRId64" nb_sectors %d"
util/iov.c +4 -0
index 2fb18e6..96517e7 100644
@@ -257,6 +257,8 @@ void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint)
     qiov->niov = 0;
     qiov->nalloc = alloc_hint;
     qiov->size = 0;
+    qiov->replay = false;
+    qiov->replay_step = 0;
 }
 
 void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov)
@@ -267,6 +269,8 @@ void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov)
     qiov->niov = niov;
     qiov->nalloc = -1;
     qiov->size = 0;
+    qiov->replay = false;
+    qiov->replay_step = 0;
     for (i = 0; i < niov; i++)
         qiov->size += iov[i].iov_len;
 }
#64 lore raw

RFC PATCH v7 18/21 replay: thread pool

In reply to Pavel Dovgalyuk

This patch modifies thread pool to allow replaying asynchronous thread tasks synchronously in replay mode.

---
 block/raw-posix.c           |    6 ++++-
 block/raw-win32.c           |    4 +++-
 include/block/thread-pool.h |    4 +++-
 replay/replay-events.c      |   11 ++++++++++
 replay/replay-internal.h    |    3 ++-
 replay/replay.h             |    2 ++
 stubs/replay.c              |    4 ++++
 tests/test-thread-pool.c    |    7 ++++--
 thread-pool.c               |   49 ++++++++++++++++++++++++++++++-------------
 9 files changed, 67 insertions(+), 23 deletions(-)
block/raw-posix.c +4 -2
index e51293a..f878e06 100644
@@ -1073,7 +1073,9 @@ static BlockAIOCB *paio_submit(BlockDriverState *bs, int fd,
 
     trace_paio_submit(acb, opaque, sector_num, nb_sectors, type);
     pool = aio_get_thread_pool(bdrv_get_aio_context(bs));
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque,
+                                  qiov ? qiov->replay : false,
+                                  qiov ? qiov->replay_step : 0);
 }
 
 static BlockAIOCB *raw_aio_submit(BlockDriverState *bs,
@@ -1986,7 +1988,7 @@ static BlockAIOCB *hdev_aio_ioctl(BlockDriverState *bs,
     acb->aio_ioctl_buf = buf;
     acb->aio_ioctl_cmd = req;
     pool = aio_get_thread_pool(bdrv_get_aio_context(bs));
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque, false, 0);
 }
 
 #elif defined(__FreeBSD__) || defined(__FreeBSD_kernel__)
block/raw-win32.c +3 -1
index 06243d7..7ac693b 100644
@@ -158,7 +158,9 @@ static BlockAIOCB *paio_submit(BlockDriverState *bs, HANDLE hfile,
 
     trace_paio_submit(acb, opaque, sector_num, nb_sectors, type);
     pool = aio_get_thread_pool(bdrv_get_aio_context(bs));
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque,
+                                  qiov ? qiov->replay : false,
+                                  qiov ? qiov->replay_step : 0);
 }
 
 int qemu_ftruncate64(int fd, int64_t length)
include/block/thread-pool.h +3 -1
index 42eb5e8..801ac00 100644
@@ -29,9 +29,11 @@ void thread_pool_free(ThreadPool *pool);
 
 BlockAIOCB *thread_pool_submit_aio(ThreadPool *pool,
         ThreadPoolFunc *func, void *arg,
-        BlockCompletionFunc *cb, void *opaque);
+        BlockCompletionFunc *cb, void *opaque,
+        bool replay, uint64_t replay_step);
 int coroutine_fn thread_pool_submit_co(ThreadPool *pool,
         ThreadPoolFunc *func, void *arg);
 void thread_pool_submit(ThreadPool *pool, ThreadPoolFunc *func, void *arg);
+void thread_pool_work(ThreadPool *pool, void *r);
 
 #endif
replay/replay-events.c +11 -0
index 1aee0a4..4da5de0 100755
@@ -12,6 +12,7 @@
 #include "qemu-common.h"
 #include "replay.h"
 #include "replay-internal.h"
+#include "block/thread-pool.h"
 
 typedef struct Event {
     int event_kind;
@@ -39,6 +40,9 @@ static void replay_run_event(Event *event)
     case REPLAY_ASYNC_EVENT_BH:
         aio_bh_call(event->opaque);
         break;
+    case REPLAY_ASYNC_EVENT_THREAD:
+        thread_pool_work((ThreadPool *)event->opaque, event->opaque2);
+        break;
     default:
         fprintf(stderr, "Replay: invalid async event ID (%d) in the queue\n",
                 event->event_kind);
@@ -127,6 +131,11 @@ void replay_add_bh_event(void *bh, uint64_t id)
     replay_add_event_internal(REPLAY_ASYNC_EVENT_BH, bh, NULL, id);
 }
 
+void replay_add_thread_event(void *opaque, void *opaque2, uint64_t id)
+{
+    replay_add_event_internal(REPLAY_ASYNC_EVENT_THREAD, opaque, opaque2, id);
+}
+
 void replay_save_events(int opt)
 {
     qemu_mutex_lock(&lock);
@@ -141,6 +150,7 @@ void replay_save_events(int opt)
             /* save event-specific data */
             switch (event->event_kind) {
             case REPLAY_ASYNC_EVENT_BH:
+            case REPLAY_ASYNC_EVENT_THREAD:
                 replay_put_qword(event->id);
                 break;
             }
@@ -170,6 +180,7 @@ void replay_read_events(int opt)
         /* Execute some events without searching them in the queue */
         switch (read_event_kind) {
         case REPLAY_ASYNC_EVENT_BH:
+        case REPLAY_ASYNC_EVENT_THREAD:
             if (read_id == -1) {
                 read_id = replay_get_qword();
             }
replay/replay-internal.h +2 -1
index 6e0c2e9..c32bd9c 100755
@@ -38,7 +38,8 @@
 /* Asynchronous events IDs */
 
 #define REPLAY_ASYNC_EVENT_BH          0
-#define REPLAY_ASYNC_COUNT             1
+#define REPLAY_ASYNC_EVENT_THREAD      1
+#define REPLAY_ASYNC_COUNT             2
 
 typedef struct ReplayState {
     /*! Cached clock values. */
replay/replay.h +2 -0
index d3f168d..c72a3fd 100755
@@ -95,5 +95,7 @@ int replay_checkpoint(unsigned int checkpoint);
 void replay_disable_events(void);
 /*! Adds BH event to the queue */
 void replay_add_bh_event(void *bh, uint64_t id);
+/*! Adds thread event to the queue */
+void replay_add_thread_event(void *pool, void *req, uint64_t id);
 
 #endif
stubs/replay.c +4 -0
index 9beecb5..770d59a 100755
@@ -35,3 +35,7 @@ uint64_t replay_get_current_step(void)
 {
     return 0;
 }
+
+void replay_add_thread_event(void *opaque, void *opaque2, uint64_t id)
+{
+}
tests/test-thread-pool.c +4 -3
index 6a0b981..f32594c 100644
@@ -56,7 +56,7 @@ static void test_submit_aio(void)
 {
     WorkerTestData data = { .n = 0, .ret = -EINPROGRESS };
     data.aiocb = thread_pool_submit_aio(pool, worker_cb, &data,
-                                        done_cb, &data);
+                                        done_cb, &data, false, 0);
 
     /* The callbacks are not called until after the first wait.  */
     active = 1;
@@ -120,7 +120,8 @@ static void test_submit_many(void)
     for (i = 0; i < 100; i++) {
         data[i].n = 0;
         data[i].ret = -EINPROGRESS;
-        thread_pool_submit_aio(pool, worker_cb, &data[i], done_cb, &data[i]);
+        thread_pool_submit_aio(pool, worker_cb, &data[i], done_cb, &data[i],
+                               false, 0);
     }
 
     active = 100;
@@ -149,7 +150,7 @@ static void do_test_cancel(bool sync)
         data[i].n = 0;
         data[i].ret = -EINPROGRESS;
         data[i].aiocb = thread_pool_submit_aio(pool, long_cb, &data[i],
-                                               done_cb, &data[i]);
+                                               done_cb, &data[i], false, 0);
     }
 
     /* Starting the threads may be left to a bottom half.  Let it
thread-pool.c +34 -15

99 lines not on this page · raw message

#65 lore raw

RFC PATCH v7 19/21 replay: initialization and deinitialization

In reply to Pavel Dovgalyuk

This patch introduces the functions for enabling the record/replay and for freeing the resources when simulator closes.

---
 block.c                  |    2 -
 exec.c                   |    1 
 replay/replay-internal.h |    2 +
 replay/replay.c          |  134 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   13 ++++
 stubs/replay.c           |    1 
 vl.c                     |    5 ++
 7 files changed, 157 insertions(+), 1 deletions(-)
block.c +1 -1
index e4cad41..1dd9837 100644
@@ -1982,7 +1982,7 @@ void bdrv_drain_all(void)
             aio_context_release(aio_context);
         }
     }
-    if (replay_mode == REPLAY_MODE_PLAY) {
+    if (replay_mode == REPLAY_MODE_PLAY && replay_checkpoints) {
         /* Skip checkpoints from the log */
         while (replay_checkpoint(8)) {
             /* Nothing */
exec.c +1 -0
index 081818e..473aec7 100644
@@ -799,6 +799,7 @@ void cpu_abort(CPUState *cpu, const char *fmt, ...)
     }
     va_end(ap2);
     va_end(ap);
+    replay_finish();
 #if defined(CONFIG_USER_ONLY)
     {
         struct sigaction act;
replay/replay-internal.h +2 -0
index c32bd9c..142e09a 100755
@@ -34,6 +34,8 @@
 
 /* for checkpoint event */
 #define EVENT_CHECKPOINT            96
+/* end of log event */
+#define EVENT_END                   127
 
 /* Asynchronous events IDs */
 
replay/replay.c +134 -0

157 lines not on this page · raw message

replay/replay.h +13 -0
index c72a3fd..9c40648 100755
@@ -17,6 +17,8 @@
 #include <time.h>
 #include "qapi-types.h"
 
+struct QemuOpts;
+
 /* replay clock kinds */
 /* rdtsc */
 #define REPLAY_CLOCK_REAL_TICKS 0
@@ -29,10 +31,21 @@
 
 extern ReplayMode replay_mode;
 extern char *replay_image_suffix;
+/*! Is set to true after finishing initialization */
+extern bool replay_checkpoints;
 
 /*! Returns replay play submode */
 ReplaySubmode replay_get_play_submode(void);
 
+/* Replay process control functions */
+
+/*! Enables recording or saving event log with specified parameters */
+void replay_configure(struct QemuOpts *opts, int mode);
+/*! Initializes timers used for snapshotting and enables events recording */
+void replay_init_timer(void);
+/*! Closes replay log file and frees other resources. */
+void replay_finish(void);
+
 /* Processing the instructions */
 
 /*! Returns number of executed instructions. */
stubs/replay.c +1 -0
index 770d59a..52076d9 100755
@@ -2,6 +2,7 @@
 #include "sysemu/sysemu.h"
 
 ReplayMode replay_mode;
+bool replay_checkpoints;
 
 ReplaySubmode replay_get_play_submode(void)
 {
vl.c +5 -0
index 663e4db..8b1b793 100644
@@ -4372,7 +4372,12 @@ int main(int argc, char **argv, char **envp)
         }
     }
 
+    replay_init_timer();
+
     main_loop();
+    if (replay_mode != REPLAY_MODE_NONE) {
+        replay_disable_events();
+    }
     bdrv_close_all();
     pause_all_vcpus();
     res_free();
#66 lore raw

RFC PATCH v7 20/21 replay: command line options

In reply to Pavel Dovgalyuk

This patch introduces command line options for enabling recording or replaying virtual machine behavior. "-record" option starts recording of the execution and saves it into the log, specified with "fname" parameter. "-replay" option is intended for replaying previously saved log.

---
 cpus.c          |    3 +-
 qemu-options.hx |   27 +++++++++++++++++++
 vl.c            |   79 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 106 insertions(+), 3 deletions(-)
cpus.c +2 -1
index 907a4bb..3442af2 100644
@@ -912,9 +912,10 @@ static void qemu_wait_io_event_common(CPUState *cpu)
 static void qemu_tcg_wait_io_event(void)
 {
     CPUState *cpu;
+    GMainContext *context = g_main_context_default();
 
     while (all_cpu_threads_idle()) {
-       /* Start accounting real time to the virtual clock if the CPUs
+        /* Start accounting real time to the virtual clock if the CPUs
           are idle.  */
         qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
         qemu_cond_wait(tcg_halt_cond, &qemu_global_mutex);
qemu-options.hx +27 -0
index 10b9568..9d28bbb 100644
@@ -3395,6 +3395,33 @@ Dump json-encoded vmstate information for current machine type to file
 in @var{file}
 ETEXI
 
+DEF("record", HAS_ARG, QEMU_OPTION_record,
+    "-record fname=<filename>[,suffix=<suffix>,snapshot=<on/off>]\n"
+    "                writes replay file for latter replaying\n",
+    QEMU_ARCH_ALL)
+STEXI
+@item -record fname=@var{file}[,suffix=@var{suffix},snapshot=@var{snapshot}]
+Writes compact execution trace into @var{file}.
+Changes for disk images are written
+into separate files with @var{suffix} added. If no @var{suffix} is
+specified, "replay_qcow" is used as suffix.
+If @var{snapshot} parameter is set as off, then original disk image will be
+modified. Default value is on.
+ETEXI
+
+DEF("replay", HAS_ARG, QEMU_OPTION_replay,
+    "-replay fname=<filename>[,suffix=<suffix>,snapshot=<on/off>]\n"
+    "                plays saved replay file\n", QEMU_ARCH_ALL)
+STEXI
+@item -replay fname=@var{filename}[,suffix=@var{suffix},snapshot=@var{snapshot}]
+Plays compact execution trace from @var{filename}.
+Changes for disk images and VM states are read
+from separate files with @var{suffix} added. If no @var{suffix} is
+specified, "replay_qcow" is used as suffix.
+If @var{snapshot} parameter is set as off, then original disk image will be
+modified. Default value is on.
+ETEXI
+
 HXCOMM This is the last statement. Insert new options before this line!
 STEXI
 @end table
vl.c +77 -2

157 lines not on this page · raw message

#67 lore raw

Re: [Qemu-devel] [RFC PATCH v7 20/21] replay: command line options

Quoted 4 lines · Pavel Dovgalyuk

This patch introduces command line options for enabling recording or replaying virtual machine behavior. "-record" option starts recording of the execution and saves it into the log, specified with "fname" parameter. "-replay" option is intended for replaying previously saved log.

What about just using "-icount", since it requires that?

Paolo

#68 lore raw

Re: [Qemu-devel] [RFC PATCH v7 20/21] replay: command line options

Quoted 8 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 4 lines · Pavel Dovgalyuk

This patch introduces command line options for enabling recording or replaying virtual machine behavior. "-record" option starts recording of the execution and saves it into the log, specified with "fname" parameter. "-replay" option is intended for replaying previously saved log.

What about just using "-icount", since it requires that?

You mean adding more attributes to -icount option? Or implicitly enabling -icount with -record/replay?

Pavel Dovgalyuk

#69 lore raw

Re: [Qemu-devel] [RFC PATCH v7 20/21] replay: command line options

Quoted 3 lines · Pavel Dovgaluk
Quoted 1 line
Quoted 1 line

What about just using "-icount", since it requires that?

You mean adding more attributes to -icount option? Or implicitly enabling -icount with -record/replay?

Adding more suboptions to -icount (e.g. rr=record|replay,rrfile=NAME).

Paolo

#70 lore raw

RFC PATCH v7 21/21 replay: recording of the user input

In reply to Pavel Dovgalyuk

This records user input (keyboard and mouse events) in record mode and replays these input events in replay mode.

---
 include/ui/input.h       |    2 +
 replay/Makefile.objs     |    1 
 replay/replay-events.c   |   48 ++++++++++++++++++++
 replay/replay-input.c    |  108 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   11 ++++-
 replay/replay.h          |    5 ++
 ui/input.c               |   80 ++++++++++++++++++++++++++--------
 7 files changed, 235 insertions(+), 20 deletions(-)
 create mode 100755 replay/replay-input.c
include/ui/input.h +2 -0
index 5d5ac00..d06a12d 100644
@@ -33,7 +33,9 @@ void qemu_input_handler_bind(QemuInputHandlerState *s,
                              const char *device_id, int head,
                              Error **errp);
 void qemu_input_event_send(QemuConsole *src, InputEvent *evt);
+void qemu_input_event_send_impl(QemuConsole *src, InputEvent *evt);
 void qemu_input_event_sync(void);
+void qemu_input_event_sync_impl(void);
 
 InputEvent *qemu_input_event_new_key(KeyValue *key, bool down);
 void qemu_input_event_send_key(QemuConsole *src, KeyValue *key, bool down);
replay/Makefile.objs +1 -0
index 257c320..3936296 100755
@@ -2,3 +2,4 @@ obj-y += replay.o
 obj-y += replay-internal.o
 obj-y += replay-events.o
 obj-y += replay-time.o
+obj-y += replay-input.o
replay/replay-events.c +48 -0

84 lines not on this page · raw message

replay/replay-input.c new file +108 -0

108 lines not on this page · raw message

replay/replay-internal.h +10 -1
index 142e09a..ac3ba82 100755
@@ -41,7 +41,9 @@
 
 #define REPLAY_ASYNC_EVENT_BH          0
 #define REPLAY_ASYNC_EVENT_THREAD      1
-#define REPLAY_ASYNC_COUNT             2
+#define REPLAY_ASYNC_EVENT_INPUT       2
+#define REPLAY_ASYNC_EVENT_INPUT_SYNC  3
+#define REPLAY_ASYNC_COUNT             4
 
 typedef struct ReplayState {
     /*! Cached clock values. */
@@ -120,4 +122,11 @@ void replay_read_events(int opt);
 /*! Adds specified async event to the queue */
 void replay_add_event(int event_id, void *opaque);
 
+/* Input events */
+
+/*! Saves input event to the log */
+void replay_save_input_event(InputEvent *evt);
+/*! Reads input event from the log */
+InputEvent *replay_read_input_event(void);
+
 #endif
replay/replay.h +5 -0
index 9c40648..f8bf4c7 100755
@@ -18,6 +18,7 @@
 #include "qapi-types.h"
 
 struct QemuOpts;
+struct InputEvent;
 
 /* replay clock kinds */
 /* rdtsc */
@@ -110,5 +111,9 @@ void replay_disable_events(void);
 void replay_add_bh_event(void *bh, uint64_t id);
 /*! Adds thread event to the queue */
 void replay_add_thread_event(void *pool, void *req, uint64_t id);
+/*! Adds input event to the queue */
+void replay_add_input_event(struct InputEvent *event);
+/*! Adds input sync event to the queue */
+void replay_add_input_sync_event(void);
 
 #endif
ui/input.c +61 -19

142 lines not on this page · raw message

#71 lore raw

Re: [Qemu-devel] [RFC PATCH v7 21/21] replay: recording of the user input

Quoted 5 lines · Pavel Dovgalyuk
ui/input.c +1 -2
+void qemu_input_event_send(QemuConsole *src, InputEvent *evt)
 {
-    QemuInputHandlerState *s;
-
     if (!runstate_is_running() && !runstate_check(RUN_STATE_SUSPENDED)) {

ui/input.c void qemu_input_event_send(QemuConsole *src, InputEvent *evt) · line 334

Quoted 10 lines
ui/input.c +7 -0
         return;
     }
 
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        /* Nothing */
+    } else if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_add_input_event(evt);
+    } else {
+        qemu_input_event_send_impl(src, evt);
+    }

Similar to other cases, please wrap this into a single function, something like

    if (replay_handle_input_event(evt)) {
        return;
    }

    /* ... original contents of qemu_input_event_send ... */

Paolo

#72 lore raw

Re: [Qemu-devel] [RFC PATCH v7 21/21] replay: recording of the user input

Quoted 21 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 5 lines · Pavel Dovgalyuk
ui/input.c +1 -2
+void qemu_input_event_send(QemuConsole *src, InputEvent *evt)
 {
-    QemuInputHandlerState *s;
-
     if (!runstate_is_running() && !runstate_check(RUN_STATE_SUSPENDED)) {

ui/input.c void qemu_input_event_send(QemuConsole *src, InputEvent *evt) · line 334

Quoted 10 lines
ui/input.c +7 -0
         return;
     }
 
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        /* Nothing */
+    } else if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_add_input_event(evt);
+    } else {
+        qemu_input_event_send_impl(src, evt);
+    }

Similar to other cases, please wrap this into a single function, something like

Quoted 5 lines
    if (replay_handle_input_event(evt)) {
        return;
    }

    /* ... original contents of qemu_input_event_send ... */

I can wrap this, but cannot get rid of _impl function.
In replay mode we have to make two things:
- Deny real user imput (qemu_input_event_send should do nothing)
- Read input from the log (qemu_input_event_send should process read data)

Pavel Dovgalyuk

A message that is not in the archive (sent off-list or not yet received): [email protected]

#74 lore raw

Re: [Qemu-devel] [RFC PATCH v7 21/21] replay: recording of the user input

Quoted 8 lines · Pavel Dovgaluk
Quoted 8 lines
Quoted 8 lines

Similar to other cases, please wrap this into a single function, something like

    if (replay_handle_input_event(evt)) {
        return;
    }
Quoted 5 lines
Quoted 1 line
Quoted 1 line
    /* ... original contents of qemu_input_event_send ... */

I can wrap this, but cannot get rid of _impl function.
In replay mode we have to make two things:
- Deny real user imput (qemu_input_event_send should do nothing)
- Read input from the log (qemu_input_event_send should process read data)

That's fine.

Paolo

#75 lore raw
In reply to Pavel Dovgalyuk
Quoted 10 lines · Pavel Dovgalyuk

This set of patches is related to the reverse execution and deterministic replay of qemu execution This implementation of deterministic replay can be used for deterministic debugging of guest code through gdb remote interface.

These patches include only core function of the replay, excluding the support for replaying serial, audio, network, and USB devices' operations. Reverse debugging and monitor commands were also excluded to be submitted later as separate patches.

Quoted 5 lines

Execution recording writes non-deterministic events log, which can be later used for replaying the execution anywhere and for unlimited number of times. It also supports checkpointing for faster rewinding during reverse debugging. Execution replaying reads the log and replays all non-deterministic events including external input, hardware clocks, and interrupts.

So it's good that all the generic migration stuff went in, and also some of the RR infrastructure. v8 should be below 20 patches, which is pretty good.

I made some generic comments:

1) thread-safety unfortunately looks a bit iffy. The RR QemuMutex should be the deepest lock in the hierarchy, which basically means that replay/ should never call anything outside it while taking the lock.

2) your code generally looks like:

   -void foo(void)
   +void foo_impl(void)
    {
        ...
    }

+void foo(void)
+{
+ if (replay_mode) ...
+ foo_impl();
+}

Instead, please use

    void foo(void)
    {
   +    replay_foo();
        ...
    }

and possibly put replay_foo() in the replay/ subdirectory.

3) Inline functions are better than macros. enums are better than #defines.

4) I've applied patch 10.

Paolo