Skip to contentSkip to conversation

Deterministic replay extensions

PATCH Superseded Pavel Dovgalyuk 4 participants 17 messages Mar 10, 2016 – Mar 14, 2016 Block layer

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v4 cover Deterministic replay extensions

This set of patches is related to the reverse execution and deterministic replay of qemu execution. It includes recording and replaying of serial devices and block devices operations.

With these patches one can record and deterministically replay behavior of the system with connected disk drives and serial communication ports (e.g., telnet terminal).

Patches for deterministic replay of the block devices intercept calls of bdrv coroutine functions at the top of block drivers stack. To record and replay block operations the drive must be configured as following:
-drive file=disk.qcow,if=none,id=img-direct
-drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay -device ide-hd,drive=img-blkreplay

blkreplay driver should be inserted between disk image and virtual driver controller. Therefore all disk requests may be recorded and replayed.

v4 changes:
- minor fixes in blkreplay layer (as suggested by Kevin Wolf)
- split qemu_clock_warp into two functions (as suggested by Paolo Bonzini)

v3 changes:
- introduced bdrv_flush callback for block drivers
- introduced block driver for recording block operations (as suggested by Kevin Wolf)
- added documentation for block record/replay

v2 changes:
- removed obsolete call of qemu_clock_warp
- fixed record/replay of aio_cancel
- simplified call sequence for blk_aio_ functions in non-replay mode (as suggested by Kevin Wolf)

---

Pavel Dovgalyuk (5):
replay: character devices
icount: remove obsolete warp call
icount: decouple warp calls
block: add flush callback
replay: introduce block devices record/replay

 block/Makefile.objs       |    2 -
 block/blkreplay.c         |  159 +++++++++++++++++++++++++++++++++++++++++++++
 block/io.c                |    7 ++
 cpus.c                    |   56 +++++++++-------
 docs/replay.txt           |   20 ++++++
 gdbstub.c                 |    2 -
 include/block/block_int.h |    7 ++
 include/qemu/timer.h      |   14 +++-
 include/sysemu/char.h     |   26 +++++++
 include/sysemu/replay.h   |   17 +++++
 main-loop.c               |    2 -
 qemu-char.c               |   56 ++++++++++++++--
 qemu-timer.c              |    4 +
 replay/Makefile.objs      |    1 
 replay/replay-char.c      |   98 ++++++++++++++++++++++++++++
 replay/replay-events.c    |   41 ++++++++++--
 replay/replay-internal.h  |   16 +++++
 replay/replay.c           |   25 +++++++
 stubs/clock-warp.c        |    2 -
 stubs/replay.c            |    4 +
 20 files changed, 512 insertions(+), 47 deletions(-)
 create mode 100755 block/blkreplay.c
 create mode 100755 replay/replay-char.c
Signature · 1 line
Pavel Dovgalyuk
#1 lore raw

PATCH v4 1/5 replay: character devices

This patch implements record and replay of character devices. It records chardevs communication in replay mode. Recorded information include data read from backend and counter of bytes written from frontend to backend to preserve frontend internal state. If character device was configured through the command line in record mode, then in replay mode it should be also added to command line. Backend of the character device could be changed in replay mode.
Replaying of devices that perform ioctl and get_msgfd operations is not supported.
gdbstub which also acts as a backend is not recorded to allow controlling the replaying through gdb.

---
 gdbstub.c                |    2 -
 include/sysemu/char.h    |   26 ++++++++++++
 include/sysemu/replay.h  |   12 ++++++
 qemu-char.c              |   56 +++++++++++++++++++++++---
 replay/Makefile.objs     |    1 
 replay/replay-char.c     |   98 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-events.c   |   17 +++++++-
 replay/replay-internal.h |   15 +++++++
 replay/replay.c          |   25 +++++++++++-
 9 files changed, 241 insertions(+), 11 deletions(-)
 create mode 100755 replay/replay-char.c
gdbstub.c +1 -1
index 61c12b1..fdcb0ee 100644
@@ -1752,7 +1752,7 @@ int gdbserver_start(const char *device)
             sigaction(SIGINT, &act, NULL);
         }
 #endif
-        chr = qemu_chr_new("gdb", device, NULL);
+        chr = qemu_chr_new_noreplay("gdb", device, NULL);
         if (!chr)
             return -1;
 
include/sysemu/char.h +26 -0
index e46884f..4c2f777 100644
@@ -86,6 +86,7 @@ struct CharDriverState {
     int is_mux;
     guint fd_in_tag;
     QemuOpts *opts;
+    bool replay;
     QTAILQ_ENTRY(CharDriverState) next;
 };
 
@@ -139,6 +140,22 @@ CharDriverState *qemu_chr_new(const char *label, const char *filename,
                               void (*init)(struct CharDriverState *s));
 
 /**
+ * @qemu_chr_new_noreplay:
+ *
+ * Create a new character backend from a URI.
+ * Character device communications are not written
+ * into the replay log.
+ *
+ * @label the name of the backend
+ * @filename the URI
+ * @init not sure..
+ *
+ * Returns: a new character backend
+ */
+CharDriverState *qemu_chr_new_noreplay(const char *label, const char *filename,
+                                       void (*init)(struct CharDriverState *s));
+
+/**
  * @qemu_chr_delete:
  *
  * Destroy a character backend and remove it from the list of
@@ -341,6 +358,15 @@ int qemu_chr_be_can_write(CharDriverState *s);
  */
 void qemu_chr_be_write(CharDriverState *s, uint8_t *buf, int len);
 
+/**
+ * @qemu_chr_be_write_impl:
+ *
+ * Implementation of back end writing. Used by replay module.
+ *
+ * @buf a buffer to receive data from the front end
+ * @len the number of bytes to receive from the front end
+ */
+void qemu_chr_be_write_impl(CharDriverState *s, uint8_t *buf, int len);
 
 /**
  * @qemu_chr_be_event:
include/sysemu/replay.h +12 -0
index e4108e8..4763e56 100644
@@ -114,4 +114,16 @@ void replay_input_event(QemuConsole *src, InputEvent *evt);
 /*! Adds input sync event to the queue */
 void replay_input_sync_event(void);
 
+/* Character device */
+
+/*! Registers char driver to save it's events */
+void replay_register_char_driver(struct CharDriverState *chr);
+/*! Saves write to char device event to the log */
+void replay_chr_be_write(struct CharDriverState *s, uint8_t *buf, int len);
+
+/* Other data */
+
+/*! Writes or reads integer value to/from replay log. */
+void replay_data_int(int *data);
+
 #endif
qemu-char.c +50 -6

101 lines not on this page · raw message

replay/Makefile.objs +1 -0
index 232193a..70e5572 100644
@@ -3,3 +3,4 @@ common-obj-y += replay-internal.o
 common-obj-y += replay-events.o
 common-obj-y += replay-time.o
 common-obj-y += replay-input.o
+common-obj-y += replay-char.o
replay/replay-char.c new file +98 -0

98 lines not on this page · raw message

replay/replay-events.c +14 -3
index 2628109..59d467f 100644
@@ -48,6 +48,9 @@ static void replay_run_event(Event *event)
     case REPLAY_ASYNC_EVENT_INPUT_SYNC:
         qemu_input_event_sync_impl();
         break;
+    case REPLAY_ASYNC_EVENT_CHAR:
+        replay_event_char_run(event->opaque);
+        break;
     default:
         error_report("Replay: invalid async event ID (%d) in the queue",
                     event->event_kind);
@@ -102,9 +105,9 @@ void replay_clear_events(void)
 }
 
 /*! Adds specified async event to the queue */
-static void replay_add_event(ReplayAsyncEventKind event_kind,
-                             void *opaque,
-                             void *opaque2, uint64_t id)
+void replay_add_event(ReplayAsyncEventKind event_kind,
+                      void *opaque,
+                      void *opaque2, uint64_t id)
 {
     assert(event_kind < REPLAY_ASYNC_COUNT);
 
@@ -168,6 +171,9 @@ static void replay_save_event(Event *event, int checkpoint)
             break;
         case REPLAY_ASYNC_EVENT_INPUT_SYNC:
             break;
+        case REPLAY_ASYNC_EVENT_CHAR:
+            replay_event_char_save(event->opaque);
+            break;
         default:
             error_report("Unknown ID %d of replay event", read_event_kind);
             exit(1);
@@ -221,6 +227,11 @@ static Event *replay_read_event(int checkpoint)
         event->event_kind = read_event_kind;
         event->opaque = 0;
         return event;
+    case REPLAY_ASYNC_EVENT_CHAR:
+        event = g_malloc0(sizeof(Event));
+        event->event_kind = read_event_kind;
+        event->opaque = replay_event_char_read();
+        return event;
     default:
         error_report("Unknown ID %d of replay event", read_event_kind);
         exit(1);
replay/replay-internal.h +15 -0
index 5438ebd..73de4ec 100644
@@ -24,6 +24,8 @@ enum ReplayEvents {
     EVENT_ASYNC,
     /* for shutdown request */
     EVENT_SHUTDOWN,
+    /* for int data */
+    EVENT_DATA_INT,
     /* for clock read/writes */
     /* some of greater codes are reserved for clocks */
     EVENT_CLOCK,
@@ -43,6 +45,7 @@ enum ReplayAsyncEventKind {
     REPLAY_ASYNC_EVENT_BH,
     REPLAY_ASYNC_EVENT_INPUT,
     REPLAY_ASYNC_EVENT_INPUT_SYNC,
+    REPLAY_ASYNC_EVENT_CHAR,
     REPLAY_ASYNC_COUNT
 };
 
@@ -124,6 +127,9 @@ bool replay_has_events(void);
 void replay_save_events(int checkpoint);
 /*! Read events from the file into the input queue */
 void replay_read_events(int checkpoint);
+/*! Adds specified async event to the queue */
+void replay_add_event(ReplayAsyncEventKind event_kind, void *opaque,
+                      void *opaque2, uint64_t id);
 
 /* Input events */
 
@@ -136,4 +142,13 @@ void replay_add_input_event(struct InputEvent *event);
 /*! Adds input sync event to the queue */
 void replay_add_input_sync_event(void);
 
+/* Character devices */
+
+/*! Called to run char device event. */
+void replay_event_char_run(void *opaque);
+/*! Writes char event to the file. */
+void replay_event_char_save(void *opaque);
+/*! Reads char event from the file. */
+void *replay_event_char_read(void);
+
 #endif
replay/replay.c +24 -1
index f8739c2..602aee9 100644
@@ -20,7 +20,7 @@
 
 /* Current version of the replay mechanism.
    Increase it when file format changes. */
-#define REPLAY_VERSION              0xe02002
+#define REPLAY_VERSION              0xe02003
 /* Size of replay log header */
 #define HEADER_SIZE                 (sizeof(uint32_t) + sizeof(uint64_t))
 
@@ -350,3 +350,26 @@ void replay_add_blocker(Error *reason)
 {
     replay_blockers = g_slist_prepend(replay_blockers, reason);
 }
+
+void replay_data_int(int *data)
+{
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        replay_account_executed_instructions();
+        replay_mutex_lock();
+        if (replay_next_event_is(EVENT_DATA_INT)) {
+            *data = replay_get_dword();
+            replay_finish_event();
+            replay_mutex_unlock();
+        } else {
+            replay_mutex_unlock();
+            error_report("Missing data int event in the replay log");
+            exit(1);
+        }
+    } else if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_save_instructions();
+        replay_mutex_lock();
+        replay_put_event(EVENT_DATA_INT);
+        replay_put_dword(*data);
+        replay_mutex_unlock();
+    }
+}
#5 lore raw

PATCH v4 2/5 icount: remove obsolete warp call

In reply to Pavel Dovgalyuk

qemu_clock_warp call in qemu_tcg_wait_io_event function is not needed anymore, because it is called in every iteration of main_loop_wait.

---
 cpus.c |    3 ---
 1 files changed, 0 insertions(+), 3 deletions(-)
cpus.c +0 -3
index bc774e2..85d0f87 100644
@@ -995,9 +995,6 @@ static void qemu_wait_io_event_common(CPUState *cpu)
 static void qemu_tcg_wait_io_event(CPUState *cpu)
 {
     while (all_cpu_threads_idle()) {
-       /* Start accounting real time to the virtual clock if the CPUs
-          are idle.  */
-        qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
         qemu_cond_wait(cpu->halt_cond, &qemu_global_mutex);
     }
 
#7 lore raw

PATCH v4 3/5 icount: decouple warp calls

In reply to Pavel Dovgalyuk

qemu_clock_warp function is called to update virtual clock when CPU is sleeping. This function includes replay checkpoint to make execution deterministic in icount mode.
Record/replay module flushes async event queue at checkpoints. Some of the events (e.g., block devices operations) include interaction with hardware. E.g., APIC polled by block devices sets one of IRQ flags. Flag to be set depends on currently executed thread (CPU or iothread). Therefore in replay mode we have to process the checkpoints in the same thread as they were recorded.
qemu_clock_warp function (and its checkpoint) may be called from different thread. This patch decouples two different execution cases of this function:
call when CPU is sleeping from iothread and call from cpu thread to update virtual clock.
First task is performed by qemu_start_warp_timer function. It sets warp timer event to the moment of nearest pending virtual timer. Second function (qemu_account_warp_timer) is called from cpu thread before execution of the code. It advances virtual clock by adding the length of period while CPU was sleeping.

---
 cpus.c                  |   53 +++++++++++++++++++++++++++--------------------
 include/qemu/timer.h    |   14 +++++++++---
 include/sysemu/replay.h |    3 ++-
 main-loop.c             |    2 +-
 qemu-timer.c            |    4 +++-
 stubs/clock-warp.c      |    2 +-
 6 files changed, 48 insertions(+), 30 deletions(-)
cpus.c +31 -22
index 85d0f87..3ab9e04 100644
@@ -373,6 +373,7 @@ static void icount_warp_rt(void)
 static void icount_dummy_timer(void *opaque)
 {
     (void)opaque;
+    icount_warp_rt();
 }
 
 void qtest_clock_warp(int64_t dest)
@@ -396,17 +397,12 @@ void qtest_clock_warp(int64_t dest)
     qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
 }
 
-void qemu_clock_warp(QEMUClockType type)
+void qemu_start_warp_timer(void)
 {
     int64_t clock;
     int64_t deadline;
 
-    /*
-     * There are too many global variables to make the "warp" behavior
-     * applicable to other clocks.  But a clock argument removes the
-     * need for if statements all over the place.
-     */
-    if (type != QEMU_CLOCK_VIRTUAL || !use_icount) {
+    if (!use_icount) {
         return;
     }
 
@@ -418,29 +414,17 @@ void qemu_clock_warp(QEMUClockType type)
     }
 
     /* warp clock deterministically in record/replay mode */
-    if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP)) {
+    if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP_START)) {
         return;
     }
 
-    if (icount_sleep) {
-        /*
-         * If the CPUs have been sleeping, advance QEMU_CLOCK_VIRTUAL timer now.
-         * This ensures that the deadline for the timer is computed correctly
-         * below.
-         * This also makes sure that the insn counter is synchronized before
-         * the CPU starts running, in case the CPU is woken by an event other
-         * than the earliest QEMU_CLOCK_VIRTUAL timer.
-         */
-        icount_warp_rt();
-        timer_del(icount_warp_timer);
-    }
     if (!all_cpu_threads_idle()) {
         return;
     }
 
     if (qtest_enabled()) {
         /* When testing, qtest commands advance icount.  */
-	return;
+        return;
     }
 
     /* We want to use the earliest deadline from ALL vm_clocks */
@@ -496,6 +480,31 @@ void qemu_clock_warp(QEMUClockType type)
     }
 }
 
+void qemu_account_warp_timer(void)
+{
+    int64_t clock;
+    int64_t warp_delta;
+
+    if (!use_icount || !icount_sleep) {
+        return;
+    }
+
+    /* Nothing to do if the VM is stopped: QEMU_CLOCK_VIRTUAL timers
+     * do not fire, so computing the deadline does not make sense.
+     */
+    if (!runstate_is_running()) {
+        return;
+    }
+
+    /* warp clock deterministically in record/replay mode */
+    if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP_ACCOUNT)) {
+        return;
+    }
+
+    timer_del(icount_warp_timer);
+    icount_warp_rt();
+}
+
 static bool icount_state_needed(void *opaque)
 {
     return use_icount;
@@ -1496,7 +1505,7 @@ static void tcg_exec_all(void)
     int r;
 
     /* Account partial waits to QEMU_CLOCK_VIRTUAL.  */
-    qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
+    qemu_account_warp_timer();
 
     if (next_cpu == NULL) {
         next_cpu = first_cpu;
include/qemu/timer.h +10 -4
index d0946cb..21ffec6 100644
@@ -210,12 +210,18 @@ void qemu_clock_notify(QEMUClockType type);
 void qemu_clock_enable(QEMUClockType type, bool enabled);
 
 /**
- * qemu_clock_warp:
- * @type: the clock type
+ * qemu_start_warp_timer:
+ *
+ * Starts a timer for virtual clock update
+ */
+void qemu_start_warp_timer(void);
+
+/**
+ * qemu_account_warp_timer:
  *
- * Warp a clock to a new value
+ * Updates virtual clock for the time CPU was sleeping
  */
-void qemu_clock_warp(QEMUClockType type);
+void qemu_account_warp_timer(void);
 
 /**
  * qemu_clock_register_reset_notifier:
include/sysemu/replay.h +2 -1
index 4763e56..6c332e5 100644
@@ -27,7 +27,8 @@ typedef enum ReplayClockKind ReplayClockKind;
 
 /* IDs of the checkpoints */
 enum ReplayCheckpoint {
-    CHECKPOINT_CLOCK_WARP,
+    CHECKPOINT_CLOCK_WARP_START,
+    CHECKPOINT_CLOCK_WARP_ACCOUNT,
     CHECKPOINT_RESET_REQUESTED,
     CHECKPOINT_SUSPEND_REQUESTED,
     CHECKPOINT_CLOCK_VIRTUAL,
main-loop.c +1 -1
index 19beae7..3a7f4cd 100644
@@ -509,7 +509,7 @@ int main_loop_wait(int nonblocking)
 
     /* CPU thread can infinitely wait for event after
        missing the warp */
-    qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
+    qemu_start_warp_timer();
     qemu_clock_run_all_timers();
 
     return ret;
qemu-timer.c +3 -1
index e98ecc9..4441fe6 100644
@@ -394,7 +394,9 @@ static bool timer_mod_ns_locked(QEMUTimerList *timer_list,
 static void timerlist_rearm(QEMUTimerList *timer_list)
 {
     /* Interrupt execution to force deadline recalculation.  */
-    qemu_clock_warp(timer_list->clock->type);
+    if (timer_list->clock->type == QEMU_CLOCK_VIRTUAL) {
+        qemu_start_warp_timer();
+    }
     timerlist_notify(timer_list);
 }
 
stubs/clock-warp.c +1 -1
index 5ae32b9..8acb58a 100644
@@ -2,7 +2,7 @@
 #include "qemu-common.h"
 #include "qemu/timer.h"
 
-void qemu_clock_warp(QEMUClockType type)
+void qemu_start_warp_timer(void)
 {
 }
 
#11 lore raw

PATCH v4 4/5 block: add flush callback

In reply to Pavel Dovgalyuk

This patch adds callback for flush request. This callback is responsible for flushing whole block devices stack. bdrv_flush function does not proceed to underlying devices. It should be performed by this callback function, if needed.

---
 block/io.c                |    7 +++++++
 include/block/block_int.h |    7 +++++++
 2 files changed, 14 insertions(+), 0 deletions(-)
block/io.c +7 -0
index a69bfc4..242ee0c 100644
@@ -2369,6 +2369,13 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
     }
 
     tracked_request_begin(&req, bs, 0, 0, BDRV_TRACKED_FLUSH);
+
+    /* Write back all layers by calling one driver function */
+    if (bs->drv->bdrv_co_flush) {
+        ret = bs->drv->bdrv_co_flush(bs);
+        goto out;
+    }
+
     /* Write back cached data to the OS even with cache=unsafe */
     BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_OS);
     if (bs->drv->bdrv_co_flush_to_os) {
include/block/block_int.h +7 -0
index 9ef823a..8f72037 100644
@@ -176,6 +176,13 @@ struct BlockDriver {
     int (*bdrv_inactivate)(BlockDriverState *bs);
 
     /*
+     * Flushes all data for all layers by calling bdrv_co_flush for underlying
+     * layers, if needed. This function is needed for deterministic
+     * synchronization of the flush finishing callback.
+     */
+    int coroutine_fn (*bdrv_co_flush)(BlockDriverState *bs);
+
+    /*
      * Flushes all data that was already written to the OS all the way down to
      * the disk (for example raw-posix calls fsync()).
      */
#12 lore raw

PATCH v4 5/5 replay: introduce block devices record/replay

In reply to Pavel Dovgalyuk

This patch introduces block driver that implement recording and replaying of block devices' operations.
All block completion operations are added to the queue. Queue is flushed at checkpoints and information about processed requests is recorded to the log. In replay phase the queue is matched with events read from the log. Therefore block devices requests are processed deterministically.

---
 block/Makefile.objs      |    2 -
 block/blkreplay.c        |  159 ++++++++++++++++++++++++++++++++++++++++++++++
 docs/replay.txt          |   20 ++++++
 include/sysemu/replay.h  |    2 +
 replay/replay-events.c   |   24 ++++++-
 replay/replay-internal.h |    1 
 stubs/replay.c           |    4 +
 7 files changed, 209 insertions(+), 3 deletions(-)
 create mode 100755 block/blkreplay.c
block/Makefile.objs +1 -1
index 58ef2ef..38fea16 100644
@@ -4,7 +4,7 @@ block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o
 block-obj-y += qed-check.o
 block-obj-$(CONFIG_VHDX) += vhdx.o vhdx-endian.o vhdx-log.o
 block-obj-y += quorum.o
-block-obj-y += parallels.o blkdebug.o blkverify.o
+block-obj-y += parallels.o blkdebug.o blkverify.o blkreplay.o
 block-obj-y += block-backend.o snapshot.o qapi.o
 block-obj-$(CONFIG_WIN32) += raw-win32.o win32-aio.o
 block-obj-$(CONFIG_POSIX) += raw-posix.o
block/blkreplay.c new file +159 -0

159 lines not on this page · raw message

docs/replay.txt +20 -0
index 149727e..acf5031 100644
@@ -166,3 +166,23 @@ Sometimes the block layer uses asynchronous callbacks for its internal purposes
 (like reading or writing VM snapshots or disk image cluster tables). In this
 case bottom halves are not marked as "replayable" and do not saved
 into the log.
+
+Block devices
+-------------
+
+Block devices record/replay module intercepts calls of
+bdrv coroutine functions at the top of block drivers stack.
+To record and replay block operations the drive must be configured
+as following:
+ -drive file=disk.qcow,if=none,id=img-direct
+ -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay
+ -device ide-hd,drive=img-blkreplay
+
+blkreplay driver should be inserted between disk image and virtual driver
+controller. Therefore all disk requests may be recorded and replayed.
+
+All block completion operations are added to the queue in the coroutines.
+Queue is flushed at checkpoints and information about processed requests
+is recorded to the log. In replay phase the queue is matched with
+events read from the log. Therefore block devices requests are processed
+deterministically.
include/sysemu/replay.h +2 -0
index 6c332e5..9085d62 100644
@@ -114,6 +114,8 @@ void replay_bh_schedule_event(QEMUBH *bh);
 void replay_input_event(QemuConsole *src, InputEvent *evt);
 /*! Adds input sync event to the queue */
 void replay_input_sync_event(void);
+/*! Adds block layer event to the queue */
+void replay_block_event(QEMUBH *bh, uint64_t id);
 
 /* Character device */
 
replay/replay-events.c +22 -2
index 59d467f..5a7042b 100644
@@ -51,6 +51,9 @@ static void replay_run_event(Event *event)
     case REPLAY_ASYNC_EVENT_CHAR:
         replay_event_char_run(event->opaque);
         break;
+    case REPLAY_ASYNC_EVENT_BLOCK:
+        aio_bh_call(event->opaque);
+        break;
     default:
         error_report("Replay: invalid async event ID (%d) in the queue",
                     event->event_kind);
@@ -135,7 +138,7 @@ void replay_add_event(ReplayAsyncEventKind event_kind,
 
 void replay_bh_schedule_event(QEMUBH *bh)
 {
-    if (replay_mode != REPLAY_MODE_NONE) {
+    if (replay_mode != REPLAY_MODE_NONE && events_enabled) {
         uint64_t id = replay_get_current_step();
         replay_add_event(REPLAY_ASYNC_EVENT_BH, bh, NULL, id);
     } else {
@@ -153,6 +156,15 @@ void replay_add_input_sync_event(void)
     replay_add_event(REPLAY_ASYNC_EVENT_INPUT_SYNC, NULL, NULL, 0);
 }
 
+void replay_block_event(QEMUBH *bh, uint64_t id)
+{
+    if (replay_mode != REPLAY_MODE_NONE && events_enabled) {
+        replay_add_event(REPLAY_ASYNC_EVENT_BLOCK, bh, NULL, id);
+    } else {
+        qemu_bh_schedule(bh);
+    }
+}
+
 static void replay_save_event(Event *event, int checkpoint)
 {
     if (replay_mode != REPLAY_MODE_PLAY) {
@@ -174,8 +186,11 @@ static void replay_save_event(Event *event, int checkpoint)
         case REPLAY_ASYNC_EVENT_CHAR:
             replay_event_char_save(event->opaque);
             break;
+        case REPLAY_ASYNC_EVENT_BLOCK:
+            replay_put_qword(event->id);
+            break;
         default:
-            error_report("Unknown ID %d of replay event", read_event_kind);
+            error_report("Unknown ID %d of replay event", event->id);
             exit(1);
         }
     }
@@ -232,6 +247,11 @@ static Event *replay_read_event(int checkpoint)
         event->event_kind = read_event_kind;
         event->opaque = replay_event_char_read();
         return event;
+    case REPLAY_ASYNC_EVENT_BLOCK:
+        if (read_id == -1) {
+            read_id = replay_get_qword();
+        }
+        break;
     default:
         error_report("Unknown ID %d of replay event", read_event_kind);
         exit(1);
replay/replay-internal.h +1 -0
index 73de4ec..f08b661 100644
@@ -46,6 +46,7 @@ enum ReplayAsyncEventKind {
     REPLAY_ASYNC_EVENT_INPUT,
     REPLAY_ASYNC_EVENT_INPUT_SYNC,
     REPLAY_ASYNC_EVENT_CHAR,
+    REPLAY_ASYNC_EVENT_BLOCK,
     REPLAY_ASYNC_COUNT
 };
 
stubs/replay.c +4 -0
index 00ca01f..6f4a8e8 100644
@@ -29,3 +29,7 @@ bool replay_events_enabled(void)
 void replay_finish(void)
 {
 }
+
+void replay_block_event(QEMUBH *bh, uint64_t id)
+{
+}
#8 lore raw

Re: [Qemu-devel] [PATCH v4 3/5] icount: decouple warp calls

In reply to Pavel Dovgalyuk
Quoted 19 lines · Pavel Dovgalyuk

qemu_clock_warp function is called to update virtual clock when CPU is sleeping. This function includes replay checkpoint to make execution deterministic in icount mode.
Record/replay module flushes async event queue at checkpoints. Some of the events (e.g., block devices operations) include interaction with hardware. E.g., APIC polled by block devices sets one of IRQ flags. Flag to be set depends on currently executed thread (CPU or iothread). Therefore in replay mode we have to process the checkpoints in the same thread as they were recorded.
qemu_clock_warp function (and its checkpoint) may be called from different thread. This patch decouples two different execution cases of this function:
call when CPU is sleeping from iothread and call from cpu thread to update virtual clock.
First task is performed by qemu_start_warp_timer function. It sets warp timer event to the moment of nearest pending virtual timer. Second function (qemu_account_warp_timer) is called from cpu thread before execution of the code. It advances virtual clock by adding the length of period while CPU was sleeping.

Quoted 1 line

Lovely. :) One question, why doesn't icount_dummy_timer need a checkpoint?

Only needs a change to the documentation:

docs/replay.txt +12 -5
index 149727e..26dfb6e 100644
@@ -134,11 +134,18 @@ of time. That's why we do not process a group of timers until the checkpoint
 event will be read from the log. Such an event allows synchronizing CPU
 execution and timer events.
 
-Another checkpoints application in record/replay is instruction counting
-while the virtual machine is idle. This function (qemu_clock_warp) is called
-from the wait loop. It changes virtual machine state and must be deterministic
-then. That is why we added checkpoint to this function to prevent its
-operation in replay mode when it does not correspond to record mode.
+Two other checkpoints govern the "warping" of the virtual clock.  While
+the virtual machine is idle, the virtual clock increments at 1 ns per
+*real time* nanosecond.  This is done by setting up a timer (called the
+warp timer) and then incrementing the virtual clock (called "warping"
+the virtual clock) as soon as the CPUs need to go out of the idle state.
+These actions change virtual machine state and must be deterministic.
+Two functions are used for this purpose, and each of them creates a
+checkpoint.  qemu_start_warp_timer checks if the CPUs are idle and if so
+starts accounting real time to virtual clock.  qemu_account_warp_timer
+is called when the CPUs get an interrupt or when a virtual clock timer
+fires, and it warps the virtual clock by the amount of real time that
+has passed since qemu_start_warp_timer.
 
 Bottom halves
 -------------

Paolo

Quoted 197 lines
---
 cpus.c                  |   53 +++++++++++++++++++++++++++--------------------
 include/qemu/timer.h    |   14 +++++++++---
 include/sysemu/replay.h |    3 ++-
 main-loop.c             |    2 +-
 qemu-timer.c            |    4 +++-
 stubs/clock-warp.c      |    2 +-
 6 files changed, 48 insertions(+), 30 deletions(-)
cpus.c +31 -22
index 85d0f87..3ab9e04 100644
@@ -373,6 +373,7 @@ static void icount_warp_rt(void)
 static void icount_dummy_timer(void *opaque)
 {
     (void)opaque;
+    icount_warp_rt();
 }
 
 void qtest_clock_warp(int64_t dest)
@@ -396,17 +397,12 @@ void qtest_clock_warp(int64_t dest)
     qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
 }
 
-void qemu_clock_warp(QEMUClockType type)
+void qemu_start_warp_timer(void)
 {
     int64_t clock;
     int64_t deadline;
 
-    /*
-     * There are too many global variables to make the "warp" behavior
-     * applicable to other clocks.  But a clock argument removes the
-     * need for if statements all over the place.
-     */
-    if (type != QEMU_CLOCK_VIRTUAL || !use_icount) {
+    if (!use_icount) {
         return;
     }
 
@@ -418,29 +414,17 @@ void qemu_clock_warp(QEMUClockType type)
     }
 
     /* warp clock deterministically in record/replay mode */
-    if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP)) {
+    if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP_START)) {
         return;
     }
 
-    if (icount_sleep) {
-        /*
-         * If the CPUs have been sleeping, advance QEMU_CLOCK_VIRTUAL timer now.
-         * This ensures that the deadline for the timer is computed correctly
-         * below.
-         * This also makes sure that the insn counter is synchronized before
-         * the CPU starts running, in case the CPU is woken by an event other
-         * than the earliest QEMU_CLOCK_VIRTUAL timer.
-         */
-        icount_warp_rt();
-        timer_del(icount_warp_timer);
-    }
     if (!all_cpu_threads_idle()) {
         return;
     }
 
     if (qtest_enabled()) {
         /* When testing, qtest commands advance icount.  */
-	return;
+        return;
     }
 
     /* We want to use the earliest deadline from ALL vm_clocks */
@@ -496,6 +480,31 @@ void qemu_clock_warp(QEMUClockType type)
     }
 }
 
+void qemu_account_warp_timer(void)
+{
+    int64_t clock;
+    int64_t warp_delta;
+
+    if (!use_icount || !icount_sleep) {
+        return;
+    }
+
+    /* Nothing to do if the VM is stopped: QEMU_CLOCK_VIRTUAL timers
+     * do not fire, so computing the deadline does not make sense.
+     */
+    if (!runstate_is_running()) {
+        return;
+    }
+
+    /* warp clock deterministically in record/replay mode */
+    if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP_ACCOUNT)) {
+        return;
+    }
+
+    timer_del(icount_warp_timer);
+    icount_warp_rt();
+}
+
 static bool icount_state_needed(void *opaque)
 {
     return use_icount;
@@ -1496,7 +1505,7 @@ static void tcg_exec_all(void)
     int r;
 
     /* Account partial waits to QEMU_CLOCK_VIRTUAL.  */
-    qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
+    qemu_account_warp_timer();
 
     if (next_cpu == NULL) {
         next_cpu = first_cpu;
include/qemu/timer.h +10 -4
index d0946cb..21ffec6 100644
@@ -210,12 +210,18 @@ void qemu_clock_notify(QEMUClockType type);
 void qemu_clock_enable(QEMUClockType type, bool enabled);
 
 /**
- * qemu_clock_warp:
- * @type: the clock type
+ * qemu_start_warp_timer:
+ *
+ * Starts a timer for virtual clock update
+ */
+void qemu_start_warp_timer(void);
+
+/**
+ * qemu_account_warp_timer:
  *
- * Warp a clock to a new value
+ * Updates virtual clock for the time CPU was sleeping
  */
-void qemu_clock_warp(QEMUClockType type);
+void qemu_account_warp_timer(void);
 
 /**
  * qemu_clock_register_reset_notifier:
include/sysemu/replay.h +2 -1
index 4763e56..6c332e5 100644
@@ -27,7 +27,8 @@ typedef enum ReplayClockKind ReplayClockKind;
 
 /* IDs of the checkpoints */
 enum ReplayCheckpoint {
-    CHECKPOINT_CLOCK_WARP,
+    CHECKPOINT_CLOCK_WARP_START,
+    CHECKPOINT_CLOCK_WARP_ACCOUNT,
     CHECKPOINT_RESET_REQUESTED,
     CHECKPOINT_SUSPEND_REQUESTED,
     CHECKPOINT_CLOCK_VIRTUAL,
main-loop.c +1 -1
index 19beae7..3a7f4cd 100644
@@ -509,7 +509,7 @@ int main_loop_wait(int nonblocking)
 
     /* CPU thread can infinitely wait for event after
        missing the warp */
-    qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
+    qemu_start_warp_timer();
     qemu_clock_run_all_timers();
 
     return ret;
qemu-timer.c +3 -1
index e98ecc9..4441fe6 100644
@@ -394,7 +394,9 @@ static bool timer_mod_ns_locked(QEMUTimerList *timer_list,
 static void timerlist_rearm(QEMUTimerList *timer_list)
 {
     /* Interrupt execution to force deadline recalculation.  */
-    qemu_clock_warp(timer_list->clock->type);
+    if (timer_list->clock->type == QEMU_CLOCK_VIRTUAL) {
+        qemu_start_warp_timer();
+    }
     timerlist_notify(timer_list);
 }
 
stubs/clock-warp.c +1 -1
index 5ae32b9..8acb58a 100644
@@ -2,7 +2,7 @@
 #include "qemu-common.h"
 #include "qemu/timer.h"
 
-void qemu_clock_warp(QEMUClockType type)
+void qemu_start_warp_timer(void)
 {
 }
 
#6 lore raw

Re: [Qemu-devel] [PATCH v4 2/5] icount: remove obsolete warp call

In reply to Pavel Dovgalyuk
Quoted 5 lines · Pavel Dovgalyuk

qemu_clock_warp call in qemu_tcg_wait_io_event function is not needed anymore, because it is called in every iteration of main_loop_wait.

Thanks again. These two patches make the icount mechanism much easier to understand.

Paolo

Quoted 19 lines
---
 cpus.c |    3 ---
 1 files changed, 0 insertions(+), 3 deletions(-)
cpus.c +0 -3
index bc774e2..85d0f87 100644
@@ -995,9 +995,6 @@ static void qemu_wait_io_event_common(CPUState *cpu)
 static void qemu_tcg_wait_io_event(CPUState *cpu)
 {
     while (all_cpu_threads_idle()) {
-       /* Start accounting real time to the virtual clock if the CPUs
-          are idle.  */
-        qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
         qemu_cond_wait(cpu->halt_cond, &qemu_global_mutex);
     }
 
#2 lore raw

Re: [Qemu-devel] [PATCH v4 1/5] replay: character devices

In reply to Pavel Dovgalyuk
Quoted 2 lines · Pavel Dovgalyuk

gdbstub which also acts as a backend is not recorded to allow controlling the replaying through gdb.

Perhaps the monitor too?

Overall the patch is nice and can definitely go in 2.6, but there are a couple changes to do...

qemu-char.c int qemu_chr_fe_write(CharDriverState *s, const uint8_t *buf, int len) · line 251

Quoted 7 lines
qemu-char.c +3 -0
@@ -245,6 +246,9 @@ int qemu_chr_fe_write(CharDriverState *s, const uint8_t *buf, int len)
         qemu_chr_fe_write_log(s, buf, ret);
     }
 
+    if (s->replay) {
+        replay_data_int(&ret);
+    }

I think this is wrong. The logic should be

    if (replaying) {
	read event(&ret);
	assert(ret <= len);
	len = ret;
    }

    qemu_mutex_lock(&s->chr_write_lock);
    ret = s->chr_write(s, buf, len);

    if (ret > 0) {
        qemu_chr_fe_write_log(s, buf, ret);
    }
    qemu_mutex_unlock(&s->chr_write_lock);

    if (recording) {
        write event(ret);
    }
Quoted 9 lines
qemu-char.c +0 -2
     qemu_mutex_unlock(&s->chr_write_lock);
     return ret;
 }
@@ -318,9 +322,19 @@ int qemu_chr_fe_read_all(CharDriverState *s, uint8_t *buf, int len)
 
 int qemu_chr_fe_ioctl(CharDriverState *s, int cmd, void *arg)
 {
-    if (!s->chr_ioctl)
-        return -ENOTSUP;

qemu-char.c int qemu_chr_fe_read_all(CharDriverState *s, uint8_t *buf, int len) · line 333

Quoted 10 lines
qemu-char.c +9 -1
-    return s->chr_ioctl(s, cmd, arg);
+    int res;
+    if (!s->chr_ioctl) {
+        res = -ENOTSUP;
+    } else {
+        res = s->chr_ioctl(s, cmd, arg);
+        if (s->replay) {
+            fprintf(stderr,
+                    "Replay: ioctl is not supported for serial devices yet\n");
+            exit(1);

Is it possible to print this warning just once per device and return -ENOTSUP instead?

replay/replay-char.c line 53

Quoted 9 lines
replay/replay-char.c new file +9 -0
+void replay_register_char_driver(CharDriverState *chr)
+{
+    if (replay_mode == REPLAY_MODE_NONE) {
+        return;
+    }
+    char_drivers = g_realloc(char_drivers,
+                             sizeof(*char_drivers) * (drivers_count + 1));
+    char_drivers[drivers_count++] = chr;
+}

You need a way to unregister character drivers when they are hot-unplugged, or at least you should block chardev-del if in record and replay mode.

replay/replay-internal.h enum ReplayEvents { · line 28

Quoted 2 lines
replay/replay-internal.h +2 -0
+    /* for int data */
+    EVENT_DATA_INT,

I think you should call the event EVENT_CHAR_WRITE (and perhaps rename REPLAY_ASYNC_EVENT_CHAR to REPLAY_ASYNC_EVENT_CHAR_READ). And as mentioned above, I think the load and save cases should be separated in qemu-char.c, so I'd prefer to have a separate function to read and write the event as well.

Thanks,

Paolo

#9 lore raw

Re: [Qemu-devel] [PATCH v4 3/5] icount: decouple warp calls

In reply to Paolo Bonzini
Quoted 21 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 19 lines · Pavel Dovgalyuk

qemu_clock_warp function is called to update virtual clock when CPU is sleeping. This function includes replay checkpoint to make execution deterministic in icount mode.
Record/replay module flushes async event queue at checkpoints. Some of the events (e.g., block devices operations) include interaction with hardware. E.g., APIC polled by block devices sets one of IRQ flags. Flag to be set depends on currently executed thread (CPU or iothread). Therefore in replay mode we have to process the checkpoints in the same thread as they were recorded.
qemu_clock_warp function (and its checkpoint) may be called from different thread. This patch decouples two different execution cases of this function:
call when CPU is sleeping from iothread and call from cpu thread to update virtual clock.
First task is performed by qemu_start_warp_timer function. It sets warp timer event to the moment of nearest pending virtual timer. Second function (qemu_account_warp_timer) is called from cpu thread before execution of the code. It advances virtual clock by adding the length of period while CPU was sleeping.

Quoted 3 lines
Quoted 1 line

Lovely. :) One question, why doesn't icount_dummy_timer need a checkpoint?

It is synchronized with CHECKPOINT_CLOCK_VIRTUAL_RT.

Quoted 1 line

Only needs a change to the documentation:

Ok, I'll change it.

Quoted 17 lines
docs/replay.txt +0 -0
index 149727e..26dfb6e 100644
@@ -134,11 +134,18 @@ of time. That's why we do not process a group of timers until the

checkpoint
event will be read from the log. Such an event allows synchronizing CPU execution and timer events.

docs/replay.txt +2 -5
-Another checkpoints application in record/replay is instruction counting
-while the virtual machine is idle. This function (qemu_clock_warp) is called
-from the wait loop. It changes virtual machine state and must be deterministic
-then. That is why we added checkpoint to this function to prevent its
-operation in replay mode when it does not correspond to record mode.
+Two other checkpoints govern the "warping" of the virtual clock.  While
+the virtual machine is idle, the virtual clock increments at 1 ns per

docs/replay.txt of time. That's why we do not process a group of timers until the checkpoint · line 148

Quoted 10 lines
docs/replay.txt +10 -0
+*real time* nanosecond.  This is done by setting up a timer (called the
+warp timer) and then incrementing the virtual clock (called "warping"
+the virtual clock) as soon as the CPUs need to go out of the idle state.
+These actions change virtual machine state and must be deterministic.
+Two functions are used for this purpose, and each of them creates a
+checkpoint.  qemu_start_warp_timer checks if the CPUs are idle and if so
+starts accounting real time to virtual clock.  qemu_account_warp_timer
+is called when the CPUs get an interrupt or when a virtual clock timer
+fires, and it warps the virtual clock by the amount of real time that
+has passed since qemu_start_warp_timer.

Pavel Dovgalyuk

#10 lore raw

Re: [Qemu-devel] [PATCH v4 3/5] icount: decouple warp calls

Quoted 23 lines · Pavel Dovgalyuk
Quoted 23 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 20 lines · Pavel Dovgalyuk

qemu_clock_warp function is called to update virtual clock when CPU is sleeping. This function includes replay checkpoint to make execution deterministic in icount mode.
Record/replay module flushes async event queue at checkpoints. Some of the events (e.g., block devices operations) include interaction with hardware. E.g., APIC polled by block devices sets one of IRQ flags. Flag to be set depends on currently executed thread (CPU or iothread). Therefore in replay mode we have to process the checkpoints in the same thread as they were recorded.
qemu_clock_warp function (and its checkpoint) may be called from different thread. This patch decouples two different execution cases of this function:
call when CPU is sleeping from iothread and call from cpu thread to update virtual clock.
First task is performed by qemu_start_warp_timer function. It sets warp timer event to the moment of nearest pending virtual timer. Second function (qemu_account_warp_timer) is called from cpu thread before execution of the code. It advances virtual clock by adding the length of period while CPU was sleeping.

Quoted 7 lines
Quoted 1 line

Lovely. :) One question, why doesn't icount_dummy_timer need a checkpoint?

It is synchronized with CHECKPOINT_CLOCK_VIRTUAL_RT.

Quoted 1 line

Only needs a change to the documentation:

Ok, I'll change it.

No problem, I can do it.

Paolo

Quoted 30 lines
Quoted 17 lines
docs/replay.txt +0 -0
index 149727e..26dfb6e 100644
@@ -134,11 +134,18 @@ of time. That's why we do not process a group of timers until the

checkpoint
event will be read from the log. Such an event allows synchronizing CPU execution and timer events.

docs/replay.txt +2 -5
-Another checkpoints application in record/replay is instruction counting
-while the virtual machine is idle. This function (qemu_clock_warp) is called
-from the wait loop. It changes virtual machine state and must be deterministic
-then. That is why we added checkpoint to this function to prevent its
-operation in replay mode when it does not correspond to record mode.
+Two other checkpoints govern the "warping" of the virtual clock.  While
+the virtual machine is idle, the virtual clock increments at 1 ns per

docs/replay.txt of time. That's why we do not process a group of timers until the checkpoint · line 148

Quoted 10 lines
docs/replay.txt +10 -0
+*real time* nanosecond.  This is done by setting up a timer (called the
+warp timer) and then incrementing the virtual clock (called "warping"
+the virtual clock) as soon as the CPUs need to go out of the idle state.
+These actions change virtual machine state and must be deterministic.
+Two functions are used for this purpose, and each of them creates a
+checkpoint.  qemu_start_warp_timer checks if the CPUs are idle and if so
+starts accounting real time to virtual clock.  qemu_account_warp_timer
+is called when the CPUs get an interrupt or when a virtual clock timer
+fires, and it warps the virtual clock by the amount of real time that
+has passed since qemu_start_warp_timer.

Pavel Dovgalyuk

#3 lore raw

Re: [Qemu-devel] [PATCH v4 1/5] replay: character devices

In reply to Paolo Bonzini
Quoted 6 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 2 lines · Pavel Dovgalyuk

gdbstub which also acts as a backend is not recorded to allow controlling the replaying through gdb.

Perhaps the monitor too?

Right. I'll check that it works.

Quoted 22 lines

Overall the patch is nice and can definitely go in 2.6, but there are a couple changes to do...

qemu-char.c int qemu_chr_fe_write(CharDriverState *s, const uint8_t *buf, int len) · line 251

Quoted 7 lines
qemu-char.c +3 -0
@@ -245,6 +246,9 @@ int qemu_chr_fe_write(CharDriverState *s, const uint8_t *buf, int len)
         qemu_chr_fe_write_log(s, buf, ret);
     }
 
+    if (s->replay) {
+        replay_data_int(&ret);
+    }

I think this is wrong. The logic should be

    if (replaying) {
	read event(&ret);
	assert(ret <= len);
	len = ret;
    }

    qemu_mutex_lock(&s->chr_write_lock);
    ret = s->chr_write(s, buf, len);
Quoted 11 lines
    if (ret > 0) {
        qemu_chr_fe_write_log(s, buf, ret);
    }
    qemu_mutex_unlock(&s->chr_write_lock);

    if (recording) {
        write event(ret);
    }
Quoted 2 lines
qemu-char.c +0 -0
     qemu_mutex_unlock(&s->chr_write_lock);
     return ret;

In this case return value in record and replay modes may differ and the behavior of caller won't be deterministic. E.g.,

static gboolean cadence_uart_xmit(GIOChannel *chan, GIOCondition cond,
void *opaque)
{
...

    ret = qemu_chr_fe_write(s->chr, s->tx_fifo, s->tx_count);
    s->tx_count -= ret;
    memmove(s->tx_fifo, s->tx_fifo + ret, s->tx_count);

...
}

Pavel Dovgalyuk

#4 lore raw

Re: [Qemu-devel] [PATCH v4 1/5] replay: character devices

Quoted 51 lines · Pavel Dovgalyuk
Quoted 6 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 2 lines · Pavel Dovgalyuk

gdbstub which also acts as a backend is not recorded to allow controlling the replaying through gdb.

Perhaps the monitor too?

Right. I'll check that it works.

Quoted 22 lines

Overall the patch is nice and can definitely go in 2.6, but there are a couple changes to do...

qemu-char.c int qemu_chr_fe_write(CharDriverState *s, const uint8_t *buf, int len) · line 251

Quoted 7 lines
qemu-char.c +3 -0
@@ -245,6 +246,9 @@ int qemu_chr_fe_write(CharDriverState *s, const uint8_t *buf, int len)
         qemu_chr_fe_write_log(s, buf, ret);
     }
 
+    if (s->replay) {
+        replay_data_int(&ret);
+    }

I think this is wrong. The logic should be

    if (replaying) {
	read event(&ret);
	assert(ret <= len);
	len = ret;
    }

    qemu_mutex_lock(&s->chr_write_lock);
    ret = s->chr_write(s, buf, len);
Quoted 11 lines
    if (ret > 0) {
        qemu_chr_fe_write_log(s, buf, ret);
    }
    qemu_mutex_unlock(&s->chr_write_lock);

    if (recording) {
        write event(ret);
    }
Quoted 2 lines
qemu-char.c +0 -0
     qemu_mutex_unlock(&s->chr_write_lock);
     return ret;

In this case return value in record and replay modes may differ and the behavior of caller won't be deterministic. E.g.,

static gboolean cadence_uart_xmit(GIOChannel *chan, GIOCondition cond,
void *opaque)
{
...

Quoted 5 lines
    ret = qemu_chr_fe_write(s->chr, s->tx_fifo, s->tx_count);
    s->tx_count -= ret;
    memmove(s->tx_fifo, s->tx_fifo + ret, s->tx_count);

...
}

What you are doing is actually worse. Say you are writing 20 bytes, and at recording time the chardev could only write 10. At replay time, you will write 20 but perhaps the chardev (which is an external program, remember) this time could write 15. Now you have written 15 characters, but you tell the device model that you have written 10. The result is that you'll write the 11th to 15th characters twice.

Likewise you could lose characters if the chardev cannot satisfy the write at replay time. With my version the latter is still possible, but duplicated characters are not.

So perhaps:

    if (replaying) {
        read event(&ret);
        assert(ret <= len);
        ret = qemu_chr_fe_write_all(s, buf, ret);
        return ret;
    }

    qemu_mutex_lock(&s->chr_write_lock);
    ret = s->chr_write(s, buf, len);

    if (ret > 0) {
        qemu_chr_fe_write_log(s, buf, ret);
    }
    qemu_mutex_unlock(&s->chr_write_lock);
    if (recording) {
        write event(ret);
    }
    return ret;

Paolo

#13 lore raw

Re: [Qemu-devel] [PATCH v4 5/5] replay: introduce block devices record/replay

In reply to Pavel Dovgalyuk

replay/replay-events.c void replay_add_event(ReplayAsyncEventKind event_kind, · line 144

Quoted 9 lines · Pavel Dovgalyuk
replay/replay-events.c +1 -1
@@ -135,7 +138,7 @@ void replay_add_event(ReplayAsyncEventKind event_kind,
 
 void replay_bh_schedule_event(QEMUBH *bh)
 {
-    if (replay_mode != REPLAY_MODE_NONE) {
+    if (replay_mode != REPLAY_MODE_NONE && events_enabled) {
         uint64_t id = replay_get_current_step();
         replay_add_event(REPLAY_ASYNC_EVENT_BH, bh, NULL, id);
     } else {

Is this hunk a fix that should be in a separate patch or squashed into a previous patch?

#15 lore raw

Re: [Qemu-devel] [PATCH v4 5/5] replay: introduce block devices record/replay

In reply to Pavel Dovgalyuk

block/blkreplay.c line 80

Quoted 9 lines · Pavel Dovgalyuk
block/blkreplay.c new file +9 -0
+static void block_request_create(uint64_t reqid, BlockDriverState *bs,
+                                 Coroutine *co)
+{
+    Request *req = g_new(Request, 1);
+    *req = (Request) {
+        .co = co,
+        .bh = aio_bh_new(bdrv_get_aio_context(bs), blkreplay_bh_cb, req),
+    };
+    replay_block_event(req->bh, reqid);

Regarding thread safety: is replay_block_event() thread-safe?

If QEMU runs with IOThreads then this might not be called under the QEMU global mutex.

Stefan

#14 lore raw

Re: [Qemu-devel] [PATCH v4 5/5] replay: introduce block devices record/replay

In reply to Stefan Hajnoczi
Quoted 4 lines

From: Stefan Hajnoczi [mailto:[email protected]]

Quoted 2 lines · Pavel Dovgalyuk
(quoted diff) +0 -0
@@ -135,7 +138,7 @@ void replay_add_event(ReplayAsyncEventKind event_kind,
 
Quoted 10 lines

replay/replay-events.c void replay_add_event(ReplayAsyncEventKind event_kind, · line 144

Quoted 7 lines
replay/replay-events.c +1 -1
 void replay_bh_schedule_event(QEMUBH *bh)
 {
-    if (replay_mode != REPLAY_MODE_NONE) {
+    if (replay_mode != REPLAY_MODE_NONE && events_enabled) {
         uint64_t id = replay_get_current_step();
         replay_add_event(REPLAY_ASYNC_EVENT_BH, bh, NULL, id);
     } else {

Is this hunk a fix that should be in a separate patch or squashed into a previous patch?

Right, this is a fix of a separate bug. It manifests itself only with this block patch, but I can separate it for clearness.

Pavel Dovgalyuk

#16 lore raw

Re: [Qemu-devel] [PATCH v4 5/5] replay: introduce block devices record/replay

In reply to Stefan Hajnoczi
Quoted 13 lines

From: Stefan Hajnoczi [mailto:[email protected]]

block/blkreplay.c line 80

Quoted 9 lines · Pavel Dovgalyuk
block/blkreplay.c new file +9 -0
+static void block_request_create(uint64_t reqid, BlockDriverState *bs,
+                                 Coroutine *co)
+{
+    Request *req = g_new(Request, 1);
+    *req = (Request) {
+        .co = co,
+        .bh = aio_bh_new(bdrv_get_aio_context(bs), blkreplay_bh_cb, req),
+    };
+    replay_block_event(req->bh, reqid);

Regarding thread safety: is replay_block_event() thread-safe?

It is. Replay module has its own mutex for file and data structures protection.

Quoted 3 lines

If QEMU runs with IOThreads then this might not be called under the QEMU global mutex.

Pavel Dovgalyuk