Skip to contentSkip to conversation

Deterministic replay extensions

PATCH Superseded Pavel Dovgalyuk 4 participants 14 messages Mar 1, 2016 – Mar 10, 2016 Block layer

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v3 cover Deterministic replay extensions

This set of patches is related to the reverse execution and deterministic replay of qemu execution. It includes recording and replaying of serial devices and block devices operations.

With these patches one can record and deterministically replay behavior of the system with connected disk drives and serial communication ports (e.g., telnet terminal).

Patches for deterministic replay of the block devices intercept calls of bdrv coroutine functions at the top of block drivers stack. To record and replay block operations the drive must be configured as following:
-drive file=disk.qcow,if=none,id=img-direct
-drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay -device ide-hd,drive=img-blkreplay

blkreplay driver should be inserted between disk image and virtual driver controller. Therefore all disk requests may be recorded and replayed.

v3 changes:
- introduced bdrv_flush callback for block drivers
- introduced block driver for recording block operations (as suggested by Kevin Wolf)
- added documentation for block record/replay

v2 changes:
- removed obsolete call of qemu_clock_warp
- fixed record/replay of aio_cancel
- simplified call sequence for blk_aio_ functions in non-replay mode (as suggested by Kevin Wolf)

---

Pavel Dovgalyuk (5):
replay: character devices
icount: remove obsolete warp call
replay: introduce new checkpoint for icount warp
block: add flush callback
replay: introduce block devices record/replay

 block/Makefile.objs       |    2 -
 block/blkreplay.c         |  156 +++++++++++++++++++++++++++++++++++++++++++++
 block/io.c                |    6 ++
 cpus.c                    |   10 +--
 docs/replay.txt           |   20 ++++++
 gdbstub.c                 |    2 -
 include/block/block_int.h |    7 ++
 include/qemu/timer.h      |    3 +
 include/sysemu/char.h     |   26 ++++++++
 include/sysemu/replay.h   |   15 ++++
 main-loop.c               |    2 -
 qemu-char.c               |   56 ++++++++++++++--
 qemu-timer.c              |    2 -
 replay/Makefile.objs      |    1 
 replay/replay-char.c      |   97 ++++++++++++++++++++++++++++
 replay/replay-events.c    |   41 ++++++++++--
 replay/replay-internal.h  |   16 +++++
 replay/replay.c           |   25 +++++++
 stubs/clock-warp.c        |    2 -
 stubs/replay.c            |    4 +
 20 files changed, 469 insertions(+), 24 deletions(-)
 create mode 100755 block/blkreplay.c
 create mode 100755 replay/replay-char.c
Signature · 1 line
Pavel Dovgalyuk
#1 lore raw

PATCH v3 1/5 replay: character devices

This patch implements record and replay of character devices. It records chardevs communication in replay mode. Recorded information include data read from backend and counter of bytes written from frontend to backend to preserve frontend internal state. If character device was configured through the command line in record mode, then in replay mode it should be also added to command line. Backend of the character device could be changed in replay mode.
Replaying of devices that perform ioctl and get_msgfd operations is not supported.
gdbstub which also acts as a backend is not recorded to allow controlling the replaying through gdb.

---
 gdbstub.c                |    2 -
 include/sysemu/char.h    |   26 ++++++++++++
 include/sysemu/replay.h  |   12 ++++++
 qemu-char.c              |   56 ++++++++++++++++++++++++---
 replay/Makefile.objs     |    1 
 replay/replay-char.c     |   97 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-events.c   |   17 +++++++-
 replay/replay-internal.h |   15 +++++++
 replay/replay.c          |   25 +++++++++++-
 9 files changed, 240 insertions(+), 11 deletions(-)
 create mode 100755 replay/replay-char.c
gdbstub.c +1 -1
index 61c12b1..fdcb0ee 100644
@@ -1752,7 +1752,7 @@ int gdbserver_start(const char *device)
             sigaction(SIGINT, &act, NULL);
         }
 #endif
-        chr = qemu_chr_new("gdb", device, NULL);
+        chr = qemu_chr_new_noreplay("gdb", device, NULL);
         if (!chr)
             return -1;
 
include/sysemu/char.h +26 -0
index e035d1c..03fc165 100644
@@ -86,6 +86,7 @@ struct CharDriverState {
     int is_mux;
     guint fd_in_tag;
     QemuOpts *opts;
+    bool replay;
     QTAILQ_ENTRY(CharDriverState) next;
 };
 
@@ -129,6 +130,22 @@ CharDriverState *qemu_chr_new(const char *label, const char *filename,
                               void (*init)(struct CharDriverState *s));
 
 /**
+ * @qemu_chr_new_noreplay:
+ *
+ * Create a new character backend from a URI.
+ * Character device communications are not written
+ * into the replay log.
+ *
+ * @label the name of the backend
+ * @filename the URI
+ * @init not sure..
+ *
+ * Returns: a new character backend
+ */
+CharDriverState *qemu_chr_new_noreplay(const char *label, const char *filename,
+                                       void (*init)(struct CharDriverState *s));
+
+/**
  * @qemu_chr_delete:
  *
  * Destroy a character backend and remove it from the list of
@@ -331,6 +348,15 @@ int qemu_chr_be_can_write(CharDriverState *s);
  */
 void qemu_chr_be_write(CharDriverState *s, uint8_t *buf, int len);
 
+/**
+ * @qemu_chr_be_write_impl:
+ *
+ * Implementation of back end writing. Used by replay module.
+ *
+ * @buf a buffer to receive data from the front end
+ * @len the number of bytes to receive from the front end
+ */
+void qemu_chr_be_write_impl(CharDriverState *s, uint8_t *buf, int len);
 
 /**
  * @qemu_chr_be_event:
include/sysemu/replay.h +12 -0
index abb4688..3c4a988 100644
@@ -117,4 +117,16 @@ void replay_input_event(QemuConsole *src, InputEvent *evt);
 /*! Adds input sync event to the queue */
 void replay_input_sync_event(void);
 
+/* Character device */
+
+/*! Registers char driver to save it's events */
+void replay_register_char_driver(struct CharDriverState *chr);
+/*! Saves write to char device event to the log */
+void replay_chr_be_write(struct CharDriverState *s, uint8_t *buf, int len);
+
+/* Other data */
+
+/*! Writes or reads integer value to/from replay log. */
+void replay_data_int(int *data);
+
 #endif
qemu-char.c +50 -6

101 lines not on this page · raw message

replay/Makefile.objs +1 -0
index 232193a..70e5572 100644
@@ -3,3 +3,4 @@ common-obj-y += replay-internal.o
 common-obj-y += replay-events.o
 common-obj-y += replay-time.o
 common-obj-y += replay-input.o
+common-obj-y += replay-char.o
replay/replay-char.c new file +97 -0

97 lines not on this page · raw message

replay/replay-events.c +14 -3
index 2628109..59d467f 100644
@@ -48,6 +48,9 @@ static void replay_run_event(Event *event)
     case REPLAY_ASYNC_EVENT_INPUT_SYNC:
         qemu_input_event_sync_impl();
         break;
+    case REPLAY_ASYNC_EVENT_CHAR:
+        replay_event_char_run(event->opaque);
+        break;
     default:
         error_report("Replay: invalid async event ID (%d) in the queue",
                     event->event_kind);
@@ -102,9 +105,9 @@ void replay_clear_events(void)
 }
 
 /*! Adds specified async event to the queue */
-static void replay_add_event(ReplayAsyncEventKind event_kind,
-                             void *opaque,
-                             void *opaque2, uint64_t id)
+void replay_add_event(ReplayAsyncEventKind event_kind,
+                      void *opaque,
+                      void *opaque2, uint64_t id)
 {
     assert(event_kind < REPLAY_ASYNC_COUNT);
 
@@ -168,6 +171,9 @@ static void replay_save_event(Event *event, int checkpoint)
             break;
         case REPLAY_ASYNC_EVENT_INPUT_SYNC:
             break;
+        case REPLAY_ASYNC_EVENT_CHAR:
+            replay_event_char_save(event->opaque);
+            break;
         default:
             error_report("Unknown ID %d of replay event", read_event_kind);
             exit(1);
@@ -221,6 +227,11 @@ static Event *replay_read_event(int checkpoint)
         event->event_kind = read_event_kind;
         event->opaque = 0;
         return event;
+    case REPLAY_ASYNC_EVENT_CHAR:
+        event = g_malloc0(sizeof(Event));
+        event->event_kind = read_event_kind;
+        event->opaque = replay_event_char_read();
+        return event;
     default:
         error_report("Unknown ID %d of replay event", read_event_kind);
         exit(1);
replay/replay-internal.h +15 -0
index 77e0d29..34eee5b 100644
@@ -25,6 +25,8 @@ enum ReplayEvents {
     EVENT_ASYNC,
     /* for shutdown request */
     EVENT_SHUTDOWN,
+    /* for int data */
+    EVENT_DATA_INT,
     /* for clock read/writes */
     /* some of greater codes are reserved for clocks */
     EVENT_CLOCK,
@@ -44,6 +46,7 @@ enum ReplayAsyncEventKind {
     REPLAY_ASYNC_EVENT_BH,
     REPLAY_ASYNC_EVENT_INPUT,
     REPLAY_ASYNC_EVENT_INPUT_SYNC,
+    REPLAY_ASYNC_EVENT_CHAR,
     REPLAY_ASYNC_COUNT
 };
 
@@ -125,6 +128,9 @@ bool replay_has_events(void);
 void replay_save_events(int checkpoint);
 /*! Read events from the file into the input queue */
 void replay_read_events(int checkpoint);
+/*! Adds specified async event to the queue */
+void replay_add_event(ReplayAsyncEventKind event_kind, void *opaque,
+                      void *opaque2, uint64_t id);
 
 /* Input events */
 
@@ -137,4 +143,13 @@ void replay_add_input_event(struct InputEvent *event);
 /*! Adds input sync event to the queue */
 void replay_add_input_sync_event(void);
 
+/* Character devices */
+
+/*! Called to run char device event. */
+void replay_event_char_run(void *opaque);
+/*! Writes char event to the file. */
+void replay_event_char_save(void *opaque);
+/*! Reads char event from the file. */
+void *replay_event_char_read(void);
+
 #endif
replay/replay.c +24 -1
index 9cac178..dfb37dc 100644
@@ -20,7 +20,7 @@
 
 /* Current version of the replay mechanism.
    Increase it when file format changes. */
-#define REPLAY_VERSION              0xe02002
+#define REPLAY_VERSION              0xe02003
 /* Size of replay log header */
 #define HEADER_SIZE                 (sizeof(uint32_t) + sizeof(uint64_t))
 
@@ -340,3 +340,26 @@ void replay_add_blocker(Error *reason)
 {
     replay_blockers = g_slist_prepend(replay_blockers, reason);
 }
+
+void replay_data_int(int *data)
+{
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        replay_account_executed_instructions();
+        replay_mutex_lock();
+        if (replay_next_event_is(EVENT_DATA_INT)) {
+            *data = replay_get_dword();
+            replay_finish_event();
+            replay_mutex_unlock();
+        } else {
+            replay_mutex_unlock();
+            error_report("Missing data int event in the replay log");
+            exit(1);
+        }
+    } else if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_save_instructions();
+        replay_mutex_lock();
+        replay_put_event(EVENT_DATA_INT);
+        replay_put_dword(*data);
+        replay_mutex_unlock();
+    }
+}
#2 lore raw

PATCH v3 2/5 icount: remove obsolete warp call

In reply to Pavel Dovgalyuk

qemu_clock_warp call in qemu_tcg_wait_io_event function is not needed anymore, because it is called in every iteration of main_loop_wait.

---
 cpus.c |    3 ---
 1 files changed, 0 insertions(+), 3 deletions(-)
cpus.c +0 -3
index 898426c..01c9809 100644
@@ -995,9 +995,6 @@ static void qemu_wait_io_event_common(CPUState *cpu)
 static void qemu_tcg_wait_io_event(CPUState *cpu)
 {
     while (all_cpu_threads_idle()) {
-       /* Start accounting real time to the virtual clock if the CPUs
-          are idle.  */
-        qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
         qemu_cond_wait(cpu->halt_cond, &qemu_global_mutex);
     }
 
Reviewed-by on v3 2/5
#3 lore raw

Re: [Qemu-devel] [PATCH v3 2/5] icount: remove obsolete warp call

Quoted 14 lines · Pavel Dovgalyuk

qemu_clock_warp call in qemu_tcg_wait_io_event function is not needed anymore, because it is called in every iteration of main_loop_wait.

---
 cpus.c |    3 ---
 1 files changed, 0 insertions(+), 3 deletions(-)
cpus.c +0 -0
index 898426c..01c9809 100644

cpus.c static void qemu_wait_io_event_common(CPUState *cpu) · line 1000

Quoted 9 lines
cpus.c +0 -3
@@ -995,9 +995,6 @@ static void qemu_wait_io_event_common(CPUState *cpu)
 static void qemu_tcg_wait_io_event(CPUState *cpu)
 {
     while (all_cpu_threads_idle()) {
-       /* Start accounting real time to the virtual clock if the CPUs
-          are idle.  */
-        qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
         qemu_cond_wait(cpu->halt_cond, &qemu_global_mutex);
     }
 
#4 lore raw

PATCH v3 3/5 replay: introduce new checkpoint for icount warp

In reply to Pavel Dovgalyuk

qemu_clock_warp function is called to update virtual clock when CPU is sleeping. This function includes replay checkpoint to make execution deterministic in icount mode.
Record/replay module flushes async event queue at checkpoints. Some of the events (e.g., block devices operations) include interaction with hardware. E.g., APIC polled by block devices sets one of IRQ flags. Flag to be set depends on currently executed thread (CPU or iothread). Therefore in replay mode we have to process the checkpoints in the same thread as they were recorded.
qemu_clock_warp function (and its checkpoint) may be called from different thread. This patch introduces new checkpoint which distinguished warp checkpoint calls from different threads.

---
 cpus.c                  |    7 ++++---
 include/qemu/timer.h    |    3 ++-
 include/sysemu/replay.h |    1 +
 main-loop.c             |    2 +-
 qemu-timer.c            |    2 +-
 stubs/clock-warp.c      |    2 +-
 6 files changed, 10 insertions(+), 7 deletions(-)
cpus.c +4 -3
index 01c9809..c2d9cfe 100644
@@ -396,7 +396,7 @@ void qtest_clock_warp(int64_t dest)
     qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
 }
 
-void qemu_clock_warp(QEMUClockType type)
+void qemu_clock_warp(QEMUClockType type, bool in_tcg)
 {
     int64_t clock;
     int64_t deadline;
@@ -418,7 +418,8 @@ void qemu_clock_warp(QEMUClockType type)
     }
 
     /* warp clock deterministically in record/replay mode */
-    if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP)) {
+    if (!replay_checkpoint(in_tcg ? CHECKPOINT_CLOCK_WARP_TCG
+                                  : CHECKPOINT_CLOCK_WARP)) {
         return;
     }
 
@@ -1496,7 +1497,7 @@ static void tcg_exec_all(void)
     int r;
 
     /* Account partial waits to QEMU_CLOCK_VIRTUAL.  */
-    qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
+    qemu_clock_warp(QEMU_CLOCK_VIRTUAL, true);
 
     if (next_cpu == NULL) {
         next_cpu = first_cpu;
include/qemu/timer.h +2 -1
index d0946cb..c58192c 100644
@@ -212,10 +212,11 @@ void qemu_clock_enable(QEMUClockType type, bool enabled);
 /**
  * qemu_clock_warp:
  * @type: the clock type
+ * @in_tcg: true if function is called from TCG CPU thread
  *
  * Warp a clock to a new value
  */
-void qemu_clock_warp(QEMUClockType type);
+void qemu_clock_warp(QEMUClockType type, bool in_tcg);
 
 /**
  * qemu_clock_register_reset_notifier:
include/sysemu/replay.h +1 -0
index 3c4a988..c879231 100644
@@ -31,6 +31,7 @@ typedef enum ReplayClockKind ReplayClockKind;
 /* IDs of the checkpoints */
 enum ReplayCheckpoint {
     CHECKPOINT_CLOCK_WARP,
+    CHECKPOINT_CLOCK_WARP_TCG,
     CHECKPOINT_RESET_REQUESTED,
     CHECKPOINT_SUSPEND_REQUESTED,
     CHECKPOINT_CLOCK_VIRTUAL,
main-loop.c +1 -1
index 19beae7..cd8415f 100644
@@ -509,7 +509,7 @@ int main_loop_wait(int nonblocking)
 
     /* CPU thread can infinitely wait for event after
        missing the warp */
-    qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
+    qemu_clock_warp(QEMU_CLOCK_VIRTUAL, false);
     qemu_clock_run_all_timers();
 
     return ret;
qemu-timer.c +1 -1
index e98ecc9..980fe7e 100644
@@ -394,7 +394,7 @@ static bool timer_mod_ns_locked(QEMUTimerList *timer_list,
 static void timerlist_rearm(QEMUTimerList *timer_list)
 {
     /* Interrupt execution to force deadline recalculation.  */
-    qemu_clock_warp(timer_list->clock->type);
+    qemu_clock_warp(timer_list->clock->type, false);
     timerlist_notify(timer_list);
 }
 
stubs/clock-warp.c +1 -1
index 5ae32b9..24ae0f8 100644
@@ -2,7 +2,7 @@
 #include "qemu-common.h"
 #include "qemu/timer.h"
 
-void qemu_clock_warp(QEMUClockType type)
+void qemu_clock_warp(QEMUClockType type, bool in_tcg)
 {
 }
 
#5 lore raw

Re: [Qemu-devel] [PATCH v3 3/5] replay: introduce new checkpoint for icount warp

Quoted 13 lines · Pavel Dovgalyuk

qemu_clock_warp function is called to update virtual clock when CPU is sleeping. This function includes replay checkpoint to make execution deterministic in icount mode.
Record/replay module flushes async event queue at checkpoints. Some of the events (e.g., block devices operations) include interaction with hardware. E.g., APIC polled by block devices sets one of IRQ flags. Flag to be set depends on currently executed thread (CPU or iothread). Therefore in replay mode we have to process the checkpoints in the same thread as they were recorded.
qemu_clock_warp function (and its checkpoint) may be called from different thread. This patch introduces new checkpoint which distinguished warp checkpoint calls from different threads.

Quoted 1 line

I think we need two different kinds of "warp" behavior, one to start the warp timer (from the main loop and when a timer is set) and one to end it (from the CPUs).

Then the need for two checkpoints is much clearer, though I suggestnaming them without a reference to TCG; something like CHECKPOINT_CLOCK_WARP_START and CHECKPOINT_CLOCK_WARP_ACCOUNT for example.

The start would be where you call qemu_clock_warp(QEMU_CLOCK_VIRTUAL, false):

    if (!use_icount) {
        return;
    }
    if (!runstate_is_running()) {
        return;
    }
    if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP_START)) {
        return;
    }
    /* I think calling icount_warp_rt here is unnecessary.  */
    if (!all_cpu_threads_idle()) {
        return;
    }
    if (qtest_enabled()) {
        /* When testing, qtest commands advance icount.  */
        return;
    }

    /* We want to use the earliest deadline from ALL vm_clocks */
    clock = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL_RT);
    deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
    ...

The end or account function, instead, would be called from tcg_exec_all() and also from icount_dummy_timer() (this is what makes the call to icount_warp_rt unnecessary above):

    if (!use_icount || !icount_isleep) {
        return;
    }
    if (!runstate_is_running()) {
        return;
    }
    if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP_END)) {
        return;
    }

    timer_del(icount_warp_timer);
    /*
     * If the CPUs have been sleeping, advance QEMU_CLOCK_VIRTUAL timer now.
     * This ensures that the deadline for the timer is computed correctly
     * below.
     * This also makes sure that the insn counter is synchronized before
     * the CPU starts running, in case the CPU is woken by an event other
     * than the earliest QEMU_CLOCK_VIRTUAL timer.
     */
    // ...include icount_warp_rt function here...

qemu_clock_warp would only be called from qemu-timer.c, and it would be simply be

    if (type == QEMU_CLOCK_VIRTUAL) {
        qemu_start_warp_timer();
    }

Separating the two boundaries of the warp this way would make the code much easier to understand, and would also make the need for a new checkpoint more obvious.

Paolo

#6 lore raw

Re: [Qemu-devel] [PATCH v3 3/5] replay: introduce new checkpoint for icount warp

Quoted 17 lines

From: Paolo Bonzini [mailto:[email protected]]

Quoted 13 lines · Pavel Dovgalyuk

qemu_clock_warp function is called to update virtual clock when CPU is sleeping. This function includes replay checkpoint to make execution deterministic in icount mode.
Record/replay module flushes async event queue at checkpoints. Some of the events (e.g., block devices operations) include interaction with hardware. E.g., APIC polled by block devices sets one of IRQ flags. Flag to be set depends on currently executed thread (CPU or iothread). Therefore in replay mode we have to process the checkpoints in the same thread as they were recorded.
qemu_clock_warp function (and its checkpoint) may be called from different thread. This patch introduces new checkpoint which distinguished warp checkpoint calls from different threads.

Quoted 1 line
Quoted 7 lines

I think we need two different kinds of "warp" behavior, one to start the warp timer (from the main loop and when a timer is set) and one to end it (from the CPUs).

Then the need for two checkpoints is much clearer, though I suggestnaming them without a reference to TCG; something like CHECKPOINT_CLOCK_WARP_START and CHECKPOINT_CLOCK_WARP_ACCOUNT for example.

Thanks, this seems reasonable.

Quoted 25 lines

The start would be where you call qemu_clock_warp(QEMU_CLOCK_VIRTUAL, false):

    if (!use_icount) {
        return;
    }
    if (!runstate_is_running()) {
        return;
    }
    if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP_START)) {
        return;
    }
    /* I think calling icount_warp_rt here is unnecessary.  */
    if (!all_cpu_threads_idle()) {
        return;
    }
    if (qtest_enabled()) {
        /* When testing, qtest commands advance icount.  */
        return;
    }

    /* We want to use the earliest deadline from ALL vm_clocks */
    clock = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL_RT);
    deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
    ...
Quoted 3 lines

The end or account function, instead, would be called from tcg_exec_all() and also from icount_dummy_timer() (this is what makes the call to icount_warp_rt unnecessary above):

Why icount_warp_rt is unnecessary? There is no code to proceed the virtual clock. Then qemu_start_warp_timer will forever setup the timer without any progress.

Pavel Dovgalyuk

#7 lore raw

Re: [Qemu-devel] [PATCH v3 3/5] replay: introduce new checkpoint for icount warp

Quoted 25 lines · Pavel Dovgalyuk
Quoted 25 lines
Quoted 25 lines

The start would be where you call qemu_clock_warp(QEMU_CLOCK_VIRTUAL, false):

    if (!use_icount) {
        return;
    }
    if (!runstate_is_running()) {
        return;
    }
    if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP_START)) {
        return;
    }
    /* I think calling icount_warp_rt here is unnecessary.  */
    if (!all_cpu_threads_idle()) {
        return;
    }
    if (qtest_enabled()) {
        /* When testing, qtest commands advance icount.  */
        return;
    }

    /* We want to use the earliest deadline from ALL vm_clocks */
    clock = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL_RT);
    deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
    ...
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

The end or account function, instead, would be called from tcg_exec_all() and also from icount_dummy_timer() (this is what makes the call to icount_warp_rt unnecessary above):

Why icount_warp_rt is unnecessary? There is no code to proceed the virtual clock. Then qemu_start_warp_timer will forever setup the timer without any progress.

If icount_warp_rt is called from icount_dummy_timer(), the virtual clock will be updated as soon as the VIRTUAL_RT clock reaches the deadline. It's a much more reasonable place to call icount_warp_rt from (if it works...).

Paolo

#8 lore raw

PATCH v3 4/5 block: add flush callback

In reply to Pavel Dovgalyuk

This patch adds callback for flush request. This callback is responsible for flushing whole block devices stack. bdrv_flush function does not proceed to underlying devices. It should be performed by this callback function, if needed.

---
 block/io.c                |    6 ++++++
 include/block/block_int.h |    7 +++++++
 2 files changed, 13 insertions(+), 0 deletions(-)
block/io.c +6 -0
index a69bfc4..9e05dfe 100644
@@ -2369,6 +2369,12 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
     }
 
     tracked_request_begin(&req, bs, 0, 0, BDRV_TRACKED_FLUSH);
+    /* Write back all layers by calling one driver function */
+    if (bs->drv->bdrv_co_flush) {
+        ret = bs->drv->bdrv_co_flush(bs);
+        goto out;
+    }
+    
     /* Write back cached data to the OS even with cache=unsafe */
     BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_OS);
     if (bs->drv->bdrv_co_flush_to_os) {
include/block/block_int.h +7 -0
index 9ef823a..9cc2c58 100644
@@ -176,6 +176,13 @@ struct BlockDriver {
     int (*bdrv_inactivate)(BlockDriverState *bs);
 
     /*
+     * Flushes all data for all layers by calling bdrv_co_flush for underlying
+     * layers, if needed. This function is needed for deterministic
+     * synchronization of the flush finishing callback.
+     */
+    int coroutine_fn (*bdrv_co_flush)(BlockDriverState *bs);
+    
+    /*
      * Flushes all data that was already written to the OS all the way down to
      * the disk (for example raw-posix calls fsync()).
      */
#9 lore raw

Re: [Qemu-devel] [PATCH v3 4/5] block: add flush callback

Am 01.03.2016 um 12:07 hat Pavel Dovgalyuk geschrieben:

Quoted 16 lines

This patch adds callback for flush request. This callback is responsible for flushing whole block devices stack. bdrv_flush function does not proceed to underlying devices. It should be performed by this callback function, if needed.

---
 block/io.c                |    6 ++++++
 include/block/block_int.h |    7 +++++++
 2 files changed, 13 insertions(+), 0 deletions(-)
block/io.c +0 -0
index a69bfc4..9e05dfe 100644

block/io.c int coroutine_fn bdrv_co_flush(BlockDriverState *bs) · line 2377

Quoted 9 lines
block/io.c +6 -0
@@ -2369,6 +2369,12 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
     }
 
     tracked_request_begin(&req, bs, 0, 0, BDRV_TRACKED_FLUSH);
+    /* Write back all layers by calling one driver function */
+    if (bs->drv->bdrv_co_flush) {
+        ret = bs->drv->bdrv_co_flush(bs);
+        goto out;
+    }
+    

Trailing whitespace.

Also, while you update the formatting, how about an empty line before the comment?

Quoted 8 lines
block/io.c +0 -0
     /* Write back cached data to the OS even with cache=unsafe */
     BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_OS);
     if (bs->drv->bdrv_co_flush_to_os) {
include/block/block_int.h +0 -0
index 9ef823a..9cc2c58 100644

include/block/block_int.h struct BlockDriver { · line 184

Quoted 9 lines
include/block/block_int.h +6 -0
@@ -176,6 +176,13 @@ struct BlockDriver {
     int (*bdrv_inactivate)(BlockDriverState *bs);
 
     /*
+     * Flushes all data for all layers by calling bdrv_co_flush for underlying
+     * layers, if needed. This function is needed for deterministic
+     * synchronization of the flush finishing callback.
+     */
+    int coroutine_fn (*bdrv_co_flush)(BlockDriverState *bs);
+    

Trailing whitespace again.

Otherwise, the patch looks good.

Kevin

#10 lore raw

PATCH v3 5/5 replay: introduce block devices record/replay

In reply to Pavel Dovgalyuk

This patch introduces block driver that implement recording and replaying of block devices' operations.
All block completion operations are added to the queue. Queue is flushed at checkpoints and information about processed requests is recorded to the log. In replay phase the queue is matched with events read from the log. Therefore block devices requests are processed deterministically.

---
 block/Makefile.objs      |    2 -
 block/blkreplay.c        |  156 ++++++++++++++++++++++++++++++++++++++++++++++
 docs/replay.txt          |   20 ++++++
 include/sysemu/replay.h  |    2 +
 replay/replay-events.c   |   24 ++++++-
 replay/replay-internal.h |    1 
 stubs/replay.c           |    4 +
 7 files changed, 206 insertions(+), 3 deletions(-)
 create mode 100755 block/blkreplay.c
block/Makefile.objs +1 -1
index 58ef2ef..38fea16 100644
@@ -4,7 +4,7 @@ block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o
 block-obj-y += qed-check.o
 block-obj-$(CONFIG_VHDX) += vhdx.o vhdx-endian.o vhdx-log.o
 block-obj-y += quorum.o
-block-obj-y += parallels.o blkdebug.o blkverify.o
+block-obj-y += parallels.o blkdebug.o blkverify.o blkreplay.o
 block-obj-y += block-backend.o snapshot.o qapi.o
 block-obj-$(CONFIG_WIN32) += raw-win32.o win32-aio.o
 block-obj-$(CONFIG_POSIX) += raw-posix.o
block/blkreplay.c new file +156 -0

156 lines not on this page · raw message

docs/replay.txt +20 -0
index 149727e..acf5031 100644
@@ -166,3 +166,23 @@ Sometimes the block layer uses asynchronous callbacks for its internal purposes
 (like reading or writing VM snapshots or disk image cluster tables). In this
 case bottom halves are not marked as "replayable" and do not saved
 into the log.
+
+Block devices
+-------------
+
+Block devices record/replay module intercepts calls of
+bdrv coroutine functions at the top of block drivers stack.
+To record and replay block operations the drive must be configured
+as following:
+ -drive file=disk.qcow,if=none,id=img-direct
+ -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay
+ -device ide-hd,drive=img-blkreplay
+
+blkreplay driver should be inserted between disk image and virtual driver
+controller. Therefore all disk requests may be recorded and replayed.
+
+All block completion operations are added to the queue in the coroutines.
+Queue is flushed at checkpoints and information about processed requests
+is recorded to the log. In replay phase the queue is matched with
+events read from the log. Therefore block devices requests are processed
+deterministically.
include/sysemu/replay.h +2 -0
index c879231..0817dc5 100644
@@ -117,6 +117,8 @@ void replay_bh_schedule_event(QEMUBH *bh);
 void replay_input_event(QemuConsole *src, InputEvent *evt);
 /*! Adds input sync event to the queue */
 void replay_input_sync_event(void);
+/*! Adds block layer event to the queue */
+void replay_block_event(QEMUBH *bh, uint64_t id);
 
 /* Character device */
 
replay/replay-events.c +22 -2
index 59d467f..5a7042b 100644
@@ -51,6 +51,9 @@ static void replay_run_event(Event *event)
     case REPLAY_ASYNC_EVENT_CHAR:
         replay_event_char_run(event->opaque);
         break;
+    case REPLAY_ASYNC_EVENT_BLOCK:
+        aio_bh_call(event->opaque);
+        break;
     default:
         error_report("Replay: invalid async event ID (%d) in the queue",
                     event->event_kind);
@@ -135,7 +138,7 @@ void replay_add_event(ReplayAsyncEventKind event_kind,
 
 void replay_bh_schedule_event(QEMUBH *bh)
 {
-    if (replay_mode != REPLAY_MODE_NONE) {
+    if (replay_mode != REPLAY_MODE_NONE && events_enabled) {
         uint64_t id = replay_get_current_step();
         replay_add_event(REPLAY_ASYNC_EVENT_BH, bh, NULL, id);
     } else {
@@ -153,6 +156,15 @@ void replay_add_input_sync_event(void)
     replay_add_event(REPLAY_ASYNC_EVENT_INPUT_SYNC, NULL, NULL, 0);
 }
 
+void replay_block_event(QEMUBH *bh, uint64_t id)
+{
+    if (replay_mode != REPLAY_MODE_NONE && events_enabled) {
+        replay_add_event(REPLAY_ASYNC_EVENT_BLOCK, bh, NULL, id);
+    } else {
+        qemu_bh_schedule(bh);
+    }
+}
+
 static void replay_save_event(Event *event, int checkpoint)
 {
     if (replay_mode != REPLAY_MODE_PLAY) {
@@ -174,8 +186,11 @@ static void replay_save_event(Event *event, int checkpoint)
         case REPLAY_ASYNC_EVENT_CHAR:
             replay_event_char_save(event->opaque);
             break;
+        case REPLAY_ASYNC_EVENT_BLOCK:
+            replay_put_qword(event->id);
+            break;
         default:
-            error_report("Unknown ID %d of replay event", read_event_kind);
+            error_report("Unknown ID %d of replay event", event->id);
             exit(1);
         }
     }
@@ -232,6 +247,11 @@ static Event *replay_read_event(int checkpoint)
         event->event_kind = read_event_kind;
         event->opaque = replay_event_char_read();
         return event;
+    case REPLAY_ASYNC_EVENT_BLOCK:
+        if (read_id == -1) {
+            read_id = replay_get_qword();
+        }
+        break;
     default:
         error_report("Unknown ID %d of replay event", read_event_kind);
         exit(1);
replay/replay-internal.h +1 -0
index 34eee5b..0363bb5 100644
@@ -47,6 +47,7 @@ enum ReplayAsyncEventKind {
     REPLAY_ASYNC_EVENT_INPUT,
     REPLAY_ASYNC_EVENT_INPUT_SYNC,
     REPLAY_ASYNC_EVENT_CHAR,
+    REPLAY_ASYNC_EVENT_BLOCK,
     REPLAY_ASYNC_COUNT
 };
 
stubs/replay.c +4 -0
index 00ca01f..6f4a8e8 100644
@@ -29,3 +29,7 @@ bool replay_events_enabled(void)
 void replay_finish(void)
 {
 }
+
+void replay_block_event(QEMUBH *bh, uint64_t id)
+{
+}
#11 lore raw

Re: [Qemu-devel] [PATCH v3 5/5] replay: introduce block devices record/replay

Am 01.03.2016 um 12:08 hat Pavel Dovgalyuk geschrieben:

Quoted 9 lines

This patch introduces block driver that implement recording and replaying of block devices' operations.
All block completion operations are added to the queue. Queue is flushed at checkpoints and information about processed requests is recorded to the log. In replay phase the queue is matched with events read from the log. Therefore block devices requests are processed deterministically.

I like this new version much better than the old one. :-)

Quoted 9 lines
block/blkreplay.c new file +9 -0
+static int blkreplay_open(BlockDriverState *bs, QDict *options, int flags,
+                          Error **errp)
+{
+    Error *local_err = NULL;
+    int ret;
+
+    /* Open the image file */
+    bs->file = bdrv_open_child(NULL, options, "image",
+                               bs, &child_file, false, &local_err);

block/blkreplay.c line 44

Quoted 10 lines
block/blkreplay.c new file +10 -0
+    if (local_err) {
+        ret = -EINVAL;
+        error_propagate(errp, local_err);
+        goto fail;
+    }
+
+    ret = 0;
+fail:
+    if (ret < 0) {
+        bdrv_unref_child(bs, bs->file);

This is unnecessary because in error cases, bdrv_open_child() returns NULL. On the other hand, bdrv_unref_child() accepts a NULL child and just doesn't do anything then, so it's harmless.

Quoted 10 lines
block/blkreplay.c new file +10 -0
+    }
+    return ret;
+}
+
+static void blkreplay_close(BlockDriverState *bs)
+{
+}
+
+static int64_t blkreplay_getlength(BlockDriverState *bs)
+{

block/blkreplay.c line 64

Quoted 10 lines
block/blkreplay.c new file +10 -0
+    return bdrv_getlength(bs->file->bs);
+}
+
+static void blkreplay_bh_cb(void *opaque)
+{
+    Request *req = opaque;
+    qemu_coroutine_enter(req->co, NULL);
+    qemu_bh_delete(req->bh);
+    g_free(req);
+}

Maybe add a comment here that explains why the BH exists (forcing the coroutine into its I/O thread, if I understand correctly).

block/blkreplay.c line 71

Quoted 6 lines
block/blkreplay.c new file +6 -0
+static void block_request_create(uint64_t reqid, BlockDriverState *bs,
+                                 Coroutine *co)
+{
+    Request *req = g_malloc0(sizeof(Request));
+    req->co = co;
+    req->bh = aio_bh_new(bdrv_get_aio_context(bs), blkreplay_bh_cb, req);

g_malloc0() zeroes all fields and then you immediately overwrite them, so this is useless work. If you just want to be protected when adding new fields, you could consider this instead:

Request *req = g_new(Request, 1);
*req = (Request) {
.co = co,
.bh = aio_bh_new(bdrv_get_aio_context(bs), blkreplay_bh_cb, req),
};

It's just a suggestion, though, feel free to change it or leave it.

Quoted 4 lines
block/blkreplay.c new file +4 -0
+    replay_block_event(req->bh, reqid);
+}
+
+static int coroutine_fn blkreplay_co_readv(BlockDriverState *bs,

block/blkreplay.c line 85

Quoted 10 lines
block/blkreplay.c new file +10 -0
+    int64_t sector_num, int nb_sectors, QEMUIOVector *qiov)
+{
+    uint64_t reqid = request_id++;
+
+    bdrv_co_readv(bs->file->bs, sector_num, nb_sectors, qiov);
+    block_request_create(reqid, bs, qemu_coroutine_self());
+    qemu_coroutine_yield();
+
+    return 0;
+}

Error handling is broken here. You need to forward the return value of bdrv_co_readv() to the caller.

block/blkreplay.c line 97

Quoted 12 lines
block/blkreplay.c new file +12 -0
+
+static int coroutine_fn blkreplay_co_writev(BlockDriverState *bs,
+    int64_t sector_num, int nb_sectors, QEMUIOVector *qiov)
+{
+    uint64_t reqid = request_id++;
+
+    bdrv_co_writev(bs->file->bs, sector_num, nb_sectors, qiov);
+    block_request_create(reqid, bs, qemu_coroutine_self());
+    qemu_coroutine_yield();
+
+    return 0;
+}

Same here.

block/blkreplay.c line 109

Quoted 11 lines
block/blkreplay.c new file +11 -0
+static int coroutine_fn blkreplay_co_write_zeroes(BlockDriverState *bs,
+    int64_t sector_num, int nb_sectors, BdrvRequestFlags flags)
+{
+    uint64_t reqid = request_id++;
+
+    bdrv_co_write_zeroes(bs->file->bs, sector_num, nb_sectors, flags);
+    block_request_create(reqid, bs, qemu_coroutine_self());
+    qemu_coroutine_yield();
+
+    return 0;
+}

And here.

block/blkreplay.c line 121

Quoted 11 lines
block/blkreplay.c new file +11 -0
+static int coroutine_fn blkreplay_co_discard(BlockDriverState *bs,
+    int64_t sector_num, int nb_sectors)
+{
+    uint64_t reqid = request_id++;
+
+    bdrv_co_discard(bs->file->bs, sector_num, nb_sectors);
+    block_request_create(reqid, bs, qemu_coroutine_self());
+    qemu_coroutine_yield();
+
+    return 0;
+}

Here, too.

block/blkreplay.c line 132

Quoted 10 lines
block/blkreplay.c new file +10 -0
+static int coroutine_fn blkreplay_co_flush(BlockDriverState *bs)
+{
+    uint64_t reqid = request_id++;
+
+    bdrv_co_flush(bs->file->bs);
+    block_request_create(reqid, bs, qemu_coroutine_self());
+    qemu_coroutine_yield();
+
+    return 0;
+}

And finally here as well.

Kevin

#12 lore raw

Re: [Qemu-devel] [PATCH v3 5/5] replay: introduce block devices record/replay

Quoted 10 lines

From: Kevin Wolf [mailto:[email protected]]
Am 01.03.2016 um 12:08 hat Pavel Dovgalyuk geschrieben:

Quoted 8 lines

This patch introduces block driver that implement recording and replaying of block devices' operations.
All block completion operations are added to the queue. Queue is flushed at checkpoints and information about processed requests is recorded to the log. In replay phase the queue is matched with events read from the log. Therefore block devices requests are processed deterministically.

Quoted 3 lines
Quoted 1 line

I like this new version much better than the old one. :-)

Thanks for reviewing!

Quoted 15 lines
Quoted 15 lines
block/blkreplay.c new file +15 -0
+static int blkreplay_open(BlockDriverState *bs, QDict *options, int flags,
+                          Error **errp)
+{
+    Error *local_err = NULL;
+    int ret;
+
+    /* Open the image file */
+    bs->file = bdrv_open_child(NULL, options, "image",
+                               bs, &child_file, false, &local_err);
+    if (local_err) {
+        ret = -EINVAL;
+        error_propagate(errp, local_err);
+        goto fail;
+    }
+
Quoted 8 lines

block/blkreplay.c line 44

Quoted 4 lines
block/blkreplay.c new file +4 -0
+    ret = 0;
+fail:
+    if (ret < 0) {
+        bdrv_unref_child(bs, bs->file);

This is unnecessary because in error cases, bdrv_open_child() returns NULL. On the other hand, bdrv_unref_child() accepts a NULL child and just doesn't do anything then, so it's harmless.

Right, but this may be useful in case of future changes (if something else will be initialized here).

Pavel Dovgalyuk

#13 lore raw
In reply to Pavel Dovgalyuk

Ping?

Pavel Dovgalyuk

Pavel Dovgalyuk писал 2016-03-01 14:07:

Quoted 65 lines

This set of patches is related to the reverse execution and deterministic replay of qemu execution. It includes recording and replaying of serial devices and block devices operations.

With these patches one can record and deterministically replay behavior of the system with connected disk drives and serial communication ports (e.g., telnet terminal).

Patches for deterministic replay of the block devices intercept calls of bdrv coroutine functions at the top of block drivers stack. To record and replay block operations the drive must be configured as following:
-drive file=disk.qcow,if=none,id=img-direct
-drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay -device ide-hd,drive=img-blkreplay

blkreplay driver should be inserted between disk image and virtual driver controller. Therefore all disk requests may be recorded and replayed.

v3 changes:
- introduced bdrv_flush callback for block drivers
- introduced block driver for recording block operations (as
suggested by Kevin Wolf)
- added documentation for block record/replay

v2 changes:
- removed obsolete call of qemu_clock_warp
- fixed record/replay of aio_cancel
- simplified call sequence for blk_aio_ functions in non-replay mode
(as suggested by Kevin Wolf)

---

Pavel Dovgalyuk (5):
replay: character devices
icount: remove obsolete warp call
replay: introduce new checkpoint for icount warp
block: add flush callback
replay: introduce block devices record/replay

 block/Makefile.objs       |    2 -
 block/blkreplay.c         |  156 
+++++++++++++++++++++++++++++++++++++++++++++
 block/io.c                |    6 ++
 cpus.c                    |   10 +--
 docs/replay.txt           |   20 ++++++
 gdbstub.c                 |    2 -
 include/block/block_int.h |    7 ++
 include/qemu/timer.h      |    3 +
 include/sysemu/char.h     |   26 ++++++++
 include/sysemu/replay.h   |   15 ++++
 main-loop.c               |    2 -
 qemu-char.c               |   56 ++++++++++++++--
 qemu-timer.c              |    2 -
 replay/Makefile.objs      |    1
 replay/replay-char.c      |   97 ++++++++++++++++++++++++++++
 replay/replay-events.c    |   41 ++++++++++--
 replay/replay-internal.h  |   16 +++++
 replay/replay.c           |   25 +++++++
 stubs/clock-warp.c        |    2 -
 stubs/replay.c            |    4 +
 20 files changed, 469 insertions(+), 24 deletions(-)
 create mode 100755 block/blkreplay.c
 create mode 100755 replay/replay-char.c