Skip to content

Deterministic replay core

Pavel Dovgalyuk qemu-devel 13 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v18 Latest21 patches 12 repliesIn review
  2. v1721 patches 2 repliesSuperseded
  3. v1621 patches 4 repliesSuperseded
Earlier versions · 10
VersionPostedPatchesRepliesStatus
v1521 patches3 repliesSuperseded
v1421 patches2 repliesSuperseded
v13 RFC21 patches3 repliesSuperseded
v12 RFC21 patches4 repliesSuperseded
v11 RFC23 patches2 repliesSuperseded
v10 RFC24 patches3 repliesSuperseded
v9 RFC23 patches20 repliesSuperseded
v8 RFC21 patches60 repliesSuperseded
v7 RFC21 patches50 repliesSuperseded
v6 RFC32 patches7 repliesSuperseded

Changes in v18 from the cover letter

* Patches were updated to match upstream version
* Added missed replay-user.c file

Patches in v18 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/21i386: partial revert of interrupt poll fixcpu-exec.c target-i386/cpu.c+11 -80 replies
2/21replay: global variables and function stubsMakefile.target docs/replay.txt qapi-schema.json +6 more+238 -00 replies
3/21replay: internal functions for replay logreplay/Makefile.objs replay/replay-internal.c replay/replay-internal.h+202 -00 replies
4/21replay: introduce mutex to protect the replay logreplay/replay-internal.c replay/replay-internal.h+34 -00 replies
5/21replay: introduce icount eventreplay/replay-internal.c replay/replay-internal.h replay/replay.c +1 more+85 -00 replies
6/21cpu-exec: allow temporary disabling icountcpu-exec.c include/exec/exec-all.h translate-all.c+6 -40 replies
7/21cpu: replay instructions sequencecpus.c replay/replay.c replay/replay.h+63 -130 replies
8/21i386: interrupt poll processingtarget-i386/seg_helper.c+3 -00 replies
9/21replay: interrupts and exceptionscpu-exec.c replay/replay-internal.h replay/replay-user.c +2 more+146 -100 replies
10/21replay: asynchronous events infrastructurereplay/Makefile.objs replay/replay-events.c replay/replay-internal.h +1 more+261 -00 replies
11/21replay: recording and replaying clock tickscpus.c qemu-timer.c replay/Makefile.objs +4 more+119 -42 replies
12/21replay: shutdown eventreplay/replay-internal.h replay/replay.c replay/replay.h +1 more+22 -00 replies
13/21icount: improve counting for record/replaycpus.c main-loop.c+11 -34 replies
14/21replay: checkpointscpus.c qemu-timer.c replay/replay-internal.h +4 more+129 -80 replies
15/21bottom halves: introduce bh call functionasync.c include/block/aio.h+11 -10 replies
16/21replay: ptimerhw/core/ptimer.c replay/replay-events.c replay/replay-internal.h +1 more+26 -20 replies
17/21typedef: add typedef for QemuOptsinclude/qemu/option.h include/qemu/typedefs.h+4 -40 replies
18/21replay: initialization and deinitializationexec.c replay/replay-internal.h replay/replay-user.c +4 more+156 -00 replies
19/21replay: replay blockers for deviceshw/bt/hci.c include/qapi/qmp/qerror.h qapi/common.json +3 more+41 -11 reply
20/21replay: command line optionsqemu-options.hx replay/replay.c vl.c+22 -50 replies
21/21replay: recording of the user inputinclude/ui/input.h replay/Makefile.objs replay/replay-events.c +4 more+232 -80 replies

Reviews trailers per patch

8 of 21 patches have a Reviewed-by or Acked-by.

PatchPaolo BonziniEric Blake
1/21
2/21R-b R-b
3/21R-b
4/21R-b
5/21R-b
6/21
7/21R-b
8/21
9/21
10/21
11/21
12/21R-b
13/21
14/21
15/21
16/21
17/21R-b
18/21R-b
19/21
20/21
21/21

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v18 thread

Cover letter [Qemu-devel] [PATCH v18 00/21] Deterministic replay core

This set of patches is related to the reverse execution and deterministic replay of qemu execution. This implementation of deterministic replay can be used for deterministic debugging of guest code through gdb remote interface.

Core set of patches does not include support for reverse debugging commands of gdb, block devices' operations, USB replay support.

These patches include only core function of the replay, excluding the support for replaying serial, audio, network, and USB devices' operations. Reverse debugging and monitor commands were also excluded to be submitted later as separate patches.

Execution recording writes non-deterministic events log, which can be later used for replaying the execution anywhere and for unlimited number of times. It also supports checkpointing for faster rewinding during reverse debugging. Execution replaying reads the log and replays all non-deterministic events including external input, hardware clocks, and interrupts.

Full version of deterministic replay has the following features:
* Deterministically replays whole system execution and all contents of the memory, state of the hadrware devices, clocks, and screen of the VM.
* Writes execution log into the file for latter replaying for multiple times on different machines.
* Supports i386, x86_64, ARM, PowerPC, and MIPS hardware platforms.
* Performs deterministic replay of all operations with keyboard and mouse input devices.
* Supports auto-checkpointing for convenient reverse debugging.

Usage of the record/replay core:
* First, record the execution, by adding the following string to the command line:
'-icount shift=7,rr=record,rrfile=replay.bin -net none'.
Block devices' images are not actually changed in the recording mode, because all of the changes are written to the temporary overlay file.
* Then you can replay it for the multiple times by using another command line option: '-icount shift=7,rr=replay,rrfile=replay.bin -net none'
* '-net none' option should also be specified if network replay patches are not applied.
* Do not add any disk images to VM, because they are not supported by the core patches.

Papers with description of deterministic replay implementation:
http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html
http://dl.acm.org/citation.cfm?id=2786805.2803179

Public repository with current version of the patches:
https://github.com/Dovgalyuk/qemu/tree/rr-17

Modifications of qemu include:
* wrappers for clock and time functions to save their return values in the log
* saving different asynchronous events (e.g. system shutdown) into the log
* synchronization of the threads from thread pool
* recording/replaying user input (mouse and keyboard)
* adding internal events for cpu and io synchronization

v18 changes:
* Patches were updated to match upstream version
* Added missed replay-user.c file

v17 changes:
* Removed useless stub functions (as suggested by Paolo Bonzini)
* Refined checkpoint-related code (as suggested by Paolo Bonzini)
* Improved icount processing (as suggested by Paolo Bonzini)
* Added checkpoint for suspend event (as suggested by Paolo Bonzini)
* Fixed linux-user configurations build
* Minor fixes

v16 changes:
* Several warnings were fixed

v15 changes:
* Tested record/replay with MIPS and PowerPC guests
* Published the patches on github
* Fixed replay mutex operation in icount mode
* Fixed timers processing in record/replay mode

v14 changes:
* Minor fixes

v13 changes:
* Introduced "ptimer trigger" event (as suggested by Paolo Bonzini)

v12 changes:
* Removed block patches from the core patches set.

v11 changes:
* Fixed instructions event processing.
* Added some mutex protection calls for replay.
* Fixed replaying read operations for qcow2.
* Fixed rtc reads on initializations stage.
* Eliminated some warnings in replay module.
* Fixed misprints in documentation for replay (as suggested by Eric Blake)

v10 changes:
* Fixed queue processing for bottom halves (as suggested by Paolo Bonzini)
* Rewritten several replay functions (as suggested by Paolo Bonzini)
* Some minor fixes.

v9 changes:
* Replaced fwrite/fread with putc/getc (as suggested by Paolo Bonzini)
* Stopping virtual machine in case of replay file end (as suggested by Paolo Bonzini)
* Removed one of the replay mutexes (as suggested by Paolo Bonzini)
* Fixed RCU queue for bottom halves (as suggested by Paolo Bonzini)
* Updated command line options' names (as suggested by Paolo Bonzini)
* Added design document for record/replay (as suggested by Paolo Bonzini)
* Simplified checkpoints for the timers
* Added cloning InputEvent objects for replay (as suggested by Paolo Bonzini)
* Added replay blockers instead of checking the command line (as suggested by Paolo Bonzini)
* Some functions renaming and extracting.

v8 changes:
* Simplified processing of the shutdown event (as suggested by Paolo Bonzini)
* Replaced stack of bottom halves in AIO context with QSIMPLEQ (as suggested by Paolo Bonzini)
* Moved replay_submode out of the series (as suggested by Paolo Bonzini)
* Moved suffix option out of the series
* Converted some of the defines into enums (as suggested by Paolo Bonzini)
* Encapsulated save_tm/read_tm calls into the single function (as suggested by Paolo Bonzini)
* Moved record/replay options to icount group (as suggested by Paolo Bonzini)
* Updated mutex protection for the events queue (as suggested by Paolo Bonzini)
* Added mutex to protect replay log file (as suggested by Paolo Bonzini)
* Minor cleanups

v7 changes:
* Removed patches that were applied to upstream.

v6 changes:
* Fixed replay stub return value (as suggested by Eric Blake)
* Fixed icount warping.
* Virtual rt clock now uses cpu_get_clock() (as suggested by Paolo Bonzini)
* Replated get_clock_realtime and get_clock calls with qemu clock requests (as suggested by Paolo Bonzini)
* Modified can_do_io logic to allow requesting icount from cpu_exec function (as suggested by Paolo Bonzini)
* Removed applied patches.

v5 changes:
* Minor changes.
* Used fixed-width integer types for read/write functions (as suggested by Alex Bennee)
* Moved savevm-related code out of the core.
* Added new traced clock for deterministic virtual clock warping (as suggested by Paolo Bonzini)
* Fixed exception_index reset for user mode (as suggested by Paolo Bonzini)
* Adopted Paolo's icount patches
* Fixed hardware interrupts replaying

v4 changes:
* Updated block drivers to support new bdrv_open interface.
* Moved migration patches into separate series (as suggested by Paolo Bonzini)
* Fixed a bug in replay_break operation.
* Fixed rtl8139 migration for replay.
* Fixed 'period' parameter processing for record mode.
* Fixed bug in 'reverse-stepi' implementation.
* Fixed replay without making any snapshots (even the starting one).
* Moved core replay patches into the separate series.
* Fixed reverse step and reverse continue support.
* Fixed several bugs in icount subsystem.
* Reusing native qemu icount for replay instructions counting.
* Separated core patches into their own series.

v3 changes:
* Fixed bug with replay of the aio write operations.
* Added virtual clock based on replay icount.
* Removed duplicated saving of interrupt_request CPU field.
* Fixed some coding style issues.
* Renamed QMP commands for controlling reverse execution (as suggested by Eric Blake)
* Replay mode and submode implemented as QAPI enumerations (as suggested by Eric Blake)
* Added description and example for replay-info command (as suggested by Eric Blake)
* Added information about the current breakpoint to the output of replay-info (as suggested by Eric Blake)
* Updated version id for HPET vmstate (as suggested by Paolo Bonzini)
* Removed static fields from parallel vmstate (as suggested by Paolo Bonzini)
* New vmstate fields for mc146818rtc, pckbd, kvmapic, serial, fdc, rtl8139 moved to subsection (as suggested by Paolo Bonzini)
* Disabled textmode cursor blinking, when virtual machine is stopped (as suggested by Paolo Bonzini)
* Extracted saving of exception_index to separate patch (as suggested by Paolo Bonzini)

v2 changes:
* Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
* Added QMP versions of replay commands (as suggested by Eric Blake)
* Removed some optional features of replay to make patches cleaner
* Minor changes and code cleanup were made

---

Pavel Dovgalyuk (21):
i386: partial revert of interrupt poll fix
replay: global variables and function stubs
replay: internal functions for replay log
replay: introduce mutex to protect the replay log
replay: introduce icount event
cpu-exec: allow temporary disabling icount
cpu: replay instructions sequence
i386: interrupt poll processing
replay: interrupts and exceptions
replay: asynchronous events infrastructure
replay: recording and replaying clock ticks
replay: shutdown event
icount: improve counting for record/replay
replay: checkpoints
bottom halves: introduce bh call function
replay: ptimer
typedef: add typedef for QemuOpts
replay: initialization and deinitialization
replay: replay blockers for devices
replay: command line options
replay: recording of the user input

 Makefile.target           |    1 
 async.c                   |    7 +
 cpu-exec.c                |   62 +++++++-
 cpus.c                    |   57 +++++---
 docs/replay.txt           |  168 ++++++++++++++++++++++
 exec.c                    |    2 
 hw/bt/hci.c               |    7 +
 hw/core/ptimer.c          |    7 +
 include/block/aio.h       |    5 +
 include/exec/exec-all.h   |    1 
 include/qapi/qmp/qerror.h |    3 
 include/qemu/option.h     |    5 -
 include/qemu/typedefs.h   |    3 
 include/ui/input.h        |    2 
 main-loop.c               |    3 
 qapi-schema.json          |   18 ++
 qapi/common.json          |    6 +
 qemu-options.hx           |    8 +
 qemu-timer.c              |   48 +++++-
 replay/Makefile.objs      |    6 +
 replay/replay-events.c    |  274 ++++++++++++++++++++++++++++++++++++
 replay/replay-input.c     |  160 +++++++++++++++++++++
 replay/replay-internal.c  |  206 +++++++++++++++++++++++++++
 replay/replay-internal.h  |  140 ++++++++++++++++++
 replay/replay-time.c      |   64 ++++++++
 replay/replay-user.c      |   36 +++++
 replay/replay.c           |  342 +++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h           |  120 ++++++++++++++++
 stubs/Makefile.objs       |    1 
 stubs/replay.c            |   32 ++++
 target-i386/cpu.c         |   10 -
 target-i386/seg_helper.c  |    3 
 translate-all.c           |    2 
 ui/input.c                |   27 +++-
 vl.c                      |   53 ++++++-
 35 files changed, 1820 insertions(+), 69 deletions(-)
 create mode 100755 docs/replay.txt
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay-events.c
 create mode 100755 replay/replay-input.c
 create mode 100755 replay/replay-internal.c
 create mode 100755 replay/replay-internal.h
 create mode 100755 replay/replay-time.c
 create mode 100755 replay/replay-user.c
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c
Signature · 1 line
Pavel Dovgalyuk