Deterministic replay core
v17 cover Deterministic replay core
This set of patches is related to the reverse execution and deterministic replay of qemu execution. This implementation of deterministic replay can be used for deterministic debugging of guest code through gdb remote interface.
Core set of patches does not include support for reverse debugging commands of gdb, block devices' operations, USB replay support.
These patches include only core function of the replay, excluding the support for replaying serial, audio, network, and USB devices' operations. Reverse debugging and monitor commands were also excluded to be submitted later as separate patches.
Execution recording writes non-deterministic events log, which can be later used for replaying the execution anywhere and for unlimited number of times. It also supports checkpointing for faster rewinding during reverse debugging. Execution replaying reads the log and replays all non-deterministic events including external input, hardware clocks, and interrupts.
Full version of deterministic replay has the following features:
* Deterministically replays whole system execution and all contents of the memory, state of the hadrware devices, clocks, and screen of the VM.
* Writes execution log into the file for latter replaying for multiple times on different machines.
* Supports i386, x86_64, ARM, PowerPC, and MIPS hardware platforms.
* Performs deterministic replay of all operations with keyboard and mouse input devices.
* Supports auto-checkpointing for convenient reverse debugging.
Usage of the record/replay core:
* First, record the execution, by adding the following string to the command line:
'-icount shift=7,rr=record,rrfile=replay.bin -net none'.
Block devices' images are not actually changed in the recording mode, because all of the changes are written to the temporary overlay file.
* Then you can replay it for the multiple times by using another command line option: '-icount shift=7,rr=replay,rrfile=replay.bin -net none'
* '-net none' option should also be specified if network replay patches are not applied.
* Do not add any disk images to VM, because they are not supported by the core patches.
Papers with description of deterministic replay implementation:
http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html
http://dl.acm.org/citation.cfm?id=2786805.2803179
Public repository with current version of the patches:
https://github.com/Dovgalyuk/qemu/tree/rr-17
Modifications of qemu include:
* wrappers for clock and time functions to save their return values in the log
* saving different asynchronous events (e.g. system shutdown) into the log
* synchronization of the threads from thread pool
* recording/replaying user input (mouse and keyboard)
* adding internal events for cpu and io synchronization
v17 changes:
* Removed useless stub functions (as suggested by Paolo Bonzini)
* Refined checkpoint-related code (as suggested by Paolo Bonzini)
* Improved icount processing (as suggested by Paolo Bonzini)
* Added checkpoint for suspend event (as suggested by Paolo Bonzini)
* Fixed linux-user configurations build
* Minor fixes
v16 changes:
* Several warnings were fixed
v15 changes:
* Tested record/replay with MIPS and PowerPC guests
* Published the patches on github
* Fixed replay mutex operation in icount mode
* Fixed timers processing in record/replay mode
v14 changes:
* Minor fixes
v13 changes:
* Introduced "ptimer trigger" event (as suggested by Paolo Bonzini)
v12 changes:
* Removed block patches from the core patches set.
v11 changes:
* Fixed instructions event processing.
* Added some mutex protection calls for replay.
* Fixed replaying read operations for qcow2.
* Fixed rtc reads on initializations stage.
* Eliminated some warnings in replay module.
* Fixed misprints in documentation for replay (as suggested by Eric Blake)
v10 changes:
* Fixed queue processing for bottom halves (as suggested by Paolo Bonzini)
* Rewritten several replay functions (as suggested by Paolo Bonzini)
* Some minor fixes.
v9 changes:
* Replaced fwrite/fread with putc/getc (as suggested by Paolo Bonzini)
* Stopping virtual machine in case of replay file end (as suggested by Paolo Bonzini)
* Removed one of the replay mutexes (as suggested by Paolo Bonzini)
* Fixed RCU queue for bottom halves (as suggested by Paolo Bonzini)
* Updated command line options' names (as suggested by Paolo Bonzini)
* Added design document for record/replay (as suggested by Paolo Bonzini)
* Simplified checkpoints for the timers
* Added cloning InputEvent objects for replay (as suggested by Paolo Bonzini)
* Added replay blockers instead of checking the command line (as suggested by Paolo Bonzini)
* Some functions renaming and extracting.
v8 changes:
* Simplified processing of the shutdown event (as suggested by Paolo Bonzini)
* Replaced stack of bottom halves in AIO context with QSIMPLEQ (as suggested by Paolo Bonzini)
* Moved replay_submode out of the series (as suggested by Paolo Bonzini)
* Moved suffix option out of the series
* Converted some of the defines into enums (as suggested by Paolo Bonzini)
* Encapsulated save_tm/read_tm calls into the single function (as suggested by Paolo Bonzini)
* Moved record/replay options to icount group (as suggested by Paolo Bonzini)
* Updated mutex protection for the events queue (as suggested by Paolo Bonzini)
* Added mutex to protect replay log file (as suggested by Paolo Bonzini)
* Minor cleanups
v7 changes:
* Removed patches that were applied to upstream.
v6 changes:
* Fixed replay stub return value (as suggested by Eric Blake)
* Fixed icount warping.
* Virtual rt clock now uses cpu_get_clock() (as suggested by Paolo Bonzini)
* Replated get_clock_realtime and get_clock calls with qemu clock requests (as suggested by Paolo Bonzini)
* Modified can_do_io logic to allow requesting icount from cpu_exec function (as suggested by Paolo Bonzini)
* Removed applied patches.
v5 changes:
* Minor changes.
* Used fixed-width integer types for read/write functions (as suggested by Alex Bennee)
* Moved savevm-related code out of the core.
* Added new traced clock for deterministic virtual clock warping (as suggested by Paolo Bonzini)
* Fixed exception_index reset for user mode (as suggested by Paolo Bonzini)
* Adopted Paolo's icount patches
* Fixed hardware interrupts replaying
v4 changes:
* Updated block drivers to support new bdrv_open interface.
* Moved migration patches into separate series (as suggested by Paolo Bonzini)
* Fixed a bug in replay_break operation.
* Fixed rtl8139 migration for replay.
* Fixed 'period' parameter processing for record mode.
* Fixed bug in 'reverse-stepi' implementation.
* Fixed replay without making any snapshots (even the starting one).
* Moved core replay patches into the separate series.
* Fixed reverse step and reverse continue support.
* Fixed several bugs in icount subsystem.
* Reusing native qemu icount for replay instructions counting.
* Separated core patches into their own series.
v3 changes:
* Fixed bug with replay of the aio write operations.
* Added virtual clock based on replay icount.
* Removed duplicated saving of interrupt_request CPU field.
* Fixed some coding style issues.
* Renamed QMP commands for controlling reverse execution (as suggested by Eric Blake)
* Replay mode and submode implemented as QAPI enumerations (as suggested by Eric Blake)
* Added description and example for replay-info command (as suggested by Eric Blake)
* Added information about the current breakpoint to the output of replay-info (as suggested by Eric Blake)
* Updated version id for HPET vmstate (as suggested by Paolo Bonzini)
* Removed static fields from parallel vmstate (as suggested by Paolo Bonzini)
* New vmstate fields for mc146818rtc, pckbd, kvmapic, serial, fdc, rtl8139 moved to subsection (as suggested by Paolo Bonzini)
* Disabled textmode cursor blinking, when virtual machine is stopped (as suggested by Paolo Bonzini)
* Extracted saving of exception_index to separate patch (as suggested by Paolo Bonzini)
v2 changes:
* Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
* Added QMP versions of replay commands (as suggested by Eric Blake)
* Removed some optional features of replay to make patches cleaner
* Minor changes and code cleanup were made
---
Pavel Dovgalyuk (21):
i386: partial revert of interrupt poll fix
replay: global variables and function stubs
replay: internal functions for replay log
replay: introduce mutex to protect the replay log
replay: introduce icount event
cpu-exec: allow temporary disabling icount
cpu: replay instructions sequence
i386: interrupt poll processing
replay: interrupts and exceptions
replay: asynchronous events infrastructure
replay: recording and replaying clock ticks
replay: shutdown event
icount: improve counting for record/replay
replay: checkpoints
bottom halves: introduce bh call function
replay: ptimer
typedef: add typedef for QemuOpts
replay: initialization and deinitialization
replay: replay blockers for devices
replay: command line options
replay: recording of the user input
Makefile.target | 1 async.c | 7 + cpu-exec.c | 63 +++++++- cpus.c | 57 +++++--- docs/replay.txt | 168 ++++++++++++++++++++++ exec.c | 2 hw/bt/hci.c | 7 + hw/core/ptimer.c | 7 + include/block/aio.h | 5 + include/exec/exec-all.h | 1 include/qapi/qmp/qerror.h | 3 include/qemu/option.h | 5 - include/qemu/typedefs.h | 3 include/ui/input.h | 2 main-loop.c | 3 qapi-schema.json | 18 ++ qemu-options.hx | 8 + qemu-timer.c | 48 +++++- replay/Makefile.objs | 6 + replay/replay-events.c | 274 ++++++++++++++++++++++++++++++++++++ replay/replay-input.c | 160 +++++++++++++++++++++ replay/replay-internal.c | 206 +++++++++++++++++++++++++++ replay/replay-internal.h | 140 ++++++++++++++++++ replay/replay-time.c | 64 ++++++++ replay/replay.c | 342 +++++++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 120 ++++++++++++++++ stubs/Makefile.objs | 1 stubs/replay.c | 32 ++++ target-i386/cpu.c | 10 - target-i386/seg_helper.c | 3 translate-all.c | 2 ui/input.c | 27 +++- vl.c | 53 ++++++- 33 files changed, 1779 insertions(+), 69 deletions(-) create mode 100755 docs/replay.txt create mode 100755 replay/Makefile.objs create mode 100755 replay/replay-events.c create mode 100755 replay/replay-input.c create mode 100755 replay/replay-internal.c create mode 100755 replay/replay-internal.h create mode 100755 replay/replay-time.c create mode 100755 replay/replay.c create mode 100755 replay/replay.h create mode 100755 stubs/replay.c
Signature · 1 line
Pavel Dovgalyuk
PATCH v17 1/21 i386: partial revert of interrupt poll fix
Processing CPU_INTERRUPT_POLL requests in cpu_has_work functions break the determinism of cpu_exec. This patch is required to make interrupts processing deterministic.
--- cpu-exec.c | 9 +++++++++ target-i386/cpu.c | 10 ++-------- 2 files changed, 11 insertions(+), 8 deletions(-)
cpu-exec.c +9 -0
@@ -28,6 +28,9 @@#include "exec/memory-internal.h" #include "qemu/rcu.h" #include "exec/tb-hash.h" +#if defined(TARGET_I386 ) && !defined(CONFIG_USER_ONLY ) +#include "hw/i386/apic.h" +#endif /* -icount align implementation. */
@@ -369,6 +372,12 @@ int cpu_exec(CPUState *cpu)volatile bool have_tb_lock = false;if (cpu->halted) { +#if defined(TARGET_I386 ) && !defined(CONFIG_USER_ONLY ) +if (cpu->interrupt_request &CPU_INTERRUPT_POLL ) { +apic_poll_irq (x86_cpu->apic_state); +cpu_reset_interrupt (cpu,CPU_INTERRUPT_POLL ); + } +#endif if (!cpu_has_work (cpu)) {return EXCP_HALTED ; }
target-i386/cpu.c +2 -8
@@ -3098,14 +3098,8 @@ static bool x86_cpu_has_work(CPUState *cs)X86CPU *cpu =X86_CPU (cs);CPUX86State *env = &cpu->env; -#if !defined(CONFIG_USER_ONLY ) -if (cs->interrupt_request &CPU_INTERRUPT_POLL ) { -apic_poll_irq (cpu->apic_state); -cpu_reset_interrupt (cs,CPU_INTERRUPT_POLL ); - } -#endif - -return ((cs->interrupt_request &CPU_INTERRUPT_HARD ) && +return ((cs->interrupt_request & (CPU_INTERRUPT_HARD | +CPU_INTERRUPT_POLL )) && (env->eflags & IF_MASK)) || (cs->interrupt_request & (CPU_INTERRUPT_NMI | CPU_INTERRUPT_INIT |
PATCH v17 2/21 replay: global variables and function stubs
In reply to Pavel DovgalyukThis patch adds global variables, defines, function declarations, and function stubs for deterministic VM replay used by external modules.
--- Makefile.target | 1 docs/replay.txt | 168 ++++++++++++++++++++++++++++++++++++++++++++++++++ qapi-schema.json | 18 +++++ replay/Makefile.objs | 2 + replay/replay.c | 14 ++++ replay/replay.h | 19 ++++++ stubs/Makefile.objs | 1 stubs/replay.c | 3 + 8 files changed, 226 insertions(+), 0 deletions(-) create mode 100755 docs/replay.txt create mode 100755 replay/Makefile.objs create mode 100755 replay/replay.c create mode 100755 replay/replay.h create mode 100755 stubs/replay.c
Makefile.target +1 -0
@@ -86,6 +86,7 @@ all: $(PROGS) stap # cpu emulator library obj-y = exec.o translate-all.o cpu-exec.o obj-y += tcg/tcg.o tcg/tcg-op.o tcg/optimize.o +obj-y += replay/ obj-$(CONFIG_TCG_INTERPRETER) += tci.o obj-$(CONFIG_TCG_INTERPRETER) += disas/tci.o obj-y += fpu/softfloat.o
docs/replay.txt new file +168 -0
168 lines not on this page · raw message
qapi-schema.json +18 -0
@@ -3794,3 +3794,21 @@ # Rocker ethernet network switch { 'include': 'qapi/rocker.json' } + +## +# ReplayMode: +# +# Mode of the replay subsystem. +# +# @none: normal execution mode. Replay or record are not enabled. +# +# @record: record mode. All non-deterministic data is written into the +# replay log. +# +# @play: replay mode. Non-deterministic data required for system execution +# is read from the log. +# +# Since: 2.5 +## +{ 'enum': 'ReplayMode', + 'data': [ 'none', 'record', 'play' ] }
replay/Makefile.objs new file +2 -0
@@ -0,0 +1,2 @@ +obj-$(CONFIG_SOFTMMU) += replay.o +obj-$(CONFIG_USER_ONLY) += replay-user.o
replay/replay.c new file +14 -0
@@ -0,0 +1,14 @@ +/* +* replay.c +* +* Copyright (c) 2010-2015 Institute for System Programming +* of the Russian Academy of Sciences. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ + +#include "replay.h" + +ReplayMode replay_mode =REPLAY_MODE_NONE ;
replay/replay.h new file +19 -0
@@ -0,0 +1,19 @@ +#ifndef REPLAY_H +#define REPLAY_H + +/* +* replay.h +* +* Copyright (c) 2010-2015 Institute for System Programming +* of the Russian Academy of Sciences. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ + +#include "qapi-types.h" + +extern ReplayMode replay_mode; + +#endif
stubs/Makefile.objs +1 -0
@@ -25,6 +25,7 @@ stub-obj-y += monitor-init.o stub-obj-y += notify-event.o stub-obj-$(CONFIG_SPICE) += qemu-chr-open-spice.o stub-obj-y += qtest.o +stub-obj-y += replay.o stub-obj-y += reset.o stub-obj-y += runstate-check.o stub-obj-y += set-fd-handler.o
stubs/replay.c new file +3 -0
@@ -0,0 +1,3 @@ +#include "replay/replay.h" + +ReplayMode replay_mode;
PATCH v17 3/21 replay: internal functions for replay log
In reply to Pavel DovgalyukThis patch adds functions to perform read and write operations with replay log.
--- replay/Makefile.objs | 1 replay/replay-internal.c | 155 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay-internal.h | 46 ++++++++++++++ 3 files changed, 202 insertions(+), 0 deletions(-) create mode 100755 replay/replay-internal.c create mode 100755 replay/replay-internal.h
replay/Makefile.objs +1 -0
@@ -1,2 +1,3 @@ obj-$(CONFIG_SOFTMMU) += replay.o +obj-$(CONFIG_SOFTMMU) += replay-internal.o obj-$(CONFIG_USER_ONLY) += replay-user.o
replay/replay-internal.c new file +155 -0
155 lines not on this page · raw message
replay/replay-internal.h new file +46 -0
46 lines not on this page · raw message
PATCH v17 4/21 replay: introduce mutex to protect the replay log
In reply to Pavel DovgalyukThis mutex will protect read/write operations for replay log. Using mutex is necessary because most of the events consist of several fields stored in the log. The mutex will help to avoid races.
--- replay/replay-internal.c | 27 +++++++++++++++++++++++++++ replay/replay-internal.h | 7 +++++++ 2 files changed, 34 insertions(+), 0 deletions(-)
replay/replay-internal.c +27 -0
@@ -17,6 +17,13 @@unsigned int replay_data_kind =-1 ;static unsigned int replay_has_unread_data; +/* Mutex to protect reading and writing events to the log. +replay_data_kind and replay_has_unread_data are also protected +by this mutex. +It also protects replay events queue which stores events to be +written or read to the log. */ +static QemuMutex lock; +/* File for replay writing */ FILE *replay_file;
@@ -153,3 +160,23 @@ void replay_finish_event(void) replay_has_unread_data =0 ;replay_fetch_data_kind (); } + +void replay_mutex_init (void ) +{ +qemu_mutex_init (&lock); +} + +void replay_mutex_destroy (void ) +{ +qemu_mutex_destroy (&lock); +} + +void replay_mutex_lock (void ) +{ +qemu_mutex_lock (&lock); +} + +void replay_mutex_unlock (void ) +{ +qemu_mutex_unlock (&lock); +}
replay/replay-internal.h +7 -0
@@ -33,6 +33,13 @@ int64_t replay_get_qword(void);void replay_get_array (uint8_t *buf,size_t *size);void replay_get_array_alloc (uint8_t **buf,size_t *size); +/* Mutex functions for protecting replay log file */ + +void replay_mutex_init (void ); +void replay_mutex_destroy (void ); +void replay_mutex_lock (void ); +void replay_mutex_unlock (void ); +/*! Checks error status of the file. */ void replay_check_error (void );
PATCH v17 5/21 replay: introduce icount event
In reply to Pavel DovgalyukThis patch adds icount event to the replay subsystem. This event corresponds to execution of several instructions and used to synchronize input events in the replay phase.
--- replay/replay-internal.c | 24 ++++++++++++++++++++++++ replay/replay-internal.h | 21 +++++++++++++++++++++ replay/replay.c | 33 +++++++++++++++++++++++++++++++++ replay/replay.h | 7 +++++++ 4 files changed, 85 insertions(+), 0 deletions(-)
replay/replay-internal.c +24 -0
@@ -10,6 +10,7 @@*/ #include "qemu-common.h" +#include "replay.h" #include "replay-internal.h" #include "qemu/error-report.h" #include "sysemu/sysemu.h"
@@ -36,6 +37,7 @@ void replay_put_byte(uint8_t byte)void replay_put_event (uint8_t event) { +assert (event <EVENT_COUNT );replay_put_byte (event); }
@@ -149,8 +151,15 @@ void replay_fetch_data_kind(void)if (replay_file) {if (!replay_has_unread_data) { replay_data_kind =replay_get_byte (); +if (replay_data_kind ==EVENT_INSTRUCTION ) { + replay_state.instructions_count =replay_get_dword (); + }replay_check_error (); replay_has_unread_data =1 ; +if (replay_data_kind >=EVENT_COUNT ) { +error_report ("Replay: unknown event kind %d" , replay_data_kind); +exit (1 ); + } } } }
@@ -180,3 +189,18 @@ void replay_mutex_unlock(void) {qemu_mutex_unlock (&lock); } + +/*! Saves cached instructions. */ +void replay_save_instructions (void ) +{ +if (replay_file && replay_mode ==REPLAY_MODE_RECORD ) { +replay_mutex_lock (); +int diff = (int )(replay_get_current_step () - replay_state.current_step); +if (first_cpu !=NULL && diff >0 ) { +replay_put_event (EVENT_INSTRUCTION ); +replay_put_dword (diff); + replay_state.current_step += diff; + } +replay_mutex_unlock (); + } +}
replay/replay-internal.h +21 -0
@@ -14,6 +14,20 @@#include <stdio.h> +enum ReplayEvents { +/* for instruction event */ +EVENT_INSTRUCTION , +EVENT_COUNT +}; + +typedef struct ReplayState { +/*! Current step - number of processed instructions and timer events. */ +uint64_t current_step; +/*! Number of instructions to be executed before other events happen. */ +int instructions_count; +}ReplayState ; +extern ReplayState replay_state; +extern unsigned int replay_data_kind;/* File for replay writing */
@@ -50,4 +64,11 @@ void replay_finish_event(void);replay_data_kind variable. */ void replay_fetch_data_kind (void ); +/*! Saves queued events (like instructions and sound). */ +void replay_save_instructions (void ); + +/*! Skips async events until some sync event will be found. +\return true, if event was found */ +bool replay_next_event_is (int event); +#endif
replay/replay.c +33 -0
@@ -9,6 +9,39 @@* */ +#include "qemu-common.h" #include "replay.h" +#include "replay-internal.h" +#include "qemu/timer.h" ReplayMode replay_mode =REPLAY_MODE_NONE ; + +ReplayState replay_state; + +bool replay_next_event_is (int event) +{ +bool res = false; + +/* nothing to skip - not all instructions used */ +if (replay_state.instructions_count !=0 ) { +assert (replay_data_kind ==EVENT_INSTRUCTION ); +return event ==EVENT_INSTRUCTION ; + } + +while (true) { +if (event == replay_data_kind) { + res = true; + } +switch (replay_data_kind) { +default : +/* clock, time_t, checkpoint and other events */ +return res; + } + } +return res; +} + +uint64_t replay_get_current_step (void ) +{ +return cpu_get_icount_raw (); +}
replay/replay.h +7 -0
@@ -12,8 +12,15 @@* */ +#include <stdbool.h> +#include <stdint.h> #include "qapi-types.h" extern ReplayMode replay_mode; +/* Processing the instructions */ + +/*! Returns number of executed instructions. */ +uint64_t replay_get_current_step (void ); +#endif
PATCH v17 6/21 cpu-exec: allow temporary disabling icount
In reply to Pavel DovgalyukThis patch is required for deterministic replay to generate an exception by trying executing an instruction without changing icount. It adds new flag to TB for disabling icount while translating it.
--- cpu-exec.c | 7 ++++--- include/exec/exec-all.h | 1 + translate-all.c | 2 +- 3 files changed, 6 insertions(+), 4 deletions(-)
cpu-exec.c +4 -3
@@ -240,7 +240,7 @@ static inline tcg_target_ulong cpu_tb_exec(CPUState *cpu, uint8_t *tb_ptr)/* Execute the code without caching the generated code. An interpreter could be used if available. */ static void cpu_exec_nocache (CPUState *cpu,int max_cycles, -TranslationBlock *orig_tb) +TranslationBlock *orig_tb,bool ignore_icount) {TranslationBlock *tb;
@@ -250,7 +250,8 @@ static void cpu_exec_nocache(CPUState *cpu, int max_cycles, max_cycles =CF_COUNT_MASK ; tb =tb_gen_code (cpu, orig_tb->pc, orig_tb->cs_base, orig_tb->flags, - max_cycles |CF_NOCACHE ); + max_cycles |CF_NOCACHE + | (ignore_icount ?CF_IGNORE_ICOUNT :0 )); tb->orig_tb = tcg_ctx.tb_ctx.tb_invalidated_flag ?NULL : orig_tb; cpu->current_tb = tb;/* execute the generated code */
@@ -555,7 +556,7 @@ int cpu_exec(CPUState *cpu)if (insns_left >0 ) {/* Execute remaining instructions. */ tb = (TranslationBlock *)(next_tb & ~TB_EXIT_MASK ); -cpu_exec_nocache (cpu, insns_left, tb); +cpu_exec_nocache (cpu, insns_left, tb, false);align_clocks (&sc, cpu); } cpu->exception_index =EXCP_INTERRUPT ;
include/exec/exec-all.h +1 -0
@@ -199,6 +199,7 @@ struct TranslationBlock {#define CF_LAST_IO 0x8000/* Last insn may be an IO access. */ #define CF_NOCACHE 0x10000/* To be freed after execution */ #define CF_USE_ICOUNT 0x20000 +#define CF_IGNORE_ICOUNT 0x40000/* Do not generate icount code */ void *tc_ptr;/* pointer to the translated code */ /* next matching tb for physical address. */
translate-all.c +1 -1
@@ -1007,7 +1007,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu,int code_gen_size; phys_pc =get_page_addr_code (env, pc); -if (use_icount) { +if (use_icount && !(cflags &CF_IGNORE_ICOUNT )) { cflags |=CF_USE_ICOUNT ; } tb =tb_alloc (pc);
PATCH v17 7/21 cpu: replay instructions sequence
In reply to Pavel DovgalyukThis patch adds calls to replay functions into the icount setup block. In record mode number of executed instructions is written to the log. In replay mode number of istructions to execute is taken from the replay log. When replayed instructions counter is expired qemu_notify_event() function is called to wake up the iothread.
--- cpus.c | 38 +++++++++++++++++++++++++------------- replay/replay.c | 34 ++++++++++++++++++++++++++++++++++ replay/replay.h | 4 ++++ 3 files changed, 63 insertions(+), 13 deletions(-)
cpus.c +25 -13
@@ -42,6 +42,7 @@#include "qemu/seqlock.h" #include "qapi-event.h" #include "hw/nmi.h" +#include "replay/replay.h" #ifndef _WIN32#include "qemu/compatfd.h"
@@ -1363,6 +1364,28 @@ int vm_stop_force_state(RunState state) } } +static int64_t tcg_get_icount_limit (void ) +{ +int64_t deadline; + +if (replay_mode !=REPLAY_MODE_PLAY ) { + deadline =qemu_clock_deadline_ns_all (QEMU_CLOCK_VIRTUAL ); + +/* Maintain prior (possibly buggy) behaviour where if no deadline +* was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than +* INT32_MAX nanoseconds ahead, we still use INT32_MAX +* nanoseconds. +*/ +if ((deadline <0 ) || (deadline >INT32_MAX )) { + deadline =INT32_MAX ; + } + +return qemu_icount_round (deadline); + }else { +return replay_get_instructions (); + } +} +static int tcg_cpu_exec (CPUState *cpu) {int ret;
@@ -1375,24 +1398,12 @@ static int tcg_cpu_exec(CPUState *cpu)#endif if (use_icount) {int64_t count; -int64_t deadline;int decr; timers_state.qemu_icount -= (cpu->icount_decr.u16.low + cpu->icount_extra); cpu->icount_decr.u16.low =0 ; cpu->icount_extra =0 ; - deadline =qemu_clock_deadline_ns_all (QEMU_CLOCK_VIRTUAL ); - -/* Maintain prior (possibly buggy) behaviour where if no deadline -* was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than -* INT32_MAX nanoseconds ahead, we still use INT32_MAX -* nanoseconds. -*/ -if ((deadline <0 ) || (deadline >INT32_MAX )) { - deadline =INT32_MAX ; - } - - count =qemu_icount_round (deadline); + count =tcg_get_icount_limit (); timers_state.qemu_icount += count; decr = (count >0xffff ) ?0xffff : count; count -= decr;
@@ -1410,6 +1421,7 @@ static int tcg_cpu_exec(CPUState *cpu) + cpu->icount_extra); cpu->icount_decr.u32 =0 ; cpu->icount_extra =0 ; +replay_account_executed_instructions (); }return ret; }
replay/replay.c +34 -0
@@ -13,6 +13,7 @@#include "replay.h" #include "replay-internal.h" #include "qemu/timer.h" +#include "qemu/main-loop.h" ReplayMode replay_mode =REPLAY_MODE_NONE ;
@@ -45,3 +46,36 @@ uint64_t replay_get_current_step(void) {return cpu_get_icount_raw (); } + +int replay_get_instructions (void ) +{ +int res =0 ; +replay_mutex_lock (); +if (replay_next_event_is (EVENT_INSTRUCTION )) { + res = replay_state.instructions_count; + } +replay_mutex_unlock (); +return res; +} + +void replay_account_executed_instructions (void ) +{ +if (replay_mode ==REPLAY_MODE_PLAY ) { +replay_mutex_lock (); +if (replay_state.instructions_count >0 ) { +int count = (int )(replay_get_current_step () + - replay_state.current_step); + replay_state.instructions_count -= count; + replay_state.current_step += count; +if (replay_state.instructions_count ==0 ) { +assert (replay_data_kind ==EVENT_INSTRUCTION ); +replay_finish_event (); +/* Wake up iothread. This is required because +timers will not expire until clock counters +will be read from the log. */ +qemu_notify_event (); + } + } +replay_mutex_unlock (); + } +}
replay/replay.h +4 -0
@@ -22,5 +22,9 @@ extern ReplayMode replay_mode;/*! Returns number of executed instructions. */ uint64_t replay_get_current_step (void ); +/*! Returns number of instructions to execute in replay mode. */ +int replay_get_instructions (void ); +/*! Updates instructions counter in replay mode. */ +void replay_account_executed_instructions (void );#endif
PATCH v17 8/21 i386: interrupt poll processing
In reply to Pavel DovgalyukThis patch updates x86_cpu_exec_interrupt function.
It can process two interrupt request at a time (poll and another one). This makes its execution non-deterministic. Determinism is requred for recorded icount execution.
--- target-i386/seg_helper.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
target-i386/seg_helper.c +3 -0
@@ -1321,6 +1321,9 @@ bool x86_cpu_exec_interrupt(CPUState *cs, int interrupt_request)if (interrupt_request &CPU_INTERRUPT_POLL ) { cs->interrupt_request &= ~CPU_INTERRUPT_POLL ;apic_poll_irq (cpu->apic_state); +/* Don't process multiple interrupt requests in a single call. +This is required to make icount-driven execution deterministic. */ +return true; }#endif if (interrupt_request &CPU_INTERRUPT_SIPI ) {
PATCH v17 9/21 replay: interrupts and exceptions
In reply to Pavel DovgalyukThis patch includes modifications of common cpu files. All interrupts and exceptions occured during recording are written into the replay log. These events allow correct replaying the execution by kicking cpu thread when one of these events is found in the log.
--- cpu-exec.c | 49 ++++++++++++++++++++++++++-------- replay/replay-internal.h | 4 +++ replay/replay.c | 67 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 17 ++++++++++++ 4 files changed, 126 insertions(+), 11 deletions(-)
cpu-exec.c +38 -11
@@ -31,6 +31,7 @@#if defined(TARGET_I386 ) && !defined(CONFIG_USER_ONLY )#include "hw/i386/apic.h" #endif +#include "replay/replay.h" /* -icount align implementation. */
@@ -372,22 +373,25 @@ int cpu_exec(CPUState *cpu)/* This must be volatile so it is not trashed by longjmp() */ volatile bool have_tb_lock = false; +/* replay_interrupt may need current_cpu */ + current_cpu = cpu; +if (cpu->halted) {#if defined(TARGET_I386 ) && !defined(CONFIG_USER_ONLY ) -if (cpu->interrupt_request &CPU_INTERRUPT_POLL ) { +if ((cpu->interrupt_request &CPU_INTERRUPT_POLL ) + &&replay_interrupt ()) {apic_poll_irq (x86_cpu->apic_state);cpu_reset_interrupt (cpu,CPU_INTERRUPT_POLL ); }#endif if (!cpu_has_work (cpu)) { + current_cpu =NULL ;return EXCP_HALTED ; } cpu->halted =0 ; } - current_cpu = cpu; -/* As long as current_cpu is null, up to the assignment just above, * requests by other threads to exit the execution loop are expected to * be issued using the exit_request global. We must make sure that our
@@ -436,10 +440,22 @@ int cpu_exec(CPUState *cpu) cpu->exception_index =-1 ;break ;#else - cc->do_interrupt (cpu); - cpu->exception_index =-1 ; +if (replay_exception ()) { + cc->do_interrupt (cpu); + cpu->exception_index =-1 ; + }else if (!replay_has_interrupt ()) { +/* give a chance to iothread in replay mode */ + ret =EXCP_INTERRUPT ; +break ; + }#endif } + }else if (replay_has_exception () + && cpu->icount_decr.u16.low + cpu->icount_extra ==0 ) { +/* try to cause an exception pending in the log */ +cpu_exec_nocache (cpu,1 ,tb_find_fast (cpu), true); + ret =-1 ; +break ; } next_tb =0 ;/* force lookup of first TB */
@@ -455,30 +471,40 @@ int cpu_exec(CPUState *cpu) cpu->exception_index =EXCP_DEBUG ;cpu_loop_exit (cpu); } -if (interrupt_request &CPU_INTERRUPT_HALT ) { +if (replay_mode ==REPLAY_MODE_PLAY + && !replay_has_interrupt ()) { +/* Do nothing */ + }else if (interrupt_request &CPU_INTERRUPT_HALT ) { +replay_interrupt (); cpu->interrupt_request &= ~CPU_INTERRUPT_HALT ; cpu->halted =1 ; cpu->exception_index =EXCP_HLT ;cpu_loop_exit (cpu); }#if defined(TARGET_I386 ) -if (interrupt_request &CPU_INTERRUPT_INIT ) { + else if (interrupt_request & CPU_INTERRUPT_INIT) { +replay_interrupt ();cpu_svm_check_intercept_param (env,SVM_EXIT_INIT ,0 );do_cpu_init (x86_cpu); cpu->exception_index =EXCP_HALTED ;cpu_loop_exit (cpu); }#else -if (interrupt_request &CPU_INTERRUPT_RESET ) { + else if (interrupt_request & CPU_INTERRUPT_RESET) { +replay_interrupt ();cpu_reset (cpu); +cpu_loop_exit (cpu); }#endif /* The target hook has 3 exit conditions: False when the interrupt isn't processed, True when it is, and we should restart on a new TB, and via longjmp via cpu_loop_exit. */ -if (cc->cpu_exec_interrupt (cpu, interrupt_request)) { - next_tb =0 ; + else { +replay_interrupt (); +if (cc->cpu_exec_interrupt (cpu, interrupt_request)) { + next_tb =0 ; + } }/* Don't use the cached interrupt_request value, do_interrupt may have updated the EXITTB flag. */
@@ -489,7 +515,8 @@ int cpu_exec(CPUState *cpu) next_tb =0 ; } } -if (unlikely (cpu->exit_request)) { +if (unlikely (cpu->exit_request + ||replay_has_interrupt ())) { cpu->exit_request =0 ; cpu->exception_index =EXCP_INTERRUPT ;cpu_loop_exit (cpu);
replay/replay-internal.h +4 -0
@@ -17,6 +17,10 @@enum ReplayEvents {/* for instruction event */ EVENT_INSTRUCTION , +/* for software interrupt */ +EVENT_INTERRUPT , +/* for emulated exceptions */ +EVENT_EXCEPTION ,EVENT_COUNT };
replay/replay.c +67 -0
70 lines not on this page · raw message
replay/replay.h +17 -0
@@ -27,4 +27,21 @@ int replay_get_instructions(void);/*! Updates instructions counter in replay mode. */ void replay_account_executed_instructions (void ); +/* Interrupts and exceptions */ + +/*! Called by exception handler to write or read +exception processing events. */ +bool replay_exception (void ); +/*! Used to determine that exception is pending. +Does not proceed to the next event in the log. */ +bool replay_has_exception (void ); +/*! Called by interrupt handlers to write or read +interrupt processing events. +\return true if interrupt should be processed */ +bool replay_interrupt (void ); +/*! Tries to read interrupt event from the file. +Returns true, when interrupt request is pending */ +bool replay_has_interrupt (void ); + +#endif
PATCH v17 10/21 replay: asynchronous events infrastructure
In reply to Pavel DovgalyukThis patch adds module for saving and replaying asynchronous events. These events include network packets, keyboard and mouse input, USB packets, thread pool and bottom halves callbacks.
All events are stored in the queue to be processed at synchronization points such as beginning of TB execution, or checkpoint in the iothread.
--- replay/Makefile.objs | 1 replay/replay-events.c | 225 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay-internal.h | 29 ++++++ replay/replay.h | 6 + 4 files changed, 261 insertions(+), 0 deletions(-) create mode 100755 replay/replay-events.c
replay/Makefile.objs +1 -0
@@ -1,3 +1,4 @@ obj-$(CONFIG_SOFTMMU) += replay.o obj-$(CONFIG_SOFTMMU) += replay-internal.o +obj-$(CONFIG_SOFTMMU) += replay-events.o obj-$(CONFIG_USER_ONLY) += replay-user.o
replay/replay-events.c new file +225 -0
225 lines not on this page · raw message
replay/replay-internal.h +29 -0
@@ -21,9 +21,19 @@ enum ReplayEvents {EVENT_INTERRUPT ,/* for emulated exceptions */ EVENT_EXCEPTION , +/* for async events */ +EVENT_ASYNC ,EVENT_COUNT }; +/* Asynchronous events IDs */ + +enum ReplayAsyncEventKind { +REPLAY_ASYNC_COUNT +}; + +typedef enum ReplayAsyncEventKind ReplayAsyncEventKind ; + typedef struct ReplayState { /*! Current step - number of processed instructions and timer events. */ uint64_t current_step;
@@ -75,4 +85,23 @@ void replay_save_instructions(void);\return true, if event was found */ bool replay_next_event_is (int event); +/* Asynchronous events queue */ + +/*! Initializes events' processing internals */ +void replay_init_events (void ); +/*! Clears internal data structures for events handling */ +void replay_finish_events (void ); +/*! Enables storing events in the queue */ +void replay_enable_events (void ); +/*! Flushes events queue */ +void replay_flush_events (void ); +/*! Clears events list before loading new VM state */ +void replay_clear_events (void ); +/*! Returns true if there are any unsaved events in the queue */ +bool replay_has_events (void ); +/*! Saves events from queue into the file */ +void replay_save_events (int checkpoint); +/*! Read events from the file into the input queue */ +void replay_read_events (int checkpoint); +#endif
replay/replay.h +6 -0
@@ -43,5 +43,11 @@ bool replay_interrupt(void);Returns true, when interrupt request is pending */ bool replay_has_interrupt (void ); +/* Asynchronous events queue */ + +/*! Disables storing events in the queue */ +void replay_disable_events (void ); +/*! Returns true when saving events is enabled */ +bool replay_events_enabled (void );#endif
PATCH v17 11/21 replay: recording and replaying clock ticks
In reply to Pavel DovgalyukClock ticks are considered as the sources of non-deterministic data for virtual machine. This patch implements saving the clock values when they are acquired (virtual, host clock).
When replaying the execution corresponding values are read from log and transfered to the module, which wants to read the values.
Such a design required the clock polling to be synchronized. Sometimes it is not true - e.g. when timeouts for timer lists are checked. In this case we use a cached value of the clock, passing it to the client code.
--- cpus.c | 3 +- qemu-timer.c | 8 ++++-- replay/Makefile.objs | 1 + replay/replay-internal.h | 11 ++++++++ replay/replay-time.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 23 +++++++++++++++++ stubs/replay.c | 13 +++++++++ 7 files changed, 119 insertions(+), 4 deletions(-) create mode 100755 replay/replay-time.c
cpus.c +2 -1
@@ -338,7 +338,8 @@ static void icount_warp_rt(void *opaque)seqlock_write_lock (&timers_state.vm_clock_seqlock);if (runstate_is_running ()) { -int64_t clock =cpu_get_clock_locked (); +int64_t clock =REPLAY_CLOCK (REPLAY_CLOCK_VIRTUAL_RT , +cpu_get_clock_locked ());int64_t warp_delta; warp_delta = clock - vm_clock_warp_start;
qemu-timer.c +5 -3
@@ -24,6 +24,7 @@#include "qemu/main-loop.h" #include "qemu/timer.h" +#include "replay/replay.h" #ifdef CONFIG_POSIX #include <pthread.h>
@@ -570,15 +571,16 @@ int64_t qemu_clock_get_ns(QEMUClockType type)return cpu_get_clock (); }case QEMU_CLOCK_HOST : - now =get_clock_realtime (); + now =REPLAY_CLOCK (REPLAY_CLOCK_HOST ,get_clock_realtime ()); last = clock->last; clock->last = now; -if (now < last || now > (last +get_max_clock_jump ())) { +if ((now < last || now > (last +get_max_clock_jump ())) + && replay_mode ==REPLAY_MODE_NONE ) {notifier_list_notify (&clock->reset_notifiers, &now); }return now;case QEMU_CLOCK_VIRTUAL_RT : -return cpu_get_clock (); +return REPLAY_CLOCK (REPLAY_CLOCK_VIRTUAL_RT ,cpu_get_clock ()); } }
replay/Makefile.objs +1 -0
@@ -1,4 +1,5 @@ obj-$(CONFIG_SOFTMMU) += replay.o obj-$(CONFIG_SOFTMMU) += replay-internal.o obj-$(CONFIG_SOFTMMU) += replay-events.o +obj-$(CONFIG_SOFTMMU) += replay-time.o obj-$(CONFIG_USER_ONLY) += replay-user.o
replay/replay-internal.h +11 -0
@@ -23,6 +23,10 @@ enum ReplayEvents {EVENT_EXCEPTION ,/* for async events */ EVENT_ASYNC , +/* for clock read/writes */ +/* some of greater codes are reserved for clocks */ +EVENT_CLOCK , +EVENT_CLOCK_LAST =EVENT_CLOCK +REPLAY_CLOCK_COUNT -1 , EVENT_COUNT };
@@ -35,6 +39,8 @@ enum ReplayAsyncEventKind {typedef enum ReplayAsyncEventKind ReplayAsyncEventKind ; typedef struct ReplayState { + /*! Cached clock values. */ + int64_t cached_clock[REPLAY_CLOCK_COUNT]; /*! Current step - number of processed instructions and timer events. */ uint64_t current_step; /*! Number of instructions to be executed before other events happen. */
@@ -85,6 +91,11 @@ void replay_save_instructions(void);\return true, if event was found */ bool replay_next_event_is (int event); +/*! Reads next clock value from the file. +If clock kind read from the file is different from the parameter, +the value is not used. */ +void replay_read_next_clock (unsigned int kind); +/* Asynchronous events queue */ /*! Initializes events' processing internals */
replay/replay-time.c new file +64 -0
64 lines not on this page · raw message
replay/replay.h +23 -0
@@ -16,6 +16,16 @@#include <stdint.h> #include "qapi-types.h" +/* replay clock kinds */ +enum ReplayClockKind { +/* host_clock */ +REPLAY_CLOCK_HOST , +/* virtual_rt_clock */ +REPLAY_CLOCK_VIRTUAL_RT , +REPLAY_CLOCK_COUNT +}; +typedef enum ReplayClockKind ReplayClockKind ; +extern ReplayMode replay_mode;/* Processing the instructions */
@@ -43,6 +53,19 @@ bool replay_interrupt(void);Returns true, when interrupt request is pending */ bool replay_has_interrupt (void ); +/* Processing clocks and other time sources */ + +/*! Save the specified clock */ +int64_t replay_save_clock (ReplayClockKind kind,int64_t clock); +/*! Read the specified clock from the log or return cached data */ +int64_t replay_read_clock (ReplayClockKind kind); +/*! Saves or reads the clock depending on the current replay mode. */ +#define REPLAY_CLOCK (clock, value) \ + (replay_mode == REPLAY_MODE_PLAY ? replay_read_clock((clock)) \ + : replay_mode == REPLAY_MODE_RECORD \ + ? replay_save_clock((clock), (value)) \ + : (value)) +/* Asynchronous events queue */ /*! Disables storing events in the queue */
stubs/replay.c +13 -0
@@ -1,3 +1,16 @@#include "replay/replay.h" +#include <stdlib.h> ReplayMode replay_mode; + +int64_t replay_save_clock (unsigned int kind,int64_t clock) +{ +abort (); +return 0 ; +} + +int64_t replay_read_clock (unsigned int kind) +{ +abort (); +return 0 ; +}
PATCH v17 12/21 replay: shutdown event
In reply to Pavel DovgalyukThis patch records and replays simulator shutdown event.
--- replay/replay-internal.h | 2 ++ replay/replay.c | 14 ++++++++++++++ replay/replay.h | 5 +++++ vl.c | 1 + 4 files changed, 22 insertions(+), 0 deletions(-)
replay/replay-internal.h +2 -0
@@ -23,6 +23,8 @@ enum ReplayEvents {EVENT_EXCEPTION ,/* for async events */ EVENT_ASYNC , +/* for shutdown request */ +EVENT_SHUTDOWN ,/* for clock read/writes */ /* some of greater codes are reserved for clocks */ EVENT_CLOCK,
replay/replay.c +14 -0
@@ -14,6 +14,7 @@#include "replay-internal.h" #include "qemu/timer.h" #include "qemu/main-loop.h" +#include "sysemu/sysemu.h" ReplayMode replay_mode =REPLAY_MODE_NONE ;
@@ -34,6 +35,10 @@ bool replay_next_event_is(int event) res = true; }switch (replay_data_kind) { +case EVENT_SHUTDOWN : +replay_finish_event (); +qemu_system_shutdown_request (); +break ;default :/* clock, time_t, checkpoint and other events */ return res;
@@ -146,3 +151,12 @@ bool replay_has_interrupt(void) }return res; } + +void replay_shutdown_request (void ) +{ +if (replay_mode ==REPLAY_MODE_RECORD ) { +replay_mutex_lock (); +replay_put_event (EVENT_SHUTDOWN ); +replay_mutex_unlock (); + } +}
replay/replay.h +5 -0
@@ -66,6 +66,11 @@ int64_t replay_read_clock(ReplayClockKind kind); ? replay_save_clock((clock), (value)) \ : (value)) +/* Events */ + +/*! Called when qemu shutdown is requested. */ +void replay_shutdown_request (void ); +/* Asynchronous events queue */ /*! Disables storing events in the queue */
vl.c +1 -0
@@ -1814,6 +1814,7 @@ void qemu_system_killed(int signal, pid_t pid)void qemu_system_shutdown_request (void ) {trace_qemu_system_shutdown_request (); +replay_shutdown_request (); shutdown_requested =1 ;qemu_notify_event (); }
PATCH v17 13/21 icount: improve counting for record/replay
In reply to Pavel Dovgalyukicount_warp_rt function is called by qemu_clock_warp and as callback of icount_warp timer. This patch adds call to qemu_clock_warp into main_loop_wait function, because icount warp may be missed in record/replay mode, when CPU is sleeping.
This patch also disables of calling this function by timer, because it is not needed after making modifications of main_loop_wait.
--- cpus.c | 11 ++++++++--- main-loop.c | 3 +++ 2 files changed, 11 insertions(+), 3 deletions(-)
cpus.c +8 -3
@@ -327,7 +327,7 @@ static int64_t qemu_icount_round(int64_t count)return (count + (1 << icount_time_shift) -1 ) >> icount_time_shift; } -static void icount_warp_rt (void *opaque) +static void icount_warp_rt (void ) {/* The icount_warp_timer is rescheduled soon after vm_clock_warp_start * changes from -1 to another value, so the race here is okay.
@@ -362,6 +362,11 @@ static void icount_warp_rt(void *opaque) } } +static void icount_dummy_timer (void *opaque) +{ + (void )opaque; +} +void qtest_clock_warp (int64_t dest) {int64_t clock =qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL );
@@ -406,7 +411,7 @@ void qemu_clock_warp(QEMUClockType type)* the CPU starts running, in case the CPU is woken by an event other * than the earliest QEMU_CLOCK_VIRTUAL timer. */ -icount_warp_rt (NULL ); +icount_warp_rt ();timer_del (icount_warp_timer); }if (!all_cpu_threads_idle ()) {
@@ -529,7 +534,7 @@ void configure_icount(QemuOpts *opts, Error **errp) icount_sleep =qemu_opt_get_bool (opts,"sleep" , true);if (icount_sleep) { icount_warp_timer =timer_new_ns (QEMU_CLOCK_VIRTUAL_RT , - icount_warp_rt,NULL ); + icount_dummy_timer,NULL ); } icount_align_option =qemu_opt_get_bool (opts,"align" , false);
main-loop.c +3 -0
@@ -505,6 +505,9 @@ int main_loop_wait(int nonblocking)slirp_pollfds_poll (gpollfds, (ret <0 ));#endif +/* CPU thread can infinitely wait for event after +missing the warp */ +qemu_clock_warp (QEMU_CLOCK_VIRTUAL );qemu_clock_run_all_timers ();return ret;
PATCH v17 14/21 replay: checkpoints
In reply to Pavel DovgalyukThis patch introduces checkpoints that synchronize cpu thread and iothread. When checkpoint is met in the code all asynchronous events from the queue are executed.
--- cpus.c | 5 +++++ qemu-timer.c | 40 ++++++++++++++++++++++++++++++++++++---- replay/replay-internal.h | 4 ++++ replay/replay.c | 34 ++++++++++++++++++++++++++++++++++ replay/replay.h | 20 ++++++++++++++++++++ stubs/replay.c | 11 +++++++++++ vl.c | 23 +++++++++++++++++++---- 7 files changed, 129 insertions(+), 8 deletions(-)
cpus.c +5 -0
@@ -402,6 +402,11 @@ void qemu_clock_warp(QEMUClockType type)return ; } +/* warp clock deterministically in record/replay mode */ +if (!replay_checkpoint (CHECKPOINT_CLOCK_WARP )) { +return ; + } +if (icount_sleep) {/* * If the CPUs have been sleeping, advance QEMU_CLOCK_VIRTUAL timer now.
qemu-timer.c +36 -4
@@ -25,6 +25,7 @@#include "qemu/main-loop.h" #include "qemu/timer.h" #include "replay/replay.h" +#include "sysemu/sysemu.h" #ifdef CONFIG_POSIX #include <pthread.h>
@@ -478,10 +479,34 @@ bool timerlist_run_timers(QEMUTimerList *timer_list)void *opaque;qemu_event_reset (&timer_list->timers_done_ev); -if (!timer_list->clock->enabled) { +if (!timer_list->clock->enabled || !timer_list->active_timers) { goto out; } +switch (timer_list->clock->type) { +case QEMU_CLOCK_REALTIME : +break ; +default : +case QEMU_CLOCK_VIRTUAL : +if ((replay_mode !=REPLAY_MODE_NONE && !runstate_is_running ()) + || !replay_checkpoint (CHECKPOINT_CLOCK_VIRTUAL )) { + goto out; + } +break ; +case QEMU_CLOCK_HOST : +if ((replay_mode !=REPLAY_MODE_NONE && !runstate_is_running ()) + || !replay_checkpoint (CHECKPOINT_CLOCK_HOST )) { + goto out; + } +break ; +case QEMU_CLOCK_VIRTUAL_RT : +if ((replay_mode !=REPLAY_MODE_NONE && !runstate_is_running ()) + || !replay_checkpoint (CHECKPOINT_CLOCK_VIRTUAL_RT )) { + goto out; + } +break ; + } + current_time =qemu_clock_get_ns (timer_list->clock->type);for (;;) {qemu_mutex_lock (&timer_list->active_timers_lock);
@@ -545,11 +570,18 @@ int64_t timerlistgroup_deadline_ns(QEMUTimerListGroup *tlg) {int64_t deadline =-1 ;QEMUClockType type; +bool play = replay_mode ==REPLAY_MODE_PLAY ;for (type =0 ; type <QEMU_CLOCK_MAX ; type++) {if (qemu_clock_use_for_deadline (tlg->tl[type]->clock->type)) { - deadline =qemu_soonest_timeout (deadline, -timerlist_deadline_ns ( - tlg->tl[type])); +if (!play || tlg->tl[type]->clock->type ==QEMU_CLOCK_REALTIME ) { + deadline =qemu_soonest_timeout (deadline, +timerlist_deadline_ns ( + tlg->tl[type])); + }else { +/* Read clock from the replay file and +do not calculate the deadline, based on virtual clock. */ +qemu_clock_get_ns (tlg->tl[type]->clock->type); + } } }return deadline;
replay/replay-internal.h +4 -0
@@ -29,6 +29,10 @@ enum ReplayEvents {/* some of greater codes are reserved for clocks */ EVENT_CLOCK ,EVENT_CLOCK_LAST =EVENT_CLOCK +REPLAY_CLOCK_COUNT -1 , +/* for checkpoint event */ +/* some of greater codes are reserved for checkpoints */ +EVENT_CHECKPOINT , +EVENT_CHECKPOINT_LAST =EVENT_CHECKPOINT +CHECKPOINT_COUNT -1 , EVENT_COUNT };
replay/replay.c +34 -0
@@ -160,3 +160,37 @@ void replay_shutdown_request(void)replay_mutex_unlock (); } } + +bool replay_checkpoint (ReplayCheckpoint checkpoint) +{ +bool res = false; +assert (EVENT_CHECKPOINT + checkpoint <=EVENT_CHECKPOINT_LAST ); +replay_save_instructions (); + +if (!replay_file) { +return true; + } + +replay_mutex_lock (); + +if (replay_mode ==REPLAY_MODE_PLAY ) { +if (replay_next_event_is (EVENT_CHECKPOINT + checkpoint)) { +replay_finish_event (); + }else if (replay_data_kind !=EVENT_ASYNC ) { + res = false; + goto out; + } +replay_read_events (checkpoint); +/* replay_read_events may leave some unread events. +Return false if not all of the events associated with +checkpoint were processed */ + res = replay_data_kind !=EVENT_ASYNC ; + }else if (replay_mode ==REPLAY_MODE_RECORD ) { +replay_put_event (EVENT_CHECKPOINT + checkpoint); +replay_save_events (checkpoint); + res = true; + } +out: +replay_mutex_unlock (); +return res; +}
replay/replay.h +20 -0
@@ -26,6 +26,20 @@ enum ReplayClockKind { };typedef enum ReplayClockKind ReplayClockKind ; +/* IDs of the checkpoints */ +enum ReplayCheckpoint { +CHECKPOINT_CLOCK_WARP , +CHECKPOINT_RESET_REQUESTED , +CHECKPOINT_SUSPEND_REQUESTED , +CHECKPOINT_CLOCK_VIRTUAL , +CHECKPOINT_CLOCK_HOST , +CHECKPOINT_CLOCK_VIRTUAL_RT , +CHECKPOINT_INIT , +CHECKPOINT_RESET , +CHECKPOINT_COUNT +}; +typedef enum ReplayCheckpoint ReplayCheckpoint ; +extern ReplayMode replay_mode;/* Processing the instructions */
@@ -70,6 +84,12 @@ int64_t replay_read_clock(ReplayClockKind kind);/*! Called when qemu shutdown is requested. */ void replay_shutdown_request (void ); +/*! Should be called at check points in the execution. +These check points are skipped, if they were not met. +Saves checkpoint in the SAVE mode and validates in the PLAY mode. +Returns 0 in PLAY mode if checkpoint was not found. +Returns 1 in all other cases. */ +bool replay_checkpoint (ReplayCheckpoint checkpoint);/* Asynchronous events queue */
stubs/replay.c +11 -0
@@ -1,5 +1,6 @@#include "replay/replay.h" #include <stdlib.h> +#include "sysemu/sysemu.h" ReplayMode replay_mode;
@@ -14,3 +15,13 @@ int64_t replay_read_clock(unsigned int kind)abort ();return 0 ; } + +bool replay_checkpoint (ReplayCheckpoint checkpoint) +{ +return 0 ; +} + +int runstate_is_running (void ) +{ +return 0 ; +}
vl.c +19 -4
@@ -122,6 +122,7 @@ int main(int argc, char **argv)#include "qapi-event.h" #include "exec/semihost.h" #include "crypto/init.h" +#include "replay/replay.h" #define MAX_VIRTIO_CONSOLES 1#define MAX_SCLP_CONSOLES 1
@@ -1661,15 +1662,21 @@ static void qemu_kill_report(void)static int qemu_reset_requested (void ) {int r = reset_requested; - reset_requested =0 ; -return r; +if (r &&replay_checkpoint (CHECKPOINT_RESET_REQUESTED )) { + reset_requested =0 ; +return r; + } +return false; }static int qemu_suspend_requested (void ) {int r = suspend_requested; - suspend_requested =0 ; -return r; +if (r &&replay_checkpoint (CHECKPOINT_SUSPEND_REQUESTED )) { + suspend_requested =0 ; +return r; + } +return false; }static WakeupReason qemu_wakeup_requested (void )
@@ -4500,6 +4507,10 @@ int main(int argc, char **argv, char **envp) }qemu_add_globals (); +/* This checkpoint is required by replay to separate prior clock +reading from the other reads, because timer polling functions query +clock values from the log. */ +replay_checkpoint (CHECKPOINT_INIT );qdev_machine_init (); current_machine->ram_size = ram_size;
@@ -4615,6 +4626,10 @@ int main(int argc, char **argv, char **envp)exit (1 ); } +/* This checkpoint is required by replay to separate prior clock +reading from the other reads, because timer polling functions query +clock values from the log. */ +replay_checkpoint (CHECKPOINT_RESET );qemu_system_reset (VMRESET_SILENT );register_global_state ();if (loadvm) {
PATCH v17 15/21 bottom halves: introduce bh call function
In reply to Pavel DovgalyukThis patch introduces aio_bh_call function. It is used to execute bottom halves as callbacks without adding them to the queue.
--- async.c | 7 ++++++- include/block/aio.h | 5 +++++ 2 files changed, 11 insertions(+), 1 deletions(-)
async.c +6 -1
@@ -59,6 +59,11 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque)return bh; } +void aio_bh_call (QEMUBH *bh) +{ + bh->cb (bh->opaque); +} +/* Multiple occurrences of aio_bh_poll cannot be called concurrently */ int aio_bh_poll (AioContext *ctx) {
@@ -84,7 +89,7 @@ int aio_bh_poll(AioContext *ctx) ret =1 ; } bh->idle =0 ; - bh->cb (bh->opaque); +aio_bh_call (bh); } }
include/block/aio.h +5 -0
@@ -207,6 +207,11 @@ void aio_notify(AioContext *ctx);void aio_notify_accept (AioContext *ctx);/** +* aio_bh_call: Executes callback function of the specified BH. +*/ +void aio_bh_call (QEMUBH *bh); + +/** * aio_bh_poll: Poll bottom halves for an AioContext. * * These are internal functions used by the QEMU main loop.
PATCH v17 16/21 replay: ptimer
In reply to Pavel DovgalyukThis patch adds deterministic replay for hardware periodic countdown timers. ptimer uses bottom halves layer to execute such an asynchronous callback. We put this callback into the replay queue instead of bottom halves one. When checkpoint is met by main loop thread, the replay queue is processed and callback is executed. Binding callback moment to one of the checkpoints makes it deterministic.
--- hw/core/ptimer.c | 7 ++++++- replay/replay-events.c | 18 +++++++++++++++++- replay/replay-internal.h | 1 + replay/replay.h | 2 ++ 4 files changed, 26 insertions(+), 2 deletions(-)
hw/core/ptimer.c +6 -1
@@ -9,6 +9,7 @@#include "qemu/timer.h" #include "hw/ptimer.h" #include "qemu/host-utils.h" +#include "replay/replay.h" struct ptimer_state {
@@ -27,7 +28,11 @@ struct ptimer_statestatic void ptimer_trigger (ptimer_state *s) {if (s->bh) { -qemu_bh_schedule (s->bh); +if (replay_mode !=REPLAY_MODE_NONE ) { +replay_add_ptimer_event (s->bh,replay_get_current_step ()); + }else { +qemu_bh_schedule (s->bh); + } } }
replay/replay-events.c +17 -1
@@ -13,6 +13,7 @@#include "qemu/error-report.h" #include "replay.h" #include "replay-internal.h" +#include "block/aio.h" typedef struct Event { ReplayAsyncEventKind event_kind;
@@ -35,6 +36,9 @@ static bool events_enabled;static void replay_run_event (Event *event) {switch (event->event_kind) { +case REPLAY_ASYNC_EVENT_PTIMER : +aio_bh_call (event->opaque); +break ;default :error_report ("Replay: invalid async event ID (%d) in the queue" , event->event_kind);
@@ -117,6 +121,11 @@ static void replay_add_event(ReplayAsyncEventKind event_kind,replay_mutex_unlock (); } +void replay_add_ptimer_event (void *bh,uint64_t id) +{ +replay_add_event (REPLAY_ASYNC_EVENT_PTIMER , bh,NULL , id); +} +static void replay_save_event (Event *event,int checkpoint) {if (replay_mode !=REPLAY_MODE_PLAY ) {
@@ -127,10 +136,12 @@ static void replay_save_event(Event *event, int checkpoint)/* save event-specific data */ switch (event->event_kind) { +case REPLAY_ASYNC_EVENT_PTIMER : +replay_put_qword (event->id); +break ;default :error_report ("Unknown ID %d of replay event" , read_event_kind);exit (1 ); -break ; } } }
@@ -166,6 +177,11 @@ static Event *replay_read_event(int checkpoint)/* Events that has not to be in the queue */ switch (read_event_kind) { +case REPLAY_ASYNC_EVENT_PTIMER : +if (read_id ==-1 ) { + read_id =replay_get_qword (); + } +break ;default :error_report ("Unknown ID %d of replay event" , read_event_kind);exit (1 );
replay/replay-internal.h +1 -0
@@ -39,6 +39,7 @@ enum ReplayEvents {/* Asynchronous events IDs */ enum ReplayAsyncEventKind { +REPLAY_ASYNC_EVENT_PTIMER ,REPLAY_ASYNC_COUNT };
replay/replay.h +2 -0
@@ -97,5 +97,7 @@ bool replay_checkpoint(ReplayCheckpoint checkpoint);void replay_disable_events (void );/*! Returns true when saving events is enabled */ bool replay_events_enabled (void ); +/*! Adds ptimer event to the queue */ +void replay_add_ptimer_event (void *bh,uint64_t id);#endif
PATCH v17 19/21 replay: replay blockers for devices
In reply to Pavel DovgalyukSome devices are not supported by record/replay subsystem. This patch introduces replay blocker which denies starting record/replay if such devices are included into the configuration.
--- hw/bt/hci.c | 7 +++++++ include/qapi/qmp/qerror.h | 3 +++ replay/replay.c | 13 +++++++++++++ replay/replay.h | 3 +++ vl.c | 10 ++++++++++ 5 files changed, 36 insertions(+), 0 deletions(-)
hw/bt/hci.c +7 -0
@@ -23,6 +23,8 @@#include "hw/usb.h" #include "sysemu/bt.h" #include "hw/bt.h" +#include "qapi/qmp/qerror.h" +#include "replay/replay.h" struct bt_hci_s {uint8_t *(*evt_packet)(void *opaque);
@@ -72,6 +74,8 @@ struct bt_hci_s {struct HCIInfo info;struct bt_device_s device; + +Error *replay_blocker; };#define DEFAULT_RSSI_DBM 20
@@ -2191,6 +2195,9 @@ struct HCIInfo *bt_new_hci(struct bt_scatternet_s *net) s->device.handle_destroy = bt_hci_destroy; +error_set (&s->replay_blocker,QERR_REPLAY_NOT_SUPPORTED ,"bt hci" ); +replay_add_blocker (s->replay_blocker); +return &s->info; }
include/qapi/qmp/qerror.h +3 -0
@@ -106,4 +106,7 @@#define QERR_UNSUPPORTED \ "this feature or command is not currently supported" +#define QERR_REPLAY_NOT_SUPPORTED \ + ERROR_CLASS_GENERIC_ERROR, "Record/replay feature is not supported for '%s'" +#endif /* QERROR_H */
replay/replay.c +13 -0
@@ -15,6 +15,7 @@#include "qemu/timer.h" #include "qemu/main-loop.h" #include "sysemu/sysemu.h" +#include "qemu/error-report.h" /* Current version of the replay mechanism. Increase it when file format changes. */
@@ -27,6 +28,7 @@ ReplayMode replay_mode = REPLAY_MODE_NONE;/* Name of replay file */ static char *replay_filename;ReplayState replay_state; +static GSList *replay_blockers;bool replay_next_event_is (int event) {
@@ -288,6 +290,12 @@ void replay_start(void)return ; } +if (replay_blockers) { +error_report ("Record/replay: %s" , +error_get_pretty (replay_blockers->data)); +exit (1 ); + } +/* Timer for snapshotting will be set up here. */ replay_enable_events ();
@@ -323,3 +331,8 @@ void replay_finish(void)replay_finish_events ();replay_mutex_destroy (); } + +void replay_add_blocker (Error *reason) +{ + replay_blockers =g_slist_prepend (replay_blockers, reason); +}
replay/replay.h +3 -0
@@ -15,6 +15,7 @@#include <stdbool.h> #include <stdint.h> #include "qapi-types.h" +#include "qapi/error.h" #include "qemu/typedefs.h" /* replay clock kinds */
@@ -51,6 +52,8 @@ void replay_configure(struct QemuOpts *opts);void replay_start (void );/*! Closes replay log file and frees other resources. */ void replay_finish (void ); +/*! Adds replay blocker with the specified error description */ +void replay_add_blocker (Error *reason);/* Processing the instructions */
vl.c +10 -0
@@ -123,6 +123,7 @@ int main(int argc, char **argv)#include "exec/semihost.h" #include "crypto/init.h" #include "replay/replay.h" +#include "qapi/qmp/qerror.h" #define MAX_VIRTIO_CONSOLES 1#define MAX_SCLP_CONSOLES 1
@@ -843,7 +844,11 @@ static void configure_rtc(QemuOpts *opts)if (!strcmp (value,"utc" )) { rtc_utc =1 ; }else if (!strcmp (value,"localtime" )) { +Error *blocker =NULL ; rtc_utc =0 ; +error_set (&blocker,QERR_REPLAY_NOT_SUPPORTED , +"-rtc base=localtime" ); +replay_add_blocker (blocker); }else {configure_rtc_date_offset (value,0 ); }
@@ -1246,6 +1251,11 @@ static void smp_parse(QemuOpts *opts)exit (1 ); } +if (smp_cpus >1 || smp_cores >1 || smp_threads >1 ) { +Error *blocker =NULL ; +error_set (&blocker,QERR_REPLAY_NOT_SUPPORTED ,"smp" ); +replay_add_blocker (blocker); + } }static void realtime_init (void )
PATCH v17 20/21 replay: command line options
In reply to Pavel DovgalyukThis patch introduces command line options for enabling recording or replaying virtual machine behavior. These options are added to icount command line parameter. They include 'rr' which switches between record and replay and 'rrfile' for specifying the filename for replay log.
--- qemu-options.hx | 8 ++++++-- replay/replay.c | 4 ++++ vl.c | 15 ++++++++++++--- 3 files changed, 22 insertions(+), 5 deletions(-)
qemu-options.hx +6 -2
@@ -3123,12 +3123,12 @@ re-inject them. ETEXI DEF("icount", HAS_ARG, QEMU_OPTION_icount, \ - "-icount [shift=N|auto][,align=on|off][,sleep=no]\n" \ + "-icount [shift=N|auto][,align=on|off][,sleep=no,rr=record|replay,rrfile=<filename>]\n" \ " enable virtual instruction counter with 2^N clock ticks per\n" \ " instruction, enable aligning the host and virtual clocks\n" \ " or disable real time cpu sleeping\n", QEMU_ARCH_ALL) STEXI -@item -icount [shift=@var{N}|auto] +@item -icount [shift=@var{N}|auto][,rr=record|replay,rrfile=@var{filename}] @findex -icount Enable virtual instruction counter. The virtual cpu will execute one instruction every 2^@var{N} ns of virtual time. If @code{auto} is specified
@@ -3157,6 +3157,10 @@ Currently this option does not work when @option{shift} is @code{auto}. Note: The sync algorithm will work for those shift values for which the guest clock runs ahead of the host clock. Typically this happens when the shift value is high (how high depends on the host machine). + +When @option{rr} option is specified deterministic record/replay is enabled. +Replay log is written into @var{filename} file in record mode and +read from this file in replay mode. ETEXI DEF("watchdog", HAS_ARG, QEMU_OPTION_watchdog, \
replay/replay.c +4 -0
@@ -295,6 +295,10 @@ void replay_start(void) error_get_pretty(replay_blockers->data));exit (1 ); } +if (!use_icount) { +error_report ("Please enable icount to use record/replay" ); +exit (1 ); + }/* Timer for snapshotting will be set up here. */
vl.c +12 -3
@@ -476,6 +476,12 @@ static QemuOptsList qemu_icount_opts = { }, { .name = "sleep", .type = QEMU_OPT_BOOL, + }, { + .name = "rr", + .type = QEMU_OPT_STRING, + }, { + .name = "rrfile", + .type = QEMU_OPT_STRING, }, { /* end of list */ } },
@@ -4027,6 +4033,8 @@ int main(int argc, char **argv, char **envp) } } +replay_configure (icount_opts); + opts =qemu_get_machine_opts (); optarg =qemu_opt_get (opts,"type" );if (optarg) {
@@ -4460,9 +4468,10 @@ int main(int argc, char **argv, char **envp) }/* open the virtual block devices */ -if (snapshot) -qemu_opts_foreach (qemu_find_opts ("drive" ), - drive_enable_snapshot,NULL ,NULL ); +if (snapshot || replay_mode !=REPLAY_MODE_NONE ) { +qemu_opts_foreach (qemu_find_opts ("drive" ), drive_enable_snapshot, +NULL ,NULL ); + }if (qemu_opts_foreach (qemu_find_opts ("drive" ), drive_init_func, &machine_class->block_default_type,NULL )) {exit (1 );
PATCH v17 21/21 replay: recording of the user input
In reply to Pavel DovgalyukThis records user input (keyboard and mouse events) in record mode and replays these input events in replay mode.
--- include/ui/input.h | 2 + replay/Makefile.objs | 1 replay/replay-events.c | 33 +++++++++ replay/replay-input.c | 160 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay-internal.h | 13 ++++ replay/replay.h | 4 + ui/input.c | 27 +++++--- 7 files changed, 232 insertions(+), 8 deletions(-) create mode 100755 replay/replay-input.c
include/ui/input.h +2 -0
@@ -33,7 +33,9 @@ void qemu_input_handler_bind(QemuInputHandlerState *s, const char *device_id, int head, Error **errp);void qemu_input_event_send (QemuConsole *src,InputEvent *evt); +void qemu_input_event_send_impl (QemuConsole *src,InputEvent *evt);void qemu_input_event_sync (void ); +void qemu_input_event_sync_impl (void );InputEvent *qemu_input_event_new_key (KeyValue *key,bool down);void qemu_input_event_send_key (QemuConsole *src,KeyValue *key,bool down);
replay/Makefile.objs +1 -0
@@ -2,4 +2,5 @@ obj-$(CONFIG_SOFTMMU) += replay.o obj-$(CONFIG_SOFTMMU) += replay-internal.o obj-$(CONFIG_SOFTMMU) += replay-events.o obj-$(CONFIG_SOFTMMU) += replay-time.o +obj-$(CONFIG_SOFTMMU) += replay-input.o obj-$(CONFIG_USER_ONLY) += replay-user.o
replay/replay-events.c +33 -0
@@ -14,6 +14,7 @@#include "replay.h" #include "replay-internal.h" #include "block/aio.h" +#include "ui/input.h" typedef struct Event { ReplayAsyncEventKind event_kind;
@@ -39,6 +40,13 @@ static void replay_run_event(Event *event)case REPLAY_ASYNC_EVENT_PTIMER :aio_bh_call (event->opaque);break ; +case REPLAY_ASYNC_EVENT_INPUT : +qemu_input_event_send_impl (NULL , (InputEvent *)event->opaque); +qapi_free_InputEvent ((InputEvent *)event->opaque); +break ; +case REPLAY_ASYNC_EVENT_INPUT_SYNC : +qemu_input_event_sync_impl (); +break ;default :error_report ("Replay: invalid async event ID (%d) in the queue" , event->event_kind);
@@ -126,6 +134,16 @@ void replay_add_ptimer_event(void *bh, uint64_t id)replay_add_event (REPLAY_ASYNC_EVENT_PTIMER , bh,NULL , id); } +void replay_add_input_event (struct InputEvent *event) +{ +replay_add_event (REPLAY_ASYNC_EVENT_INPUT , event,NULL ,0 ); +} + +void replay_add_input_sync_event (void ) +{ +replay_add_event (REPLAY_ASYNC_EVENT_INPUT_SYNC ,NULL ,NULL ,0 ); +} +static void replay_save_event (Event *event,int checkpoint) {if (replay_mode !=REPLAY_MODE_PLAY ) {
@@ -139,6 +157,11 @@ static void replay_save_event(Event *event, int checkpoint)case REPLAY_ASYNC_EVENT_PTIMER :replay_put_qword (event->id);break ; +case REPLAY_ASYNC_EVENT_INPUT : +replay_save_input_event (event->opaque); +break ; +case REPLAY_ASYNC_EVENT_INPUT_SYNC : +break ;default :error_report ("Unknown ID %d of replay event" , read_event_kind);exit (1 );
@@ -182,6 +205,16 @@ static Event *replay_read_event(int checkpoint) read_id =replay_get_qword (); }break ; +case REPLAY_ASYNC_EVENT_INPUT : + event =g_malloc0 (sizeof (Event)); + event->event_kind = read_event_kind; + event->opaque =replay_read_input_event (); +return event; +case REPLAY_ASYNC_EVENT_INPUT_SYNC : + event =g_malloc0 (sizeof (Event)); + event->event_kind = read_event_kind; + event->opaque =0 ; +return event;default :error_report ("Unknown ID %d of replay event" , read_event_kind);exit (1 );
replay/replay-input.c new file +160 -0
160 lines not on this page · raw message
replay/replay-internal.h +13 -0
@@ -42,6 +42,8 @@ enum ReplayEvents {enum ReplayAsyncEventKind {REPLAY_ASYNC_EVENT_PTIMER , +REPLAY_ASYNC_EVENT_INPUT , +REPLAY_ASYNC_EVENT_INPUT_SYNC ,REPLAY_ASYNC_COUNT };
@@ -124,4 +126,15 @@ void replay_save_events(int checkpoint);/*! Read events from the file into the input queue */ void replay_read_events (int checkpoint); +/* Input events */ + +/*! Saves input event to the log */ +void replay_save_input_event (InputEvent *evt); +/*! Reads input event from the log */ +InputEvent *replay_read_input_event (void ); +/*! Adds input event to the queue */ +void replay_add_input_event (struct InputEvent *event); +/*! Adds input sync event to the queue */ +void replay_add_input_sync_event (void ); +#endif
replay/replay.h +4 -0
@@ -112,5 +112,9 @@ void replay_disable_events(void);bool replay_events_enabled (void );/*! Adds ptimer event to the queue */ void replay_add_ptimer_event (void *bh,uint64_t id); +/*! Adds input event to the queue */ +void replay_input_event (QemuConsole *src,InputEvent *evt); +/*! Adds input sync event to the queue */ +void replay_input_sync_event (void );#endif
ui/input.c +19 -8
@@ -6,6 +6,7 @@#include "trace.h" #include "ui/input.h" #include "ui/console.h" +#include "replay/replay.h" struct QemuInputHandlerState {DeviceState *dev;
@@ -300,14 +301,10 @@ static void qemu_input_queue_sync(struct QemuInputEventQueueHead *queue)QTAILQ_INSERT_TAIL (queue, item, node); } -void qemu_input_event_send (QemuConsole *src,InputEvent *evt) +void qemu_input_event_send_impl (QemuConsole *src,InputEvent *evt) {QemuInputHandlerState *s; -if (!runstate_is_running () && !runstate_check (RUN_STATE_SUSPENDED )) { -return ; - } -qemu_input_event_trace (src, evt);/* pre processing */
@@ -324,14 +321,19 @@ void qemu_input_event_send(QemuConsole *src, InputEvent *evt) s->events++; } -void qemu_input_event_sync (void ) +void qemu_input_event_send (QemuConsole *src,InputEvent *evt) { -QemuInputHandlerState *s; -if (!runstate_is_running () && !runstate_check (RUN_STATE_SUSPENDED )) {return ; } +replay_input_event (src, evt); +} + +void qemu_input_event_sync_impl (void ) +{ +QemuInputHandlerState *s; +trace_input_event_sync (); QTAILQ_FOREACH(s, &handlers, node) {
@@ -345,6 +347,15 @@ void qemu_input_event_sync(void) } } +void qemu_input_event_sync (void ) +{ +if (!runstate_is_running () && !runstate_check (RUN_STATE_SUSPENDED )) { +return ; + } + +replay_input_sync_event (); +} +InputEvent *qemu_input_event_new_key (KeyValue *key,bool down) {InputEvent *evt =g_new0 (InputEvent,1 );
PATCH v17 17/21 typedef: add typedef for QemuOpts
In reply to Pavel DovgalyukThis patch moves typedefs for QemuOpts and related types to qemu/typedefs.h file.
--- include/qemu/option.h | 5 +---- include/qemu/typedefs.h | 3 +++ 2 files changed, 4 insertions(+), 4 deletions(-)
include/qemu/option.h +1 -4
@@ -30,6 +30,7 @@#include "qemu/queue.h" #include "qapi/error.h" #include "qapi/qmp/qdict.h" +#include "qemu/typedefs.h" const char *get_opt_name (char *buf,int buf_size,const char *p,char delim);const char *get_opt_value (char *buf,int buf_size,const char *p);
@@ -44,10 +45,6 @@ void parse_option_size(const char *name, const char *value,bool has_help_option (const char *param);bool is_valid_option_list (const char *param); -typedef struct QemuOpt QemuOpt ; -typedef struct QemuOpts QemuOpts ; -typedef struct QemuOptsList QemuOptsList ; -enum QemuOptType {QEMU_OPT_STRING =0 ,/* no parsing (use string as-is) */ QEMU_OPT_BOOL ,/* on/off */
include/qemu/typedefs.h +3 -0
@@ -68,6 +68,9 @@ typedef struct QEMUBH QEMUBH;typedef struct QemuConsole QemuConsole ;typedef struct QEMUFile QEMUFile ;typedef struct QEMUMachine QEMUMachine ; +typedef struct QemuOpt QemuOpt ; +typedef struct QemuOpts QemuOpts ; +typedef struct QemuOptsList QemuOptsList ;typedef struct QEMUSGList QEMUSGList ;typedef struct QEMUSizedBuffer QEMUSizedBuffer ;typedef struct QEMUTimerListGroup QEMUTimerListGroup ;
PATCH v17 18/21 replay: initialization and deinitialization
In reply to Pavel DovgalyukThis patch introduces the functions for enabling the record/replay and for freeing the resources when simulator closes.
--- exec.c | 2 + replay/replay-internal.h | 2 + replay/replay.c | 129 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 10 ++++ stubs/replay.c | 5 ++ vl.c | 4 + 6 files changed, 152 insertions(+), 0 deletions(-)
exec.c +2 -0
@@ -51,6 +51,7 @@#include "qemu/main-loop.h" #include "exec/cputlb.h" #include "translate-all.h" +#include "replay/replay.h" #include "exec/memory-internal.h" #include "exec/ram_addr.h"
@@ -841,6 +842,7 @@ void cpu_abort(CPUState *cpu, const char *fmt, ...) }va_end (ap2);va_end (ap); +replay_finish ();#if defined(CONFIG_USER_ONLY ) {struct sigaction act;
replay/replay-internal.h +2 -0
@@ -33,6 +33,8 @@ enum ReplayEvents {/* some of greater codes are reserved for checkpoints */ EVENT_CHECKPOINT ,EVENT_CHECKPOINT_LAST =EVENT_CHECKPOINT +CHECKPOINT_COUNT -1 , +/* end of log event */ +EVENT_END , EVENT_COUNT };
replay/replay.c +129 -0
140 lines not on this page · raw message
replay/replay.h +10 -0
@@ -15,6 +15,7 @@#include <stdbool.h> #include <stdint.h> #include "qapi-types.h" +#include "qemu/typedefs.h" /* replay clock kinds */ enum ReplayClockKind {
@@ -42,6 +43,15 @@ typedef enum ReplayCheckpoint ReplayCheckpoint;extern ReplayMode replay_mode; +/* Replay process control functions */ + +/*! Enables recording or saving event log with specified parameters */ +void replay_configure (struct QemuOpts *opts); +/*! Initializes timers used for snapshotting and enables events recording */ +void replay_start (void ); +/*! Closes replay log file and frees other resources. */ +void replay_finish (void ); +/* Processing the instructions */ /*! Returns number of executed instructions. */
stubs/replay.c +5 -0
@@ -25,3 +25,8 @@ int runstate_is_running(void) {return 0 ; } + +bool replay_events_enabled (void ) +{ +return false; +}
vl.c +4 -0
@@ -4626,6 +4626,8 @@ int main(int argc, char **argv, char **envp)exit (1 ); } +replay_start (); +/* This checkpoint is required by replay to separate prior clock reading from the other reads, because timer polling functions query clock values from the log. */
@@ -4667,6 +4669,8 @@ int main(int argc, char **argv, char **envp) }main_loop (); +replay_disable_events (); +bdrv_close_all ();pause_all_vcpus ();res_free ();
Paolo,
Are these patches good enough?
Pavel Dovgalyuk
Quoted 263 lines
-----Original Message-----
From: Pavel Dovgalyuk [mailto:[email protected]]
Sent: Monday, September 07, 2015 11:40 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v17 00/21] Deterministic replay coreThis set of patches is related to the reverse execution and deterministic replay of qemu execution. This implementation of deterministic replay can be used for deterministic debugging of guest code through gdb remote interface.
Core set of patches does not include support for reverse debugging commands of gdb, block devices' operations, USB replay support.
These patches include only core function of the replay, excluding the support for replaying serial, audio, network, and USB devices' operations. Reverse debugging and monitor commands were also excluded to be submitted later as separate patches.
Execution recording writes non-deterministic events log, which can be later used for replaying the execution anywhere and for unlimited number of times. It also supports checkpointing for faster rewinding during reverse debugging. Execution replaying reads the log and replays all non-deterministic events including external input, hardware clocks, and interrupts.
Full version of deterministic replay has the following features:
* Deterministically replays whole system execution and all contents of the memory, state of the hadrware devices, clocks, and screen of the VM.
* Writes execution log into the file for latter replaying for multiple times on different machines.
* Supports i386,x86_64, ARM, PowerPC, and MIPS hardware platforms.
* Performs deterministic replay of all operations with keyboard and mouse input devices.
* Supports auto-checkpointing for convenient reverse debugging.Usage of the record/replay core:
* First, record the execution, by adding the following string to the command line:
'-icount shift=7,rr=record,rrfile=replay.bin -net none'.
Block devices' images are not actually changed in the recording mode, because all of the changes are written to the temporary overlay file.
* Then you can replay it for the multiple times by using another command line option: '-icount shift=7,rr=replay,rrfile=replay.bin -net none'
* '-net none' option should also be specified if network replay patches are not applied.
* Do not add any disk images to VM, because they are not supported by the core patches.Papers with description of deterministic replay implementation:
http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html
http://dl.acm.org/citation.cfm?id=2786805.2803179Public repository with current version of the patches:
https://github.com/Dovgalyuk/qemu/tree/rr-17Modifications of qemu include:
* wrappers for clock and time functions to save their return values in the log
* saving different asynchronous events (e.g. system shutdown) into the log
* synchronization of the threads from thread pool
* recording/replaying user input (mouse and keyboard)
* adding internal events for cpu and io synchronizationv17 changes:
* Removed useless stub functions (as suggested by Paolo Bonzini)
* Refined checkpoint-related code (as suggested by Paolo Bonzini)
* Improved icount processing (as suggested by Paolo Bonzini)
* Added checkpoint for suspend event (as suggested by Paolo Bonzini)
* Fixed linux-user configurations build
* Minor fixesv16 changes:
* Several warnings were fixedv15 changes:
* Tested record/replay with MIPS and PowerPC guests
* Published the patches on github
* Fixed replay mutex operation in icount mode
* Fixed timers processing in record/replay modev14 changes:
* Minor fixesv13 changes:
* Introduced "ptimer trigger" event (as suggested by Paolo Bonzini)v12 changes:
* Removed block patches from the core patches set.v11 changes:
* Fixed instructions event processing.
* Added some mutex protection calls for replay.
* Fixed replaying read operations for qcow2.
* Fixed rtc reads on initializations stage.
* Eliminated some warnings in replay module.
* Fixed misprints in documentation for replay (as suggested by Eric Blake)v10 changes:
* Fixed queue processing for bottom halves (as suggested by Paolo Bonzini)
* Rewritten several replay functions (as suggested by Paolo Bonzini)
* Some minor fixes.v9 changes:
* Replaced fwrite/fread with putc/getc (as suggested by Paolo Bonzini)
* Stopping virtual machine in case of replay file end (as suggested by Paolo Bonzini)
* Removed one of the replay mutexes (as suggested by Paolo Bonzini)
* Fixed RCU queue for bottom halves (as suggested by Paolo Bonzini)
* Updated command line options' names (as suggested by Paolo Bonzini)
* Added design document for record/replay (as suggested by Paolo Bonzini)
* Simplified checkpoints for the timers
* Added cloning InputEvent objects for replay (as suggested by Paolo Bonzini)
* Added replay blockers instead of checking the command line (as suggested by Paolo Bonzini)
* Some functions renaming and extracting.v8 changes:
* Simplified processing of the shutdown event (as suggested by Paolo Bonzini)
* Replaced stack of bottom halves in AIO context with QSIMPLEQ (as suggested by Paolo
Bonzini)
* Movedreplay_submodeout of the series (as suggested by Paolo Bonzini)
* Moved suffix option out of the series
* Converted some of the defines into enums (as suggested by Paolo Bonzini)
* Encapsulated save_tm/read_tm calls into the single function (as suggested by Paolo Bonzini)
* Moved record/replay options to icount group (as suggested by Paolo Bonzini)
* Updated mutex protection for the events queue (as suggested by Paolo Bonzini)
* Added mutex to protect replay log file (as suggested by Paolo Bonzini)
* Minor cleanupsv7 changes:
* Removed patches that were applied to upstream.v6 changes:
* Fixed replay stub return value (as suggested by Eric Blake)
* Fixed icount warping.
* Virtual rt clock now usescpu_get_clock()(as suggested by Paolo Bonzini)
* Replatedget_clock_realtimeandget_clockcalls with qemu clock requests (as suggested by
Paolo Bonzini)
* Modifiedcan_do_iologic to allow requesting icount fromcpu_execfunction (as suggested by
Paolo Bonzini)
* Removed applied patches.v5 changes:
* Minor changes.
* Used fixed-width integer types for read/write functions (as suggested by Alex Bennee)
* Moved savevm-related code out of the core.
* Added new traced clock for deterministic virtual clock warping (as suggested by Paolo
Bonzini)
* Fixedexception_indexreset for user mode (as suggested by Paolo Bonzini)
* Adopted Paolo's icount patches
* Fixed hardware interrupts replayingv4 changes:
* Updated block drivers to support newbdrv_openinterface.
* Moved migration patches into separate series (as suggested by Paolo Bonzini)
* Fixed a bug inreplay_breakoperation.
* Fixed rtl8139 migration for replay.
* Fixed 'period' parameter processing for record mode.
* Fixed bug in 'reverse-stepi' implementation.
* Fixed replay without making any snapshots (even the starting one).
* Moved core replay patches into the separate series.
* Fixed reverse step and reverse continue support.
* Fixed several bugs in icount subsystem.
* Reusing native qemu icount for replay instructions counting.
* Separated core patches into their own series.v3 changes:
* Fixed bug with replay of the aio write operations.
* Added virtual clock based on replay icount.
* Removed duplicated saving ofinterrupt_requestCPU field.
* Fixed some coding style issues.
* Renamed QMP commands for controlling reverse execution (as suggested by Eric Blake)
* Replay mode and submode implemented as QAPI enumerations (as suggested by Eric Blake)
* Added description and example for replay-info command (as suggested by Eric Blake)
* Added information about the current breakpoint to the output of replay-info (as suggested
by Eric Blake)
* Updated version id for HPET vmstate (as suggested by Paolo Bonzini)
* Removed static fields from parallel vmstate (as suggested by Paolo Bonzini)
* New vmstate fields for mc146818rtc, pckbd, kvmapic, serial, fdc, rtl8139 moved to
subsection (as suggested by Paolo Bonzini)
* Disabled textmode cursor blinking, when virtual machine is stopped (as suggested by Paolo
Bonzini)
* Extracted saving ofexception_indexto separate patch (as suggested by Paolo Bonzini)v2 changes:
* Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
* Added QMP versions of replay commands (as suggested by Eric Blake)
* Removed some optional features of replay to make patches cleaner
* Minor changes and code cleanup were made---
Pavel Dovgalyuk (21):
i386: partial revert of interrupt poll fix
replay: global variables and function stubs
replay: internal functions for replay log
replay: introduce mutex to protect the replay log
replay: introduce icount event
cpu-exec: allow temporary disabling icount
cpu: replay instructions sequence
i386: interrupt poll processing
replay: interrupts and exceptions
replay: asynchronous events infrastructure
replay: recording and replaying clock ticks
replay: shutdown event
icount: improve counting for record/replay
replay: checkpoints
bottom halves: introduce bh call function
replay: ptimer
typedef: add typedef for QemuOpts
replay: initialization and deinitialization
replay: replay blockers for devices
replay: command line options
replay: recording of the user inputMakefile.target | 1 async.c | 7 + cpu-exec.c | 63 +++++++- cpus.c | 57 +++++--- docs/replay.txt | 168 ++++++++++++++++++++++ exec.c | 2 hw/bt/hci.c | 7 + hw/core/ptimer.c | 7 + include/block/aio.h | 5 + include/exec/exec-all.h | 1 include/qapi/qmp/qerror.h | 3 include/qemu/option.h | 5 - include/qemu/typedefs.h | 3 include/ui/input.h | 2 main-loop.c | 3 qapi-schema.json | 18 ++ qemu-options.hx | 8 + qemu-timer.c | 48 +++++- replay/Makefile.objs | 6 + replay/replay-events.c | 274 ++++++++++++++++++++++++++++++++++++ replay/replay-input.c | 160 +++++++++++++++++++++ replay/replay-internal.c | 206 +++++++++++++++++++++++++++ replay/replay-internal.h | 140 ++++++++++++++++++ replay/replay-time.c | 64 ++++++++ replay/replay.c | 342 +++++++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 120 ++++++++++++++++ stubs/Makefile.objs | 1 stubs/replay.c | 32 ++++ target-i386/cpu.c | 10 - target-i386/seg_helper.c | 3 translate-all.c | 2 ui/input.c | 27 +++- vl.c | 53 ++++++- 33 files changed, 1779 insertions(+), 69 deletions(-) create mode 100755 docs/replay.txt create mode 100755 replay/Makefile.objs create mode 100755 replay/replay-events.c create mode 100755 replay/replay-input.c create mode 100755 replay/replay-internal.c create mode 100755 replay/replay-internal.h create mode 100755 replay/replay-time.c create mode 100755 replay/replay.c create mode 100755 replay/replay.h create mode 100755 stubs/replay.c--
Pavel Dovgalyuk
Quoted 3 lines · Pavel Dovgaluk
Paolo,
Are these patches good enough?
Haven't reviewed them as I already have a 50 patch queue. I will look at them next week.
I can already tell you that I'd like some comments in front of clock warp calls, but that can be added as a follow up.
Paolo