Deterministic replay core
Subsystems (2)
v8 cover Deterministic replay core
This set of patches is related to the reverse execution and deterministic replay of qemu execution This implementation of deterministic replay can be used for deterministic debugging of guest code through gdb remote interface.
These patches include only core function of the replay, excluding the support for replaying serial, audio, network, and USB devices' operations. Reverse debugging and monitor commands were also excluded to be submitted later as separate patches.
Execution recording writes non-deterministic events log, which can be later used for replaying the execution anywhere and for unlimited number of times. It also supports checkpointing for faster rewinding during reverse debugging. Execution replaying reads the log and replays all non-deterministic events including external input, hardware clocks, and interrupts.
Deterministic replay has the following features:
* Deterministically replays whole system execution and all contents of the memory, state of the hadrware devices, clocks, and screen of the VM.
* Writes execution log into the file for latter replaying for multiple times on different machines.
* Supports i386, x86_64, and ARM hardware platforms.
* Performs deterministic replay of all operations with keyboard and mouse input devices.
* Supports auto-checkpointing for convenient reverse debugging.
Usage of the record/replay:
* First, record the execution, by adding the following string to the command line:
'-record fname=replay.bin -icount 7 -net none'. Block devices' images are not actually changed in the recording mode, because all of the changes are written to the temporary overlay file.
* Then you can replay it for the multiple times by using another command line option: '-replay fname=replay.bin -icount 7 -net none'
* '-net none' option should also be specified if network replay patches are not applied.
Paper with short description of deterministic replay implementation:
http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html
Modifications of qemu include:
* wrappers for clock and time functions to save their return values in the log
* saving different asynchronous events (e.g. system shutdown) into the log
* synchronization of the bottom halves execution
* synchronization of the threads from thread pool
* recording/replaying user input (mouse and keyboard)
* adding internal events for cpu and io synchronization
v8 changes:
* Simplified processing of the shutdown event (as suggested by Paolo Bonzini)
* Replaced stack of bottom halves in AIO context with QSIMPLEQ (as suggested by Paolo Bonzini)
* Moved replay_submode out of the series (as suggested by Paolo Bonzini)
* Moved suffix option out of the series
* Converted some of the defines into enums (as suggested by Paolo Bonzini)
* Encapsulated save_tm/read_tm calls into the single function (as suggested by Paolo Bonzini)
* Moved record/replay options to icount group (as suggested by Paolo Bonzini)
* Updated mutex protection for the events queue (as suggested by Paolo Bonzini)
* Added mutex to protect replay log file (as suggested by Paolo Bonzini)
* Minor cleanups
v7 changes:
* Removed patches that were applied to upstream.
v6 changes:
* Fixed replay stub return value (as suggested by Eric Blake)
* Fixed icount warping.
* Virtual rt clock now uses cpu_get_clock() (as suggested by Paolo Bonzini)
* Replated get_clock_realtime and get_clock calls with qemu clock requests (as suggested by Paolo Bonzini)
* Modified can_do_io logic to allow requesting icount from cpu_exec function (as suggested by Paolo Bonzini)
* Removed applied patches.
v5 changes:
* Minor changes.
* Used fixed-width integer types for read/write functions (as suggested by Alex Bennee)
* Moved savevm-related code out of the core.
* Added new traced clock for deterministic virtual clock warping (as suggested by Paolo Bonzini)
* Fixed exception_index reset for user mode (as suggested by Paolo Bonzini)
* Adopted Paolo's icount patches
* Fixed hardware interrupts replaying
v4 changes:
* Updated block drivers to support new bdrv_open interface.
* Moved migration patches into separate series (as suggested by Paolo Bonzini)
* Fixed a bug in replay_break operation.
* Fixed rtl8139 migration for replay.
* Fixed 'period' parameter processing for record mode.
* Fixed bug in 'reverse-stepi' implementation.
* Fixed replay without making any snapshots (even the starting one).
* Moved core replay patches into the separate series.
* Fixed reverse step and reverse continue support.
* Fixed several bugs in icount subsystem.
* Reusing native qemu icount for replay instructions counting.
* Separated core patches into their own series.
v3 changes:
* Fixed bug with replay of the aio write operations.
* Added virtual clock based on replay icount.
* Removed duplicated saving of interrupt_request CPU field.
* Fixed some coding style issues.
* Renamed QMP commands for controlling reverse execution (as suggested by Eric Blake)
* Replay mode and submode implemented as QAPI enumerations (as suggested by Eric Blake)
* Added description and example for replay-info command (as suggested by Eric Blake)
* Added information about the current breakpoint to the output of replay-info (as suggested by Eric Blake)
* Updated version id for HPET vmstate (as suggested by Paolo Bonzini)
* Removed static fields from parallel vmstate (as suggested by Paolo Bonzini)
* New vmstate fields for mc146818rtc, pckbd, kvmapic, serial, fdc, rtl8139 moved to subsection (as suggested by Paolo Bonzini)
* Disabled textmode cursor blinking, when virtual machine is stopped (as suggested by Paolo Bonzini)
* Extracted saving of exception_index to separate patch (as suggested by Paolo Bonzini)
v2 changes:
* Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
* Added QMP versions of replay commands (as suggested by Eric Blake)
* Removed some optional features of replay to make patches cleaner
* Minor changes and code cleanup were made
---
Pavel Dovgalyuk (21):
i386: partial revert of interrupt poll fix
replay: global variables and function stubs
sysemu: system functions for replay
replay: internal functions for replay log
replay: introduce mutex to protect the replay log
replay: introduce icount event
cpu-exec: allow temporary disabling icount
cpu: replay instructions sequence
replay: interrupts and exceptions
replay: asynchronous events infrastructure
replay: recording and replaying clock ticks
replay: recording and replaying different timers
replay: shutdown event
replay: checkpoints
aio: replace stack of bottom halves with queue
replay: bottom halves
replay: replay aio requests
replay: thread pool
replay: initialization and deinitialization
replay: command line options
replay: recording of the user input
Makefile.target | 1 async.c | 49 ++++-- block.c | 92 ++++++++++- block/block-backend.c | 30 +++- block/qcow2.c | 4 block/raw-posix.c | 6 - block/raw-win32.c | 4 cpu-exec.c | 60 ++++++- cpus.c | 45 ++++- dma-helpers.c | 10 + exec.c | 1 hw/block/virtio-blk.c | 10 + hw/ide/ahci.c | 4 hw/ide/atapi.c | 10 + hw/ide/core.c | 18 +- hw/timer/arm_timer.c | 2 hw/timer/mc146818rtc.c | 3 hw/timer/pl031.c | 3 hw/usb/hcd-uhci.c | 2 include/block/aio.h | 22 ++- include/block/block.h | 15 ++ include/block/thread-pool.h | 4 include/exec/exec-all.h | 2 include/qemu-common.h | 3 include/qemu/main-loop.h | 1 include/qemu/queue.h | 7 + include/qemu/timer.h | 16 ++ include/qom/cpu.h | 10 + include/sysemu/block-backend.h | 10 + include/sysemu/cpus.h | 1 include/ui/input.h | 2 main-loop.c | 10 + qapi-schema.json | 18 ++ qemu-io-cmds.c | 2 qemu-options.hx | 8 + qemu-timer.c | 56 +++++-- replay/Makefile.objs | 5 + replay/replay-events.c | 310 ++++++++++++++++++++++++++++++++++++ replay/replay-input.c | 130 +++++++++++++++ replay/replay-internal.c | 183 +++++++++++++++++++++ replay/replay-internal.h | 149 +++++++++++++++++ replay/replay-time.c | 216 +++++++++++++++++++++++++ replay/replay.c | 341 ++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 132 +++++++++++++++ stubs/Makefile.objs | 1 stubs/replay.c | 37 ++++ target-i386/cpu.c | 10 - target-i386/seg_helper.c | 4 tests/test-thread-pool.c | 7 - thread-pool.c | 49 ++++-- trace-events | 2 translate-all.c | 10 + ui/input.c | 68 ++++++-- util/iov.c | 4 vl.c | 54 ++++++ 55 files changed, 2101 insertions(+), 152 deletions(-) create mode 100755 replay/Makefile.objs create mode 100755 replay/replay-events.c create mode 100755 replay/replay-input.c create mode 100755 replay/replay-internal.c create mode 100755 replay/replay-internal.h create mode 100755 replay/replay-time.c create mode 100755 replay/replay.c create mode 100755 replay/replay.h create mode 100755 stubs/replay.c
Signature · 1 line
Pavel Dovgalyuk
RFC PATCH v8 1/21 i386: partial revert of interrupt poll fix
Processing CPU_INTERRUPT_POLL requests in cpu_has_work functions break the determinism of cpu_exec. This patch is required to make interrupts processing deterministic.
--- cpu-exec.c | 6 ++++++ target-i386/cpu.c | 10 ++-------- 2 files changed, 8 insertions(+), 8 deletions(-)
cpu-exec.c +6 -0
@@ -338,6 +338,12 @@ int cpu_exec(CPUArchState *env)volatile bool have_tb_lock = false;if (cpu->halted) { +#ifdef TARGET_I386 +if (cpu->interrupt_request &CPU_INTERRUPT_POLL ) { +apic_poll_irq (x86_cpu->apic_state); +cpu_reset_interrupt (cpu,CPU_INTERRUPT_POLL ); + } +#endif if (!cpu_has_work (cpu)) {return EXCP_HALTED ; }
target-i386/cpu.c +2 -8
@@ -2970,14 +2970,8 @@ static bool x86_cpu_has_work(CPUState *cs)X86CPU *cpu =X86_CPU (cs);CPUX86State *env = &cpu->env; -#if !defined(CONFIG_USER_ONLY ) -if (cs->interrupt_request &CPU_INTERRUPT_POLL ) { -apic_poll_irq (cpu->apic_state); -cpu_reset_interrupt (cs,CPU_INTERRUPT_POLL ); - } -#endif - -return ((cs->interrupt_request &CPU_INTERRUPT_HARD ) && +return ((cs->interrupt_request & (CPU_INTERRUPT_HARD | +CPU_INTERRUPT_POLL )) && (env->eflags & IF_MASK)) || (cs->interrupt_request & (CPU_INTERRUPT_NMI | CPU_INTERRUPT_INIT |
RFC PATCH v8 2/21 replay: global variables and function stubs
In reply to Pavel DovgalyukThis patch adds global variables, defines, functions declarations, and function stubs for deterministic VM replay used by external modules.
--- Makefile.target | 1 + qapi-schema.json | 18 ++++++++++++++++++ replay/Makefile.objs | 1 + replay/replay.c | 14 ++++++++++++++ replay/replay.h | 19 +++++++++++++++++++ stubs/Makefile.objs | 1 + stubs/replay.c | 3 +++ 7 files changed, 57 insertions(+), 0 deletions(-) create mode 100755 replay/Makefile.objs create mode 100755 replay/replay.c create mode 100755 replay/replay.h create mode 100755 stubs/replay.c
Makefile.target +1 -0
@@ -83,6 +83,7 @@ all: $(PROGS) stap ######################################################### # cpu emulator library obj-y = exec.o translate-all.o cpu-exec.o +obj-y += replay/ obj-y += tcg/tcg.o tcg/optimize.o obj-$(CONFIG_TCG_INTERPRETER) += tci.o obj-$(CONFIG_TCG_INTERPRETER) += disas/tci.o
qapi-schema.json +18 -0
@@ -3523,3 +3523,21 @@ # Since: 2.1 ## { 'command': 'rtc-reset-reinjection' } + +## +# ReplayMode: +# +# Mode of the replay subsystem. +# +# @none: normal execution mode. Replay or record are not enabled. +# +# @record: record mode. All non-deterministic data is written into the +# replay log. +# +# @play: replay mode. Non-deterministic data required for system execution +# is read from the log. +# +# Since: 2.3 +## +{ 'enum': 'ReplayMode', + 'data': [ 'none', 'record', 'play' ] }
replay/Makefile.objs new file +1 -0
@@ -0,0 +1 @@ +obj-y += replay.o
replay/replay.c new file +14 -0
@@ -0,0 +1,14 @@ +/* +* replay.c +* +* Copyright (c) 2010-2015 Institute for System Programming +* of the Russian Academy of Sciences. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ + +#include "replay.h" + +ReplayMode replay_mode =REPLAY_MODE_NONE ;
replay/replay.h new file +19 -0
@@ -0,0 +1,19 @@ +#ifndef REPLAY_H +#define REPLAY_H + +/* +* replay.h +* +* Copyright (c) 2010-2015 Institute for System Programming +* of the Russian Academy of Sciences. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ + +#include "qapi-types.h" + +extern ReplayMode replay_mode; + +#endif
stubs/Makefile.objs +1 -0
@@ -27,6 +27,7 @@ stub-obj-y += notify-event.o stub-obj-y += pci-drive-hot-add.o stub-obj-$(CONFIG_SPICE) += qemu-chr-open-spice.o stub-obj-y += qtest.o +stub-obj-y += replay.o stub-obj-y += reset.o stub-obj-y += runstate-check.o stub-obj-y += set-fd-handler.o
stubs/replay.c new file +3 -0
@@ -0,0 +1,3 @@ +#include "replay/replay.h" + +ReplayMode replay_mode;
Re: [Qemu-devel] [RFC PATCH v8 02/21] replay: global variables and function stubs
Quoted 127 lines · Pavel Dovgalyuk
This patch adds global variables, defines, functions declarations, and function stubs for deterministic VM replay used by external modules.
Signed-off-by Pavel Dovgalyuk <[email protected]>--- Makefile.target | 1 + qapi-schema.json | 18 ++++++++++++++++++ replay/Makefile.objs | 1 + replay/replay.c | 14 ++++++++++++++ replay/replay.h | 19 +++++++++++++++++++ stubs/Makefile.objs | 1 + stubs/replay.c | 3 +++ 7 files changed, 57 insertions(+), 0 deletions(-) create mode 100755 replay/Makefile.objs create mode 100755 replay/replay.c create mode 100755 replay/replay.h create mode 100755 stubs/replay.cMakefile.target +1
-0@@ -83,6 +83,7 @@ all: $(PROGS) stap ######################################################### # cpu emulator library obj-y = exec.o translate-all.o cpu-exec.o +obj-y += replay/ obj-y += tcg/tcg.o tcg/optimize.o obj-$(CONFIG_TCG_INTERPRETER) += tci.o obj-$(CONFIG_TCG_INTERPRETER) += disas/tci.oqapi-schema.json +18
-0@@ -3523,3 +3523,21 @@ # Since: 2.1 ## { 'command': 'rtc-reset-reinjection' } + +## +# ReplayMode: +# +# Mode of the replay subsystem. +# +# @none: normal execution mode. Replay or record are not enabled. +# +# @record: record mode. All non-deterministic data is written into the +# replay log. +# +# @play: replay mode. Non-deterministic data required for system execution +# is read from the log. +# +# Since: 2.3 +## +{ 'enum': 'ReplayMode', + 'data': [ 'none', 'record', 'play' ] }replay/Makefile.objs new file +1
-0@@ -0,0 +1 @@ +obj-y += replay.oreplay/replay.c new file +14
-0@@ -0,0 +1,14 @@ +/* +* replay.c +* +* Copyright (c) 2010-2015 Institute for System Programming +* of the Russian Academy of Sciences. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ + +#include "replay.h" + +ReplayMode replay_mode =REPLAY_MODE_NONE ;replay/replay.h new file +19
-0@@ -0,0 +1,19 @@ +#ifndef REPLAY_H +#define REPLAY_H + +/* +* replay.h +* +* Copyright (c) 2010-2015 Institute for System Programming +* of the Russian Academy of Sciences. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ + +#include "qapi-types.h" + +extern ReplayMode replay_mode; + +#endif stubs/Makefile.objs +1
-0@@ -27,6 +27,7 @@ stub-obj-y += notify-event.o stub-obj-y += pci-drive-hot-add.o stub-obj-$(CONFIG_SPICE) += qemu-chr-open-spice.o stub-obj-y += qtest.o +stub-obj-y += replay.o stub-obj-y += reset.o stub-obj-y += runstate-check.o stub-obj-y += set-fd-handler.ostubs/replay.c new file +0
-0
stubs/replay.c line 3
Quoted 3 lines
stubs/replay.c new file +3
-0@@ -0,0 +1,3 @@ +#include "replay/replay.h" + +ReplayMode replay_mode;
Re: [Qemu-devel] [RFC PATCH v8 02/21] replay: global variables and function stubs
In reply to Pavel DovgalyukQuoted 4 lines · Pavel Dovgalyuk
This patch adds global variables, defines, functions declarations, and function stubs for deterministic VM replay used by external modules.
Signed-off-by Pavel Dovgalyuk <[email protected]>
Quoted 13 lines
--- Makefile.target | 1 + qapi-schema.json | 18 ++++++++++++++++++ replay/Makefile.objs | 1 + replay/replay.c | 14 ++++++++++++++ replay/replay.h | 19 +++++++++++++++++++ stubs/Makefile.objs | 1 + stubs/replay.c | 3 +++ 7 files changed, 57 insertions(+), 0 deletions(-) create mode 100755 replay/Makefile.objs create mode 100755 replay/replay.c create mode 100755 replay/replay.h create mode 100755 stubs/replay.c
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v8 3/21 sysemu: system functions for replay
In reply to Pavel DovgalyukThis patch removes "static" specifier from several qemu function to make them visible to the replay module. It also invents several system functions that will be used by replay.
--- cpus.c | 4 ++-- include/exec/exec-all.h | 1 + include/qom/cpu.h | 10 ++++++++++ include/sysemu/cpus.h | 1 + translate-all.c | 8 ++++++++ 5 files changed, 22 insertions(+), 2 deletions(-)
cpus.c +2 -2
@@ -88,7 +88,7 @@ static bool cpu_thread_is_idle(CPUState *cpu)return true; } -static bool all_cpu_threads_idle (void ) +bool all_cpu_threads_idle (void ) {CPUState *cpu;
@@ -1121,7 +1121,7 @@ bool qemu_cpu_is_self(CPUState *cpu)return qemu_thread_is_self (cpu->thread); } -static bool qemu_in_vcpu_thread (void ) +bool qemu_in_vcpu_thread (void ) {return current_cpu &&qemu_cpu_is_self (current_cpu); }
include/exec/exec-all.h +1 -0
@@ -214,6 +214,7 @@ static inline unsigned int tb_phys_hash_func(tb_page_addr_t pc)void tb_free (TranslationBlock *tb);void tb_flush (CPUArchState *env); +void tb_flush_all (void );void tb_phys_invalidate (TranslationBlock *tb,tb_page_addr_t page_addr);#if defined(USE_DIRECT_JUMP )
include/qom/cpu.h +10 -0
@@ -480,6 +480,16 @@ static inline bool cpu_has_work(CPUState *cpu)bool qemu_cpu_is_self (CPUState *cpu);/** +* qemu_in_vcpu_thread: +* +* Checks whether the caller is executing on the vCPU thread +* of the current vCPU. +* +* Returns: %true if called from vCPU's thread, %false otherwise. +*/ +bool qemu_in_vcpu_thread (void ); + +/** * qemu_cpu_kick: * @cpu: The vCPU to kick. *
include/sysemu/cpus.h +1 -0
@@ -6,6 +6,7 @@ void qemu_init_cpu_loop(void);void resume_all_vcpus (void );void pause_all_vcpus (void );void cpu_stop_current (void ); +bool all_cpu_threads_idle (void );void cpu_synchronize_all_states (void );void cpu_synchronize_all_post_reset (void );
translate-all.c +8 -0
@@ -812,6 +812,14 @@ void tb_flush(CPUArchState *env1) tcg_ctx.tb_ctx.tb_flush_count++; } +void tb_flush_all (void ) +{ +CPUState *cpu; +for (cpu = first_cpu ; cpu !=NULL ; cpu =CPU_NEXT (cpu)) { +tb_flush (cpu->env_ptr); + } +} +#ifdef DEBUG_TB_CHECK static void tb_invalidate_check (target_ulong address)
Re: [Qemu-devel] [RFC PATCH v8 03/21] sysemu: system functions for replay
Quoted 90 lines · Pavel Dovgalyuk
This patch removes "static" specifier from several qemu function to make them visible to the replay module. It also invents several system functions that will be used by replay.
Signed-off-by Pavel Dovgalyuk <[email protected]>--- cpus.c | 4 ++-- include/exec/exec-all.h | 1 + include/qom/cpu.h | 10 ++++++++++ include/sysemu/cpus.h | 1 + translate-all.c | 8 ++++++++ 5 files changed, 22 insertions(+), 2 deletions(-)cpus.c +2
-2@@ -88,7 +88,7 @@ static bool cpu_thread_is_idle(CPUState *cpu)return true; } -static bool all_cpu_threads_idle (void ) +bool all_cpu_threads_idle (void ) {CPUState *cpu;@@ -1121,7 +1121,7 @@ bool qemu_cpu_is_self(CPUState *cpu)return qemu_thread_is_self (cpu->thread); } -static bool qemu_in_vcpu_thread (void ) +bool qemu_in_vcpu_thread (void ) {return current_cpu &&qemu_cpu_is_self (current_cpu); }include/exec/exec-all.h +1
-0@@ -214,6 +214,7 @@ static inline unsigned int tb_phys_hash_func(tb_page_addr_t pc)void tb_free (TranslationBlock *tb);void tb_flush (CPUArchState *env); +void tb_flush_all (void );void tb_phys_invalidate (TranslationBlock *tb,tb_page_addr_t page_addr);#if defined(USE_DIRECT_JUMP )include/qom/cpu.h +10
-0@@ -480,6 +480,16 @@ static inline bool cpu_has_work(CPUState *cpu)bool qemu_cpu_is_self (CPUState *cpu);/** +* qemu_in_vcpu_thread: +* +* Checks whether the caller is executing on the vCPU thread +* of the current vCPU. +* +* Returns: %true if called from vCPU's thread, %false otherwise. +*/ +bool qemu_in_vcpu_thread (void ); + +/** * qemu_cpu_kick: * @cpu: The vCPU to kick. * include/sysemu/cpus.h +1
-0@@ -6,6 +6,7 @@ void qemu_init_cpu_loop(void);void resume_all_vcpus (void );void pause_all_vcpus (void );void cpu_stop_current (void ); +bool all_cpu_threads_idle (void );void cpu_synchronize_all_states (void );void cpu_synchronize_all_post_reset (void );translate-all.c +1
-0@@ -812,6 +812,14 @@ void tb_flush(CPUArchState *env1) tcg_ctx.tb_ctx.tb_flush_count++; } +void tb_flush_all (void )
translate-all.c void tb_flush(CPUArchState *env1) · line 825
Quoted 10 lines
translate-all.c +7
-0+{ +CPUState *cpu; +for (cpu = first_cpu ; cpu !=NULL ; cpu =CPU_NEXT (cpu)) { +tb_flush (cpu->env_ptr); + } +} +#ifdef DEBUG_TB_CHECK static void tb_invalidate_check (target_ulong address)
RFC PATCH v8 4/21 replay: internal functions for replay log
In reply to Pavel DovgalyukThis patch adds functions to perform read and write operations with replay log.
--- replay/Makefile.objs | 1 replay/replay-internal.c | 141 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay-internal.h | 50 ++++++++++++++++ 3 files changed, 192 insertions(+), 0 deletions(-) create mode 100755 replay/replay-internal.c create mode 100755 replay/replay-internal.h
replay/Makefile.objs +1 -0
@@ -1 +1,2 @@ obj-y += replay.o +obj-y += replay-internal.o
replay/replay-internal.c new file +141 -0
141 lines not on this page · raw message
replay/replay-internal.h new file +50 -0
50 lines not on this page · raw message
Re: [Qemu-devel] [RFC PATCH v8 04/21] replay: internal functions for replay log
Quoted 39 lines · Pavel Dovgalyuk
This patch adds functions to perform read and write operations with replay log.
Signed-off-by Pavel Dovgalyuk <[email protected]>--- replay/Makefile.objs | 1 replay/replay-internal.c | 141 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay-internal.h | 50 ++++++++++++++++ 3 files changed, 192 insertions(+), 0 deletions(-) create mode 100755 replay/replay-internal.c create mode 100755 replay/replay-internal.hreplay/Makefile.objs +1
-0@@ -1 +1,2 @@ obj-y += replay.o +obj-y += replay-internal.oreplay/replay-internal.c new file +14
-0@@ -0,0 +1,141 @@ +/* +* replay-internal.c +* +* Copyright (c) 2010-2015 Institute for System Programming +* of the Russian Academy of Sciences. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ + +#include "qemu-common.h" +#include "replay-internal.h" +
replay/replay-internal.c line 24
Quoted 10 lines
replay/replay-internal.c new file +10
-0+unsigned int replay_data_kind =-1 ; +unsigned int replay_has_unread_data; + +/* File for replay writing */ +FILE *replay_file; + +void replay_put_byte (uint8_t byte) +{ +if (replay_file) { +fwrite (&byte,sizeof (byte),1 , replay_file);
putc(byte, replay_file);
replay/replay-internal.c line 37
Quoted 13 lines
replay/replay-internal.c new file +13
-0+ } +} + +void replay_put_event (uint8_t event) +{ +replay_put_byte (event); +} + + +void replay_put_word (uint16_t word) +{ +if (replay_file) { +fwrite (&word,sizeof (word),1 , replay_file);
Let's make this use a guaranteed endianness. QEMU usually uses big endian. So either use putc, or use cpu_to_be16 and fwrite.
replay/replay-internal.c line 44
Quoted 7 lines
replay/replay-internal.c new file +7
-0+ } +} + +void replay_put_dword (uint32_t dword) +{ +if (replay_file) { +fwrite (&dword,sizeof (dword),1 , replay_file);
Same here.
replay/replay-internal.c line 51
Quoted 7 lines
replay/replay-internal.c new file +7
-0+ } +} + +void replay_put_qword (int64_t qword) +{ +if (replay_file) { +fwrite (&qword,sizeof (qword),1 , replay_file);
And same here.
replay/replay-internal.c line 58
Quoted 7 lines
replay/replay-internal.c new file +7
-0+ } +} + +void replay_put_array (const uint8_t *buf,size_t size) +{ +if (replay_file) { +fwrite (&size,sizeof (size),1 , replay_file);
replay_put_dword (a limit of 2^32 elements should not be a problem :)).
replay/replay-internal.c line 67
Quoted 9 lines
replay/replay-internal.c new file +9
-0+fwrite (buf,1 , size, replay_file); + } +} + +uint8_t replay_get_byte (void ) +{ +uint8_t byte; +if (replay_file) { +fread (&byte,sizeof (byte),1 , replay_file);
byte = getc(replay_file)
replay/replay-internal.c line 76
Quoted 9 lines
replay/replay-internal.c new file +9
-0+ } +return byte; +} + +uint16_t replay_get_word (void ) +{ +uint16_t word; +if (replay_file) { +fread (&word,sizeof (word),1 , replay_file);
Same comment about endianness.
Quoted 33 lines
replay/replay-internal.c new file +33
-0+ } + +return word; +} + +uint32_t replay_get_dword (void ) +{ +uint32_t dword; +if (replay_file) { +fread (&dword,sizeof (dword),1 , replay_file); + } + +return dword; +} + +int64_t replay_get_qword (void ) +{ +int64_t qword; +if (replay_file) { +fread (&qword,sizeof (qword),1 , replay_file); + } + +return qword; +} + +void replay_get_array (uint8_t *buf,size_t *size) +{ +if (replay_file) { +fread (size,sizeof (*size),1 , replay_file); +fread (buf,1 , *size, replay_file); + } +} +
replay/replay-internal.c line 119
Quoted 10 lines
replay/replay-internal.c new file +10
-0+void replay_get_array_alloc (uint8_t **buf,size_t *size) +{ +if (replay_file) { +fread (size,sizeof (*size),1 , replay_file); + *buf =g_malloc (*size); +fread (*buf,1 , *size, replay_file); + } +} + +void replay_check_error (void )
Could this be static? (I haven't checked).
replay/replay-internal.c line 124
Quoted 5 lines
replay/replay-internal.c new file +5
-0+{ +if (replay_file) { +if (feof (replay_file)) { +fprintf (stderr,"replay file is over\n" ); +exit (1 );
Perhaps qemu_system_vmstop_request_prepare + qemu_system_vmstop_request(RUN_STATE_PAUSED) instead of exit? Those two functions are thread-safe.
replay/replay-internal.c line 126
Quoted 2 lines
replay/replay-internal.c new file +2
-0+ }else if (ferror (replay_file)) { +fprintf (stderr,"replay file is over or something goes wrong\n" );
Same here (RUN_STATE_INTERNAL_ERROR?).
Quoted 59 lines
replay/replay-internal.c new file +15
-0+exit (1 ); + } + } +} + +void replay_fetch_data_kind (void ) +{ +if (replay_file) { +if (!replay_has_unread_data) { + replay_data_kind =replay_get_byte (); +replay_check_error (); + replay_has_unread_data =1 ; + } + } +}replay/replay-internal.h new file +38
-0@@ -0,0 +1,50 @@ +#ifndef REPLAY_INTERNAL_H +#define REPLAY_INTERNAL_H + +/* +* replay-internal.h +* +* Copyright (c) 2010-2015 Institute for System Programming +* of the Russian Academy of Sciences. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ + +#include <stdio.h> + +extern unsigned int replay_data_kind; +extern unsigned int replay_has_unread_data; + +/* File for replay writing */ +extern FILE *replay_file; + +void replay_put_byte (uint8_t byte); +void replay_put_event (uint8_t event); +void replay_put_word (uint16_t word); +void replay_put_dword (uint32_t dword); +void replay_put_qword (int64_t qword); +void replay_put_array (const uint8_t *buf,size_t size); + +uint8_t replay_get_byte (void ); +uint16_t replay_get_word (void ); +uint32_t replay_get_dword (void ); +int64_t replay_get_qword (void ); +void replay_get_array (uint8_t *buf,size_t *size); +void replay_get_array_alloc (uint8_t **buf,size_t *size); + +/*! Checks error status of the file. */ +void replay_check_error (void );
replay/replay-internal.h line 48
Quoted 10 lines
replay/replay-internal.h new file +10
-0+ +/*! Reads data type from the file and stores it in the +replay_data_kind variable. */ +void replay_fetch_data_kind (void ); + +/*! Saves queued events (like instructions and sound). */ +void replay_save_instructions (void ); +/*! Checks that the next data is corresponding to the desired kind. +Terminates the program in case of error. */ +void validate_data_kind (int kind);
Please move the last two prototypes to the patches that introduce the functions.
Quoted 3 lines
replay/replay-internal.h new file +2
-0+ +#endif
Re: [Qemu-devel] [RFC PATCH v8 04/21] replay: internal functions for replay log
Quoted 10 lines
From: Paolo Bonzini [mailto:[email protected]]
Quoted 6 lines · Pavel Dovgalyuk
This patch adds functions to perform read and write operations with replay log.
Signed-off-by Pavel Dovgalyuk <[email protected]>---
+voidreplay_check_error(void)Could this be static? (I haven't checked).
No, because it is used from several replay files.
Quoted 9 lines
replay/replay-internal.c line 124
Quoted 5 lines
replay/replay-internal.c new file +5
-0+{ +if (replay_file) { +if (feof (replay_file)) { +fprintf (stderr,"replay file is over\n" ); +exit (1 );Perhaps
qemu_system_vmstop_request_prepare+qemu_system_vmstop_request(RUN_STATE_PAUSED)instead of exit? Those two functions are thread-safe.
There is no need to stop when replay file is over (because we cannot replay more). Should we send shutdown request instead?
Pavel Dovgalyuk
Re: [Qemu-devel] [RFC PATCH v8 04/21] replay: internal functions for replay log
Quoted 3 lines · Pavel Dovgaluk
Quoted 1 line
Could this be static? (I haven't checked).
No, because it is used from several replay files.
I wonder if that's a layering violation.
Quoted 6 lines
Quoted 3 lines
Perhaps
qemu_system_vmstop_request_prepare+qemu_system_vmstop_request(RUN_STATE_PAUSED)instead of exit? Those two functions are thread-safe.There is no need to stop when replay file is over (because we cannot replay more). Should we send shutdown request instead?
I thought about it. I think no, because shutdown is irreversible (see runstate_needs_reset). Just pausing seemed to be the right compromise, and then the next "cont" can run the VM out of replay mode.
Paolo
Re: [Qemu-devel] [RFC PATCH v8 04/21] replay: internal functions for replay log
I believe thats what we concluded too
Cheers
Mark.
Quoted 15 lines · Paolo Bonzini
Quoted 3 lines · Pavel Dovgaluk
Quoted 1 line
Could this be static? (I haven't checked).
No, because it is used from several replay files.
I wonder if that's a layering violation.
Quoted 4 lines
Quoted 3 lines
Perhaps
qemu_system_vmstop_request_prepare+qemu_system_vmstop_request(RUN_STATE_PAUSED)instead of exit? Those two functions are thread-safe.
Quoted 8 lines
Quoted 2 lines
There is no need to stop when replay file is over (because we cannot replay more). Should we send shutdown request instead?
I thought about it. I think no, because shutdown is irreversible (see
runstate_needs_reset). Just pausing seemed to be the right compromise, and then the next "cont" can run the VM out of replay mode.Paolo
+44 (0)20 7100 3485 x 210
+33 (0)5 33 52 01 77x 210
+33 (0)603762104 mark.burton
RFC PATCH v8 5/21 replay: introduce mutex to protect the replay log
In reply to Pavel DovgalyukThis mutex will protect read/write operations for replay log. Using mutex is necessary because most of the events consist of several fields stored in the log. The mutex will help to avoid races.
--- replay/replay-internal.c | 25 +++++++++++++++++++++++++ replay/replay-internal.h | 7 +++++++ 2 files changed, 32 insertions(+), 0 deletions(-)
replay/replay-internal.c +25 -0
@@ -15,6 +15,11 @@unsigned int replay_data_kind =-1 ;unsigned int replay_has_unread_data; +/* Mutex to protect reading and writing events to the log. +replay_data_kind and replay_has_unread_data are also protected +by this mutex. */ +static QemuMutex lock; +/* File for replay writing */ FILE *replay_file;
@@ -139,3 +144,23 @@ void replay_fetch_data_kind(void) } } } + +void replay_mutex_init (void ) +{ +qemu_mutex_init (&lock); +} + +void replay_mutex_destroy (void ) +{ +qemu_mutex_destroy (&lock); +} + +void replay_mutex_lock (void ) +{ +qemu_mutex_lock (&lock); +} + +void replay_mutex_unlock (void ) +{ +qemu_mutex_unlock (&lock); +}
replay/replay-internal.h +7 -0
@@ -34,6 +34,13 @@ int64_t replay_get_qword(void);void replay_get_array (uint8_t *buf,size_t *size);void replay_get_array_alloc (uint8_t **buf,size_t *size); +/* Mutex functions for protecting replay log file */ + +void replay_mutex_init (void ); +void replay_mutex_destroy (void ); +void replay_mutex_lock (void ); +void replay_mutex_unlock (void ); +/*! Checks error status of the file. */ void replay_check_error (void );
Re: [Qemu-devel] [RFC PATCH v8 05/21] replay: introduce mutex to protect the replay log
Quoted 30 lines · Pavel Dovgalyuk
This mutex will protect read/write operations for replay log. Using mutex is necessary because most of the events consist of several fields stored in the log. The mutex will help to avoid races.
Signed-off-by Pavel Dovgalyuk <[email protected]>--- replay/replay-internal.c | 25 +++++++++++++++++++++++++ replay/replay-internal.h | 7 +++++++ 2 files changed, 32 insertions(+), 0 deletions(-)replay/replay-internal.c +5
-0@@ -15,6 +15,11 @@unsigned int replay_data_kind =-1 ;unsigned int replay_has_unread_data; +/* Mutex to protect reading and writing events to the log. +replay_data_kind and replay_has_unread_data are also protected +by this mutex. */ +static QemuMutex lock; +/* File for replay writing */ FILE *replay_file;@@ -139,3 +144,23 @@ void replay_fetch_data_kind(void) } } }
replay/replay-internal.c void replay_fetch_data_kind(void) · line 156
Quoted 10 lines
replay/replay-internal.c +10
-0+ +void replay_mutex_init (void ) +{ +qemu_mutex_init (&lock); +} + +void replay_mutex_destroy (void ) +{ +qemu_mutex_destroy (&lock); +}
Destroy is probably not necessary.
Quoted 18 lines
replay/replay-internal.c +9
-0+void replay_mutex_lock (void ) +{ +qemu_mutex_lock (&lock); +} + +void replay_mutex_unlock (void ) +{ +qemu_mutex_unlock (&lock); +}replay/replay-internal.h +0
-0@@ -34,6 +34,13 @@ int64_t replay_get_qword(void);void replay_get_array (uint8_t *buf,size_t *size);void replay_get_array_alloc (uint8_t **buf,size_t *size);
replay/replay-internal.h int64_t replay_get_qword(void); · line 46
Quoted 10 lines
replay/replay-internal.h +7
-0+/* Mutex functions for protecting replay log file */ + +void replay_mutex_init (void ); +void replay_mutex_destroy (void ); +void replay_mutex_lock (void ); +void replay_mutex_unlock (void ); +/*! Checks error status of the file. */ void replay_check_error (void );
RFC PATCH v8 6/21 replay: introduce icount event
In reply to Pavel DovgalyukThis patch adds icount event to the replay subsystem. This event corresponds to execution of several instructions and used to synchronize input events in the replay phase.
--- replay/replay-internal.c | 16 ++++++++++++++++ replay/replay-internal.h | 20 ++++++++++++++++++++ replay/replay.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 7 +++++++ 4 files changed, 89 insertions(+), 0 deletions(-)
replay/replay-internal.c +16 -0
@@ -10,6 +10,7 @@*/ #include "qemu-common.h" +#include "replay.h" #include "replay-internal.h" unsigned int replay_data_kind =-1 ;
@@ -164,3 +165,18 @@ void replay_mutex_unlock(void) {qemu_mutex_unlock (&lock); } + +/*! Saves cached instructions. */ +void replay_save_instructions (void ) +{ +if (replay_file && replay_mode ==REPLAY_MODE_RECORD ) { +int diff = (int )(replay_get_current_step () - replay_state.current_step); +if (first_cpu !=NULL && diff >0 ) { +replay_mutex_lock (); +replay_put_event (EVENT_INSTRUCTION ); +replay_put_dword (diff); + replay_state.current_step += diff; +replay_mutex_unlock (); + } + } +}
replay/replay-internal.h +20 -0
@@ -14,6 +14,19 @@#include <stdio.h> +enum ReplayEvents { +/* for instruction event */ +EVENT_INSTRUCTION +}; + +typedef struct ReplayState { +/*! Current step - number of processed instructions and timer events. */ +uint64_t current_step; +/*! Number of instructions to be executed before other events happen. */ +int instructions_count; +}ReplayState ; +extern ReplayState replay_state; +extern unsigned int replay_data_kind;extern unsigned int replay_has_unread_data;
@@ -54,4 +67,11 @@ void replay_save_instructions(void);Terminates the program in case of error. */ void validate_data_kind (int kind); +/*! Skips async events until some sync event will be found. */ +bool skip_async_events (int stop_event); +/*! Skips async events invocations from the input, +until required data kind is found. If the requested data is not found +reports an error and stops the execution. */ +void skip_async_events_until (unsigned int kind); +#endif
replay/replay.c +46 -0
52 lines not on this page · raw message
replay/replay.h +7 -0
@@ -12,8 +12,15 @@* */ +#include <stdbool.h> +#include <stdint.h> #include "qapi-types.h" extern ReplayMode replay_mode; +/* Processing the instructions */ + +/*! Returns number of executed instructions. */ +uint64_t replay_get_current_step (void ); +#endif
Re: [Qemu-devel] [RFC PATCH v8 06/21] replay: introduce icount event
Quoted 147 lines · Pavel Dovgalyuk
This patch adds icount event to the replay subsystem. This event corresponds to execution of several instructions and used to synchronize input events in the replay phase.
Signed-off-by Pavel Dovgalyuk <[email protected]>--- replay/replay-internal.c | 16 ++++++++++++++++ replay/replay-internal.h | 20 ++++++++++++++++++++ replay/replay.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 7 +++++++ 4 files changed, 89 insertions(+), 0 deletions(-)replay/replay-internal.c +16
-0@@ -10,6 +10,7 @@*/ #include "qemu-common.h" +#include "replay.h" #include "replay-internal.h" unsigned int replay_data_kind =-1 ;@@ -164,3 +165,18 @@ void replay_mutex_unlock(void) {qemu_mutex_unlock (&lock); } + +/*! Saves cached instructions. */ +void replay_save_instructions (void ) +{ +if (replay_file && replay_mode ==REPLAY_MODE_RECORD ) { +int diff = (int )(replay_get_current_step () - replay_state.current_step); +if (first_cpu !=NULL && diff >0 ) { +replay_mutex_lock (); +replay_put_event (EVENT_INSTRUCTION ); +replay_put_dword (diff); + replay_state.current_step += diff; +replay_mutex_unlock (); + } + } +}replay/replay-internal.h +20
-0@@ -14,6 +14,19 @@#include <stdio.h> +enum ReplayEvents { +/* for instruction event */ +EVENT_INSTRUCTION +}; + +typedef struct ReplayState { +/*! Current step - number of processed instructions and timer events. */ +uint64_t current_step; +/*! Number of instructions to be executed before other events happen. */ +int instructions_count; +}ReplayState ; +extern ReplayState replay_state; +extern unsigned int replay_data_kind;extern unsigned int replay_has_unread_data;@@ -54,4 +67,11 @@ void replay_save_instructions(void);Terminates the program in case of error. */ void validate_data_kind (int kind); +/*! Skips async events until some sync event will be found. */ +bool skip_async_events (int stop_event); +/*! Skips async events invocations from the input, +until required data kind is found. If the requested data is not found +reports an error and stops the execution. */ +void skip_async_events_until (unsigned int kind); +#endif replay/replay.c +46
-0@@ -9,6 +9,52 @@* */ +#include "qemu-common.h" #include "replay.h" +#include "replay-internal.h" +#include "qemu/timer.h" ReplayMode replay_mode =REPLAY_MODE_NONE ; + +ReplayState replay_state; + +bool skip_async_events (int stop_event) +{ +bool res = false; + +/* nothing to skip - not all instructions used */ +if (replay_state.instructions_count !=0 + && replay_has_unread_data) { +return stop_event ==EVENT_INSTRUCTION ; + } + +while (true) { +replay_fetch_data_kind (); +if (stop_event == replay_data_kind) { + res = true; + } +switch (replay_data_kind) { +case EVENT_INSTRUCTION : + replay_state.instructions_count =replay_get_dword (); +return res; +default : +/* clock, time_t, checkpoint and other events */ +return res; + } + } +return res; +} + +void skip_async_events_until (unsigned int kind) +{ +if (!skip_async_events (kind)) { +fprintf (stderr,"%" PRId64": Read data kind %d instead of expected %d\n" , +replay_get_current_step (), replay_data_kind, kind); +exit (1 ); + } +} + +uint64_t replay_get_current_step (void ) +{ +return cpu_get_icount_raw (); +}replay/replay.h +2
-0@@ -12,8 +12,15 @@* */ +#include <stdbool.h> +#include <stdint.h>
replay/replay.h line 26
Quoted 10 lines
replay/replay.h +5
-0#include "qapi-types.h" extern ReplayMode replay_mode; +/* Processing the instructions */ + +/*! Returns number of executed instructions. */ +uint64_t replay_get_current_step (void ); +#endif
RFC PATCH v8 7/21 cpu-exec: allow temporary disabling icount
In reply to Pavel DovgalyukThis patch is required for deterministic replay to generate an exception by trying executing an instruction without changing icount. It adds new flag to TB for disabling icount while translating it.
--- cpu-exec.c | 7 ++++--- include/exec/exec-all.h | 1 + translate-all.c | 2 +- 3 files changed, 6 insertions(+), 4 deletions(-)
cpu-exec.c +4 -3
@@ -200,7 +200,7 @@ static inline tcg_target_ulong cpu_tb_exec(CPUState *cpu, uint8_t *tb_ptr)/* Execute the code without caching the generated code. An interpreter could be used if available. */ static void cpu_exec_nocache (CPUArchState *env,int max_cycles, -TranslationBlock *orig_tb) +TranslationBlock *orig_tb,bool ignore_icount) {CPUState *cpu =ENV_GET_CPU (env);TranslationBlock *tb;
@@ -216,7 +216,8 @@ static void cpu_exec_nocache(CPUArchState *env, int max_cycles,/* tb_gen_code can flush our orig_tb, invalidate it now */ tb_phys_invalidate (orig_tb,-1 ); tb =tb_gen_code (cpu, pc, cs_base, flags, - max_cycles |CF_NOCACHE ); + max_cycles |CF_NOCACHE + | (ignore_icount ?CF_IGNORE_ICOUNT :0 )); cpu->current_tb = tb;/* execute the generated code */ trace_exec_tb_nocache (tb, tb->pc);
@@ -525,7 +526,7 @@ int cpu_exec(CPUArchState *env) }else {if (insns_left >0 ) {/* Execute remaining instructions. */ -cpu_exec_nocache (env, insns_left, tb); +cpu_exec_nocache (env, insns_left, tb, false);align_clocks (&sc, cpu); } cpu->exception_index =EXCP_INTERRUPT ;
include/exec/exec-all.h +1 -0
@@ -148,6 +148,7 @@ struct TranslationBlock {#define CF_LAST_IO 0x8000/* Last insn may be an IO access. */ #define CF_NOCACHE 0x10000/* To be freed after execution */ #define CF_USE_ICOUNT 0x20000 +#define CF_IGNORE_ICOUNT 0x40000/* Do not generate icount code */ void *tc_ptr;/* pointer to the translated code */ /* next matching tb for physical address. */
translate-all.c +1 -1
@@ -1053,7 +1053,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu,int code_gen_size; phys_pc =get_page_addr_code (env, pc); -if (use_icount) { +if (use_icount && !(cflags &CF_IGNORE_ICOUNT )) { cflags |=CF_USE_ICOUNT ; } tb =tb_alloc (pc);
RFC PATCH v8 8/21 cpu: replay instructions sequence
In reply to Pavel DovgalyukThis patch adds calls to replay functions into the icount setup block. In record mode number of executed instructions is written to the log. In replay mode number of istructions to execute is taken from the replay log.
--- cpu-exec.c | 1 + cpus.c | 28 ++++++++++++++++++---------- replay/replay.c | 24 ++++++++++++++++++++++++ replay/replay.h | 4 ++++ 4 files changed, 47 insertions(+), 10 deletions(-)
cpu-exec.c +1 -0
@@ -531,6 +531,7 @@ int cpu_exec(CPUArchState *env) } cpu->exception_index =EXCP_INTERRUPT ; next_tb =0 ; +qemu_notify_event ();cpu_loop_exit (cpu); }break ;
cpus.c +18 -10
@@ -41,6 +41,7 @@#include "qemu/seqlock.h" #include "qapi-event.h" #include "hw/nmi.h" +#include "replay/replay.h" #ifndef _WIN32#include "qemu/compatfd.h"
@@ -1342,18 +1343,22 @@ static int tcg_cpu_exec(CPUArchState *env) + cpu->icount_extra); cpu->icount_decr.u16.low =0 ; cpu->icount_extra =0 ; - deadline =qemu_clock_deadline_ns_all (QEMU_CLOCK_VIRTUAL ); +if (replay_mode !=REPLAY_MODE_PLAY ) { + deadline =qemu_clock_deadline_ns_all (QEMU_CLOCK_VIRTUAL ); -/* Maintain prior (possibly buggy) behaviour where if no deadline -* was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than -* INT32_MAX nanoseconds ahead, we still use INT32_MAX -* nanoseconds. -*/ -if ((deadline <0 ) || (deadline >INT32_MAX )) { - deadline =INT32_MAX ; - } +/* Maintain prior (possibly buggy) behaviour where if no deadline +* was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than +* INT32_MAX nanoseconds ahead, we still use INT32_MAX +* nanoseconds. +*/ +if ((deadline <0 ) || (deadline >INT32_MAX )) { + deadline =INT32_MAX ; + } - count =qemu_icount_round (deadline); + count =qemu_icount_round (deadline); + }else { + count =replay_get_instructions (); + } timers_state.qemu_icount += count; decr = (count >0xffff ) ?0xffff : count; count -= decr;
@@ -1371,6 +1376,9 @@ static int tcg_cpu_exec(CPUArchState *env) + cpu->icount_extra); cpu->icount_decr.u32 =0 ; cpu->icount_extra =0 ; +if (replay_mode ==REPLAY_MODE_PLAY ) { +replay_exec_instructions (); + } }return ret; }
replay/replay.c +24 -0
@@ -58,3 +58,27 @@ uint64_t replay_get_current_step(void) {return cpu_get_icount_raw (); } + +int replay_get_instructions (void ) +{ +int res =0 ; +replay_mutex_lock (); +if (skip_async_events (EVENT_INSTRUCTION )) { + res = replay_state.instructions_count; + } +replay_mutex_unlock (); +return res; +} + +void replay_exec_instructions (void ) +{ +if (replay_state.instructions_count >0 ) { +int count = (int )(replay_get_current_step () + - replay_state.current_step); + replay_state.instructions_count -= count; + replay_state.current_step += count; +if (replay_state.instructions_count ==0 && count !=0 ) { + replay_has_unread_data =0 ; + } + } +}
replay/replay.h +4 -0
@@ -22,5 +22,9 @@ extern ReplayMode replay_mode;/*! Returns number of executed instructions. */ uint64_t replay_get_current_step (void ); +/*! Returns number of instructions to execute in replay mode. */ +int replay_get_instructions (void ); +/*! Updates instructions counter in replay mode. */ +void replay_exec_instructions (void );#endif
Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence
Quoted 17 lines · Pavel Dovgalyuk
This patch adds calls to replay functions into the icount setup block. In record mode number of executed instructions is written to the log. In replay mode number of istructions to execute is taken from the replay log.
Signed-off-by Pavel Dovgalyuk <[email protected]>--- cpu-exec.c | 1 + cpus.c | 28 ++++++++++++++++++---------- replay/replay.c | 24 ++++++++++++++++++++++++ replay/replay.h | 4 ++++ 4 files changed, 47 insertions(+), 10 deletions(-)cpu-exec.c +0
-0
cpu-exec.c int cpu_exec(CPUArchState *env) · line 534
Quoted 4 lines
cpu-exec.c +1
-0@@ -531,6 +531,7 @@ int cpu_exec(CPUArchState *env) } cpu->exception_index =EXCP_INTERRUPT ; next_tb =0 ; +qemu_notify_event ();
Why is this needed?
Quoted 35 lines
cpu-exec.c +0
-0cpu_loop_exit (cpu); }break ;cpus.c +7
-9@@ -41,6 +41,7 @@#include "qemu/seqlock.h" #include "qapi-event.h" #include "hw/nmi.h" +#include "replay/replay.h" #ifndef _WIN32#include "qemu/compatfd.h" @@ -1342,18 +1343,22 @@ static int tcg_cpu_exec(CPUArchState *env) + cpu->icount_extra); cpu->icount_decr.u16.low =0 ; cpu->icount_extra =0 ; - deadline =qemu_clock_deadline_ns_all (QEMU_CLOCK_VIRTUAL ); +if (replay_mode !=REPLAY_MODE_PLAY ) { + deadline =qemu_clock_deadline_ns_all (QEMU_CLOCK_VIRTUAL ); -/* Maintain prior (possibly buggy) behaviour where if no deadline -* was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than -* INT32_MAX nanoseconds ahead, we still use INT32_MAX -* nanoseconds. -*/ -if ((deadline <0 ) || (deadline >INT32_MAX )) { - deadline =INT32_MAX ; - } +/* Maintain prior (possibly buggy) behaviour where if no deadline +* was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than +* INT32_MAX nanoseconds ahead, we still use INT32_MAX +* nanoseconds.
cpus.c static int tcg_cpu_exec(CPUArchState *env) · line 1361
Quoted 10 lines
cpus.c +8
-1+*/ +if ((deadline <0 ) || (deadline >INT32_MAX )) { + deadline =INT32_MAX ; + } - count =qemu_icount_round (deadline); + count =qemu_icount_round (deadline); + }else { + count =replay_get_instructions (); + }
Please extract the "if" to a separate function tcg_get_icount_limit().
cpus.c static int tcg_cpu_exec(CPUArchState *env) · line 1380
Quoted 9 lines
cpus.c +2
-0timers_state.qemu_icount += count; decr = (count >0xffff ) ?0xffff : count; count -= decr;@@ -1371,6 +1376,9 @@ static int tcg_cpu_exec(CPUArchState *env) + cpu->icount_extra); cpu->icount_decr.u32 =0 ; cpu->icount_extra =0 ; +if (replay_mode ==REPLAY_MODE_PLAY ) { +replay_exec_instructions ();
replay_account_executed_instructions()
Quoted 22 lines
cpus.c +1
-0+ } }return ret; }replay/replay.c +10
-0@@ -58,3 +58,27 @@ uint64_t replay_get_current_step(void) {return cpu_get_icount_raw (); } + +int replay_get_instructions (void ) +{ +int res =0 ; +replay_mutex_lock (); +if (skip_async_events (EVENT_INSTRUCTION )) { + res = replay_state.instructions_count; + } +replay_mutex_unlock (); +return res;
replay/replay.c uint64_t replay_get_current_step(void) · line 80
Quoted 10 lines
replay/replay.c +10
-0+} + +void replay_exec_instructions (void ) +{ +if (replay_state.instructions_count >0 ) { +int count = (int )(replay_get_current_step () + - replay_state.current_step); + replay_state.instructions_count -= count; + replay_state.current_step += count; +if (replay_state.instructions_count ==0 && count !=0 ) {
If replay_state.instructions_count is now zero, count must be nonzero (because replay_state.instructions_count was > 0) before.
replay/replay.c uint64_t replay_get_current_step(void) · line 82
Quoted 2 lines
replay/replay.c +2
-0+ replay_has_unread_data =0 ; + }
Can replay_state.instructions_count be < count at all? If not, and if replay_state.instructions_count is zero, then count must also be zero.
If so, I suggest rewriting as
int count = (int)(replay_get_current_step()
- replay_state.current_step);
assert(replay_state.instructions_count >= count);
replay_state.instructions_count -= count;
replay_state.current_step += count;
if (replay_state.instructions_count == 0) {
replay_has_unread_data = 0;
}Paolo
Quoted 17 lines
replay/replay.c +2
-0+ } +}replay/replay.h +4
-0@@ -22,5 +22,9 @@ extern ReplayMode replay_mode;/*! Returns number of executed instructions. */ uint64_t replay_get_current_step (void ); +/*! Returns number of instructions to execute in replay mode. */ +int replay_get_instructions (void ); +/*! Updates instructions counter in replay mode. */ +void replay_exec_instructions (void );#endif
Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence
Quoted 14 lines
From: Paolo Bonzini [mailto:[email protected]]
Quoted 12 lines · Pavel Dovgalyuk
This patch adds calls to replay functions into the icount setup block. In record mode number of executed instructions is written to the log. In replay mode number of istructions to execute is taken from the replay log.
Signed-off-by Pavel Dovgalyuk <[email protected]>--- cpu-exec.c | 1 + cpus.c | 28 ++++++++++++++++++---------- replay/replay.c | 24 ++++++++++++++++++++++++ replay/replay.h | 4 ++++ 4 files changed, 47 insertions(+), 10 deletions(-)
Quoted 11 lines
cpu-exec.c int cpu_exec(CPUArchState *env) · line 534
Quoted 9 lines
cpu-exec.c +1
-0@@ -531,6 +531,7 @@ int cpu_exec(CPUArchState *env) } cpu->exception_index =EXCP_INTERRUPT ; next_tb =0 ; +qemu_notify_event ();Why is this needed?
It is needed to wake up iothread in replay mode. Otherwise it waits for additional time and replay becomes too slow.
Quoted 4 lines
cpu-exec.c int cpu_exec(CPUArchState *env) · line 537
Quoted 4 lines
cpu-exec.c +0
-0cpu_loop_exit (cpu); }break ;
Pavel Dovgalyuk
Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence
Quoted 7 lines · Pavel Dovgaluk
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
cpu->exception_index = EXCP_INTERRUPT; next_tb = 0;+
qemu_notify_event();Why is this needed?
It is needed to wake up iothread in replay mode. Otherwise it waits for additional time and replay becomes too slow.
What event (something from a timerlist?) is ready, that has not been notified to the iothread yet? qemu_notify_event() should never be needed in common case. It's probably missing somewhere else.
Paolo
Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence
Quoted 6 lines
From: Paolo Bonzini [mailto:[email protected]]
Quoted 4 lines · Pavel Dovgaluk
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
cpu->exception_index = EXCP_INTERRUPT; next_tb = 0;+
qemu_notify_event();
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Why is this needed?
It is needed to wake up iothread in replay mode. Otherwise it waits for additional time and replay becomes too slow.
What event (something from a timerlist?) is ready, that has not been notified to the iothread yet?
qemu_notify_event()should never be needed in common case. It's probably missing somewhere else.
I think in this case there are no events at all - just reading timers values that were made while recording.
We have to replay these reads by waking iothread.
Pavel Dovgalyuk
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence
Quoted 10 lines · Pavel Dovgaluk
Quoted 10 lines
From: Paolo Bonzini [mailto:[email protected]]
Quoted 7 lines · Pavel Dovgaluk
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
cpu->exception_index = EXCP_INTERRUPT; next_tb = 0;+
qemu_notify_event();Why is this needed?
It is needed to wake up iothread in replay mode. Otherwise it waits for additional time and replay becomes too slow.
Quoted 7 lines
Quoted 3 lines
What event (something from a timerlist?) is ready, that has not been notified to the iothread yet?
qemu_notify_event()should never be needed in common case. It's probably missing somewhere else.I think in this case there are no events at all - just reading timers values that were made while recording.
We have to replay these reads by waking iothread.
I think the right place for this is in replay_read_next_clock then.
Paolo
Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence
Quoted 16 lines
From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini
Quoted 14 lines · Pavel Dovgaluk
Quoted 13 lines
From: Paolo Bonzini [mailto:[email protected]]
Quoted 7 lines · Pavel Dovgaluk
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
cpu->exception_index = EXCP_INTERRUPT; next_tb = 0;+
qemu_notify_event();Why is this needed?
It is needed to wake up iothread in replay mode. Otherwise it waits for additional time and replay becomes too slow.
What event (something from a timerlist?) is ready, that has not been notified to the iothread yet?
qemu_notify_event()should never be needed in common case. It's probably missing somewhere else.
Quoted 5 lines
Quoted 3 lines
I think in this case there are no events at all - just reading timers values that were made while recording.
We have to replay these reads by waking iothread.I think the right place for this is in
replay_read_next_clockthen.
It doesn't fit. Log file is not read until all instructions are executed. And the next read from the file should be performed by iothread which should be notified and waked up.
Pavel Dovgalyuk
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence
Quoted 8 lines · Pavel Dovgaluk
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
I think in this case there are no events at all - just reading timers values that were made while recording.
We have to replay these reads by waking iothread.I think the right place for this is in
replay_read_next_clockthen.It doesn't fit. Log file is not read until all instructions are executed. And the next read from the file should be performed by iothread which should be notified and waked up.
I still don't understand. If you're getting EXCP_INTERRUPT it means:
1) that cpu_signal was called
2) in turn this means that qemu_cpu_kick was called
3) in turn this means that the iothread is trying to run via qemu_mutex_lock_iothread (the iothread_requesting_mutex stuff). So why do you need an explicit qemu_notify_event?
Paolo
Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence
Quoted 6 lines
From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini
Quoted 4 lines · Pavel Dovgaluk
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
I think in this case there are no events at all - just reading timers values that were made while recording.
We have to replay these reads by waking iothread.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
I think the right place for this is in
replay_read_next_clockthen.It doesn't fit. Log file is not read until all instructions are executed. And the next read from the file should be performed by iothread which should be notified and waked up.
I still don't understand. If you're getting
EXCP_INTERRUPTit means:1) that
cpu_signalwas called
No, it isn't. That is the branch when icount is expired. And when it is expired in replay mode we have to wake up iothread, because nobody will care about this.
Quoted 6 lines
2) in turn this means that
qemu_cpu_kickwas called3) in turn this means that the iothread is trying to run via
qemu_mutex_lock_iothread(theiothread_requesting_mutexstuff). So why do you need an explicitqemu_notify_event?
Pavel Dovgalyuk
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence
Quoted 11 lines · Pavel Dovgaluk
Quoted 11 lines
From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini
Quoted 8 lines · Pavel Dovgaluk
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
I think in this case there are no events at all - just reading timers values that were made while recording.
We have to replay these reads by waking iothread.I think the right place for this is in
replay_read_next_clockthen.It doesn't fit. Log file is not read until all instructions are executed. And the next read from the file should be performed by iothread which should be notified and waked up.
Quoted 7 lines
Quoted 3 lines
I still don't understand. If you're getting
EXCP_INTERRUPTit means:1) that
cpu_signalwas calledNo, it isn't. That is the branch when icount is expired. And when it is expired in replay mode we have to wake up iothread, because nobody will care about this.
Then it's done here in qemu_tcg_cpu_thread_fn:
if (use_icount) {
int64_t deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
if (deadline == 0) {
qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
}
}If you need to move the 4 lines inside the if elsewhere, that I guess it's okay.
Paolo
Quoted 11 lines
Quoted 6 lines
2) in turn this means that
qemu_cpu_kickwas called3) in turn this means that the iothread is trying to run via
qemu_mutex_lock_iothread(theiothread_requesting_mutexstuff). So why do you need an explicitqemu_notify_event?Pavel Dovgalyuk
Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence
Quoted 15 lines
From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini
Quoted 13 lines · Pavel Dovgaluk
Quoted 13 lines
From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini
Quoted 8 lines · Pavel Dovgaluk
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
I think in this case there are no events at all - just reading timers values that were made while recording.
We have to replay these reads by waking iothread.I think the right place for this is in
replay_read_next_clockthen.It doesn't fit. Log file is not read until all instructions are executed. And the next read from the file should be performed by iothread which should be notified and waked up.
I still don't understand. If you're getting
EXCP_INTERRUPTit means:
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
1) that
cpu_signalwas calledNo, it isn't. That is the branch when icount is expired. And when it is expired in replay mode we have to wake up iothread, because nobody will care about this.
Then it's done here in
qemu_tcg_cpu_thread_fn:
Do you mean that I should put iothread notification right here?
Or that this code duplicates my patch?
If the second one then I guess that it doesn't help and I need to make additional checks about it.
Quoted 10 lines
if (use_icount) { int64_t deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL); if (deadline == 0) { qemu_clock_notify(QEMU_CLOCK_VIRTUAL); } }If you need to move the 4 lines inside the if elsewhere, that I guess it's okay.
Pavel Dovgalyuk
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence
Quoted 17 lines · Pavel Dovgaluk
Quoted 17 lines
From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini
Quoted 15 lines · Pavel Dovgaluk
Quoted 14 lines
From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini
Quoted 8 lines · Pavel Dovgaluk
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
I think in this case there are no events at all - just reading timers values that were made while recording.
We have to replay these reads by waking iothread.I think the right place for this is in
replay_read_next_clockthen.It doesn't fit. Log file is not read until all instructions are executed. And the next read from the file should be performed by iothread which should be notified and waked up.
I still don't understand. If you're getting
EXCP_INTERRUPTit means:1) that
cpu_signalwas called
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
No, it isn't. That is the branch when icount is expired. And when it is expired in replay mode we have to wake up iothread, because nobody will care about this.
Then it's done here in
qemu_tcg_cpu_thread_fn:Do you mean that I should put iothread notification right here?
It already notifies the iothread.
Quoted 2 lines
Or that this code duplicates my patch?
If the second one then I guess that it doesn't help and I need to make additional checks about it.
Yes. You can modify your patch to do:
int64_t deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
if (deadline == 0) {
qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
}instead of qemu_notify_event(), and remove these lines from qemu_tcg_cpu_thread_fn.
Paolo
Quoted 15 lines
Quoted 10 lines
if (use_icount) { int64_t deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL); if (deadline == 0) { qemu_clock_notify(QEMU_CLOCK_VIRTUAL); } }If you need to move the 4 lines inside the if elsewhere, that I guess it's okay.
Pavel Dovgalyuk
Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence
Quoted 35 lines
From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini
Quoted 17 lines · Pavel Dovgaluk
Quoted 17 lines
From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini
Quoted 15 lines · Pavel Dovgaluk
Quoted 14 lines
From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini
Quoted 8 lines · Pavel Dovgaluk
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
I think in this case there are no events at all - just reading timers values that were made while recording.
We have to replay these reads by waking iothread.I think the right place for this is in
replay_read_next_clockthen.It doesn't fit. Log file is not read until all instructions are executed. And the next read from the file should be performed by iothread which should be notified and waked up.
I still don't understand. If you're getting
EXCP_INTERRUPTit means:1) that
cpu_signalwas calledQuoted 7 lines
Quoted 5 lines
Quoted 3 lines
No, it isn't. That is the branch when icount is expired. And when it is expired in replay mode we have to wake up iothread, because nobody will care about this.
Then it's done here in
qemu_tcg_cpu_thread_fn:Do you mean that I should put iothread notification right here?
It already notifies the iothread.
Quoted 2 lines
Or that this code duplicates my patch?
If the second one then I guess that it doesn't help and I need to make additional checksabout it.
Yes. You can modify your patch to do:
Quoted 7 lines
int64_t deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL); if (deadline == 0) { qemu_clock_notify(QEMU_CLOCK_VIRTUAL); }instead of
qemu_notify_event(), and remove these lines fromqemu_tcg_cpu_thread_fn.
I tried this one. But there is one problem.
Expiring of the virtual timers is not the only reason of icount expiration in replay mode.
It may be caused by host timers deadline or poll timeout in record mode. In this case qemu_clock_notify(QEMU_CLOCK_VIRTUAL) will not be called in replay mode and we'll waste time for iothread sleeping.
Pavel Dovgalyuk
Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence
Quoted 11 lines · Pavel Dovgaluk
Quoted 7 lines
Quoted 7 lines
int64_t deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL); if (deadline == 0) { qemu_clock_notify(QEMU_CLOCK_VIRTUAL); }instead of
qemu_notify_event(), and remove these lines fromqemu_tcg_cpu_thread_fn.I tried this one. But there is one problem.
Expiring of the virtual timers is not the only reason of icount expiration in replay mode.
It may be caused by host timers deadline or poll timeout in record mode. In this caseqemu_clock_notify(QEMU_CLOCK_VIRTUAL)will not be called in replay mode and we'll waste time for iothread sleeping.
Sure, but unconditional qemu_notify_event() is also wrong. So, to sum up:
- it's okay to move code from qemu_tcg_cpu_thread_fn to cpu-exec.c
- it's okay to add more qemu_clock_notify calls than just qemu_clock_notify(QEMU_CLOCK_VIRTUAL), each with its own condition
- it's better if all these, after being moved to cpu-exec.c, are also extracted in a separate function
- it's not okay to do an unconditional qemu_notify_event() in cpu-exec.c, even if it's under "if (replay_mode != NONE)".
Thanks for your understanding! :)
Paolo
Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence
Quoted 19 lines
From: Paolo Bonzini [mailto:[email protected]]
Quoted 11 lines · Pavel Dovgaluk
Quoted 7 lines
Quoted 7 lines
int64_t deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL); if (deadline == 0) { qemu_clock_notify(QEMU_CLOCK_VIRTUAL); }instead of
qemu_notify_event(), and remove these lines fromqemu_tcg_cpu_thread_fn.I tried this one. But there is one problem.
Expiring of the virtual timers is not the only reason of icount expiration in replay mode. It may be caused by host timers deadline or poll timeout in record mode. In this caseqemu_clock_notify(QEMU_CLOCK_VIRTUAL)will not be called in replay mode and we'll waste timefor iothread sleeping.
Sure, but unconditional
qemu_notify_event()is also wrong. So, to sum up:- it's okay to move code from
qemu_tcg_cpu_thread_fntocpu-exec.c
Quoted 8 lines
- it's okay to add more
qemu_clock_notifycalls than justqemu_clock_notify(QEMU_CLOCK_VIRTUAL), each with its own condition- it's better if all these, after being moved to
cpu-exec.c, are also extracted in a separate function- it's not okay to do an unconditional
qemu_notify_event()incpu-exec.c, even if it's under "if (replay_mode!= NONE)".
How can I wake up iothread if there are no pending timers? deadline will (almost) never become zero in my case, because there is another kind of event in the log (not the timer one).
Should I fetch the event and call qemu_notify_event() from replay module?
Quoted 2 lines
Thanks for your understanding! :)
Pavel Dovgalyuk
Re: [Qemu-devel] [RFC PATCH v8 08/21] cpu: replay instructions sequence
Quoted 4 lines · Pavel Dovgaluk
How can I wake up iothread if there are no pending timers? deadline will (almost) never become zero in my case, because there is another kind of event in the log (not the timer one).
Should I fetch the event and callqemu_notify_event()from replay module?
Yes, a few mails ago I suggested that (maybe I got the wrong function in the replay module).
Paolo
RFC PATCH v8 9/21 replay: interrupts and exceptions
In reply to Pavel DovgalyukThis patch includes modifications of common cpu files. All interrupts and exceptions occured during recording are written into the replay log. These events allow correct replaying the execution by kicking cpu thread when one of these events is found in the log.
--- cpu-exec.c | 48 +++++++++++++++++++++++++------- replay/replay-internal.h | 6 +++- replay/replay.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 17 +++++++++++ target-i386/seg_helper.c | 4 +++ 5 files changed, 132 insertions(+), 12 deletions(-)
cpu-exec.c +37 -11
@@ -24,6 +24,7 @@#include "qemu/atomic.h" #include "sysemu/qtest.h" #include "qemu/timer.h" +#include "replay/replay.h" /* -icount align implementation. */
@@ -338,22 +339,25 @@ int cpu_exec(CPUArchState *env)/* This must be volatile so it is not trashed by longjmp() */ volatile bool have_tb_lock = false; +/* replay_interrupt may need current_cpu */ + current_cpu = cpu; +if (cpu->halted) {#ifdef TARGET_I386 -if (cpu->interrupt_request &CPU_INTERRUPT_POLL ) { +if ((cpu->interrupt_request &CPU_INTERRUPT_POLL ) + &&replay_interrupt ()) {apic_poll_irq (x86_cpu->apic_state);cpu_reset_interrupt (cpu,CPU_INTERRUPT_POLL ); }#endif if (!cpu_has_work (cpu)) { + current_cpu =NULL ;return EXCP_HALTED ; } cpu->halted =0 ; } - current_cpu = cpu; -/* As long as current_cpu is null, up to the assignment just above, * requests by other threads to exit the execution loop are expected to * be issued using the exit_request global. We must make sure that our
@@ -400,10 +404,21 @@ int cpu_exec(CPUArchState *env) cpu->exception_index =-1 ;break ;#else - cc->do_interrupt (cpu); - cpu->exception_index =-1 ; +if (replay_exception ()) { + cc->do_interrupt (cpu); + cpu->exception_index =-1 ; + }else if (!replay_has_interrupt ()) { +/* give a chance to iothread in replay mode */ + ret =EXCP_INTERRUPT ; +break ; + }#endif } + }else if (replay_has_exception () + && cpu->icount_decr.u16.low + cpu->icount_extra ==0 ) { +/* try to cause an exception pending in the log */ +cpu_exec_nocache (env,1 ,tb_find_fast (env), true); +break ; } next_tb =0 ;/* force lookup of first TB */
@@ -419,30 +434,40 @@ int cpu_exec(CPUArchState *env) cpu->exception_index =EXCP_DEBUG ;cpu_loop_exit (cpu); } -if (interrupt_request &CPU_INTERRUPT_HALT ) { +if (replay_mode ==REPLAY_MODE_PLAY + && !replay_has_interrupt ()) { +/* Do nothing */ + }else if (interrupt_request &CPU_INTERRUPT_HALT ) { +replay_interrupt (); cpu->interrupt_request &= ~CPU_INTERRUPT_HALT ; cpu->halted =1 ; cpu->exception_index =EXCP_HLT ;cpu_loop_exit (cpu); }#if defined(TARGET_I386 ) -if (interrupt_request &CPU_INTERRUPT_INIT ) { + else if (interrupt_request & CPU_INTERRUPT_INIT) { +replay_interrupt ();cpu_svm_check_intercept_param (env,SVM_EXIT_INIT ,0 );do_cpu_init (x86_cpu); cpu->exception_index =EXCP_HALTED ;cpu_loop_exit (cpu); }#else -if (interrupt_request &CPU_INTERRUPT_RESET ) { + else if (interrupt_request & CPU_INTERRUPT_RESET) { +replay_interrupt ();cpu_reset (cpu); +cpu_loop_exit (cpu); }#endif /* The target hook has 3 exit conditions: False when the interrupt isn't processed, True when it is, and we should restart on a new TB, and via longjmp via cpu_loop_exit. */ -if (cc->cpu_exec_interrupt (cpu, interrupt_request)) { - next_tb =0 ; + else { +replay_interrupt (); +if (cc->cpu_exec_interrupt (cpu, interrupt_request)) { + next_tb =0 ; + } }/* Don't use the cached interrupt_request value, do_interrupt may have updated the EXITTB flag. */
@@ -453,7 +478,8 @@ int cpu_exec(CPUArchState *env) next_tb =0 ; } } -if (unlikely (cpu->exit_request)) { +if (unlikely (cpu->exit_request + ||replay_has_interrupt ())) { cpu->exit_request =0 ; cpu->exception_index =EXCP_INTERRUPT ;cpu_loop_exit (cpu);
replay/replay-internal.h +5 -1
@@ -16,7 +16,11 @@enum ReplayEvents {/* for instruction event */ -EVENT_INSTRUCTION +EVENT_INSTRUCTION , +/* for software interrupt */ +EVENT_INTERRUPT , +/* for emulated exceptions */ +EVENT_EXCEPTION }; typedef struct ReplayState {
replay/replay.c +69 -0
72 lines not on this page · raw message
replay/replay.h +17 -0
@@ -27,4 +27,21 @@ int replay_get_instructions(void);/*! Updates instructions counter in replay mode. */ void replay_exec_instructions (void ); +/* Interrupts and exceptions */ + +/*! Called by exception handler to write or read +exception processing events. */ +bool replay_exception (void ); +/*! Used to determine that exception is pending. +Does not proceed to the next event in the log. */ +bool replay_has_exception (void ); +/*! Called by interrupt handlers to write or read +interrupt processing events. +\return true if interrupt should be processed */ +bool replay_interrupt (void ); +/*! Tries to read interrupt event from the file. +Returns true, when interrupt request is pending */ +bool replay_has_interrupt (void ); + +#endif
target-i386/seg_helper.c +4 -0
@@ -22,6 +22,7 @@#include "qemu/log.h" #include "exec/helper-proto.h" #include "exec/cpu_ldst.h" +#include "replay/replay.h" //#define DEBUG_PCALL
@@ -1294,6 +1295,9 @@ bool x86_cpu_exec_interrupt(CPUState *cs, int interrupt_request)if (interrupt_request &CPU_INTERRUPT_POLL ) { cs->interrupt_request &= ~CPU_INTERRUPT_POLL ;apic_poll_irq (cpu->apic_state); +if (replay_mode !=REPLAY_MODE_NONE ) { +return true; + } }#endif if (interrupt_request &CPU_INTERRUPT_SIPI ) {
Re: [Qemu-devel] [RFC PATCH v8 09/21] replay: interrupts and exceptions
replay/replay.c void replay_exec_instructions(void) · line 91
Quoted 4 lines · Pavel Dovgalyuk
replay/replay.c +4
-0+if (replay_mode ==REPLAY_MODE_RECORD ) { +replay_save_instructions (); +replay_put_event (EVENT_EXCEPTION ); +return true;
Missing mutex lock/unlock.
replay/replay.c void replay_exec_instructions(void) · line 102
Quoted 11 lines
replay/replay.c +11
-0+ }else if (replay_mode ==REPLAY_MODE_PLAY ) { +bool res = false; +replay_exec_instructions (); +replay_mutex_lock (); +if (skip_async_events (EVENT_EXCEPTION )) { + replay_has_unread_data =0 ; + res = true; + } +replay_mutex_unlock (); +return res; + }
bool res;replay_exec_instructions();
res = replay_has_exception();
if (res) {
replay_has_unread_data = 0;
}return res;
Same for replay_interrupt().
Perhaps worth factoring out two functions replay_cpu_event and replay_has_cpu_event? You choose.
target-i386/seg_helper.c bool x86_cpu_exec_interrupt(CPUState *cs, int interrupt_request) · line 1302
Quoted 10 lines
target-i386/seg_helper.c +3
-0@@ -1294,6 +1295,9 @@ bool x86_cpu_exec_interrupt(CPUState *cs, int interrupt_request)if (interrupt_request &CPU_INTERRUPT_POLL ) { cs->interrupt_request &= ~CPU_INTERRUPT_POLL ;apic_poll_irq (cpu->apic_state); +if (replay_mode !=REPLAY_MODE_NONE ) { +return true; + } }#endif
Can you explain this? It probably needs a comment.
Paolo
Re: [Qemu-devel] [RFC PATCH v8 09/21] replay: interrupts and exceptions
Quoted 8 lines
From: Paolo Bonzini [mailto:[email protected]]
replay/replay.c void replay_exec_instructions(void) · line 91
Quoted 4 lines · Pavel Dovgalyuk
replay/replay.c +4
-0+if (replay_mode ==REPLAY_MODE_RECORD ) { +replay_save_instructions (); +replay_put_event (EVENT_EXCEPTION ); +return true;Missing mutex lock/unlock.
I think not. We just have to write number of already executed instructions. This number is not linked to exception event. They could be read in replay while processing some other event.
Quoted 11 lines
target-i386/seg_helper.c bool x86_cpu_exec_interrupt(CPUState *cs, int interrupt_request) · line 1302
Quoted 9 lines
target-i386/seg_helper.c +3
-0@@ -1294,6 +1295,9 @@ bool x86_cpu_exec_interrupt(CPUState *cs, int interrupt_request)if (interrupt_request &CPU_INTERRUPT_POLL ) { cs->interrupt_request &= ~CPU_INTERRUPT_POLL ;apic_poll_irq (cpu->apic_state); +if (replay_mode !=REPLAY_MODE_NONE ) { +return true; + } }#endif Can you explain this? It probably needs a comment.
Each function call should process one interrupt at once, because it corresponds to single event in the log.
Pavel Dovgalyuk
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [Qemu-devel] [RFC PATCH v8 09/21] replay: interrupts and exceptions
Quoted 4 lines · Pavel Dovgaluk
I think not. We just have to write number of already executed instructions. This number is not linked to exception event. They could be read in replay while processing some other event.
I was referring to replay_put_event(EVENT_EXCEPTION) only.
In principle you could run EVENT_EXCEPTION concurrently with other event writes, for example:
+ replay_mutex_lock(); + replay_put_event(EVENT_CLOCK + kind); + replay_put_qword(clock); + replay_mutex_unlock();
and get
EVENT_CLOCK + kind (1 byte)
EVENT_EXCEPTION (1 byte)
clock (8 bytes)
in the replay stream.
I know it's really unlikely, perhaps even impossible in the current QEMU architecture that is dominated by the big QEMU lock. But the right thing to do is to lock the mutex around all event writes. Even if the write itself is atomic, it could happen in the middle of another write if you do not lock the mutex.
Paolo
RFC PATCH v8 10/21 replay: asynchronous events infrastructure
In reply to Pavel DovgalyukThis patch adds module for saving and replaying asynchronous events. These events include network packets, keyboard and mouse input, USB packets, thread pool and bottom halves callbacks.
All events are stored in the queue to be processed at synchronization points such as beginning of TB execution, or checkpoint in the iothread.
--- replay/Makefile.objs | 1 replay/replay-events.c | 228 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay-internal.h | 33 ++++++- replay/replay.h | 4 + 4 files changed, 265 insertions(+), 1 deletions(-) create mode 100755 replay/replay-events.c
replay/Makefile.objs +1 -0
@@ -1,2 +1,3 @@ obj-y += replay.o obj-y += replay-internal.o +obj-y += replay-events.o
replay/replay-events.c new file +228 -0
228 lines not on this page · raw message
replay/replay-internal.h +32 -1
@@ -20,9 +20,19 @@ enum ReplayEvents {/* for software interrupt */ EVENT_INTERRUPT ,/* for emulated exceptions */ -EVENT_EXCEPTION +EVENT_EXCEPTION , +/* for async events */ +EVENT_ASYNC }; +/* Asynchronous events IDs */ + +enum ReplayAsyncEventKind { +REPLAY_ASYNC_COUNT +}; + +typedef enum ReplayAsyncEventKind ReplayAsyncEventKind ; + typedef struct ReplayState { /*! Current step - number of processed instructions and timer events. */ uint64_t current_step;
@@ -78,4 +88,25 @@ bool skip_async_events(int stop_event);reports an error and stops the execution. */ void skip_async_events_until (unsigned int kind); +/* Asynchronous events queue */ + +/*! Initializes events' processing internals */ +void replay_init_events (void ); +/*! Clears internal data structures for events handling */ +void replay_finish_events (void ); +/*! Enables storing events in the queue */ +void replay_enable_events (void ); +/*! Flushes events queue */ +void replay_flush_events (void ); +/*! Clears events list before loading new VM state */ +void replay_clear_events (void ); +/*! Returns true if there are any unsaved events in the queue */ +bool replay_has_events (void ); +/*! Saves events from queue into the file */ +void replay_save_events (int opt); +/*! Read events from the file into the input queue */ +void replay_read_events (int opt); +/*! Adds specified async event to the queue */ +void replay_add_event (ReplayAsyncEventKind event_id,void *opaque); +#endif
replay/replay.h +4 -0
@@ -43,5 +43,9 @@ bool replay_interrupt(void);Returns true, when interrupt request is pending */ bool replay_has_interrupt (void ); +/* Asynchronous events queue */ + +/*! Disables storing events in the queue */ +void replay_disable_events (void );#endif
Re: [Qemu-devel] [RFC PATCH v8 10/21] replay: asynchronous events infrastructure
Quoted 46 lines · Pavel Dovgalyuk
This patch adds module for saving and replaying asynchronous events. These events include network packets, keyboard and mouse input, USB packets, thread pool and bottom halves callbacks.
All events are stored in the queue to be processed at synchronization points such as beginning of TB execution, or checkpoint in the iothread.Signed-off-by Pavel Dovgalyuk <[email protected]>--- replay/Makefile.objs | 1 replay/replay-events.c | 228 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay-internal.h | 33 ++++++- replay/replay.h | 4 + 4 files changed, 265 insertions(+), 1 deletions(-) create mode 100755 replay/replay-events.creplay/Makefile.objs +1
-0@@ -1,2 +1,3 @@ obj-y += replay.o obj-y += replay-internal.o +obj-y += replay-events.oreplay/replay-events.c new file +17
-0@@ -0,0 +1,228 @@ +/* +* replay-events.c +* +* Copyright (c) 2010-2015 Institute for System Programming +* of the Russian Academy of Sciences. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ + +#include "qemu-common.h" +#include "replay.h" +#include "replay-internal.h" + +typedef struct Event { + ReplayAsyncEventKind event_kind;
replay/replay-events.c line 27
Quoted 10 lines
replay/replay-events.c new file +10
-0+void *opaque; +void *opaque2; +uint64_t id; + +QTAILQ_ENTRY (Event) events; +} Event; + +static QTAILQ_HEAD(, Event) events_list = QTAILQ_HEAD_INITIALIZER(events_list); +/* Mutex to protect events_list modifications */ +static QemuMutex lock;
Can you just reuse the replay_lock? Otherwise you have to document the lock hierarchy. But if a single coarser lock is enough, that would be nice.
replay/replay-events.c line 30
Quoted 3 lines
replay/replay-events.c new file +3
-0+static unsigned int read_event_kind =-1 ; +static uint64_t read_id =-1 ; +static int read_opt =-1 ;
Please document what "opt" means.
replay/replay-events.c line 42
Quoted 12 lines
replay/replay-events.c new file +12
-0+ +static bool replay_events_enabled = false; + +/* Functions */ + +static void replay_run_event (Event *event) +{ +switch (event->event_kind) { +default : +fprintf (stderr,"Replay: invalid async event ID (%d) in the queue\n" , + event->event_kind); +exit (1 );
Please treat this the same as ferror(). fprintf(stderr) should be replaced by error_report(), which btw takes a string that is not terminated by \n.
Quoted 87 lines
replay/replay-events.c new file +87
-0+break ; + } +} + +void replay_enable_events (void ) +{ + replay_events_enabled = true; +} + +bool replay_has_events (void ) +{ +return !QTAILQ_EMPTY (&events_list); +} + +void replay_flush_events (void ) +{ +qemu_mutex_lock (&lock); +while (!QTAILQ_EMPTY (&events_list)) { +Event *event =QTAILQ_FIRST (&events_list); +qemu_mutex_unlock (&lock); +replay_run_event (event); +qemu_mutex_lock (&lock); +QTAILQ_REMOVE (&events_list, event, events); +g_free (event); + } +qemu_mutex_unlock (&lock); +} + +void replay_disable_events (void ) +{ + replay_events_enabled = false; +/* Flush events queue before waiting of completion */ +replay_flush_events (); +} + +void replay_clear_events (void ) +{ +qemu_mutex_lock (&lock); +while (!QTAILQ_EMPTY (&events_list)) { +Event *event =QTAILQ_FIRST (&events_list); +QTAILQ_REMOVE (&events_list, event, events); + +g_free (event); + } +qemu_mutex_unlock (&lock); +} + +static void replay_add_event_internal (ReplayAsyncEventKind event_kind, +void *opaque, +void *opaque2,uint64_t id) +{ +if (event_kind >=REPLAY_ASYNC_COUNT ) { +fprintf (stderr,"Replay: invalid async event ID (%d)\n" , event_kind); +exit (1 ); + } +if (!replay_file || replay_mode ==REPLAY_MODE_NONE + || !replay_events_enabled) { +Event e; + e.event_kind = event_kind; + e.opaque = opaque; + e.opaque2 = opaque2; + e.id = id; +replay_run_event (&e); +return ; + } + +Event *event =g_malloc0 (sizeof (Event)); + event->event_kind = event_kind; + event->opaque = opaque; + event->opaque2 = opaque2; + event->id = id; + +qemu_mutex_lock (&lock); +QTAILQ_INSERT_TAIL (&events_list, event, events); +qemu_mutex_unlock (&lock); +} + +void replay_add_event (ReplayAsyncEventKind event_kind,void *opaque) +{ +replay_add_event_internal (event_kind, opaque,NULL ,0 ); +} + +/* Called with replay mutex locked */ +void replay_save_events (int opt) +{ +qemu_mutex_lock (&lock); +while (!QTAILQ_EMPTY (&events_list)) {
replay/replay-events.c line 139
Quoted 10 lines
replay/replay-events.c new file +10
-0+Event *event =QTAILQ_FIRST (&events_list); +if (replay_mode !=REPLAY_MODE_PLAY ) { +/* put the event into the file */ +replay_put_event (EVENT_ASYNC ); +replay_put_byte (opt); +replay_put_byte (event->event_kind); + +/* save event-specific data */ +switch (event->event_kind) { + }
Please extract body of "if" to a separate function replay_save_event.
Quoted 22 lines
replay/replay-events.c new file +22
-0+ } + +qemu_mutex_unlock (&lock); +replay_mutex_unlock (); +replay_run_event (event); +replay_mutex_lock (); +qemu_mutex_lock (&lock); +QTAILQ_REMOVE (&events_list, event, events); +g_free (event); + } +qemu_mutex_unlock (&lock); +} + +/* Called with replay mutex locked */ +void replay_read_events (int opt) +{ +replay_fetch_data_kind (); +while (replay_data_kind ==EVENT_ASYNC ) { +if (read_event_kind ==-1 ) { + read_opt =replay_get_byte (); + read_event_kind =replay_get_byte (); + read_id =-1 ;
replay/replay-events.c line 171
Quoted 10 lines
replay/replay-events.c new file +10
-0+replay_check_error (); + } + +if (opt != read_opt) { +break ; + } +/* Execute some events without searching them in the queue */ +switch (read_event_kind) { +default : +fprintf (stderr,"Unknown ID %d of replay event\n" , read_event_kind);
Again, please treat this the same as ferror().
Quoted 20 lines
replay/replay-events.c new file +20
-0+exit (1 ); +break ; + } + +qemu_mutex_lock (&lock); + +Event *event =NULL ; +Event *curr =NULL ; + QTAILQ_FOREACH(curr, &events_list, events) { +if (curr->event_kind == read_event_kind + && (read_id ==-1 || read_id == curr->id)) { + event = curr; +break ; + } + } + +if (event) { +/* read event-specific reading data */ + +QTAILQ_REMOVE (&events_list, event, events);
replay/replay-events.c line 201
Quoted 10 lines
replay/replay-events.c new file +10
-0+ +qemu_mutex_unlock (&lock); + +/* reset unread data and other parameters to allow +reading other data from the log while +running the event */ + replay_has_unread_data =0 ; + read_event_kind =-1 ; + read_id =-1 ; + read_opt =-1 ;
Everything up to here can be extracted to a separate function that returns Event *. Then this function is:
while (replay_data_kind == EVENT_ASYNC) {
event = replay_read_event(opt);
if (!event) {
break;
} replay_mutex_unlock();
replay_run_event(event);
replay_mutex_lock();
g_free(event);
replay_fetch_data_kind();}
Quoted 31 lines
replay/replay-events.c new file +26
-0+replay_mutex_unlock (); +replay_run_event (event); +replay_mutex_lock (); +g_free (event); + +replay_fetch_data_kind (); + }else { +qemu_mutex_unlock (&lock); +/* No such event found in the queue */ +break ; + } + } +} + +void replay_init_events (void ) +{ + read_event_kind =-1 ; +qemu_mutex_init (&lock); +} + +void replay_finish_events (void ) +{ + replay_events_enabled = false; +replay_clear_events (); +qemu_mutex_destroy (&lock); +}replay/replay-internal.h +0
-0
replay/replay-internal.h enum ReplayEvents { · line 23
Quoted 5 lines
replay/replay-internal.h +1
-1@@ -20,9 +20,19 @@ enum ReplayEvents {/* for software interrupt */ EVENT_INTERRUPT ,/* for emulated exceptions */ -EVENT_EXCEPTION + EVENT_EXCEPTION,
Leave a trailing comma, and the patches become nicer. :)
replay/replay-internal.h enum ReplayEvents { · line 25
Quoted 2 lines
replay/replay-internal.h +2
-0+/* for async events */ +EVENT_ASYNC
Perhaps add EVENT_COUNT and add a sanity check on load?
Paolo
Quoted 54 lines
replay/replay-internal.h +29
-0}; +/* Asynchronous events IDs */ + +enum ReplayAsyncEventKind { +REPLAY_ASYNC_COUNT +}; + +typedef enum ReplayAsyncEventKind ReplayAsyncEventKind ; + typedef struct ReplayState { /*! Current step - number of processed instructions and timer events. */ uint64_t current_step;@@ -78,4 +88,25 @@ bool skip_async_events(int stop_event);reports an error and stops the execution. */ void skip_async_events_until (unsigned int kind); +/* Asynchronous events queue */ + +/*! Initializes events' processing internals */ +void replay_init_events (void ); +/*! Clears internal data structures for events handling */ +void replay_finish_events (void ); +/*! Enables storing events in the queue */ +void replay_enable_events (void ); +/*! Flushes events queue */ +void replay_flush_events (void ); +/*! Clears events list before loading new VM state */ +void replay_clear_events (void ); +/*! Returns true if there are any unsaved events in the queue */ +bool replay_has_events (void ); +/*! Saves events from queue into the file */ +void replay_save_events (int opt); +/*! Read events from the file into the input queue */ +void replay_read_events (int opt); +/*! Adds specified async event to the queue */ +void replay_add_event (ReplayAsyncEventKind event_id,void *opaque); +#endif replay/replay.h +4
-0@@ -43,5 +43,9 @@ bool replay_interrupt(void);Returns true, when interrupt request is pending */ bool replay_has_interrupt (void ); +/* Asynchronous events queue */ + +/*! Disables storing events in the queue */ +void replay_disable_events (void );#endif
RFC PATCH v8 11/21 replay: recording and replaying clock ticks
In reply to Pavel DovgalyukClock ticks are considered as the sources of non-deterministic data for virtual machine. This patch implements saving the clock values when they are acquired (virtual, host clock, rdtsc, and some other timers). When replaying the execution corresponding values are read from log and transfered to the module, which wants to read the values.
Such a design required the clock polling to be synchronized. Sometimes it is not true - e.g. when timeouts for timer lists are checked. In this case we use a cached value of the clock, passing it to the client code.
--- cpus.c | 3 +- include/qemu/timer.h | 10 +++++ qemu-timer.c | 7 ++-- replay/Makefile.objs | 1 + replay/replay-internal.h | 13 +++++++ replay/replay-time.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 25 ++++++++++++++ stubs/replay.c | 9 +++++ 8 files changed, 147 insertions(+), 5 deletions(-) create mode 100755 replay/replay-time.c
cpus.c +2 -1
@@ -353,7 +353,8 @@ static void icount_warp_rt(void *opaque)seqlock_write_lock (&timers_state.vm_clock_seqlock);if (runstate_is_running ()) { -int64_t clock =cpu_get_clock_locked (); +int64_t clock =REPLAY_CLOCK (REPLAY_CLOCK_VIRTUAL_RT , +cpu_get_clock_locked ());int64_t warp_delta; warp_delta = clock - vm_clock_warp_start;
include/qemu/timer.h +10 -0
@@ -4,6 +4,7 @@#include "qemu/typedefs.h" #include "qemu-common.h" #include "qemu/notify.h" +#include "replay/replay.h" /* timers */
@@ -760,6 +761,8 @@ int64_t cpu_icount_to_ns(int64_t icount);/*******************************************/ /* host CPU ticks (if available) */ +#define cpu_get_real_ticks cpu_get_real_ticks_impl +#if defined(_ARCH_PPC)static inline int64_t cpu_get_real_ticks (void )
@@ -913,6 +916,13 @@ static inline int64_t cpu_get_real_ticks (void) }#endif +#undef cpu_get_real_ticks + +static inline int64_t cpu_get_real_ticks (void ) +{ +return REPLAY_CLOCK (REPLAY_CLOCK_REAL_TICKS ,cpu_get_real_ticks_impl ()); +} +#ifdef CONFIG_PROFILER static inline int64_t profile_getclock (void ) {
qemu-timer.c +4 -3
@@ -25,6 +25,7 @@#include "sysemu/sysemu.h" #include "monitor/monitor.h" #include "ui/console.h" +#include "replay/replay.h" #include "hw/hw.h"
@@ -566,15 +567,15 @@ int64_t qemu_clock_get_ns(QEMUClockType type)return cpu_get_clock (); }case QEMU_CLOCK_HOST : - now =get_clock_realtime (); + now =REPLAY_CLOCK (REPLAY_CLOCK_HOST ,get_clock_realtime ()); last = clock->last; clock->last = now; -if (now < last) { +if (now < last && replay_mode ==REPLAY_MODE_NONE ) {notifier_list_notify (&clock->reset_notifiers, &now); }return now;case QEMU_CLOCK_VIRTUAL_RT : -return cpu_get_clock (); +return REPLAY_CLOCK (REPLAY_CLOCK_VIRTUAL_RT ,cpu_get_clock ()); } }
replay/Makefile.objs +1 -0
@@ -1,3 +1,4 @@ obj-y += replay.o obj-y += replay-internal.o obj-y += replay-events.o +obj-y += replay-time.o
replay/replay-internal.h +12 -1
@@ -22,7 +22,10 @@ enum ReplayEvents {/* for emulated exceptions */ EVENT_EXCEPTION ,/* for async events */ - EVENT_ASYNC +EVENT_ASYNC , +/* for clock read/writes */ +/* some of grteater codes are reserved for clocks */ + EVENT_CLOCK };/* Asynchronous events IDs */
@@ -34,6 +37,8 @@ enum ReplayAsyncEventKind {typedef enum ReplayAsyncEventKind ReplayAsyncEventKind ; typedef struct ReplayState { + /*! Cached clock values. */ + int64_t cached_clock[REPLAY_CLOCK_COUNT]; /*! Current step - number of processed instructions and timer events. */ uint64_t current_step; /*! Number of instructions to be executed before other events happen. */
@@ -88,6 +93,12 @@ bool skip_async_events(int stop_event);reports an error and stops the execution. */ void skip_async_events_until (unsigned int kind); +/*! Reads next clock value from the file. +If clock kind read from the file is different from the parameter, +the value is not used. +If the parameter is -1, the clock value is read to the cache anyway. */ +void replay_read_next_clock (unsigned int kind); +/* Asynchronous events queue */ /*! Initializes events' processing internals */
replay/replay-time.c new file +84 -0
84 lines not on this page · raw message
replay/replay.h +25 -0
@@ -16,6 +16,18 @@#include <stdint.h> #include "qapi-types.h" +/* replay clock kinds */ +enum ReplayClockKind { +/* rdtsc */ +REPLAY_CLOCK_REAL_TICKS , +/* host_clock */ +REPLAY_CLOCK_HOST , +/* virtual_rt_clock */ +REPLAY_CLOCK_VIRTUAL_RT , +REPLAY_CLOCK_COUNT +}; +typedef enum ReplayClockKind ReplayClockKind ; +extern ReplayMode replay_mode;/* Processing the instructions */
@@ -43,6 +55,19 @@ bool replay_interrupt(void);Returns true, when interrupt request is pending */ bool replay_has_interrupt (void ); +/* Processing clocks and other time sources */ + +/*! Save the specified clock */ +int64_t replay_save_clock (ReplayClockKind kind,int64_t clock); +/*! Read the specified clock from the log or return cached data */ +int64_t replay_read_clock (ReplayClockKind kind); +/*! Saves or reads the clock depending on the current replay mode. */ +#define REPLAY_CLOCK (clock, value) \ + (replay_mode == REPLAY_MODE_PLAY ? replay_read_clock((clock)) \ + : replay_mode == REPLAY_MODE_RECORD \ + ? replay_save_clock((clock), (value)) \ + : (value)) +/* Asynchronous events queue */ /*! Disables storing events in the queue */
stubs/replay.c +9 -0
@@ -1,3 +1,12 @@#include "replay/replay.h" ReplayMode replay_mode; + +int64_t replay_save_clock (unsigned int kind,int64_t clock) +{ +} + +int64_t replay_read_clock (unsigned int kind) +{ +return 0 ; +}
Re: [Qemu-devel] [RFC PATCH v8 11/21] replay: recording and replaying clock ticks
Quoted 58 lines · Pavel Dovgalyuk
Clock ticks are considered as the sources of non-deterministic data for virtual machine. This patch implements saving the clock values when they are acquired (virtual, host clock, rdtsc, and some other timers). When replaying the execution corresponding values are read from log and transfered to the module, which wants to read the values.
Such a design required the clock polling to be synchronized. Sometimes it is not true - e.g. when timeouts for timer lists are checked. In this case we use a cached value of the clock, passing it to the client code.Signed-off-by Pavel Dovgalyuk <[email protected]>--- cpus.c | 3 +- include/qemu/timer.h | 10 +++++ qemu-timer.c | 7 ++-- replay/Makefile.objs | 1 + replay/replay-internal.h | 13 +++++++ replay/replay-time.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 25 ++++++++++++++ stubs/replay.c | 9 +++++ 8 files changed, 147 insertions(+), 5 deletions(-) create mode 100755 replay/replay-time.ccpus.c +2
-1@@ -353,7 +353,8 @@ static void icount_warp_rt(void *opaque)seqlock_write_lock (&timers_state.vm_clock_seqlock);if (runstate_is_running ()) { -int64_t clock =cpu_get_clock_locked (); +int64_t clock =REPLAY_CLOCK (REPLAY_CLOCK_VIRTUAL_RT , +cpu_get_clock_locked ());int64_t warp_delta; warp_delta = clock - vm_clock_warp_start;include/qemu/timer.h +3
-0@@ -4,6 +4,7 @@#include "qemu/typedefs.h" #include "qemu-common.h" #include "qemu/notify.h" +#include "replay/replay.h" /* timers */ @@ -760,6 +761,8 @@ int64_t cpu_icount_to_ns(int64_t icount);/*******************************************/ /* host CPU ticks (if available) */ +#define cpu_get_real_ticks cpu_get_real_ticks_impl +#if defined(_ARCH_PPC)static inline int64_t cpu_get_real_ticks (void )
include/qemu/timer.h static inline int64_t cpu_get_real_ticks (void) · line 921
Quoted 6 lines
include/qemu/timer.h +3
-0@@ -913,6 +916,13 @@ static inline int64_t cpu_get_real_ticks (void) }#endif +#undef cpu_get_real_ticks + +static inline int64_t cpu_get_real_ticks (void )
cpu_get_real_ticks should never be used. Please instead wrap cpu_get_ticks() with REPLAY_CLOCK.
Quoted 73 lines
include/qemu/timer.h +4
-0+{ +return REPLAY_CLOCK (REPLAY_CLOCK_REAL_TICKS ,cpu_get_real_ticks_impl ()); +} +#ifdef CONFIG_PROFILER static inline int64_t profile_getclock (void ) {qemu-timer.c +4
-3@@ -25,6 +25,7 @@#include "sysemu/sysemu.h" #include "monitor/monitor.h" #include "ui/console.h" +#include "replay/replay.h" #include "hw/hw.h" @@ -566,15 +567,15 @@ int64_t qemu_clock_get_ns(QEMUClockType type)return cpu_get_clock (); }case QEMU_CLOCK_HOST : - now =get_clock_realtime (); + now =REPLAY_CLOCK (REPLAY_CLOCK_HOST ,get_clock_realtime ()); last = clock->last; clock->last = now; -if (now < last) { +if (now < last && replay_mode ==REPLAY_MODE_NONE ) {notifier_list_notify (&clock->reset_notifiers, &now); }return now;case QEMU_CLOCK_VIRTUAL_RT : -return cpu_get_clock (); +return REPLAY_CLOCK (REPLAY_CLOCK_VIRTUAL_RT ,cpu_get_clock ()); } }replay/Makefile.objs +1
-0@@ -1,3 +1,4 @@ obj-y += replay.o obj-y += replay-internal.o obj-y += replay-events.o +obj-y += replay-time.oreplay/replay-internal.h +6
-1@@ -22,7 +22,10 @@ enum ReplayEvents {/* for emulated exceptions */ EVENT_EXCEPTION ,/* for async events */ - EVENT_ASYNC +EVENT_ASYNC , +/* for clock read/writes */ +/* some of grteater codes are reserved for clocks */ + EVENT_CLOCK };/* Asynchronous events IDs */ @@ -34,6 +37,8 @@ enum ReplayAsyncEventKind {typedef enum ReplayAsyncEventKind ReplayAsyncEventKind ; typedef struct ReplayState { + /*! Cached clock values. */ + int64_t cached_clock[REPLAY_CLOCK_COUNT]; /*! Current step - number of processed instructions and timer events. */ uint64_t current_step; /*! Number of instructions to be executed before other events happen. */
replay/replay-internal.h bool skip_async_events(int stop_event); · line 99
Quoted 7 lines
replay/replay-internal.h +4
-0@@ -88,6 +93,12 @@ bool skip_async_events(int stop_event);reports an error and stops the execution. */ void skip_async_events_until (unsigned int kind); +/*! Reads next clock value from the file. +If clock kind read from the file is different from the parameter, +the value is not used. +If the parameter is -1, the clock value is read to the cache anyway. */
In what case could the clock kind not match?
Paolo
Quoted 156 lines
replay/replay-internal.h +2
-0+void replay_read_next_clock (unsigned int kind); +/* Asynchronous events queue */ /*! Initializes events' processing internals */ replay/replay-time.c new file +84
-0@@ -0,0 +1,84 @@ +/* +* replay-time.c +* +* Copyright (c) 2010-2015 Institute for System Programming +* of the Russian Academy of Sciences. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ + +#include "qemu-common.h" +#include "replay.h" +#include "replay-internal.h" + + +int64_t replay_save_clock (ReplayClockKind kind,int64_t clock) +{ +replay_save_instructions (); + +if (kind >=REPLAY_CLOCK_COUNT ) { +fprintf (stderr,"invalid clock ID %d for replay\n" , kind); +exit (1 ); + } + +if (replay_file) { +replay_mutex_lock (); +replay_put_event (EVENT_CLOCK + kind); +replay_put_qword (clock); +replay_mutex_unlock (); + } + +return clock; +} + +void replay_read_next_clock (ReplayClockKind kind) +{ +replay_fetch_data_kind (); +if (replay_file) { +unsigned int read_kind = replay_data_kind -EVENT_CLOCK ; + +if (kind !=-1 && read_kind != kind) { +return ; + } +if (read_kind >=REPLAY_CLOCK_COUNT ) { +fprintf (stderr, +"invalid clock ID %d was read from replay\n" , read_kind); +exit (1 ); + } + +int64_t clock =replay_get_qword (); + +replay_check_error (); + replay_has_unread_data =0 ; + + replay_state.cached_clock[read_kind] = clock; + } +} + +/*! Reads next clock event from the input. */ +int64_t replay_read_clock (ReplayClockKind kind) +{ +if (kind >=REPLAY_CLOCK_COUNT ) { +fprintf (stderr,"invalid clock ID %d for replay\n" , kind); +exit (1 ); + } + +replay_exec_instructions (); + +if (replay_file) { +int64_t ret; +replay_mutex_lock (); +if (skip_async_events (EVENT_CLOCK + kind)) { +replay_read_next_clock (kind); + } + ret = replay_state.cached_clock[kind]; +replay_mutex_unlock (); + +return ret; + } + +fprintf (stderr,"REPLAY INTERNAL ERROR %d\n" , __LINE__); +exit (1 ); +}replay/replay.h +25
-0@@ -16,6 +16,18 @@#include <stdint.h> #include "qapi-types.h" +/* replay clock kinds */ +enum ReplayClockKind { +/* rdtsc */ +REPLAY_CLOCK_REAL_TICKS , +/* host_clock */ +REPLAY_CLOCK_HOST , +/* virtual_rt_clock */ +REPLAY_CLOCK_VIRTUAL_RT , +REPLAY_CLOCK_COUNT +}; +typedef enum ReplayClockKind ReplayClockKind ; +extern ReplayMode replay_mode;/* Processing the instructions */ @@ -43,6 +55,19 @@ bool replay_interrupt(void);Returns true, when interrupt request is pending */ bool replay_has_interrupt (void ); +/* Processing clocks and other time sources */ + +/*! Save the specified clock */ +int64_t replay_save_clock (ReplayClockKind kind,int64_t clock); +/*! Read the specified clock from the log or return cached data */ +int64_t replay_read_clock (ReplayClockKind kind); +/*! Saves or reads the clock depending on the current replay mode. */ +#define REPLAY_CLOCK (clock, value) \ + (replay_mode == REPLAY_MODE_PLAY ? replay_read_clock((clock)) \ + : replay_mode == REPLAY_MODE_RECORD \ + ? replay_save_clock((clock), (value)) \ + : (value)) +/* Asynchronous events queue */ /*! Disables storing events in the queue */ stubs/replay.c +9
-0@@ -1,3 +1,12 @@#include "replay/replay.h" ReplayMode replay_mode; + +int64_t replay_save_clock (unsigned int kind,int64_t clock) +{ +} + +int64_t replay_read_clock (unsigned int kind) +{ +return 0 ; +}
Re: [Qemu-devel] [RFC PATCH v8 11/21] replay: recording and replaying clock ticks
Quoted 63 lines
From: Paolo Bonzini [mailto:[email protected]]
Quoted 61 lines · Pavel Dovgalyuk
Clock ticks are considered as the sources of non-deterministic data for virtual machine. This patch implements saving the clock values when they are acquired (virtual, host clock, rdtsc, and some other timers). When replaying the execution corresponding values are read from log and transfered to the module, which wants to read the values.
Such a design required the clock polling to be synchronized. Sometimes it is not true - e.g. when timeouts for timer lists are checked. In this case we use a cached value of the clock, passing it to the client code.Signed-off-by Pavel Dovgalyuk <[email protected]>--- cpus.c | 3 +- include/qemu/timer.h | 10 +++++ qemu-timer.c | 7 ++-- replay/Makefile.objs | 1 + replay/replay-internal.h | 13 +++++++ replay/replay-time.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 25 ++++++++++++++ stubs/replay.c | 9 +++++ 8 files changed, 147 insertions(+), 5 deletions(-) create mode 100755 replay/replay-time.ccpus.c +2
-1@@ -353,7 +353,8 @@ static void icount_warp_rt(void *opaque)seqlock_write_lock (&timers_state.vm_clock_seqlock);if (runstate_is_running ()) { -int64_t clock =cpu_get_clock_locked (); +int64_t clock =REPLAY_CLOCK (REPLAY_CLOCK_VIRTUAL_RT , +cpu_get_clock_locked ());int64_t warp_delta; warp_delta = clock - vm_clock_warp_start;include/qemu/timer.h +3
-0@@ -4,6 +4,7 @@#include "qemu/typedefs.h" #include "qemu-common.h" #include "qemu/notify.h" +#include "replay/replay.h" /* timers */ @@ -760,6 +761,8 @@ int64_t cpu_icount_to_ns(int64_t icount);/*******************************************/ /* host CPU ticks (if available) */ +#define cpu_get_real_ticks cpu_get_real_ticks_impl +#if defined(_ARCH_PPC)static inline int64_t cpu_get_real_ticks (void )@@ -913,6 +916,13 @@ static inline int64_t cpu_get_real_ticks (void) }#endif
Quoted 6 lines
include/qemu/timer.h static inline int64_t cpu_get_real_ticks (void) · line 921
Quoted 3 lines
include/qemu/timer.h +3
-0+#undef cpu_get_real_ticks + +static inline int64_t cpu_get_real_ticks (void )
cpu_get_real_ticksshould never be used. Please instead wrapcpu_get_ticks()withREPLAY_CLOCK.
I don't quite understand this comment.
Do you mean that I should move REPLAY_CLOCK to the cpu_get_real_ticks usages instead of it's implementation?
Quoted 77 lines
Quoted 77 lines
include/qemu/timer.h +4
-0+{ +return REPLAY_CLOCK (REPLAY_CLOCK_REAL_TICKS ,cpu_get_real_ticks_impl ()); +} +#ifdef CONFIG_PROFILER static inline int64_t profile_getclock (void ) {qemu-timer.c +4
-3@@ -25,6 +25,7 @@#include "sysemu/sysemu.h" #include "monitor/monitor.h" #include "ui/console.h" +#include "replay/replay.h" #include "hw/hw.h" @@ -566,15 +567,15 @@ int64_t qemu_clock_get_ns(QEMUClockType type)return cpu_get_clock (); }case QEMU_CLOCK_HOST : - now =get_clock_realtime (); + now =REPLAY_CLOCK (REPLAY_CLOCK_HOST ,get_clock_realtime ()); last = clock->last; clock->last = now; -if (now < last) { +if (now < last && replay_mode ==REPLAY_MODE_NONE ) {notifier_list_notify (&clock->reset_notifiers, &now); }return now;case QEMU_CLOCK_VIRTUAL_RT : -return cpu_get_clock (); +return REPLAY_CLOCK (REPLAY_CLOCK_VIRTUAL_RT ,cpu_get_clock ()); } }replay/Makefile.objs +1
-0@@ -1,3 +1,4 @@ obj-y += replay.o obj-y += replay-internal.o obj-y += replay-events.o +obj-y += replay-time.oreplay/replay-internal.h +6
-1@@ -22,7 +22,10 @@ enum ReplayEvents {/* for emulated exceptions */ EVENT_EXCEPTION ,/* for async events */ - EVENT_ASYNC +EVENT_ASYNC , +/* for clock read/writes */ +/* some of grteater codes are reserved for clocks */ + EVENT_CLOCK };/* Asynchronous events IDs */ @@ -34,6 +37,8 @@ enum ReplayAsyncEventKind {typedef enum ReplayAsyncEventKind ReplayAsyncEventKind ; typedef struct ReplayState { + /*! Cached clock values. */ + int64_t cached_clock[REPLAY_CLOCK_COUNT]; /*! Current step - number of processed instructions and timer events. */ uint64_t current_step; /*! Number of instructions to be executed before other events happen. */@@ -88,6 +93,12 @@ bool skip_async_events(int stop_event);reports an error and stops the execution. */ void skip_async_events_until (unsigned int kind);
Quoted 7 lines
replay/replay-internal.h bool skip_async_events(int stop_event); · line 99
Quoted 4 lines
replay/replay-internal.h +4
-0+/*! Reads next clock value from the file. +If clock kind read from the file is different from the parameter, +the value is not used. +If the parameter is -1, the clock value is read to the cache anyway. */ In what case could the clock kind not match?
It was used in full version which had to skip clock from the log while loading the VM state.
Quoted 5 lines
replay/replay-internal.h bool skip_async_events(int stop_event); · line 103
Quoted 5 lines
replay/replay-internal.h +2
-0+void replay_read_next_clock (unsigned int kind); +/* Asynchronous events queue */
Pavel Dovgalyuk
Re: [Qemu-devel] [RFC PATCH v8 11/21] replay: recording and replaying clock ticks
Quoted 70 lines · Pavel Dovgaluk
Quoted 63 lines
From: Paolo Bonzini [mailto:[email protected]]
Quoted 61 lines · Pavel Dovgalyuk
Clock ticks are considered as the sources of non-deterministic data for virtual machine. This patch implements saving the clock values when they are acquired (virtual, host clock, rdtsc, and some other timers). When replaying the execution corresponding values are read from log and transfered to the module, which wants to read the values.
Such a design required the clock polling to be synchronized. Sometimes it is not true - e.g. when timeouts for timer lists are checked. In this case we use a cached value of the clock, passing it to the client code.Signed-off-by Pavel Dovgalyuk <[email protected]>--- cpus.c | 3 +- include/qemu/timer.h | 10 +++++ qemu-timer.c | 7 ++-- replay/Makefile.objs | 1 + replay/replay-internal.h | 13 +++++++ replay/replay-time.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 25 ++++++++++++++ stubs/replay.c | 9 +++++ 8 files changed, 147 insertions(+), 5 deletions(-) create mode 100755 replay/replay-time.ccpus.c +2
-1@@ -353,7 +353,8 @@ static void icount_warp_rt(void *opaque)seqlock_write_lock (&timers_state.vm_clock_seqlock);if (runstate_is_running ()) { -int64_t clock =cpu_get_clock_locked (); +int64_t clock =REPLAY_CLOCK (REPLAY_CLOCK_VIRTUAL_RT , +cpu_get_clock_locked ());int64_t warp_delta; warp_delta = clock - vm_clock_warp_start;include/qemu/timer.h +3
-0@@ -4,6 +4,7 @@#include "qemu/typedefs.h" #include "qemu-common.h" #include "qemu/notify.h" +#include "replay/replay.h" /* timers */ @@ -760,6 +761,8 @@ int64_t cpu_icount_to_ns(int64_t icount);/*******************************************/ /* host CPU ticks (if available) */ +#define cpu_get_real_ticks cpu_get_real_ticks_impl +#if defined(_ARCH_PPC)static inline int64_t cpu_get_real_ticks (void )@@ -913,6 +916,13 @@ static inline int64_t cpu_get_real_ticks (void) }#endif Quoted 6 lines
include/qemu/timer.h static inline int64_t cpu_get_real_ticks (void) · line 921
Quoted 3 lines
include/qemu/timer.h +3
-0+#undef cpu_get_real_ticks + +static inline int64_t cpu_get_real_ticks (void )
cpu_get_real_ticksshould never be used. Please instead wrapcpu_get_ticks()withREPLAY_CLOCK.
Quoted 2 lines
I don't quite understand this comment.
Do you mean that I should moveREPLAY_CLOCKto thecpu_get_real_ticksusages instead of it's implementation?
Only to the cpu_get_ticks usage. The others are okay.
Quoted 9 lines
Quoted 7 lines
include/qemu/timer.h static inline int64_t cpu_get_real_ticks (void) · line 925
Quoted 4 lines
include/qemu/timer.h +4
-0+/*! Reads next clock value from the file. +If clock kind read from the file is different from the parameter, +the value is not used. +If the parameter is -1, the clock value is read to the cache anyway. */ In what case could the clock kind not match?
It was used in full version which had to skip clock from the log while loading the VM state.
So can it be removed for now?
Paolo
Quoted 9 lines
Quoted 5 lines
include/qemu/timer.h static inline int64_t cpu_get_real_ticks (void) · line 928
Quoted 5 lines
include/qemu/timer.h +2
-0+void replay_read_next_clock (unsigned int kind); +/* Asynchronous events queue */ Pavel Dovgalyuk
Re: [Qemu-devel] [RFC PATCH v8 11/21] replay: recording and replaying clock ticks
Quoted 72 lines
From: Paolo Bonzini [mailto:[email protected]]
Quoted 70 lines · Pavel Dovgaluk
Quoted 69 lines
From: Paolo Bonzini [mailto:[email protected]]
Quoted 58 lines · Pavel Dovgalyuk
Clock ticks are considered as the sources of non-deterministic data for virtual machine. This patch implements saving the clock values when they are acquired (virtual, host clock, rdtsc, and some other timers). When replaying the execution corresponding values are read from log and transfered to the module, which wants to read the values.
Such a design required the clock polling to be synchronized. Sometimes it is not true - e.g. when timeouts for timer lists are checked. In this case we use a cached value of the clock, passing it to the client code.Signed-off-by Pavel Dovgalyuk <[email protected]>--- cpus.c | 3 +- include/qemu/timer.h | 10 +++++ qemu-timer.c | 7 ++-- replay/Makefile.objs | 1 + replay/replay-internal.h | 13 +++++++ replay/replay-time.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 25 ++++++++++++++ stubs/replay.c | 9 +++++ 8 files changed, 147 insertions(+), 5 deletions(-) create mode 100755 replay/replay-time.ccpus.c +2
-1@@ -353,7 +353,8 @@ static void icount_warp_rt(void *opaque)seqlock_write_lock (&timers_state.vm_clock_seqlock);if (runstate_is_running ()) { -int64_t clock =cpu_get_clock_locked (); +int64_t clock =REPLAY_CLOCK (REPLAY_CLOCK_VIRTUAL_RT , +cpu_get_clock_locked ());int64_t warp_delta; warp_delta = clock - vm_clock_warp_start;include/qemu/timer.h +3
-0@@ -4,6 +4,7 @@#include "qemu/typedefs.h" #include "qemu-common.h" #include "qemu/notify.h" +#include "replay/replay.h" /* timers */ @@ -760,6 +761,8 @@ int64_t cpu_icount_to_ns(int64_t icount);/*******************************************/ /* host CPU ticks (if available) */ +#define cpu_get_real_ticks cpu_get_real_ticks_impl +#if defined(_ARCH_PPC)static inline int64_t cpu_get_real_ticks (void )include/qemu/timer.h static inline int64_t cpu_get_real_ticks (void) · line 921
Quoted 6 lines
include/qemu/timer.h +3
-0@@ -913,6 +916,13 @@ static inline int64_t cpu_get_real_ticks (void) }#endif +#undef cpu_get_real_ticks + +static inline int64_t cpu_get_real_ticks (void )
cpu_get_real_ticksshould never be used. Please instead wrapcpu_get_ticks()withREPLAY_CLOCK.
Quoted 5 lines
Quoted 2 lines
I don't quite understand this comment.
Do you mean that I should moveREPLAY_CLOCKto thecpu_get_real_ticksusages instead of it'simplementation?
Only to the
cpu_get_ticksusage. The others are okay.
cpu_get_ticks cannot call cpu_get_real_ticks in icount mode.
And other functions can. Then we should put REPLAY_CLOCK into those functions?
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
include/qemu/timer.h +4
-0+/*! Reads next clock value from the file. +If clock kind read from the file is different from the parameter, +the value is not used. +If the parameter is -1, the clock value is read to the cache anyway. */
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
In what case could the clock kind not match?
It was used in full version which had to skip clock from the log while loading the VM state.
So can it be removed for now?
I think it can.
Pavel Dovgalyuk
Re: [Qemu-devel] [RFC PATCH v8 11/21] replay: recording and replaying clock ticks
Quoted 3 lines · Pavel Dovgaluk
Quoted 1 line
Only to the
cpu_get_ticksusage. The others are okay.
cpu_get_tickscannot callcpu_get_real_ticksin icount mode.
You're right, but...
Quoted 1 line
And other functions can.
... which functions? cpu_enable_ticks and cpu_disable_ticks call it, but the result is only used by cpu_get_ticks. As you said, this cannot be called in icount mode.
A couple of PPC device models call it, but they probably should call cpu_get_ticks instead. So it looks like handling REAL_TICKS is altogether unnecessary.
Paolo
RFC PATCH v8 12/21 replay: recording and replaying different timers
In reply to Pavel DovgalyukThis patch introduces functions for recording and replaying realtime sources, that do not use qemu-clock interface. These include return value of time() function in time_t and struct tm forms. Patch also adds warning to get_timedate function to prevent its usage in recording mode, because it may lead to non-determinism.
--- hw/timer/mc146818rtc.c | 3 + hw/timer/pl031.c | 3 + include/qemu-common.h | 1 replay/replay-internal.h | 4 + replay/replay-time.c | 132 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 8 +++ vl.c | 17 +++++- 7 files changed, 163 insertions(+), 5 deletions(-)
hw/timer/mc146818rtc.c +2 -1
@@ -28,6 +28,7 @@#include "qapi/visitor.h" #include "qapi-event.h" #include "qmp-commands.h" +#include "replay/replay.h" #ifdef TARGET_I386 #include "hw/i386/apic.h"
@@ -703,7 +704,7 @@ static void rtc_set_date_from_host(ISADevice *dev)RTCState *s =MC146818_RTC (dev);struct tm tm; -qemu_get_timedate (&tm,0 ); +replay_get_timedate (&tm); s->base_rtc =mktimegm (&tm); s->last_update =qemu_clock_get_ns (rtc_clock);
hw/timer/pl031.c +2 -1
@@ -14,6 +14,7 @@#include "hw/sysbus.h" #include "qemu/timer.h" #include "sysemu/sysemu.h" +#include "replay/replay.h" //#define DEBUG_PL031
@@ -200,7 +201,7 @@ static int pl031_init(SysBusDevice *dev)sysbus_init_mmio (dev, &s->iomem);sysbus_init_irq (dev, &s->irq); -qemu_get_timedate (&tm,0 ); +replay_get_timedate (&tm); s->tick_offset =mktimegm (&tm) -qemu_clock_get_ns (rtc_clock) /get_ticks_per_sec ();
include/qemu-common.h +1 -0
@@ -129,6 +129,7 @@ void dump_drift_info(FILE *f, fprintf_function cpu_fprintf);int qemu_main (int argc,char **argv,char **envp);#endif +void qemu_get_timedate_no_warning (struct tm *tm,int offset);void qemu_get_timedate (struct tm *tm,int offset);int qemu_timedate_diff (struct tm *tm);
replay/replay-internal.h +4 -0
@@ -23,6 +23,10 @@ enum ReplayEvents {EVENT_EXCEPTION ,/* for async events */ EVENT_ASYNC , +/* for time_t event */ +EVENT_TIME_T , +/* for tm event */ +EVENT_TM ,/* for clock read/writes */ /* some of grteater codes are reserved for clocks */ EVENT_CLOCK
replay/replay-time.c +132 -0
135 lines not on this page · raw message
replay/replay.h +8 -0
@@ -14,6 +14,7 @@#include <stdbool.h> #include <stdint.h> +#include <time.h> #include "qapi-types.h" /* replay clock kinds */
@@ -67,6 +68,13 @@ int64_t replay_read_clock(ReplayClockKind kind); : replay_mode == REPLAY_MODE_RECORD \ ? replay_save_clock((clock), (value)) \ : (value)) +/*! Returns result of time() function execution in normal and record modes. +In play mode returns value read from the log. */ +time_t replay_time (void ); +/*! This function is used for obtaining current timedate. +In record mode it also saves in into the replay log. +In replay mode it just read from the log. */ +void replay_get_timedate (struct tm *tm);/* Asynchronous events queue */
vl.c +14 -3
@@ -118,6 +118,7 @@ int main(int argc, char **argv)#include "qapi/opts-visitor.h" #include "qom/object_interfaces.h" #include "qapi-event.h" +#include "replay/replay.h" #define DEFAULT_RAM_SIZE 128
@@ -715,7 +716,7 @@ void vm_start(void)/***********************************************************/ /* host time/date access */ -void qemu_get_timedate (struct tm *tm,int offset) +void qemu_get_timedate_no_warning (struct tm *tm,int offset) {time_t ti;
@@ -732,6 +733,16 @@ void qemu_get_timedate(struct tm *tm, int offset) } } +/* host time/date access with replay warning */ +void qemu_get_timedate (struct tm *tm,int offset) +{ +if (replay_mode ==REPLAY_MODE_RECORD ) { +fprintf (stderr,"REPLAY WARNING! qemu_get_timedate " +"function may lead to non-determinism\n" ); + } +qemu_get_timedate_no_warning (tm, offset); +} +int qemu_timedate_diff (struct tm *tm) {time_t seconds;
@@ -747,7 +758,7 @@ int qemu_timedate_diff(struct tm *tm)else seconds =mktimegm (tm) + rtc_date_offset; -return seconds -time (NULL ); +return seconds -replay_time (); }static void configure_rtc_date_offset (const char *startdate,int legacy)
@@ -785,7 +796,7 @@ static void configure_rtc_date_offset(const char *startdate, int legacy) "'2006-06-17T16:01:21' or '2006-06-17'\n");exit (1 ); } - rtc_date_offset =time (NULL ) - rtc_start_date; + rtc_date_offset =replay_time () - rtc_start_date; } }
Re: [Qemu-devel] [RFC PATCH v8 12/21] replay: recording and replaying different timers
Quoted 30 lines · Pavel Dovgalyuk
This patch introduces functions for recording and replaying realtime sources, that do not use qemu-clock interface. These include return value of
time()function intime_tand struct tm forms. Patch also adds warning toget_timedatefunction to prevent its usage in recording mode, because it may lead to non-determinism.Signed-off-by Pavel Dovgalyuk <[email protected]>--- hw/timer/mc146818rtc.c | 3 + hw/timer/pl031.c | 3 + include/qemu-common.h | 1 replay/replay-internal.h | 4 + replay/replay-time.c | 132 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 8 +++ vl.c | 17 +++++- 7 files changed, 163 insertions(+), 5 deletions(-)hw/timer/mc146818rtc.c +1
-0@@ -28,6 +28,7 @@#include "qapi/visitor.h" #include "qapi-event.h" #include "qmp-commands.h" +#include "replay/replay.h" #ifdef TARGET_I386 #include "hw/i386/apic.h"
hw/timer/mc146818rtc.c static void rtc_set_date_from_host(ISADevice *dev) · line 706
Quoted 4 lines
hw/timer/mc146818rtc.c +0
-1@@ -703,7 +704,7 @@ static void rtc_set_date_from_host(ISADevice *dev)RTCState *s =MC146818_RTC (dev);struct tm tm; -qemu_get_timedate (&tm,0 );
What about just making qemu_get_timedate use qemu_clock_get_ns(QEMU_CLOCK_HOST) instead of time()? This would just work using the infrastructure of the previous patch.
Paolo
Quoted 275 lines
hw/timer/mc146818rtc.c +1
-0+replay_get_timedate (&tm); s->base_rtc =mktimegm (&tm); s->last_update =qemu_clock_get_ns (rtc_clock);hw/timer/pl031.c +2
-1@@ -14,6 +14,7 @@#include "hw/sysbus.h" #include "qemu/timer.h" #include "sysemu/sysemu.h" +#include "replay/replay.h" //#define DEBUG_PL031 @@ -200,7 +201,7 @@ static int pl031_init(SysBusDevice *dev)sysbus_init_mmio (dev, &s->iomem);sysbus_init_irq (dev, &s->irq); -qemu_get_timedate (&tm,0 ); +replay_get_timedate (&tm); s->tick_offset =mktimegm (&tm) -qemu_clock_get_ns (rtc_clock) /get_ticks_per_sec ();include/qemu-common.h +1
-0@@ -129,6 +129,7 @@ void dump_drift_info(FILE *f, fprintf_function cpu_fprintf);int qemu_main (int argc,char **argv,char **envp);#endif +void qemu_get_timedate_no_warning (struct tm *tm,int offset);void qemu_get_timedate (struct tm *tm,int offset);int qemu_timedate_diff (struct tm *tm);replay/replay-internal.h +4
-0@@ -23,6 +23,10 @@ enum ReplayEvents {EVENT_EXCEPTION ,/* for async events */ EVENT_ASYNC , +/* for time_t event */ +EVENT_TIME_T , +/* for tm event */ +EVENT_TM ,/* for clock read/writes */ /* some of grteater codes are reserved for clocks */ EVENT_CLOCK replay/replay-time.c +132
-0@@ -82,3 +82,135 @@ int64_t replay_read_clock(ReplayClockKind kind)fprintf (stderr,"REPLAY INTERNAL ERROR %d\n" , __LINE__);exit (1 ); } + +/*! Saves time_t value to the log */ +static void replay_save_time_t (time_t tm) +{ +replay_save_instructions (); + +if (replay_file) { +replay_mutex_lock (); +replay_put_event (EVENT_TIME_T ); +if (sizeof (tm) ==4 ) { +replay_put_dword (tm); + }else if (sizeof (tm) ==8 ) { +replay_put_qword (tm); + }else { +fprintf (stderr,"invalid time_t sizeof: %u\n" , + (unsigned )sizeof (tm)); +exit (1 ); + } +replay_mutex_unlock (); + } +} + +/*! Reads time_t value from the log. Stops execution in case of error */ +static time_t replay_read_time_t (void ) +{ +replay_exec_instructions (); + +if (replay_file) { +time_t tm; + +replay_mutex_lock (); +skip_async_events_until (EVENT_TIME_T ); + +if (sizeof (tm) ==4 ) { + tm =replay_get_dword (); + }else if (sizeof (tm) ==8 ) { + tm =replay_get_qword (); + }else { +fprintf (stderr,"invalid time_t sizeof: %u\n" , + (unsigned )sizeof (tm)); +exit (1 ); + } + +replay_check_error (); + + replay_has_unread_data =0 ; +replay_mutex_unlock (); + +return tm; + } + +fprintf (stderr,"REPLAY INTERNAL ERROR %d\n" , __LINE__); +exit (1 ); +} + +static void replay_save_tm (struct tm *tm) +{ +replay_save_instructions (); + +if (replay_file) { +replay_mutex_lock (); +replay_put_event (EVENT_TM ); + +replay_put_dword (tm->tm_sec); +replay_put_dword (tm->tm_min); +replay_put_dword (tm->tm_hour); +replay_put_dword (tm->tm_mday); +replay_put_dword (tm->tm_mon); +replay_put_dword (tm->tm_year); +replay_put_dword (tm->tm_wday); +replay_put_dword (tm->tm_yday); +replay_put_dword (tm->tm_isdst); +replay_mutex_unlock (); + } +} + +static void replay_read_tm (struct tm *tm) +{ +replay_exec_instructions (); + +if (replay_file) { +replay_mutex_lock (); +skip_async_events_until (EVENT_TM ); + + tm->tm_sec =replay_get_dword (); + tm->tm_min =replay_get_dword (); + tm->tm_hour =replay_get_dword (); + tm->tm_mday =replay_get_dword (); + tm->tm_mon =replay_get_dword (); + tm->tm_year =replay_get_dword (); + tm->tm_wday =replay_get_dword (); + tm->tm_yday =replay_get_dword (); + tm->tm_isdst =replay_get_dword (); + +replay_check_error (); + replay_has_unread_data =0 ; +replay_mutex_unlock (); + +return ; + } + +fprintf (stderr,"REPLAY INTERNAL ERROR %d\n" , __LINE__); +exit (1 ); +} + +void replay_get_timedate (struct tm *tm) +{ +if (replay_mode ==REPLAY_MODE_RECORD ) { +qemu_get_timedate_no_warning (tm,0 ); +replay_save_tm (tm); + }else if (replay_mode ==REPLAY_MODE_PLAY ) { +replay_read_tm (tm); + }else { +qemu_get_timedate_no_warning (tm,0 ); + } +} + +time_t replay_time (void ) +{ +time_t systime; + +if (replay_mode ==REPLAY_MODE_RECORD ) { + systime =time (NULL ); +replay_save_time_t (systime); + }else if (replay_mode ==REPLAY_MODE_PLAY ) { + systime =replay_read_time_t (); + }else { + systime =time (NULL ); + } + +return systime; +}replay/replay.h +8
-0@@ -14,6 +14,7 @@#include <stdbool.h> #include <stdint.h> +#include <time.h> #include "qapi-types.h" /* replay clock kinds */ @@ -67,6 +68,13 @@ int64_t replay_read_clock(ReplayClockKind kind); : replay_mode == REPLAY_MODE_RECORD \ ? replay_save_clock((clock), (value)) \ : (value)) +/*! Returns result of time() function execution in normal and record modes. +In play mode returns value read from the log. */ +time_t replay_time (void ); +/*! This function is used for obtaining current timedate. +In record mode it also saves in into the replay log. +In replay mode it just read from the log. */ +void replay_get_timedate (struct tm *tm);/* Asynchronous events queue */ vl.c +14
-3@@ -118,6 +118,7 @@ int main(int argc, char **argv)#include "qapi/opts-visitor.h" #include "qom/object_interfaces.h" #include "qapi-event.h" +#include "replay/replay.h" #define DEFAULT_RAM_SIZE 128@@ -715,7 +716,7 @@ void vm_start(void)/***********************************************************/ /* host time/date access */ -void qemu_get_timedate (struct tm *tm,int offset) +void qemu_get_timedate_no_warning (struct tm *tm,int offset) {time_t ti;@@ -732,6 +733,16 @@ void qemu_get_timedate(struct tm *tm, int offset) } } +/* host time/date access with replay warning */ +void qemu_get_timedate (struct tm *tm,int offset) +{ +if (replay_mode ==REPLAY_MODE_RECORD ) { +fprintf (stderr,"REPLAY WARNING! qemu_get_timedate " +"function may lead to non-determinism\n" ); + } +qemu_get_timedate_no_warning (tm, offset); +} +int qemu_timedate_diff (struct tm *tm) {time_t seconds;@@ -747,7 +758,7 @@ int qemu_timedate_diff(struct tm *tm)else seconds =mktimegm (tm) + rtc_date_offset; -return seconds -time (NULL ); +return seconds -replay_time (); }static void configure_rtc_date_offset (const char *startdate,int legacy)@@ -785,7 +796,7 @@ static void configure_rtc_date_offset(const char *startdate, int legacy) "'2006-06-17T16:01:21' or '2006-06-17'\n");exit (1 ); } - rtc_date_offset =time (NULL ) - rtc_start_date; + rtc_date_offset =replay_time () - rtc_start_date; } }
Re: [Qemu-devel] [RFC PATCH v8 12/21] replay: recording and replaying different timers
Quoted 35 lines
From: Paolo Bonzini [mailto:[email protected]]
Quoted 33 lines · Pavel Dovgalyuk
This patch introduces functions for recording and replaying realtime sources, that do not use qemu-clock interface. These include return value of
time()function intime_tand struct tm forms. Patch also adds warning toget_timedatefunction to prevent its usage in recording mode, because it may lead to non-determinism.Signed-off-by Pavel Dovgalyuk <[email protected]>--- hw/timer/mc146818rtc.c | 3 + hw/timer/pl031.c | 3 + include/qemu-common.h | 1 replay/replay-internal.h | 4 + replay/replay-time.c | 132 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 8 +++ vl.c | 17 +++++- 7 files changed, 163 insertions(+), 5 deletions(-)hw/timer/mc146818rtc.c +1
-0@@ -28,6 +28,7 @@#include "qapi/visitor.h" #include "qapi-event.h" #include "qmp-commands.h" +#include "replay/replay.h" #ifdef TARGET_I386 #include "hw/i386/apic.h" @@ -703,7 +704,7 @@ static void rtc_set_date_from_host(ISADevice *dev)RTCState *s =MC146818_RTC (dev);struct tm tm;
Quoted 5 lines
hw/timer/mc146818rtc.c static void rtc_set_date_from_host(ISADevice *dev) · line 706
Quoted 1 line
hw/timer/mc146818rtc.c +0
-1-qemu_get_timedate (&tm,0 );What about just making
qemu_get_timedateuseqemu_clock_get_ns(QEMU_CLOCK_HOST)instead oftime()? This would just work using the infrastructure of the previous patch.
I can get rid of these calls, but localtime() function used in qemu_get_timedate() will not work deterministically. Should we save its' result then?
Pavel Dovgalyuk
Re: [Qemu-devel] [RFC PATCH v8 12/21] replay: recording and replaying different timers
Quoted 7 lines · Pavel Dovgaluk
Quoted 4 lines
Quoted 4 lines
What about just making
qemu_get_timedateuseqemu_clock_get_ns(QEMU_CLOCK_HOST)instead oftime()? This would just work using the infrastructure of the previous patch.I can get rid of these calls, but
localtime()function used inqemu_get_timedate()will not work deterministically. Should we save its' result then?
It is deterministic if the user's timezone remains the same. You could just outlaw "-rtc base=localtime".
Also, if you choose to record the output of qemu_get_timedate(), you have to do the same for qemu_timedate_diff(). Switching qemu_timedate_diff to use QEMU_CLOCK_HOST would also work for qemu_timedate_diff().
Paolo
RFC PATCH v8 13/21 replay: shutdown event
In reply to Pavel DovgalyukThis patch records and replays simulator shutdown event.
--- replay/replay-internal.h | 2 ++ replay/replay.c | 12 ++++++++++++ replay/replay.h | 5 +++++ vl.c | 1 + 4 files changed, 20 insertions(+), 0 deletions(-)
replay/replay-internal.h +2 -0
@@ -27,6 +27,8 @@ enum ReplayEvents {EVENT_TIME_T ,/* for tm event */ EVENT_TM , +/* for shutdown request */ +EVENT_SHUTDOWN ,/* for clock read/writes */ /* some of grteater codes are reserved for clocks */ EVENT_CLOCK
replay/replay.c +12 -0
@@ -13,6 +13,7 @@#include "replay.h" #include "replay-internal.h" #include "qemu/timer.h" +#include "sysemu/sysemu.h" ReplayMode replay_mode =REPLAY_MODE_NONE ;
@@ -34,6 +35,10 @@ bool skip_async_events(int stop_event) res = true; }switch (replay_data_kind) { +case EVENT_SHUTDOWN : + replay_has_unread_data =0 ; +qemu_system_shutdown_request (); +break ;case EVENT_INSTRUCTION : replay_state.instructions_count =replay_get_dword ();return res;
@@ -151,3 +156,10 @@ bool replay_has_interrupt(void) }return res; } + +void replay_shutdown_request (void ) +{ +if (replay_mode ==REPLAY_MODE_RECORD ) { +replay_put_event (EVENT_SHUTDOWN ); + } +}
replay/replay.h +5 -0
@@ -76,6 +76,11 @@ time_t replay_time(void);In replay mode it just read from the log. */ void replay_get_timedate (struct tm *tm); +/* Events */ + +/*! Called when qemu shutdown is requested. */ +void replay_shutdown_request (void ); +/* Asynchronous events queue */ /*! Disables storing events in the queue */
vl.c +1 -0
@@ -1720,6 +1720,7 @@ void qemu_system_killed(int signal, pid_t pid)void qemu_system_shutdown_request (void ) {trace_qemu_system_shutdown_request (); +replay_shutdown_request (); shutdown_requested =1 ;qemu_notify_event (); }
Re: [Qemu-devel] [RFC PATCH v8 13/21] replay: shutdown event
Quoted 81 lines · Pavel Dovgalyuk
This patch records and replays simulator shutdown event.
Signed-off-by Pavel Dovgalyuk <[email protected]>--- replay/replay-internal.h | 2 ++ replay/replay.c | 12 ++++++++++++ replay/replay.h | 5 +++++ vl.c | 1 + 4 files changed, 20 insertions(+), 0 deletions(-)replay/replay-internal.h +2
-0@@ -27,6 +27,8 @@ enum ReplayEvents {EVENT_TIME_T ,/* for tm event */ EVENT_TM , +/* for shutdown request */ +EVENT_SHUTDOWN ,/* for clock read/writes */ /* some of grteater codes are reserved for clocks */ EVENT_CLOCK replay/replay.c +12
-0@@ -13,6 +13,7 @@#include "replay.h" #include "replay-internal.h" #include "qemu/timer.h" +#include "sysemu/sysemu.h" ReplayMode replay_mode =REPLAY_MODE_NONE ;@@ -34,6 +35,10 @@ bool skip_async_events(int stop_event) res = true; }switch (replay_data_kind) { +case EVENT_SHUTDOWN : + replay_has_unread_data =0 ; +qemu_system_shutdown_request (); +break ;case EVENT_INSTRUCTION : replay_state.instructions_count =replay_get_dword ();return res;@@ -151,3 +156,10 @@ bool replay_has_interrupt(void) }return res; } + +void replay_shutdown_request (void ) +{ +if (replay_mode ==REPLAY_MODE_RECORD ) { +replay_put_event (EVENT_SHUTDOWN ); + } +}replay/replay.h +5
-0@@ -76,6 +76,11 @@ time_t replay_time(void);In replay mode it just read from the log. */ void replay_get_timedate (struct tm *tm); +/* Events */ + +/*! Called when qemu shutdown is requested. */ +void replay_shutdown_request (void ); +/* Asynchronous events queue */ /*! Disables storing events in the queue */ vl.c +0
-0
vl.c void qemu_system_killed(int signal, pid_t pid) · line 1726
Quoted 7 lines
vl.c +1
-0@@ -1720,6 +1720,7 @@ void qemu_system_killed(int signal, pid_t pid)void qemu_system_shutdown_request (void ) {trace_qemu_system_shutdown_request (); +replay_shutdown_request (); shutdown_requested =1 ;qemu_notify_event (); }
RFC PATCH v8 14/21 replay: checkpoints
In reply to Pavel DovgalyukThis patch introduces checkpoints that synchronize cpu thread and iothread. When checkpoint is met in the code all asynchronous events from the queue are executed.
--- block.c | 11 ++++++++++ cpus.c | 7 ++++++- include/qemu/timer.h | 6 ++++-- main-loop.c | 5 +++++ qemu-timer.c | 49 ++++++++++++++++++++++++++++++++++++++-------- replay/replay-internal.h | 5 ++++- replay/replay.c | 36 ++++++++++++++++++++++++++++++++++ replay/replay.h | 21 ++++++++++++++++++++ stubs/replay.c | 11 ++++++++++ vl.c | 4 +++- 10 files changed, 142 insertions(+), 13 deletions(-)
block.c +11 -0
@@ -1994,6 +1994,11 @@ void bdrv_drain_all(void)BlockDriverState *bs;while (busy) { +if (!replay_checkpoint (CHECKPOINT_BDRV_DRAIN )) { +/* Do not wait anymore, we stopped at some place in +the middle of execution during replay */ +return ; + } busy = false; QTAILQ_FOREACH(bs, &bdrv_states, device_list) {
@@ -2004,6 +2009,12 @@ void bdrv_drain_all(void)aio_context_release (aio_context); } } +if (replay_mode ==REPLAY_MODE_PLAY ) { +/* Skip checkpoints from the log */ +while (replay_checkpoint (CHECKPOINT_BDRV_DRAIN )) { +/* Nothing */ + } + } }/* make a BlockDriverState anonymous by removing from bdrv_state and
cpus.c +6 -1
@@ -388,7 +388,7 @@ void qtest_clock_warp(int64_t dest) timers_state.qemu_icount_bias += warp;seqlock_write_unlock (&timers_state.vm_clock_seqlock); -qemu_clock_run_timers (QEMU_CLOCK_VIRTUAL ); +qemu_clock_run_timers (QEMU_CLOCK_VIRTUAL , false); clock =qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ); }qemu_clock_notify (QEMU_CLOCK_VIRTUAL );
@@ -408,6 +408,11 @@ void qemu_clock_warp(QEMUClockType type)return ; } +/* warp clock deterministically in record/replay mode */ +if (!replay_checkpoint (CHECKPOINT_CLOCK_WARP )) { +return ; + } +/* * If the CPUs have been sleeping, advance QEMU_CLOCK_VIRTUAL timer now. * This ensures that the deadline for the timer is computed correctly below.
include/qemu/timer.h +4 -2
@@ -240,13 +240,14 @@ void qemu_clock_unregister_reset_notifier(QEMUClockType type,/** * qemu_clock_run_timers: * @type: clock on which to operate +* @run_all: true, when called from qemu_clock_run_all_timers * * Run all the timers associated with the default timer list * of a clock. * * Returns: true if any timer ran. */ -bool qemu_clock_run_timers (QEMUClockType type); +bool qemu_clock_run_timers (QEMUClockType type,bool run_all);/** * qemu_clock_run_all_timers:
@@ -337,12 +338,13 @@ QEMUClockType timerlist_get_clock(QEMUTimerList *timer_list);/** * timerlist_run_timers: * @timer_list: the timer list to use +* @run_all: true, when called from qemu_clock_run_all_timers * * Call all expired timers associated with the timer list. * * Returns: true if any timer expired */ -bool timerlist_run_timers (QEMUTimerList *timer_list); +bool timerlist_run_timers (QEMUTimerList *timer_list,bool run_all);/** * timerlist_notify:
main-loop.c +5 -0
@@ -497,6 +497,11 @@ int main_loop_wait(int nonblocking)slirp_pollfds_poll (gpollfds, (ret <0 ));#endif +/* CPU thread can infinitely wait for event after +missing the warp */ +if (replay_mode ==REPLAY_MODE_PLAY ) { +qemu_clock_warp (QEMU_CLOCK_VIRTUAL ); + }qemu_clock_run_all_timers ();return ret;
qemu-timer.c +41 -8
88 lines not on this page · raw message
replay/replay-internal.h +4 -1
@@ -31,7 +31,10 @@ enum ReplayEvents {EVENT_SHUTDOWN ,/* for clock read/writes */ /* some of grteater codes are reserved for clocks */ - EVENT_CLOCK +EVENT_CLOCK , +/* for checkpoint event */ +/* some of grteater codes are reserved for checkpoints */ + EVENT_CHECKPOINT = EVENT_CLOCK + REPLAY_CLOCK_COUNT };/* Asynchronous events IDs */
replay/replay.c +36 -0
@@ -163,3 +163,39 @@ void replay_shutdown_request(void)replay_put_event (EVENT_SHUTDOWN ); } } + +bool replay_checkpoint (ReplayCheckpoint checkpoint) +{ +bool res = false; +replay_save_instructions (); + +if (replay_file) { +if (replay_mode ==REPLAY_MODE_PLAY ) { +replay_mutex_lock (); +if (!skip_async_events (EVENT_CHECKPOINT + checkpoint)) { +if (replay_data_kind ==EVENT_ASYNC ) { +replay_read_events (checkpoint); +replay_fetch_data_kind (); + res = replay_data_kind !=EVENT_ASYNC ; +replay_mutex_unlock (); +return res; + } +replay_mutex_unlock (); +return res; + } + replay_has_unread_data =0 ; +replay_read_events (checkpoint); +replay_fetch_data_kind (); + res = replay_data_kind !=EVENT_ASYNC ; +replay_mutex_unlock (); +return res; + }else if (replay_mode ==REPLAY_MODE_RECORD ) { +replay_mutex_lock (); +replay_put_event (EVENT_CHECKPOINT + checkpoint); +replay_save_events (checkpoint); +replay_mutex_unlock (); + } + } + +return true; +}
replay/replay.h +21 -0
@@ -29,6 +29,21 @@ enum ReplayClockKind { };typedef enum ReplayClockKind ReplayClockKind ; +/* IDs of the checkpoints */ +enum ReplayCheckpoint { +CHECKPOINT_BDRV_DRAIN , +CHECKPOINT_CLOCK_WARP , +CHECKPOINT_RESET_REQUESTED , +CHECKPOINT_CLOCK_VIRTUAL , +CHECKPOINT_CLOCK_VIRTUAL_ALL , +CHECKPOINT_CLOCK_HOST , +CHECKPOINT_CLOCK_HOST_ALL , +CHECKPOINT_CLOCK_VIRTUAL_RT , +CHECKPOINT_CLOCK_VIRTUAL_RT_ALL , +CHECKPOINT_COUNT +}; +typedef enum ReplayCheckpoint ReplayCheckpoint ; +extern ReplayMode replay_mode;/* Processing the instructions */
@@ -80,6 +95,12 @@ void replay_get_timedate(struct tm *tm);/*! Called when qemu shutdown is requested. */ void replay_shutdown_request (void ); +/*! Should be called at check points in the execution. +These check points are skipped, if they were not met. +Saves checkpoint in the SAVE mode and validates in the PLAY mode. +Returns 0 in PLAY mode if checkpoint was not found. +Returns 1 in all other cases. */ +bool replay_checkpoint (ReplayCheckpoint checkpoint);/* Asynchronous events queue */
stubs/replay.c +11 -0
@@ -1,4 +1,5 @@#include "replay/replay.h" +#include "sysemu/sysemu.h" ReplayMode replay_mode;
@@ -10,3 +11,13 @@ int64_t replay_read_clock(unsigned int kind) {return 0 ; } + +bool replay_checkpoint (ReplayCheckpoint checkpoint) +{ +return 0 ; +} + +int runstate_is_running (void ) +{ +return 0 ; +}
vl.c +3 -1
@@ -1767,7 +1767,9 @@ static bool main_loop_should_exit(void)return true; } } -if (qemu_reset_requested ()) { +if (qemu_reset_requested_get () + &&replay_checkpoint (CHECKPOINT_RESET_REQUESTED )) { +qemu_reset_requested ();pause_all_vcpus ();cpu_synchronize_all_states ();qemu_system_reset (VMRESET_REPORT );
Re: [Qemu-devel] [RFC PATCH v8 14/21] replay: checkpoints
Quoted 5 lines · Pavel Dovgalyuk
This patch introduces checkpoints that synchronize cpu thread and iothread. When checkpoint is met in the code all asynchronous events from the queue are executed.
Signed-off-by Pavel Dovgalyuk <[email protected]>
I cannot understand this patch.
Please document the design in docs/ and explain why you have done the checkpoints this way.
In particular I cannot understand why you have these six checkpoints:
+ CHECKPOINT_CLOCK_VIRTUAL, + CHECKPOINT_CLOCK_VIRTUAL_ALL, + CHECKPOINT_CLOCK_HOST, + CHECKPOINT_CLOCK_HOST_ALL, + CHECKPOINT_CLOCK_VIRTUAL_RT, + CHECKPOINT_CLOCK_VIRTUAL_RT_ALL,
1) why do you need separate checkpoints for each clock?
2) why do you need separate checkpoints for _ALL and single-aio-context?
Paolo
Quoted 357 lines
--- block.c | 11 ++++++++++ cpus.c | 7 ++++++- include/qemu/timer.h | 6 ++++-- main-loop.c | 5 +++++ qemu-timer.c | 49 ++++++++++++++++++++++++++++++++++++++-------- replay/replay-internal.h | 5 ++++- replay/replay.c | 36 ++++++++++++++++++++++++++++++++++ replay/replay.h | 21 ++++++++++++++++++++ stubs/replay.c | 11 ++++++++++ vl.c | 4 +++- 10 files changed, 142 insertions(+), 13 deletions(-)block.c +11
-0@@ -1994,6 +1994,11 @@ void bdrv_drain_all(void)BlockDriverState *bs;while (busy) { +if (!replay_checkpoint (CHECKPOINT_BDRV_DRAIN )) { +/* Do not wait anymore, we stopped at some place in +the middle of execution during replay */ +return ; + } busy = false; QTAILQ_FOREACH(bs, &bdrv_states, device_list) {@@ -2004,6 +2009,12 @@ void bdrv_drain_all(void)aio_context_release (aio_context); } } +if (replay_mode ==REPLAY_MODE_PLAY ) { +/* Skip checkpoints from the log */ +while (replay_checkpoint (CHECKPOINT_BDRV_DRAIN )) { +/* Nothing */ + } + } }/* make a BlockDriverState anonymous by removing from bdrv_state and cpus.c +6
-1@@ -388,7 +388,7 @@ void qtest_clock_warp(int64_t dest) timers_state.qemu_icount_bias += warp;seqlock_write_unlock (&timers_state.vm_clock_seqlock); -qemu_clock_run_timers (QEMU_CLOCK_VIRTUAL ); +qemu_clock_run_timers (QEMU_CLOCK_VIRTUAL , false); clock =qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ); }qemu_clock_notify (QEMU_CLOCK_VIRTUAL );@@ -408,6 +408,11 @@ void qemu_clock_warp(QEMUClockType type)return ; } +/* warp clock deterministically in record/replay mode */ +if (!replay_checkpoint (CHECKPOINT_CLOCK_WARP )) { +return ; + } +/* * If the CPUs have been sleeping, advance QEMU_CLOCK_VIRTUAL timer now. * This ensures that the deadline for the timer is computed correctly below. include/qemu/timer.h +4
-2@@ -240,13 +240,14 @@ void qemu_clock_unregister_reset_notifier(QEMUClockType type,/** * qemu_clock_run_timers: * @type: clock on which to operate +* @run_all: true, when called from qemu_clock_run_all_timers * * Run all the timers associated with the default timer list * of a clock. * * Returns: true if any timer ran. */ -bool qemu_clock_run_timers (QEMUClockType type); +bool qemu_clock_run_timers (QEMUClockType type,bool run_all);/** * qemu_clock_run_all_timers: @@ -337,12 +338,13 @@ QEMUClockType timerlist_get_clock(QEMUTimerList *timer_list);/** * timerlist_run_timers: * @timer_list: the timer list to use +* @run_all: true, when called from qemu_clock_run_all_timers * * Call all expired timers associated with the timer list. * * Returns: true if any timer expired */ -bool timerlist_run_timers (QEMUTimerList *timer_list); +bool timerlist_run_timers (QEMUTimerList *timer_list,bool run_all);/** * timerlist_notify: main-loop.c +5
-0@@ -497,6 +497,11 @@ int main_loop_wait(int nonblocking)slirp_pollfds_poll (gpollfds, (ret <0 ));#endif +/* CPU thread can infinitely wait for event after +missing the warp */ +if (replay_mode ==REPLAY_MODE_PLAY ) { +qemu_clock_warp (QEMU_CLOCK_VIRTUAL ); + }qemu_clock_run_all_timers ();return ret;qemu-timer.c +41
-8@@ -465,7 +465,7 @@ bool timer_expired(QEMUTimer *timer_head, int64_t current_time)return timer_expired_ns (timer_head, current_time * timer_head->scale); } -bool timerlist_run_timers (QEMUTimerList *timer_list) +bool timerlist_run_timers (QEMUTimerList *timer_list,bool run_all) {QEMUTimer *ts;int64_t current_time;@@ -473,6 +473,32 @@ bool timerlist_run_timers(QEMUTimerList *timer_list)QEMUTimerCB *cb;void *opaque; +switch (timer_list->clock->type) { +case QEMU_CLOCK_REALTIME : +break ; +default : +case QEMU_CLOCK_VIRTUAL : +if ((replay_mode !=REPLAY_MODE_NONE && !runstate_is_running ()) + || !replay_checkpoint (run_all ?CHECKPOINT_CLOCK_VIRTUAL_ALL + :CHECKPOINT_CLOCK_VIRTUAL )) { +return false; + } +break ; +case QEMU_CLOCK_HOST : +if ((replay_mode !=REPLAY_MODE_NONE && !runstate_is_running ()) + || !replay_checkpoint (run_all ?CHECKPOINT_CLOCK_HOST_ALL + :CHECKPOINT_CLOCK_HOST )) { +return false; + } +case QEMU_CLOCK_VIRTUAL_RT : +if ((replay_mode !=REPLAY_MODE_NONE && !runstate_is_running ()) + || !replay_checkpoint (run_all ?CHECKPOINT_CLOCK_VIRTUAL_RT_ALL + :CHECKPOINT_CLOCK_VIRTUAL_RT )) { +return false; + } +break ; + } +qemu_event_reset (&timer_list->timers_done_ev);if (!timer_list->clock->enabled) { goto out;@@ -505,9 +531,9 @@ out:return progress; } -bool qemu_clock_run_timers (QEMUClockType type) +bool qemu_clock_run_timers (QEMUClockType type,bool run_all) { -return timerlist_run_timers (main_loop_tlg.tl[type]); +return timerlist_run_timers (main_loop_tlg.tl[type], run_all); } void timerlistgroup_init(QEMUTimerListGroup *tlg,@@ -532,7 +558,7 @@ bool timerlistgroup_run_timers(QEMUTimerListGroup *tlg)QEMUClockType type;bool progress = false;for (type =0 ; type <QEMU_CLOCK_MAX ; type++) { - progress |=timerlist_run_timers (tlg->tl[type]); + progress |=timerlist_run_timers (tlg->tl[type], false); }return progress; }@@ -541,11 +567,18 @@ int64_t timerlistgroup_deadline_ns(QEMUTimerListGroup *tlg) {int64_t deadline =-1 ;QEMUClockType type; +bool play = replay_mode ==REPLAY_MODE_PLAY ;for (type =0 ; type <QEMU_CLOCK_MAX ; type++) {if (qemu_clock_use_for_deadline (tlg->tl[type]->clock->type)) { - deadline =qemu_soonest_timeout (deadline, -timerlist_deadline_ns ( - tlg->tl[type])); +if (!play || tlg->tl[type]->clock->type ==QEMU_CLOCK_REALTIME ) { + deadline =qemu_soonest_timeout (deadline, +timerlist_deadline_ns ( + tlg->tl[type])); + }else { +/* Read clock from the replay file and +do not calculate the deadline, based on virtual clock. */ +qemu_clock_get_ns (tlg->tl[type]->clock->type); + } } }return deadline;@@ -615,7 +648,7 @@ bool qemu_clock_run_all_timers(void)QEMUClockType type;for (type =0 ; type <QEMU_CLOCK_MAX ; type++) { - progress |=qemu_clock_run_timers (type); + progress |=qemu_clock_run_timers (type, true); }return progress;replay/replay-internal.h +4
-1@@ -31,7 +31,10 @@ enum ReplayEvents {EVENT_SHUTDOWN ,/* for clock read/writes */ /* some of grteater codes are reserved for clocks */ - EVENT_CLOCK +EVENT_CLOCK , +/* for checkpoint event */ +/* some of grteater codes are reserved for checkpoints */ + EVENT_CHECKPOINT = EVENT_CLOCK + REPLAY_CLOCK_COUNT };/* Asynchronous events IDs */ replay/replay.c +36
-0@@ -163,3 +163,39 @@ void replay_shutdown_request(void)replay_put_event (EVENT_SHUTDOWN ); } } + +bool replay_checkpoint (ReplayCheckpoint checkpoint) +{ +bool res = false; +replay_save_instructions (); + +if (replay_file) { +if (replay_mode ==REPLAY_MODE_PLAY ) { +replay_mutex_lock (); +if (!skip_async_events (EVENT_CHECKPOINT + checkpoint)) { +if (replay_data_kind ==EVENT_ASYNC ) { +replay_read_events (checkpoint); +replay_fetch_data_kind (); + res = replay_data_kind !=EVENT_ASYNC ; +replay_mutex_unlock (); +return res; + } +replay_mutex_unlock (); +return res; + } + replay_has_unread_data =0 ; +replay_read_events (checkpoint); +replay_fetch_data_kind (); + res = replay_data_kind !=EVENT_ASYNC ; +replay_mutex_unlock (); +return res; + }else if (replay_mode ==REPLAY_MODE_RECORD ) { +replay_mutex_lock (); +replay_put_event (EVENT_CHECKPOINT + checkpoint); +replay_save_events (checkpoint); +replay_mutex_unlock (); + } + } + +return true; +}replay/replay.h +21
-0@@ -29,6 +29,21 @@ enum ReplayClockKind { };typedef enum ReplayClockKind ReplayClockKind ; +/* IDs of the checkpoints */ +enum ReplayCheckpoint { +CHECKPOINT_BDRV_DRAIN , +CHECKPOINT_CLOCK_WARP , +CHECKPOINT_RESET_REQUESTED , +CHECKPOINT_CLOCK_VIRTUAL , +CHECKPOINT_CLOCK_VIRTUAL_ALL , +CHECKPOINT_CLOCK_HOST , +CHECKPOINT_CLOCK_HOST_ALL , +CHECKPOINT_CLOCK_VIRTUAL_RT , +CHECKPOINT_CLOCK_VIRTUAL_RT_ALL , +CHECKPOINT_COUNT +}; +typedef enum ReplayCheckpoint ReplayCheckpoint ; +extern ReplayMode replay_mode;/* Processing the instructions */ @@ -80,6 +95,12 @@ void replay_get_timedate(struct tm *tm);/*! Called when qemu shutdown is requested. */ void replay_shutdown_request (void ); +/*! Should be called at check points in the execution. +These check points are skipped, if they were not met. +Saves checkpoint in the SAVE mode and validates in the PLAY mode. +Returns 0 in PLAY mode if checkpoint was not found. +Returns 1 in all other cases. */ +bool replay_checkpoint (ReplayCheckpoint checkpoint);/* Asynchronous events queue */ stubs/replay.c +11
-0@@ -1,4 +1,5 @@#include "replay/replay.h" +#include "sysemu/sysemu.h" ReplayMode replay_mode;@@ -10,3 +11,13 @@ int64_t replay_read_clock(unsigned int kind) {return 0 ; } + +bool replay_checkpoint (ReplayCheckpoint checkpoint) +{ +return 0 ; +} + +int runstate_is_running (void ) +{ +return 0 ; +}vl.c +3
-1@@ -1767,7 +1767,9 @@ static bool main_loop_should_exit(void)return true; } } -if (qemu_reset_requested ()) { +if (qemu_reset_requested_get () + &&replay_checkpoint (CHECKPOINT_RESET_REQUESTED )) { +qemu_reset_requested ();pause_all_vcpus ();cpu_synchronize_all_states ();qemu_system_reset (VMRESET_REPORT );
RFC PATCH v8 15/21 aio: replace stack of bottom halves with queue
In reply to Pavel DovgalyukBottom halves in AIO context are stored and removes in LIFO order. It makes their execution non-deterministic. This patch replaces the stack with queue to preserve the order of bottom halves processing.
--- async.c | 25 +++++++++---------------- include/block/aio.h | 4 ++-- include/qemu/queue.h | 7 +++++++ 3 files changed, 18 insertions(+), 18 deletions(-)
async.c +9 -16
@@ -35,7 +35,7 @@ struct QEMUBH {AioContext *ctx;QEMUBHFunc *cb;void *opaque; -QEMUBH *next; +QSIMPLEQ_ENTRY (QEMUBH) next;bool scheduled;bool idle;bool deleted;
@@ -51,10 +51,7 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque) .opaque = opaque, };qemu_mutex_lock (&ctx->bh_lock); - bh->next = ctx->first_bh; -/* Make sure that the members are ready before putting bh into list */ -smp_wmb (); - ctx->first_bh = bh; +QSIMPLEQ_INSERT_TAIL_RCU (&ctx->bh_queue, bh, next);qemu_mutex_unlock (&ctx->bh_lock);return bh; }
@@ -62,16 +59,15 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque)/* Multiple occurrences of aio_bh_poll cannot be called concurrently */ int aio_bh_poll (AioContext *ctx) { -QEMUBH *bh, **bhp, *next; +QEMUBH *bh, *next;int ret; ctx->walking_bh++; ret =0 ; -for (bh = ctx->first_bh; bh; bh = next) { + QSIMPLEQ_FOREACH_SAFE(bh, &ctx->bh_queue, next, next) {/* Make sure that fetching bh happens before accessing its members */ smp_read_barrier_depends (); - next = bh->next;if (!bh->deleted && bh->scheduled) { bh->scheduled =0 ;/* Paired with write barrier in bh schedule to ensure reading for
@@ -90,14 +86,10 @@ int aio_bh_poll(AioContext *ctx)/* remove deleted bhs */ if (!ctx->walking_bh) {qemu_mutex_lock (&ctx->bh_lock); - bhp = &ctx->first_bh; -while (*bhp) { - bh = *bhp; + QSIMPLEQ_FOREACH_SAFE(bh, &ctx->bh_queue, next, next) {if (bh->deleted) { - *bhp = bh->next; +QSIMPLEQ_REMOVE (&ctx->bh_queue, bh,QEMUBH , next);g_free (bh); - }else { - bhp = &bh->next; } }qemu_mutex_unlock (&ctx->bh_lock);
@@ -161,7 +153,7 @@ aio_compute_timeout(AioContext *ctx)int timeout =-1 ;QEMUBH *bh; -for (bh = ctx->first_bh; bh; bh = bh->next) { + QSIMPLEQ_FOREACH(bh, &ctx->bh_queue, next) {if (!bh->deleted && bh->scheduled) {if (bh->idle) {/* idle bottom halves will be polled at least
@@ -204,7 +196,7 @@ aio_ctx_check(GSource *source)AioContext *ctx = (AioContext *) source;QEMUBH *bh; -for (bh = ctx->first_bh; bh; bh = bh->next) { + QSIMPLEQ_FOREACH(bh, &ctx->bh_queue, next) {if (!bh->deleted && bh->scheduled) {return true; }
@@ -311,6 +303,7 @@ AioContext *aio_context_new(Error **errp)qemu_mutex_init (&ctx->bh_lock);rfifolock_init (&ctx->lock, aio_rfifolock_cb, ctx);timerlistgroup_init (&ctx->tlg, aio_timerlist_notify, ctx); +QSIMPLEQ_INIT (&ctx->bh_queue);return ctx; }
include/block/aio.h +2 -2
@@ -71,8 +71,8 @@ struct AioContext {/* lock to protect between bh's adders and deleter */ QemuMutex bh_lock; -/* Anchor of the list of Bottom Halves belonging to the context */ -struct QEMUBH *first_bh; +/* List of Bottom Halves belonging to the context */ + QSIMPLEQ_HEAD(, QEMUBH) bh_queue;/* A simple lock used to protect the first_bh list, and ensure that * no callbacks are removed while we're walking and dispatching callbacks.
include/qemu/queue.h +7 -0
@@ -268,6 +268,13 @@ struct { \ (head)->sqh_last = &(elm)->field.sqe_next; \ } while (/*CONSTCOND*/0) +#define QSIMPLEQ_INSERT_TAIL_RCU(head, elm, field) do { \ + (elm)->field.sqe_next = NULL; \ + *(head)->sqh_last = (elm); \ + (head)->sqh_last = &(elm)->field.sqe_next; \ + smp_wmb(); \ +} while (/*CONSTCOND*/0) +#define QSIMPLEQ_INSERT_AFTER (head, listelm, elm, field) do { \ if (((elm)->field.sqe_next = (listelm)->field.sqe_next) == NULL) \ (head)->sqh_last = &(elm)->field.sqe_next; \
Re: [Qemu-devel] [RFC PATCH v8 15/21] aio: replace stack of bottom halves with queue
Quoted 43 lines · Pavel Dovgalyuk
Bottom halves in AIO context are stored and removes in LIFO order. It makes their execution non-deterministic. This patch replaces the stack with queue to preserve the order of bottom halves processing.
Signed-off-by Pavel Dovgalyuk <[email protected]>--- async.c | 25 +++++++++---------------- include/block/aio.h | 4 ++-- include/qemu/queue.h | 7 +++++++ 3 files changed, 18 insertions(+), 18 deletions(-)async.c +3
-6@@ -35,7 +35,7 @@ struct QEMUBH {AioContext *ctx;QEMUBHFunc *cb;void *opaque; -QEMUBH *next; +QSIMPLEQ_ENTRY (QEMUBH) next;bool scheduled;bool idle;bool deleted;@@ -51,10 +51,7 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque) .opaque = opaque, };qemu_mutex_lock (&ctx->bh_lock); - bh->next = ctx->first_bh; -/* Make sure that the members are ready before putting bh into list */ -smp_wmb (); - ctx->first_bh = bh; +QSIMPLEQ_INSERT_TAIL_RCU (&ctx->bh_queue, bh, next);qemu_mutex_unlock (&ctx->bh_lock);return bh; }@@ -62,16 +59,15 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque)/* Multiple occurrences of aio_bh_poll cannot be called concurrently */ int aio_bh_poll (AioContext *ctx) { -QEMUBH *bh, **bhp, *next; +QEMUBH *bh, *next;
async.c QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque) · line 74
Quoted 10 lines
async.c +1
-2int ret; ctx->walking_bh++; ret =0 ; -for (bh = ctx->first_bh; bh; bh = next) { + QSIMPLEQ_FOREACH_SAFE(bh, &ctx->bh_queue, next, next) {/* Make sure that fetching bh happens before accessing its members */ smp_read_barrier_depends (); - next = bh->next;
Must use QSIMPLEQ_FOREACH. Otherwise, the access of next is before the smp_read_barrier_depends().
Quoted 66 lines
async.c +5
-8if (!bh->deleted && bh->scheduled) { bh->scheduled =0 ;/* Paired with write barrier in bh schedule to ensure reading for @@ -90,14 +86,10 @@ int aio_bh_poll(AioContext *ctx)/* remove deleted bhs */ if (!ctx->walking_bh) {qemu_mutex_lock (&ctx->bh_lock); - bhp = &ctx->first_bh; -while (*bhp) { - bh = *bhp; + QSIMPLEQ_FOREACH_SAFE(bh, &ctx->bh_queue, next, next) {if (bh->deleted) { - *bhp = bh->next; +QSIMPLEQ_REMOVE (&ctx->bh_queue, bh,QEMUBH , next);g_free (bh); - }else { - bhp = &bh->next; } }qemu_mutex_unlock (&ctx->bh_lock);@@ -161,7 +153,7 @@ aio_compute_timeout(AioContext *ctx)int timeout =-1 ;QEMUBH *bh; -for (bh = ctx->first_bh; bh; bh = bh->next) { + QSIMPLEQ_FOREACH(bh, &ctx->bh_queue, next) {if (!bh->deleted && bh->scheduled) {if (bh->idle) {/* idle bottom halves will be polled at least @@ -204,7 +196,7 @@ aio_ctx_check(GSource *source)AioContext *ctx = (AioContext *) source;QEMUBH *bh; -for (bh = ctx->first_bh; bh; bh = bh->next) { + QSIMPLEQ_FOREACH(bh, &ctx->bh_queue, next) {if (!bh->deleted && bh->scheduled) {return true; }@@ -311,6 +303,7 @@ AioContext *aio_context_new(Error **errp)qemu_mutex_init (&ctx->bh_lock);rfifolock_init (&ctx->lock, aio_rfifolock_cb, ctx);timerlistgroup_init (&ctx->tlg, aio_timerlist_notify, ctx); +QSIMPLEQ_INIT (&ctx->bh_queue);return ctx; }include/block/aio.h +2
-2@@ -71,8 +71,8 @@ struct AioContext {/* lock to protect between bh's adders and deleter */ QemuMutex bh_lock; -/* Anchor of the list of Bottom Halves belonging to the context */ -struct QEMUBH *first_bh; +/* List of Bottom Halves belonging to the context */ + QSIMPLEQ_HEAD(, QEMUBH) bh_queue;/* A simple lock used to protect the first_bh list, and ensure that * no callbacks are removed while we're walking and dispatching callbacks. include/qemu/queue.h +0
-0
include/qemu/queue.h struct { \ · line 275
Quoted 8 lines
include/qemu/queue.h +5
-0@@ -268,6 +268,13 @@ struct { \ (head)->sqh_last = &(elm)->field.sqe_next; \ } while (/*CONSTCOND*/0) +#define QSIMPLEQ_INSERT_TAIL_RCU (head, elm, field) do { \ + (elm)->field.sqe_next = NULL; \ + *(head)->sqh_last = (elm); \ + (head)->sqh_last = &(elm)->field.sqe_next; \ + smp_wmb(); \
smp_wmb() must be after the (elm) assignment.
Paolo
Quoted 6 lines
include/qemu/queue.h +2
-0+} while (/*CONSTCOND*/0) +#define QSIMPLEQ_INSERT_AFTER (head, listelm, elm, field) do { \ if (((elm)->field.sqe_next = (listelm)->field.sqe_next) == NULL) \ (head)->sqh_last = &(elm)->field.sqe_next; \
RFC PATCH v8 16/21 replay: bottom halves
In reply to Pavel DovgalyukThis patch introduces bottom half event for replay queue. It saves the events into the queue and process them at the checkpoints and instructions execution.
--- async.c | 24 +++++++++++++++++++++++- dma-helpers.c | 4 +++- hw/ide/ahci.c | 4 +++- hw/ide/core.c | 4 +++- hw/timer/arm_timer.c | 2 +- hw/usb/hcd-uhci.c | 2 +- include/block/aio.h | 18 ++++++++++++++++++ include/qemu/main-loop.h | 1 + main-loop.c | 5 +++++ replay/replay-events.c | 16 ++++++++++++++++ replay/replay-internal.h | 1 + replay/replay.h | 2 ++ stubs/replay.c | 4 ++++ 13 files changed, 81 insertions(+), 6 deletions(-)
async.c +23 -1
@@ -27,6 +27,7 @@#include "block/thread-pool.h" #include "qemu/main-loop.h" #include "qemu/atomic.h" +#include "replay/replay.h" /***********************************************************/ /* bottom halves (can be seen as timers which expire ASAP) */
@@ -39,6 +40,8 @@ struct QEMUBH {bool scheduled;bool idle;bool deleted; +bool replay; +uint64_t id; };QEMUBH *aio_bh_new (AioContext *ctx,QEMUBHFunc *cb,void *opaque)
@@ -56,6 +59,21 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque)return bh; } +QEMUBH *aio_bh_new_replay (AioContext *ctx,QEMUBHFunc *cb,void *opaque, +uint64_t id) +{ +QEMUBH *bh =aio_bh_new (ctx, cb, opaque); + bh->replay = true; + bh->id = id; +return bh; +} + +void aio_bh_call (void *opaque) +{ +QEMUBH *bh = (QEMUBH *)opaque; + bh->cb (bh->opaque); +} +/* Multiple occurrences of aio_bh_poll cannot be called concurrently */ int aio_bh_poll (AioContext *ctx) {
@@ -77,7 +95,11 @@ int aio_bh_poll(AioContext *ctx)if (!bh->idle) ret =1 ; bh->idle =0 ; - bh->cb (bh->opaque); +if (!bh->replay) { +aio_bh_call (bh); + }else { +replay_add_bh_event (bh, bh->id); + } } }
dma-helpers.c +3 -1
@@ -13,6 +13,7 @@#include "qemu/range.h" #include "qemu/thread.h" #include "qemu/main-loop.h" +#include "replay/replay.h" /* #define DEBUG_IOMMU */
@@ -96,7 +97,8 @@ static void continue_after_map_failure(void *opaque) {DMAAIOCB *dbs = (DMAAIOCB *)opaque; - dbs->bh =qemu_bh_new (reschedule_dma, dbs); + dbs->bh =qemu_bh_new_replay (reschedule_dma, dbs, +replay_get_current_step ());qemu_bh_schedule (dbs->bh); }
hw/ide/ahci.c +3 -1
@@ -33,6 +33,7 @@#include "internal.h" #include <hw/ide/pci.h> #include <hw/ide/ahci.h> +#include "replay/replay.h" #define DEBUG_AHCI 0
@@ -1243,7 +1244,8 @@ static void ahci_cmd_done(IDEDMA *dma)if (!ad->check_bh) {/* maybe we still have something to process, check later */ - ad->check_bh =qemu_bh_new (ahci_check_cmd_bh, ad); + ad->check_bh =qemu_bh_new_replay (ahci_check_cmd_bh, ad, +replay_get_current_step ());qemu_bh_schedule (ad->check_bh); } }
hw/ide/core.c +3 -1
@@ -32,6 +32,7 @@#include "sysemu/dma.h" #include "hw/block/block.h" #include "sysemu/block-backend.h" +#include "replay/replay.h" #include <hw/ide/internal.h>
@@ -448,7 +449,8 @@ BlockAIOCB *ide_issue_trim(BlockBackend *blk, iocb =blk_aio_get (&trim_aiocb_info, blk, cb, opaque); iocb->blk = blk; - iocb->bh =qemu_bh_new (ide_trim_bh_cb, iocb); + iocb->bh =qemu_bh_new_replay (ide_trim_bh_cb, iocb, +replay_get_current_step ()); iocb->ret =0 ; iocb->qiov = qiov; iocb->i =-1 ;
hw/timer/arm_timer.c +1 -1
@@ -168,7 +168,7 @@ static arm_timer_state *arm_timer_init(uint32_t freq) s->freq = freq; s->control =TIMER_CTRL_IE ; - bh =qemu_bh_new (arm_timer_tick, s); + bh =qemu_bh_new_replay (arm_timer_tick, s,0 ); s->timer =ptimer_init (bh);vmstate_register (NULL ,-1 , &vmstate_arm_timer, s);return s;
hw/usb/hcd-uhci.c +1 -1
@@ -1221,7 +1221,7 @@ static int usb_uhci_common_initfn(PCIDevice *dev) USB_SPEED_MASK_LOW | USB_SPEED_MASK_FULL); } } - s->bh =qemu_bh_new (uhci_bh, s); + s->bh =qemu_bh_new_replay (uhci_bh, s,0 ); s->frame_timer =timer_new_ns (QEMU_CLOCK_VIRTUAL , uhci_frame_timer, s); s->num_ports_vmstate =NB_PORTS ;QTAILQ_INIT (&s->queues);
include/block/aio.h +18 -0
@@ -35,6 +35,8 @@ struct BlockAIOCB {const AIOCBInfo *aiocb_info;BlockDriverState *bs;BlockCompletionFunc *cb; +bool replay; +uint64_t replay_step;void *opaque;int refcnt; };
@@ -144,6 +146,17 @@ void aio_context_release(AioContext *ctx);QEMUBH *aio_bh_new (AioContext *ctx,QEMUBHFunc *cb,void *opaque);/** +* aio_bh_new_replay: Allocate a new bottom half structure for replay. +* +* This function calls aio_bh_new function and also fills replay parameters +* of the BH structure. BH created with this function in record/replay mode +* are executed through the replay queue only at checkpoints and instructions +* executions. +*/ +QEMUBH *aio_bh_new_replay (AioContext *ctx,QEMUBHFunc *cb,void *opaque, +uint64_t id); + +/** * aio_notify: Force processing of pending events. * * Similar to signaling a condition variable, aio_notify forces
@@ -159,6 +172,11 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque);void aio_notify (AioContext *ctx);/** +* aio_bh_call: Executes callback function of the specified BH. +*/ +void aio_bh_call (void *opaque); + +/** * aio_bh_poll: Poll bottom halves for an AioContext. * * These are internal functions used by the QEMU main loop.
include/qemu/main-loop.h +1 -0
@@ -306,6 +306,7 @@ void qemu_iohandler_fill(GArray *pollfds);void qemu_iohandler_poll (GArray *pollfds,int rc);QEMUBH *qemu_bh_new (QEMUBHFunc *cb,void *opaque); +QEMUBH *qemu_bh_new_replay (QEMUBHFunc *cb,void *opaque,uint64_t id);void qemu_bh_schedule_idle (QEMUBH *bh);#endif
main-loop.c +5 -0
@@ -513,3 +513,8 @@ QEMUBH *qemu_bh_new(QEMUBHFunc *cb, void *opaque) {return aio_bh_new (qemu_aio_context, cb, opaque); } + +QEMUBH *qemu_bh_new_replay (QEMUBHFunc *cb,void *opaque,uint64_t id) +{ +return aio_bh_new_replay (qemu_aio_context, cb, opaque, id); +}
replay/replay-events.c +16 -0
@@ -36,6 +36,9 @@ static bool replay_events_enabled = false;static void replay_run_event (Event *event) {switch (event->event_kind) { +case REPLAY_ASYNC_EVENT_BH : +aio_bh_call (event->opaque); +break ;default :fprintf (stderr,"Replay: invalid async event ID (%d) in the queue\n" , event->event_kind);
@@ -122,6 +125,11 @@ void replay_add_event(ReplayAsyncEventKind event_kind, void *opaque)replay_add_event_internal (event_kind, opaque,NULL ,0 ); } +void replay_add_bh_event (void *bh,uint64_t id) +{ +replay_add_event_internal (REPLAY_ASYNC_EVENT_BH , bh,NULL , id); +} +/* Called with replay mutex locked */ void replay_save_events (int opt) {
@@ -136,6 +144,9 @@ void replay_save_events(int opt)/* save event-specific data */ switch (event->event_kind) { +case REPLAY_ASYNC_EVENT_BH : +replay_put_qword (event->id); +break ; } }
@@ -167,6 +178,11 @@ void replay_read_events(int opt) }/* Execute some events without searching them in the queue */ switch (read_event_kind) { +case REPLAY_ASYNC_EVENT_BH : +if (read_id ==-1 ) { + read_id =replay_get_qword (); + } +break ;default :fprintf (stderr,"Unknown ID %d of replay event\n" , read_event_kind);exit (1 );
replay/replay-internal.h +1 -0
@@ -40,6 +40,7 @@ enum ReplayEvents {/* Asynchronous events IDs */ enum ReplayAsyncEventKind { +REPLAY_ASYNC_EVENT_BH ,REPLAY_ASYNC_COUNT };
replay/replay.h +2 -0
@@ -106,5 +106,7 @@ bool replay_checkpoint(ReplayCheckpoint checkpoint);/*! Disables storing events in the queue */ void replay_disable_events (void ); +/*! Adds BH event to the queue */ +void replay_add_bh_event (void *bh,uint64_t id);#endif
stubs/replay.c +4 -0
@@ -21,3 +21,7 @@ int runstate_is_running(void) {return 0 ; } + +void replay_add_bh_event (void *bh,uint64_t id) +{ +}
Re: [Qemu-devel] [RFC PATCH v8 16/21] replay: bottom halves
Quoted 2 lines · Pavel Dovgalyuk
This patch introduces bottom half event for replay queue. It saves the events into the queue and process them at the checkpoints and instructions execution.
Which bottom halves must _not_ go through aio/qemu_bh_new_replay?
async.c QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque) · line 63
Quoted 2 lines
async.c +2
-0+QEMUBH *aio_bh_new_replay (AioContext *ctx,QEMUBHFunc *cb,void *opaque, +uint64_t id)
id is superfluous, it can always be replay_get_current_step().
It seems to me that the device models can always use qemu_bh_new_replay.
There are few if any uses of qemu_bh_new outside device models.
Perhaps convert those to aio_bh_new, and make qemu_bh_new always do special replay treatment?
async.c QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque) · line 69
Quoted 6 lines
async.c +6
-0+{ +QEMUBH *bh =aio_bh_new (ctx, cb, opaque); + bh->replay = true; + bh->id = id; +return bh; +}
Paolo
Re: [Qemu-devel] [RFC PATCH v8 16/21] replay: bottom halves
Quoted 6 lines
From: Paolo Bonzini [mailto:[email protected]]
Quoted 2 lines · Pavel Dovgalyuk
This patch introduces bottom half event for replay queue. It saves the events into the queue and process them at the checkpoints and instructions execution.
Which bottom halves must _not_ go through aio/qemu_bh_new_replay?
The block layer and migration internal ones.
They should not be replayed because are used for simulator purposes and not for VM execution.
Quoted 10 lines
async.c QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque) · line 63
Quoted 3 lines
async.c +2
-0+QEMUBH *aio_bh_new_replay (AioContext *ctx,QEMUBHFunc *cb,void *opaque, +uint64_t id)id is superfluous, it can always be
replay_get_current_step().It seems to me that the device models can always use
qemu_bh_new_replay.
There are few if any uses ofqemu_bh_newoutside device models.
Perhaps convert those toaio_bh_new, and makeqemu_bh_newalways do special replay treatment?
I've created some kind of design document.
Then I'll probably leave these changes until that document review.
Pavel Dovgalyuk
RFC PATCH v8 17/21 replay: replay aio requests
In reply to Pavel DovgalyukThis patch adds identifier to aio requests. ID is used for creating bottom halves and identifying them while replaying.
The patch also introduces several functions that make possible replaying of the aio requests.
--- block.c | 81 ++++++++++++++++++++++++++++++++++++---- block/block-backend.c | 30 ++++++++++++++- block/qcow2.c | 4 ++ dma-helpers.c | 6 ++- hw/block/virtio-blk.c | 10 ++--- hw/ide/atapi.c | 10 +++-- hw/ide/core.c | 14 ++++--- include/block/block.h | 15 +++++++ include/qemu-common.h | 2 + include/sysemu/block-backend.h | 10 +++++ qemu-io-cmds.c | 2 - stubs/replay.c | 5 ++ trace-events | 2 + util/iov.c | 4 ++ 14 files changed, 167 insertions(+), 28 deletions(-)
block.c +72 -9
@@ -83,7 +83,8 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,BdrvRequestFlags flags, BlockCompletionFunc *cb, void *opaque, - bool is_write); + bool is_write, + bool aio_replay); static void coroutine_fn bdrv_co_do_rw(void *opaque); static int coroutine_fn bdrv_co_do_write_zeroes(BlockDriverState *bs,int64_t sector_num,int nb_sectors,BdrvRequestFlags flags);
@@ -4425,7 +4426,19 @@ BlockAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,trace_bdrv_aio_readv (bs, sector_num, nb_sectors, opaque);return bdrv_co_aio_rw_vector (bs, sector_num, qiov, nb_sectors,0 , - cb, opaque, false); + cb, opaque, false, false); +} + +BlockAIOCB *bdrv_aio_readv_replay (BlockDriverState *bs, +int64_t sector_num, +QEMUIOVector *qiov,int nb_sectors, +BlockCompletionFunc *cb, +void *opaque) +{ +trace_bdrv_aio_readv_replay (bs, sector_num, nb_sectors, opaque); + +return bdrv_co_aio_rw_vector (bs, sector_num, qiov, nb_sectors,0 , + cb, opaque, false, true); } BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
@@ -4435,7 +4448,19 @@ BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,trace_bdrv_aio_writev (bs, sector_num, nb_sectors, opaque);return bdrv_co_aio_rw_vector (bs, sector_num, qiov, nb_sectors,0 , - cb, opaque, true); + cb, opaque, true, false); +} + +BlockAIOCB *bdrv_aio_writev_replay (BlockDriverState *bs, +int64_t sector_num, +QEMUIOVector *qiov,int nb_sectors, +BlockCompletionFunc *cb, +void *opaque) +{ +trace_bdrv_aio_writev_replay (bs, sector_num, nb_sectors, opaque); + +return bdrv_co_aio_rw_vector (bs, sector_num, qiov, nb_sectors,0 , + cb, opaque, true, true); } BlockAIOCB *bdrv_aio_write_zeroes(BlockDriverState *bs,
@@ -4446,7 +4471,7 @@ BlockAIOCB *bdrv_aio_write_zeroes(BlockDriverState *bs,return bdrv_co_aio_rw_vector (bs, sector_num,NULL , nb_sectors,BDRV_REQ_ZERO_WRITE | flags, - cb, opaque, true); + cb, opaque, true, true); }
@@ -4593,7 +4618,8 @@ static int multiwrite_merge(BlockDriverState *bs, BlockRequest *reqs,* requests. However, the fields opaque and error are left unmodified as they * are used to signal failure for a single request to the caller. */ -int bdrv_aio_multiwrite (BlockDriverState *bs,BlockRequest *reqs,int num_reqs) +int bdrv_aio_multiwrite (BlockDriverState *bs,BlockRequest *reqs,int num_reqs, +bool replay) {MultiwriteCB *mcb;int i;
@@ -4631,7 +4657,7 @@ int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs)bdrv_co_aio_rw_vector (bs, reqs[i].sector, reqs[i].qiov, reqs[i].nb_sectors, reqs[i].flags, multiwrite_cb, mcb, - true); + true, replay); }return 0 ;
@@ -4776,7 +4802,12 @@ static void coroutine_fn bdrv_co_do_rw(void *opaque) acb->req.nb_sectors, acb->req.qiov, acb->req.flags); } - acb->bh =aio_bh_new (bdrv_get_aio_context (bs), bdrv_co_em_bh, acb); +if (acb->common.replay) { + acb->bh =aio_bh_new_replay (bdrv_get_aio_context (bs), bdrv_co_em_bh, + acb, acb->common.replay_step); + }else { + acb->bh =aio_bh_new (bdrv_get_aio_context (bs), bdrv_co_em_bh, acb); + }qemu_bh_schedule (acb->bh); }
@@ -4787,7 +4818,8 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,BdrvRequestFlags flags, BlockCompletionFunc *cb, void *opaque, - bool is_write) + bool is_write, + bool aio_replay) { Coroutine *co;BlockAIOCBCoroutine *acb;
@@ -4798,6 +4830,11 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs, acb->req.qiov = qiov; acb->req.flags = flags; acb->is_write = is_write; + acb->done =NULL ; + acb->common.replay = aio_replay; +if (aio_replay) { + acb->common.replay_step =replay_get_current_step (); + } co =qemu_coroutine_create (bdrv_co_do_rw);qemu_coroutine_enter (co, acb);
@@ -4811,7 +4848,12 @@ static void coroutine_fn bdrv_aio_flush_co_entry(void *opaque)BlockDriverState *bs = acb->common.bs; acb->req.error =bdrv_co_flush (bs); - acb->bh =aio_bh_new (bdrv_get_aio_context (bs), bdrv_co_em_bh, acb); +if (acb->common.replay) { + acb->bh =aio_bh_new_replay (bdrv_get_aio_context (bs), bdrv_co_em_bh, + acb, acb->common.replay_step); + }else { + acb->bh =aio_bh_new (bdrv_get_aio_context (bs), bdrv_co_em_bh, acb); + }qemu_bh_schedule (acb->bh); }
@@ -4831,6 +4873,25 @@ BlockAIOCB *bdrv_aio_flush(BlockDriverState *bs,return &acb->common; } +BlockAIOCB *bdrv_aio_flush_replay (BlockDriverState *bs, +BlockCompletionFunc *cb,void *opaque) +{ +trace_bdrv_aio_flush (bs, opaque); + +Coroutine *co; +BlockAIOCBCoroutine *acb; + + acb =qemu_aio_get (&bdrv_em_co_aiocb_info, bs, cb, opaque); + acb->done =NULL ; + acb->common.replay = true; + acb->common.replay_step =replay_get_current_step (); + + co =qemu_coroutine_create (bdrv_aio_flush_co_entry); +qemu_coroutine_enter (co, acb); + +return &acb->common; +} + static void coroutine_fn bdrv_aio_discard_co_entry(void *opaque) {BlockAIOCBCoroutine *acb = opaque;
@@ -4881,6 +4942,8 @@ void *qemu_aio_get(const AIOCBInfo *aiocb_info, BlockDriverState *bs, acb->cb = cb; acb->opaque = opaque; acb->refcnt =1 ; + acb->replay_step =0 ; + acb->replay = false;return acb; }
block/block-backend.c +29 -1
@@ -447,6 +447,14 @@ BlockAIOCB *blk_aio_readv(BlockBackend *blk, int64_t sector_num,return bdrv_aio_readv (blk->bs, sector_num, iov, nb_sectors, cb, opaque); } +BlockAIOCB *blk_aio_readv_replay (BlockBackend *blk,int64_t sector_num, +QEMUIOVector *iov,int nb_sectors, +BlockCompletionFunc *cb,void *opaque) +{ +return bdrv_aio_readv_replay (blk->bs, sector_num, iov, nb_sectors, + cb, opaque); +} +BlockAIOCB *blk_aio_writev (BlockBackend *blk,int64_t sector_num,QEMUIOVector *iov,int nb_sectors,BlockCompletionFunc *cb,void *opaque)
@@ -454,12 +462,26 @@ BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,return bdrv_aio_writev (blk->bs, sector_num, iov, nb_sectors, cb, opaque); } +BlockAIOCB *blk_aio_writev_replay (BlockBackend *blk,int64_t sector_num, +QEMUIOVector *iov,int nb_sectors, +BlockCompletionFunc *cb,void *opaque) +{ +return bdrv_aio_writev_replay (blk->bs, sector_num, iov, nb_sectors, + cb, opaque); +} +BlockAIOCB *blk_aio_flush (BlockBackend *blk,BlockCompletionFunc *cb,void *opaque) {return bdrv_aio_flush (blk->bs, cb, opaque); } +BlockAIOCB *blk_aio_flush_replay (BlockBackend *blk, +BlockCompletionFunc *cb,void *opaque) +{ +return bdrv_aio_flush_replay (blk->bs, cb, opaque); +} +BlockAIOCB *blk_aio_discard (BlockBackend *blk,int64_t sector_num,int nb_sectors,BlockCompletionFunc *cb,void *opaque)
@@ -479,7 +501,13 @@ void blk_aio_cancel_async(BlockAIOCB *acb)int blk_aio_multiwrite (BlockBackend *blk,BlockRequest *reqs,int num_reqs) { -return bdrv_aio_multiwrite (blk->bs, reqs, num_reqs); +return bdrv_aio_multiwrite (blk->bs, reqs, num_reqs, false); +} + +int blk_aio_multiwrite_replay (BlockBackend *blk,BlockRequest *reqs, +int num_reqs) +{ +return bdrv_aio_multiwrite (blk->bs, reqs, num_reqs, true); }int blk_ioctl (BlockBackend *blk,unsigned long int req,void *buf)
block/qcow2.c +4 -0
@@ -1140,6 +1140,8 @@ static coroutine_fn int qcow2_co_readv(BlockDriverState *bs, int64_t sector_num,uint8_t *cluster_data =NULL ;qemu_iovec_init (&hd_qiov, qiov->niov); + hd_qiov.replay = qiov->replay; + hd_qiov.replay_step = qiov->replay_step;qemu_co_mutex_lock (&s->lock);
@@ -1297,6 +1299,8 @@ static coroutine_fn int qcow2_co_writev(BlockDriverState *bs, remaining_sectors);qemu_iovec_init (&hd_qiov, qiov->niov); + hd_qiov.replay = qiov->replay; + hd_qiov.replay_step = qiov->replay_step; s->cluster_cache_offset =-1 ;/* disable compressed cache */
dma-helpers.c +4 -2
@@ -212,6 +212,8 @@ BlockAIOCB *dma_blk_io( dbs->io_func = io_func; dbs->bh =NULL ;qemu_iovec_init (&dbs->iov, sg->nsg); + dbs->iov.replay = true; + dbs->iov.replay_step =replay_get_current_step ();dma_blk_cb (dbs,0 );return &dbs->common; }
@@ -221,7 +223,7 @@ BlockAIOCB *dma_blk_read(BlockBackend *blk, QEMUSGList *sg, uint64_t sector, void (*cb)(void *opaque, int ret), void *opaque) { - return dma_blk_io(blk, sg, sector, blk_aio_readv, cb, opaque, + return dma_blk_io(blk, sg, sector, blk_aio_readv_replay, cb, opaque,DMA_DIRECTION_FROM_DEVICE ); }
@@ -229,7 +231,7 @@ BlockAIOCB *dma_blk_write(BlockBackend *blk, QEMUSGList *sg, uint64_t sector, void (*cb)(void *opaque, int ret), void *opaque) { - return dma_blk_io(blk, sg, sector, blk_aio_writev, cb, opaque, + return dma_blk_io(blk, sg, sector, blk_aio_writev_replay, cb, opaque,DMA_DIRECTION_TO_DEVICE ); }
hw/block/virtio-blk.c +5 -5
@@ -265,7 +265,7 @@ void virtio_submit_multiwrite(BlockBackend *blk, MultiReqBuffer *mrb)return ; } - ret =blk_aio_multiwrite (blk, mrb->blkreq, mrb->num_writes); + ret =blk_aio_multiwrite_replay (blk, mrb->blkreq, mrb->num_writes);if (ret !=0 ) {for (i =0 ; i < mrb->num_writes; i++) {if (mrb->blkreq[i].error) {
@@ -286,7 +286,7 @@ static void virtio_blk_handle_flush(VirtIOBlockReq *req, MultiReqBuffer *mrb)* Make sure all outstanding writes are posted to the backing device. */ virtio_submit_multiwrite (req->dev->blk, mrb); -blk_aio_flush (req->dev->blk, virtio_blk_flush_complete, req); +blk_aio_flush_replay (req->dev->blk, virtio_blk_flush_complete, req); } static bool virtio_blk_sect_range_ok(VirtIOBlock *dev,
@@ -357,9 +357,9 @@ static void virtio_blk_handle_read(VirtIOBlockReq *req)block_acct_start (blk_get_stats (req->dev->blk), &req->acct, req->qiov.size,BLOCK_ACCT_READ ); -blk_aio_readv (req->dev->blk, sector, &req->qiov, - req->qiov.size /BDRV_SECTOR_SIZE , - virtio_blk_rw_complete, req); +blk_aio_readv_replay (req->dev->blk, sector, &req->qiov, + req->qiov.size /BDRV_SECTOR_SIZE , + virtio_blk_rw_complete, req); }void virtio_blk_handle_request (VirtIOBlockReq *req,MultiReqBuffer *mrb)
hw/ide/atapi.c +6 -4
@@ -25,7 +25,7 @@#include "hw/ide/internal.h" #include "hw/scsi/scsi.h" -#include "sysemu/block-backend.h" +#include "replay/replay.h" static void ide_atapi_cmd_read_dma_cb (void *opaque,int ret);
@@ -350,10 +350,12 @@ static void ide_atapi_cmd_read_dma_cb(void *opaque, int ret) s->bus->dma->iov.iov_base = (void *)(s->io_buffer + data_offset); s->bus->dma->iov.iov_len = n *4 *512 ;qemu_iovec_init_external (&s->bus->dma->qiov, &s->bus->dma->iov,1 ); + s->bus->dma->qiov.replay = true; + s->bus->dma->qiov.replay_step =replay_get_current_step (); - s->bus->dma->aiocb =blk_aio_readv (s->blk, (int64_t )s->lba <<2 , - &s->bus->dma->qiov, n *4 , - ide_atapi_cmd_read_dma_cb, s); + s->bus->dma->aiocb =blk_aio_readv_replay (s->blk, (int64_t )s->lba <<2 , + &s->bus->dma->qiov, n *4 , + ide_atapi_cmd_read_dma_cb, s);return ; eot:
hw/ide/core.c +9 -5
@@ -630,11 +630,13 @@ void ide_sector_read(IDEState *s) s->iov.iov_base = s->io_buffer; s->iov.iov_len = n *BDRV_SECTOR_SIZE ;qemu_iovec_init_external (&s->qiov, &s->iov,1 ); + s->qiov.replay = true; + s->qiov.replay_step =replay_get_current_step ();block_acct_start (blk_get_stats (s->blk), &s->acct, n *BDRV_SECTOR_SIZE ,BLOCK_ACCT_READ ); - s->pio_aiocb =blk_aio_readv (s->blk, sector_num, &s->qiov, n, - ide_sector_read_cb, s); + s->pio_aiocb =blk_aio_readv_replay (s->blk, sector_num, &s->qiov, n, + ide_sector_read_cb, s); }static void dma_buf_commit (IDEState *s,uint32_t tx_bytes)
@@ -888,11 +890,13 @@ void ide_sector_write(IDEState *s) s->iov.iov_base = s->io_buffer; s->iov.iov_len = n *BDRV_SECTOR_SIZE ;qemu_iovec_init_external (&s->qiov, &s->iov,1 ); + s->qiov.replay = true; + s->qiov.replay_step =replay_get_current_step ();block_acct_start (blk_get_stats (s->blk), &s->acct, n *BDRV_SECTOR_SIZE ,BLOCK_ACCT_READ ); - s->pio_aiocb =blk_aio_writev (s->blk, sector_num, &s->qiov, n, - ide_sector_write_cb, s); + s->pio_aiocb =blk_aio_writev_replay (s->blk, sector_num, &s->qiov, n, + ide_sector_write_cb, s); }static void ide_flush_cb (void *opaque,int ret)
@@ -928,7 +932,7 @@ void ide_flush_cache(IDEState *s) s->status |=BUSY_STAT ;block_acct_start (blk_get_stats (s->blk), &s->acct,0 ,BLOCK_ACCT_FLUSH ); - s->pio_aiocb =blk_aio_flush (s->blk, ide_flush_cb, s); + s->pio_aiocb =blk_aio_flush_replay (s->blk, ide_flush_cb, s); }static void ide_cfata_metadata_inquiry (IDEState *s)
include/block/block.h +14 -1
@@ -293,11 +293,24 @@ typedef void BlockDriverDirtyHandler(BlockDriverState *bs, int64_t sector,BlockAIOCB *bdrv_aio_readv (BlockDriverState *bs,int64_t sector_num,QEMUIOVector *iov,int nb_sectors,BlockCompletionFunc *cb,void *opaque); +BlockAIOCB *bdrv_aio_readv_replay (BlockDriverState *bs, +int64_t sector_num, +QEMUIOVector *iov,int nb_sectors, +BlockCompletionFunc *cb, +void *opaque);BlockAIOCB *bdrv_aio_writev (BlockDriverState *bs,int64_t sector_num,QEMUIOVector *iov,int nb_sectors,BlockCompletionFunc *cb,void *opaque); +BlockAIOCB *bdrv_aio_writev_replay (BlockDriverState *bs, +int64_t sector_num, +QEMUIOVector *iov,int nb_sectors, +BlockCompletionFunc *cb, +void *opaque);BlockAIOCB *bdrv_aio_flush (BlockDriverState *bs,BlockCompletionFunc *cb,void *opaque); +BlockAIOCB *bdrv_aio_flush_replay (BlockDriverState *bs, +BlockCompletionFunc *cb, +void *opaque);BlockAIOCB *bdrv_aio_discard (BlockDriverState *bs,int64_t sector_num,int nb_sectors,BlockCompletionFunc *cb,void *opaque);
@@ -318,7 +331,7 @@ typedef struct BlockRequest { } BlockRequest;int bdrv_aio_multiwrite (BlockDriverState *bs,BlockRequest *reqs, -int num_reqs); +int num_reqs,bool replay);/* sg packet commands */ int bdrv_ioctl (BlockDriverState *bs,unsigned long int req,void *buf);
include/qemu-common.h +2 -0
@@ -320,6 +320,8 @@ typedef struct QEMUIOVector {int niov;int nalloc;size_t size; +bool replay; +uint64_t replay_step; } QEMUIOVector;void qemu_iovec_init (QEMUIOVector *qiov,int alloc_hint);
include/sysemu/block-backend.h +10 -0
@@ -94,17 +94,27 @@ void blk_get_geometry(BlockBackend *blk, uint64_t *nb_sectors_ptr);BlockAIOCB *blk_aio_readv (BlockBackend *blk,int64_t sector_num,QEMUIOVector *iov,int nb_sectors,BlockCompletionFunc *cb,void *opaque); +BlockAIOCB *blk_aio_readv_replay (BlockBackend *blk,int64_t sector_num, +QEMUIOVector *iov,int nb_sectors, +BlockCompletionFunc *cb,void *opaque);BlockAIOCB *blk_aio_writev (BlockBackend *blk,int64_t sector_num,QEMUIOVector *iov,int nb_sectors,BlockCompletionFunc *cb,void *opaque); +BlockAIOCB *blk_aio_writev_replay (BlockBackend *blk,int64_t sector_num, +QEMUIOVector *iov,int nb_sectors, +BlockCompletionFunc *cb,void *opaque);BlockAIOCB *blk_aio_flush (BlockBackend *blk,BlockCompletionFunc *cb,void *opaque); +BlockAIOCB *blk_aio_flush_replay (BlockBackend *blk, +BlockCompletionFunc *cb,void *opaque);BlockAIOCB *blk_aio_discard (BlockBackend *blk,int64_t sector_num,int nb_sectors,BlockCompletionFunc *cb,void *opaque);void blk_aio_cancel (BlockAIOCB *acb);void blk_aio_cancel_async (BlockAIOCB *acb);int blk_aio_multiwrite (BlockBackend *blk,BlockRequest *reqs,int num_reqs); +int blk_aio_multiwrite_replay (BlockBackend *blk,BlockRequest *reqs, +int num_reqs);int blk_ioctl (BlockBackend *blk,unsigned long int req,void *buf);BlockAIOCB *blk_aio_ioctl (BlockBackend *blk,unsigned long int req,void *buf,BlockCompletionFunc *cb,void *opaque);
qemu-io-cmds.c +1 -1
@@ -583,7 +583,7 @@ static int do_aio_multiwrite(BlockDriverState *bs, BlockRequest* reqs, *total += reqs[i].qiov->size; } - ret =bdrv_aio_multiwrite (bs, reqs, num_reqs); + ret =bdrv_aio_multiwrite (bs, reqs, num_reqs, false);if (ret <0 ) {return ret; }
stubs/replay.c +5 -0
@@ -25,3 +25,8 @@ int runstate_is_running(void)void replay_add_bh_event (void *bh,uint64_t id) { } + +uint64_t replay_get_current_step (void ) +{ +return 0 ; +}
trace-events +2 -0
@@ -64,7 +64,9 @@ bdrv_aio_multiwrite(void *mcb, int num_callbacks, int num_reqs) "mcb %p num_call bdrv_aio_discard(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p" bdrv_aio_flush(void *bs, void *opaque) "bs %p opaque %p" bdrv_aio_readv(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p" +bdrv_aio_readv_replay(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p" bdrv_aio_writev(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p" +bdrv_aio_writev_replay(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p" bdrv_aio_write_zeroes(void *bs, int64_t sector_num, int nb_sectors, int flags, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d flags %#x opaque %p" bdrv_lock_medium(void *bs, bool locked) "bs %p locked %d" bdrv_co_readv(void *bs, int64_t sector_num, int nb_sector) "bs %p sector_num %"PRId64" nb_sectors %d"
util/iov.c +4 -0
@@ -257,6 +257,8 @@ void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint) qiov->niov =0 ; qiov->nalloc = alloc_hint; qiov->size =0 ; + qiov->replay = false; + qiov->replay_step =0 ; }void qemu_iovec_init_external (QEMUIOVector *qiov,struct iovec *iov,int niov)
@@ -267,6 +269,8 @@ void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov) qiov->niov = niov; qiov->nalloc =-1 ; qiov->size =0 ; + qiov->replay = false; + qiov->replay_step =0 ;for (i =0 ; i < niov; i++) qiov->size += iov[i].iov_len; }
Re: [Qemu-devel] [RFC PATCH v8 17/21] replay: replay aio requests
Quoted 6 lines · Pavel Dovgalyuk
This patch adds identifier to aio requests. ID is used for creating bottom halves and identifying them while replaying.
The patch also introduces several functions that make possible replaying of the aio requests.Signed-off-by Pavel Dovgalyuk <[email protected]>
Ah, some obscure parts of patch 16 now get clearer. :)
But it's still not clear what is the design. You have to document this too, then I (or others) can start reviewing the patches.
Paolo
Quoted 557 lines
--- block.c | 81 ++++++++++++++++++++++++++++++++++++---- block/block-backend.c | 30 ++++++++++++++- block/qcow2.c | 4 ++ dma-helpers.c | 6 ++- hw/block/virtio-blk.c | 10 ++--- hw/ide/atapi.c | 10 +++-- hw/ide/core.c | 14 ++++--- include/block/block.h | 15 +++++++ include/qemu-common.h | 2 + include/sysemu/block-backend.h | 10 +++++ qemu-io-cmds.c | 2 - stubs/replay.c | 5 ++ trace-events | 2 + util/iov.c | 4 ++ 14 files changed, 167 insertions(+), 28 deletions(-)block.c +72
-9@@ -83,7 +83,8 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,BdrvRequestFlags flags, BlockCompletionFunc *cb, void *opaque, - bool is_write); + bool is_write, + bool aio_replay); static void coroutine_fn bdrv_co_do_rw(void *opaque); static int coroutine_fn bdrv_co_do_write_zeroes(BlockDriverState *bs,int64_t sector_num,int nb_sectors,BdrvRequestFlags flags);@@ -4425,7 +4426,19 @@ BlockAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,trace_bdrv_aio_readv (bs, sector_num, nb_sectors, opaque);return bdrv_co_aio_rw_vector (bs, sector_num, qiov, nb_sectors,0 , - cb, opaque, false); + cb, opaque, false, false); +} + +BlockAIOCB *bdrv_aio_readv_replay (BlockDriverState *bs, +int64_t sector_num, +QEMUIOVector *qiov,int nb_sectors, +BlockCompletionFunc *cb, +void *opaque) +{ +trace_bdrv_aio_readv_replay (bs, sector_num, nb_sectors, opaque); + +return bdrv_co_aio_rw_vector (bs, sector_num, qiov, nb_sectors,0 , + cb, opaque, false, true); } BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,@@ -4435,7 +4448,19 @@ BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,trace_bdrv_aio_writev (bs, sector_num, nb_sectors, opaque);return bdrv_co_aio_rw_vector (bs, sector_num, qiov, nb_sectors,0 , - cb, opaque, true); + cb, opaque, true, false); +} + +BlockAIOCB *bdrv_aio_writev_replay (BlockDriverState *bs, +int64_t sector_num, +QEMUIOVector *qiov,int nb_sectors, +BlockCompletionFunc *cb, +void *opaque) +{ +trace_bdrv_aio_writev_replay (bs, sector_num, nb_sectors, opaque); + +return bdrv_co_aio_rw_vector (bs, sector_num, qiov, nb_sectors,0 , + cb, opaque, true, true); } BlockAIOCB *bdrv_aio_write_zeroes(BlockDriverState *bs,@@ -4446,7 +4471,7 @@ BlockAIOCB *bdrv_aio_write_zeroes(BlockDriverState *bs,return bdrv_co_aio_rw_vector (bs, sector_num,NULL , nb_sectors,BDRV_REQ_ZERO_WRITE | flags, - cb, opaque, true); + cb, opaque, true, true); }@@ -4593,7 +4618,8 @@ static int multiwrite_merge(BlockDriverState *bs, BlockRequest *reqs,* requests. However, the fields opaque and error are left unmodified as they * are used to signal failure for a single request to the caller. */ -int bdrv_aio_multiwrite (BlockDriverState *bs,BlockRequest *reqs,int num_reqs) +int bdrv_aio_multiwrite (BlockDriverState *bs,BlockRequest *reqs,int num_reqs, +bool replay) {MultiwriteCB *mcb;int i;@@ -4631,7 +4657,7 @@ int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs)bdrv_co_aio_rw_vector (bs, reqs[i].sector, reqs[i].qiov, reqs[i].nb_sectors, reqs[i].flags, multiwrite_cb, mcb, - true); + true, replay); }return 0 ;@@ -4776,7 +4802,12 @@ static void coroutine_fn bdrv_co_do_rw(void *opaque) acb->req.nb_sectors, acb->req.qiov, acb->req.flags); } - acb->bh =aio_bh_new (bdrv_get_aio_context (bs), bdrv_co_em_bh, acb); +if (acb->common.replay) { + acb->bh =aio_bh_new_replay (bdrv_get_aio_context (bs), bdrv_co_em_bh, + acb, acb->common.replay_step); + }else { + acb->bh =aio_bh_new (bdrv_get_aio_context (bs), bdrv_co_em_bh, acb); + }qemu_bh_schedule (acb->bh); }@@ -4787,7 +4818,8 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,BdrvRequestFlags flags, BlockCompletionFunc *cb, void *opaque, - bool is_write) + bool is_write, + bool aio_replay) { Coroutine *co;BlockAIOCBCoroutine *acb;@@ -4798,6 +4830,11 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs, acb->req.qiov = qiov; acb->req.flags = flags; acb->is_write = is_write; + acb->done =NULL ; + acb->common.replay = aio_replay; +if (aio_replay) { + acb->common.replay_step =replay_get_current_step (); + } co =qemu_coroutine_create (bdrv_co_do_rw);qemu_coroutine_enter (co, acb);@@ -4811,7 +4848,12 @@ static void coroutine_fn bdrv_aio_flush_co_entry(void *opaque)BlockDriverState *bs = acb->common.bs; acb->req.error =bdrv_co_flush (bs); - acb->bh =aio_bh_new (bdrv_get_aio_context (bs), bdrv_co_em_bh, acb); +if (acb->common.replay) { + acb->bh =aio_bh_new_replay (bdrv_get_aio_context (bs), bdrv_co_em_bh, + acb, acb->common.replay_step); + }else { + acb->bh =aio_bh_new (bdrv_get_aio_context (bs), bdrv_co_em_bh, acb); + }qemu_bh_schedule (acb->bh); }@@ -4831,6 +4873,25 @@ BlockAIOCB *bdrv_aio_flush(BlockDriverState *bs,return &acb->common; } +BlockAIOCB *bdrv_aio_flush_replay (BlockDriverState *bs, +BlockCompletionFunc *cb,void *opaque) +{ +trace_bdrv_aio_flush (bs, opaque); + +Coroutine *co; +BlockAIOCBCoroutine *acb; + + acb =qemu_aio_get (&bdrv_em_co_aiocb_info, bs, cb, opaque); + acb->done =NULL ; + acb->common.replay = true; + acb->common.replay_step =replay_get_current_step (); + + co =qemu_coroutine_create (bdrv_aio_flush_co_entry); +qemu_coroutine_enter (co, acb); + +return &acb->common; +} + static void coroutine_fn bdrv_aio_discard_co_entry(void *opaque) {BlockAIOCBCoroutine *acb = opaque;@@ -4881,6 +4942,8 @@ void *qemu_aio_get(const AIOCBInfo *aiocb_info, BlockDriverState *bs, acb->cb = cb; acb->opaque = opaque; acb->refcnt =1 ; + acb->replay_step =0 ; + acb->replay = false;return acb; }block/block-backend.c +29
-1@@ -447,6 +447,14 @@ BlockAIOCB *blk_aio_readv(BlockBackend *blk, int64_t sector_num,return bdrv_aio_readv (blk->bs, sector_num, iov, nb_sectors, cb, opaque); } +BlockAIOCB *blk_aio_readv_replay (BlockBackend *blk,int64_t sector_num, +QEMUIOVector *iov,int nb_sectors, +BlockCompletionFunc *cb,void *opaque) +{ +return bdrv_aio_readv_replay (blk->bs, sector_num, iov, nb_sectors, + cb, opaque); +} +BlockAIOCB *blk_aio_writev (BlockBackend *blk,int64_t sector_num,QEMUIOVector *iov,int nb_sectors,BlockCompletionFunc *cb,void *opaque)@@ -454,12 +462,26 @@ BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,return bdrv_aio_writev (blk->bs, sector_num, iov, nb_sectors, cb, opaque); } +BlockAIOCB *blk_aio_writev_replay (BlockBackend *blk,int64_t sector_num, +QEMUIOVector *iov,int nb_sectors, +BlockCompletionFunc *cb,void *opaque) +{ +return bdrv_aio_writev_replay (blk->bs, sector_num, iov, nb_sectors, + cb, opaque); +} +BlockAIOCB *blk_aio_flush (BlockBackend *blk,BlockCompletionFunc *cb,void *opaque) {return bdrv_aio_flush (blk->bs, cb, opaque); } +BlockAIOCB *blk_aio_flush_replay (BlockBackend *blk, +BlockCompletionFunc *cb,void *opaque) +{ +return bdrv_aio_flush_replay (blk->bs, cb, opaque); +} +BlockAIOCB *blk_aio_discard (BlockBackend *blk,int64_t sector_num,int nb_sectors,BlockCompletionFunc *cb,void *opaque)@@ -479,7 +501,13 @@ void blk_aio_cancel_async(BlockAIOCB *acb)int blk_aio_multiwrite (BlockBackend *blk,BlockRequest *reqs,int num_reqs) { -return bdrv_aio_multiwrite (blk->bs, reqs, num_reqs); +return bdrv_aio_multiwrite (blk->bs, reqs, num_reqs, false); +} + +int blk_aio_multiwrite_replay (BlockBackend *blk,BlockRequest *reqs, +int num_reqs) +{ +return bdrv_aio_multiwrite (blk->bs, reqs, num_reqs, true); }int blk_ioctl (BlockBackend *blk,unsigned long int req,void *buf)block/qcow2.c +4
-0@@ -1140,6 +1140,8 @@ static coroutine_fn int qcow2_co_readv(BlockDriverState *bs, int64_t sector_num,uint8_t *cluster_data =NULL ;qemu_iovec_init (&hd_qiov, qiov->niov); + hd_qiov.replay = qiov->replay; + hd_qiov.replay_step = qiov->replay_step;qemu_co_mutex_lock (&s->lock);@@ -1297,6 +1299,8 @@ static coroutine_fn int qcow2_co_writev(BlockDriverState *bs, remaining_sectors);qemu_iovec_init (&hd_qiov, qiov->niov); + hd_qiov.replay = qiov->replay; + hd_qiov.replay_step = qiov->replay_step; s->cluster_cache_offset =-1 ;/* disable compressed cache */ dma-helpers.c +4
-2@@ -212,6 +212,8 @@ BlockAIOCB *dma_blk_io( dbs->io_func = io_func; dbs->bh =NULL ;qemu_iovec_init (&dbs->iov, sg->nsg); + dbs->iov.replay = true; + dbs->iov.replay_step =replay_get_current_step ();dma_blk_cb (dbs,0 );return &dbs->common; }@@ -221,7 +223,7 @@ BlockAIOCB *dma_blk_read(BlockBackend *blk, QEMUSGList *sg, uint64_t sector, void (*cb)(void *opaque, int ret), void *opaque) { - return dma_blk_io(blk, sg, sector, blk_aio_readv, cb, opaque, + return dma_blk_io(blk, sg, sector, blk_aio_readv_replay, cb, opaque,DMA_DIRECTION_FROM_DEVICE ); }@@ -229,7 +231,7 @@ BlockAIOCB *dma_blk_write(BlockBackend *blk, QEMUSGList *sg, uint64_t sector, void (*cb)(void *opaque, int ret), void *opaque) { - return dma_blk_io(blk, sg, sector, blk_aio_writev, cb, opaque, + return dma_blk_io(blk, sg, sector, blk_aio_writev_replay, cb, opaque,DMA_DIRECTION_TO_DEVICE ); }hw/block/virtio-blk.c +5
-5@@ -265,7 +265,7 @@ void virtio_submit_multiwrite(BlockBackend *blk, MultiReqBuffer *mrb)return ; } - ret =blk_aio_multiwrite (blk, mrb->blkreq, mrb->num_writes); + ret =blk_aio_multiwrite_replay (blk, mrb->blkreq, mrb->num_writes);if (ret !=0 ) {for (i =0 ; i < mrb->num_writes; i++) {if (mrb->blkreq[i].error) {@@ -286,7 +286,7 @@ static void virtio_blk_handle_flush(VirtIOBlockReq *req, MultiReqBuffer *mrb)* Make sure all outstanding writes are posted to the backing device. */ virtio_submit_multiwrite (req->dev->blk, mrb); -blk_aio_flush (req->dev->blk, virtio_blk_flush_complete, req); +blk_aio_flush_replay (req->dev->blk, virtio_blk_flush_complete, req); } static bool virtio_blk_sect_range_ok(VirtIOBlock *dev,@@ -357,9 +357,9 @@ static void virtio_blk_handle_read(VirtIOBlockReq *req)block_acct_start (blk_get_stats (req->dev->blk), &req->acct, req->qiov.size,BLOCK_ACCT_READ ); -blk_aio_readv (req->dev->blk, sector, &req->qiov, - req->qiov.size /BDRV_SECTOR_SIZE , - virtio_blk_rw_complete, req); +blk_aio_readv_replay (req->dev->blk, sector, &req->qiov, + req->qiov.size /BDRV_SECTOR_SIZE , + virtio_blk_rw_complete, req); }void virtio_blk_handle_request (VirtIOBlockReq *req,MultiReqBuffer *mrb)hw/ide/atapi.c +6
-4@@ -25,7 +25,7 @@#include "hw/ide/internal.h" #include "hw/scsi/scsi.h" -#include "sysemu/block-backend.h" +#include "replay/replay.h" static void ide_atapi_cmd_read_dma_cb (void *opaque,int ret);@@ -350,10 +350,12 @@ static void ide_atapi_cmd_read_dma_cb(void *opaque, int ret) s->bus->dma->iov.iov_base = (void *)(s->io_buffer + data_offset); s->bus->dma->iov.iov_len = n *4 *512 ;qemu_iovec_init_external (&s->bus->dma->qiov, &s->bus->dma->iov,1 ); + s->bus->dma->qiov.replay = true; + s->bus->dma->qiov.replay_step =replay_get_current_step (); - s->bus->dma->aiocb =blk_aio_readv (s->blk, (int64_t )s->lba <<2 , - &s->bus->dma->qiov, n *4 , - ide_atapi_cmd_read_dma_cb, s); + s->bus->dma->aiocb =blk_aio_readv_replay (s->blk, (int64_t )s->lba <<2 , + &s->bus->dma->qiov, n *4 , + ide_atapi_cmd_read_dma_cb, s);return ; eot:hw/ide/core.c +9
-5@@ -630,11 +630,13 @@ void ide_sector_read(IDEState *s) s->iov.iov_base = s->io_buffer; s->iov.iov_len = n *BDRV_SECTOR_SIZE ;qemu_iovec_init_external (&s->qiov, &s->iov,1 ); + s->qiov.replay = true; + s->qiov.replay_step =replay_get_current_step ();block_acct_start (blk_get_stats (s->blk), &s->acct, n *BDRV_SECTOR_SIZE ,BLOCK_ACCT_READ ); - s->pio_aiocb =blk_aio_readv (s->blk, sector_num, &s->qiov, n, - ide_sector_read_cb, s); + s->pio_aiocb =blk_aio_readv_replay (s->blk, sector_num, &s->qiov, n, + ide_sector_read_cb, s); }static void dma_buf_commit (IDEState *s,uint32_t tx_bytes)@@ -888,11 +890,13 @@ void ide_sector_write(IDEState *s) s->iov.iov_base = s->io_buffer; s->iov.iov_len = n *BDRV_SECTOR_SIZE ;qemu_iovec_init_external (&s->qiov, &s->iov,1 ); + s->qiov.replay = true; + s->qiov.replay_step =replay_get_current_step ();block_acct_start (blk_get_stats (s->blk), &s->acct, n *BDRV_SECTOR_SIZE ,BLOCK_ACCT_READ ); - s->pio_aiocb =blk_aio_writev (s->blk, sector_num, &s->qiov, n, - ide_sector_write_cb, s); + s->pio_aiocb =blk_aio_writev_replay (s->blk, sector_num, &s->qiov, n, + ide_sector_write_cb, s); }static void ide_flush_cb (void *opaque,int ret)@@ -928,7 +932,7 @@ void ide_flush_cache(IDEState *s) s->status |=BUSY_STAT ;block_acct_start (blk_get_stats (s->blk), &s->acct,0 ,BLOCK_ACCT_FLUSH ); - s->pio_aiocb =blk_aio_flush (s->blk, ide_flush_cb, s); + s->pio_aiocb =blk_aio_flush_replay (s->blk, ide_flush_cb, s); }static void ide_cfata_metadata_inquiry (IDEState *s)include/block/block.h +14
-1@@ -293,11 +293,24 @@ typedef void BlockDriverDirtyHandler(BlockDriverState *bs, int64_t sector,BlockAIOCB *bdrv_aio_readv (BlockDriverState *bs,int64_t sector_num,QEMUIOVector *iov,int nb_sectors,BlockCompletionFunc *cb,void *opaque); +BlockAIOCB *bdrv_aio_readv_replay (BlockDriverState *bs, +int64_t sector_num, +QEMUIOVector *iov,int nb_sectors, +BlockCompletionFunc *cb, +void *opaque);BlockAIOCB *bdrv_aio_writev (BlockDriverState *bs,int64_t sector_num,QEMUIOVector *iov,int nb_sectors,BlockCompletionFunc *cb,void *opaque); +BlockAIOCB *bdrv_aio_writev_replay (BlockDriverState *bs, +int64_t sector_num, +QEMUIOVector *iov,int nb_sectors, +BlockCompletionFunc *cb, +void *opaque);BlockAIOCB *bdrv_aio_flush (BlockDriverState *bs,BlockCompletionFunc *cb,void *opaque); +BlockAIOCB *bdrv_aio_flush_replay (BlockDriverState *bs, +BlockCompletionFunc *cb, +void *opaque);BlockAIOCB *bdrv_aio_discard (BlockDriverState *bs,int64_t sector_num,int nb_sectors,BlockCompletionFunc *cb,void *opaque);@@ -318,7 +331,7 @@ typedef struct BlockRequest { } BlockRequest;int bdrv_aio_multiwrite (BlockDriverState *bs,BlockRequest *reqs, -int num_reqs); +int num_reqs,bool replay);/* sg packet commands */ int bdrv_ioctl (BlockDriverState *bs,unsigned long int req,void *buf);include/qemu-common.h +2
-0@@ -320,6 +320,8 @@ typedef struct QEMUIOVector {int niov;int nalloc;size_t size; +bool replay; +uint64_t replay_step; } QEMUIOVector;void qemu_iovec_init (QEMUIOVector *qiov,int alloc_hint);include/sysemu/block-backend.h +10
-0@@ -94,17 +94,27 @@ void blk_get_geometry(BlockBackend *blk, uint64_t *nb_sectors_ptr);BlockAIOCB *blk_aio_readv (BlockBackend *blk,int64_t sector_num,QEMUIOVector *iov,int nb_sectors,BlockCompletionFunc *cb,void *opaque); +BlockAIOCB *blk_aio_readv_replay (BlockBackend *blk,int64_t sector_num, +QEMUIOVector *iov,int nb_sectors, +BlockCompletionFunc *cb,void *opaque);BlockAIOCB *blk_aio_writev (BlockBackend *blk,int64_t sector_num,QEMUIOVector *iov,int nb_sectors,BlockCompletionFunc *cb,void *opaque); +BlockAIOCB *blk_aio_writev_replay (BlockBackend *blk,int64_t sector_num, +QEMUIOVector *iov,int nb_sectors, +BlockCompletionFunc *cb,void *opaque);BlockAIOCB *blk_aio_flush (BlockBackend *blk,BlockCompletionFunc *cb,void *opaque); +BlockAIOCB *blk_aio_flush_replay (BlockBackend *blk, +BlockCompletionFunc *cb,void *opaque);BlockAIOCB *blk_aio_discard (BlockBackend *blk,int64_t sector_num,int nb_sectors,BlockCompletionFunc *cb,void *opaque);void blk_aio_cancel (BlockAIOCB *acb);void blk_aio_cancel_async (BlockAIOCB *acb);int blk_aio_multiwrite (BlockBackend *blk,BlockRequest *reqs,int num_reqs); +int blk_aio_multiwrite_replay (BlockBackend *blk,BlockRequest *reqs, +int num_reqs);int blk_ioctl (BlockBackend *blk,unsigned long int req,void *buf);BlockAIOCB *blk_aio_ioctl (BlockBackend *blk,unsigned long int req,void *buf,BlockCompletionFunc *cb,void *opaque);qemu-io-cmds.c +1
-1@@ -583,7 +583,7 @@ static int do_aio_multiwrite(BlockDriverState *bs, BlockRequest* reqs, *total += reqs[i].qiov->size; } - ret =bdrv_aio_multiwrite (bs, reqs, num_reqs); + ret =bdrv_aio_multiwrite (bs, reqs, num_reqs, false);if (ret <0 ) {return ret; }stubs/replay.c +5
-0@@ -25,3 +25,8 @@ int runstate_is_running(void)void replay_add_bh_event (void *bh,uint64_t id) { } + +uint64_t replay_get_current_step (void ) +{ +return 0 ; +}trace-events +2
-0@@ -64,7 +64,9 @@ bdrv_aio_multiwrite(void *mcb, int num_callbacks, int num_reqs) "mcb %p num_call bdrv_aio_discard(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p" bdrv_aio_flush(void *bs, void *opaque) "bs %p opaque %p" bdrv_aio_readv(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p" +bdrv_aio_readv_replay(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p" bdrv_aio_writev(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p" +bdrv_aio_writev_replay(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p" bdrv_aio_write_zeroes(void *bs, int64_t sector_num, int nb_sectors, int flags, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d flags %#x opaque %p" bdrv_lock_medium(void *bs, bool locked) "bs %p locked %d" bdrv_co_readv(void *bs, int64_t sector_num, int nb_sector) "bs %p sector_num %"PRId64" nb_sectors %d"util/iov.c +4
-0@@ -257,6 +257,8 @@ void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint) qiov->niov =0 ; qiov->nalloc = alloc_hint; qiov->size =0 ; + qiov->replay = false; + qiov->replay_step =0 ; }void qemu_iovec_init_external (QEMUIOVector *qiov,struct iovec *iov,int niov)@@ -267,6 +269,8 @@ void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov) qiov->niov = niov; qiov->nalloc =-1 ; qiov->size =0 ; + qiov->replay = false; + qiov->replay_step =0 ;for (i =0 ; i < niov; i++) qiov->size += iov[i].iov_len; }
RFC PATCH v8 18/21 replay: thread pool
In reply to Pavel DovgalyukThis patch modifies thread pool to allow replaying asynchronous thread tasks synchronously in replay mode.
--- block/raw-posix.c | 6 ++++- block/raw-win32.c | 4 +++- include/block/thread-pool.h | 4 +++- replay/replay-events.c | 11 ++++++++++ replay/replay-internal.h | 1 + replay/replay.h | 2 ++ stubs/replay.c | 4 ++++ tests/test-thread-pool.c | 7 ++++-- thread-pool.c | 49 ++++++++++++++++++++++++++++++------------- 9 files changed, 66 insertions(+), 22 deletions(-)
block/raw-posix.c +4 -2
@@ -1073,7 +1073,9 @@ static BlockAIOCB *paio_submit(BlockDriverState *bs, int fd,trace_paio_submit (acb, opaque, sector_num, nb_sectors, type); pool =aio_get_thread_pool (bdrv_get_aio_context (bs)); -return thread_pool_submit_aio (pool, aio_worker, acb, cb, opaque); +return thread_pool_submit_aio (pool, aio_worker, acb, cb, opaque, + qiov ? qiov->replay : false, + qiov ? qiov->replay_step :0 ); } static BlockAIOCB *raw_aio_submit(BlockDriverState *bs,
@@ -1986,7 +1988,7 @@ static BlockAIOCB *hdev_aio_ioctl(BlockDriverState *bs, acb->aio_ioctl_buf = buf; acb->aio_ioctl_cmd = req; pool =aio_get_thread_pool (bdrv_get_aio_context (bs)); - return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque); + return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque, false, 0); }#elif defined(__FreeBSD__) || defined(__FreeBSD_kernel__)
block/raw-win32.c +3 -1
@@ -158,7 +158,9 @@ static BlockAIOCB *paio_submit(BlockDriverState *bs, HANDLE hfile,trace_paio_submit (acb, opaque, sector_num, nb_sectors, type); pool =aio_get_thread_pool (bdrv_get_aio_context (bs)); -return thread_pool_submit_aio (pool, aio_worker, acb, cb, opaque); +return thread_pool_submit_aio (pool, aio_worker, acb, cb, opaque, + qiov ? qiov->replay : false, + qiov ? qiov->replay_step :0 ); }int qemu_ftruncate64 (int fd,int64_t length)
include/block/thread-pool.h +3 -1
@@ -29,9 +29,11 @@ void thread_pool_free(ThreadPool *pool);BlockAIOCB *thread_pool_submit_aio (ThreadPool *pool,ThreadPoolFunc *func,void *arg, -BlockCompletionFunc *cb,void *opaque); +BlockCompletionFunc *cb,void *opaque, +bool replay,uint64_t replay_step); int coroutine_fn thread_pool_submit_co(ThreadPool *pool, ThreadPoolFunc *func, void *arg);void thread_pool_submit (ThreadPool *pool,ThreadPoolFunc *func,void *arg); +void thread_pool_work (ThreadPool *pool,void *r);#endif
replay/replay-events.c +11 -0
@@ -12,6 +12,7 @@#include "qemu-common.h" #include "replay.h" #include "replay-internal.h" +#include "block/thread-pool.h" typedef struct Event { ReplayAsyncEventKind event_kind;
@@ -39,6 +40,9 @@ static void replay_run_event(Event *event)case REPLAY_ASYNC_EVENT_BH :aio_bh_call (event->opaque);break ; +case REPLAY_ASYNC_EVENT_THREAD : +thread_pool_work ((ThreadPool *)event->opaque, event->opaque2); +break ;default :fprintf (stderr,"Replay: invalid async event ID (%d) in the queue\n" , event->event_kind);
@@ -130,6 +134,11 @@ void replay_add_bh_event(void *bh, uint64_t id)replay_add_event_internal (REPLAY_ASYNC_EVENT_BH , bh,NULL , id); } +void replay_add_thread_event (void *opaque,void *opaque2,uint64_t id) +{ +replay_add_event_internal (REPLAY_ASYNC_EVENT_THREAD , opaque, opaque2, id); +} +/* Called with replay mutex locked */ void replay_save_events (int opt) {
@@ -145,6 +154,7 @@ void replay_save_events(int opt)/* save event-specific data */ switch (event->event_kind) {case REPLAY_ASYNC_EVENT_BH : +case REPLAY_ASYNC_EVENT_THREAD :replay_put_qword (event->id);break ; }
@@ -179,6 +189,7 @@ void replay_read_events(int opt)/* Execute some events without searching them in the queue */ switch (read_event_kind) {case REPLAY_ASYNC_EVENT_BH : +case REPLAY_ASYNC_EVENT_THREAD :if (read_id ==-1 ) { read_id =replay_get_qword (); }
replay/replay-internal.h +1 -0
@@ -41,6 +41,7 @@ enum ReplayEvents {enum ReplayAsyncEventKind {REPLAY_ASYNC_EVENT_BH , +REPLAY_ASYNC_EVENT_THREAD ,REPLAY_ASYNC_COUNT };
replay/replay.h +2 -0
@@ -108,5 +108,7 @@ bool replay_checkpoint(ReplayCheckpoint checkpoint);void replay_disable_events (void );/*! Adds BH event to the queue */ void replay_add_bh_event (void *bh,uint64_t id); +/*! Adds thread event to the queue */ +void replay_add_thread_event (void *pool,void *req,uint64_t id);#endif
stubs/replay.c +4 -0
@@ -30,3 +30,7 @@ uint64_t replay_get_current_step(void) {return 0 ; } + +void replay_add_thread_event (void *opaque,void *opaque2,uint64_t id) +{ +}
tests/test-thread-pool.c +4 -3
@@ -56,7 +56,7 @@ static void test_submit_aio(void) {WorkerTestData data = { .n =0 , .ret = -EINPROGRESS }; data.aiocb =thread_pool_submit_aio (pool, worker_cb, &data, - done_cb, &data); + done_cb, &data, false,0 );/* The callbacks are not called until after the first wait. */ active =1 ;
@@ -120,7 +120,8 @@ static void test_submit_many(void)for (i =0 ; i <100 ; i++) { data[i].n =0 ; data[i].ret = -EINPROGRESS ; -thread_pool_submit_aio (pool, worker_cb, &data[i], done_cb, &data[i]); +thread_pool_submit_aio (pool, worker_cb, &data[i], done_cb, &data[i], + false,0 ); } active =100 ;
@@ -149,7 +150,7 @@ static void do_test_cancel(bool sync) data[i].n =0 ; data[i].ret = -EINPROGRESS ; data[i].aiocb =thread_pool_submit_aio (pool, long_cb, &data[i], - done_cb, &data[i]); + done_cb, &data[i], false,0 ); }/* Starting the threads may be left to a bottom half. Let it
thread-pool.c +34 -15
99 lines not on this page · raw message
Re: [Qemu-devel] [RFC PATCH v8 18/21] replay: thread pool
block/raw-posix.c static BlockAIOCB *paio_submit(BlockDriverState *bs, int fd, · line 1078
Quoted 4 lines · Pavel Dovgalyuk
block/raw-posix.c +3
-1-return thread_pool_submit_aio (pool, aio_worker, acb, cb, opaque); +return thread_pool_submit_aio (pool, aio_worker, acb, cb, opaque, + qiov ? qiov->replay : false, + qiov ? qiov->replay_step :0 );
Adding the replay/replay_step to the QEMUIOVector is... weird. You're paying for the fact that we do not have a BlockRequest struct that is used throughout.
I guess we can live with this until other problems are solved, then we can discuss this with the block folks and go for a more complete solution.
Paolo
RFC PATCH v8 19/21 replay: initialization and deinitialization
In reply to Pavel DovgalyukThis patch introduces the functions for enabling the record/replay and for freeing the resources when simulator closes.
--- block.c | 2 - exec.c | 1 replay/replay-internal.c | 1 replay/replay-internal.h | 4 + replay/replay.c | 140 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 13 ++++ stubs/replay.c | 1 vl.c | 5 ++ 8 files changed, 165 insertions(+), 2 deletions(-)
block.c +1 -1
@@ -2010,7 +2010,7 @@ void bdrv_drain_all(void)aio_context_release (aio_context); } } -if (replay_mode ==REPLAY_MODE_PLAY ) { +if (replay_mode ==REPLAY_MODE_PLAY && replay_checkpoints) {/* Skip checkpoints from the log */ while (replay_checkpoint (CHECKPOINT_BDRV_DRAIN )) {/* Nothing */
exec.c +1 -0
@@ -783,6 +783,7 @@ void cpu_abort(CPUState *cpu, const char *fmt, ...) }va_end (ap2);va_end (ap); +replay_finish ();#if defined(CONFIG_USER_ONLY ) {struct sigaction act;
replay/replay-internal.c +1 -0
@@ -33,6 +33,7 @@ void replay_put_byte(uint8_t byte)void replay_put_event (uint8_t event) { +assert (event <=EVENT_END );replay_put_byte (event); }
replay/replay-internal.h +3 -1
@@ -34,7 +34,9 @@ enum ReplayEvents {EVENT_CLOCK ,/* for checkpoint event */ /* some of grteater codes are reserved for checkpoints */ - EVENT_CHECKPOINT = EVENT_CLOCK + REPLAY_CLOCK_COUNT +EVENT_CHECKPOINT =EVENT_CLOCK +REPLAY_CLOCK_COUNT , +/* end of log event */ + EVENT_END = EVENT_CHECKPOINT + CHECKPOINT_COUNT };/* Asynchronous events IDs */
replay/replay.c +140 -0
158 lines not on this page · raw message
replay/replay.h +13 -0
@@ -17,6 +17,8 @@#include <time.h> #include "qapi-types.h" +struct QemuOpts ; +/* replay clock kinds */ enum ReplayClockKind {/* rdtsc */
@@ -45,6 +47,17 @@ enum ReplayCheckpoint {typedef enum ReplayCheckpoint ReplayCheckpoint ;extern ReplayMode replay_mode; +/*! Is set to true after finishing initialization */ +extern bool replay_checkpoints; + +/* Replay process control functions */ + +/*! Enables recording or saving event log with specified parameters */ +void replay_configure (struct QemuOpts *opts); +/*! Initializes timers used for snapshotting and enables events recording */ +void replay_init_timer (void ); +/*! Closes replay log file and frees other resources. */ +void replay_finish (void );/* Processing the instructions */
stubs/replay.c +1 -0
@@ -2,6 +2,7 @@#include "sysemu/sysemu.h" ReplayMode replay_mode; +bool replay_checkpoints;int64_t replay_save_clock (unsigned int kind,int64_t clock) {
vl.c +5 -0
@@ -4376,7 +4376,12 @@ int main(int argc, char **argv, char **envp) } } +replay_init_timer (); +main_loop (); +if (replay_mode !=REPLAY_MODE_NONE ) { +replay_disable_events (); + }bdrv_close_all ();pause_all_vcpus ();res_free ();
Re: [Qemu-devel] [RFC PATCH v8 19/21] replay: initialization and deinitialization
Quoted 45 lines · Pavel Dovgalyuk
This patch introduces the functions for enabling the record/replay and for freeing the resources when simulator closes.
Signed-off-by Pavel Dovgalyuk <[email protected]>--- block.c | 2 - exec.c | 1 replay/replay-internal.c | 1 replay/replay-internal.h | 4 + replay/replay.c | 140 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 13 ++++ stubs/replay.c | 1 vl.c | 5 ++ 8 files changed, 165 insertions(+), 2 deletions(-)block.c +1
-1@@ -2010,7 +2010,7 @@ void bdrv_drain_all(void)aio_context_release (aio_context); } } -if (replay_mode ==REPLAY_MODE_PLAY ) { +if (replay_mode ==REPLAY_MODE_PLAY && replay_checkpoints) {/* Skip checkpoints from the log */ while (replay_checkpoint (CHECKPOINT_BDRV_DRAIN )) {/* Nothing */ exec.c +1
-0@@ -783,6 +783,7 @@ void cpu_abort(CPUState *cpu, const char *fmt, ...) }va_end (ap2);va_end (ap); +replay_finish ();#if defined(CONFIG_USER_ONLY ) {struct sigaction act;replay/replay-internal.c +0
-0
replay/replay-internal.c void replay_put_byte(uint8_t byte) · line 36
Quoted 4 lines
replay/replay-internal.c +1
-0@@ -33,6 +33,7 @@ void replay_put_byte(uint8_t byte)void replay_put_event (uint8_t event) { +assert (event <=EVENT_END );
Please move (way) earlier in the series.
Quoted 8 lines
replay/replay-internal.c +0
-0replay_put_byte (event); }replay/replay-internal.h +0
-0
replay/replay-internal.h enum ReplayEvents { · line 39
Quoted 7 lines
replay/replay-internal.h +3
-1@@ -34,7 +34,9 @@ enum ReplayEvents {EVENT_CLOCK ,/* for checkpoint event */ /* some of grteater codes are reserved for checkpoints */ -EVENT_CHECKPOINT =EVENT_CLOCK +REPLAY_CLOCK_COUNT +EVENT_CHECKPOINT =EVENT_CLOCK +REPLAY_CLOCK_COUNT , +/* end of log event */ +EVENT_END =EVENT_CHECKPOINT +CHECKPOINT_COUNT
Please add EVENT_END in the beginning of the series. If you do:
EVENT_CLOCK,
EVENT_CLOCK_LAST = EVENT_CLOCK + REPLAY_CLOCK_COUNT - 1, EVENT_CHECKPOINT,
EVENT_CHECKPOINT_LAST = EVENT_CHECKPOINT + CHECKPOINT_COUNT - 1, EVENT_END,
the patches look quite nice.
Quoted 8 lines
replay/replay-internal.h +0
-0};/* Asynchronous events IDs */ replay/replay.c +0
-0
replay/replay.c line 20
Quoted 6 lines
replay/replay.c +3
-0@@ -15,9 +15,18 @@#include "qemu/timer.h" #include "sysemu/sysemu.h" +/* Current version of the replay mechanism. +Increase it when file format changes. */ +#define REPLAY_VERSION 0xe02002
Any meaning? Perhaps make it 'QRR\0' or something identifiable?
replay/replay.c line 29
Quoted 9 lines
replay/replay.c +6
-0+/* Size of replay log header */ +#define HEADER_SIZE (sizeof(uint32_t) + sizeof(uint64_t)) +ReplayMode replay_mode =REPLAY_MODE_NONE ; +/* Name of replay file */ +static char *replay_filename;ReplayState replay_state; +bool replay_checkpoints;
Why is replay_checkpoints only defined and used in this patch? Is it a duplicate of replay_events_enabled?
Quoted 14 lines
replay/replay.c +4
-0bool skip_async_events (int stop_event) {@@ -167,6 +176,10 @@ void replay_shutdown_request(void)bool replay_checkpoint (ReplayCheckpoint checkpoint) {bool res = false; +if (!replay_checkpoints) { +return true; + } +replay_save_instructions ();if (replay_file) {
replay/replay.c bool replay_checkpoint(ReplayCheckpoint checkpoint) · line 221
Quoted 10 lines
replay/replay.c +7
-0@@ -199,3 +212,130 @@ bool replay_checkpoint(ReplayCheckpoint checkpoint)return true; } + +static void replay_enable (const char *fname,int mode) +{ +const char *fmode =NULL ; +if (replay_file) { +fprintf (stderr, +"Replay: some record/replay operation is already started\n" );
Would this have to be an assertion?
Quoted 26 lines
replay/replay.c +26
-0+return ; + } + +switch (mode) { +case REPLAY_MODE_RECORD : + fmode ="wb" ; +break ; +case REPLAY_MODE_PLAY : + fmode ="rb" ; +break ; +default : +fprintf (stderr,"Replay: internal error: invalid replay mode\n" ); +exit (1 ); + } + +atexit (replay_finish); + +replay_mutex_init (); + + replay_file =fopen (fname, fmode); +if (replay_file ==NULL ) { +fprintf (stderr,"Replay: open %s: %s\n" , fname,strerror (errno)); +exit (1 ); + } + + replay_filename =g_strdup (fname);
replay/replay.c bool replay_checkpoint(ReplayCheckpoint checkpoint) · line 257
Quoted 10 lines
replay/replay.c +10
-0+ + replay_mode = mode; + replay_has_unread_data =0 ; + replay_data_kind =-1 ; + replay_state.instructions_count =0 ; + replay_state.current_step =0 ; + +/* skip file header for RECORD and check it for PLAY */ +if (replay_mode ==REPLAY_MODE_RECORD ) { +fseek (replay_file,HEADER_SIZE ,SEEK_SET );
Why can't you write the header here?
Quoted 40 lines
replay/replay.c +40
-0+ }else if (replay_mode ==REPLAY_MODE_PLAY ) { +unsigned int version =replay_get_dword (); +uint64_t offset =replay_get_qword (); +if (version !=REPLAY_VERSION ) { +fprintf (stderr,"Replay: invalid input log file version\n" ); +exit (1 ); + } +/* go to the beginning */ +fseek (replay_file,12 ,SEEK_SET ); + } + +replay_init_events (); +} + +void replay_configure (QemuOpts *opts) +{ +const char *fname; +const char *rr; +ReplayMode mode =REPLAY_MODE_NONE ; + + rr =qemu_opt_get (opts,"rr" ); +if (!rr) { +/* Just enabling icount */ +return ; + }else if (!strcmp (rr,"record" )) { + mode =REPLAY_MODE_RECORD ; + }else if (!strcmp (rr,"replay" )) { + mode =REPLAY_MODE_PLAY ; + }else { +fprintf (stderr,"Invalid icount rr option: %s\n" , rr); +exit (1 ); + } + + fname =qemu_opt_get (opts,"rrfname" ); +if (!fname) { +fprintf (stderr,"File name not specified for replay\n" ); +exit (1 ); + } + +replay_enable (fname, mode);
replay/replay.c bool replay_checkpoint(ReplayCheckpoint checkpoint) · line 307
Quoted 10 lines
replay/replay.c +10
-0+} + +void replay_init_timer (void ) +{ +if (replay_mode ==REPLAY_MODE_NONE ) { +return ; + } + + replay_checkpoints = true; +replay_enable_events ();
Move if to replay_enable_events() and drop replay_init_timer() altogether?
Quoted 11 lines
replay/replay.c +11
-0+} + +void replay_finish (void ) +{ +if (replay_mode ==REPLAY_MODE_NONE ) { +return ; + } + +replay_save_instructions (); + +/* finalize the file */
replay/replay.c bool replay_checkpoint(ReplayCheckpoint checkpoint) · line 328
Quoted 10 lines
replay/replay.c +10
-0+if (replay_file) { +if (replay_mode ==REPLAY_MODE_RECORD ) { +uint64_t offset =0 ; +/* write end event */ +replay_put_event (EVENT_END ); + +/* write header */ +fseek (replay_file,0 ,SEEK_SET ); +replay_put_dword (REPLAY_VERSION ); +replay_put_qword (offset);
Why is this done here? You're writing a constant zero.
Quoted 18 lines
replay/replay.c +13
-0+ } + +fclose (replay_file); + replay_file =NULL ; + } +if (replay_filename) { +g_free (replay_filename); + replay_filename =NULL ; + } + +replay_mutex_destroy (); +replay_finish_events (); +}replay/replay.h +0
-0
replay/replay.h line 20
Quoted 4 lines
replay/replay.h +1
-0@@ -17,6 +17,8 @@#include <time.h> #include "qapi-types.h" +struct QemuOpts ;
#include instead?
Quoted 38 lines
replay/replay.h +11
-0/* replay clock kinds */ enum ReplayClockKind {/* rdtsc */ @@ -45,6 +47,17 @@ enum ReplayCheckpoint {typedef enum ReplayCheckpoint ReplayCheckpoint ;extern ReplayMode replay_mode; +/*! Is set to true after finishing initialization */ +extern bool replay_checkpoints; + +/* Replay process control functions */ + +/*! Enables recording or saving event log with specified parameters */ +void replay_configure (struct QemuOpts *opts); +/*! Initializes timers used for snapshotting and enables events recording */ +void replay_init_timer (void ); +/*! Closes replay log file and frees other resources. */ +void replay_finish (void );/* Processing the instructions */ stubs/replay.c +1
-0@@ -2,6 +2,7 @@#include "sysemu/sysemu.h" ReplayMode replay_mode; +bool replay_checkpoints;int64_t replay_save_clock (unsigned int kind,int64_t clock) {vl.c +0
-0
vl.c int main(int argc, char **argv, char **envp) · line 4379
Quoted 4 lines
vl.c +1
-0@@ -4376,7 +4376,12 @@ int main(int argc, char **argv, char **envp) } } +replay_init_timer ();
replay_enable_events()
vl.c int main(int argc, char **argv, char **envp) · line 4383
Quoted 3 lines
vl.c +2
-0main_loop (); +if (replay_mode !=REPLAY_MODE_NONE ) { +replay_disable_events ();
Move "if" to replay_disable_events.
Paolo
Quoted 5 lines
vl.c +1
-0+ }bdrv_close_all ();pause_all_vcpus ();res_free ();
Re: [Qemu-devel] [RFC PATCH v8 19/21] replay: initialization and deinitialization
Quoted 18 lines
From: Paolo Bonzini [mailto:[email protected]]
Quoted 16 lines · Pavel Dovgalyuk
This patch introduces the functions for enabling the record/replay and for freeing the resources when simulator closes.
Signed-off-by Pavel Dovgalyuk <[email protected]>};
/* Asynchronous events IDs */
replay/replay.c +0
-0@@ -15,9 +15,18 @@#include "qemu/timer.h" #include "sysemu/sysemu.h"
Quoted 5 lines
replay/replay.c line 20
Quoted 3 lines
replay/replay.c +3
-0+/* Current version of the replay mechanism. +Increase it when file format changes. */ +#define REPLAY_VERSION 0xe02002Any meaning? Perhaps make it 'QRR\0' or something identifiable?
Version id has no special meaning, but has to be distinct from other RR versions. When format of the log file changes we have to update version id to prevent reading incompatible files.
Quoted 11 lines
Quoted 11 lines
replay/replay.c +11
-0+ replay_file =fopen (fname, fmode); +if (replay_file ==NULL ) { +fprintf (stderr,"Replay: open %s: %s\n" , fname,strerror (errno)); +exit (1 ); + } + + replay_filename =g_strdup (fname); + + replay_mode = mode; + replay_has_unread_data =0 ; + replay_data_kind =-1 ;
Quoted 8 lines
replay/replay.c bool replay_checkpoint(ReplayCheckpoint checkpoint) · line 257
Quoted 6 lines
replay/replay.c +6
-0+ replay_state.instructions_count =0 ; + replay_state.current_step =0 ; + +/* skip file header for RECORD and check it for PLAY */ +if (replay_mode ==REPLAY_MODE_RECORD ) { +fseek (replay_file,HEADER_SIZE ,SEEK_SET );Why can't you write the header here?
We don't write header here to detect broken log files at the replay stage.
Quoted 7 lines
Quoted 7 lines
replay/replay.c +7
-0+replay_save_instructions (); + +/* finalize the file */ +if (replay_file) { +if (replay_mode ==REPLAY_MODE_RECORD ) { +uint64_t offset =0 ; +/* write end event */
Quoted 8 lines
replay/replay.c bool replay_checkpoint(ReplayCheckpoint checkpoint) · line 328
Quoted 6 lines
replay/replay.c +6
-0+replay_put_event (EVENT_END ); + +/* write header */ +fseek (replay_file,0 ,SEEK_SET ); +replay_put_dword (REPLAY_VERSION ); +replay_put_qword (offset);Why is this done here? You're writing a constant zero.
Right, this offset is the field reserved for snapshots table. It will be introduced in other patches.
Pavel Dovgalyuk
Re: [Qemu-devel] [RFC PATCH v8 19/21] replay: initialization and deinitialization
Quoted 48 lines · Pavel Dovgaluk
Quoted 18 lines
From: Paolo Bonzini [mailto:[email protected]]
Quoted 16 lines · Pavel Dovgalyuk
This patch introduces the functions for enabling the record/replay and for freeing the resources when simulator closes.
Signed-off-by Pavel Dovgalyuk <[email protected]>};
/* Asynchronous events IDs */
replay/replay.c +0
-0@@ -15,9 +15,18 @@#include "qemu/timer.h" #include "sysemu/sysemu.h" Quoted 5 lines
replay/replay.c line 20
Quoted 3 lines
replay/replay.c +3
-0+/* Current version of the replay mechanism. +Increase it when file format changes. */ +#define REPLAY_VERSION 0xe02002Any meaning? Perhaps make it 'QRR\0' or something identifiable?
Version id has no special meaning, but has to be distinct from other RR versions. When format of the log file changes we have to update version id to prevent reading incompatible files.
Quoted 11 lines
Quoted 11 lines
replay/replay.c +11
-0+ replay_file =fopen (fname, fmode); +if (replay_file ==NULL ) { +fprintf (stderr,"Replay: open %s: %s\n" , fname,strerror (errno)); +exit (1 ); + } + + replay_filename =g_strdup (fname); + + replay_mode = mode; + replay_has_unread_data =0 ; + replay_data_kind =-1 ;Quoted 8 lines
replay/replay.c bool replay_checkpoint(ReplayCheckpoint checkpoint) · line 257
Quoted 6 lines
replay/replay.c +6
-0+ replay_state.instructions_count =0 ; + replay_state.current_step =0 ; + +/* skip file header for RECORD and check it for PLAY */ +if (replay_mode ==REPLAY_MODE_RECORD ) { +fseek (replay_file,HEADER_SIZE ,SEEK_SET );Why can't you write the header here?
Quoted 1 line
We don't write header here to detect broken log files at the replay stage.
I guess this is also a provision for something that you'll do in the future.
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
replay/replay.c +7
-0+replay_save_instructions (); + +/* finalize the file */ +if (replay_file) { +if (replay_mode ==REPLAY_MODE_RECORD ) { +uint64_t offset =0 ; +/* write end event */ Quoted 8 lines
replay/replay.c bool replay_checkpoint(ReplayCheckpoint checkpoint) · line 328
Quoted 6 lines
replay/replay.c +6
-0+replay_put_event (EVENT_END ); + +/* write header */ +fseek (replay_file,0 ,SEEK_SET ); +replay_put_dword (REPLAY_VERSION ); +replay_put_qword (offset);Why is this done here? You're writing a constant zero.
Quoted 2 lines
Right, this offset is the field reserved for snapshots table. It will be introduced in other patches.
Okay, then please add comments.
Paolo
RFC PATCH v8 20/21 replay: command line options
In reply to Pavel DovgalyukThis patch introduces command line options for enabling recording or replaying virtual machine behavior. "-record" option starts recording of the execution and saves it into the log, specified with "fname" parameter. "-replay" option is intended for replaying previously saved log.
--- cpus.c | 3 ++- qemu-options.hx | 8 ++++++-- vl.c | 27 ++++++++++++++++++++++++++- 3 files changed, 34 insertions(+), 4 deletions(-)
cpus.c +2 -1
@@ -912,9 +912,10 @@ static void qemu_wait_io_event_common(CPUState *cpu)static void qemu_tcg_wait_io_event (void ) {CPUState *cpu; +GMainContext *context =g_main_context_default ();while (all_cpu_threads_idle ()) { -/* Start accounting real time to the virtual clock if the CPUs +/* Start accounting real time to the virtual clock if the CPUs are idle. */ qemu_clock_warp (QEMU_CLOCK_VIRTUAL );qemu_cond_wait (tcg_halt_cond, &qemu_global_mutex);
qemu-options.hx +6 -2
@@ -3037,11 +3037,11 @@ re-inject them. ETEXI DEF("icount", HAS_ARG, QEMU_OPTION_icount, \ - "-icount [shift=N|auto][,align=on|off]\n" \ + "-icount [shift=N|auto][,align=on|off][,rr=record|replay,rrfname=<filename>]\n" \ " enable virtual instruction counter with 2^N clock ticks per\n" \ " instruction and enable aligning the host and virtual clocks\n", QEMU_ARCH_ALL) STEXI -@item -icount [shift=@var{N}|auto] +@item -icount [shift=@var{N}|auto][,rr=record|replay,rrfname=@var{filename}] @findex -icount Enable virtual instruction counter. The virtual cpu will execute one instruction every 2^@var{N} ns of virtual time. If @code{auto} is specified
@@ -3063,6 +3063,10 @@ Currently this option does not work when @option{shift} is @code{auto}. Note: The sync algorithm will work for those shift values for which the guest clock runs ahead of the host clock. Typically this happens when the shift value is high (how high depends on the host machine). + +When @option{rr} option is specified deterministic record/replay is enabled. +Replay log is written into @var{filename} file in record mode and +read from this file in replay mode. ETEXI DEF("watchdog", HAS_ARG, QEMU_OPTION_watchdog, \
vl.c +26 -1
@@ -475,6 +475,12 @@ static QemuOptsList qemu_icount_opts = { }, { .name = "align", .type = QEMU_OPT_BOOL, + }, { + .name = "rr", + .type = QEMU_OPT_STRING, + }, { + .name = "rrfname", + .type = QEMU_OPT_STRING, }, { /* end of list */ } },
@@ -2752,6 +2758,8 @@ int main(int argc, char **argv, char **envp) {int i;int snapshot, linux_boot; +int not_compatible_replay_param =0 ; +const char *icount_option =NULL ;const char *initrd_filename;const char *kernel_filename, *kernel_cmdline;const char *boot_order;
@@ -2949,6 +2957,7 @@ int main(int argc, char **argv, char **envp)break ;case QEMU_OPTION_pflash:drive_add (IF_PFLASH ,-1 , optarg,PFLASH_OPTS ); + not_compatible_replay_param++;break ;case QEMU_OPTION_snapshot: snapshot =1 ;
@@ -3105,6 +3114,7 @@ int main(int argc, char **argv, char **envp)#endif case QEMU_OPTION_bt:add_device_config (DEV_BT , optarg); + not_compatible_replay_param++;break ;case QEMU_OPTION_audio_help:AUD_help ();
@@ -3244,6 +3254,7 @@ int main(int argc, char **argv, char **envp)if (!opts) {exit (1 ); } + not_compatible_replay_param++;break ;case QEMU_OPTION_fsdev: olist =qemu_find_opts ("fsdev" );
@@ -3372,6 +3383,7 @@ int main(int argc, char **argv, char **envp)if (strncmp (optarg,"mon:" ,4 ) ==0 ) { default_monitor =0 ; } + not_compatible_replay_param++;break ;case QEMU_OPTION_debugcon:add_device_config (DEV_DEBUGCON , optarg);
@@ -3489,6 +3501,7 @@ int main(int argc, char **argv, char **envp)if (!qemu_opts_parse (qemu_find_opts ("smp-opts" ), optarg,1 )) {exit (1 ); } + not_compatible_replay_param++;break ;case QEMU_OPTION_vnc:#ifdef CONFIG_VNC
@@ -3628,6 +3641,7 @@ int main(int argc, char **argv, char **envp)if (!icount_opts) {exit (1 ); } +replay_configure (icount_opts);break ;case QEMU_OPTION_incoming: incoming = optarg;
@@ -3786,6 +3800,12 @@ int main(int argc, char **argv, char **envp)exit (1 ); } +if (not_compatible_replay_param && (replay_mode !=REPLAY_MODE_NONE )) { +fprintf (stderr,"options -smp, -pflash, -chardev, -bt, -parallel " +"are not compatible with record/replay\n" ); +exit (1 ); + } +if (qemu_opts_foreach (qemu_find_opts ("sandbox" ), parse_sandbox,NULL ,0 )) {exit (1 ); }
@@ -4136,6 +4156,11 @@ int main(int argc, char **argv, char **envp)qemu_opts_del (icount_opts); } +if (replay_mode !=REPLAY_MODE_NONE && !use_icount) { +fprintf (stderr,"Please enable icount to use record/replay\n" ); +exit (1 ); + } +/* clean up network at qemu process termination */ atexit (&net_cleanup);
@@ -4172,7 +4197,7 @@ int main(int argc, char **argv, char **envp) }/* open the virtual block devices */ -if (snapshot) +if (snapshot || replay_mode !=REPLAY_MODE_NONE )qemu_opts_foreach (qemu_find_opts ("drive" ), drive_enable_snapshot,NULL ,0 );if (qemu_opts_foreach (qemu_find_opts ("drive" ), drive_init_func, &machine_class->block_default_type,1 ) !=0 ) {
Re: [Qemu-devel] [RFC PATCH v8 20/21] replay: command line options
Quoted 18 lines · Pavel Dovgalyuk
This patch introduces command line options for enabling recording or replaying virtual machine behavior. "-record" option starts recording of the execution and saves it into the log, specified with "fname" parameter. "-replay" option is intended for replaying previously saved log.
Signed-off-by Pavel Dovgalyuk <[email protected]>--- cpus.c | 3 ++- qemu-options.hx | 8 ++++++-- vl.c | 27 ++++++++++++++++++++++++++- 3 files changed, 34 insertions(+), 4 deletions(-)cpus.c +0
-0@@ -912,9 +912,10 @@ static void qemu_wait_io_event_common(CPUState *cpu)static void qemu_tcg_wait_io_event (void )
cpus.c static void qemu_wait_io_event_common(CPUState *cpu) · line 921
Quoted 10 lines
cpus.c +2
-1{CPUState *cpu; +GMainContext *context =g_main_context_default ();while (all_cpu_threads_idle ()) { -/* Start accounting real time to the virtual clock if the CPUs +/* Start accounting real time to the virtual clock if the CPUs are idle. */ qemu_clock_warp (QEMU_CLOCK_VIRTUAL );qemu_cond_wait (tcg_halt_cond, &qemu_global_mutex);
Extraneous change.
qemu-options.hx re-inject them. · line 3040
Quoted 10 lines
qemu-options.hx +1
-1@@ -3037,11 +3037,11 @@ re-inject them. ETEXI DEF("icount", HAS_ARG, QEMU_OPTION_icount, \ - "-icount [shift=N|auto][,align=on|off]\n" \ + "-icount [shift=N|auto][,align=on|off][,rr=record|replay,rrfname=<filename>]\n" \
rrfile?
Quoted 46 lines
qemu-options.hx +5
-1" enable virtual instruction counter with 2^N clock ticks per\n" \ " instruction and enable aligning the host and virtual clocks\n", QEMU_ARCH_ALL) STEXI -@item -icount [shift=@var{N}|auto] +@item -icount [shift=@var{N}|auto][,rr=record|replay,rrfname=@var{filename}] @findex -icount Enable virtual instruction counter. The virtual cpu will execute one instruction every 2^@var{N} ns of virtual time. If @code{auto} is specified@@ -3063,6 +3063,10 @@ Currently this option does not work when @option{shift} is @code{auto}. Note: The sync algorithm will work for those shift values for which the guest clock runs ahead of the host clock. Typically this happens when the shift value is high (how high depends on the host machine). + +When @option{rr} option is specified deterministic record/replay is enabled. +Replay log is written into @var{filename} file in record mode and +read from this file in replay mode. ETEXI DEF("watchdog", HAS_ARG, QEMU_OPTION_watchdog, \vl.c +8
-0@@ -475,6 +475,12 @@ static QemuOptsList qemu_icount_opts = { }, { .name = "align", .type = QEMU_OPT_BOOL, + }, { + .name = "rr", + .type = QEMU_OPT_STRING, + }, { + .name = "rrfname", + .type = QEMU_OPT_STRING, }, { /* end of list */ } },@@ -2752,6 +2758,8 @@ int main(int argc, char **argv, char **envp) {int i;int snapshot, linux_boot; +int not_compatible_replay_param =0 ; +const char *icount_option =NULL ;const char *initrd_filename;const char *kernel_filename, *kernel_cmdline;const char *boot_order;
vl.c int main(int argc, char **argv, char **envp) · line 2960
Quoted 4 lines
vl.c +1
-0@@ -2949,6 +2957,7 @@ int main(int argc, char **argv, char **envp)break ;case QEMU_OPTION_pflash:drive_add (IF_PFLASH ,-1 , optarg,PFLASH_OPTS ); + not_compatible_replay_param++;
Why not compatible?
vl.c int main(int argc, char **argv, char **envp) · line 3117
Quoted 8 lines
vl.c +1
-0break ;case QEMU_OPTION_snapshot: snapshot =1 ;@@ -3105,6 +3114,7 @@ int main(int argc, char **argv, char **envp)#endif case QEMU_OPTION_bt:add_device_config (DEV_BT , optarg); + not_compatible_replay_param++;
Could it be enough to add a migration blocker?
vl.c int main(int argc, char **argv, char **envp) · line 3257
Quoted 8 lines
vl.c +1
-0break ;case QEMU_OPTION_audio_help:AUD_help ();@@ -3244,6 +3254,7 @@ int main(int argc, char **argv, char **envp)if (!opts) {exit (1 ); } + not_compatible_replay_param++;
Why not compatible?
vl.c int main(int argc, char **argv, char **envp) · line 3386
Quoted 8 lines
vl.c +1
-0break ;case QEMU_OPTION_fsdev: olist =qemu_find_opts ("fsdev" );@@ -3372,6 +3383,7 @@ int main(int argc, char **argv, char **envp)if (strncmp (optarg,"mon:" ,4 ) ==0 ) { default_monitor =0 ; } + not_compatible_replay_param++;
Add a migration blocker?
vl.c int main(int argc, char **argv, char **envp) · line 3504
Quoted 8 lines
vl.c +1
-0break ;case QEMU_OPTION_debugcon:add_device_config (DEV_DEBUGCON , optarg);@@ -3489,6 +3501,7 @@ int main(int argc, char **argv, char **envp)if (!qemu_opts_parse (qemu_find_opts ("smp-opts" ), optarg,1 )) {exit (1 ); } + not_compatible_replay_param++;
Check smp_cpus instead in replay_configure?
vl.c int main(int argc, char **argv, char **envp) · line 3644
Quoted 8 lines
vl.c +1
-0break ;case QEMU_OPTION_vnc:#ifdef CONFIG_VNC @@ -3628,6 +3641,7 @@ int main(int argc, char **argv, char **envp)if (!icount_opts) {exit (1 ); } +replay_configure (icount_opts);
Perhaps move this outside the switch so that you have a fuller view of the command line?
vl.c int main(int argc, char **argv, char **envp) · line 3807
Quoted 12 lines
vl.c +5
-0break ;case QEMU_OPTION_incoming: incoming = optarg;@@ -3786,6 +3800,12 @@ int main(int argc, char **argv, char **envp)exit (1 ); } +if (not_compatible_replay_param && (replay_mode !=REPLAY_MODE_NONE )) { +fprintf (stderr,"options -smp, -pflash, -chardev, -bt, -parallel " +"are not compatible with record/replay\n" ); +exit (1 ); + }
... so this (or some replacement) can be done in replay_configure.
vl.c int main(int argc, char **argv, char **envp) · line 4162
Quoted 11 lines
vl.c +4
-0if (qemu_opts_foreach (qemu_find_opts ("sandbox" ), parse_sandbox,NULL ,0 )) {exit (1 ); }@@ -4136,6 +4156,11 @@ int main(int argc, char **argv, char **envp)qemu_opts_del (icount_opts); } +if (replay_mode !=REPLAY_MODE_NONE && !use_icount) { +fprintf (stderr,"Please enable icount to use record/replay\n" ); +exit (1 ); + }
Same here.
Paolo
Quoted 13 lines
vl.c +1
-1/* clean up network at qemu process termination */ atexit (&net_cleanup);@@ -4172,7 +4197,7 @@ int main(int argc, char **argv, char **envp) }/* open the virtual block devices */ -if (snapshot) +if (snapshot || replay_mode !=REPLAY_MODE_NONE )qemu_opts_foreach (qemu_find_opts ("drive" ), drive_enable_snapshot,NULL ,0 );if (qemu_opts_foreach (qemu_find_opts ("drive" ), drive_init_func, &machine_class->block_default_type,1 ) !=0 ) {
Re: [Qemu-devel] [RFC PATCH v8 20/21] replay: command line options
Quoted 7 lines
From: Paolo Bonzini [mailto:[email protected]]
Quoted 5 lines · Pavel Dovgalyuk
This patch introduces command line options for enabling recording or replaying virtual machine behavior. "-record" option starts recording of the execution and saves it into the log, specified with "fname" parameter. "-replay" option is intended for replaying previously saved log.
Quoted 11 lines
vl.c int main(int argc, char **argv, char **envp) · line 3117
Quoted 9 lines
Signed-off-by Pavel Dovgalyuk <[email protected]>break; case QEMU_OPTION_snapshot: snapshot = 1;vl.c +1
-0@@ -3105,6 +3114,7 @@ int main(int argc, char **argv, char **envp)#endif case QEMU_OPTION_bt:add_device_config (DEV_BT , optarg); + not_compatible_replay_param++;Could it be enough to add a migration blocker?
Record/replay core does not use migration subsystem. That is why it should check the hardware by itself.
Quoted 11 lines
vl.c int main(int argc, char **argv, char **envp) · line 3257
Quoted 9 lines
vl.c +1
-0break ;case QEMU_OPTION_audio_help:AUD_help ();@@ -3244,6 +3254,7 @@ int main(int argc, char **argv, char **envp)if (!opts) {exit (1 ); } + not_compatible_replay_param++;Why not compatible?
Replay for audio adapter will be added in latter patches. Trying to record/replay machine with audio using current set of patches will break the replay.
Pavel Dovgalyuk
Re: [Qemu-devel] [RFC PATCH v8 20/21] replay: command line options
Quoted 19 lines · Pavel Dovgaluk
Quoted 7 lines
From: Paolo Bonzini [mailto:[email protected]]
Quoted 5 lines · Pavel Dovgalyuk
This patch introduces command line options for enabling recording or replaying virtual machine behavior. "-record" option starts recording of the execution and saves it into the log, specified with "fname" parameter. "-replay" option is intended for replaying previously saved log.
Quoted 11 lines
vl.c int main(int argc, char **argv, char **envp) · line 3117
Quoted 9 lines
Signed-off-by Pavel Dovgalyuk <[email protected]>break; case QEMU_OPTION_snapshot: snapshot = 1;vl.c +1
-0@@ -3105,6 +3114,7 @@ int main(int argc, char **argv, char **envp)#endif case QEMU_OPTION_bt:add_device_config (DEV_BT , optarg); + not_compatible_replay_param++;Could it be enough to add a migration blocker?
Quoted 2 lines
Record/replay core does not use migration subsystem. That is why it should check the hardware by itself.
Right. But is there anything that supports migration but not replay? If the two overlap, you can just check migration blockers to see if record/replay can be allowed.
Quoted 12 lines
Quoted 11 lines
vl.c int main(int argc, char **argv, char **envp) · line 3257
Quoted 9 lines
vl.c +1
-0break ;case QEMU_OPTION_audio_help:AUD_help ();@@ -3244,6 +3254,7 @@ int main(int argc, char **argv, char **envp)if (!opts) {exit (1 ); } + not_compatible_replay_param++;Why not compatible?
Quoted 3 lines
Replay for audio adapter will be added in latter patches. Trying to record/replay machine with audio using current set of patches will break the replay.
For this case you can try adding a mechanism similar to migration blockers (replay blockers).
Paolo
Re: [Qemu-devel] [RFC PATCH v8 20/21] replay: command line options
Quoted 14 lines
From: Paolo Bonzini [mailto:[email protected]]
Quoted 12 lines · Pavel Dovgaluk
Quoted 11 lines
From: Paolo Bonzini [mailto:[email protected]]
vl.c int main(int argc, char **argv, char **envp) · line 3257
Quoted 8 lines
break; case QEMU_OPTION_audio_help: AUD_help ();vl.c +1
-0@@ -3244,6 +3254,7 @@ int main(int argc, char **argv, char **envp)if (!opts) {exit (1 ); } + not_compatible_replay_param++;Why not compatible?
Quoted 6 lines
Quoted 3 lines
Replay for audio adapter will be added in latter patches. Trying to record/replay machine with audio using current set of patches will break the replay.
For this case you can try adding a mechanism similar to migration blockers (replay blockers).
Do you mean adding flag to vmsd similar to unmigratable?
Pavel Dovgalyuk
Re: [Qemu-devel] [RFC PATCH v8 20/21] replay: command line options
Quoted 8 lines · Pavel Dovgaluk
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Replay for audio adapter will be added in latter patches. Trying to record/replay machine with audio using current set of patches will break the replay.
For this case you can try adding a mechanism similar to migration blockers (replay blockers).
Do you mean adding flag to vmsd similar to unmigratable?
No, see this in migration/migration.c:
static GSList *migration_blockers;
void migrate_add_blocker(Error *reason)
{
migration_blockers = g_slist_prepend(migration_blockers, reason);
}void migrate_del_blocker(Error *reason)
{
migration_blockers = g_slist_remove(migration_blockers, reason);
}...
if (migration_blockers) {
*errp = error_copy(migration_blockers->data);
return;
}Paolo
RFC PATCH v8 21/21 replay: recording of the user input
In reply to Pavel DovgalyukThis records user input (keyboard and mouse events) in record mode and replays these input events in replay mode.
--- include/ui/input.h | 2 + replay/Makefile.objs | 1 replay/replay-events.c | 55 +++++++++++++++++++ replay/replay-input.c | 130 ++++++++++++++++++++++++++++++++++++++++++++++ replay/replay-internal.h | 13 +++++ replay/replay.h | 5 ++ ui/input.c | 68 +++++++++++++++++------- 7 files changed, 255 insertions(+), 19 deletions(-) create mode 100755 replay/replay-input.c
include/ui/input.h +2 -0
@@ -33,7 +33,9 @@ void qemu_input_handler_bind(QemuInputHandlerState *s, const char *device_id, int head, Error **errp);void qemu_input_event_send (QemuConsole *src,InputEvent *evt); +void qemu_input_event_send_impl (QemuConsole *src,InputEvent *evt);void qemu_input_event_sync (void ); +void qemu_input_event_sync_impl (void );InputEvent *qemu_input_event_new_key (KeyValue *key,bool down);void qemu_input_event_send_key (QemuConsole *src,KeyValue *key,bool down);
replay/Makefile.objs +1 -0
@@ -2,3 +2,4 @@ obj-y += replay.o obj-y += replay-internal.o obj-y += replay-events.o obj-y += replay-time.o +obj-y += replay-input.o
replay/replay-events.c +55 -0
91 lines not on this page · raw message
replay/replay-input.c new file +130 -0
130 lines not on this page · raw message
replay/replay-internal.h +13 -0
@@ -44,6 +44,8 @@ enum ReplayEvents {enum ReplayAsyncEventKind {REPLAY_ASYNC_EVENT_BH ,REPLAY_ASYNC_EVENT_THREAD , +REPLAY_ASYNC_EVENT_INPUT , +REPLAY_ASYNC_EVENT_INPUT_SYNC ,REPLAY_ASYNC_COUNT };
@@ -133,4 +135,15 @@ void replay_read_events(int opt);/*! Adds specified async event to the queue */ void replay_add_event (ReplayAsyncEventKind event_id,void *opaque); +/* Input events */ + +/*! Saves input event to the log */ +void replay_save_input_event (InputEvent *evt); +/*! Reads input event from the log */ +InputEvent *replay_read_input_event (void ); +/*! Adds input event to the queue */ +void replay_add_input_event (struct InputEvent *event); +/*! Adds input sync event to the queue */ +void replay_add_input_sync_event (void ); +#endif
replay/replay.h +5 -0
@@ -16,6 +16,7 @@#include <stdint.h> #include <time.h> #include "qapi-types.h" +#include "qemu/typedefs.h" struct QemuOpts ;
@@ -123,5 +124,9 @@ void replay_disable_events(void);void replay_add_bh_event (void *bh,uint64_t id);/*! Adds thread event to the queue */ void replay_add_thread_event (void *pool,void *req,uint64_t id); +/*! Adds input event to the queue */ +void replay_input_event (QemuConsole *src,InputEvent *evt); +/*! Adds input sync event to the queue */ +void replay_input_sync_event (void );#endif
ui/input.c +49 -19
130 lines not on this page · raw message
Re: [Qemu-devel] [RFC PATCH v8 21/21] replay: recording of the user input
replay/replay-input.c line 115
Quoted 6 lines · Pavel Dovgalyuk
replay/replay-input.c new file +6
-0+void replay_input_event (QemuConsole *src,InputEvent *evt) +{ +if (replay_mode ==REPLAY_MODE_PLAY ) { +/* Nothing */ + }else if (replay_mode ==REPLAY_MODE_RECORD ) { +replay_add_input_event (evt);
Does replay_add_input_event ultimately call qemu_input_event_send_impl?
replay/replay-input.c line 120
Quoted 5 lines
replay/replay-input.c new file +5
-0+ }else { +qemu_input_event_send_impl (src, evt); + } +} +
Perhaps make this and replay_input_sync_event return a bool and in the caller do:
if (replay_input_event(src, evt)) {
qemu_input_event_send_impl(src, evt):
}ui/input.c InputEvent *qemu_input_event_new_key(KeyValue *key, bool down) · line 377
Quoted 8 lines
ui/input.c +8
-0+if (replay_mode !=REPLAY_MODE_PLAY ) { + evt =qemu_input_event_new_key (key, down); +if (QTAILQ_EMPTY (&kbd_queue)) { +qemu_input_event_send (src, evt); +qemu_input_event_sync (); +if (replay_mode !=REPLAY_MODE_RECORD ) { +qapi_free_InputEvent (evt); + }
This is wrong. You have different lifetimes for different modes. Please make a copy of the event in the implementation of record mode.
Also, you do not need the "if" for replay mode. The functions would just do nothing.
ui/input.c InputEvent *qemu_input_event_new_key(KeyValue *key, bool down) · line 382
Quoted 5 lines
ui/input.c +5
-0+ }else { +if (replay_mode !=REPLAY_MODE_NONE ) { +fprintf (stderr,"Input queue is not supported " +"in record/replay mode\n" ); +exit (1 );
Why? For record mode should just work since qemu_input_event_send is called in qemu_input_queue_process.
Replay mode can just do nothing, by returning early from qemu_input_queue_event/qemu_input_queue_sync.
Paolo
Quoted 5 lines
ui/input.c +4
-0+ } +qemu_input_queue_event (&kbd_queue, src, evt); +qemu_input_queue_sync (&kbd_queue); + } }
Re: [Qemu-devel] [RFC PATCH v8 21/21] replay: recording of the user input
Quoted 10 lines
From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini
replay/replay-input.c line 115
Quoted 6 lines · Pavel Dovgalyuk
replay/replay-input.c new file +6
-0+void replay_input_event (QemuConsole *src,InputEvent *evt) +{ +if (replay_mode ==REPLAY_MODE_PLAY ) { +/* Nothing */ + }else if (replay_mode ==REPLAY_MODE_RECORD ) { +replay_add_input_event (evt);Does
replay_add_input_eventultimately callqemu_input_event_send_impl?
No, it just adds event to the queue.
Quoted 9 lines
replay/replay-input.c line 120
Quoted 5 lines
replay/replay-input.c new file +5
-0+ }else { +qemu_input_event_send_impl (src, evt); + } +} +Perhaps make this and
replay_input_sync_eventreturn a bool and in the caller do:
Quoted 3 lines
if (replay_input_event(src, evt)) {
qemu_input_event_send_impl(src, evt):
}No, we can't. qemu_input_event_send_impl is called when the queue is saved to the log.
Pavel Dovgalyuk
Re: [Qemu-devel] [RFC PATCH v8 21/21] replay: recording of the user input
In reply to Paolo BonziniQuoted 11 lines
From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini
ui/input.c InputEvent *qemu_input_event_new_key(KeyValue *key, bool down) · line 377
Quoted 8 lines · Pavel Dovgalyuk
ui/input.c +8
-0+if (replay_mode !=REPLAY_MODE_PLAY ) { + evt =qemu_input_event_new_key (key, down); +if (QTAILQ_EMPTY (&kbd_queue)) { +qemu_input_event_send (src, evt); +qemu_input_event_sync (); +if (replay_mode !=REPLAY_MODE_RECORD ) { +qapi_free_InputEvent (evt); + }
Quoted 2 lines
This is wrong. You have different lifetimes for different modes. Please make a copy of the event in the implementation of record mode.
What is the correct way for cloning the QAPI type?
I should invent the cloning visitor or just create a switch for correct cloning of the InputEvent union?
Pavel Dovgalyuk
Re: [Qemu-devel] [RFC PATCH v8 21/21] replay: recording of the user input
Quoted 5 lines · Pavel Dovgaluk
Quoted 3 lines
Quoted 3 lines
This is wrong. You have different lifetimes for different modes. Please make a copy of the event in the implementation of record mode.
What is the correct way for cloning the QAPI type?
I should invent the cloning visitor or just create a switch for correct cloning of the InputEvent union?
You can use the existing visitors to clone objects, see qapi_copy_SocketAddress in qemu-char.c.
Paolo
Ping?
Pavel Dovgalyuk
Quoted 225 lines
-----Original Message-----
From: Pavel Dovgalyuk [mailto:[email protected]]
Sent: Thursday, January 22, 2015 11:52 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [RFC PATCH v8 00/21] Deterministic replay coreThis set of patches is related to the reverse execution and deterministic replay of qemu execution This implementation of deterministic replay can be used for deterministic debugging of guest code through gdb remote interface.
These patches include only core function of the replay, excluding the support for replaying serial, audio, network, and USB devices' operations. Reverse debugging and monitor commands were also excluded to be submitted later as separate patches.
Execution recording writes non-deterministic events log, which can be later used for replaying the execution anywhere and for unlimited number of times. It also supports checkpointing for faster rewinding during reverse debugging. Execution replaying reads the log and replays all non-deterministic events including external input, hardware clocks, and interrupts.
Deterministic replay has the following features:
* Deterministically replays whole system execution and all contents of the memory, state of the hadrware devices, clocks, and screen of the VM.
* Writes execution log into the file for latter replaying for multiple times on different machines.
* Supports i386,x86_64, and ARM hardware platforms.
* Performs deterministic replay of all operations with keyboard and mouse input devices.
* Supports auto-checkpointing for convenient reverse debugging.Usage of the record/replay:
* First, record the execution, by adding the following string to the command line:
'-record fname=replay.bin -icount 7 -net none'. Block devices' images are not actually changed in the recording mode, because all of the changes are written to the temporary overlay file.
* Then you can replay it for the multiple times by using another command line option: '-replay fname=replay.bin -icount 7 -net none'
* '-net none' option should also be specified if network replay patches are not applied.Paper with short description of deterministic replay implementation:
http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.htmlModifications of qemu include:
* wrappers for clock and time functions to save their return values in the log
* saving different asynchronous events (e.g. system shutdown) into the log
* synchronization of the bottom halves execution
* synchronization of the threads from thread pool
* recording/replaying user input (mouse and keyboard)
* adding internal events for cpu and io synchronizationv8 changes:
* Simplified processing of the shutdown event (as suggested by Paolo Bonzini)
* Replaced stack of bottom halves in AIO context with QSIMPLEQ (as suggested by Paolo
Bonzini)
* Movedreplay_submodeout of the series (as suggested by Paolo Bonzini)
* Moved suffix option out of the series
* Converted some of the defines into enums (as suggested by Paolo Bonzini)
* Encapsulated save_tm/read_tm calls into the single function (as suggested by Paolo Bonzini)
* Moved record/replay options to icount group (as suggested by Paolo Bonzini)
* Updated mutex protection for the events queue (as suggested by Paolo Bonzini)
* Added mutex to protect replay log file (as suggested by Paolo Bonzini)
* Minor cleanupsv7 changes:
* Removed patches that were applied to upstream.v6 changes:
* Fixed replay stub return value (as suggested by Eric Blake)
* Fixed icount warping.
* Virtual rt clock now usescpu_get_clock()(as suggested by Paolo Bonzini)
* Replatedget_clock_realtimeandget_clockcalls with qemu clock requests (as suggested by
Paolo Bonzini)
* Modifiedcan_do_iologic to allow requesting icount fromcpu_execfunction (as suggested by
Paolo Bonzini)
* Removed applied patches.v5 changes:
* Minor changes.
* Used fixed-width integer types for read/write functions (as suggested by Alex Bennee)
* Moved savevm-related code out of the core.
* Added new traced clock for deterministic virtual clock warping (as suggested by Paolo
Bonzini)
* Fixedexception_indexreset for user mode (as suggested by Paolo Bonzini)
* Adopted Paolo's icount patches
* Fixed hardware interrupts replayingv4 changes:
* Updated block drivers to support newbdrv_openinterface.
* Moved migration patches into separate series (as suggested by Paolo Bonzini)
* Fixed a bug inreplay_breakoperation.
* Fixed rtl8139 migration for replay.
* Fixed 'period' parameter processing for record mode.
* Fixed bug in 'reverse-stepi' implementation.
* Fixed replay without making any snapshots (even the starting one).
* Moved core replay patches into the separate series.
* Fixed reverse step and reverse continue support.
* Fixed several bugs in icount subsystem.
* Reusing native qemu icount for replay instructions counting.
* Separated core patches into their own series.v3 changes:
* Fixed bug with replay of the aio write operations.
* Added virtual clock based on replay icount.
* Removed duplicated saving ofinterrupt_requestCPU field.
* Fixed some coding style issues.
* Renamed QMP commands for controlling reverse execution (as suggested by Eric Blake)
* Replay mode and submode implemented as QAPI enumerations (as suggested by Eric Blake)
* Added description and example for replay-info command (as suggested by Eric Blake)
* Added information about the current breakpoint to the output of replay-info (as suggested
by Eric Blake)
* Updated version id for HPET vmstate (as suggested by Paolo Bonzini)
* Removed static fields from parallel vmstate (as suggested by Paolo Bonzini)
* New vmstate fields for mc146818rtc, pckbd, kvmapic, serial, fdc, rtl8139 moved to
subsection (as suggested by Paolo Bonzini)
* Disabled textmode cursor blinking, when virtual machine is stopped (as suggested by Paolo
Bonzini)
* Extracted saving ofexception_indexto separate patch (as suggested by Paolo Bonzini)v2 changes:
* Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
* Added QMP versions of replay commands (as suggested by Eric Blake)
* Removed some optional features of replay to make patches cleaner
* Minor changes and code cleanup were made---
Pavel Dovgalyuk (21):
i386: partial revert of interrupt poll fix
replay: global variables and function stubs
sysemu: system functions for replay
replay: internal functions for replay log
replay: introduce mutex to protect the replay log
replay: introduce icount event
cpu-exec: allow temporary disabling icount
cpu: replay instructions sequence
replay: interrupts and exceptions
replay: asynchronous events infrastructure
replay: recording and replaying clock ticks
replay: recording and replaying different timers
replay: shutdown event
replay: checkpoints
aio: replace stack of bottom halves with queue
replay: bottom halves
replay: replay aio requests
replay: thread pool
replay: initialization and deinitialization
replay: command line options
replay: recording of the user inputMakefile.target | 1 async.c | 49 ++++-- block.c | 92 ++++++++++- block/block-backend.c | 30 +++- block/qcow2.c | 4 block/raw-posix.c | 6 - block/raw-win32.c | 4 cpu-exec.c | 60 ++++++- cpus.c | 45 ++++- dma-helpers.c | 10 + exec.c | 1 hw/block/virtio-blk.c | 10 + hw/ide/ahci.c | 4 hw/ide/atapi.c | 10 + hw/ide/core.c | 18 +- hw/timer/arm_timer.c | 2 hw/timer/mc146818rtc.c | 3 hw/timer/pl031.c | 3 hw/usb/hcd-uhci.c | 2 include/block/aio.h | 22 ++- include/block/block.h | 15 ++ include/block/thread-pool.h | 4 include/exec/exec-all.h | 2 include/qemu-common.h | 3 include/qemu/main-loop.h | 1 include/qemu/queue.h | 7 + include/qemu/timer.h | 16 ++ include/qom/cpu.h | 10 + include/sysemu/block-backend.h | 10 + include/sysemu/cpus.h | 1 include/ui/input.h | 2 main-loop.c | 10 + qapi-schema.json | 18 ++ qemu-io-cmds.c | 2 qemu-options.hx | 8 + qemu-timer.c | 56 +++++-- replay/Makefile.objs | 5 + replay/replay-events.c | 310 ++++++++++++++++++++++++++++++++++++ replay/replay-input.c | 130 +++++++++++++++ replay/replay-internal.c | 183 +++++++++++++++++++++ replay/replay-internal.h | 149 +++++++++++++++++ replay/replay-time.c | 216 +++++++++++++++++++++++++ replay/replay.c | 341 ++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 132 +++++++++++++++ stubs/Makefile.objs | 1 stubs/replay.c | 37 ++++ target-i386/cpu.c | 10 - target-i386/seg_helper.c | 4 tests/test-thread-pool.c | 7 - thread-pool.c | 49 ++++-- trace-events | 2 translate-all.c | 10 + ui/input.c | 68 ++++++-- util/iov.c | 4 vl.c | 54 ++++++ 55 files changed, 2101 insertions(+), 152 deletions(-) create mode 100755 replay/Makefile.objs create mode 100755 replay/replay-events.c create mode 100755 replay/replay-input.c create mode 100755 replay/replay-internal.c create mode 100755 replay/replay-internal.h create mode 100755 replay/replay-time.c create mode 100755 replay/replay.c create mode 100755 replay/replay.h create mode 100755 stubs/replay.c--
Pavel Dovgalyuk
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 1 line · Pavel Dovgaluk
Ping?
Reviewed 13 patches out of 21. Made some comments, but overall I'm really pleased. Thanks for persisting!
Will continue tomorrow.
Paolo
Quoted 230 lines
Pavel Dovgalyuk
Quoted 225 lines
-----Original Message-----
From: Pavel Dovgalyuk [mailto:[email protected]]
Sent: Thursday, January 22, 2015 11:52 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [RFC PATCH v8 00/21] Deterministic replay coreThis set of patches is related to the reverse execution and deterministic replay of qemu execution This implementation of deterministic replay can be used for deterministic debugging of guest code through gdb remote interface.
These patches include only core function of the replay, excluding the support for replaying serial, audio, network, and USB devices' operations. Reverse debugging and monitor commands were also excluded to be submitted later as separate patches.
Execution recording writes non-deterministic events log, which can be later used for replaying the execution anywhere and for unlimited number of times. It also supports checkpointing for faster rewinding during reverse debugging. Execution replaying reads the log and replays all non-deterministic events including external input, hardware clocks, and interrupts.
Deterministic replay has the following features:
* Deterministically replays whole system execution and all contents of the memory, state of the hadrware devices, clocks, and screen of the VM.
* Writes execution log into the file for latter replaying for multiple times on different machines.
* Supports i386,x86_64, and ARM hardware platforms.
* Performs deterministic replay of all operations with keyboard and mouse input devices.
* Supports auto-checkpointing for convenient reverse debugging.Usage of the record/replay:
* First, record the execution, by adding the following string to the command line:
'-record fname=replay.bin -icount 7 -net none'. Block devices' images are not actually changed in the recording mode, because all of the changes are written to the temporary overlay file.
* Then you can replay it for the multiple times by using another command line option: '-replay fname=replay.bin -icount 7 -net none'
* '-net none' option should also be specified if network replay patches are not applied.Paper with short description of deterministic replay implementation:
http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.htmlModifications of qemu include:
* wrappers for clock and time functions to save their return values in the log
* saving different asynchronous events (e.g. system shutdown) into the log
* synchronization of the bottom halves execution
* synchronization of the threads from thread pool
* recording/replaying user input (mouse and keyboard)
* adding internal events for cpu and io synchronizationv8 changes:
* Simplified processing of the shutdown event (as suggested by Paolo Bonzini)
* Replaced stack of bottom halves in AIO context with QSIMPLEQ (as suggested by Paolo
Bonzini)
* Movedreplay_submodeout of the series (as suggested by Paolo Bonzini)
* Moved suffix option out of the series
* Converted some of the defines into enums (as suggested by Paolo Bonzini)
* Encapsulated save_tm/read_tm calls into the single function (as suggested by Paolo Bonzini)
* Moved record/replay options to icount group (as suggested by Paolo Bonzini)
* Updated mutex protection for the events queue (as suggested by Paolo Bonzini)
* Added mutex to protect replay log file (as suggested by Paolo Bonzini)
* Minor cleanupsv7 changes:
* Removed patches that were applied to upstream.v6 changes:
* Fixed replay stub return value (as suggested by Eric Blake)
* Fixed icount warping.
* Virtual rt clock now usescpu_get_clock()(as suggested by Paolo Bonzini)
* Replatedget_clock_realtimeandget_clockcalls with qemu clock requests (as suggested by
Paolo Bonzini)
* Modifiedcan_do_iologic to allow requesting icount fromcpu_execfunction (as suggested by
Paolo Bonzini)
* Removed applied patches.v5 changes:
* Minor changes.
* Used fixed-width integer types for read/write functions (as suggested by Alex Bennee)
* Moved savevm-related code out of the core.
* Added new traced clock for deterministic virtual clock warping (as suggested by Paolo
Bonzini)
* Fixedexception_indexreset for user mode (as suggested by Paolo Bonzini)
* Adopted Paolo's icount patches
* Fixed hardware interrupts replayingv4 changes:
* Updated block drivers to support newbdrv_openinterface.
* Moved migration patches into separate series (as suggested by Paolo Bonzini)
* Fixed a bug inreplay_breakoperation.
* Fixed rtl8139 migration for replay.
* Fixed 'period' parameter processing for record mode.
* Fixed bug in 'reverse-stepi' implementation.
* Fixed replay without making any snapshots (even the starting one).
* Moved core replay patches into the separate series.
* Fixed reverse step and reverse continue support.
* Fixed several bugs in icount subsystem.
* Reusing native qemu icount for replay instructions counting.
* Separated core patches into their own series.v3 changes:
* Fixed bug with replay of the aio write operations.
* Added virtual clock based on replay icount.
* Removed duplicated saving ofinterrupt_requestCPU field.
* Fixed some coding style issues.
* Renamed QMP commands for controlling reverse execution (as suggested by Eric Blake)
* Replay mode and submode implemented as QAPI enumerations (as suggested by Eric Blake)
* Added description and example for replay-info command (as suggested by Eric Blake)
* Added information about the current breakpoint to the output of replay-info (as suggested
by Eric Blake)
* Updated version id for HPET vmstate (as suggested by Paolo Bonzini)
* Removed static fields from parallel vmstate (as suggested by Paolo Bonzini)
* New vmstate fields for mc146818rtc, pckbd, kvmapic, serial, fdc, rtl8139 moved to
subsection (as suggested by Paolo Bonzini)
* Disabled textmode cursor blinking, when virtual machine is stopped (as suggested by Paolo
Bonzini)
* Extracted saving ofexception_indexto separate patch (as suggested by Paolo Bonzini)v2 changes:
* Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
* Added QMP versions of replay commands (as suggested by Eric Blake)
* Removed some optional features of replay to make patches cleaner
* Minor changes and code cleanup were made---
Pavel Dovgalyuk (21):
i386: partial revert of interrupt poll fix
replay: global variables and function stubs
sysemu: system functions for replay
replay: internal functions for replay log
replay: introduce mutex to protect the replay log
replay: introduce icount event
cpu-exec: allow temporary disabling icount
cpu: replay instructions sequence
replay: interrupts and exceptions
replay: asynchronous events infrastructure
replay: recording and replaying clock ticks
replay: recording and replaying different timers
replay: shutdown event
replay: checkpoints
aio: replace stack of bottom halves with queue
replay: bottom halves
replay: replay aio requests
replay: thread pool
replay: initialization and deinitialization
replay: command line options
replay: recording of the user inputMakefile.target | 1 async.c | 49 ++++-- block.c | 92 ++++++++++- block/block-backend.c | 30 +++- block/qcow2.c | 4 block/raw-posix.c | 6 - block/raw-win32.c | 4 cpu-exec.c | 60 ++++++- cpus.c | 45 ++++- dma-helpers.c | 10 + exec.c | 1 hw/block/virtio-blk.c | 10 + hw/ide/ahci.c | 4 hw/ide/atapi.c | 10 + hw/ide/core.c | 18 +- hw/timer/arm_timer.c | 2 hw/timer/mc146818rtc.c | 3 hw/timer/pl031.c | 3 hw/usb/hcd-uhci.c | 2 include/block/aio.h | 22 ++- include/block/block.h | 15 ++ include/block/thread-pool.h | 4 include/exec/exec-all.h | 2 include/qemu-common.h | 3 include/qemu/main-loop.h | 1 include/qemu/queue.h | 7 + include/qemu/timer.h | 16 ++ include/qom/cpu.h | 10 + include/sysemu/block-backend.h | 10 + include/sysemu/cpus.h | 1 include/ui/input.h | 2 main-loop.c | 10 + qapi-schema.json | 18 ++ qemu-io-cmds.c | 2 qemu-options.hx | 8 + qemu-timer.c | 56 +++++-- replay/Makefile.objs | 5 + replay/replay-events.c | 310 ++++++++++++++++++++++++++++++++++++ replay/replay-input.c | 130 +++++++++++++++ replay/replay-internal.c | 183 +++++++++++++++++++++ replay/replay-internal.h | 149 +++++++++++++++++ replay/replay-time.c | 216 +++++++++++++++++++++++++ replay/replay.c | 341 ++++++++++++++++++++++++++++++++++++++++ replay/replay.h | 132 +++++++++++++++ stubs/Makefile.objs | 1 stubs/replay.c | 37 ++++ target-i386/cpu.c | 10 - target-i386/seg_helper.c | 4 tests/test-thread-pool.c | 7 - thread-pool.c | 49 ++++-- trace-events | 2 translate-all.c | 10 + ui/input.c | 68 ++++++-- util/iov.c | 4 vl.c | 54 ++++++ 55 files changed, 2101 insertions(+), 152 deletions(-) create mode 100755 replay/Makefile.objs create mode 100755 replay/replay-events.c create mode 100755 replay/replay-input.c create mode 100755 replay/replay-internal.c create mode 100755 replay/replay-internal.h create mode 100755 replay/replay-time.c create mode 100755 replay/replay.c create mode 100755 replay/replay.h create mode 100755 stubs/replay.c--
Pavel Dovgalyuk
Quoted 9 lines · Paolo Bonzini
Quoted 1 line · Pavel Dovgaluk
Ping?
Reviewed 13 patches out of 21. Made some comments, but overall I'm really pleased. Thanks for persisting!
Will continue tomorrow.
The second part has a few more problems:
1) some simple and stylistic (patch 15, 19-21)
2) the invasive changes in patches 16-18 are, well, invasive. Lacking a design and example document it is hard to review them. The block maintainers, also, probably have better ideas of what to do.
3) A design document is also needed for checkpoints (patch 14)
Paolo
Quoted 4 lines · Pavel Dovgalyuk
These patches include only core function of the replay, excluding the support for replaying serial, audio, network, and USB devices' operations. Reverse debugging and monitor commands were also excluded to be submitted later as separate patches.
BTW, I am interested in the serial part of the RR functionality, since it is probably simpler than block.
Paolo
Quoted 9 lines
From: Paolo Bonzini [mailto:[email protected]] On Behalf Of Paolo Bonzini
Quoted 4 lines · Pavel Dovgalyuk
These patches include only core function of the replay, excluding the support for replaying serial, audio, network, and USB devices' operations. Reverse debugging and monitor commands were also excluded to be submitted later as separate patches.
BTW, I am interested in the serial part of the RR functionality, since it is probably simpler than block.
Right, it's quite simple.
But replaying block layer gives us a usable version of record/replay. That is why serial devices replay is not included in the very first series.
Pavel Dovgalyuk