Fixing record/replay and adding reverse debugging
v13 cover Fixing record/replay and adding reverse debugging
GDB remote protocol supports reverse debugging of the targets. It includes 'reverse step' and 'reverse continue' operations. The first one finds the previous step of the execution, and the second one is intended to stop at the last breakpoint that would happen when the program is executed normally.
Reverse debugging is possible in the replay mode, when at least one snapshot was created at the record or replay phase. QEMU can use these snapshots for travelling back in time with GDB.
Running the execution in replay mode allows using GDB reverse debugging commands:
- reverse-stepi (or rsi): Steps one instruction to the past. QEMU loads on of the prior snapshots and proceeds to the desired instruction forward. When that step is reaches, execution stops.
- reverse-continue (or rc): Runs execution "backwards". QEMU tries to find breakpoint or watchpoint by loaded prior snapshot and replaying the execution. Then QEMU loads snapshots again and replays to the latest breakpoint. When there are no breakpoints in the examined section of the execution, QEMU finds one more snapshot and tries again. After the first snapshot is processed, execution stops at this snapshot.
The set of patches include the following modifications:
- gdbstub update for reverse debugging support
- functions that automatically perform reverse step and reverse continue operations
- hmp/qmp commands for manipulating the replay process
- improvement of the snapshotting for saving the execution step in the snapshot parameters
- other record/replay fixes
The patches are available in the repository:
https://github.com/ispras/qemu/tree/rr-190221
v13 changes:
- rebased to make QAPI stuff applicable
- minor reverse step/reverse continue fix
v12 changes:
- style fixes
v11 changes:
- added can_do_io resetting before jumping to the next block in the chain
- rebase to the latest master
v10 changes:
- added patch for correct deadline calculation with external timers
- updated icount-related documentation in json files (suggested by Markus Armbruster)
- fixed replay shutdown
- renamed some functions and variables to make them consistent with the documentation and displayed messages
- minor changes
v9 changes:
- moved rr qapi stuff to the separate file (suggested by Markus Armbruster)
- minor coding style fixes
v8 changes:
- rebased to the new master
- added missing fix for prior rr patch
- updated 'since' version number in json-related patches
v7 changes:
- rebased to the new master with upstreamed patches from the series
- several improvements in hmp/qmp commands handling (suggested by Markus Armbruster)
- fixed record/replay with '-rtc base' option enabled
- added document with virtual hardware requirements
v6 changes:
- rebased to the new version of master
- fixed build of linux-user configurations
- added new clock for slirp and vnc timers
v5 changes:
- multiple fixes of record/replay bugs appeared after QEMU core update
- changed reverse debugging to 'since 3.1'
v4 changes:
- changed 'since 2.13' to 'since 3.0' in json (as suggested by Eric Blake)
v3 changes:
- Fixed PS/2 bug with save/load vm, which caused failures of the replay.
- Rebased to the new code base.
- Minor fixes.
v2 changes:
- documented reverse debugging
- fixed start vmstate loading in record mode
- documented qcow2 changes (as suggested by Eric Blake)
- made icount SnapshotInfo field optional (as suggested by Eric Blake)
- renamed qmp commands (as suggested by Eric Blake)
- minor changes
---
Pavel Dovgalyuk (24):
block: implement bdrv_snapshot_goto for blkreplay
replay: disable default snapshot for record/replay
replay: update docs for record/replay with block devices
replay: don't drain/flush bdrv queue while RR is working
replay: finish record/replay before closing the disks
qcow2: introduce icount field for snapshots
migration: introduce icount field for snapshots
replay: provide an accessor for rr filename
qapi: introduce replay.json for record/replay-related stuff
replay: introduce info hmp/qmp command
replay: introduce breakpoint at the specified step
replay: implement replay-seek command
replay: refine replay-time module
replay: flush rr queue before loading the vmstate
gdbstub: add reverse step support in replay mode
gdbstub: add reverse continue support in replay mode
replay: describe reverse debugging in docs/replay.txt
replay: add BH oneshot event for block layer
replay: init rtc after enabling the replay
replay: document development rules
util/qemu-timer: refactor deadline calculation for external timers
replay: fix replay shutdown
replay: rename step-related variables and functions
icount: clean up cpu_can_io before jumping to the next block
[email protected] (1):
replay: add missing fix for internal function
MAINTAINERS | 1 accel/tcg/tcg-runtime.c | 2 accel/tcg/translator.c | 1 block/blkreplay.c | 8 + block/block-backend.c | 8 + block/io.c | 32 ++++ block/iscsi.c | 5 - block/nfs.c | 5 - block/null.c | 4 - block/nvme.c | 6 + block/qapi.c | 18 ++ block/qcow2-snapshot.c | 9 + block/qcow2.h | 2 block/rbd.c | 5 - block/vxhs.c | 5 - blockdev.c | 10 + cpus.c | 30 +++- docs/devel/replay.txt | 46 ++++++ docs/interop/qcow2.txt | 4 + docs/replay.txt | 45 ++++++ exec.c | 8 + gdbstub.c | 52 +++++++ hmp-commands-info.hx | 14 ++ hmp-commands.hx | 53 +++++++ hmp.h | 4 + include/block/snapshot.h | 1 include/qemu/timer.h | 7 - include/sysemu/replay.h | 30 ++++ migration/savevm.c | 11 ++ qapi/Makefile.objs | 2 qapi/block-core.json | 8 + qapi/block.json | 3 qapi/misc.json | 18 -- qapi/qapi-schema.json | 1 qapi/replay.json | 121 +++++++++++++++++ qtest.c | 2 replay/Makefile.objs | 3 replay/replay-debugging.c | 327 +++++++++++++++++++++++++++++++++++++++++++++ replay/replay-events.c | 18 ++ replay/replay-internal.c | 10 + replay/replay-internal.h | 17 +- replay/replay-snapshot.c | 6 - replay/replay-time.c | 36 ++--- replay/replay.c | 56 ++++++-- stubs/Makefile.objs | 1 stubs/replay-user.c | 9 + stubs/replay.c | 10 + tests/ptimer-test-stubs.c | 2 tests/ptimer-test.c | 4 - util/qemu-timer.c | 41 +++++- vl.c | 21 ++- 51 files changed, 1009 insertions(+), 133 deletions(-) create mode 100644 docs/devel/replay.txt create mode 100644 qapi/replay.json create mode 100644 replay/replay-debugging.c create mode 100644 stubs/replay-user.c
Signature · 1 line
Pavel Dovgalyuk
PATCH v13 1/25 replay: add missing fix for internal function
From: [email protected] <[email protected]>
This is a fix which was missed by patch 74c0b816adfc6aa1b01b4426fdf385e32e35cbac, which added current_step parameter to the replay_advance_current_step function.
--- replay/replay-internal.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
replay/replay-internal.c +1 -1
@@ -229,7 +229,7 @@ void replay_mutex_unlock(void)void replay_advance_current_step (uint64_t current_step) { -int diff = (int )(replay_get_current_step () - replay_state.current_step); +int diff = (int )(current_step - replay_state.current_step);/* Time can only go forward */ assert (diff >=0 );
PATCH v13 2/25 block: implement bdrv_snapshot_goto for blkreplay
In reply to Pavel DovgalyukFrom: Pavel Dovgalyuk <[email protected]>
This patch enables making snapshots with blkreplay used in block devices.
This function is required to make bdrv_snapshot_goto without calling .bdrv_open which is not implemented.
--- block/blkreplay.c | 8 ++++++++ 1 file changed, 8 insertions(+)
block/blkreplay.c +8 -0
@@ -126,6 +126,12 @@ static int coroutine_fn blkreplay_co_flush(BlockDriverState *bs)return ret; } +static int blkreplay_snapshot_goto (BlockDriverState *bs, +const char *snapshot_id) +{ +return bdrv_snapshot_goto (bs->file->bs, snapshot_id,NULL ); +} +static BlockDriver bdrv_blkreplay = { .format_name ="blkreplay" , .instance_size =0 ,
@@ -140,6 +146,8 @@ static BlockDriver bdrv_blkreplay = { .bdrv_co_pwrite_zeroes = blkreplay_co_pwrite_zeroes, .bdrv_co_pdiscard = blkreplay_co_pdiscard, .bdrv_co_flush = blkreplay_co_flush, + + .bdrv_snapshot_goto = blkreplay_snapshot_goto, };static void bdrv_blkreplay_init (void )
PATCH v13 3/25 replay: disable default snapshot for record/replay
In reply to Pavel DovgalyukFrom: Pavel Dovgalyuk <[email protected]>
This patch disables setting '-snapshot' option on by default in record/replay mode. This is needed for creating vmstates in record and replay modes.
--- vl.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
vl.c +8 -2
@@ -3149,7 +3149,13 @@ int main(int argc, char **argv, char **envp)drive_add (IF_PFLASH ,-1 , optarg,PFLASH_OPTS );break ;case QEMU_OPTION_snapshot: - snapshot =1 ; + { +Error *blocker =NULL ; + snapshot =1 ; +error_setg (&blocker,QERR_REPLAY_NOT_SUPPORTED , +"-snapshot" ); +replay_add_blocker (blocker); + }break ;case QEMU_OPTION_numa: opts = qemu_opts_parse_noisily(qemu_find_opts("numa"),
@@ -4383,7 +4389,7 @@ int main(int argc, char **argv, char **envp)qapi_free_BlockdevOptions (bdo->bdo);g_free (bdo); } -if (snapshot || replay_mode !=REPLAY_MODE_NONE ) { +if (snapshot) {qemu_opts_foreach (qemu_find_opts ("drive" ), drive_enable_snapshot,NULL ,NULL ); }
PATCH v13 4/25 replay: update docs for record/replay with block devices
In reply to Pavel DovgalyukThis patch updates the description of the command lines for using record/replay with attached block devices.
--- docs/replay.txt | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
docs/replay.txt +9 -3
@@ -27,7 +27,7 @@ Usage of the record/replay: * First, record the execution with the following command line: qemu-system-i386 \ -icount shift=7,rr=record,rrfile=replay.bin \ - -drive file=disk.qcow2,if=none,id=img-direct \ + -drive file=disk.qcow2,if=none,snapshot,id=img-direct \ -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \ -device ide-hd,drive=img-blkreplay \ -netdev user,id=net1 -device rtl8139,netdev=net1 \
@@ -35,7 +35,7 @@ Usage of the record/replay: * After recording, you can replay it by using another command line: qemu-system-i386 \ -icount shift=7,rr=replay,rrfile=replay.bin \ - -drive file=disk.qcow2,if=none,id=img-direct \ + -drive file=disk.qcow2,if=none,snapshot,id=img-direct \ -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \ -device ide-hd,drive=img-blkreplay \ -netdev user,id=net1 -device rtl8139,netdev=net1 \
@@ -223,7 +223,7 @@ Block devices record/replay module intercepts calls of bdrv coroutine functions at the top of block drivers stack. To record and replay block operations the drive must be configured as following: - -drive file=disk.qcow2,if=none,id=img-direct + -drive file=disk.qcow2,if=none,snapshot,id=img-direct -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay -device ide-hd,drive=img-blkreplay
@@ -252,6 +252,12 @@ This snapshot is created at start of recording and restored at start of replaying. It also can be loaded while replaying to roll back the execution. +'snapshot' flag of the disk image must be removed to save the snapshots +in the overlay (or original image) instead of using the temporary overlay. + -drive file=disk.ovl,if=none,id=img-direct + -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay + -device ide-hd,drive=img-blkreplay + Use QEMU monitor to create additional snapshots. 'savevm <name>' command created the snapshot and 'loadvm <name>' restores it. To prevent corruption of the original disk image, use overlay files linked to the original images.
PATCH v13 5/25 replay: don't drain/flush bdrv queue while RR is working
In reply to Pavel DovgalyukIn record/replay mode bdrv queue is controlled by replay mechanism. It does not allow saving or loading the snapshots
when bdrv queue is not empty. Stopping the VM is not blocked by nonempty queue, but flushing the queue is still impossible there, because it may cause deadlocks in replay mode.
This patch disables bdrv_drain_all and bdrv_flush_all in record/replay mode.
Stopping the machine when the IO requests are not finished is needed for the debugging. E.g., breakpoint may be set at the specified step, and forcing the IO requests to finish may break the determinism of the execution.
--- block/io.c | 28 ++++++++++++++++++++++++++++ cpus.c | 2 -- 2 files changed, 28 insertions(+), 2 deletions(-)
block/io.c +28 -0
@@ -32,6 +32,7 @@#include "qemu/cutils.h" #include "qapi/error.h" #include "qemu/error-report.h" +#include "sysemu/replay.h" #define NOT_DONE 0x7fffffff/* used while emulated sync operation in progress */
@@ -538,6 +539,15 @@ void bdrv_drain_all_begin(void)return ; } +/* +* bdrv queue is managed by record/replay, +* waiting for finishing the I/O requests may +* be infinite +*/ +if (replay_events_enabled ()) { +return ; + } +/* AIO_WAIT_WHILE() with a NULL context can only be called from the main * loop AioContext, so make sure we're in the main context. */ assert (qemu_get_current_aio_context () ==qemu_get_aio_context ());
@@ -566,6 +576,15 @@ void bdrv_drain_all_end(void) {BlockDriverState *bs =NULL ; +/* +* bdrv queue is managed by record/replay, +* waiting for finishing the I/O requests may +* be endless +*/ +if (replay_events_enabled ()) { +return ; + } +while ((bs =bdrv_next_all_states (bs))) {AioContext *aio_context =bdrv_get_aio_context (bs);
@@ -1998,6 +2017,15 @@ int bdrv_flush_all(void)BlockDriverState *bs =NULL ;int result =0 ; +/* +* bdrv queue is managed by record/replay, +* creating new flush request for stopping +* the VM may break the determinism +*/ +if (replay_events_enabled ()) { +return result; + } +for (bs =bdrv_first (&it); bs; bs =bdrv_next (&it)) {AioContext *aio_context =bdrv_get_aio_context (bs);int ret;
cpus.c +0 -2
@@ -1078,7 +1078,6 @@ static int do_vm_stop(RunState state, bool send_stop) }bdrv_drain_all (); -replay_disable_events (); ret =bdrv_flush_all ();return ret;
@@ -2149,7 +2148,6 @@ int vm_prepare_start(void)/* We are sending this now, but the CPUs will be resumed shortly later */ qapi_event_send_resume (); -replay_enable_events ();cpu_enable_ticks ();runstate_set (RUN_STATE_RUNNING );vm_state_notify (1 ,RUN_STATE_RUNNING );
PATCH v13 6/25 replay: finish record/replay before closing the disks
In reply to Pavel DovgalyukAfter recent updates block devices cannot be closed on qemu exit. This happens due to the block request polling when replay is not finished. Therefore now we stop execution recording before closing the block devices.
--- replay/replay.c | 2 ++ vl.c | 1 + 2 files changed, 3 insertions(+)
replay/replay.c +2 -0
@@ -385,6 +385,8 @@ void replay_finish(void)g_free (replay_snapshot); replay_snapshot =NULL ; + replay_mode =REPLAY_MODE_NONE ; +replay_finish_events (); }
vl.c +1 -0
@@ -4587,6 +4587,7 @@ int main(int argc, char **argv, char **envp)/* No more vcpu or device emulation activity beyond this point */ vm_shutdown (); +replay_finish ();job_cancel_sync_all ();bdrv_close_all ();
PATCH v13 7/25 qcow2: introduce icount field for snapshots
In reply to Pavel DovgalyukThis patch introduces the icount field for saving within the snapshot. It is required for navigation between the snapshots in record/replay mode.
--
v2:
- documented format changes in docs/interop/qcow2.txt (suggested by Eric Blake)
v10:
- updated the documentation
--- block/qcow2-snapshot.c | 7 +++++++ block/qcow2.h | 2 ++ docs/interop/qcow2.txt | 4 ++++ 3 files changed, 13 insertions(+)
block/qcow2-snapshot.c +7 -0
@@ -103,6 +103,12 @@ int qcow2_read_snapshots(BlockDriverState *bs) sn->disk_size = bs->total_sectors *BDRV_SECTOR_SIZE ; } +if (extra_data_size >=24 ) { + sn->icount =be64_to_cpu (extra.icount); + }else { + sn->icount =-1ULL ; + } +/* Read snapshot ID */ sn->id_str =g_malloc (id_str_size +1 ); ret =bdrv_pread (bs->file, offset, sn->id_str, id_str_size);
@@ -209,6 +215,7 @@ static int qcow2_write_snapshots(BlockDriverState *bs)memset (&extra,0 ,sizeof (extra)); extra.vm_state_size_large =cpu_to_be64 (sn->vm_state_size); extra.disk_size =cpu_to_be64 (sn->disk_size); + extra.icount =cpu_to_be64 (sn->icount); id_str_size =strlen (sn->id_str); name_size =strlen (sn->name);
block/qcow2.h +2 -0
@@ -159,6 +159,7 @@ typedef struct QEMU_PACKED QCowSnapshotHeader { typedef struct QEMU_PACKED QCowSnapshotExtraData {uint64_t vm_state_size_large;uint64_t disk_size; +uint64_t icount; } QCowSnapshotExtraData;
@@ -172,6 +173,7 @@ typedef struct QCowSnapshot {uint32_t date_sec;uint32_t date_nsec;uint64_t vm_clock_nsec; +uint64_t icount; } QCowSnapshot;struct Qcow2Cache ;
docs/interop/qcow2.txt +4 -0
@@ -540,6 +540,10 @@ Snapshot table entry: Byte 48 - 55: Virtual disk size of the snapshot in bytes + Byte 56 - 63: icount value which corresponds to + the record/replay instruction count + when the snapshot was taken + Version 3 images must include extra data at least up to byte 55.
PATCH v13 8/25 migration: introduce icount field for snapshots
In reply to Pavel DovgalyukSaving icount as a parameters of the snapshot allows navigation between them in the execution replay scenario.
This information can be used for finding a specific snapshot for proceeding the recorded execution to the specific moment of the time. E.g., 'reverse step' action (introduced in one of the following patches) needs to load the nearest snapshot which is prior to the current moment of time .
--
v2:
- made icount in SnapshotInfo optional (suggested by Eric Blake)
v7:
- added more comments for icount member (suggested by Markus Armbruster)
v9:
- updated icount comment
v10:
- updated icount comment again
--- block/qapi.c | 18 ++++++++++++++---- block/qcow2-snapshot.c | 2 ++ blockdev.c | 10 ++++++++++ include/block/snapshot.h | 1 + migration/savevm.c | 5 +++++ qapi/block-core.json | 7 ++++++- qapi/block.json | 3 ++- 7 files changed, 40 insertions(+), 6 deletions(-)
block/qapi.c +14 -4
@@ -210,6 +210,8 @@ int bdrv_query_snapshot_info_list(BlockDriverState *bs, info->date_nsec = sn_tab[i].date_nsec; info->vm_clock_sec = sn_tab[i].vm_clock_nsec /1000000000 ; info->vm_clock_nsec = sn_tab[i].vm_clock_nsec %1000000000 ; + info->icount = sn_tab[i].icount; + info->has_icount = sn_tab[i].icount !=-1ULL ; info_list =g_new0 (SnapshotInfoList,1 ); info_list->value = info;
@@ -668,14 +670,15 @@ void bdrv_snapshot_dump(fprintf_function func_fprintf, void *f, QEMUSnapshotInfo *sn) {char buf1[128 ], date_buf[128 ], clock_buf[128 ]; +char icount_buf[128 ] = {0 };struct tm tm;time_t ti;int64_t secs;if (!sn) {func_fprintf (f, -"%-10s%-20s%7s%20s%15s" , -"ID" ,"TAG" ,"VM SIZE" ,"DATE" ,"VM CLOCK" ); +"%-10s%-18s%7s%20s%13s%11s" , +"ID" ,"TAG" ,"VM SIZE" ,"DATE" ,"VM CLOCK" ,"ICOUNT" ); }else { ti = sn->date_sec;localtime_r (&ti, &tm);
@@ -688,13 +691,18 @@ void bdrv_snapshot_dump(fprintf_function func_fprintf, void *f, (int )((secs /60 ) %60 ), (int )(secs %60 ), (int)((sn->vm_clock_nsec / 1000000) % 1000)); +if (sn->icount !=-1ULL ) { +snprintf (icount_buf,sizeof (icount_buf), +"%" PRId64, sn->icount); + }func_fprintf (f, -"%-10s%-20s%7s%20s%15s" , +"%-10s%-18s%7s%20s%13s%11s" , sn->id_str, sn->name,get_human_readable_size (buf1,sizeof (buf1), sn->vm_state_size), date_buf, - clock_buf); + clock_buf, + icount_buf); } }
@@ -862,6 +870,8 @@ void bdrv_image_info_dump(fprintf_function func_fprintf, void *f, .date_nsec = elem->value->date_nsec, .vm_clock_nsec = elem->value->vm_clock_sec *1000000000ULL + elem->value->vm_clock_nsec, + .icount = elem->value->has_icount ? + elem->value->icount : -1ULL, };pstrcpy (sn.id_str,sizeof (sn.id_str), elem->value->id);
block/qcow2-snapshot.c +2 -0
@@ -379,6 +379,7 @@ int qcow2_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info) sn->date_sec = sn_info->date_sec; sn->date_nsec = sn_info->date_nsec; sn->vm_clock_nsec = sn_info->vm_clock_nsec; + sn->icount = sn_info->icount;/* Allocate the L1 table of the snapshot and copy the current one there. */ l1_table_offset =qcow2_alloc_clusters (bs, s->l1_size *sizeof (uint64_t ));
@@ -698,6 +699,7 @@ int qcow2_snapshot_list(BlockDriverState *bs, QEMUSnapshotInfo **psn_tab) sn_info->date_sec = sn->date_sec; sn_info->date_nsec = sn->date_nsec; sn_info->vm_clock_nsec = sn->vm_clock_nsec; + sn_info->icount = sn->icount; } *psn_tab = sn_tab;return s->nb_snapshots;
blockdev.c +10 -0
@@ -57,6 +57,7 @@#include "block/trace.h" #include "sysemu/arch_init.h" #include "sysemu/qtest.h" +#include "sysemu/replay.h" #include "qemu/cutils.h" #include "qemu/help_option.h" #include "qemu/throttle-options.h"
@@ -1239,6 +1240,10 @@ SnapshotInfo *qmp_blockdev_snapshot_delete_internal_sync(const char *device, info->vm_state_size = sn.vm_state_size; info->vm_clock_nsec = sn.vm_clock_nsec %1000000000 ; info->vm_clock_sec = sn.vm_clock_nsec /1000000000 ; +if (sn.icount !=-1ULL ) { + info->icount = sn.icount; + info->has_icount = true; + }return info;
@@ -1447,6 +1452,11 @@ static void internal_snapshot_prepare(BlkActionState *common, sn->date_sec = tv.tv_sec; sn->date_nsec = tv.tv_usec *1000 ; sn->vm_clock_nsec =qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ); +if (replay_mode !=REPLAY_MODE_NONE ) { + sn->icount =replay_get_current_step (); + }else { + sn->icount =-1ULL ; + } ret1 =bdrv_snapshot_create (bs, sn);if (ret1 <0 ) {
include/block/snapshot.h +1 -0
@@ -42,6 +42,7 @@ typedef struct QEMUSnapshotInfo {uint32_t date_sec;/* UTC date of the snapshot */ uint32_t date_nsec;uint64_t vm_clock_nsec;/* VM clock relative to boot */ +uint64_t icount;/* record/replay step */ } QEMUSnapshotInfo; int bdrv_snapshot_find(BlockDriverState *bs, QEMUSnapshotInfo *sn_info,
migration/savevm.c +5 -0
@@ -2510,6 +2510,11 @@ int save_snapshot(const char *name, Error **errp) sn->date_sec = tv.tv_sec; sn->date_nsec = tv.tv_usec *1000 ; sn->vm_clock_nsec =qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ); +if (replay_mode !=REPLAY_MODE_NONE ) { + sn->icount =replay_get_current_step (); + }else { + sn->icount =-1ULL ; + }if (name) { ret =bdrv_snapshot_find (bs, old_sn, name);
qapi/block-core.json +6 -1
@@ -26,13 +26,18 @@ # # @vm-clock-nsec: fractional part in nano seconds to be used with vm-clock-sec # +# @icount: Current instruction count. Appears when execution record/replay +# is enabled. Used for "time-traveling" to match the moment +# in the recorded execution with the snapshots. (since 4.0) +# # Since: 1.3 # ## { 'struct': 'SnapshotInfo', 'data': { 'id': 'str', 'name': 'str', 'vm-state-size': 'int', 'date-sec': 'int', 'date-nsec': 'int', - 'vm-clock-sec': 'int', 'vm-clock-nsec': 'int' } } + 'vm-clock-sec': 'int', 'vm-clock-nsec': 'int', + '*icount': 'int' } } ## # @ImageInfoSpecificQCow2EncryptionBase:
qapi/block.json +2 -1
@@ -176,7 +176,8 @@ # "date-sec": 1000012, # "date-nsec": 10, # "vm-clock-sec": 100, -# "vm-clock-nsec": 20 +# "vm-clock-nsec": 20, +# "icount": 220414 # } # } #
PATCH v13 9/25 replay: provide an accessor for rr filename
In reply to Pavel DovgalyukThis patch adds an accessor function for the name of the record/replay log file. Adding an accessor instead of making variable global, prevents accidental modification of this variable by other modules.
--- include/sysemu/replay.h | 2 ++ replay/replay.c | 5 +++++ 2 files changed, 7 insertions(+)
include/sysemu/replay.h +2 -0
@@ -71,6 +71,8 @@ void replay_start(void);void replay_finish (void );/*! Adds replay blocker with the specified error description */ void replay_add_blocker (Error *reason); +/* Returns name of the replay log file */ +const char *replay_get_filename (void );/* Processing the instructions */
replay/replay.c +5 -0
@@ -394,3 +394,8 @@ void replay_add_blocker(Error *reason) { replay_blockers =g_slist_prepend (replay_blockers, reason); } + +const char *replay_get_filename (void ) +{ +return replay_filename; +}
PATCH v13 10/25 qapi: introduce replay.json for record/replay-related stuff
In reply to Pavel DovgalyukThis patch adds replay.json file. It will be
used for adding record/replay-related data structures and commands.
--
v10:
- minor changes
v13:
- rebased to the new QAPI files
--- MAINTAINERS | 1 + include/sysemu/replay.h | 2 +- qapi/Makefile.objs | 2 +- qapi/misc.json | 18 ------------------ qapi/qapi-schema.json | 1 + qapi/replay.json | 26 ++++++++++++++++++++++++++ 6 files changed, 30 insertions(+), 20 deletions(-) create mode 100644 qapi/replay.json
MAINTAINERS +1 -0
@@ -2126,6 +2126,7 @@ F: net/filter-replay.c F: include/sysemu/replay.h F: docs/replay.txt F: stubs/replay.c +F: qapi/replay.json IOVA Tree M: Peter Xu <[email protected]>
include/sysemu/replay.h +1 -1
@@ -13,7 +13,7 @@*/ #include "sysemu.h" -#include "qapi/qapi-types-misc.h" +#include "qapi/qapi-types-replay.h" #include "qapi/qapi-types-ui.h" /* replay clock kinds */
qapi/Makefile.objs +1 -1
@@ -6,7 +6,7 @@ util-obj-y += qmp-event.o util-obj-y += qapi-util.o QAPI_COMMON_MODULES = block-core block char common crypto introspect -QAPI_COMMON_MODULES += job migration misc net rdma rocker run-state +QAPI_COMMON_MODULES += job migration misc net replay rdma rocker run-state QAPI_COMMON_MODULES += sockets tpm trace transaction ui QAPI_TARGET_MODULES = target QAPI_MODULES = $(QAPI_COMMON_MODULES) $(QAPI_TARGET_MODULES)
qapi/misc.json +0 -18
@@ -2876,24 +2876,6 @@ { 'event': 'ACPI_DEVICE_OST', 'data': { 'info': 'ACPIOSTInfo' } } -## -# @ReplayMode: -# -# Mode of the replay subsystem. -# -# @none: normal execution mode. Replay or record are not enabled. -# -# @record: record mode. All non-deterministic data is written into the -# replay log. -# -# @play: replay mode. Non-deterministic data required for system execution -# is read from the log. -# -# Since: 2.5 -## -{ 'enum': 'ReplayMode', - 'data': [ 'none', 'record', 'play' ] } - ## # @xen-load-devices-state: #
qapi/qapi-schema.json +1 -0
@@ -96,5 +96,6 @@ { 'include': 'transaction.json' } { 'include': 'trace.json' } { 'include': 'introspect.json' } +{ 'include': 'replay.json' } { 'include': 'misc.json' } { 'include': 'target.json' }
qapi/replay.json new file +26 -0
@@ -0,0 +1,26 @@ +# -*- Mode: Python -*- +# + +## +# = Record/replay +## + +{ 'include': 'common.json' } + +## +# @ReplayMode: +# +# Mode of the replay subsystem. +# +# @none: normal execution mode. Replay or record are not enabled. +# +# @record: record mode. All non-deterministic data is written into the +# replay log. +# +# @play: replay mode. Non-deterministic data required for system execution +# is read from the log. +# +# Since: 2.5 +## +{ 'enum': 'ReplayMode', + 'data': [ 'none', 'record', 'play' ] }
PATCH v13 11/25 replay: introduce info hmp/qmp command
In reply to Pavel DovgalyukThis patch introduces 'info replay' monitor command and corresponding qmp request.
These commands request the current record/replay mode, replay log file name, and the instruction count (number of recorded/replayed instructions). The instruction count can be used with the replay_seek/replay_break commands added in the next two patches.
--
v2:
- renamed info_replay qmp into query-replay (suggested by Eric Blake)
v7:
- added empty line (suggested by Markus Armbruster)
v9:
- changed 'step' parameter name to 'icount'
- moved json stuff to replay.json and updated the descriptions (suggested by Markus Armbruster)
v10:
- updated descriptions and messages for rr stuff
--- hmp-commands-info.hx | 14 ++++++++++++++ hmp.h | 1 + qapi/block-core.json | 3 ++- qapi/replay.json | 39 +++++++++++++++++++++++++++++++++++++++ replay/Makefile.objs | 3 ++- replay/replay-debugging.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 101 insertions(+), 2 deletions(-) create mode 100644 replay/replay-debugging.c
hmp-commands-info.hx +14 -0
@@ -916,6 +916,20 @@ STEXI @item info sev @findex info sev Show SEV information. +ETEXI + + { + .name = "replay", + .args_type = "", + .params = "", + .help = "show record/replay information", + .cmd = hmp_info_replay, + }, + +STEXI +@item info replay +@findex info replay +Display the record/replay information: mode and the current icount. ETEXI STEXI
hmp.h +1 -0
@@ -148,5 +148,6 @@ void hmp_hotpluggable_cpus(Monitor *mon, const QDict *qdict);void hmp_info_vm_generation_id (Monitor *mon,const QDict *qdict);void hmp_info_memory_size_summary (Monitor *mon,const QDict *qdict);void hmp_info_sev (Monitor *mon,const QDict *qdict); +void hmp_info_replay (Monitor *mon,const QDict *qdict);#endif
qapi/block-core.json +2 -1
@@ -28,7 +28,8 @@ # # @icount: Current instruction count. Appears when execution record/replay # is enabled. Used for "time-traveling" to match the moment -# in the recorded execution with the snapshots. (since 4.0) +# in the recorded execution with the snapshots. This counter may +# be obtained through @query-replay command (since 4.0) # # Since: 1.3 #
qapi/replay.json +39 -0
@@ -24,3 +24,42 @@ ## { 'enum': 'ReplayMode', 'data': [ 'none', 'record', 'play' ] } + +## +# @ReplayInfo: +# +# Record/replay information. +# +# @mode: current mode. +# +# @filename: name of the record/replay log file. +# It is present only in record or replay modes, when the log +# is recorded or replayed. +# +# @icount: current number of executed instructions. +# +# Since: 4.0 +# +## +{ 'struct': 'ReplayInfo', + 'data': { 'mode': 'ReplayMode', '*filename': 'str', 'icount': 'int' } } + +## +# @query-replay: +# +# Retrieve the record/replay information. +# It includes current instruction count which may be used for +# @replay-break and @replay-seek commands. +# +# Returns: record/replay information. +# +# Since: 4.0 +# +# Example: +# +# -> { "execute": "query-replay" } +# <- { "return": { "mode": "play", "filename": "log.rr", "icount": 220414 } } +# +## +{ 'command': 'query-replay', + 'returns': 'ReplayInfo' }
replay/Makefile.objs +2 -1
@@ -6,4 +6,5 @@ common-obj-y += replay-input.o common-obj-y += replay-char.o common-obj-y += replay-snapshot.o common-obj-y += replay-net.o -common-obj-y += replay-audio.o +common-obj-y += replay-audio.o +common-obj-y += replay-debugging.o
replay/replay-debugging.c new file +43 -0
43 lines not on this page · raw message
PATCH v13 12/25 replay: introduce breakpoint at the specified step
In reply to Pavel DovgalyukThis patch introduces replay_break, replay_delete_break qmp and hmp commands.
These commands allow stopping at the specified instruction. It may be useful for debugging when there are some known events that should be investigated.replay_break command has one argument - number of instructions executed since the start of the replay. replay_delete_break removes previously set breakpoint.
--
v2:
- renamed replay_break qmp command into replay-break (suggested by Eric Blake)
v7:
- introduces replay_delete_break command
v9:
- changed 'step' parameter name to 'icount'
- moved json stuff to replay.json and updated the description (suggested by Markus Armbruster)
v10:
- updated descriptions (suggested by Markus Armbruster)
--- hmp-commands.hx | 34 ++++++++++++++++++ hmp.h | 2 + qapi/replay.json | 36 +++++++++++++++++++ replay/replay-debugging.c | 86 +++++++++++++++++++++++++++++++++++++++++++++ replay/replay-internal.h | 4 ++ replay/replay.c | 17 +++++++++ 6 files changed, 179 insertions(+)
hmp-commands.hx +34 -0
@@ -1887,6 +1887,40 @@ ETEXI STEXI @item qom-set @var{path} @var{property} @var{value} Set QOM property @var{property} of object at location @var{path} to value @var{value} +ETEXI + + { + .name = "replay_break", + .args_type = "icount:i", + .params = "icount", + .help = "set breakpoint at the specified instruction count", + .cmd = hmp_replay_break, + }, + +STEXI +@item replay_break @var{icount} +@findex replay_break +Set replay breakpoint at instruction count @var{icount}. +Execution stops when the specified instruction is reached. +There can be at most one breakpoint. When breakpoint is set, any prior +one is removed. The breakpoint may be set only in replay mode and only +"in the future", i.e. at instruction counts greater than the current one. +The current instruction count can be observed with 'info replay'. +ETEXI + + { + .name = "replay_delete_break", + .args_type = "", + .params = "", + .help = "removes replay breakpoint", + .cmd = hmp_replay_delete_break, + }, + +STEXI +@item replay_delete_break +@findex replay_delete_break +Remove replay breakpoint which was previously set with replay_break. +The command is ignored when there are no replay breakpoints. ETEXI {
hmp.h +2 -0
@@ -149,5 +149,7 @@ void hmp_info_vm_generation_id(Monitor *mon, const QDict *qdict);void hmp_info_memory_size_summary (Monitor *mon,const QDict *qdict);void hmp_info_sev (Monitor *mon,const QDict *qdict);void hmp_info_replay (Monitor *mon,const QDict *qdict); +void hmp_replay_break (Monitor *mon,const QDict *qdict); +void hmp_replay_delete_break (Monitor *mon,const QDict *qdict);#endif
qapi/replay.json +36 -0
@@ -63,3 +63,39 @@ ## { 'command': 'query-replay', 'returns': 'ReplayInfo' } + +## +# @replay-break: +# +# Set replay breakpoint at instruction count @icount. +# Execution stops when the specified instruction is reached. +# There can be at most one breakpoint. When breakpoint is set, any prior +# one is removed. The breakpoint may be set only in replay mode and only +# "in the future", i.e. at instruction counts greater than the current one. +# The current instruction count can be observed with @query-replay. +# +# @icount: instruction count to stop at +# +# Since: 4.0 +# +# Example: +# +# -> { "execute": "replay-break", "data": { "icount": 220414 } } +# +## +{ 'command': 'replay-break', 'data': { 'icount': 'int' } } + +## +# @replay-delete-break: +# +# Remove replay breakpoint which was set with @replay-break. +# The command is ignored when there are no replay breakpoints. +# +# Since: 4.0 +# +# Example: +# +# -> { "execute": "replay-delete-break" } +# +## +{ 'command': 'replay-delete-break' }
replay/replay-debugging.c +86 -0
95 lines not on this page · raw message
replay/replay-internal.h +4 -0
@@ -91,6 +91,10 @@ extern ReplayState replay_state;/* File for replay writing */ extern FILE *replay_file; +/* Instruction count of the replay breakpoint */ +extern uint64_t replay_break_icount; +/* Timer for the replay breakpoint callback */ +extern QEMUTimer *replay_break_timer;void replay_put_byte (uint8_t byte);void replay_put_event (uint8_t event);
replay/replay.c +17 -0
@@ -34,6 +34,10 @@ static char *replay_filename;ReplayState replay_state;static GSList *replay_blockers; +/* Replay breakpoints */ +uint64_t replay_break_icount =-1ULL ; +QEMUTimer *replay_break_timer; +bool replay_next_event_is (int event) {bool res = false;
@@ -73,6 +77,13 @@ int replay_get_instructions(void)replay_mutex_lock ();if (replay_next_event_is (EVENT_INSTRUCTION )) { res = replay_state.instructions_count; +if (replay_break_icount !=-1LL ) { +uint64_t current =replay_get_current_step (); +assert (replay_break_icount >= current); +if (current + res > replay_break_icount) { + res = replay_break_icount - current; + } + } }replay_mutex_unlock ();return res;
@@ -99,6 +110,12 @@ void replay_account_executed_instructions(void)will be read from the log. */ qemu_notify_event (); } +/* Execution reached the break step */ +if (replay_break_icount == replay_state.current_step) { +/* Cannot make callback directly from the vCPU thread */ +timer_mod_ns (replay_break_timer, +qemu_clock_get_ns (QEMU_CLOCK_REALTIME )); + } } } }
PATCH v13 13/25 replay: implement replay-seek command
In reply to Pavel DovgalyukThis patch adds hmp/qmp commands replay_seek/replay-seek that proceed the execution to the specified instruction count.
The command automatically loads nearest snapshot and replays the execution to find the desired instruction count.
--
v2:
- renamed replay_seek qmp command into replay-seek (suggested by Eric Blake)
v7:
- small fixes related to Markus Armbruster's review
v9:
- changed 'step' parameter name to 'icount'
- moved json stuff to replay.json and updated the description (suggested by Markus Armbruster)
v10:
- updated the descriptions
--- hmp-commands.hx | 19 +++++++++ hmp.h | 1 qapi/replay.json | 20 ++++++++++ replay/replay-debugging.c | 92 +++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 132 insertions(+)
hmp-commands.hx +19 -0
@@ -1921,6 +1921,25 @@ STEXI @findex replay_delete_break Remove replay breakpoint which was previously set with replay_break. The command is ignored when there are no replay breakpoints. +ETEXI + + { + .name = "replay_seek", + .args_type = "icount:i", + .params = "icount", + .help = "replay execution to the specified instruction count", + .cmd = hmp_replay_seek, + }, + +STEXI +@item replay_seek @var{icount} +@findex replay_seek +Automatically proceed to the instruction count @var{icount}, when +replaying the execution. The command automatically loads nearest +snapshot and replays the execution to find the desired instruction. +When there is no preceding snapshot or the execution is not replayed, +then the command fails. +icount for the reference may be observed with 'info replay' command. ETEXI {
hmp.h +1 -0
@@ -151,5 +151,6 @@ void hmp_info_sev(Monitor *mon, const QDict *qdict);void hmp_info_replay (Monitor *mon,const QDict *qdict);void hmp_replay_break (Monitor *mon,const QDict *qdict);void hmp_replay_delete_break (Monitor *mon,const QDict *qdict); +void hmp_replay_seek (Monitor *mon,const QDict *qdict);#endif
qapi/replay.json +20 -0
@@ -99,3 +99,23 @@ # ## { 'command': 'replay-delete-break' } + +## +# @replay-seek: +# +# Automatically proceed to the instruction count @icount, when +# replaying the execution. The command automatically loads nearest +# snapshot and replays the execution to find the desired instruction. +# When there is no preceding snapshot or the execution is not replayed, +# then the command fails. +# icount for the reference may be obtained with @query-replay command. +# +# @icount: target instruction count +# +# Since: 4.0 +# +# Example: +# +# -> { "execute": "replay-seek", "data": { "icount": 220414 } } +## +{ 'command': 'replay-seek', 'data': { 'icount': 'int' } }
replay/replay-debugging.c +92 -0
101 lines not on this page · raw message
PATCH v13 14/25 replay: refine replay-time module
In reply to Pavel DovgalyukThis patch removes refactoring artifacts from the replay/replay-time.c
--- replay/replay-time.c | 36 ++++++++++++++++-------------------- 1 file changed, 16 insertions(+), 20 deletions(-)
replay/replay-time.c +16 -20
@@ -15,18 +15,19 @@#include "replay-internal.h" #include "qemu/error-report.h" -int64_t replay_save_clock (ReplayClockKind kind,int64_t clock,int64_t raw_icount) +int64_t replay_save_clock (ReplayClockKind kind,int64_t clock, +int64_t raw_icount) { -if (replay_file) { -g_assert (replay_mutex_locked ()); +g_assert (replay_file); +g_assert (replay_mutex_locked ()); -/* Due to the caller's locking requirements we get the icount from it -* instead of using replay_save_instructions(). -*/ -replay_advance_current_step (raw_icount); -replay_put_event (EVENT_CLOCK + kind); -replay_put_qword (clock); - } +/* +* Due to the caller's locking requirements we get the icount from it +* instead of using replay_save_instructions(). +*/ +replay_advance_current_step (raw_icount); +replay_put_event (EVENT_CLOCK + kind); +replay_put_qword (clock);return clock; }
@@ -48,20 +49,15 @@ void replay_read_next_clock(ReplayClockKind kind)/*! Reads next clock event from the input. */ int64_t replay_read_clock (ReplayClockKind kind) { +int64_t ret;g_assert (replay_file &&replay_mutex_locked ());replay_account_executed_instructions (); -if (replay_file) { -int64_t ret; -if (replay_next_event_is (EVENT_CLOCK + kind)) { -replay_read_next_clock (kind); - } - ret = replay_state.cached_clock[kind]; - -return ret; +if (replay_next_event_is (EVENT_CLOCK + kind)) { +replay_read_next_clock (kind); } + ret = replay_state.cached_clock[kind]; -error_report ("REPLAY INTERNAL ERROR %d" , __LINE__); -exit (1 ); +return ret; }
PATCH v13 15/25 replay: flush rr queue before loading the vmstate
In reply to Pavel DovgalyukNon-empty record/replay queue prevents saving and loading the VM state, because it includes pending bottom halves and block coroutines. But when the new VM state is loaded, we don't have to preserve the consistency of the current state anymore. Therefore this patch just flushes the queue allowing the coroutines to finish.
--- include/sysemu/replay.h | 2 ++ migration/savevm.c | 6 ++++++ replay/replay-internal.h | 2 -- 3 files changed, 8 insertions(+), 2 deletions(-)
include/sysemu/replay.h +2 -0
@@ -140,6 +140,8 @@ void replay_disable_events(void);void replay_enable_events (void );/*! Returns true when saving events is enabled */ bool replay_events_enabled (void ); +/* Flushes events queue */ +void replay_flush_events (void );/*! Adds bottom half event to the queue */ void replay_bh_schedule_event (QEMUBH *bh);/*! Adds input event to the queue */
migration/savevm.c +6 -0
@@ -2702,6 +2702,12 @@ int load_snapshot(const char *name, Error **errp)return -EINVAL ; } +/* +* Flush the record/replay queue. Now the VM state is going +* to change. Therefore we don't need to preserve its consistency +*/ +replay_flush_events (); +/* Flush all IO requests so they don't interfere with the new state. */ bdrv_drain_all_begin ();
replay/replay-internal.h +0 -2
@@ -146,8 +146,6 @@ void replay_read_next_clock(unsigned int kind);void replay_init_events (void );/*! Clears internal data structures for events handling */ void replay_finish_events (void ); -/*! Flushes events queue */ -void replay_flush_events (void );/*! Returns true if there are any unsaved events in the queue */ bool replay_has_events (void );/*! Saves events from queue into the file */
PATCH v13 16/25 gdbstub: add reverse step support in replay mode
In reply to Pavel DovgalyukGDB remote protocol supports two reverse debugging commands:
reverse step and reverse continue.
This patch adds support of the first one to the gdbstub. Reverse step is intended to step one instruction in the backwards direction. This is not possible in regular execution.
But replayed execution is deterministic, therefore we can load one of the prior snapshots and proceed to the desired step. It is equivalent to stepping one instruction back.
There should be at least one snapshot preceding the debugged part of the replay log.
--- accel/tcg/translator.c | 1 + cpus.c | 14 +++++++++++--- exec.c | 7 +++++++ gdbstub.c | 44 +++++++++++++++++++++++++++++++++++++++++--- include/sysemu/replay.h | 11 +++++++++++ replay/replay-debugging.c | 33 +++++++++++++++++++++++++++++++++ stubs/replay.c | 5 +++++ 7 files changed, 109 insertions(+), 6 deletions(-)
accel/tcg/translator.c +1 -0
@@ -17,6 +17,7 @@#include "exec/gen-icount.h" #include "exec/log.h" #include "exec/translator.h" +#include "sysemu/replay.h" /* Pairs with tcg_clear_temp_count. To be called by #TranslatorOps.{translate_insn,tb_stop} if
cpus.c +11 -3
@@ -1104,9 +1104,17 @@ static bool cpu_can_run(CPUState *cpu)static void cpu_handle_guest_debug (CPUState *cpu) { -gdb_set_stop_cpu (cpu); -qemu_system_debug_request (); - cpu->stopped = true; +if (!replay_running_debug ()) { +gdb_set_stop_cpu (cpu); +qemu_system_debug_request (); + cpu->stopped = true; + }else { +if (!cpu->singlestep_enabled) { +cpu_single_step (cpu,SSTEP_ENABLE ); + }else { +cpu_single_step (cpu,0 ); + } + } }#ifdef CONFIG_LINUX
exec.c +7 -0
@@ -2742,6 +2742,13 @@ static void check_watchpoint(int offset, int len, MemTxAttrs attrs, int flags) QTAILQ_FOREACH(wp, &cpu->watchpoints, entry) {if (cpu_watchpoint_address_matches (wp, vaddr, len) && (wp->flags & flags)) { +if (replay_running_debug ()) { +/* +* Don't process the watchpoints when we are +* in a reverse debugging operation. +*/ +return ; + }if (flags ==BP_MEM_READ ) { wp->flags |=BP_WATCHPOINT_HIT_READ ; }else {
gdbstub.c +41 -3
82 lines not on this page · raw message
include/sysemu/replay.h +11 -0
@@ -73,6 +73,17 @@ void replay_finish(void);void replay_add_blocker (Error *reason);/* Returns name of the replay log file */ const char *replay_get_filename (void ); +/* +* Start making one step in backward direction. +* Used by gdbstub for backwards debugging. +* Returns true on success. +*/ +bool replay_reverse_step (void ); +/* +* Returns true if replay module is processing +* reverse_continue or reverse_step request +*/ +bool replay_running_debug (void );/* Processing the instructions */
replay/replay-debugging.c +33 -0
@@ -21,6 +21,13 @@#include "block/snapshot.h" #include "migration/snapshot.h" +static bool replay_is_debugging; + +bool replay_running_debug (void ) +{ +return replay_is_debugging; +} +void hmp_info_replay (Monitor *mon,const QDict *qdict) {if (replay_mode ==REPLAY_MODE_NONE ) {
@@ -219,3 +226,29 @@ void hmp_replay_seek(Monitor *mon, const QDict *qdict)return ; } } + +static void replay_stop_vm_debug (void *opaque) +{ + replay_is_debugging = false; +vm_stop (RUN_STATE_DEBUG ); +replay_delete_break (); +} + +bool replay_reverse_step (void ) +{ +Error *err =NULL ; + +assert (replay_mode ==REPLAY_MODE_PLAY ); + +if (replay_get_current_step () !=0 ) { +replay_seek (replay_get_current_step () -1 , replay_stop_vm_debug, &err); +if (err) { +error_free (err); +return false; + } + replay_is_debugging = true; +return true; + } + +return false; +}
stubs/replay.c +5 -0
@@ -80,3 +80,8 @@ void replay_mutex_lock(void)void replay_mutex_unlock (void ) { } + +bool replay_reverse_step (void ) +{ +return false; +}
PATCH v13 17/25 gdbstub: add reverse continue support in replay mode
In reply to Pavel DovgalyukThis patch adds support of the reverse continue operation for gdbstub. Reverse continue finds the last breakpoint that would happen in normal execution from the beginning to the current moment. Implementation of the reverse continue replays the execution twice:
to find the breakpoints that were hit and to seek to the last breakpoint. Reverse continue loads the previous snapshot and tries to find the breakpoint since that moment. If there are no such breakpoints, it proceeds to the earlier snapshot, and so on. When no breakpoints or watchpoints were hit at all, execution stops at the beginning of the replay log.
--- cpus.c | 5 +++ exec.c | 1 + gdbstub.c | 10 ++++++ include/sysemu/replay.h | 8 +++++ replay/replay-debugging.c | 71 +++++++++++++++++++++++++++++++++++++++++++++ stubs/replay.c | 5 +++ 6 files changed, 99 insertions(+), 1 deletion(-)
cpus.c +5 -0
@@ -1110,6 +1110,11 @@ static void cpu_handle_guest_debug(CPUState *cpu) cpu->stopped = true; }else {if (!cpu->singlestep_enabled) { +/* +* Report about the breakpoint and +* make a single step to skip it +*/ +replay_breakpoint ();cpu_single_step (cpu,SSTEP_ENABLE ); }else {cpu_single_step (cpu,0 );
exec.c +1 -0
@@ -2747,6 +2747,7 @@ static void check_watchpoint(int offset, int len, MemTxAttrs attrs, int flags)* Don't process the watchpoints when we are * in a reverse debugging operation. */ +replay_breakpoint ();return ; }if (flags ==BP_MEM_READ ) {
gdbstub.c +9 -1
@@ -1444,6 +1444,14 @@ static int gdb_handle_packet(GDBState *s, const char *line_buf)put_packet (s,"E14" );break ; } +case 'c' : +if (replay_reverse_continue ()) { +gdb_continue (s); +return RS_IDLE ; + }else { +put_packet (s,"E14" ); +break ; + }default : goto unknown_command; }
@@ -1754,7 +1762,7 @@ static int gdb_handle_packet(GDBState *s, const char *line_buf)pstrcat (buf,sizeof (buf),";multiprocess+" );if (replay_mode ==REPLAY_MODE_PLAY ) { -pstrcat (buf,sizeof (buf),";ReverseStep+" ); +pstrcat (buf,sizeof (buf),";ReverseStep+;ReverseContinue+" ); }put_packet (s, buf);
include/sysemu/replay.h +8 -0
@@ -79,11 +79,19 @@ const char *replay_get_filename(void);* Returns true on success. */ bool replay_reverse_step (void ); +/* +* Start searching the last breakpoint/watchpoint. +* Used by gdbstub for backwards debugging. +* Returns true if the process successfully started. +*/ +bool replay_reverse_continue (void );/* * Returns true if replay module is processing * reverse_continue or reverse_step request */ bool replay_running_debug (void ); +/* Called in reverse debugging mode to collect breakpoint information */ +void replay_breakpoint (void );/* Processing the instructions */
replay/replay-debugging.c +71 -0
80 lines not on this page · raw message
stubs/replay.c +5 -0
@@ -85,3 +85,8 @@ bool replay_reverse_step(void) {return false; } + +bool replay_reverse_continue (void ) +{ +return false; +}
PATCH v13 18/25 replay: describe reverse debugging in docs/replay.txt
In reply to Pavel DovgalyukThis patch updates the documentation and describes usage of the reverse debugging in QEMU+GDB.
--- docs/replay.txt | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
docs/replay.txt +33 -0
@@ -293,6 +293,39 @@ for recording and replaying must contain identical number of ports in record and replay modes, but their backends may differ. E.g., '-serial stdio' in record mode, and '-serial null' in replay mode. +Reverse debugging +----------------- + +Reverse debugging allows "executing" the program in reverse direction. +GDB remote protocol supports "reverse step" and "reverse continue" +commands. The first one steps single instruction backwards in time, +and the second one finds the last breakpoint in the past. + +Recorded executions may be used to enable reverse debugging. QEMU can't +execute the code in backwards direction, but can load a snapshot and +replay forward to find the desired position or breakpoint. + +The following GDB commands are supported: + - reverse-stepi (or rsi) - step one instruction backwards + - reverse-continue (or rc) - find last breakpoint in the past + +Reverse step loads the nearest snapshot and replays the execution until +the required instruction is met. + +Reverse continue may include several passes of examining the execution +between the snapshots. Each of the passes include the following steps: + 1. loading the snapshot + 2. replaying to examine the breakpoints + 3. if breakpoint or watchpoint was met + - loading the snaphot again + - replaying to the required breakpoint + 4. else + - proceeding to the p.1 with the earlier snapshot + +Therefore usage of the reverse debugging requires at least one snapshot +created in advance. See the "Snapshotting" section to learn about running +record/replay and creating the snapshot in these modes. + Replay log format -----------------
PATCH v13 19/25 replay: add BH oneshot event for block layer
In reply to Pavel DovgalyukReplay is capable of recording normal BH events, but sometimes there are single use callbacks scheduled with aio_bh_schedule_oneshot function. This patch enables recording and replaying such callbacks. Block layer uses these events for calling the completion function. Replaying these calls makes the execution deterministic.
--
v6:
- moved stub function to the separate file for fixing linux-user build
v10:
- replaced all block layer aio_bh_schedule_oneshot calls
--- block/block-backend.c | 8 +++++--- block/io.c | 4 ++-- block/iscsi.c | 5 +++-- block/nfs.c | 5 +++-- block/null.c | 4 +++- block/nvme.c | 6 ++++-- block/rbd.c | 5 +++-- block/vxhs.c | 5 +++-- include/sysemu/replay.h | 3 +++ replay/replay-events.c | 16 ++++++++++++++++ replay/replay-internal.h | 1 + replay/replay.c | 2 +- stubs/Makefile.objs | 1 + stubs/replay-user.c | 9 +++++++++ 14 files changed, 57 insertions(+), 17 deletions(-) create mode 100644 stubs/replay-user.c
block/block-backend.c +5 -3
@@ -17,6 +17,7 @@#include "block/throttle-groups.h" #include "sysemu/blockdev.h" #include "sysemu/sysemu.h" +#include "sysemu/replay.h" #include "qapi/error.h" #include "qapi/qapi-events-block.h" #include "qemu/id.h"
@@ -1293,7 +1294,8 @@ BlockAIOCB *blk_abort_aio_request(BlockBackend *blk, acb->blk = blk; acb->ret = ret; -aio_bh_schedule_oneshot (blk_get_aio_context (blk), error_callback_bh, acb); +replay_bh_schedule_oneshot_event (blk_get_aio_context (blk), + error_callback_bh, acb);return &acb->common; }
@@ -1349,8 +1351,8 @@ static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, int bytes, acb->has_returned = true;if (acb->rwco.ret !=NOT_DONE ) { -aio_bh_schedule_oneshot (blk_get_aio_context (blk), - blk_aio_complete_bh, acb); +replay_bh_schedule_oneshot_event (blk_get_aio_context (blk), + blk_aio_complete_bh, acb); }return &acb->common;
block/io.c +2 -2
@@ -340,8 +340,8 @@ static void coroutine_fn bdrv_co_yield_to_drain(BlockDriverState *bs,if (bs) {bdrv_inc_in_flight (bs); } -aio_bh_schedule_oneshot (bdrv_get_aio_context (bs), - bdrv_co_drain_bh_cb, &data); +replay_bh_schedule_oneshot_event (bdrv_get_aio_context (bs), + bdrv_co_drain_bh_cb, &data);qemu_coroutine_yield ();/* If we are resumed from some other event (such as an aio completion or a
block/iscsi.c +3 -2
@@ -38,6 +38,7 @@#include "qemu/iov.h" #include "qemu/option.h" #include "qemu/uuid.h" +#include "sysemu/replay.h" #include "qapi/error.h" #include "qapi/qapi-commands-misc.h" #include "qapi/qmp/qdict.h"
@@ -275,8 +276,8 @@ iscsi_co_generic_cb(struct iscsi_context *iscsi, int status, out:if (iTask->co) { -aio_bh_schedule_oneshot (iTask->iscsilun->aio_context, - iscsi_co_generic_bh_cb, iTask); +replay_bh_schedule_oneshot_event (iTask->iscsilun->aio_context, + iscsi_co_generic_bh_cb, iTask); }else { iTask->complete =1 ; }
block/nfs.c +3 -2
@@ -36,6 +36,7 @@#include "qemu/uri.h" #include "qemu/cutils.h" #include "sysemu/sysemu.h" +#include "sysemu/replay.h" #include "qapi/qapi-visit-block-core.h" #include "qapi/qmp/qdict.h" #include "qapi/qmp/qstring.h"
@@ -256,8 +257,8 @@ nfs_co_generic_cb(int ret, struct nfs_context *nfs, void *data,if (task->ret <0 ) {error_report ("NFS Error: %s" ,nfs_get_error (nfs)); } -aio_bh_schedule_oneshot (task->client->aio_context, - nfs_co_generic_bh_cb, task); +replay_bh_schedule_oneshot_event (task->client->aio_context, + nfs_co_generic_bh_cb, task); } static int coroutine_fn nfs_co_preadv(BlockDriverState *bs, uint64_t offset,
block/null.c +3 -1
@@ -16,6 +16,7 @@#include "qapi/qmp/qstring.h" #include "qemu/option.h" #include "block/block_int.h" +#include "sysemu/replay.h" #define NULL_OPT_LATENCY "latency-ns"#define NULL_OPT_ZEROES "read-zeroes"
@@ -178,7 +179,8 @@ static inline BlockAIOCB *null_aio_common(BlockDriverState *bs,timer_mod_ns (&acb->timer,qemu_clock_get_ns (QEMU_CLOCK_REALTIME ) + s->latency_ns); }else { -aio_bh_schedule_oneshot (bdrv_get_aio_context (bs), null_bh_cb, acb); +replay_bh_schedule_oneshot_event (bdrv_get_aio_context (bs), + null_bh_cb, acb); }return &acb->common; }
block/nvme.c +4 -2
@@ -21,6 +21,7 @@#include "qemu/option.h" #include "qemu/vfio-helpers.h" #include "block/block_int.h" +#include "sysemu/replay.h" #include "trace.h" #include "block/nvme.h"
@@ -346,7 +347,8 @@ static bool nvme_process_completion(BDRVNVMeState *s, NVMeQueuePair *q)smp_mb_release (); *q->cq.doorbell =cpu_to_le32 (q->cq.head);if (!qemu_co_queue_empty (&q->free_req_queue)) { -aio_bh_schedule_oneshot (s->aio_context, nvme_free_req_queue_cb, q); +replay_bh_schedule_oneshot_event (s->aio_context, + nvme_free_req_queue_cb, q); } } q->busy = false;
@@ -897,7 +899,7 @@ static void nvme_rw_cb(void *opaque, int ret)/* The rw coroutine hasn't yielded, don't try to enter. */ return ; } -aio_bh_schedule_oneshot (data->ctx, nvme_rw_cb_bh, data); +replay_bh_schedule_oneshot_event (data->ctx, nvme_rw_cb_bh, data); } static coroutine_fn int nvme_co_prw_aligned(BlockDriverState *bs,
block/rbd.c +3 -2
@@ -21,6 +21,7 @@#include "block/qdict.h" #include "crypto/secret.h" #include "qemu/cutils.h" +#include "sysemu/replay.h" #include "qapi/qmp/qstring.h" #include "qapi/qmp/qdict.h" #include "qapi/qmp/qjson.h"
@@ -858,8 +859,8 @@ static void rbd_finish_aiocb(rbd_completion_t c, RADOSCB *rcb) rcb->ret =rbd_aio_get_return_value (c);rbd_aio_release (c); -aio_bh_schedule_oneshot (bdrv_get_aio_context (acb->common.bs), - rbd_finish_bh, rcb); +replay_bh_schedule_oneshot_event (bdrv_get_aio_context (acb->common.bs), + rbd_finish_bh, rcb); } static int rbd_aio_discard_wrapper(rbd_image_t image,
block/vxhs.c +3 -2
@@ -21,6 +21,7 @@#include "qapi/error.h" #include "qemu/uuid.h" #include "crypto/tlscredsx509.h" +#include "sysemu/replay.h" #define VXHS_OPT_FILENAME "filename"#define VXHS_OPT_VDISK_ID "vdisk-id"
@@ -104,8 +105,8 @@ static void vxhs_iio_callback(void *ctx, uint32_t opcode, uint32_t error)trace_vxhs_iio_callback (error); } -aio_bh_schedule_oneshot (bdrv_get_aio_context (acb->common.bs), - vxhs_complete_aio_bh, acb); +replay_bh_schedule_oneshot_event (bdrv_get_aio_context (acb->common.bs), + vxhs_complete_aio_bh, acb);break ;default :
include/sysemu/replay.h +3 -0
@@ -163,6 +163,9 @@ bool replay_events_enabled(void);void replay_flush_events (void );/*! Adds bottom half event to the queue */ void replay_bh_schedule_event (QEMUBH *bh); +/* Adds oneshot bottom half event to the queue */ +void replay_bh_schedule_oneshot_event (AioContext *ctx, +QEMUBHFunc *cb,void *opaque);/*! Adds input event to the queue */ void replay_input_event (QemuConsole *src,InputEvent *evt);/*! Adds input sync event to the queue */
replay/replay-events.c +16 -0
@@ -37,6 +37,9 @@ static void replay_run_event(Event *event)case REPLAY_ASYNC_EVENT_BH :aio_bh_call (event->opaque);break ; +case REPLAY_ASYNC_EVENT_BH_ONESHOT : + ((QEMUBHFunc *)event->opaque)(event->opaque2); +break ;case REPLAY_ASYNC_EVENT_INPUT :qemu_input_event_send_impl (NULL , (InputEvent *)event->opaque);qapi_free_InputEvent ((InputEvent *)event->opaque);
@@ -132,6 +135,17 @@ void replay_bh_schedule_event(QEMUBH *bh) } } +void replay_bh_schedule_oneshot_event (AioContext *ctx, +QEMUBHFunc *cb,void *opaque) +{ +if (events_enabled) { +uint64_t id =replay_get_current_step (); +replay_add_event (REPLAY_ASYNC_EVENT_BH_ONESHOT , cb, opaque, id); + }else { +aio_bh_schedule_oneshot (ctx, cb, opaque); + } +} +void replay_add_input_event (struct InputEvent *event) {replay_add_event (REPLAY_ASYNC_EVENT_INPUT , event,NULL ,0 );
@@ -162,6 +176,7 @@ static void replay_save_event(Event *event, int checkpoint)/* save event-specific data */ switch (event->event_kind) {case REPLAY_ASYNC_EVENT_BH : +case REPLAY_ASYNC_EVENT_BH_ONESHOT :replay_put_qword (event->id);break ;case REPLAY_ASYNC_EVENT_INPUT :
@@ -217,6 +232,7 @@ static Event *replay_read_event(int checkpoint)/* Events that has not to be in the queue */ switch (replay_state.read_event_kind) {case REPLAY_ASYNC_EVENT_BH : +case REPLAY_ASYNC_EVENT_BH_ONESHOT :if (replay_state.read_event_id ==-1 ) { replay_state.read_event_id =replay_get_qword (); }
replay/replay-internal.h +1 -0
@@ -51,6 +51,7 @@ enum ReplayEvents {enum ReplayAsyncEventKind {REPLAY_ASYNC_EVENT_BH , +REPLAY_ASYNC_EVENT_BH_ONESHOT ,REPLAY_ASYNC_EVENT_INPUT ,REPLAY_ASYNC_EVENT_INPUT_SYNC ,REPLAY_ASYNC_EVENT_CHAR_READ ,
replay/replay.c +1 -1
@@ -22,7 +22,7 @@/* Current version of the replay mechanism. Increase it when file format changes. */ -#define REPLAY_VERSION 0xe02007 +#define REPLAY_VERSION 0xe02008/* Size of replay log header */ #define HEADER_SIZE (sizeof(uint32_t) + sizeof(uint64_t))
stubs/Makefile.objs +1 -0
@@ -20,6 +20,7 @@ stub-obj-y += monitor.o stub-obj-y += notify-event.o stub-obj-y += qtest.o stub-obj-y += replay.o +stub-obj-y += replay-user.o stub-obj-y += runstate-check.o stub-obj-y += set-fd-handler.o stub-obj-y += sysbus.o
stubs/replay-user.c new file +9 -0
@@ -0,0 +1,9 @@ +#include "qemu/osdep.h" +#include "sysemu/replay.h" +#include "sysemu/sysemu.h" + +void replay_bh_schedule_oneshot_event (AioContext *ctx, +QEMUBHFunc *cb,void *opaque) +{ +aio_bh_schedule_oneshot (ctx, cb, opaque); +}
Re: [Qemu-devel] [PATCH v13 19/25] replay: add BH oneshot event for block layer
Am 21.02.2019 um 12:05 hat Pavel Dovgalyuk geschrieben:
Quoted 23 lines
Replay is capable of recording normal BH events, but sometimes there are single use callbacks scheduled with
aio_bh_schedule_oneshotfunction. This patch enables recording and replaying such callbacks. Block layer uses these events for calling the completion function. Replaying these calls makes the execution deterministic.Signed-off-by Pavel Dovgalyuk <[email protected]>--
v6:
- moved stub function to the separate file for fixing linux-user build
v10:
- replaced all block layeraio_bh_schedule_oneshotcalls--- block/block-backend.c | 8 +++++--- block/io.c | 4 ++-- block/iscsi.c | 5 +++-- block/nfs.c | 5 +++-- block/null.c | 4 +++- block/nvme.c | 6 ++++-- block/rbd.c | 5 +++-- block/vxhs.c | 5 +++--
Quoted 8 lines
include/sysemu/replay.h | 3 +++ replay/replay-events.c | 16 ++++++++++++++++ replay/replay-internal.h | 1 + replay/replay.c | 2 +- stubs/Makefile.objs | 1 + stubs/replay-user.c | 9 +++++++++ 14 files changed, 57 insertions(+), 17 deletions(-) create mode 100644 stubs/replay-user.c
This still doesn't catch all instances, e.g. everything that goes through aio_co_schedule() is missing.
But I fully expect this to get broken anyway all the time because nobody understands which function to use, and if it works for your special case now and we'll fix other stuff as you encouter it, maybe that's good enough for you.
block/block-backend.c static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, int bytes, · line 1356
Quoted 9 lines
block/block-backend.c +2
-2@@ -1349,8 +1351,8 @@ static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, int bytes, acb->has_returned = true;if (acb->rwco.ret !=NOT_DONE ) { -aio_bh_schedule_oneshot (blk_get_aio_context (blk), - blk_aio_complete_bh, acb); +replay_bh_schedule_oneshot_event (blk_get_aio_context (blk), + blk_aio_complete_bh, acb); }
This, and a few other places that you convert, are in fast paths and add some calls that are unnecessary for non-replay cases.
I wonder if we could make replay optional in ./configure and then make replay_bh_schedule_oneshot_event() a static inline function that can get optimised away at compile time if the feature is disabled.
Kevin
Re: [Qemu-devel] [PATCH v13 19/25] replay: add BH oneshot event for block layer
Quoted 10 lines
From: Kevin Wolf [mailto:[email protected]]
Am 21.02.2019 um 12:05 hat Pavel Dovgalyuk geschrieben:Quoted 8 lines
Replay is capable of recording normal BH events, but sometimes there are single use callbacks scheduled with
aio_bh_schedule_oneshotfunction. This patch enables recording and replaying such callbacks. Block layer uses these events for calling the completion function. Replaying these calls makes the execution deterministic.Signed-off-by Pavel Dovgalyuk <[email protected]>
Quoted 8 lines
Quoted 6 lines
--
v6:
- moved stub function to the separate file for fixing linux-user build
v10:
- replaced all block layeraio_bh_schedule_oneshotcallsThis still doesn't catch all instances, e.g. everything that goes through
aio_co_schedule()is missing.
It seems, that everything else is synchronized with blkreplay driver which is mandatory when using block devices in rr mode.
Quoted 4 lines
But I fully expect this to get broken anyway all the time because nobody understands which function to use, and if it works for your special case now and we'll fix other stuff as you encouter it, maybe that's good enough for you.
This problem exists in every subsystem and it is ok for now, when record/replay is not mature enough, and not familiar for others.
When virtual devices are updated, developers may miss correct loadvm/savevm implementation. For example, loading the audio device state may miss shift the phase of the output signal. Nobody will notice that bug in the migration process, but it reveals when we use record/replay.
We can't cover everything with record/replay tests. Most of the new bugs can be revealed in complex configurations after billions of executed instructions. But when this feature will be available out of the box, we'll
at least get more smoke testing.
Quoted 12 lines
Review static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, int
Quoted 2 lines
(quoted diff) +0
-0@@ -1349,8 +1351,8 @@ static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, intbytes,
block/block-backend.c static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, int bytes, · line 1356
Quoted 8 lines
block/block-backend.c +2
-2acb->has_returned = true;if (acb->rwco.ret !=NOT_DONE ) { -aio_bh_schedule_oneshot (blk_get_aio_context (blk), - blk_aio_complete_bh, acb); +replay_bh_schedule_oneshot_event (blk_get_aio_context (blk), + blk_aio_complete_bh, acb); }
Quoted 2 lines
This, and a few other places that you convert, are in fast paths and add some calls that are unnecessary for non-replay cases.
I don't think that this can make a noticeable slowdown, but we can run the tests if you want.
We have the test suite which performs disk-intensive computation. It was created to measure the effect of running BH callbacks through the virtual timer infrastructure.
Quoted 3 lines
I wonder if we could make replay optional in ./configure and then make
replay_bh_schedule_oneshot_event()a static inline function that can get optimised away at compile time if the feature is disabled.
It is coupled with icount. However, some icount calls are also lie on the fast paths and are completely useless when icount is not enabled.
Pavel Dovgalyuk
Re: [Qemu-devel] [PATCH v13 19/25] replay: add BH oneshot event for block layer
Am 04.03.2019 um 13:17 hat Pavel Dovgalyuk geschrieben:
Quoted 16 lines
Quoted 16 lines
From: Kevin Wolf [mailto:[email protected]]
Am 21.02.2019 um 12:05 hat Pavel Dovgalyuk geschrieben:Quoted 14 lines
Replay is capable of recording normal BH events, but sometimes there are single use callbacks scheduled with
aio_bh_schedule_oneshotfunction. This patch enables recording and replaying such callbacks. Block layer uses these events for calling the completion function. Replaying these calls makes the execution deterministic.Signed-off-by Pavel Dovgalyuk <[email protected]>--
v6:
- moved stub function to the separate file for fixing linux-user build
v10:
- replaced all block layeraio_bh_schedule_oneshotcalls
Quoted 5 lines
Quoted 2 lines
This still doesn't catch all instances, e.g. everything that goes through
aio_co_schedule()is missing.It seems, that everything else is synchronized with blkreplay driver which is mandatory when using block devices in rr mode.
Ah, yes, this is a good point. blkreplay goes through replay_block_event(), which is where things get synchronised, right?
Does this mean that most of the places where you replaced a BH with your new function don't actually need it either because they are called through blkreplay and will go through replay_block_event() before reaching the guest?
Quoted 12 lines
Quoted 4 lines
But I fully expect this to get broken anyway all the time because nobody understands which function to use, and if it works for your special case now and we'll fix other stuff as you encouter it, maybe that's good enough for you.
This problem exists in every subsystem and it is ok for now, when record/replay is not mature enough, and not familiar for others. When virtual devices are updated, developers may miss correct loadvm/savevm implementation. For example, loading the audio device state may miss shift the phase of the output signal. Nobody will notice that bug in the migration process, but it reveals when we use record/replay.
Quoted 4 lines
We can't cover everything with record/replay tests. Most of the new bugs can be revealed in complex configurations after billions of executed instructions. But when this feature will be available out of the box, we'll at least get more smoke testing.
Ok.
Quoted 11 lines
Quoted 11 lines
Review static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, int
Quoted 1 line
(quoted diff) +0
-0@@ -1349,8 +1351,8 @@ static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, intbytes,
Quoted 8 lines
block/block-backend.c +2
-2acb->has_returned = true;if (acb->rwco.ret !=NOT_DONE ) { -aio_bh_schedule_oneshot (blk_get_aio_context (blk), - blk_aio_complete_bh, acb); +replay_bh_schedule_oneshot_event (blk_get_aio_context (blk), + blk_aio_complete_bh, acb); }
Quoted 8 lines
Quoted 2 lines
This, and a few other places that you convert, are in fast paths and add some calls that are unnecessary for non-replay cases.
I don't think that this can make a noticeable slowdown, but we can run the tests if you want.
We have the test suite which performs disk-intensive computation. It was created to measure the effect of running BH callbacks through the virtual timer infrastructure.
I think this requires quite fast storage to possibly make a difference. Or if you don't have that, maybe a ramdisk or even a null-co:// backend could do the trick. Maybe null-co:// is actually the best option.
Anyway, if it's not too much work for you, running some tests would be good.
Quoted 6 lines
Quoted 3 lines
I wonder if we could make replay optional in ./configure and then make
replay_bh_schedule_oneshot_event()a static inline function that can get optimised away at compile time if the feature is disabled.It is coupled with icount. However, some icount calls are also lie on the fast paths and are completely useless when icount is not enabled.
Well, the common fast path is KVM, which doesn't have icount at all, so that might make it less critical. :-)
I get your point, though maybe that just means that both should be possible to be disabled at configure time.
Kevin
Re: [Qemu-devel] [PATCH v13 19/25] replay: add BH oneshot event for block layer
Quoted 18 lines
From: Kevin Wolf [mailto:[email protected]]
Am 04.03.2019 um 13:17 hat Pavel Dovgalyuk geschrieben:Quoted 16 lines
Quoted 16 lines
From: Kevin Wolf [mailto:[email protected]]
Am 21.02.2019 um 12:05 hat Pavel Dovgalyuk geschrieben:Quoted 14 lines
Replay is capable of recording normal BH events, but sometimes there are single use callbacks scheduled with
aio_bh_schedule_oneshotfunction. This patch enables recording and replaying such callbacks. Block layer uses these events for calling the completion function. Replaying these calls makes the execution deterministic.Signed-off-by Pavel Dovgalyuk <[email protected]>--
v6:
- moved stub function to the separate file for fixing linux-user build
v10:
- replaced all block layeraio_bh_schedule_oneshotcalls
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
This still doesn't catch all instances, e.g. everything that goes through
aio_co_schedule()is missing.It seems, that everything else is synchronized with blkreplay driver which is mandatory when using block devices in rr mode.
Ah, yes, this is a good point. blkreplay goes through
replay_block_event(), which is where things get synchronised, right?
Right.
Quoted 4 lines
Does this mean that most of the places where you replaced a BH with your new function don't actually need it either because they are called through blkreplay and will go through
replay_block_event()before reaching the guest?
I'm not sure, maybe some of them are going through the blkreplay driver.
Quoted 15 lines
Quoted 1 line
Quoted 1 line
Review static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset,
Quoted 1 line
(quoted diff) +0
-0@@ -1349,8 +1351,8 @@ static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset,int
Quoted 13 lines
Quoted 12 lines
bytes,
block/block-backend.c static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, int bytes, · line 1356
Quoted 8 lines
block/block-backend.c +2
-2acb->has_returned = true;if (acb->rwco.ret !=NOT_DONE ) { -aio_bh_schedule_oneshot (blk_get_aio_context (blk), - blk_aio_complete_bh, acb); +replay_bh_schedule_oneshot_event (blk_get_aio_context (blk), + blk_aio_complete_bh, acb); }This, and a few other places that you convert, are in fast paths and add some calls that are unnecessary for non-replay cases.
Quoted 7 lines
Quoted 5 lines
I don't think that this can make a noticeable slowdown, but we can run the tests if you want.
We have the test suite which performs disk-intensive computation. It was created to measure the effect of running BH callbacks through the virtual timer infrastructure.I think this requires quite fast storage to possibly make a difference.
True.
Quoted 2 lines
Or if you don't have that, maybe a ramdisk or even a null-co:// backend could do the trick. Maybe null-co:// is actually the best option.
We've got tests with file copying and compression on qcow2 disks. How the null backend can be applied there?
Quoted 7 lines
Anyway, if it's not too much work for you, running some tests would be good.
Quoted 4 lines
Quoted 3 lines
I wonder if we could make replay optional in ./configure and then make
replay_bh_schedule_oneshot_event()a static inline function that can get optimised away at compile time if the feature is disabled.
Quoted 5 lines
Quoted 2 lines
It is coupled with icount. However, some icount calls are also lie on the fast paths and are completely useless when icount is not enabled.
Well, the common fast path is KVM, which doesn't have icount at all, so that might make it less critical. :-)
I see.
Quoted 2 lines
I get your point, though maybe that just means that both should be possible to be disabled at configure time.
Right, it may be reasonable for some TCG use cases. This could be a separate patch set, because everything changes.
Pavel Dovgalyuk
Re: [Qemu-devel] [PATCH v13 19/25] replay: add BH oneshot event for block layer
Am 05.03.2019 um 12:04 hat Pavel Dovgalyuk geschrieben:
Quoted 23 lines
Quoted 15 lines
Quoted 1 line
Quoted 1 line
Review static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset,
Quoted 1 line
(quoted diff) +0
-0@@ -1349,8 +1351,8 @@ static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset,int
Quoted 13 lines
Quoted 12 lines
bytes,
block/block-backend.c static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, int bytes, · line 1356
Quoted 8 lines
block/block-backend.c +2
-2acb->has_returned = true;if (acb->rwco.ret !=NOT_DONE ) { -aio_bh_schedule_oneshot (blk_get_aio_context (blk), - blk_aio_complete_bh, acb); +replay_bh_schedule_oneshot_event (blk_get_aio_context (blk), + blk_aio_complete_bh, acb); }This, and a few other places that you convert, are in fast paths and add some calls that are unnecessary for non-replay cases.
Quoted 7 lines
Quoted 5 lines
I don't think that this can make a noticeable slowdown, but we can run the tests if you want.
We have the test suite which performs disk-intensive computation. It was created to measure the effect of running BH callbacks through the virtual timer infrastructure.I think this requires quite fast storage to possibly make a difference.
Quoted 7 lines
True.
Quoted 2 lines
Or if you don't have that, maybe a ramdisk or even a null-co:// backend could do the trick. Maybe null-co:// is actually the best option.
We've got tests with file copying and compression on qcow2 disks. How the null backend can be applied there?
With qcow2, it can't really. null-co:// would work for running something like fio directly against a virtual disk, without any image format involved. Getting the image format out of the way makes things even a little bit faster.
Maybe we should run a micro-benchmark fio with null-co just in addition to your higher level tests?
Quoted 13 lines
Quoted 7 lines
Anyway, if it's not too much work for you, running some tests would be good.
Quoted 4 lines
Quoted 3 lines
I wonder if we could make replay optional in ./configure and then make
replay_bh_schedule_oneshot_event()a static inline function that can get optimised away at compile time if the feature is disabled.Quoted 5 lines
Quoted 2 lines
It is coupled with icount. However, some icount calls are also lie on the fast paths and are completely useless when icount is not enabled.
Well, the common fast path is KVM, which doesn't have icount at all, so that might make it less critical. :-)
Quoted 7 lines
I see.
Quoted 2 lines
I get your point, though maybe that just means that both should be possible to be disabled at configure time.
Right, it may be reasonable for some TCG use cases. This could be a separate patch set, because everything changes.
Yes, I agree.
Kevin
Re: [Qemu-devel] [PATCH v13 19/25] replay: add BH oneshot event for block layer
Quoted 34 lines
From: Kevin Wolf [mailto:[email protected]]
Am 05.03.2019 um 12:04 hat Pavel Dovgalyuk geschrieben:Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Review static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t
Quoted 1 line
(quoted diff) +0
-0@@ -1349,8 +1351,8 @@ static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_toffset,
Quoted 22 lines
Quoted 14 lines
int
Quoted 13 lines
Quoted 12 lines
bytes,
block/block-backend.c static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, int bytes, · line 1356
Quoted 8 lines
block/block-backend.c +2
-2acb->has_returned = true;if (acb->rwco.ret !=NOT_DONE ) { -aio_bh_schedule_oneshot (blk_get_aio_context (blk), - blk_aio_complete_bh, acb); +replay_bh_schedule_oneshot_event (blk_get_aio_context (blk), + blk_aio_complete_bh, acb); }This, and a few other places that you convert, are in fast paths and add some calls that are unnecessary for non-replay cases.
Quoted 7 lines
Quoted 5 lines
I don't think that this can make a noticeable slowdown, but we can run the tests if you want.
We have the test suite which performs disk-intensive computation. It was created to measure the effect of running BH callbacks through the virtual timer infrastructure.I think this requires quite fast storage to possibly make a difference.
Quoted 7 lines
True.
Quoted 2 lines
Or if you don't have that, maybe a ramdisk or even a null-co:// backend could do the trick. Maybe null-co:// is actually the best option.
We've got tests with file copying and compression on qcow2 disks. How the null backend can be applied there?
Quoted 7 lines
With qcow2, it can't really. null-co:// would work for running something like fio directly against a virtual disk, without any image format involved. Getting the image format out of the way makes things even a little bit faster.
Maybe we should run a micro-benchmark fio with null-co just in addition to your higher level tests?
We run something like that:
-drive file={},if=none,id=drv,snapshot -device ide-hd,drive=drv -drive
file={},if=none,id=tst,snapshot -device ide-hd,drive=tst -net none -monitor stdio --enable-kvm -m 2G
I don't really get your idea. What should be added to the command line to run null-co benchmark?
Pavel Dovgalyuk
Re: [Qemu-devel] [PATCH v13 19/25] replay: add BH oneshot event for block layer
Am 06.03.2019 um 09:33 hat Pavel Dovgalyuk geschrieben:
Quoted 42 lines
Quoted 34 lines
From: Kevin Wolf [mailto:[email protected]]
Am 05.03.2019 um 12:04 hat Pavel Dovgalyuk geschrieben:Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Review static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t
Quoted 1 line
(quoted diff) +0
-0@@ -1349,8 +1351,8 @@ static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_toffset,
Quoted 22 lines
Quoted 14 lines
int
Quoted 13 lines
Quoted 12 lines
bytes,
Review static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t · line 1355
Quoted 8 lines
(quoted diff) +2
-2acb->has_returned = true; if (acb->rwco.ret != NOT_DONE) { - aio_bh_schedule_oneshot(blk_get_aio_context(blk), - blk_aio_complete_bh, acb); + replay_bh_schedule_oneshot_event(blk_get_aio_context(blk), + blk_aio_complete_bh, acb); }This, and a few other places that you convert, are in fast paths and add some calls that are unnecessary for non-replay cases.
Quoted 7 lines
Quoted 5 lines
I don't think that this can make a noticeable slowdown, but we can run the tests if you want.
We have the test suite which performs disk-intensive computation. It was created to measure the effect of running BH callbacks through the virtual timer infrastructure.I think this requires quite fast storage to possibly make a difference.
Quoted 7 lines
True.
Quoted 2 lines
Or if you don't have that, maybe a ramdisk or even a null-co:// backend could do the trick. Maybe null-co:// is actually the best option.
We've got tests with file copying and compression on qcow2 disks. How the null backend can be applied there?
Quoted 7 lines
With qcow2, it can't really. null-co:// would work for running something like fio directly against a virtual disk, without any image format involved. Getting the image format out of the way makes things even a little bit faster.
Maybe we should run a micro-benchmark fio with null-co just in addition to your higher level tests?
Quoted 6 lines
We run something like that:
-drive file={},if=none,id=drv,snapshot -device ide-hd,drive=drv -drive
file={},if=none,id=tst,snapshot -device ide-hd,drive=tst -net none -monitor stdio --enable-kvm -m 2GI don't really get your idea. What should be added to the command line to run null-co benchmark?
Something like:
-drive file=null-co://,if=none,id=null -device virtio-blk,drive=null
(Testing with IDE is useless, IDE is slow, you'll never get very high IOPS or bandwidth with it, neither before nor after your series.)
Kevin
Re: [Qemu-devel] [PATCH v13 19/25] replay: add BH oneshot event for block layer
Quoted 47 lines
From: Kevin Wolf [mailto:[email protected]]
Am 06.03.2019 um 09:33 hat Pavel Dovgalyuk geschrieben:Quoted 39 lines
Quoted 39 lines
From: Kevin Wolf [mailto:[email protected]]
Am 05.03.2019 um 12:04 hat Pavel Dovgalyuk geschrieben:Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Review static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t
Quoted 1 line
(quoted diff) +0
-0@@ -1349,8 +1351,8 @@ static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_toffset,
Quoted 22 lines
Quoted 14 lines
int
Quoted 13 lines
Quoted 12 lines
bytes,
Review static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t · line 1355
Quoted 8 lines
(quoted diff) +2
-2acb->has_returned = true; if (acb->rwco.ret != NOT_DONE) { - aio_bh_schedule_oneshot(blk_get_aio_context(blk), - blk_aio_complete_bh, acb); + replay_bh_schedule_oneshot_event(blk_get_aio_context(blk), + blk_aio_complete_bh, acb); }This, and a few other places that you convert, are in fast paths and add some calls that are unnecessary for non-replay cases.
Quoted 7 lines
Quoted 5 lines
I don't think that this can make a noticeable slowdown, but we can run the tests if you want.
We have the test suite which performs disk-intensive computation. It was created to measure the effect of running BH callbacks through the virtual timer infrastructure.I think this requires quite fast storage to possibly make a difference.
Quoted 7 lines
True.
Quoted 2 lines
Or if you don't have that, maybe a ramdisk or even a null-co:// backend could do the trick. Maybe null-co:// is actually the best option.
We've got tests with file copying and compression on qcow2 disks. How the null backend can be applied there?
With qcow2, it can't really. null-co:// would work for running something like fio directly against a virtual disk, without any image format involved. Getting the image format out of the way makes things even a little bit faster.
Quoted 6 lines
Quoted 2 lines
Maybe we should run a micro-benchmark fio with null-co just in addition to your higher level tests?
We run something like that:
-drive file={},if=none,id=drv,snapshot -device ide-hd,drive=drv -drive
file={},if=none,id=tst,snapshot -device ide-hd,drive=tst -net none -monitor stdio --enable-
Quoted 8 lines
kvm -m 2G
Quoted 3 lines
I don't really get your idea. What should be added to the command line to run null-co benchmark?
Something like:
-drive file=null-co://,if=none,id=null -device virtio-blk,drive=null
And this drive should be destination of the copy operations, right?
Quoted 2 lines
(Testing with IDE is useless, IDE is slow, you'll never get very high IOPS or bandwidth with it, neither before nor after your series.)
SCSI or something else?
Pavel Dovgalyuk
Re: [Qemu-devel] [PATCH v13 19/25] replay: add BH oneshot event for block layer
Am 06.03.2019 um 10:18 hat Pavel Dovgalyuk geschrieben:
Quoted 49 lines
Quoted 49 lines
From: Kevin Wolf [mailto:[email protected]]
Am 06.03.2019 um 09:33 hat Pavel Dovgalyuk geschrieben:Quoted 39 lines
Quoted 39 lines
From: Kevin Wolf [mailto:[email protected]]
Am 05.03.2019 um 12:04 hat Pavel Dovgalyuk geschrieben:Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Review static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t
Quoted 1 line
(quoted diff) +0
-0@@ -1349,8 +1351,8 @@ static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_toffset,
Quoted 22 lines
Quoted 14 lines
int
Quoted 13 lines
Quoted 12 lines
bytes,
Review static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t · line 1355
Quoted 8 lines
(quoted diff) +2
-2acb->has_returned = true; if (acb->rwco.ret != NOT_DONE) { - aio_bh_schedule_oneshot(blk_get_aio_context(blk), - blk_aio_complete_bh, acb); + replay_bh_schedule_oneshot_event(blk_get_aio_context(blk), + blk_aio_complete_bh, acb); }This, and a few other places that you convert, are in fast paths and add some calls that are unnecessary for non-replay cases.
Quoted 7 lines
Quoted 5 lines
I don't think that this can make a noticeable slowdown, but we can run the tests if you want.
We have the test suite which performs disk-intensive computation. It was created to measure the effect of running BH callbacks through the virtual timer infrastructure.I think this requires quite fast storage to possibly make a difference.
Quoted 7 lines
True.
Quoted 2 lines
Or if you don't have that, maybe a ramdisk or even a null-co:// backend could do the trick. Maybe null-co:// is actually the best option.
We've got tests with file copying and compression on qcow2 disks. How the null backend can be applied there?
With qcow2, it can't really. null-co:// would work for running something like fio directly against a virtual disk, without any image format involved. Getting the image format out of the way makes things even a little bit faster.
Quoted 6 lines
Quoted 2 lines
Maybe we should run a micro-benchmark fio with null-co just in addition to your higher level tests?
We run something like that:
-drive file={},if=none,id=drv,snapshot -device ide-hd,drive=drv -drive
file={},if=none,id=tst,snapshot -device ide-hd,drive=tst -net none -monitor stdio --enable-kvm -m 2G
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
I don't really get your idea. What should be added to the command line to run null-co benchmark?
Something like:
-drive file=null-co://,if=none,id=null -device virtio-blk,drive=null
And this drive should be destination of the copy operations, right?
I don't know your exact benchmark, but this drive should be where the high I/O rates are, yes.
For getting meaningful numbers, you should have I/O only on the fast test disk (you're talking about a copy, where is source?), you should use direct I/O to get the page cache of the guest out of the way, and you should make sure that multiple requests are issued in parallel.
Quoted 4 lines
Quoted 2 lines
(Testing with IDE is useless, IDE is slow, you'll never get very high IOPS or bandwidth with it, neither before nor after your series.)
SCSI or something else?
I think virtio-blk is still the fastest we have.
Kevin
Re: [Qemu-devel] [PATCH v13 19/25] replay: add BH oneshot event for block layer
Quoted 56 lines
From: Kevin Wolf [mailto:[email protected]]
Am 06.03.2019 um 10:18 hat Pavel Dovgalyuk geschrieben:Quoted 5 lines
Quoted 5 lines
From: Kevin Wolf [mailto:[email protected]]
Am 06.03.2019 um 09:33 hat Pavel Dovgalyuk geschrieben:Quoted 3 lines
Quoted 3 lines
From: Kevin Wolf [mailto:[email protected]]
Am 05.03.2019 um 12:04 hat Pavel Dovgalyuk geschrieben:Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Review static BlockAIOCB *blk_aio_prwv(BlockBackend *blk,
Quoted 1 line
(quoted diff) +0
-0@@ -1349,8 +1351,8 @@ static BlockAIOCB *blk_aio_prwv(BlockBackend *blk,
int64_tQuoted 36 lines
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
offset,
Quoted 22 lines
Quoted 14 lines
int
Quoted 13 lines
Quoted 12 lines
bytes,
Review static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, · line 1355
Quoted 8 lines
(quoted diff) +2
-2acb->has_returned = true; if (acb->rwco.ret != NOT_DONE) { - aio_bh_schedule_oneshot(blk_get_aio_context(blk), - blk_aio_complete_bh, acb); + replay_bh_schedule_oneshot_event(blk_get_aio_context(blk), + blk_aio_complete_bh, acb); }This, and a few other places that you convert, are in fast paths and add some calls that are unnecessary for non-replay cases.
Quoted 7 lines
Quoted 5 lines
I don't think that this can make a noticeable slowdown, but we can run the tests if you want.
We have the test suite which performs disk-intensive computation. It was created to measure the effect of running BH callbacks through the virtual timer infrastructure.I think this requires quite fast storage to possibly make a difference.
Quoted 7 lines
True.
Quoted 2 lines
Or if you don't have that, maybe a ramdisk or even a null-co:// backend could do the trick. Maybe null-co:// is actually the best option.
We've got tests with file copying and compression on qcow2 disks. How the null backend can be applied there?
With qcow2, it can't really. null-co:// would work for running something like fio directly against a virtual disk, without any image format involved. Getting the image format out of the way makes things even a little bit faster.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Maybe we should run a micro-benchmark fio with null-co just in addition to your higher level tests?
We run something like that:
-drive file={},if=none,id=drv,snapshot -device ide-hd,drive=drv -drive
file={},if=none,id=tst,snapshot -device ide-hd,drive=tst -net none -monitor stdio --enable-
Quoted 5 lines
Quoted 5 lines
kvm -m 2G
Quoted 3 lines
I don't really get your idea. What should be added to the command line to run null-co benchmark?
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Something like:
-drive file=null-co://,if=none,id=null -device virtio-blk,drive=null
And this drive should be destination of the copy operations, right?
I don't know your exact benchmark, but this drive should be where the high I/O rates are, yes.
Ok.
Quoted 2 lines
For getting meaningful numbers, you should have I/O only on the fast test disk (you're talking about a copy, where is source?),
We used a qcow2 image as a source.
Quoted 3 lines
you should
use direct I/O to get the page cache of the guest out of the way, and you should make sure that multiple requests are issued in parallel.
Is this possible, if we have only conventional HDDs?
Pavel Dovgalyuk
Re: [Qemu-devel] [PATCH v13 19/25] replay: add BH oneshot event for block layer
Am 06.03.2019 um 10:37 hat Pavel Dovgalyuk geschrieben:
Quoted 65 lines
Quoted 56 lines
From: Kevin Wolf [mailto:[email protected]]
Am 06.03.2019 um 10:18 hat Pavel Dovgalyuk geschrieben:Quoted 5 lines
Quoted 5 lines
From: Kevin Wolf [mailto:[email protected]]
Am 06.03.2019 um 09:33 hat Pavel Dovgalyuk geschrieben:Quoted 3 lines
Quoted 3 lines
From: Kevin Wolf [mailto:[email protected]]
Am 05.03.2019 um 12:04 hat Pavel Dovgalyuk geschrieben:Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Review static BlockAIOCB *blk_aio_prwv(BlockBackend *blk,
Quoted 1 line
(quoted diff) +0
-0@@ -1349,8 +1351,8 @@ static BlockAIOCB *blk_aio_prwv(BlockBackend *blk,
int64_tQuoted 36 lines
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
offset,
Quoted 22 lines
Quoted 14 lines
int
Quoted 13 lines
Quoted 12 lines
bytes,
Review static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, · line 1355
Quoted 8 lines
(quoted diff) +2
-2acb->has_returned = true; if (acb->rwco.ret != NOT_DONE) { - aio_bh_schedule_oneshot(blk_get_aio_context(blk), - blk_aio_complete_bh, acb); + replay_bh_schedule_oneshot_event(blk_get_aio_context(blk), + blk_aio_complete_bh, acb); }This, and a few other places that you convert, are in fast paths and add some calls that are unnecessary for non-replay cases.
Quoted 7 lines
Quoted 5 lines
I don't think that this can make a noticeable slowdown, but we can run the tests if you want.
We have the test suite which performs disk-intensive computation. It was created to measure the effect of running BH callbacks through the virtual timer infrastructure.I think this requires quite fast storage to possibly make a difference.
Quoted 7 lines
True.
Quoted 2 lines
Or if you don't have that, maybe a ramdisk or even a null-co:// backend could do the trick. Maybe null-co:// is actually the best option.
We've got tests with file copying and compression on qcow2 disks. How the null backend can be applied there?
With qcow2, it can't really. null-co:// would work for running something like fio directly against a virtual disk, without any image format involved. Getting the image format out of the way makes things even a little bit faster.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Maybe we should run a micro-benchmark fio with null-co just in addition to your higher level tests?
We run something like that:
-drive file={},if=none,id=drv,snapshot -device ide-hd,drive=drv -drive
file={},if=none,id=tst,snapshot -device ide-hd,drive=tst -net none -monitor stdio --enable-
Quoted 5 lines
Quoted 5 lines
kvm -m 2G
Quoted 3 lines
I don't really get your idea. What should be added to the command line to run null-co benchmark?
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Something like:
-drive file=null-co://,if=none,id=null -device virtio-blk,drive=null
And this drive should be destination of the copy operations, right?
I don't know your exact benchmark, but this drive should be where the high I/O rates are, yes.
Quoted 6 lines
Ok.
Quoted 2 lines
For getting meaningful numbers, you should have I/O only on the fast test disk (you're talking about a copy, where is source?),
We used a qcow2 image as a source.
So the source is going to slow down the I/O and you won't actually test whether the possible maximum changes.
Quoted 5 lines
Quoted 3 lines
you should
use direct I/O to get the page cache of the guest out of the way, and you should make sure that multiple requests are issued in parallel.Is this possible, if we have only conventional HDDs?
null-co:// doesn't access your disk at all, so if this is the only virtual disk that has I/O, the conventional HDD doesn't hurt. But you're right that you probably can't use your physical disk for high performance benchmarks then.
I'm going to suggest once more to use fio for storage testing. Actually, maybe I can find the time to do this myself on my system, too.
But anyway, I think it's good if you're at least aware that the test you used so far was a low-performance test where any possible performance degradations would be dwarved by both the slow physical disk and the slow IDE interface to communicate with the guest (which also enforces to have only one request at the same time).
Kevin
Re: [Qemu-devel] [PATCH v13 19/25] replay: add BH oneshot event for block layer
Quoted 29 lines
From: Kevin Wolf [mailto:[email protected]]
Am 06.03.2019 um 10:37 hat Pavel Dovgalyuk geschrieben:Quoted 11 lines
Quoted 10 lines
From: Kevin Wolf [mailto:[email protected]]
Am 06.03.2019 um 10:18 hat Pavel Dovgalyuk geschrieben:Quoted 5 lines
Quoted 3 lines
Something like:
-drive file=null-co://,if=none,id=null -device virtio-blk,drive=null
And this drive should be destination of the copy operations, right?
I don't know your exact benchmark, but this drive should be where the high I/O rates are, yes.
Quoted 6 lines
Ok.
Quoted 2 lines
For getting meaningful numbers, you should have I/O only on the fast test disk (you're talking about a copy, where is source?),
We used a qcow2 image as a source.
So the source is going to slow down the I/O and you won't actually test whether the possible maximum changes.
Quoted 5 lines
Quoted 3 lines
you should
use direct I/O to get the page cache of the guest out of the way, and you should make sure that multiple requests are issued in parallel.Is this possible, if we have only conventional HDDs?
Quoted 7 lines
null-co:// doesn't access your disk at all, so if this is the only virtual disk that has I/O, the conventional HDD doesn't hurt. But you're right that you probably can't use your physical disk for high performance benchmarks then.
I'm going to suggest once more to use fio for storage testing. Actually, maybe I can find the time to do this myself on my system, too.
Ok, I'll try it.
Quoted 5 lines
But anyway, I think it's good if you're at least aware that the test you used so far was a low-performance test where any possible performance degradations would be dwarved by both the slow physical disk and the slow IDE interface to communicate with the guest (which also enforces to have only one request at the same time).
Pavel Dovgalyuk
Re: [Qemu-devel] [PATCH v13 19/25] replay: add BH oneshot event for block layer
In reply to Kevin WolfQuoted 29 lines
From: Kevin Wolf [mailto:[email protected]]
Am 06.03.2019 um 10:37 hat Pavel Dovgalyuk geschrieben:Quoted 11 lines
Quoted 10 lines
From: Kevin Wolf [mailto:[email protected]]
Am 06.03.2019 um 10:18 hat Pavel Dovgalyuk geschrieben:Quoted 5 lines
Quoted 3 lines
Something like:
-drive file=null-co://,if=none,id=null -device virtio-blk,drive=null
And this drive should be destination of the copy operations, right?
I don't know your exact benchmark, but this drive should be where the high I/O rates are, yes.
Quoted 6 lines
Ok.
Quoted 2 lines
For getting meaningful numbers, you should have I/O only on the fast test disk (you're talking about a copy, where is source?),
We used a qcow2 image as a source.
So the source is going to slow down the I/O and you won't actually test whether the possible maximum changes.
Quoted 5 lines
Quoted 3 lines
you should
use direct I/O to get the page cache of the guest out of the way, and you should make sure that multiple requests are issued in parallel.Is this possible, if we have only conventional HDDs?
Quoted 7 lines
null-co:// doesn't access your disk at all, so if this is the only virtual disk that has I/O, the conventional HDD doesn't hurt. But you're right that you probably can't use your physical disk for high performance benchmarks then.
I'm going to suggest once more to use fio for storage testing. Actually, maybe I can find the time to do this myself on my system, too.
We've made some testing with the following fio configs:
[readtest]
blocksize=4k
filename=/dev/vda
rw=randread
direct=1
buffered=0
ioengine=libaio
iodepth=32
[writetest]
blocksize=4k
filename=/dev/vda
rw=randwrite
direct=1
buffered=0
ioengine=libaio
iodepth=32
One with read, one with write, and one with both.
master branch:
1 read : io=1024.0MB, bw=475545KB/s, iops=118886, runt= 2205msec
2 write: io=1024.0MB, bw=445444KB/s, iops=111361, runt= 2354msec
3 read : io=1024.0MB, bw=229850KB/s, iops=57462, runt= 4562msec
write: io=1024.0MB, bw=227210KB/s, iops=56802, runt= 4615msec
rr branch:
1 read : io=1024.0MB, bw=479021KB/s, iops=119755, runt= 2189msec 2 write: io=1024.0MB, bw=440763KB/s, iops=110190, runt= 2379msec
3 read : io=1024.0MB, bw=230456KB/s, iops=57614, runt= 4550msec
write: io=1024.0MB, bw=228548KB/s, iops=57136, runt= 4588msec
It seems that the difference can't be measured in our environment.
Pavel Dovgalyuk
PATCH v13 20/25 replay: init rtc after enabling the replay
In reply to Pavel DovgalyukThis patch postpones the call of 'configure_rtc' function. This call uses host clock to configure the rtc, but host clock access should be recorded when using icount record/replay mode. Therefore now rtc is configured after switching record/replay mode on.
--- vl.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
vl.c +7 -3
@@ -2961,6 +2961,7 @@ int main(int argc, char **argv, char **envp)DisplayState *ds;QemuOpts *opts, *machine_opts;QemuOpts *icount_opts =NULL , *accel_opts =NULL ; +QemuOpts *rtc_opts =NULL ;QemuOptsList *olist;int optind;const char *optarg;
@@ -3742,9 +3743,9 @@ int main(int argc, char **argv, char **envp) old_param =1 ;break ;case QEMU_OPTION_rtc: - opts =qemu_opts_parse_noisily (qemu_find_opts ("rtc" ), optarg, - false); -if (!opts) { + rtc_opts =qemu_opts_parse_noisily (qemu_find_opts ("rtc" ), + optarg, false); +if (!rtc_opts) {exit (1 ); }break ;
@@ -3950,6 +3951,9 @@ int main(int argc, char **argv, char **envp)loc_set_none ();replay_configure (icount_opts); +if (rtc_opts) { +configure_rtc (rtc_opts); + }if (incoming && !preconfig_exit_requested) { error_report("'preconfig' and 'incoming' options are "
PATCH v13 21/25 replay: document development rules
In reply to Pavel DovgalyukThis patch introduces docs/devel/replay.txt which describes the rules that should be followed to make virtual devices usable in record/replay mode.
--
v9: fixed external virtual clock description (reported by Artem Pisarenko)
--- docs/devel/replay.txt | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 docs/devel/replay.txt
docs/devel/replay.txt new file +46 -0
@@ -0,0 +1,46 @@ +Record/replay mechanism, that could be enabled through icount mode, expects +the virtual devices to satisfy the following requirements. + +The main idea behind this document is that everything that affects +the guest state during execution in icount mode should be deterministic. + +Timers +====== + +All virtual devices should use virtual clock for timers that change the guest +state. Virtual clock is deterministic, therefore such timers are deterministic +too. + +Virtual devices can also use realtime clock for the events that do not change +the guest state directly. When the clock ticking should depend on VM execution +speed, use virtual clock with EXTERNAL attribute. It is not deterministic, +but its speed depends on the guest execution. This clock is used by +the virtual devices (e.g., slirp routing device) that lie outside the +replayed guest. + +Bottom halves +============= + +Bottom half callbacks, that affect the guest state, should be invoked through +replay_bh_schedule_event or replay_bh_schedule_oneshot_event functions. +Their invocations are saved in record mode and synchronized with the existing +log in replay mode. + +Saving/restoring the VM state +============================= + +All fields in the device state structure (including virtual timers) +should be restored by loadvm to the same values they had before savevm. + +Avoid accessing other devices' state, because the order of saving/restoring +is not defined. It means that you should not call functions like +'update_irq' in post_load callback. Save everything explicitly to avoid +the dependencies that may make restoring the VM state non-deterministic. + +Stopping the VM +=============== + +Stopping the guest should not interfere with its state (with the exception +of the network connections, that could be broken by the remote timeouts). +VM can be stopped at any moment of replay by the user. Restarting the VM +after that stop should not break the replay by the unneeded guest state change.
PATCH v13 22/25 util/qemu-timer: refactor deadline calculation for external timers
In reply to Pavel Dovgalyukicount-based record/replay uses qemu_clock_deadline_ns_all to measure the period until vCPU may be interrupted.
This function takes in account the virtual timers, because they belong to the virtual devices that may generate interrupt request or affect the virtual machine state.
However, there are a subset of virtual timers, that are marked with 'external' flag. These do not change the virtual machine state and only based on virtual clock. Calculating the deadling using the external timers breaks the determinism, because they do not belong to the replayed part of the virtual machine.
This patch fixes the deadline calculation for this case.
--- cpus.c | 9 ++++----- include/qemu/timer.h | 7 +++---- qtest.c | 2 +- tests/ptimer-test-stubs.c | 2 +- tests/ptimer-test.c | 4 ++-- util/qemu-timer.c | 41 +++++++++++++++++++++++++++++++++-------- 6 files changed, 44 insertions(+), 21 deletions(-)
cpus.c +4 -5
@@ -550,7 +550,7 @@ void qtest_clock_warp(int64_t dest)assert (qtest_enabled ()); aio_context =qemu_get_aio_context ();while (clock < dest) { -int64_t deadline =qemu_clock_deadline_ns_all (QEMU_CLOCK_VIRTUAL ); +int64_t deadline =virtual_clock_deadline_ns ();int64_t warp =qemu_soonest_timeout (dest - clock, deadline); seqlock_write_lock(&timers_state.vm_clock_seqlock,
@@ -610,7 +610,7 @@ void qemu_start_warp_timer(void)/* We want to use the earliest deadline from ALL vm_clocks */ clock =qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL_RT ); - deadline =qemu_clock_deadline_ns_all (QEMU_CLOCK_VIRTUAL ); + deadline =virtual_clock_deadline_ns ();if (deadline <0 ) {static bool notified;if (!icount_sleep && !notified) {
@@ -1355,7 +1355,7 @@ static int64_t tcg_get_icount_limit(void)int64_t deadline;if (replay_mode !=REPLAY_MODE_PLAY ) { - deadline =qemu_clock_deadline_ns_all (QEMU_CLOCK_VIRTUAL ); + deadline =virtual_clock_deadline_ns ();/* Maintain prior (possibly buggy) behaviour where if no deadline * was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than
@@ -1376,8 +1376,7 @@ static void handle_icount_deadline(void) {assert (qemu_in_vcpu_thread ());if (use_icount) { -int64_t deadline = -qemu_clock_deadline_ns_all (QEMU_CLOCK_VIRTUAL ); +int64_t deadline =virtual_clock_deadline_ns ();if (deadline ==0 ) {/* Wake up other AioContexts. */
include/qemu/timer.h +3 -4
@@ -176,16 +176,15 @@ bool qemu_clock_expired(QEMUClockType type);bool qemu_clock_use_for_deadline (QEMUClockType type);/** -* qemu_clock_deadline_ns_all: -* @type: the clock type +* virtual_clock_deadline_ns: * * Calculate the deadline across all timer lists associated -* with a clock (as opposed to just the default one) +* with virtual clock (excluding external timers) * in nanoseconds, or -1 if no timer is set to expire. * * Returns: time until expiry in nanoseconds or -1 */ -int64_t qemu_clock_deadline_ns_all (QEMUClockType type); +int64_t virtual_clock_deadline_ns (void );/** * qemu_clock_get_main_loop_timerlist:
qtest.c +1 -1
@@ -655,7 +655,7 @@ static void qtest_process_command(CharBackend *chr, gchar **words)int ret =qemu_strtoi64 (words[1 ],NULL ,0 , &ns);g_assert (ret ==0 ); }else { - ns =qemu_clock_deadline_ns_all (QEMU_CLOCK_VIRTUAL ); + ns =virtual_clock_deadline_ns (); }qtest_clock_warp (qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ) + ns);qtest_send_prefix (chr);
tests/ptimer-test-stubs.c +1 -1
@@ -88,7 +88,7 @@ int64_t qemu_clock_get_ns(QEMUClockType type)return ptimer_test_time_ns; } -int64_t qemu_clock_deadline_ns_all (QEMUClockType type) +int64_t virtual_clock_deadline_ns (void ) {QEMUTimerList *timer_list = main_loop_tlg.tl[type];QEMUTimer *t = timer_list->active_timers.next;
tests/ptimer-test.c +2 -2
@@ -50,13 +50,13 @@ static void ptimer_test_set_qemu_time_ns(int64_t ns)static void qemu_clock_step (uint64_t ns) { -int64_t deadline =qemu_clock_deadline_ns_all (QEMU_CLOCK_VIRTUAL ); +int64_t deadline =virtual_clock_deadline_ns ();int64_t advanced_time =qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ) + ns;while (deadline !=-1 && deadline <= advanced_time) {ptimer_test_set_qemu_time_ns (deadline);ptimer_test_expire_qemu_timers (deadline,QEMU_CLOCK_VIRTUAL ); - deadline =qemu_clock_deadline_ns_all (QEMU_CLOCK_VIRTUAL ); + deadline =virtual_clock_deadline_ns (); }ptimer_test_set_qemu_time_ns (advanced_time);
util/qemu-timer.c +33 -8
58 lines not on this page · raw message
PATCH v13 23/25 replay: fix replay shutdown
In reply to Pavel DovgalyukThis patch fixes shutdown of the replay process, which is terminated with the assert when shutdown event is read from the log. replay_finish_event reads new data_kind and therefore the value of data_kind should be preserved to be valid at qemu_system_shutdown_request call.
--- replay/replay.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
replay/replay.c +4 -4
@@ -49,14 +49,14 @@ bool replay_next_event_is(int event) }while (true) { -if (event == replay_state.data_kind) { +unsigned int data_kind = replay_state.data_kind; +if (event == data_kind) { res = true; } -switch (replay_state.data_kind) { +switch (data_kind) { case EVENT_SHUTDOWN ... EVENT_SHUTDOWN_LAST:replay_finish_event (); -qemu_system_shutdown_request (replay_state.data_kind - -EVENT_SHUTDOWN ); +qemu_system_shutdown_request (data_kind -EVENT_SHUTDOWN );break ;default :/* clock, time_t, checkpoint and other events */
PATCH v13 24/25 replay: rename step-related variables and functions
In reply to Pavel DovgalyukThis patch renames replay_get_current_step() and related variables to make these names consistent with hmp/qmp commands.
--- blockdev.c | 2 +- include/sysemu/replay.h | 2 +- migration/savevm.c | 2 +- replay/replay-debugging.c | 30 ++++++++++++++++-------------- replay/replay-events.c | 4 ++-- replay/replay-internal.c | 10 +++++----- replay/replay-internal.h | 10 +++++----- replay/replay-snapshot.c | 6 +++--- replay/replay-time.c | 2 +- replay/replay.c | 26 +++++++++++++------------- 10 files changed, 48 insertions(+), 46 deletions(-)
blockdev.c +1 -1
@@ -1453,7 +1453,7 @@ static void internal_snapshot_prepare(BlkActionState *common, sn->date_nsec = tv.tv_usec *1000 ; sn->vm_clock_nsec =qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL );if (replay_mode !=REPLAY_MODE_NONE ) { - sn->icount =replay_get_current_step (); + sn->icount =replay_get_current_icount (); }else { sn->icount =-1ULL ; }
include/sysemu/replay.h +1 -1
@@ -96,7 +96,7 @@ void replay_breakpoint(void);/* Processing the instructions */ /*! Returns number of executed instructions. */ -uint64_t replay_get_current_step (void ); +uint64_t replay_get_current_icount (void );/*! Returns number of instructions to execute in replay mode. */ int replay_get_instructions (void );/*! Updates instructions counter in replay mode. */
migration/savevm.c +1 -1
@@ -2511,7 +2511,7 @@ int save_snapshot(const char *name, Error **errp) sn->date_nsec = tv.tv_usec *1000 ; sn->vm_clock_nsec =qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL );if (replay_mode !=REPLAY_MODE_NONE ) { - sn->icount =replay_get_current_step (); + sn->icount =replay_get_current_icount (); }else { sn->icount =-1ULL ; }
replay/replay-debugging.c +16 -14
@@ -38,7 +38,7 @@ void hmp_info_replay(Monitor *mon, const QDict *qdict)monitor_printf (mon,"%s execution '%s': instruction count = %" PRId64"\n" , replay_mode ==REPLAY_MODE_RECORD ?"Recording" :"Replaying" , -replay_get_filename (),replay_get_current_step ()); +replay_get_filename (),replay_get_current_icount ()); } }
@@ -51,7 +51,7 @@ ReplayInfo *qmp_query_replay(Error **errp) retval->filename =g_strdup (replay_get_filename ()); retval->has_filename = true; } - retval->icount =replay_get_current_step (); + retval->icount =replay_get_current_icount ();return retval; }
@@ -59,7 +59,7 @@ static void replay_break(uint64_t icount, QEMUTimerCB callback, void *opaque) {assert (replay_mode ==REPLAY_MODE_PLAY );assert (replay_mutex_locked ()); -assert (replay_break_icount >=replay_get_current_step ()); +assert (replay_break_icount >=replay_get_current_icount ());assert (callback); replay_break_icount = icount;
@@ -94,7 +94,7 @@ static void replay_stop_vm(void *opaque)void qmp_replay_break (int64_t icount,Error **errp) {if (replay_mode ==REPLAY_MODE_PLAY ) { -if (icount >=replay_get_current_step ()) { +if (icount >=replay_get_current_icount ()) {replay_break (icount, replay_stop_vm,NULL ); }else { error_setg(errp,
@@ -196,14 +196,14 @@ static void replay_seek(int64_t icount, QEMUTimerCB callback, Error **errp) snapshot =replay_find_nearest_snapshot (icount, &snapshot_icount);if (snapshot) { -if (icount <replay_get_current_step () - ||replay_get_current_step () < snapshot_icount) { +if (icount <replay_get_current_icount () + ||replay_get_current_icount () < snapshot_icount) {vm_stop (RUN_STATE_RESTORE_VM );load_snapshot (snapshot, errp); }g_free (snapshot); } -if (replay_get_current_step () <= icount) { +if (replay_get_current_icount () <= icount) {replay_break (icount, callback,NULL );vm_start (); }else {
@@ -242,8 +242,9 @@ bool replay_reverse_step(void)assert (replay_mode ==REPLAY_MODE_PLAY ); -if (replay_get_current_step () !=0 ) { -replay_seek (replay_get_current_step () -1 , replay_stop_vm_debug, &err); +if (replay_get_current_icount () !=0 ) { +replay_seek (replay_get_current_icount () -1 , + replay_stop_vm_debug, &err);if (err) {error_free (err);return false;
@@ -283,7 +284,7 @@ static void replay_continue_stop(void *opaque)error_free (err);replay_continue_end (); } - replay_last_snapshot =replay_get_current_step (); + replay_last_snapshot =replay_get_current_icount ();return ; }else {/* Seek to the very first step */
@@ -303,15 +304,16 @@ bool replay_reverse_continue(void)assert (replay_mode ==REPLAY_MODE_PLAY ); -if (replay_get_current_step () !=0 ) { -replay_seek (replay_get_current_step () -1 , replay_continue_stop, &err); +if (replay_get_current_icount () !=0 ) { +replay_seek (replay_get_current_icount () -1 , + replay_continue_stop, &err);if (err) {error_free (err);return false; } replay_last_breakpoint =-1LL ; replay_is_debugging = true; - replay_last_snapshot =replay_get_current_step (); + replay_last_snapshot =replay_get_current_icount ();return true; }
@@ -321,5 +323,5 @@ bool replay_reverse_continue(void)void replay_breakpoint (void ) {assert (replay_mode ==REPLAY_MODE_PLAY ); - replay_last_breakpoint =replay_get_current_step (); + replay_last_breakpoint =replay_get_current_icount (); }
replay/replay-events.c +2 -2
@@ -128,7 +128,7 @@ void replay_add_event(ReplayAsyncEventKind event_kind,void replay_bh_schedule_event (QEMUBH *bh) {if (events_enabled) { -uint64_t id =replay_get_current_step (); +uint64_t id =replay_get_current_icount ();replay_add_event (REPLAY_ASYNC_EVENT_BH , bh,NULL , id); }else {qemu_bh_schedule (bh);
@@ -139,7 +139,7 @@ void replay_bh_schedule_oneshot_event(AioContext *ctx, QEMUBHFunc *cb, void *opaque) {if (events_enabled) { -uint64_t id =replay_get_current_step (); +uint64_t id =replay_get_current_icount ();replay_add_event (REPLAY_ASYNC_EVENT_BH_ONESHOT , cb, opaque, id); }else {aio_bh_schedule_oneshot (ctx, cb, opaque);
replay/replay-internal.c +5 -5
@@ -173,7 +173,7 @@ void replay_fetch_data_kind(void)if (!replay_state.has_unread_data) { replay_state.data_kind =replay_get_byte ();if (replay_state.data_kind ==EVENT_INSTRUCTION ) { - replay_state.instructions_count =replay_get_dword (); + replay_state.instruction_count =replay_get_dword (); }replay_check_error (); replay_state.has_unread_data =1 ;
@@ -227,9 +227,9 @@ void replay_mutex_unlock(void) } } -void replay_advance_current_step (uint64_t current_step) +void replay_advance_current_icount (uint64_t current_icount) { -int diff = (int )(current_step - replay_state.current_step); +int diff = (int )(current_icount - replay_state.current_icount);/* Time can only go forward */ assert (diff >=0 );
@@ -237,7 +237,7 @@ void replay_advance_current_step(uint64_t current_step)if (diff >0 ) {replay_put_event (EVENT_INSTRUCTION );replay_put_dword (diff); - replay_state.current_step += diff; + replay_state.current_icount += diff; } }
@@ -246,6 +246,6 @@ void replay_save_instructions(void) {if (replay_file && replay_mode ==REPLAY_MODE_RECORD ) {g_assert (replay_mutex_locked ()); -replay_advance_current_step (replay_get_current_step ()); +replay_advance_current_icount (replay_get_current_icount ()); } }
replay/replay-internal.h +5 -5
@@ -65,10 +65,10 @@ typedef enum ReplayAsyncEventKind ReplayAsyncEventKind; typedef struct ReplayState { /*! Cached clock values. */ int64_t cached_clock[REPLAY_CLOCK_COUNT]; - /*! Current step - number of processed instructions and timer events. */ - uint64_t current_step; + /*! Current icount - number of processed instructions. */ + uint64_t current_icount; /*! Number of instructions to be executed before other events happen. */ - int instructions_count; + int instruction_count; /*! Type of the currently executed event. */ unsigned int data_kind; /*! Flag which indicates that event is not processed yet. */
@@ -127,8 +127,8 @@ void replay_finish_event(void);data_kind variable. */ void replay_fetch_data_kind (void ); -/*! Advance replay_state.current_step to the specified value. */ -void replay_advance_current_step (uint64_t current_step); +/*! Advance replay_state.current_icount to the specified value. */ +void replay_advance_current_icount (uint64_t current_icount);/*! Saves queued events (like instructions and sound). */ void replay_save_instructions (void );
replay/replay-snapshot.c +3 -3
@@ -42,7 +42,7 @@ static int replay_post_load(void *opaque, int version_id) }else if (replay_mode ==REPLAY_MODE_RECORD ) {/* This is only useful for loading the initial state. Therefore reset all the counters. */ - state->instructions_count =0 ; + state->instruction_count =0 ; state->block_request_id =0 ; }
@@ -57,8 +57,8 @@ static const VMStateDescription vmstate_replay = { .post_load = replay_post_load, .fields = (VMStateField []) {VMSTATE_INT64_ARRAY (cached_clock, ReplayState,REPLAY_CLOCK_COUNT ), -VMSTATE_UINT64 (current_step, ReplayState), -VMSTATE_INT32 (instructions_count, ReplayState), +VMSTATE_UINT64 (current_icount, ReplayState), +VMSTATE_INT32 (instruction_count, ReplayState),VMSTATE_UINT32 (data_kind, ReplayState),VMSTATE_UINT32 (has_unread_data, ReplayState),VMSTATE_UINT64 (file_offset, ReplayState),
replay/replay-time.c +1 -1
@@ -25,7 +25,7 @@ int64_t replay_save_clock(ReplayClockKind kind, int64_t clock,* Due to the caller's locking requirements we get the icount from it * instead of using replay_save_instructions(). */ -replay_advance_current_step (raw_icount); +replay_advance_current_icount (raw_icount);replay_put_event (EVENT_CLOCK + kind);replay_put_qword (clock);
replay/replay.c +13 -13
@@ -43,7 +43,7 @@ bool replay_next_event_is(int event)bool res = false;/* nothing to skip - not all instructions used */ -if (replay_state.instructions_count !=0 ) { +if (replay_state.instruction_count !=0 ) {assert (replay_state.data_kind ==EVENT_INSTRUCTION );return event ==EVENT_INSTRUCTION ; }
@@ -66,7 +66,7 @@ bool replay_next_event_is(int event)return res; } -uint64_t replay_get_current_step (void ) +uint64_t replay_get_current_icount (void ) {return cpu_get_icount_raw (); }
@@ -76,9 +76,9 @@ int replay_get_instructions(void)int res =0 ;replay_mutex_lock ();if (replay_next_event_is (EVENT_INSTRUCTION )) { - res = replay_state.instructions_count; + res = replay_state.instruction_count;if (replay_break_icount !=-1LL ) { -uint64_t current =replay_get_current_step (); +uint64_t current =replay_get_current_icount ();assert (replay_break_icount >= current);if (current + res > replay_break_icount) { res = replay_break_icount - current;
@@ -93,16 +93,16 @@ void replay_account_executed_instructions(void) {if (replay_mode ==REPLAY_MODE_PLAY ) {g_assert (replay_mutex_locked ()); -if (replay_state.instructions_count >0 ) { -int count = (int )(replay_get_current_step () - - replay_state.current_step); +if (replay_state.instruction_count >0 ) { +int count = (int )(replay_get_current_icount () + - replay_state.current_icount);/* Time can only go forward */ assert (count >=0 ); - replay_state.instructions_count -= count; - replay_state.current_step += count; -if (replay_state.instructions_count ==0 ) { + replay_state.instruction_count -= count; + replay_state.current_icount += count; +if (replay_state.instruction_count ==0 ) {assert (replay_state.data_kind ==EVENT_INSTRUCTION );replay_finish_event ();/* Wake up iothread. This is required because
@@ -111,7 +111,7 @@ void replay_account_executed_instructions(void)qemu_notify_event (); }/* Execution reached the break step */ -if (replay_break_icount == replay_state.current_step) { +if (replay_break_icount == replay_state.current_icount) {/* Cannot make callback directly from the vCPU thread */ timer_mod_ns (replay_break_timer,qemu_clock_get_ns (QEMU_CLOCK_REALTIME ));
@@ -290,8 +290,8 @@ static void replay_enable(const char *fname, int mode)replay_mutex_init (); replay_state.data_kind =-1 ; - replay_state.instructions_count =0 ; - replay_state.current_step =0 ; + replay_state.instruction_count =0 ; + replay_state.current_icount =0 ; replay_state.has_unread_data =0 ;/* skip file header for RECORD and check it for PLAY */
PATCH v13 25/25 icount: clean up cpu_can_io before jumping to the next block
In reply to Pavel DovgalyukMost of IO instructions can be executed only at the end of the block in icount mode. Therefore translator can set cpu_can_io flag when translating the last instruction.
But when the blocks are chained, then this flag is not reset and may remain set at the beginning of the next block.
This patch resets the flag before "chaining" the translation blocks.
--- accel/tcg/tcg-runtime.c | 2 ++ 1 file changed, 2 insertions(+)
accel/tcg/tcg-runtime.c +2 -0
@@ -151,6 +151,8 @@ void *HELPER(lookup_tb_ptr)(CPUArchState *env)target_ulong cs_base, pc;uint32_t flags; +/* We are going to jump to the next block. can_do_io should be reset */ + cpu->can_do_io = !use_icount; tb =tb_lookup__cpu_state (cpu, &pc, &cs_base, &flags,curr_cflags ());if (tb ==NULL ) {return tcg_ctx->code_gen_epilogue;
Ping.
Can anyone review block-related patches?
Pavel Dovgalyuk
Quoted 198 lines
-----Original Message-----
From: Pavel Dovgalyuk [mailto:[email protected]]
Sent: Thursday, February 21, 2019 2:04 PM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v13 00/25] Fixing record/replay and adding reverse debuggingGDB remote protocol supports reverse debugging of the targets. It includes 'reverse step' and 'reverse continue' operations. The first one finds the previous step of the execution, and the second one is intended to stop at the last breakpoint that would happen when the program is executed normally.
Reverse debugging is possible in the replay mode, when at least one snapshot was created at the record or replay phase. QEMU can use these snapshots for travelling back in time with GDB.
Running the execution in replay mode allows using GDB reverse debugging commands:
- reverse-stepi (or rsi): Steps one instruction to the past. QEMU loads on of the prior snapshots and proceeds to the desired instruction forward. When that step is reaches, execution stops.
- reverse-continue (or rc): Runs execution "backwards". QEMU tries to find breakpoint or watchpoint by loaded prior snapshot and replaying the execution. Then QEMU loads snapshots again and replays to the latest breakpoint. When there are no breakpoints in the examined section of the execution, QEMU finds one more snapshot and tries again. After the first snapshot is processed, execution stops at this snapshot.The set of patches include the following modifications:
- gdbstub update for reverse debugging support
- functions that automatically perform reverse step and reverse continue operations
- hmp/qmp commands for manipulating the replay process
- improvement of the snapshotting for saving the execution step in the snapshot parameters
- other record/replay fixesThe patches are available in the repository:
https://github.com/ispras/qemu/tree/rr-190221v13 changes:
- rebased to make QAPI stuff applicable
- minor reverse step/reverse continue fixv12 changes:
- style fixesv11 changes:
- addedcan_do_ioresetting before jumping to the next block in the chain
- rebase to the latest masterv10 changes:
- added patch for correct deadline calculation with external timers
- updated icount-related documentation in json files (suggested by Markus Armbruster)
- fixed replay shutdown
- renamed some functions and variables to make them consistent with the documentation and displayed messages
- minor changesv9 changes:
- moved rr qapi stuff to the separate file (suggested by Markus Armbruster)
- minor coding style fixesv8 changes:
- rebased to the new master
- added missing fix for prior rr patch
- updated 'since' version number in json-related patchesv7 changes:
- rebased to the new master with upstreamed patches from the series
- several improvements in hmp/qmp commands handling (suggested by Markus Armbruster)
- fixed record/replay with '-rtc base' option enabled
- added document with virtual hardware requirementsv6 changes:
- rebased to the new version of master
- fixed build of linux-user configurations
- added new clock for slirp and vnc timersv5 changes:
- multiple fixes of record/replay bugs appeared after QEMU core update
- changed reverse debugging to 'since 3.1'v4 changes:
- changed 'since 2.13' to 'since 3.0' in json (as suggested by Eric Blake)v3 changes:
- Fixed PS/2 bug with save/load vm, which caused failures of the replay.
- Rebased to the new code base.
- Minor fixes.v2 changes:
- documented reverse debugging
- fixed start vmstate loading in record mode
- documented qcow2 changes (as suggested by Eric Blake)
- made icount SnapshotInfo field optional (as suggested by Eric Blake)
- renamed qmp commands (as suggested by Eric Blake)
- minor changes---
Pavel Dovgalyuk (24):
block: implementbdrv_snapshot_gotofor blkreplay
replay: disable default snapshot for record/replay
replay: update docs for record/replay with block devices
replay: don't drain/flush bdrv queue while RR is working
replay: finish record/replay before closing the disks
qcow2: introduce icount field for snapshots
migration: introduce icount field for snapshots
replay: provide an accessor for rr filename
qapi: introduce replay.json for record/replay-related stuff
replay: introduce info hmp/qmp command
replay: introduce breakpoint at the specified step
replay: implement replay-seek command
replay: refine replay-time module
replay: flush rr queue before loading the vmstate
gdbstub: add reverse step support in replay mode
gdbstub: add reverse continue support in replay mode
replay: describe reverse debugging indocs/replay.txt
replay: add BH oneshot event for block layer
replay: init rtc after enabling the replay
replay: document development rules
util/qemu-timer: refactor deadline calculation for external timers
replay: fix replay shutdown
replay: rename step-related variables and functions
icount: clean upcpu_can_iobefore jumping to the next block[email protected] (1):
replay: add missing fix for internal functionMAINTAINERS | 1 accel/tcg/tcg-runtime.c | 2 accel/tcg/translator.c | 1 block/blkreplay.c | 8 + block/block-backend.c | 8 + block/io.c | 32 ++++ block/iscsi.c | 5 - block/nfs.c | 5 - block/null.c | 4 - block/nvme.c | 6 + block/qapi.c | 18 ++ block/qcow2-snapshot.c | 9 + block/qcow2.h | 2 block/rbd.c | 5 - block/vxhs.c | 5 - blockdev.c | 10 + cpus.c | 30 +++- docs/devel/replay.txt | 46 ++++++ docs/interop/qcow2.txt | 4 + docs/replay.txt | 45 ++++++ exec.c | 8 + gdbstub.c | 52 +++++++ hmp-commands-info.hx | 14 ++ hmp-commands.hx | 53 +++++++ hmp.h | 4 + include/block/snapshot.h | 1 include/qemu/timer.h | 7 - include/sysemu/replay.h | 30 ++++ migration/savevm.c | 11 ++ qapi/Makefile.objs | 2 qapi/block-core.json | 8 + qapi/block.json | 3 qapi/misc.json | 18 -- qapi/qapi-schema.json | 1 qapi/replay.json | 121 +++++++++++++++++ qtest.c | 2 replay/Makefile.objs | 3 replay/replay-debugging.c | 327 +++++++++++++++++++++++++++++++++++++++++++++ replay/replay-events.c | 18 ++ replay/replay-internal.c | 10 + replay/replay-internal.h | 17 +- replay/replay-snapshot.c | 6 - replay/replay-time.c | 36 ++--- replay/replay.c | 56 ++++++-- stubs/Makefile.objs | 1 stubs/replay-user.c | 9 + stubs/replay.c | 10 + tests/ptimer-test-stubs.c | 2 tests/ptimer-test.c | 4 - util/qemu-timer.c | 41 +++++- vl.c | 21 ++- 51 files changed, 1009 insertions(+), 133 deletions(-) create mode 100644 docs/devel/replay.txt create mode 100644 qapi/replay.json create mode 100644 replay/replay-debugging.c create mode 100644 stubs/replay-user.c--
Pavel Dovgalyuk