reverse debugging
PATCH v4 1/19 block: implement bdrv_snapshot_goto for blkreplay
In reply to Pavel DovgalyukFrom: Pavel Dovgalyuk <[email protected]>
This patch enables making snapshots with blkreplay used in block devices.
This function is required to make bdrv_snapshot_goto without calling .bdrv_open which is not implemented.
--- block/blkreplay.c | 8 ++++++++ 1 file changed, 8 insertions(+)
block/blkreplay.c +8 -0
@@ -130,6 +130,12 @@ static int coroutine_fn blkreplay_co_flush(BlockDriverState *bs)return ret; } +static int blkreplay_snapshot_goto (BlockDriverState *bs, +const char *snapshot_id) +{ +return bdrv_snapshot_goto (bs->file->bs, snapshot_id,NULL ); +} +static BlockDriver bdrv_blkreplay = { .format_name ="blkreplay" , .instance_size =0 ,
@@ -145,6 +151,8 @@ static BlockDriver bdrv_blkreplay = { .bdrv_co_pwrite_zeroes = blkreplay_co_pwrite_zeroes, .bdrv_co_pdiscard = blkreplay_co_pdiscard, .bdrv_co_flush = blkreplay_co_flush, + + .bdrv_snapshot_goto = blkreplay_snapshot_goto, };static void bdrv_blkreplay_init (void )
v4 cover reverse debugging
GDB remote protocol supports reverse debugging of the targets. It includes 'reverse step' and 'reverse continue' operations. The first one finds the previous step of the execution, and the second one is intended to stop at the last breakpoint that would happen when the program is executed normally.
Reverse debugging is possible in the replay mode, when at least one snapshot was created at the record or replay phase. QEMU can use these snapshots for travelling back in time with GDB.
Running the execution in replay mode allows using GDB reverse debugging commands:
- reverse-stepi (or rsi): Steps one instruction to the past. QEMU loads on of the prior snapshots and proceeds to the desired instruction forward. When that step is reaches, execution stops.
- reverse-continue (or rc): Runs execution "backwards". QEMU tries to find breakpoint or watchpoint by loaded prior snapshot and replaying the execution. Then QEMU loads snapshots again and replays to the latest breakpoint. When there are no breakpoints in the examined section of the execution, QEMU finds one more snapshot and tries again. After the first snapshot is processed, execution stops at this snapshot.
The set of patches include the following modifications:
- gdbstub update for reverse debugging support
- functions that automatically perform reverse step and reverse continue operations
- hmp/qmp commands for manipulating the replay process
- improvement of the snapshotting for saving the execution step in the snapshot parameters
- other record/replay fixes
The patches are available in the repository:
https://github.com/ispras/qemu/tree/rr-180524
v4 changes:
- changed 'since 2.13' to 'since 3.0' in json (as suggested by Eric Blake)
v3 changes:
- Fixed PS/2 bug with save/load vm, which caused failures of the replay. The patch was sent separately.
- Rebased to the new code base.
- Minor fixes.
v2 changes:
- documented reverse debugging
- fixed start vmstate loading in record mode
- documented qcow2 changes (as suggested by Eric Blake)
- made icount SnapshotInfo field optional (as suggested by Eric Blake)
- renamed qmp commands (as suggested by Eric Blake)
- minor changes
---
Pavel Dovgalyuk (19):
block: implement bdrv_snapshot_goto for blkreplay
replay: disable default snapshot for record/replay
replay: update docs for record/replay with block devices
replay: don't drain/flush bdrv queue while RR is working
replay: finish record/replay before closing the disks
qcow2: introduce icount field for snapshots
migration: introduce icount field for snapshots
replay: introduce info hmp/qmp command
replay: introduce breakpoint at the specified step
replay: implement replay-seek command to proceed to the desired step
replay: flush events when exiting
timer: remove replay clock probe in deadline calculation
replay: refine replay-time module
translator: fix breakpoint processing
replay: flush rr queue before loading the vmstate
gdbstub: add reverse step support in replay mode
gdbstub: add reverse continue support in replay mode
replay: describe reverse debugging in docs/replay.txt
replay: allow loading any snapshots before recording
accel/tcg/translator.c | 9 + block/blkreplay.c | 8 + block/io.c | 22 +++ block/qapi.c | 17 ++- block/qcow2-snapshot.c | 9 + block/qcow2.h | 2 blockdev.c | 10 ++ cpus.c | 19 ++- docs/interop/qcow2.txt | 4 + docs/replay.txt | 45 +++++++ exec.c | 6 + gdbstub.c | 50 +++++++- hmp-commands-info.hx | 14 ++ hmp-commands.hx | 30 +++++ hmp.h | 3 include/block/snapshot.h | 1 include/sysemu/replay.h | 18 +++ migration/savevm.c | 15 +- qapi/block-core.json | 5 + qapi/block.json | 3 qapi/misc.json | 68 +++++++++++ replay/Makefile.objs | 3 replay/replay-debugging.c | 287 +++++++++++++++++++++++++++++++++++++++++++++ replay/replay-events.c | 14 -- replay/replay-internal.h | 10 +- replay/replay-snapshot.c | 17 ++- replay/replay-time.c | 27 ++-- replay/replay.c | 22 +++ stubs/replay.c | 10 ++ util/qemu-timer.c | 11 -- vl.c | 18 ++- 31 files changed, 696 insertions(+), 81 deletions(-) create mode 100644 replay/replay-debugging.c
Signature · 1 line
Pavel Dovgalyuk
PATCH v4 2/19 replay: disable default snapshot for record/replay
From: Pavel Dovgalyuk <[email protected]>
This patch disables setting '-snapshot' option on by default in record/replay mode. This is needed for creating vmstates in record and replay modes.
--- vl.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
vl.c +8 -2
@@ -3187,7 +3187,13 @@ int main(int argc, char **argv, char **envp)drive_add (IF_PFLASH ,-1 , optarg,PFLASH_OPTS );break ;case QEMU_OPTION_snapshot: - snapshot =1 ; + { +Error *blocker =NULL ; + snapshot =1 ; +error_setg (&blocker,QERR_REPLAY_NOT_SUPPORTED , +"-snapshot" ); +replay_add_blocker (blocker); + }break ;case QEMU_OPTION_numa: opts = qemu_opts_parse_noisily(qemu_find_opts("numa"),
@@ -4476,7 +4482,7 @@ int main(int argc, char **argv, char **envp)qapi_free_BlockdevOptions (bdo->bdo);g_free (bdo); } -if (snapshot || replay_mode !=REPLAY_MODE_NONE ) { +if (snapshot) {qemu_opts_foreach (qemu_find_opts ("drive" ), drive_enable_snapshot,NULL ,NULL ); }
PATCH v4 3/19 replay: update docs for record/replay with block devices
In reply to Pavel DovgalyukThis patch updates the description of the command lines for using record/replay with attached block devices.
--- docs/replay.txt | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
docs/replay.txt +9 -3
@@ -27,7 +27,7 @@ Usage of the record/replay: * First, record the execution with the following command line: qemu-system-i386 \ -icount shift=7,rr=record,rrfile=replay.bin \ - -drive file=disk.qcow2,if=none,id=img-direct \ + -drive file=disk.qcow2,if=none,snapshot,id=img-direct \ -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \ -device ide-hd,drive=img-blkreplay \ -netdev user,id=net1 -device rtl8139,netdev=net1 \
@@ -35,7 +35,7 @@ Usage of the record/replay: * After recording, you can replay it by using another command line: qemu-system-i386 \ -icount shift=7,rr=replay,rrfile=replay.bin \ - -drive file=disk.qcow2,if=none,id=img-direct \ + -drive file=disk.qcow2,if=none,snapshot,id=img-direct \ -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \ -device ide-hd,drive=img-blkreplay \ -netdev user,id=net1 -device rtl8139,netdev=net1 \
@@ -223,7 +223,7 @@ Block devices record/replay module intercepts calls of bdrv coroutine functions at the top of block drivers stack. To record and replay block operations the drive must be configured as following: - -drive file=disk.qcow2,if=none,id=img-direct + -drive file=disk.qcow2,if=none,snapshot,id=img-direct -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay -device ide-hd,drive=img-blkreplay
@@ -252,6 +252,12 @@ This snapshot is created at start of recording and restored at start of replaying. It also can be loaded while replaying to roll back the execution. +'snapshot' flag of the disk image must be removed to save the snapshots +in the overlay (or original image) instead of using the temporary overlay. + -drive file=disk.ovl,if=none,id=img-direct + -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay + -device ide-hd,drive=img-blkreplay + Use QEMU monitor to create additional snapshots. 'savevm <name>' command created the snapshot and 'loadvm <name>' restores it. To prevent corruption of the original disk image, use overlay files linked to the original images.
PATCH v4 4/19 replay: don't drain/flush bdrv queue while RR is working
In reply to Pavel DovgalyukIn record/replay mode bdrv queue is controlled by replay mechanism. It does not allow saving or loading the snapshots
when bdrv queue is not empty. Stopping the VM is not blocked by nonempty queue, but flushing the queue is still impossible there, because it may cause deadlocks in replay mode.
This patch disables bdrv_drain_all and bdrv_flush_all in record/replay mode.
--- block/io.c | 22 ++++++++++++++++++++++ cpus.c | 2 -- 2 files changed, 22 insertions(+), 2 deletions(-)
block/io.c +22 -0
@@ -32,6 +32,7 @@#include "qemu/cutils.h" #include "qapi/error.h" #include "qemu/error-report.h" +#include "sysemu/replay.h" #define NOT_DONE 0x7fffffff/* used while emulated sync operation in progress */
@@ -408,6 +409,13 @@ void bdrv_drain_all_begin(void)BdrvNextIterator it;GSList *aio_ctxs =NULL , *ctx; +/* bdrv queue is managed by record/replay, +waiting for finishing the I/O requests may +be infinite */ +if (replay_events_enabled ()) { +return ; + } +/* BDRV_POLL_WHILE() for a node can only be called from its own I/O thread * or the main loop AioContext. We potentially use BDRV_POLL_WHILE() on * nodes in several different AioContexts, so make sure we're in the main
@@ -459,6 +467,13 @@ void bdrv_drain_all_end(void)BlockDriverState *bs;BdrvNextIterator it; +/* bdrv queue is managed by record/replay, +waiting for finishing the I/O requests may +be endless */ +if (replay_events_enabled ()) { +return ; + } +for (bs =bdrv_first (&it); bs; bs =bdrv_next (&it)) {AioContext *aio_context =bdrv_get_aio_context (bs);
@@ -1841,6 +1856,13 @@ int bdrv_flush_all(void)BlockDriverState *bs =NULL ;int result =0 ; +/* bdrv queue is managed by record/replay, +creating new flush request for stopping +the VM may break the determinism */ +if (replay_events_enabled ()) { +return result; + } +for (bs =bdrv_first (&it); bs; bs =bdrv_next (&it)) {AioContext *aio_context =bdrv_get_aio_context (bs);int ret;
cpus.c +0 -2
@@ -1016,7 +1016,6 @@ static int do_vm_stop(RunState state, bool send_stop) }bdrv_drain_all (); -replay_disable_events (); ret =bdrv_flush_all ();return ret;
@@ -2063,7 +2062,6 @@ int vm_prepare_start(void)/* We are sending this now, but the CPUs will be resumed shortly later */ qapi_event_send_resume (&error_abort); -replay_enable_events ();cpu_enable_ticks ();runstate_set (RUN_STATE_RUNNING );vm_state_notify (1 ,RUN_STATE_RUNNING );
PATCH v4 6/19 qcow2: introduce icount field for snapshots
In reply to Pavel DovgalyukThis patch introduces the icount field for saving within the snapshot. It is required for navigation between the snapshots in record/replay mode.
--
v2:
- documented format changes in docs/interop/qcow2.txt (suggested by Eric Blake)
--- block/qcow2-snapshot.c | 7 +++++++ block/qcow2.h | 2 ++ docs/interop/qcow2.txt | 4 ++++ 3 files changed, 13 insertions(+)
block/qcow2-snapshot.c +7 -0
@@ -103,6 +103,12 @@ int qcow2_read_snapshots(BlockDriverState *bs) sn->disk_size = bs->total_sectors *BDRV_SECTOR_SIZE ; } +if (extra_data_size >=24 ) { + sn->icount =be64_to_cpu (extra.icount); + }else { + sn->icount =-1ULL ; + } +/* Read snapshot ID */ sn->id_str =g_malloc (id_str_size +1 ); ret =bdrv_pread (bs->file, offset, sn->id_str, id_str_size);
@@ -209,6 +215,7 @@ static int qcow2_write_snapshots(BlockDriverState *bs)memset (&extra,0 ,sizeof (extra)); extra.vm_state_size_large =cpu_to_be64 (sn->vm_state_size); extra.disk_size =cpu_to_be64 (sn->disk_size); + extra.icount =cpu_to_be64 (sn->icount); id_str_size =strlen (sn->id_str); name_size =strlen (sn->name);
block/qcow2.h +2 -0
@@ -147,6 +147,7 @@ typedef struct QEMU_PACKED QCowSnapshotHeader { typedef struct QEMU_PACKED QCowSnapshotExtraData {uint64_t vm_state_size_large;uint64_t disk_size; +uint64_t icount; } QCowSnapshotExtraData;
@@ -160,6 +161,7 @@ typedef struct QCowSnapshot {uint32_t date_sec;uint32_t date_nsec;uint64_t vm_clock_nsec; +uint64_t icount; } QCowSnapshot;struct Qcow2Cache ;
docs/interop/qcow2.txt +4 -0
@@ -506,6 +506,10 @@ Snapshot table entry: Byte 48 - 55: Virtual disk size of the snapshot in bytes + Byte 56 - 63: icount value which corresponds to + the record/replay step when the snapshot + was taken + Version 3 images must include extra data at least up to byte 55.
PATCH v4 5/19 replay: finish record/replay before closing the disks
In reply to Pavel DovgalyukAfter recent updates block devices cannot be closed on qemu exit. This happens due to the block request polling when replay is not finished. Therefore now we stop execution recording before closing the block devices.
--- replay/replay.c | 2 ++ vl.c | 1 + 2 files changed, 3 insertions(+)
replay/replay.c +2 -0
@@ -366,6 +366,8 @@ void replay_finish(void)g_free (replay_snapshot); replay_snapshot =NULL ; + replay_mode =REPLAY_MODE_NONE ; +replay_finish_events (); }
vl.c +1 -0
@@ -4688,6 +4688,7 @@ int main(int argc, char **argv, char **envp)/* No more vcpu or device emulation activity beyond this point */ vm_shutdown (); +replay_finish ();job_cancel_sync_all ();bdrv_close_all ();
PATCH v4 7/19 migration: introduce icount field for snapshots
In reply to Pavel DovgalyukSaving icount as a parameters of the snapshot allows navigation between them in the execution replay scenario.
This information can be used for finding a specific snapshot for rewinding the recorded execution to the specific moment of the time. E.g., 'reverse step' action needs to load the nearest snapshot which is prior to the current moment of time .
--
v2:
- made icount in SnapshotInfo optional (suggested by Eric Blake)
--- block/qapi.c | 17 +++++++++++++---- block/qcow2-snapshot.c | 2 ++ blockdev.c | 10 ++++++++++ include/block/snapshot.h | 1 + migration/savevm.c | 5 +++++ qapi/block-core.json | 5 ++++- qapi/block.json | 3 ++- 7 files changed, 37 insertions(+), 6 deletions(-)
block/qapi.c +13 -4
@@ -210,6 +210,7 @@ int bdrv_query_snapshot_info_list(BlockDriverState *bs, info->date_nsec = sn_tab[i].date_nsec; info->vm_clock_sec = sn_tab[i].vm_clock_nsec /1000000000 ; info->vm_clock_nsec = sn_tab[i].vm_clock_nsec %1000000000 ; + info->icount = sn_tab[i].icount; info_list =g_new0 (SnapshotInfoList,1 ); info_list->value = info;
@@ -648,14 +649,15 @@ void bdrv_snapshot_dump(fprintf_function func_fprintf, void *f, QEMUSnapshotInfo *sn) {char buf1[128 ], date_buf[128 ], clock_buf[128 ]; +char icount_buf[128 ] = {0 };struct tm tm;time_t ti;int64_t secs;if (!sn) {func_fprintf (f, -"%-10s%-20s%7s%20s%15s" , -"ID" ,"TAG" ,"VM SIZE" ,"DATE" ,"VM CLOCK" ); +"%-10s%-18s%7s%20s%13s%11s" , +"ID" ,"TAG" ,"VM SIZE" ,"DATE" ,"VM CLOCK" ,"ICOUNT" ); }else { ti = sn->date_sec;localtime_r (&ti, &tm);
@@ -668,13 +670,18 @@ void bdrv_snapshot_dump(fprintf_function func_fprintf, void *f, (int )((secs /60 ) %60 ), (int )(secs %60 ), (int)((sn->vm_clock_nsec / 1000000) % 1000)); +if (sn->icount !=-1ULL ) { +snprintf (icount_buf,sizeof (icount_buf), +"%" PRId64, sn->icount); + }func_fprintf (f, -"%-10s%-20s%7s%20s%15s" , +"%-10s%-18s%7s%20s%13s%11s" , sn->id_str, sn->name,get_human_readable_size (buf1,sizeof (buf1), sn->vm_state_size), date_buf, - clock_buf); + clock_buf, + icount_buf); } }
@@ -842,6 +849,8 @@ void bdrv_image_info_dump(fprintf_function func_fprintf, void *f, .date_nsec = elem->value->date_nsec, .vm_clock_nsec = elem->value->vm_clock_sec *1000000000ULL + elem->value->vm_clock_nsec, + .icount = elem->value->has_icount ? + elem->value->icount : -1ULL, };pstrcpy (sn.id_str,sizeof (sn.id_str), elem->value->id);
block/qcow2-snapshot.c +2 -0
@@ -379,6 +379,7 @@ int qcow2_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info) sn->date_sec = sn_info->date_sec; sn->date_nsec = sn_info->date_nsec; sn->vm_clock_nsec = sn_info->vm_clock_nsec; + sn->icount = sn_info->icount;/* Allocate the L1 table of the snapshot and copy the current one there. */ l1_table_offset =qcow2_alloc_clusters (bs, s->l1_size *sizeof (uint64_t ));
@@ -698,6 +699,7 @@ int qcow2_snapshot_list(BlockDriverState *bs, QEMUSnapshotInfo **psn_tab) sn_info->date_sec = sn->date_sec; sn_info->date_nsec = sn->date_nsec; sn_info->vm_clock_nsec = sn->vm_clock_nsec; + sn_info->icount = sn->icount; } *psn_tab = sn_tab;return s->nb_snapshots;
blockdev.c +10 -0
@@ -56,6 +56,7 @@#include "block/trace.h" #include "sysemu/arch_init.h" #include "sysemu/qtest.h" +#include "sysemu/replay.h" #include "qemu/cutils.h" #include "qemu/help_option.h" #include "qemu/throttle-options.h"
@@ -1348,6 +1349,10 @@ SnapshotInfo *qmp_blockdev_snapshot_delete_internal_sync(const char *device, info->vm_state_size = sn.vm_state_size; info->vm_clock_nsec = sn.vm_clock_nsec %1000000000 ; info->vm_clock_sec = sn.vm_clock_nsec /1000000000 ; +if (sn.icount !=-1ULL ) { + info->icount = sn.icount; + info->has_icount = true; + }return info;
@@ -1556,6 +1561,11 @@ static void internal_snapshot_prepare(BlkActionState *common, sn->date_sec = tv.tv_sec; sn->date_nsec = tv.tv_usec *1000 ; sn->vm_clock_nsec =qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ); +if (replay_mode !=REPLAY_MODE_NONE ) { + sn->icount =replay_get_current_step (); + }else { + sn->icount =-1ULL ; + } ret1 =bdrv_snapshot_create (bs, sn);if (ret1 <0 ) {
include/block/snapshot.h +1 -0
@@ -42,6 +42,7 @@ typedef struct QEMUSnapshotInfo {uint32_t date_sec;/* UTC date of the snapshot */ uint32_t date_nsec;uint64_t vm_clock_nsec;/* VM clock relative to boot */ +uint64_t icount;/* record/replay step */ } QEMUSnapshotInfo; int bdrv_snapshot_find(BlockDriverState *bs, QEMUSnapshotInfo *sn_info,
migration/savevm.c +5 -0
@@ -2457,6 +2457,11 @@ int save_snapshot(const char *name, Error **errp) sn->date_sec = tv.tv_sec; sn->date_nsec = tv.tv_usec *1000 ; sn->vm_clock_nsec =qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ); +if (replay_mode !=REPLAY_MODE_NONE ) { + sn->icount =replay_get_current_step (); + }else { + sn->icount =-1ULL ; + }if (name) { ret =bdrv_snapshot_find (bs, old_sn, name);
qapi/block-core.json +4 -1
@@ -26,13 +26,16 @@ # # @vm-clock-nsec: fractional part in nano seconds to be used with vm-clock-sec # +# @icount: current instruction count for execution record/replay (since 3.0) +# # Since: 1.3 # ## { 'struct': 'SnapshotInfo', 'data': { 'id': 'str', 'name': 'str', 'vm-state-size': 'int', 'date-sec': 'int', 'date-nsec': 'int', - 'vm-clock-sec': 'int', 'vm-clock-nsec': 'int' } } + 'vm-clock-sec': 'int', 'vm-clock-nsec': 'int', + '*icount': 'int' } } ## # @ImageInfoSpecificQCow2EncryptionBase:
qapi/block.json +2 -1
@@ -148,7 +148,8 @@ # "date-sec": 1000012, # "date-nsec": 10, # "vm-clock-sec": 100, -# "vm-clock-nsec": 20 +# "vm-clock-nsec": 20, +# "icount": 220414 # } # } #
PATCH v4 8/19 replay: introduce info hmp/qmp command
In reply to Pavel DovgalyukThis patch introduces 'info replay' monitor command and corresponding qmp request.
These commands request the current record/replay mode, replay log file name, and the execution step (number or recorded/replayed instructions).
--
v2:
- renamed info_replay qmp into query-replay (suggested by Eric Blake)
--- hmp-commands-info.hx | 14 ++++++++++++++ hmp.h | 1 + qapi/misc.json | 35 +++++++++++++++++++++++++++++++++++ replay/Makefile.objs | 3 ++- replay/replay-debugging.c | 41 +++++++++++++++++++++++++++++++++++++++++ replay/replay-internal.h | 2 ++ replay/replay.c | 3 +-- 7 files changed, 96 insertions(+), 3 deletions(-) create mode 100644 replay/replay-debugging.c
hmp-commands-info.hx +14 -0
@@ -883,6 +883,20 @@ STEXI Show SEV information. ETEXI + { + .name = "replay", + .args_type = "", + .params = "", + .help = "show parameters of the record/replay", + .cmd = hmp_info_replay, + }, + +STEXI +@item info replay +@findex info replay +Display the current record/replay mode and the currently executing step. +ETEXI + STEXI @end table ETEXI
hmp.h +1 -0
@@ -146,5 +146,6 @@ void hmp_hotpluggable_cpus(Monitor *mon, const QDict *qdict);void hmp_info_vm_generation_id (Monitor *mon,const QDict *qdict);void hmp_info_memory_size_summary (Monitor *mon,const QDict *qdict);void hmp_info_sev (Monitor *mon,const QDict *qdict); +void hmp_info_replay (Monitor *mon,const QDict *qdict);#endif
qapi/misc.json +35 -0
@@ -3098,6 +3098,41 @@ 'data': [ 'none', 'record', 'play' ] } ## +# @ReplayInfo: +# +# Status of the record/replay mode. +# +# @mode: current mode. +# +# @filename: name of the record/replay log file. +# +# @step: current step number. +# +# Since: 3.0 +# +## +{ 'struct': 'ReplayInfo', + 'data': { 'mode': 'ReplayMode', '*filename': 'str', 'step': 'int' } } + +## +# @query-replay: +# +# Retrieves the status of the execution record/replay. +# +# Returns: structure with the properties of the record/replay. +# +# Since: 3.0 +# +# Example: +# +# -> { "execute": "query-replay" } +# <- { "return": { "mode": "play", "filename": "log.rr", "step": 220414 } } +# +## +{ 'command': 'query-replay', + 'returns': 'ReplayInfo' } + +## # @xen-load-devices-state: # # Load the state of all devices from file. The RAM and the block devices
replay/Makefile.objs +2 -1
@@ -6,4 +6,5 @@ common-obj-y += replay-input.o common-obj-y += replay-char.o common-obj-y += replay-snapshot.o common-obj-y += replay-net.o -common-obj-y += replay-audio.o +common-obj-y += replay-audio.o +common-obj-y += replay-debugging.o
replay/replay-debugging.c new file +41 -0
41 lines not on this page · raw message
replay/replay-internal.h +2 -0
@@ -91,6 +91,8 @@ extern ReplayState replay_state;/* File for replay writing */ extern FILE *replay_file; +/*! Name of replay file */ +extern char *replay_filename;void replay_put_byte (uint8_t byte);void replay_put_event (uint8_t event);
replay/replay.c +1 -2
@@ -29,8 +29,7 @@ReplayMode replay_mode =REPLAY_MODE_NONE ;char *replay_snapshot; -/* Name of replay file */ -static char *replay_filename; +char *replay_filename;ReplayState replay_state;static GSList *replay_blockers;
PATCH v4 9/19 replay: introduce breakpoint at the specified step
In reply to Pavel DovgalyukThis patch introduces replay_break qmp and hmp commands. These commands allow stopping at the specified instruction. It may be useful for debugging when there are some known events that should be investigated.
The commands have one argument - number of instructions executed since the start of the replay.
--
v2:
- renamed replay_break qmp command into replay-break (suggested by Eric Blake)
--- hmp-commands.hx | 15 ++++++++++++ hmp.h | 1 + include/sysemu/replay.h | 3 ++ qapi/misc.json | 17 ++++++++++++++ replay/replay-debugging.c | 55 +++++++++++++++++++++++++++++++++++++++++++++ replay/replay-internal.h | 4 +++ replay/replay.c | 17 ++++++++++++++ 7 files changed, 112 insertions(+)
hmp-commands.hx +15 -0
@@ -1849,6 +1849,21 @@ Set QOM property @var{property} of object at location @var{path} to value @var{v ETEXI { + .name = "replay_break", + .args_type = "step:i", + .params = "step", + .help = "sets breakpoint on the specified step of the replay", + .cmd = hmp_replay_break, + }, + +STEXI +@item replay_break @var{step} +@findex replay_break +Set breakpoint on the specified step of the replay. +Execution stops when the specified step is reached. +ETEXI + + { .name = "info", .args_type = "item:s?", .params = "[subcommand]",
hmp.h +1 -0
@@ -147,5 +147,6 @@ void hmp_info_vm_generation_id(Monitor *mon, const QDict *qdict);void hmp_info_memory_size_summary (Monitor *mon,const QDict *qdict);void hmp_info_sev (Monitor *mon,const QDict *qdict);void hmp_info_replay (Monitor *mon,const QDict *qdict); +void hmp_replay_break (Monitor *mon,const QDict *qdict);#endif
include/sysemu/replay.h +3 -0
@@ -71,6 +71,9 @@ void replay_start(void);void replay_finish (void );/*! Adds replay blocker with the specified error description */ void replay_add_blocker (Error *reason); +/*! Sets breakpoint at the specified step. +If step = -1LL the existing breakpoint is removed. */ +void replay_break (int64_t step,QEMUTimerCB callback,void *opaque);/* Processing the instructions */
qapi/misc.json +17 -0
@@ -3133,6 +3133,23 @@ 'returns': 'ReplayInfo' } ## +# @replay-break: +# +# Set breakpoint on the specified step of the replay. +# Execution stops when the specified step is reached. +# +# @step: execution step to stop at +# +# Since: 3.0 +# +# Example: +# +# -> { "execute": "replay-break", "data": { "step": 220414 } } +# +## +{ 'command': 'replay-break', 'data': { 'step': 'int' } } + +## # @xen-load-devices-state: # # Load the state of all devices from file. The RAM and the block devices
replay/replay-debugging.c +55 -0
64 lines not on this page · raw message
replay/replay-internal.h +4 -0
@@ -93,6 +93,10 @@ extern ReplayState replay_state;extern FILE *replay_file;/*! Name of replay file */ extern char *replay_filename; +/*! Step of the replay breakpoint */ +extern int64_t replay_break_step; +/*! Timer for the replay breakpoint callback */ +extern QEMUTimer *replay_break_timer;void replay_put_byte (uint8_t byte);void replay_put_event (uint8_t event);
replay/replay.c +17 -0
@@ -33,6 +33,10 @@ char *replay_filename;ReplayState replay_state;static GSList *replay_blockers; +/* Replay breakpoints */ +int64_t replay_break_step =-1LL ; +QEMUTimer *replay_break_timer; +bool replay_next_event_is (int event) {bool res = false;
@@ -72,6 +76,13 @@ int replay_get_instructions(void)replay_mutex_lock ();if (replay_next_event_is (EVENT_INSTRUCTION )) { res = replay_state.instructions_count; +if (replay_break_step !=-1LL ) { +uint64_t current =replay_get_current_step (); +assert (replay_break_step >= current); +if (current + res > replay_break_step) { + res = replay_break_step - current; + } + } }replay_mutex_unlock ();return res;
@@ -98,6 +109,12 @@ void replay_account_executed_instructions(void)will be read from the log. */ qemu_notify_event (); } +/* Execution reached the break step */ +if (replay_break_step == replay_state.current_step) { +/* Cannot make callback directly from the vCPU thread */ +timer_mod_ns (replay_break_timer, +qemu_clock_get_ns (QEMU_CLOCK_REALTIME )); + } } } }
PATCH v4 10/19 replay: implement replay-seek command to proceed to the desired step
In reply to Pavel DovgalyukThis patch adds hmp/qmp commands replay_seek/replay-seek that proceed the execution to the specified step.
The commands automatically loads nearest snapshot and replay the execution to find the desired step.
--
v2:
- renamed replay_seek qmp command into replay-seek (suggested by Eric Blake)
--- hmp-commands.hx | 15 ++++++++ hmp.h | 1 + qapi/misc.json | 16 ++++++++ replay/replay-debugging.c | 89 +++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 121 insertions(+)
hmp-commands.hx +15 -0
@@ -1864,6 +1864,21 @@ Execution stops when the specified step is reached. ETEXI { + .name = "replay_seek", + .args_type = "step:i", + .params = "step", + .help = "rewinds replay to the specified step", + .cmd = hmp_replay_seek, + }, + +STEXI +@item replay_seek @var{step} +@findex replay_seek +Automatically proceeds to the specified step, when replaying +the execution. +ETEXI + + { .name = "info", .args_type = "item:s?", .params = "[subcommand]",
hmp.h +1 -0
@@ -148,5 +148,6 @@ void hmp_info_memory_size_summary(Monitor *mon, const QDict *qdict);void hmp_info_sev (Monitor *mon,const QDict *qdict);void hmp_info_replay (Monitor *mon,const QDict *qdict);void hmp_replay_break (Monitor *mon,const QDict *qdict); +void hmp_replay_seek (Monitor *mon,const QDict *qdict);#endif
qapi/misc.json +16 -0
@@ -3150,6 +3150,22 @@ { 'command': 'replay-break', 'data': { 'step': 'int' } } ## +# @replay-seek: +# +# Automatically proceeds to the specified step, when replaying +# the execution. +# +# @step: destination execution step +# +# Since: 3.0 +# +# Example: +# +# -> { "execute": "replay-seek", "data": { "step": 220414 } } +## +{ 'command': 'replay-seek', 'data': { 'step': 'int' } } + +## # @xen-load-devices-state: # # Load the state of all devices from file. The RAM and the block devices
replay/replay-debugging.c +89 -0
98 lines not on this page · raw message
PATCH v4 11/19 replay: flush events when exiting
In reply to Pavel DovgalyukThis patch adds events processing when emulation finishes instead of just cleaning the queue. Now the bdrv coroutines will be in consistent state when emulator closes. It allows correct polling of the block layer at exit.
--- replay/replay-events.c | 14 +------------- replay/replay-internal.h | 2 -- 2 files changed, 1 insertion(+), 15 deletions(-)
replay/replay-events.c +1 -13
@@ -94,18 +94,6 @@ void replay_disable_events(void) } } -void replay_clear_events (void ) -{ -g_assert (replay_mutex_locked ()); - -while (!QTAILQ_EMPTY (&events_list)) { -Event *event =QTAILQ_FIRST (&events_list); -QTAILQ_REMOVE (&events_list, event, events); - -g_free (event); - } -} -/*! Adds specified async event to the queue */ void replay_add_event(ReplayAsyncEventKind event_kind, void *opaque,
@@ -308,7 +296,7 @@ void replay_init_events(void)void replay_finish_events (void ) { events_enabled = false; -replay_clear_events (); +replay_flush_events (); }bool replay_events_enabled (void )
replay/replay-internal.h +0 -2
@@ -148,8 +148,6 @@ void replay_init_events(void);void replay_finish_events (void );/*! Flushes events queue */ void replay_flush_events (void ); -/*! Clears events list before loading new VM state */ -void replay_clear_events (void );/*! Returns true if there are any unsaved events in the queue */ bool replay_has_events (void );/*! Saves events from queue into the file */
PATCH v4 12/19 timer: remove replay clock probe in deadline calculation
In reply to Pavel DovgalyukCiro Santilli reported that commit a5ed352596a8b7eb2f9acce34371b944ac3056c4 breaks the execution replay. It happens due to the probing the clock for the new instances of iothread.
However, this probing was made in replay mode for the timer lists that are empty.
This patch removes clock probing in replay mode.
It is an artifact of the old version with another thread model.
--- util/qemu-timer.c | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-)
util/qemu-timer.c +2 -9
@@ -578,17 +578,10 @@ int64_t timerlistgroup_deadline_ns(QEMUTimerListGroup *tlg) {int64_t deadline =-1 ;QEMUClockType type; -bool play = replay_mode ==REPLAY_MODE_PLAY ;for (type =0 ; type <QEMU_CLOCK_MAX ; type++) {if (qemu_clock_use_for_deadline (type)) { -if (!play || type ==QEMU_CLOCK_REALTIME ) { - deadline =qemu_soonest_timeout (deadline, -timerlist_deadline_ns (tlg->tl[type])); - }else { -/* Read clock from the replay file and -do not calculate the deadline, based on virtual clock. */ -qemu_clock_get_ns (type); - } + deadline =qemu_soonest_timeout (deadline, +timerlist_deadline_ns (tlg->tl[type])); } }return deadline;
PATCH v4 13/19 replay: refine replay-time module
In reply to Pavel DovgalyukThis patch removes refactoring artifacts from the replay/replay-time.c
--- replay/replay-time.c | 27 ++++++++++----------------- 1 file changed, 10 insertions(+), 17 deletions(-)
replay/replay-time.c +10 -17
@@ -17,14 +17,12 @@int64_t replay_save_clock (ReplayClockKind kind,int64_t clock) { +g_assert (replay_file); +g_assert (replay_mutex_locked ()); -if (replay_file) { -g_assert (replay_mutex_locked ()); - -replay_save_instructions (); -replay_put_event (EVENT_CLOCK + kind); -replay_put_qword (clock); - } +replay_save_instructions (); +replay_put_event (EVENT_CLOCK + kind); +replay_put_qword (clock);return clock; }
@@ -46,20 +44,15 @@ void replay_read_next_clock(ReplayClockKind kind)/*! Reads next clock event from the input. */ int64_t replay_read_clock (ReplayClockKind kind) { +int64_t ret;g_assert (replay_file &&replay_mutex_locked ());replay_account_executed_instructions (); -if (replay_file) { -int64_t ret; -if (replay_next_event_is (EVENT_CLOCK + kind)) { -replay_read_next_clock (kind); - } - ret = replay_state.cached_clock[kind]; - -return ret; +if (replay_next_event_is (EVENT_CLOCK + kind)) { +replay_read_next_clock (kind); } + ret = replay_state.cached_clock[kind]; -error_report ("REPLAY INTERNAL ERROR %d" , __LINE__); -exit (1 ); +return ret; }
PATCH v4 14/19 translator: fix breakpoint processing
In reply to Pavel DovgalyukQEMU cannot pass through the breakpoints when 'si' command is used in remote gdb. This patch disables inserting the breakpoints when we are already single stepping though the gdb remote protocol. This patch also fixes icount calculation for the blocks that include breakpoints - instruction with breakpoint is not executed and shouldn't be used in icount calculation.
--- accel/tcg/translator.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
accel/tcg/translator.c +6 -2
@@ -34,6 +34,8 @@ void translator_loop_temp_check(DisasContextBase *db)void translator_loop (const TranslatorOps *ops,DisasContextBase *db,CPUState *cpu,TranslationBlock *tb) { +int bp_insn =0 ; +/* Initialize DisasContext */ db->tb = tb; db->pc_first = tb->pc;
@@ -71,11 +73,13 @@ void translator_loop(const TranslatorOps *ops, DisasContextBase *db,tcg_debug_assert (db->is_jmp ==DISAS_NEXT );/* no early exit */ /* Pass breakpoint hits to target for further processing */ -if (unlikely (!QTAILQ_EMPTY (&cpu->breakpoints))) { +if (!db->singlestep_enabled + &&unlikely (!QTAILQ_EMPTY (&cpu->breakpoints))) {CPUBreakpoint *bp; QTAILQ_FOREACH(bp, &cpu->breakpoints, entry) {if (bp->pc == db->pc_next) {if (ops->breakpoint_check (db, cpu, bp)) { + bp_insn =1 ;break ; } }
@@ -118,7 +122,7 @@ void translator_loop(const TranslatorOps *ops, DisasContextBase *db,/* Emit code to exit the TB, as indicated by db->is_jmp. */ ops->tb_stop (db, cpu); -gen_tb_end (db->tb, db->num_insns); +gen_tb_end (db->tb, db->num_insns - bp_insn);/* The disas_log hook may use these values rather than recompute. */ db->tb->size = db->pc_next - db->pc_first;
PATCH v4 15/19 replay: flush rr queue before loading the vmstate
In reply to Pavel DovgalyukNon-empty record/replay queue prevents saving and loading the VM state, because it includes pending bottom halves and block coroutines. But when the new VM state is loaded, we don't have to preserve the consistency of the current state anymore. Therefore this patch just flushes the queue allowing the coroutines to finish.
--- include/sysemu/replay.h | 2 ++ migration/savevm.c | 10 ++++------ replay/replay-internal.h | 2 -- 3 files changed, 6 insertions(+), 8 deletions(-)
include/sysemu/replay.h +2 -0
@@ -132,6 +132,8 @@ void replay_disable_events(void);void replay_enable_events (void );/*! Returns true when saving events is enabled */ bool replay_events_enabled (void ); +/*! Flushes events queue */ +void replay_flush_events (void );/*! Adds bottom half event to the queue */ void replay_bh_schedule_event (QEMUBH *bh);/*! Adds input event to the queue */
migration/savevm.c +4 -6
@@ -2610,12 +2610,6 @@ int load_snapshot(const char *name, Error **errp)AioContext *aio_context;MigrationIncomingState *mis =migration_incoming_get_current (); -if (!replay_can_snapshot ()) { -error_report ("Record/replay does not allow loading snapshot " -"right now. Try once more later." ); -return -EINVAL ; - } -if (!bdrv_all_can_snapshot (&bs)) { error_setg(errp, "Device '%s' is writable but does not support snapshots",
@@ -2649,6 +2643,10 @@ int load_snapshot(const char *name, Error **errp)return -EINVAL ; } +/* Flush the record/replay queue. Now the VM state is going +to change. Therefore we don't need to preserve its consistency */ +replay_flush_events (); +/* Flush all IO requests so they don't interfere with the new state. */ bdrv_drain_all_begin ();
replay/replay-internal.h +0 -2
@@ -146,8 +146,6 @@ void replay_read_next_clock(unsigned int kind);void replay_init_events (void );/*! Clears internal data structures for events handling */ void replay_finish_events (void ); -/*! Flushes events queue */ -void replay_flush_events (void );/*! Returns true if there are any unsaved events in the queue */ bool replay_has_events (void );/*! Saves events from queue into the file */
PATCH v4 16/19 gdbstub: add reverse step support in replay mode
In reply to Pavel DovgalyukGDB remote protocol supports two reverse debugging commands:
reverse step and reverse continue.
This patch adds support of the first one to the gdbstub. Reverse step is intended to step one instruction in the backwards direction. This is not possible in regular execution.
But replayed execution is deterministic, therefore we can load one of the prior snapshots and proceed to the desired step. It is equivalent to stepping one instruction back.
There should be at least one snapshot preceding the debugged part of the replay log.
--- accel/tcg/translator.c | 1 + cpus.c | 14 +++++++++++--- exec.c | 5 +++++ gdbstub.c | 42 +++++++++++++++++++++++++++++++++++++++--- include/sysemu/replay.h | 7 +++++++ replay/replay-debugging.c | 33 +++++++++++++++++++++++++++++++++ stubs/replay.c | 5 +++++ 7 files changed, 101 insertions(+), 6 deletions(-)
accel/tcg/translator.c +1 -0
@@ -17,6 +17,7 @@#include "exec/gen-icount.h" #include "exec/log.h" #include "exec/translator.h" +#include "sysemu/replay.h" /* Pairs with tcg_clear_temp_count. To be called by #TranslatorOps.{translate_insn,tb_stop} if
cpus.c +11 -3
@@ -1042,9 +1042,17 @@ static bool cpu_can_run(CPUState *cpu)static void cpu_handle_guest_debug (CPUState *cpu) { -gdb_set_stop_cpu (cpu); -qemu_system_debug_request (); - cpu->stopped = true; +if (!replay_running_debug ()) { +gdb_set_stop_cpu (cpu); +qemu_system_debug_request (); + cpu->stopped = true; + }else { +if (!cpu->singlestep_enabled) { +cpu_single_step (cpu,SSTEP_ENABLE ); + }else { +cpu_single_step (cpu,0 ); + } + } }#ifdef CONFIG_LINUX
exec.c +5 -0
@@ -2581,6 +2581,11 @@ static void check_watchpoint(int offset, int len, MemTxAttrs attrs, int flags) QTAILQ_FOREACH(wp, &cpu->watchpoints, entry) {if (cpu_watchpoint_address_matches (wp, vaddr, len) && (wp->flags & flags)) { +if (replay_running_debug ()) { +/* Don't process the watchpoints when we are +in a reverse debugging operation. */ +return ; + }if (flags ==BP_MEM_READ ) { wp->flags |=BP_WATCHPOINT_HIT_READ ; }else {
gdbstub.c +39 -3
80 lines not on this page · raw message
include/sysemu/replay.h +7 -0
@@ -74,6 +74,13 @@ void replay_add_blocker(Error *reason);/*! Sets breakpoint at the specified step. If step = -1LL the existing breakpoint is removed. */ void replay_break (int64_t step,QEMUTimerCB callback,void *opaque); +/*! Start making one step in backward direction. +Used by gdbstub for backwards debugging. +Returns true on success. */ +bool replay_reverse_step (void ); +/*! Returns true if replay module is processing +reverse_continue or reverse_step request */ +bool replay_running_debug (void );/* Processing the instructions */
replay/replay-debugging.c +33 -0
@@ -21,6 +21,13 @@#include "block/snapshot.h" #include "migration/snapshot.h" +static bool replay_is_debugging; + +bool replay_running_debug (void ) +{ +return replay_is_debugging; +} +void hmp_info_replay (Monitor *mon,const QDict *qdict) {if (replay_mode ==REPLAY_MODE_NONE ) {
@@ -183,3 +190,29 @@ void hmp_replay_seek(Monitor *mon, const QDict *qdict)return ; } } + +static void replay_stop_vm_debug (void *opaque) +{ + replay_is_debugging = false; +vm_stop (RUN_STATE_DEBUG ); +replay_break (-1LL ,NULL ,NULL ); +} + +bool replay_reverse_step (void ) +{ +Error *err =NULL ; + +assert (replay_mode ==REPLAY_MODE_PLAY ); + +if (replay_get_current_step () !=0 ) { +replay_seek (replay_get_current_step () -1 , &err, replay_stop_vm_debug); +if (err) { +error_free (err); +return false; + } + replay_is_debugging = true; +return true; + } + +return false; +}
stubs/replay.c +5 -0
@@ -80,3 +80,8 @@ void replay_mutex_lock(void)void replay_mutex_unlock (void ) { } + +bool replay_reverse_step (void ) +{ +return false; +}
PATCH v4 17/19 gdbstub: add reverse continue support in replay mode
In reply to Pavel DovgalyukThis patch adds support of the reverse continue operation for gdbstub. Reverse continue finds the last breakpoint that would happen in normal execution from the beginning to the current moment. Implementation of the reverse continue replays the execution twice:
to find the breakpoints that were hit and to seek to the last breakpoint. Reverse continue loads the previous snapshot and tries to find the breakpoint since that moment. If there are no such breakpoints, it proceeds to the earlier snapshot, and so on. When no breakpoints or watchpoints were hit at all, execution stops at the beginning of the replay log.
--- cpus.c | 3 ++ exec.c | 1 + gdbstub.c | 10 ++++++- include/sysemu/replay.h | 6 ++++ replay/replay-debugging.c | 69 +++++++++++++++++++++++++++++++++++++++++++++ stubs/replay.c | 5 +++ 6 files changed, 93 insertions(+), 1 deletion(-)
cpus.c +3 -0
@@ -1048,6 +1048,9 @@ static void cpu_handle_guest_debug(CPUState *cpu) cpu->stopped = true; }else {if (!cpu->singlestep_enabled) { +/* Report about the breakpoint and +make a single step to skip it */ +replay_breakpoint ();cpu_single_step (cpu,SSTEP_ENABLE ); }else {cpu_single_step (cpu,0 );
exec.c +1 -0
@@ -2584,6 +2584,7 @@ static void check_watchpoint(int offset, int len, MemTxAttrs attrs, int flags)if (replay_running_debug ()) {/* Don't process the watchpoints when we are in a reverse debugging operation. */ +replay_breakpoint ();return ; }if (flags ==BP_MEM_READ ) {
gdbstub.c +9 -1
@@ -1111,6 +1111,14 @@ static int gdb_handle_packet(GDBState *s, const char *line_buf)put_packet (s,"E14" );break ; } +case 'c' : +if (replay_reverse_continue ()) { +gdb_continue (s); +return RS_IDLE ; + }else { +put_packet (s,"E14" ); +break ; + }default : goto unknown_command; }
@@ -1381,7 +1389,7 @@ static int gdb_handle_packet(GDBState *s, const char *line_buf)pstrcat (buf,sizeof (buf),";qXfer:features:read+" ); }if (replay_mode ==REPLAY_MODE_PLAY ) { -pstrcat (buf,sizeof (buf),";ReverseStep+" ); +pstrcat (buf,sizeof (buf),";ReverseStep+;ReverseContinue+" ); }put_packet (s, buf);break ;
include/sysemu/replay.h +6 -0
@@ -78,9 +78,15 @@ void replay_break(int64_t step, QEMUTimerCB callback, void *opaque);Used by gdbstub for backwards debugging. Returns true on success. */ bool replay_reverse_step (void ); +/*! Start searching the last breakpoint/watchpoint. +Used by gdbstub for backwards debugging. +Returns true if the process successfully started. */ +bool replay_reverse_continue (void );/*! Returns true if replay module is processing reverse_continue or reverse_step request */ bool replay_running_debug (void ); +/*! Called in reverse debugging mode to collect breakpoint information */ +void replay_breakpoint (void );/* Processing the instructions */
replay/replay-debugging.c +69 -0
78 lines not on this page · raw message
stubs/replay.c +5 -0
@@ -85,3 +85,8 @@ bool replay_reverse_step(void) {return false; } + +bool replay_reverse_continue (void ) +{ +return false; +}
PATCH v4 18/19 replay: describe reverse debugging in docs/replay.txt
In reply to Pavel DovgalyukThis patch updates the documentation and describes usage of the reverse debugging in QEMU+GDB.
--- docs/replay.txt | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
docs/replay.txt +33 -0
@@ -293,6 +293,39 @@ for recording and replaying must contain identical number of ports in record and replay modes, but their backends may differ. E.g., '-serial stdio' in record mode, and '-serial null' in replay mode. +Reverse debugging +----------------- + +Reverse debugging allows "executing" the program in reverse direction. +GDB remote protocol supports "reverse step" and "reverse continue" +commands. The first one steps single instruction backwards in time, +and the second one finds the last breakpoint in the past. + +Recorded executions may be used to enable reverse debugging. QEMU can't +execute the code in backwards direction, but can load a snapshot and +replay forward to find the desired position or breakpoint. + +The following GDB commands are supported: + - reverse-stepi (or rsi) - step one instruction backwards + - reverse-continue (or rc) - find last breakpoint in the past + +Reverse step loads the nearest snapshot and replays the execution until +the required instruction is met. + +Reverse continue may include several passes of examining the execution +between the snapshots. Each of the passes include the following steps: + 1. loading the snapshot + 2. replaying to examine the breakpoints + 3. if breakpoint or watchpoint was met + - loading the snaphot again + - replaying to the required breakpoint + 4. else + - proceeding to the p.1 with the earlier snapshot + +Therefore usage of the reverse debugging requires at least one snapshot +created in advance. See the "Snapshotting" section to learn about running +record/replay and creating the snapshot in these modes. + Replay log format -----------------
PATCH v4 19/19 replay: allow loading any snapshots before recording
In reply to Pavel DovgalyukThis patch enables using -loadvm in recording mode to allow starting the execution recording from any of the available snapshots. It also fixes loading of the record/replay state, therefore snapshots created in replay mode may also be used for starting the new recording.
--- replay/replay-snapshot.c | 17 ++++++++++++----- vl.c | 7 ++++--- 2 files changed, 16 insertions(+), 8 deletions(-)
replay/replay-snapshot.c +12 -5
@@ -33,11 +33,18 @@ static int replay_pre_save(void *opaque)static int replay_post_load (void *opaque,int version_id) {ReplayState *state = opaque; -fseek (replay_file, state->file_offset,SEEK_SET ); -qemu_clock_set_last (QEMU_CLOCK_HOST , state->host_clock_last); -/* If this was a vmstate, saved in recording mode, -we need to initialize replay data fields. */ -replay_fetch_data_kind (); +if (replay_mode ==REPLAY_MODE_PLAY ) { +fseek (replay_file, state->file_offset,SEEK_SET ); +qemu_clock_set_last (QEMU_CLOCK_HOST , state->host_clock_last); +/* If this was a vmstate, saved in recording mode, +we need to initialize replay data fields. */ +replay_fetch_data_kind (); + }else if (replay_mode ==REPLAY_MODE_RECORD ) { +/* This is only useful for loading the initial state. +Therefore reset all the counters. */ + state->instructions_count =0 ; + state->block_request_id =0 ; + }return 0 ; }
vl.c +4 -3
@@ -4651,15 +4651,16 @@ int main(int argc, char **argv, char **envp)replay_checkpoint (CHECKPOINT_RESET );qemu_system_reset (SHUTDOWN_CAUSE_NONE );register_global_state (); -if (replay_mode !=REPLAY_MODE_NONE ) { -replay_vmstate_init (); - }else if (loadvm) { +if (loadvm) {Error *local_err =NULL ;if (load_snapshot (loadvm, &local_err) <0 ) {error_report_err (local_err); autostart =0 ; } } +if (replay_mode !=REPLAY_MODE_NONE ) { +replay_vmstate_init (); + }qdev_prop_check_globals ();if (vmstate_dump_file) {
Ping?
Отправлено с помощью BlueMail
На 28 Май 2018 г., 10:13, в 10:13, Pavel Dovgalyuk <[email protected]> написал:п>GDB remote protocol supports reverse debugging of the targets.
Quoted 114 lines
It includes 'reverse step' and 'reverse continue' operations. The first one finds the previous step of the execution, and the second one is intended to stop at the last breakpoint that would happen when the program is executed normally.
Reverse debugging is possible in the replay mode, when at least one snapshot was created at the record or replay phase. QEMU can use these snapshots for travelling back in time with GDB.
Running the execution in replay mode allows using GDB reverse debugging commands:
- reverse-stepi (or rsi): Steps one instruction to the past. QEMU loads on of the prior snapshots and proceeds to the desired instruction forward. When that step is reaches, execution stops.
- reverse-continue (or rc): Runs execution "backwards".
QEMU tries to find breakpoint or watchpoint by loaded prior snapshot and replaying the execution. Then QEMU loads snapshots again and replays to the latest breakpoint. When there are no breakpoints in the examined section of the execution, QEMU finds one more snapshot and tries again. After the first snapshot is processed, execution stops at this snapshot.The set of patches include the following modifications:
- gdbstub update for reverse debugging support
- functions that automatically perform reverse step and reverse continue operations
- hmp/qmp commands for manipulating the replay process
- improvement of the snapshotting for saving the execution step in the snapshot parameters
- other record/replay fixesThe patches are available in the repository:
https://github.com/ispras/qemu/tree/rr-180524v4 changes:
- changed 'since 2.13' to 'since 3.0' in json (as suggested by Eric Blake)v3 changes:
- Fixed PS/2 bug with save/load vm, which caused failures of the replay.
The patch was sent separately.
- Rebased to the new code base.
- Minor fixes.v2 changes:
- documented reverse debugging
- fixed start vmstate loading in record mode
- documented qcow2 changes (as suggested by Eric Blake)
- made icount SnapshotInfo field optional (as suggested by Eric Blake)
- renamed qmp commands (as suggested by Eric Blake)
- minor changes---
Pavel Dovgalyuk (19):
block: implementbdrv_snapshot_gotofor blkreplay
replay: disable default snapshot for record/replay
replay: update docs for record/replay with block devices
replay: don't drain/flush bdrv queue while RR is working
replay: finish record/replay before closing the disks
qcow2: introduce icount field for snapshots
migration: introduce icount field for snapshots
replay: introduce info hmp/qmp command
replay: introduce breakpoint at the specified step
replay: implement replay-seek command to proceed to the desired step
replay: flush events when exiting
timer: remove replay clock probe in deadline calculation
replay: refine replay-time module
translator: fix breakpoint processing
replay: flush rr queue before loading the vmstate
gdbstub: add reverse step support in replay mode
gdbstub: add reverse continue support in replay mode
replay: describe reverse debugging indocs/replay.txt
replay: allow loading any snapshots before recordingaccel/tcg/translator.c | 9 + block/blkreplay.c | 8 + block/io.c | 22 +++ block/qapi.c | 17 ++- block/qcow2-snapshot.c | 9 + block/qcow2.h | 2 blockdev.c | 10 ++ cpus.c | 19 ++- docs/interop/qcow2.txt | 4 + docs/replay.txt | 45 +++++++ exec.c | 6 + gdbstub.c | 50 +++++++- hmp-commands-info.hx | 14 ++ hmp-commands.hx | 30 +++++ hmp.h | 3 include/block/snapshot.h | 1 include/sysemu/replay.h | 18 +++ migration/savevm.c | 15 +- qapi/block-core.json | 5 + qapi/block.json | 3 qapi/misc.json | 68 +++++++++++ replay/Makefile.objs | 3 replay/replay-debugging.c | 287 +++++++++++++++++++++++++++++++++++++++++++++ replay/replay-events.c | 14 -- replay/replay-internal.h | 10 +- replay/replay-snapshot.c | 17 ++- replay/replay-time.c | 27 ++-- replay/replay.c | 22 +++ stubs/replay.c | 10 ++ util/qemu-timer.c | 11 -- vl.c | 18 ++- 31 files changed, 696 insertions(+), 81 deletions(-)create mode 100644
replay/replay-debugging.c--
Pavel Dovgalyuk
Quoted 1 line · Pavel Dovgalyuk
Ping?
I started having a look but I ran into this straight away. First I recorded a boot of the kernel:
./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=record,rrfile=replay.bin
Then played back:
./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=replay,rrfile=replay.bin -s -S
And did the following on gdb:
(gdb) i0x0000000040000004 in ?? ()
=> 0x40000004: mov x1, xzr 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr (gdb) 0x0000000040000008 in ?? () => 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 (gdb) 0x000000004000000c in ?? () => 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 (gdb) 0x0000000040000010 in ?? () => 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined (gdb) 0x0000000040000014 in ?? () => 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined 0x4000001c: .inst 0x00000000 ; undefined
(gdb) p/x $x4
$1 = 0x40080000
(gdb) reverse-stepi
warning: Remote failure reply: E14
Surely this is the simple case and doesn't require any snapshots for block devices as there are none. Am I missing something?
Quoted 120 lines
Отправлено с помощью BlueMail
На 28 Май 2018 г., 10:13, в 10:13, Pavel Dovgalyuk <[email protected]> написал:п>GDB remote protocol supports reverse debugging of the targets.
Quoted 114 lines
It includes 'reverse step' and 'reverse continue' operations. The first one finds the previous step of the execution, and the second one is intended to stop at the last breakpoint that would happen when the program is executed normally.
Reverse debugging is possible in the replay mode, when at least one snapshot was created at the record or replay phase. QEMU can use these snapshots for travelling back in time with GDB.
Running the execution in replay mode allows using GDB reverse debugging commands:
- reverse-stepi (or rsi): Steps one instruction to the past. QEMU loads on of the prior snapshots and proceeds to the desired instruction forward. When that step is reaches, execution stops.
- reverse-continue (or rc): Runs execution "backwards".
QEMU tries to find breakpoint or watchpoint by loaded prior snapshot and replaying the execution. Then QEMU loads snapshots again and replays to the latest breakpoint. When there are no breakpoints in the examined section of the execution, QEMU finds one more snapshot and tries again. After the first snapshot is processed, execution stops at this snapshot.The set of patches include the following modifications:
- gdbstub update for reverse debugging support
- functions that automatically perform reverse step and reverse continue operations
- hmp/qmp commands for manipulating the replay process
- improvement of the snapshotting for saving the execution step in the snapshot parameters
- other record/replay fixesThe patches are available in the repository:
https://github.com/ispras/qemu/tree/rr-180524v4 changes:
- changed 'since 2.13' to 'since 3.0' in json (as suggested by Eric Blake)v3 changes:
- Fixed PS/2 bug with save/load vm, which caused failures of the replay.
The patch was sent separately.
- Rebased to the new code base.
- Minor fixes.v2 changes:
- documented reverse debugging
- fixed start vmstate loading in record mode
- documented qcow2 changes (as suggested by Eric Blake)
- made icount SnapshotInfo field optional (as suggested by Eric Blake)
- renamed qmp commands (as suggested by Eric Blake)
- minor changes---
Pavel Dovgalyuk (19):
block: implementbdrv_snapshot_gotofor blkreplay
replay: disable default snapshot for record/replay
replay: update docs for record/replay with block devices
replay: don't drain/flush bdrv queue while RR is working
replay: finish record/replay before closing the disks
qcow2: introduce icount field for snapshots
migration: introduce icount field for snapshots
replay: introduce info hmp/qmp command
replay: introduce breakpoint at the specified step
replay: implement replay-seek command to proceed to the desired step
replay: flush events when exiting
timer: remove replay clock probe in deadline calculation
replay: refine replay-time module
translator: fix breakpoint processing
replay: flush rr queue before loading the vmstate
gdbstub: add reverse step support in replay mode
gdbstub: add reverse continue support in replay mode
replay: describe reverse debugging indocs/replay.txt
replay: allow loading any snapshots before recordingaccel/tcg/translator.c | 9 + block/blkreplay.c | 8 + block/io.c | 22 +++ block/qapi.c | 17 ++- block/qcow2-snapshot.c | 9 + block/qcow2.h | 2 blockdev.c | 10 ++ cpus.c | 19 ++- docs/interop/qcow2.txt | 4 + docs/replay.txt | 45 +++++++ exec.c | 6 + gdbstub.c | 50 +++++++- hmp-commands-info.hx | 14 ++ hmp-commands.hx | 30 +++++ hmp.h | 3 include/block/snapshot.h | 1 include/sysemu/replay.h | 18 +++ migration/savevm.c | 15 +- qapi/block-core.json | 5 + qapi/block.json | 3 qapi/misc.json | 68 +++++++++++ replay/Makefile.objs | 3 replay/replay-debugging.c | 287 +++++++++++++++++++++++++++++++++++++++++++++ replay/replay-events.c | 14 -- replay/replay-internal.h | 10 +- replay/replay-snapshot.c | 17 ++- replay/replay-time.c | 27 ++-- replay/replay.c | 22 +++ stubs/replay.c | 10 ++ util/qemu-timer.c | 11 -- vl.c | 18 ++- 31 files changed, 696 insertions(+), 81 deletions(-)create mode 100644
replay/replay-debugging.c--
Pavel Dovgalyuk
--
Alex Bennée
Quoted 12 lines
From: Alex Bennée [mailto:[email protected]]
Quoted 1 line · Pavel Dovgalyuk
Ping?
I started having a look but I ran into this straight away. First I recorded a boot of the kernel:
./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-
version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=record,rrfile=replay.bin
Quoted 6 lines
Then played back:
./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-
version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=replay,rrfile=replay.bin - s -S
This looks ok, but...
Quoted 27 lines
And did the following on gdb:
(gdb) i
0x0000000040000004in ?? ()=> 0x40000004: mov x1, xzr 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr (gdb) 0x0000000040000008 in ?? () => 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 (gdb) 0x000000004000000c in ?? () => 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 (gdb) 0x0000000040000010 in ?? () => 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined (gdb) 0x0000000040000014 in ?? () => 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined 0x4000001c: .inst 0x00000000 ; undefined
Quoted 7 lines
(gdb) p/x $x4
$1 =0x40080000
(gdb) reverse-stepi
warning: Remote failure reply: E14Surely this is the simple case and doesn't require any snapshots for block devices as there are none. Am I missing something?
Reverse debugging requires the snapshotting. QEMU can't revert the VM state without the snapshots. You can try adding an empty qcow2 image to allow snapshotting there.
Pavel Dovgalyuk
Quoted 53 lines · Pavel Dovgalyuk
Quoted 12 lines
From: Alex Bennée [mailto:[email protected]]
Quoted 1 line · Pavel Dovgalyuk
Ping?
I started having a look but I ran into this straight away. First I recorded a boot of the kernel:
./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-
version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=record,rrfile=replay.binQuoted 6 lines
Then played back:
./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-
version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=replay,rrfile=replay.bin - s -SThis looks ok, but...
Quoted 32 lines
And did the following on gdb:
(gdb) i
0x0000000040000004in ?? ()=> 0x40000004: mov x1, xzr 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr (gdb) 0x0000000040000008 in ?? () => 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 (gdb) 0x000000004000000c in ?? () => 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 (gdb) 0x0000000040000010 in ?? () => 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined (gdb) 0x0000000040000014 in ?? () => 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined 0x4000001c: .inst 0x00000000 ; undefined(gdb) p/x $x4
$1 =0x40080000
(gdb) reverse-stepi
warning: Remote failure reply: E14
Quoted 5 lines
Quoted 2 lines
Surely this is the simple case and doesn't require any snapshots for block devices as there are none. Am I missing something?
Reverse debugging requires the snapshotting. QEMU can't revert the VM state without the snapshots. You can try adding an empty qcow2 image to allow snapshotting there.
I got confused with block device snapshots and rr snapshots. Let me try again ;-)
--
Alex Bennée
Quoted 53 lines · Pavel Dovgalyuk
Quoted 12 lines
From: Alex Bennée [mailto:[email protected]]
Quoted 1 line · Pavel Dovgalyuk
Ping?
I started having a look but I ran into this straight away. First I recorded a boot of the kernel:
./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-
version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=record,rrfile=replay.binQuoted 6 lines
Then played back:
./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-
version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=replay,rrfile=replay.bin - s -SThis looks ok, but...
Quoted 32 lines
And did the following on gdb:
(gdb) i
0x0000000040000004in ?? ()=> 0x40000004: mov x1, xzr 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr (gdb) 0x0000000040000008 in ?? () => 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 (gdb) 0x000000004000000c in ?? () => 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 (gdb) 0x0000000040000010 in ?? () => 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined (gdb) 0x0000000040000014 in ?? () => 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined 0x4000001c: .inst 0x00000000 ; undefined(gdb) p/x $x4
$1 =0x40080000
(gdb) reverse-stepi
warning: Remote failure reply: E14
Quoted 5 lines
Quoted 2 lines
Surely this is the simple case and doesn't require any snapshots for block devices as there are none. Am I missing something?
Reverse debugging requires the snapshotting. QEMU can't revert the VM state without the snapshots. You can try adding an empty qcow2 image to allow snapshotting there.
I suspect a recent patch has broken locking again:
Starting program: /home/alex/lsrc/qemu/qemu.git/aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=replay,rrfile=replay.bin,rrsnapshot=debug -drive file=rr.qcow2,if=none,snapshot,id=rr -s -S [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffd8536700 (LWP 32452)]
[New Thread 0x7fffd5bb3700 (LWP 32453)]
[New Thread 0x7fffd4eab700 (LWP 32454)]
**
ERROR:replay/replay-time.c:49:replay_read_clock: assertion failed: (replay_file && replay_mutex_locked())
Once I have the linux-user TCG tests merged I'm planning on focusing on the system emulation tests and we should be able to add some record/replay tests to defend the behaviour.
--
Alex Bennée
Quoted 67 lines
From: Alex Bennée [mailto:[email protected]]
Quoted 11 lines · Pavel Dovgalyuk
Quoted 11 lines
From: Alex Bennée [mailto:[email protected]]
Quoted 1 line · Pavel Dovgalyuk
Ping?
I started having a look but I ran into this straight away. First I recorded a boot of the kernel:
./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-
version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icountshift=7,rr=record,rrfile=replay.bin
Quoted 6 lines
Quoted 6 lines
Then played back:
./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-
version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icountshift=7,rr=replay,rrfile=replay.bin -
Quoted 36 lines
Quoted 1 line
s -S
This looks ok, but...
Quoted 32 lines
And did the following on gdb:
(gdb) i
0x0000000040000004in ?? ()=> 0x40000004: mov x1, xzr 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr (gdb) 0x0000000040000008 in ?? () => 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 (gdb) 0x000000004000000c in ?? () => 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 (gdb) 0x0000000040000010 in ?? () => 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined (gdb) 0x0000000040000014 in ?? () => 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined 0x4000001c: .inst 0x00000000 ; undefined(gdb) p/x $x4
$1 =0x40080000
(gdb) reverse-stepi
warning: Remote failure reply: E14Quoted 4 lines
Quoted 2 lines
Surely this is the simple case and doesn't require any snapshots for block devices as there are none. Am I missing something?
Reverse debugging requires the snapshotting. QEMU can't revert the VM state without the
snapshots.
Quoted 1 line
You can try adding an empty qcow2 image to allow snapshotting there.
I suspect a recent patch has broken locking again:
Quoted 13 lines
Starting program:
/home/alex/lsrc/qemu/qemu.git/aarch64-softmmu/qemu-system-aarch64-machine virt,graphics=on,gic-version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=replay,rrfile=replay.bin,rrsnapshot=debug -drive file=rr.qcow2,if=none,snapshot,id=rr -s -S
[Thread debugging usinglibthread_dbenabled]
Using hostlibthread_dblibrary "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread0x7fffd8536700(LWP 32452)]
[New Thread0x7fffd5bb3700(LWP 32453)]
[New Thread0x7fffd4eab700(LWP 32454)]
**
ERROR:replay/replay-time.c:49:replay_read_clock: assertion failed: (replay_file&&replay_mutex_locked())
Have you recorded it with the attached disk before replaying?
Are you using the latest version?
If the both answers are 'yes', then can you share the kernel? My i386 runs work normally.
Quoted 3 lines
Once I have the linux-user TCG tests merged I'm planning on focusing on the system emulation tests and we should be able to add some record/replay tests to defend the behaviour.
That will be great.
There are some Ciro's attempts on that: https://github.com/cirosantilli/qemu-test/blob/master/arm/rr
Pavel Dovgalyuk
Quoted 81 lines · Pavel Dovgalyuk
Quoted 67 lines
From: Alex Bennée [mailto:[email protected]]
Quoted 11 lines · Pavel Dovgalyuk
Quoted 11 lines
From: Alex Bennée [mailto:[email protected]]
Quoted 1 line · Pavel Dovgalyuk
Ping?
I started having a look but I ran into this straight away. First I recorded a boot of the kernel:
./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-
version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icountshift=7,rr=record,rrfile=replay.bin
Quoted 6 lines
Quoted 6 lines
Then played back:
./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-
version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icountshift=7,rr=replay,rrfile=replay.bin -
Quoted 36 lines
Quoted 1 line
s -S
This looks ok, but...
Quoted 32 lines
And did the following on gdb:
(gdb) i
0x0000000040000004in ?? ()=> 0x40000004: mov x1, xzr 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr (gdb) 0x0000000040000008 in ?? () => 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 (gdb) 0x000000004000000c in ?? () => 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 (gdb) 0x0000000040000010 in ?? () => 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined (gdb) 0x0000000040000014 in ?? () => 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined 0x4000001c: .inst 0x00000000 ; undefined(gdb) p/x $x4
$1 =0x40080000
(gdb) reverse-stepi
warning: Remote failure reply: E14Quoted 4 lines
Quoted 2 lines
Surely this is the simple case and doesn't require any snapshots for block devices as there are none. Am I missing something?
Reverse debugging requires the snapshotting. QEMU can't revert the VM state without the
snapshots.
Quoted 1 line
You can try adding an empty qcow2 image to allow snapshotting there.
I suspect a recent patch has broken locking again:
Quoted 13 lines
Starting program:
/home/alex/lsrc/qemu/qemu.git/aarch64-softmmu/qemu-system-aarch64-machine virt,graphics=on,gic-version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=replay,rrfile=replay.bin,rrsnapshot=debug -drive file=rr.qcow2,if=none,snapshot,id=rr -s -S
[Thread debugging usinglibthread_dbenabled]
Using hostlibthread_dblibrary "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread0x7fffd8536700(LWP 32452)]
[New Thread0x7fffd5bb3700(LWP 32453)]
[New Thread0x7fffd4eab700(LWP 32454)]
**
ERROR:replay/replay-time.c:49:replay_read_clock: assertion failed: (replay_file&&replay_mutex_locked())
Quoted 1 line
Have you recorded it with the attached disk before replaying?
Yes. I assume the -drive doesn't actually have to be visible to the guest, it's just the mechanism rr needs for saving snapshots?
Quoted 3 lines
Are you using the latest version?
If the both answers are 'yes', then can you share the kernel? My i386 runs work normally.
I'll have a go with x86 first as aarch64 hasn't been proven yet.
Quoted 9 lines
Quoted 4 lines
Once I have the linux-user TCG tests merged I'm planning on focusing on the system emulation tests and we should be able to add some record/replay tests to defend the behaviour.
That will be great.
There are some Ciro's attempts on that: https://github.com/cirosantilli/qemu-test/blob/master/arm/rrPavel Dovgalyuk
--
Alex Bennée
Quoted 66 lines
From: Alex Bennée [mailto:[email protected]]
Quoted 63 lines · Pavel Dovgalyuk
Quoted 63 lines
From: Alex Bennée [mailto:[email protected]]
Quoted 11 lines · Pavel Dovgalyuk
Quoted 11 lines
From: Alex Bennée [mailto:[email protected]]
Quoted 1 line · Pavel Dovgalyuk
Ping?
I started having a look but I ran into this straight away. First I recorded a boot of the kernel:
./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-
version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icountshift=7,rr=record,rrfile=replay.bin
Quoted 6 lines
Quoted 6 lines
Then played back:
./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-
version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icountshift=7,rr=replay,rrfile=replay.bin -
Quoted 36 lines
Quoted 1 line
s -S
This looks ok, but...
Quoted 32 lines
And did the following on gdb:
(gdb) i
0x0000000040000004in ?? ()=> 0x40000004: mov x1, xzr 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr (gdb) 0x0000000040000008 in ?? () => 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 (gdb) 0x000000004000000c in ?? () => 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 (gdb) 0x0000000040000010 in ?? () => 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined (gdb) 0x0000000040000014 in ?? () => 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined 0x4000001c: .inst 0x00000000 ; undefined(gdb) p/x $x4
$1 =0x40080000
(gdb) reverse-stepi
warning: Remote failure reply: E14Quoted 4 lines
Quoted 2 lines
Surely this is the simple case and doesn't require any snapshots for block devices as there are none. Am I missing something?
Reverse debugging requires the snapshotting. QEMU can't revert the VM state without the
snapshots.
Quoted 11 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
You can try adding an empty qcow2 image to allow snapshotting there.
I suspect a recent patch has broken locking again:
Starting program:
/home/alex/lsrc/qemu/qemu.git/aarch64-softmmu/qemu-system-aarch64-machine
Quoted 1 line
Quoted 1 line
virt,graphics=on,gic-version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -
display
Quoted 3 lines
Quoted 3 lines
none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=replay,rrfile=replay.bin,rrsnapshot=debug -drive file=rr.qcow2,if=none,snapshot,id=rr -s -S
Just noticed. If you are using VM snapshots, then you should disable "snapshot" option of the drive. Like that: -drive file=rr.qcow2,if=none
BTW, similar command line for aarch64 worked for me. I just removed "-display=none" for convenience.
Pavel Dovgalyuk
Quoted 88 lines
From: Alex Bennée [mailto:[email protected]]
Quoted 61 lines · Pavel Dovgalyuk
Quoted 61 lines
From: Alex Bennée [mailto:[email protected]]
Quoted 11 lines · Pavel Dovgalyuk
Quoted 11 lines
From: Alex Bennée [mailto:[email protected]]
Quoted 1 line · Pavel Dovgalyuk
Ping?
I started having a look but I ran into this straight away. First I recorded a boot of the kernel:
./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-
version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icountshift=7,rr=record,rrfile=replay.bin
Quoted 6 lines
Quoted 6 lines
Then played back:
./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-
version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icountshift=7,rr=replay,rrfile=replay.bin -
Quoted 39 lines
Quoted 1 line
s -S
This looks ok, but...
Quoted 34 lines
And did the following on gdb:
(gdb) i
0x0000000040000004in ?? ()=> 0x40000004: mov x1, xzr 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr (gdb) 0x0000000040000008 in ?? () => 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 (gdb) 0x000000004000000c in ?? () => 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 (gdb) 0x0000000040000010 in ?? () => 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined (gdb) 0x0000000040000014 in ?? () => 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined 0x4000001c: .inst 0x00000000 ; undefined(gdb) p/x $x4
$1 =0x40080000
(gdb) reverse-stepi
warning: Remote failure reply: E14Surely this is the simple case and doesn't require any snapshots for block devices as there are none. Am I missing something?
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Reverse debugging requires the snapshotting. QEMU can't revert the VM state without the
snapshots.
Quoted 1 line
You can try adding an empty qcow2 image to allow snapshotting there.
I suspect a recent patch has broken locking again:
Starting program:
/home/alex/lsrc/qemu/qemu.git/aarch64-softmmu/qemu-system-aarch64-machine
Quoted 1 line
Quoted 1 line
virt,graphics=on,gic-version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -
display
Quoted 12 lines
Quoted 11 lines
none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=replay,rrfile=replay.bin,rrsnapshot=debug -drive file=rr.qcow2,if=none,snapshot,id=rr -s -S
[Thread debugging usinglibthread_dbenabled]
Using hostlibthread_dblibrary "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread0x7fffd8536700(LWP 32452)]
[New Thread0x7fffd5bb3700(LWP 32453)]
[New Thread0x7fffd4eab700(LWP 32454)]
**
ERROR:replay/replay-time.c:49:replay_read_clock: assertion failed: (replay_file&&replay_mutex_locked())Quoted 1 line
Have you recorded it with the attached disk before replaying?
Quoted 8 lines
Yes. I assume the -drive doesn't actually have to be visible to the guest, it's just the mechanism rr needs for saving snapshots?
Quoted 3 lines
Are you using the latest version?
If the both answers are 'yes', then can you share the kernel? My i386 runs work normally.I'll have a go with x86 first as aarch64 hasn't been proven yet.
Any news about that?
Pavel Dovgalyuk
Quoted 91 lines · Pavel Dovgalyuk
Quoted 91 lines
From: Alex Bennée [mailto:[email protected]]
Quoted 61 lines · Pavel Dovgalyuk
Quoted 61 lines
From: Alex Bennée [mailto:[email protected]]
Quoted 11 lines · Pavel Dovgalyuk
Quoted 11 lines
From: Alex Bennée [mailto:[email protected]]
Quoted 1 line · Pavel Dovgalyuk
Ping?
I started having a look but I ran into this straight away. First I recorded a boot of the kernel:
./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-
version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icountshift=7,rr=record,rrfile=replay.bin
Quoted 6 lines
Quoted 6 lines
Then played back:
./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-
version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icountshift=7,rr=replay,rrfile=replay.bin -
Quoted 39 lines
Quoted 1 line
s -S
This looks ok, but...
Quoted 34 lines
And did the following on gdb:
(gdb) i
0x0000000040000004in ?? ()=> 0x40000004: mov x1, xzr 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr (gdb) 0x0000000040000008 in ?? () => 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 (gdb) 0x000000004000000c in ?? () => 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 (gdb) 0x0000000040000010 in ?? () => 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined (gdb) 0x0000000040000014 in ?? () => 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined 0x4000001c: .inst 0x00000000 ; undefined(gdb) p/x $x4
$1 =0x40080000
(gdb) reverse-stepi
warning: Remote failure reply: E14Surely this is the simple case and doesn't require any snapshots for block devices as there are none. Am I missing something?
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Reverse debugging requires the snapshotting. QEMU can't revert the VM state without the
snapshots.
Quoted 1 line
You can try adding an empty qcow2 image to allow snapshotting there.
I suspect a recent patch has broken locking again:
Starting program:
/home/alex/lsrc/qemu/qemu.git/aarch64-softmmu/qemu-system-aarch64-machine
Quoted 1 line
Quoted 1 line
virt,graphics=on,gic-version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -
display
Quoted 12 lines
Quoted 11 lines
none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=replay,rrfile=replay.bin,rrsnapshot=debug -drive file=rr.qcow2,if=none,snapshot,id=rr -s -S
[Thread debugging usinglibthread_dbenabled]
Using hostlibthread_dblibrary "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread0x7fffd8536700(LWP 32452)]
[New Thread0x7fffd5bb3700(LWP 32453)]
[New Thread0x7fffd4eab700(LWP 32454)]
**
ERROR:replay/replay-time.c:49:replay_read_clock: assertion failed: (replay_file&&replay_mutex_locked())Quoted 1 line
Have you recorded it with the attached disk before replaying?
Yes. I assume the -drive doesn't actually have to be visible to the guest, it's just the mechanism rr needs for saving snapshots?
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Are you using the latest version?
If the both answers are 'yes', then can you share the kernel? My i386 runs work normally.I'll have a go with x86 first as aarch64 hasn't been proven yet.
Any news about that?
Sorry I got caught up with the pre-softfreeze rush. I'm hoping to get to it this week.
Quoted 3 lines
Pavel Dovgalyuk
--
Alex Bennée