Fixing record/replay and adding reverse debugging
v7 cover Fixing record/replay and adding reverse debugging
GDB remote protocol supports reverse debugging of the targets. It includes 'reverse step' and 'reverse continue' operations. The first one finds the previous step of the execution, and the second one is intended to stop at the last breakpoint that would happen when the program is executed normally.
Reverse debugging is possible in the replay mode, when at least one snapshot was created at the record or replay phase. QEMU can use these snapshots for travelling back in time with GDB.
Running the execution in replay mode allows using GDB reverse debugging commands:
- reverse-stepi (or rsi): Steps one instruction to the past. QEMU loads on of the prior snapshots and proceeds to the desired instruction forward. When that step is reaches, execution stops.
- reverse-continue (or rc): Runs execution "backwards". QEMU tries to find breakpoint or watchpoint by loaded prior snapshot and replaying the execution. Then QEMU loads snapshots again and replays to the latest breakpoint. When there are no breakpoints in the examined section of the execution, QEMU finds one more snapshot and tries again. After the first snapshot is processed, execution stops at this snapshot.
The set of patches include the following modifications:
- gdbstub update for reverse debugging support
- functions that automatically perform reverse step and reverse continue operations
- hmp/qmp commands for manipulating the replay process
- improvement of the snapshotting for saving the execution step in the snapshot parameters
- other record/replay fixes
The patches are available in the repository:
https://github.com/ispras/qemu/tree/rr-181010
v7 changes:
- rebased to the new master with upstreamed patches from the series
- several improvements in hmp/qmp commands handling (suggested by Markus Armbruster)
- fixed record/replay with '-rtc base' option enabled
- added document with virtual hardware requirements
v6 changes:
- rebased to the new version of master
- fixed build of linux-user configurations
- added new clock for slirp and vnc timers
v5 changes:
- multiple fixes of record/replay bugs appeared after QEMU core update
- changed reverse debugging to 'since 3.1'
v4 changes:
- changed 'since 2.13' to 'since 3.0' in json (as suggested by Eric Blake)
v3 changes:
- Fixed PS/2 bug with save/load vm, which caused failures of the replay.
- Rebased to the new code base.
- Minor fixes.
v2 changes:
- documented reverse debugging
- fixed start vmstate loading in record mode
- documented qcow2 changes (as suggested by Eric Blake)
- made icount SnapshotInfo field optional (as suggested by Eric Blake)
- renamed qmp commands (as suggested by Eric Blake)
- minor changes
---
Pavel Dovgalyuk (19):
block: implement bdrv_snapshot_goto for blkreplay
replay: disable default snapshot for record/replay
replay: update docs for record/replay with block devices
replay: don't drain/flush bdrv queue while RR is working
replay: finish record/replay before closing the disks
qcow2: introduce icount field for snapshots
migration: introduce icount field for snapshots
replay: provide and accessor for rr filename
replay: introduce info hmp/qmp command
replay: introduce breakpoint at the specified step
replay: implement replay-seek command to proceed to the desired step
replay: refine replay-time module
replay: flush rr queue before loading the vmstate
gdbstub: add reverse step support in replay mode
gdbstub: add reverse continue support in replay mode
replay: describe reverse debugging in docs/replay.txt
replay: add BH oneshot event for block layer
replay: init rtc after enabling the replay
replay: document development rules
accel/tcg/translator.c | 1 block/blkreplay.c | 8 + block/block-backend.c | 3 block/io.c | 22 +++ block/qapi.c | 17 ++ block/qcow2-snapshot.c | 9 + block/qcow2.h | 2 blockdev.c | 10 + cpus.c | 19 ++- docs/devel/replay.txt | 45 ++++++ docs/interop/qcow2.txt | 4 + docs/replay.txt | 45 ++++++ exec.c | 6 + gdbstub.c | 50 +++++++ hmp-commands-info.hx | 14 ++ hmp-commands.hx | 44 ++++++ hmp.h | 4 + include/block/snapshot.h | 1 include/sysemu/replay.h | 20 +++ migration/savevm.c | 15 +- qapi/block-core.json | 7 + qapi/block.json | 3 qapi/misc.json | 77 +++++++++++ replay/Makefile.objs | 3 replay/replay-debugging.c | 319 +++++++++++++++++++++++++++++++++++++++++++++ replay/replay-events.c | 16 ++ replay/replay-internal.h | 7 + replay/replay-time.c | 32 ++--- replay/replay.c | 26 ++++ stubs/Makefile.objs | 1 stubs/replay-user.c | 9 + stubs/replay.c | 10 + vl.c | 22 ++- 33 files changed, 818 insertions(+), 53 deletions(-) create mode 100644 docs/devel/replay.txt create mode 100644 replay/replay-debugging.c create mode 100644 stubs/replay-user.c
Signature · 1 line
Pavel Dovgalyuk
PATCH v7 1/19 block: implement bdrv_snapshot_goto for blkreplay
From: Pavel Dovgalyuk <[email protected]>
This patch enables making snapshots with blkreplay used in block devices.
This function is required to make bdrv_snapshot_goto without calling .bdrv_open which is not implemented.
--- block/blkreplay.c | 8 ++++++++ 1 file changed, 8 insertions(+)
block/blkreplay.c +8 -0
@@ -126,6 +126,12 @@ static int coroutine_fn blkreplay_co_flush(BlockDriverState *bs)return ret; } +static int blkreplay_snapshot_goto (BlockDriverState *bs, +const char *snapshot_id) +{ +return bdrv_snapshot_goto (bs->file->bs, snapshot_id,NULL ); +} +static BlockDriver bdrv_blkreplay = { .format_name ="blkreplay" , .instance_size =0 ,
@@ -140,6 +146,8 @@ static BlockDriver bdrv_blkreplay = { .bdrv_co_pwrite_zeroes = blkreplay_co_pwrite_zeroes, .bdrv_co_pdiscard = blkreplay_co_pdiscard, .bdrv_co_flush = blkreplay_co_flush, + + .bdrv_snapshot_goto = blkreplay_snapshot_goto, };static void bdrv_blkreplay_init (void )
PATCH v7 2/19 replay: disable default snapshot for record/replay
In reply to Pavel DovgalyukFrom: Pavel Dovgalyuk <[email protected]>
This patch disables setting '-snapshot' option on by default in record/replay mode. This is needed for creating vmstates in record and replay modes.
--- vl.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
vl.c +8 -2
@@ -3124,7 +3124,13 @@ int main(int argc, char **argv, char **envp)drive_add (IF_PFLASH ,-1 , optarg,PFLASH_OPTS );break ;case QEMU_OPTION_snapshot: - snapshot =1 ; + { +Error *blocker =NULL ; + snapshot =1 ; +error_setg (&blocker,QERR_REPLAY_NOT_SUPPORTED , +"-snapshot" ); +replay_add_blocker (blocker); + }break ;case QEMU_OPTION_numa: opts = qemu_opts_parse_noisily(qemu_find_opts("numa"),
@@ -4416,7 +4422,7 @@ int main(int argc, char **argv, char **envp)qapi_free_BlockdevOptions (bdo->bdo);g_free (bdo); } -if (snapshot || replay_mode !=REPLAY_MODE_NONE ) { +if (snapshot) {qemu_opts_foreach (qemu_find_opts ("drive" ), drive_enable_snapshot,NULL ,NULL ); }
PATCH v7 3/19 replay: update docs for record/replay with block devices
In reply to Pavel DovgalyukThis patch updates the description of the command lines for using record/replay with attached block devices.
--- docs/replay.txt | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
docs/replay.txt +9 -3
@@ -27,7 +27,7 @@ Usage of the record/replay: * First, record the execution with the following command line: qemu-system-i386 \ -icount shift=7,rr=record,rrfile=replay.bin \ - -drive file=disk.qcow2,if=none,id=img-direct \ + -drive file=disk.qcow2,if=none,snapshot,id=img-direct \ -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \ -device ide-hd,drive=img-blkreplay \ -netdev user,id=net1 -device rtl8139,netdev=net1 \
@@ -35,7 +35,7 @@ Usage of the record/replay: * After recording, you can replay it by using another command line: qemu-system-i386 \ -icount shift=7,rr=replay,rrfile=replay.bin \ - -drive file=disk.qcow2,if=none,id=img-direct \ + -drive file=disk.qcow2,if=none,snapshot,id=img-direct \ -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \ -device ide-hd,drive=img-blkreplay \ -netdev user,id=net1 -device rtl8139,netdev=net1 \
@@ -223,7 +223,7 @@ Block devices record/replay module intercepts calls of bdrv coroutine functions at the top of block drivers stack. To record and replay block operations the drive must be configured as following: - -drive file=disk.qcow2,if=none,id=img-direct + -drive file=disk.qcow2,if=none,snapshot,id=img-direct -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay -device ide-hd,drive=img-blkreplay
@@ -252,6 +252,12 @@ This snapshot is created at start of recording and restored at start of replaying. It also can be loaded while replaying to roll back the execution. +'snapshot' flag of the disk image must be removed to save the snapshots +in the overlay (or original image) instead of using the temporary overlay. + -drive file=disk.ovl,if=none,id=img-direct + -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay + -device ide-hd,drive=img-blkreplay + Use QEMU monitor to create additional snapshots. 'savevm <name>' command created the snapshot and 'loadvm <name>' restores it. To prevent corruption of the original disk image, use overlay files linked to the original images.
PATCH v7 4/19 replay: don't drain/flush bdrv queue while RR is working
In reply to Pavel DovgalyukIn record/replay mode bdrv queue is controlled by replay mechanism. It does not allow saving or loading the snapshots
when bdrv queue is not empty. Stopping the VM is not blocked by nonempty queue, but flushing the queue is still impossible there, because it may cause deadlocks in replay mode.
This patch disables bdrv_drain_all and bdrv_flush_all in record/replay mode.
--- block/io.c | 22 ++++++++++++++++++++++ cpus.c | 2 -- 2 files changed, 22 insertions(+), 2 deletions(-)
block/io.c +22 -0
@@ -32,6 +32,7 @@#include "qemu/cutils.h" #include "qapi/error.h" #include "qemu/error-report.h" +#include "sysemu/replay.h" #define NOT_DONE 0x7fffffff/* used while emulated sync operation in progress */
@@ -538,6 +539,13 @@ void bdrv_drain_all_begin(void)return ; } +/* bdrv queue is managed by record/replay, +waiting for finishing the I/O requests may +be infinite */ +if (replay_events_enabled ()) { +return ; + } +/* AIO_WAIT_WHILE() with a NULL context can only be called from the main * loop AioContext, so make sure we're in the main context. */ assert (qemu_get_current_aio_context () ==qemu_get_aio_context ());
@@ -566,6 +574,13 @@ void bdrv_drain_all_end(void) {BlockDriverState *bs =NULL ; +/* bdrv queue is managed by record/replay, +waiting for finishing the I/O requests may +be endless */ +if (replay_events_enabled ()) { +return ; + } +while ((bs =bdrv_next_all_states (bs))) {AioContext *aio_context =bdrv_get_aio_context (bs);
@@ -1997,6 +2012,13 @@ int bdrv_flush_all(void)BlockDriverState *bs =NULL ;int result =0 ; +/* bdrv queue is managed by record/replay, +creating new flush request for stopping +the VM may break the determinism */ +if (replay_events_enabled ()) { +return result; + } +for (bs =bdrv_first (&it); bs; bs =bdrv_next (&it)) {AioContext *aio_context =bdrv_get_aio_context (bs);int ret;
cpus.c +0 -2
@@ -1078,7 +1078,6 @@ static int do_vm_stop(RunState state, bool send_stop) }bdrv_drain_all (); -replay_disable_events (); ret =bdrv_flush_all ();return ret;
@@ -2135,7 +2134,6 @@ int vm_prepare_start(void)/* We are sending this now, but the CPUs will be resumed shortly later */ qapi_event_send_resume (); -replay_enable_events ();cpu_enable_ticks ();runstate_set (RUN_STATE_RUNNING );vm_state_notify (1 ,RUN_STATE_RUNNING );
Re: [Qemu-devel] [PATCH v7 04/19] replay: don't drain/flush bdrv queue while RR is working
Am 10.10.2018 um 15:33 hat Pavel Dovgalyuk geschrieben:
Quoted 9 lines
In record/replay mode bdrv queue is controlled by replay mechanism. It does not allow saving or loading the snapshots
when bdrv queue is not empty. Stopping the VM is not blocked by nonempty queue, but flushing the queue is still impossible there, because it may cause deadlocks in replay mode.
This patch disablesbdrv_drain_allandbdrv_flush_allin record/replay mode.Signed-off-by Pavel Dovgalyuk <[email protected]>
Disabling bdrv_flush_all() shouldn't make a big difference in replay mode, so I agree with that.
But does the bdrv_drain_all() change mean that bdrv_drain_all_begin() can return while there are still requests in flight? This sounds like something that could break some existing code, because the meaning of the function is that if it returns success, no requests are in flight any more.
What would move the request forward in the replay case once it has been created? Does this also involve some user interaction? Or may we process the next event in a drain callback of blkreplay or something?
Kevin
Re: [Qemu-devel] [PATCH v7 04/19] replay: don't drain/flush bdrv queue while RR is working
Quoted 21 lines
From: Kevin Wolf [mailto:[email protected]]
Am 10.10.2018 um 15:33 hat Pavel Dovgalyuk geschrieben:Quoted 9 lines
In record/replay mode bdrv queue is controlled by replay mechanism. It does not allow saving or loading the snapshots
when bdrv queue is not empty. Stopping the VM is not blocked by nonempty queue, but flushing the queue is still impossible there, because it may cause deadlocks in replay mode.
This patch disablesbdrv_drain_allandbdrv_flush_allin record/replay mode.Signed-off-by Pavel Dovgalyuk <[email protected]>Disabling
bdrv_flush_all()shouldn't make a big difference in replay mode, so I agree with that.But does the
bdrv_drain_all()change mean thatbdrv_drain_all_begin()can return while there are still requests in flight? This sounds like something that could break some existing code, because the meaning of the function is that if it returns success, no requests are in flight any more.
Quoted 3 lines
What would move the request forward in the replay case once it has been created? Does this also involve some user interaction? Or may we process the next event in a drain callback of blkreplay or something?
You are right - there are some operation that can't be performed at any time during the replay, because of unfinished block requests.
One of these is creating the VM snapshot. I've added a check in another patch, which prevents snapshotting while the event queue (which holds the block requests) is not empty.
There could also be other things, that will be fixed when we try to use it.
Moving replay forward until the queue is empty is not a good idea, because step-by-step debugging should be available and work without skipping the instructions.
Pavel Dovgalyuk
Re: [Qemu-devel] [PATCH v7 04/19] replay: don't drain/flush bdrv queue while RR is working
Am 30.11.2018 um 08:55 hat Pavel Dovgalyuk geschrieben:
Quoted 28 lines
Quoted 21 lines
From: Kevin Wolf [mailto:[email protected]]
Am 10.10.2018 um 15:33 hat Pavel Dovgalyuk geschrieben:Quoted 9 lines
In record/replay mode bdrv queue is controlled by replay mechanism. It does not allow saving or loading the snapshots
when bdrv queue is not empty. Stopping the VM is not blocked by nonempty queue, but flushing the queue is still impossible there, because it may cause deadlocks in replay mode.
This patch disablesbdrv_drain_allandbdrv_flush_allin record/replay mode.Signed-off-by Pavel Dovgalyuk <[email protected]>Disabling
bdrv_flush_all()shouldn't make a big difference in replay mode, so I agree with that.But does the
bdrv_drain_all()change mean thatbdrv_drain_all_begin()can return while there are still requests in flight? This sounds like something that could break some existing code, because the meaning of the function is that if it returns success, no requests are in flight any more.Quoted 3 lines
What would move the request forward in the replay case once it has been created? Does this also involve some user interaction? Or may we process the next event in a drain callback of blkreplay or something?
You are right - there are some operation that can't be performed at any time during the replay, because of unfinished block requests.
Quoted 8 lines
One of these is creating the VM snapshot. I've added a check in another patch, which prevents snapshotting while the event queue (which holds the block requests) is not empty.
There could also be other things, that will be fixed when we try to use it.
Moving replay forward until the queue is empty is not a good idea, because step-by-step debugging should be available and work without skipping the instructions.
So if the idea is that in replay mode, bdrv_drain_all_begin() is only called when there are no requests in flight anyway (because callers catch this situation earler), can we make this an assertion that the block device is already quiesced?
Skipping drain without knowing that there are no requests in flight feels simply wrong, and I'm almost sure it will result in bugs. On the other hand, if we know that no requests are in flight, there's no real point in skipping.
So I think I still don't understand the situation where skipping a drain is the correct solution.
Kevin
Re: [Qemu-devel] [PATCH v7 04/19] replay: don't drain/flush bdrv queue while RR is working
Quoted 36 lines
From: Kevin Wolf [mailto:[email protected]]
Am 30.11.2018 um 08:55 hat Pavel Dovgalyuk geschrieben:Quoted 34 lines
Quoted 21 lines
From: Kevin Wolf [mailto:[email protected]]
Am 10.10.2018 um 15:33 hat Pavel Dovgalyuk geschrieben:Quoted 9 lines
In record/replay mode bdrv queue is controlled by replay mechanism. It does not allow saving or loading the snapshots
when bdrv queue is not empty. Stopping the VM is not blocked by nonempty queue, but flushing the queue is still impossible there, because it may cause deadlocks in replay mode.
This patch disablesbdrv_drain_allandbdrv_flush_allin record/replay mode.Signed-off-by Pavel Dovgalyuk <[email protected]>Disabling
bdrv_flush_all()shouldn't make a big difference in replay mode, so I agree with that.But does the
bdrv_drain_all()change mean thatbdrv_drain_all_begin()can return while there are still requests in flight? This sounds like something that could break some existing code, because the meaning of the function is that if it returns success, no requests are in flight any more.Quoted 3 lines
What would move the request forward in the replay case once it has been created? Does this also involve some user interaction? Or may we process the next event in a drain callback of blkreplay or something?
You are right - there are some operation that can't be performed at any time during the replay, because of unfinished block requests.
One of these is creating the VM snapshot. I've added a check in another patch, which prevents snapshotting while the event queue (which holds the block requests) is not empty.
There could also be other things, that will be fixed when we try to use it.
Quoted 7 lines
Quoted 2 lines
Moving replay forward until the queue is empty is not a good idea, because step-by-step debugging should be available and work without skipping the instructions.
So if the idea is that in replay mode,
bdrv_drain_all_begin()is only called when there are no requests in flight anyway (because callers catch this situation earler), can we make this an assertion that the block device is already quiesced?
Maybe it's behavior changed in the latest version?
We observed that bdrv_drain_all hangs, when there is a block request in the queue. Therefore we decided that stopping with non-empty queue is better than skipping some steps.
Quoted 4 lines
Skipping drain without knowing that there are no requests in flight feels simply wrong, and I'm almost sure it will result in bugs. On the other hand, if we know that no requests are in flight, there's no real point in skipping.
What bugs can happen?
Pavel Dovgalyuk
PATCH v7 5/19 replay: finish record/replay before closing the disks
In reply to Pavel DovgalyukAfter recent updates block devices cannot be closed on qemu exit. This happens due to the block request polling when replay is not finished. Therefore now we stop execution recording before closing the block devices.
--- replay/replay.c | 2 ++ vl.c | 1 + 2 files changed, 3 insertions(+)
replay/replay.c +2 -0
@@ -378,6 +378,8 @@ void replay_finish(void)g_free (replay_snapshot); replay_snapshot =NULL ; + replay_mode =REPLAY_MODE_NONE ; +replay_finish_events (); }
vl.c +1 -0
@@ -4627,6 +4627,7 @@ int main(int argc, char **argv, char **envp)/* No more vcpu or device emulation activity beyond this point */ vm_shutdown (); +replay_finish ();job_cancel_sync_all ();bdrv_close_all ();
PATCH v7 6/19 qcow2: introduce icount field for snapshots
In reply to Pavel DovgalyukThis patch introduces the icount field for saving within the snapshot. It is required for navigation between the snapshots in record/replay mode.
--
v2:
- documented format changes in docs/interop/qcow2.txt (suggested by Eric Blake)
--- block/qcow2-snapshot.c | 7 +++++++ block/qcow2.h | 2 ++ docs/interop/qcow2.txt | 4 ++++ 3 files changed, 13 insertions(+)
block/qcow2-snapshot.c +7 -0
@@ -103,6 +103,12 @@ int qcow2_read_snapshots(BlockDriverState *bs) sn->disk_size = bs->total_sectors *BDRV_SECTOR_SIZE ; } +if (extra_data_size >=24 ) { + sn->icount =be64_to_cpu (extra.icount); + }else { + sn->icount =-1ULL ; + } +/* Read snapshot ID */ sn->id_str =g_malloc (id_str_size +1 ); ret =bdrv_pread (bs->file, offset, sn->id_str, id_str_size);
@@ -209,6 +215,7 @@ static int qcow2_write_snapshots(BlockDriverState *bs)memset (&extra,0 ,sizeof (extra)); extra.vm_state_size_large =cpu_to_be64 (sn->vm_state_size); extra.disk_size =cpu_to_be64 (sn->disk_size); + extra.icount =cpu_to_be64 (sn->icount); id_str_size =strlen (sn->id_str); name_size =strlen (sn->name);
block/qcow2.h +2 -0
@@ -153,6 +153,7 @@ typedef struct QEMU_PACKED QCowSnapshotHeader { typedef struct QEMU_PACKED QCowSnapshotExtraData {uint64_t vm_state_size_large;uint64_t disk_size; +uint64_t icount; } QCowSnapshotExtraData;
@@ -166,6 +167,7 @@ typedef struct QCowSnapshot {uint32_t date_sec;uint32_t date_nsec;uint64_t vm_clock_nsec; +uint64_t icount; } QCowSnapshot;struct Qcow2Cache ;
docs/interop/qcow2.txt +4 -0
@@ -506,6 +506,10 @@ Snapshot table entry: Byte 48 - 55: Virtual disk size of the snapshot in bytes + Byte 56 - 63: icount value which corresponds to + the record/replay step when the snapshot + was taken + Version 3 images must include extra data at least up to byte 55.
Re: [Qemu-devel] [PATCH v7 06/19] qcow2: introduce icount field for snapshots
Am 10.10.2018 um 15:34 hat Pavel Dovgalyuk geschrieben:
Quoted 10 lines
This patch introduces the icount field for saving within the snapshot. It is required for navigation between the snapshots in record/replay mode.
Signed-off-by Pavel Dovgalyuk <[email protected]>--
v2:
- documented format changes indocs/interop/qcow2.txt(suggested by Eric Blake)
PATCH v7 7/19 migration: introduce icount field for snapshots
In reply to Pavel DovgalyukSaving icount as a parameters of the snapshot allows navigation between them in the execution replay scenario.
This information can be used for finding a specific snapshot for rewinding the recorded execution to the specific moment of the time. E.g., 'reverse step' action needs to load the nearest snapshot which is prior to the current moment of time .
--
v2:
- made icount in SnapshotInfo optional (suggested by Eric Blake)
v7:
- added more comments for icount member (suggested by Markus Armbruster)
--- block/qapi.c | 17 +++++++++++++---- block/qcow2-snapshot.c | 2 ++ blockdev.c | 10 ++++++++++ include/block/snapshot.h | 1 + migration/savevm.c | 5 +++++ qapi/block-core.json | 7 ++++++- qapi/block.json | 3 ++- 7 files changed, 39 insertions(+), 6 deletions(-)
block/qapi.c +13 -4
@@ -210,6 +210,7 @@ int bdrv_query_snapshot_info_list(BlockDriverState *bs, info->date_nsec = sn_tab[i].date_nsec; info->vm_clock_sec = sn_tab[i].vm_clock_nsec /1000000000 ; info->vm_clock_nsec = sn_tab[i].vm_clock_nsec %1000000000 ; + info->icount = sn_tab[i].icount; info_list =g_new0 (SnapshotInfoList,1 ); info_list->value = info;
@@ -663,14 +664,15 @@ void bdrv_snapshot_dump(fprintf_function func_fprintf, void *f, QEMUSnapshotInfo *sn) {char buf1[128 ], date_buf[128 ], clock_buf[128 ]; +char icount_buf[128 ] = {0 };struct tm tm;time_t ti;int64_t secs;if (!sn) {func_fprintf (f, -"%-10s%-20s%7s%20s%15s" , -"ID" ,"TAG" ,"VM SIZE" ,"DATE" ,"VM CLOCK" ); +"%-10s%-18s%7s%20s%13s%11s" , +"ID" ,"TAG" ,"VM SIZE" ,"DATE" ,"VM CLOCK" ,"ICOUNT" ); }else { ti = sn->date_sec;localtime_r (&ti, &tm);
@@ -683,13 +685,18 @@ void bdrv_snapshot_dump(fprintf_function func_fprintf, void *f, (int )((secs /60 ) %60 ), (int )(secs %60 ), (int)((sn->vm_clock_nsec / 1000000) % 1000)); +if (sn->icount !=-1ULL ) { +snprintf (icount_buf,sizeof (icount_buf), +"%" PRId64, sn->icount); + }func_fprintf (f, -"%-10s%-20s%7s%20s%15s" , +"%-10s%-18s%7s%20s%13s%11s" , sn->id_str, sn->name,get_human_readable_size (buf1,sizeof (buf1), sn->vm_state_size), date_buf, - clock_buf); + clock_buf, + icount_buf); } }
@@ -857,6 +864,8 @@ void bdrv_image_info_dump(fprintf_function func_fprintf, void *f, .date_nsec = elem->value->date_nsec, .vm_clock_nsec = elem->value->vm_clock_sec *1000000000ULL + elem->value->vm_clock_nsec, + .icount = elem->value->has_icount ? + elem->value->icount : -1ULL, };pstrcpy (sn.id_str,sizeof (sn.id_str), elem->value->id);
block/qcow2-snapshot.c +2 -0
@@ -379,6 +379,7 @@ int qcow2_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info) sn->date_sec = sn_info->date_sec; sn->date_nsec = sn_info->date_nsec; sn->vm_clock_nsec = sn_info->vm_clock_nsec; + sn->icount = sn_info->icount;/* Allocate the L1 table of the snapshot and copy the current one there. */ l1_table_offset =qcow2_alloc_clusters (bs, s->l1_size *sizeof (uint64_t ));
@@ -698,6 +699,7 @@ int qcow2_snapshot_list(BlockDriverState *bs, QEMUSnapshotInfo **psn_tab) sn_info->date_sec = sn->date_sec; sn_info->date_nsec = sn->date_nsec; sn_info->vm_clock_nsec = sn->vm_clock_nsec; + sn_info->icount = sn->icount; } *psn_tab = sn_tab;return s->nb_snapshots;
blockdev.c +10 -0
@@ -57,6 +57,7 @@#include "block/trace.h" #include "sysemu/arch_init.h" #include "sysemu/qtest.h" +#include "sysemu/replay.h" #include "qemu/cutils.h" #include "qemu/help_option.h" #include "qemu/throttle-options.h"
@@ -1239,6 +1240,10 @@ SnapshotInfo *qmp_blockdev_snapshot_delete_internal_sync(const char *device, info->vm_state_size = sn.vm_state_size; info->vm_clock_nsec = sn.vm_clock_nsec %1000000000 ; info->vm_clock_sec = sn.vm_clock_nsec /1000000000 ; +if (sn.icount !=-1ULL ) { + info->icount = sn.icount; + info->has_icount = true; + }return info;
@@ -1447,6 +1452,11 @@ static void internal_snapshot_prepare(BlkActionState *common, sn->date_sec = tv.tv_sec; sn->date_nsec = tv.tv_usec *1000 ; sn->vm_clock_nsec =qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ); +if (replay_mode !=REPLAY_MODE_NONE ) { + sn->icount =replay_get_current_step (); + }else { + sn->icount =-1ULL ; + } ret1 =bdrv_snapshot_create (bs, sn);if (ret1 <0 ) {
include/block/snapshot.h +1 -0
@@ -42,6 +42,7 @@ typedef struct QEMUSnapshotInfo {uint32_t date_sec;/* UTC date of the snapshot */ uint32_t date_nsec;uint64_t vm_clock_nsec;/* VM clock relative to boot */ +uint64_t icount;/* record/replay step */ } QEMUSnapshotInfo; int bdrv_snapshot_find(BlockDriverState *bs, QEMUSnapshotInfo *sn_info,
migration/savevm.c +5 -0
@@ -2462,6 +2462,11 @@ int save_snapshot(const char *name, Error **errp) sn->date_sec = tv.tv_sec; sn->date_nsec = tv.tv_usec *1000 ; sn->vm_clock_nsec =qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ); +if (replay_mode !=REPLAY_MODE_NONE ) { + sn->icount =replay_get_current_step (); + }else { + sn->icount =-1ULL ; + }if (name) { ret =bdrv_snapshot_find (bs, old_sn, name);
qapi/block-core.json +6 -1
@@ -26,13 +26,18 @@ # # @vm-clock-nsec: fractional part in nano seconds to be used with vm-clock-sec # +# @icount: Current instruction count. Appears when execution record/replay +# is enabled. Used for "time-traveling" to match the moment +# in the recorded execution with the snapshots (since 3.1) +# # Since: 1.3 # ## { 'struct': 'SnapshotInfo', 'data': { 'id': 'str', 'name': 'str', 'vm-state-size': 'int', 'date-sec': 'int', 'date-nsec': 'int', - 'vm-clock-sec': 'int', 'vm-clock-nsec': 'int' } } + 'vm-clock-sec': 'int', 'vm-clock-nsec': 'int', + '*icount': 'int' } } ## # @ImageInfoSpecificQCow2EncryptionBase:
qapi/block.json +2 -1
@@ -176,7 +176,8 @@ # "date-sec": 1000012, # "date-nsec": 10, # "vm-clock-sec": 100, -# "vm-clock-nsec": 20 +# "vm-clock-nsec": 20, +# "icount": 220414 # } # } #
Re: [Qemu-devel] [PATCH v7 07/19] migration: introduce icount field for snapshots
Am 10.10.2018 um 15:34 hat Pavel Dovgalyuk geschrieben:
Quoted 14 lines
Saving icount as a parameters of the snapshot allows navigation between them in the execution replay scenario.
This information can be used for finding a specific snapshot for rewinding the recorded execution to the specific moment of the time. E.g., 'reverse step' action needs to load the nearest snapshot which is prior to the current moment of time .Signed-off-by Pavel Dovgalyuk <[email protected]>block/qapi.c +0
-0
block/qapi.c int bdrv_query_snapshot_info_list(BlockDriverState *bs, · line 213
Quoted 4 lines
block/qapi.c +1
-0@@ -210,6 +210,7 @@ int bdrv_query_snapshot_info_list(BlockDriverState *bs, info->date_nsec = sn_tab[i].date_nsec; info->vm_clock_sec = sn_tab[i].vm_clock_nsec /1000000000 ; info->vm_clock_nsec = sn_tab[i].vm_clock_nsec %1000000000 ; + info->icount = sn_tab[i].icount;
Now that it's optional, info->has_icount needs to be set, too.
Quoted 8 lines
block/qapi.c +0
-0info_list =g_new0 (SnapshotInfoList,1 ); info_list->value = info;qapi/block-core.json +0
-0
qapi/block-core.json line 31
Quoted 6 lines
qapi/block-core.json +3
-0@@ -26,13 +26,18 @@ # # @vm-clock-nsec: fractional part in nano seconds to be used with vm-clock-sec # +# @icount: Current instruction count. Appears when execution record/replay +# is enabled. Used for "time-traveling" to match the moment +# in the recorded execution with the snapshots (since 3.1)
"(since 4.0)" now.
Kevin
PATCH v7 8/19 replay: provide and accessor for rr filename
In reply to Pavel DovgalyukThis patch adds an accessor function for the name of the record/replay log file. Adding an accessor instead of making variable global, prevents accidental modification of this variable by other modules.
--- include/sysemu/replay.h | 2 ++ replay/replay.c | 5 +++++ 2 files changed, 7 insertions(+)
include/sysemu/replay.h +2 -0
@@ -71,6 +71,8 @@ void replay_start(void);void replay_finish (void );/*! Adds replay blocker with the specified error description */ void replay_add_blocker (Error *reason); +/* Returns name of the replay log file */ +const char *replay_get_filename (void );/* Processing the instructions */
replay/replay.c +5 -0
@@ -387,3 +387,8 @@ void replay_add_blocker(Error *reason) { replay_blockers =g_slist_prepend (replay_blockers, reason); } + +const char *replay_get_filename (void ) +{ +return replay_filename; +}
PATCH v7 9/19 replay: introduce info hmp/qmp command
In reply to Pavel DovgalyukThis patch introduces 'info replay' monitor command and corresponding qmp request.
These commands request the current record/replay mode, replay log file name, and the execution step (number or recorded/replayed instructions). User may use step number for replay_seek/replay_break commands and for controlling the execution of replay.
--
v2:
- renamed info_replay qmp into query-replay (suggested by Eric Blake)
v7:
- added empty line (suggested by Markus Armbruster)
--- hmp-commands-info.hx | 14 ++++++++++++++ hmp.h | 1 + qapi/misc.json | 35 +++++++++++++++++++++++++++++++++++ replay/Makefile.objs | 3 ++- replay/replay-debugging.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 94 insertions(+), 1 deletion(-) create mode 100644 replay/replay-debugging.c
hmp-commands-info.hx +14 -0
@@ -918,6 +918,20 @@ STEXI Show SEV information. ETEXI + { + .name = "replay", + .args_type = "", + .params = "", + .help = "show parameters of the record/replay", + .cmd = hmp_info_replay, + }, + +STEXI +@item info replay +@findex info replay +Display the current record/replay mode and the currently executing step. +ETEXI + STEXI @end table ETEXI
hmp.h +1 -0
@@ -148,5 +148,6 @@ void hmp_hotpluggable_cpus(Monitor *mon, const QDict *qdict);void hmp_info_vm_generation_id (Monitor *mon,const QDict *qdict);void hmp_info_memory_size_summary (Monitor *mon,const QDict *qdict);void hmp_info_sev (Monitor *mon,const QDict *qdict); +void hmp_info_replay (Monitor *mon,const QDict *qdict);#endif
qapi/misc.json +35 -0
@@ -3105,6 +3105,41 @@ 'data': [ 'none', 'record', 'play' ] } ## +# @ReplayInfo: +# +# Status of the record/replay mode. +# +# @mode: current mode. +# +# @filename: name of the record/replay log file. +# +# @step: current step number. +# +# Since: 3.1 +# +## +{ 'struct': 'ReplayInfo', + 'data': { 'mode': 'ReplayMode', '*filename': 'str', 'step': 'int' } } + +## +# @query-replay: +# +# Retrieves the status of the execution record/replay. +# +# Returns: structure with the properties of the record/replay. +# +# Since: 3.1 +# +# Example: +# +# -> { "execute": "query-replay" } +# <- { "return": { "mode": "play", "filename": "log.rr", "step": 220414 } } +# +## +{ 'command': 'query-replay', + 'returns': 'ReplayInfo' } + +## # @xen-load-devices-state: # # Load the state of all devices from file. The RAM and the block devices
replay/Makefile.objs +2 -1
@@ -6,4 +6,5 @@ common-obj-y += replay-input.o common-obj-y += replay-char.o common-obj-y += replay-snapshot.o common-obj-y += replay-net.o -common-obj-y += replay-audio.o +common-obj-y += replay-audio.o +common-obj-y += replay-debugging.o
replay/replay-debugging.c new file +42 -0
42 lines not on this page · raw message
PATCH v7 10/19 replay: introduce breakpoint at the specified step
In reply to Pavel DovgalyukThis patch introduces replay_break, replay_delete_break qmp and hmp commands.
These commands allow stopping at the specified instruction. It may be useful for debugging when there are some known events that should be investigated.replay_break command has one argument - number of instructions executed since the start of the replay. replay_delete_break removes previously set breakpoint.
--
v2:
- renamed replay_break qmp command into replay-break (suggested by Eric Blake)
v7:
- introduces replay_delete_break command
--- hmp-commands.hx | 29 ++++++++++++++++ hmp.h | 2 + qapi/misc.json | 31 +++++++++++++++++ replay/replay-debugging.c | 84 +++++++++++++++++++++++++++++++++++++++++++++ replay/replay-internal.h | 4 ++ replay/replay.c | 17 +++++++++ 6 files changed, 167 insertions(+)
hmp-commands.hx +29 -0
@@ -1888,6 +1888,35 @@ Set QOM property @var{property} of object at location @var{path} to value @var{v ETEXI { + .name = "replay_break", + .args_type = "step:i", + .params = "step", + .help = "sets breakpoint on the specified step of the replay", + .cmd = hmp_replay_break, + }, + +STEXI +@item replay_break @var{step} +@findex replay_break +Set breakpoint on the specified step of the replay. +Execution stops when the specified step is reached. +ETEXI + + { + .name = "replay_delete_break", + .args_type = "", + .params = "", + .help = "removes replay breakpoint", + .cmd = hmp_replay_delete_break, + }, + +STEXI +@item replay_delete_break +@findex replay_delete_break +Removes replay breakpoint which was previously set with replay_break. +ETEXI + + { .name = "info", .args_type = "item:s?", .params = "[subcommand]",
hmp.h +2 -0
@@ -149,5 +149,7 @@ void hmp_info_vm_generation_id(Monitor *mon, const QDict *qdict);void hmp_info_memory_size_summary (Monitor *mon,const QDict *qdict);void hmp_info_sev (Monitor *mon,const QDict *qdict);void hmp_info_replay (Monitor *mon,const QDict *qdict); +void hmp_replay_break (Monitor *mon,const QDict *qdict); +void hmp_replay_delete_break (Monitor *mon,const QDict *qdict);#endif
qapi/misc.json +31 -0
@@ -3140,6 +3140,37 @@ 'returns': 'ReplayInfo' } ## +# @replay-break: +# +# Set breakpoint on the specified step of the replay. +# Execution stops when the specified step is reached. +# +# @step: execution step to stop at +# +# Since: 3.1 +# +# Example: +# +# -> { "execute": "replay-break", "data": { "step": 220414 } } +# +## +{ 'command': 'replay-break', 'data': { 'step': 'int' } } + +## +# @replay-delete-break: +# +# Removes replay breakpoint. +# +# Since: 3.1 +# +# Example: +# +# -> { "execute": "replay-delete-break" } +# +## +{ 'command': 'replay-delete-break' } + +## # @xen-load-devices-state: # # Load the state of all devices from file. The RAM and the block devices
replay/replay-debugging.c +84 -0
93 lines not on this page · raw message
replay/replay-internal.h +4 -0
@@ -91,6 +91,10 @@ extern ReplayState replay_state;/* File for replay writing */ extern FILE *replay_file; +/* Step of the replay breakpoint */ +extern uint64_t replay_break_step; +/* Timer for the replay breakpoint callback */ +extern QEMUTimer *replay_break_timer;void replay_put_byte (uint8_t byte);void replay_put_event (uint8_t event);
replay/replay.c +17 -0
@@ -34,6 +34,10 @@ static char *replay_filename;ReplayState replay_state;static GSList *replay_blockers; +/* Replay breakpoints */ +uint64_t replay_break_step =-1ULL ; +QEMUTimer *replay_break_timer; +bool replay_next_event_is (int event) {bool res = false;
@@ -73,6 +77,13 @@ int replay_get_instructions(void)replay_mutex_lock ();if (replay_next_event_is (EVENT_INSTRUCTION )) { res = replay_state.instructions_count; +if (replay_break_step !=-1LL ) { +uint64_t current =replay_get_current_step (); +assert (replay_break_step >= current); +if (current + res > replay_break_step) { + res = replay_break_step - current; + } + } }replay_mutex_unlock ();return res;
@@ -99,6 +110,12 @@ void replay_account_executed_instructions(void)will be read from the log. */ qemu_notify_event (); } +/* Execution reached the break step */ +if (replay_break_step == replay_state.current_step) { +/* Cannot make callback directly from the vCPU thread */ +timer_mod_ns (replay_break_timer, +qemu_clock_get_ns (QEMU_CLOCK_REALTIME )); + } } } }
PATCH v7 11/19 replay: implement replay-seek command to proceed to the desired step
In reply to Pavel DovgalyukThis patch adds hmp/qmp commands replay_seek/replay-seek that proceed the execution to the specified step.
The commands automatically loads nearest snapshot and replay the execution to find the desired step.
--
v2:
- renamed replay_seek qmp command into replay-seek (suggested by Eric Blake)
v7:
- small fixes related to Markus Armbruster's review
--- hmp-commands.hx | 15 +++++++ hmp.h | 1 qapi/misc.json | 11 +++++ replay/replay-debugging.c | 91 +++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 118 insertions(+)
hmp-commands.hx +15 -0
@@ -1917,6 +1917,21 @@ Removes replay breakpoint which was previously set with replay_break. ETEXI { + .name = "replay_seek", + .args_type = "step:i", + .params = "step", + .help = "rewinds replay to the specified step", + .cmd = hmp_replay_seek, + }, + +STEXI +@item replay_seek @var{step} +@findex replay_seek +Automatically proceeds to the specified step, when replaying +the execution. +ETEXI + + { .name = "info", .args_type = "item:s?", .params = "[subcommand]",
hmp.h +1 -0
@@ -151,5 +151,6 @@ void hmp_info_sev(Monitor *mon, const QDict *qdict);void hmp_info_replay (Monitor *mon,const QDict *qdict);void hmp_replay_break (Monitor *mon,const QDict *qdict);void hmp_replay_delete_break (Monitor *mon,const QDict *qdict); +void hmp_replay_seek (Monitor *mon,const QDict *qdict);#endif
qapi/misc.json +11 -0
@@ -3171,6 +3171,17 @@ { 'command': 'replay-delete-break' } ## +# @replay-seek: +# +# Automatically proceeds to the specified step when replaying +# the execution. +# +# @step: destination execution step +# -> { "execute": "replay-seek", "data": { "step": 220414 } } +## +{ 'command': 'replay-seek', 'data': { 'step': 'int' } } + +## # @xen-load-devices-state: # # Load the state of all devices from file. The RAM and the block devices
replay/replay-debugging.c +91 -0
100 lines not on this page · raw message
PATCH v7 12/19 replay: refine replay-time module
In reply to Pavel DovgalyukThis patch removes refactoring artifacts from the replay/replay-time.c
--- replay/replay-time.c | 32 +++++++++++++------------------- 1 file changed, 13 insertions(+), 19 deletions(-)
replay/replay-time.c +13 -19
@@ -17,16 +17,15 @@int64_t replay_save_clock (ReplayClockKind kind,int64_t clock,int64_t raw_icount) { -if (replay_file) { -g_assert (replay_mutex_locked ()); +g_assert (replay_file); +g_assert (replay_mutex_locked ()); -/* Due to the caller's locking requirements we get the icount from it instead -* of using replay_save_instructions(). -*/ -replay_advance_current_step (raw_icount); -replay_put_event (EVENT_CLOCK + kind); -replay_put_qword (clock); - } +/* Due to the caller's locking requirements we get the icount from it instead +* of using replay_save_instructions(). +*/ +replay_advance_current_step (raw_icount); +replay_put_event (EVENT_CLOCK + kind); +replay_put_qword (clock);return clock; }
@@ -48,20 +47,15 @@ void replay_read_next_clock(ReplayClockKind kind)/*! Reads next clock event from the input. */ int64_t replay_read_clock (ReplayClockKind kind) { +int64_t ret;g_assert (replay_file &&replay_mutex_locked ());replay_account_executed_instructions (); -if (replay_file) { -int64_t ret; -if (replay_next_event_is (EVENT_CLOCK + kind)) { -replay_read_next_clock (kind); - } - ret = replay_state.cached_clock[kind]; - -return ret; +if (replay_next_event_is (EVENT_CLOCK + kind)) { +replay_read_next_clock (kind); } + ret = replay_state.cached_clock[kind]; -error_report ("REPLAY INTERNAL ERROR %d" , __LINE__); -exit (1 ); +return ret; }
PATCH v7 13/19 replay: flush rr queue before loading the vmstate
In reply to Pavel DovgalyukNon-empty record/replay queue prevents saving and loading the VM state, because it includes pending bottom halves and block coroutines. But when the new VM state is loaded, we don't have to preserve the consistency of the current state anymore. Therefore this patch just flushes the queue allowing the coroutines to finish.
--- include/sysemu/replay.h | 2 ++ migration/savevm.c | 10 ++++------ replay/replay-internal.h | 2 -- 3 files changed, 6 insertions(+), 8 deletions(-)
include/sysemu/replay.h +2 -0
@@ -140,6 +140,8 @@ void replay_disable_events(void);void replay_enable_events (void );/*! Returns true when saving events is enabled */ bool replay_events_enabled (void ); +/* Flushes events queue */ +void replay_flush_events (void );/*! Adds bottom half event to the queue */ void replay_bh_schedule_event (QEMUBH *bh);/*! Adds input event to the queue */
migration/savevm.c +4 -6
@@ -2615,12 +2615,6 @@ int load_snapshot(const char *name, Error **errp)AioContext *aio_context;MigrationIncomingState *mis =migration_incoming_get_current (); -if (!replay_can_snapshot ()) { -error_report ("Record/replay does not allow loading snapshot " -"right now. Try once more later." ); -return -EINVAL ; - } -if (!bdrv_all_can_snapshot (&bs)) { error_setg(errp, "Device '%s' is writable but does not support snapshots",
@@ -2654,6 +2648,10 @@ int load_snapshot(const char *name, Error **errp)return -EINVAL ; } +/* Flush the record/replay queue. Now the VM state is going +to change. Therefore we don't need to preserve its consistency */ +replay_flush_events (); +/* Flush all IO requests so they don't interfere with the new state. */ bdrv_drain_all_begin ();
replay/replay-internal.h +0 -2
@@ -146,8 +146,6 @@ void replay_read_next_clock(unsigned int kind);void replay_init_events (void );/*! Clears internal data structures for events handling */ void replay_finish_events (void ); -/*! Flushes events queue */ -void replay_flush_events (void );/*! Returns true if there are any unsaved events in the queue */ bool replay_has_events (void );/*! Saves events from queue into the file */
PATCH v7 14/19 gdbstub: add reverse step support in replay mode
In reply to Pavel DovgalyukGDB remote protocol supports two reverse debugging commands:
reverse step and reverse continue.
This patch adds support of the first one to the gdbstub. Reverse step is intended to step one instruction in the backwards direction. This is not possible in regular execution.
But replayed execution is deterministic, therefore we can load one of the prior snapshots and proceed to the desired step. It is equivalent to stepping one instruction back.
There should be at least one snapshot preceding the debugged part of the replay log.
--- accel/tcg/translator.c | 1 + cpus.c | 14 +++++++++++--- exec.c | 5 +++++ gdbstub.c | 42 +++++++++++++++++++++++++++++++++++++++--- include/sysemu/replay.h | 7 +++++++ replay/replay-debugging.c | 33 +++++++++++++++++++++++++++++++++ stubs/replay.c | 5 +++++ 7 files changed, 101 insertions(+), 6 deletions(-)
accel/tcg/translator.c +1 -0
@@ -17,6 +17,7 @@#include "exec/gen-icount.h" #include "exec/log.h" #include "exec/translator.h" +#include "sysemu/replay.h" /* Pairs with tcg_clear_temp_count. To be called by #TranslatorOps.{translate_insn,tb_stop} if
cpus.c +11 -3
@@ -1104,9 +1104,17 @@ static bool cpu_can_run(CPUState *cpu)static void cpu_handle_guest_debug (CPUState *cpu) { -gdb_set_stop_cpu (cpu); -qemu_system_debug_request (); - cpu->stopped = true; +if (!replay_running_debug ()) { +gdb_set_stop_cpu (cpu); +qemu_system_debug_request (); + cpu->stopped = true; + }else { +if (!cpu->singlestep_enabled) { +cpu_single_step (cpu,SSTEP_ENABLE ); + }else { +cpu_single_step (cpu,0 ); + } + } }#ifdef CONFIG_LINUX
exec.c +5 -0
@@ -2738,6 +2738,11 @@ static void check_watchpoint(int offset, int len, MemTxAttrs attrs, int flags) QTAILQ_FOREACH(wp, &cpu->watchpoints, entry) {if (cpu_watchpoint_address_matches (wp, vaddr, len) && (wp->flags & flags)) { +if (replay_running_debug ()) { +/* Don't process the watchpoints when we are +in a reverse debugging operation. */ +return ; + }if (flags ==BP_MEM_READ ) { wp->flags |=BP_WATCHPOINT_HIT_READ ; }else {
gdbstub.c +39 -3
80 lines not on this page · raw message
include/sysemu/replay.h +7 -0
@@ -73,6 +73,13 @@ void replay_finish(void);void replay_add_blocker (Error *reason);/* Returns name of the replay log file */ const char *replay_get_filename (void ); +/* Start making one step in backward direction. +Used by gdbstub for backwards debugging. +Returns true on success. */ +bool replay_reverse_step (void ); +/* Returns true if replay module is processing +reverse_continue or reverse_step request */ +bool replay_running_debug (void );/* Processing the instructions */
replay/replay-debugging.c +33 -0
@@ -21,6 +21,13 @@#include "block/snapshot.h" #include "migration/snapshot.h" +static bool replay_is_debugging; + +bool replay_running_debug (void ) +{ +return replay_is_debugging; +} +void hmp_info_replay (Monitor *mon,const QDict *qdict) {if (replay_mode ==REPLAY_MODE_NONE ) {
@@ -215,3 +222,29 @@ void hmp_replay_seek(Monitor *mon, const QDict *qdict)return ; } } + +static void replay_stop_vm_debug (void *opaque) +{ + replay_is_debugging = false; +vm_stop (RUN_STATE_DEBUG ); +replay_break (-1LL ,NULL ,NULL ); +} + +bool replay_reverse_step (void ) +{ +Error *err =NULL ; + +assert (replay_mode ==REPLAY_MODE_PLAY ); + +if (replay_get_current_step () !=0 ) { +replay_seek (replay_get_current_step () -1 , replay_stop_vm_debug, &err); +if (err) { +error_free (err); +return false; + } + replay_is_debugging = true; +return true; + } + +return false; +}
stubs/replay.c +5 -0
@@ -80,3 +80,8 @@ void replay_mutex_lock(void)void replay_mutex_unlock (void ) { } + +bool replay_reverse_step (void ) +{ +return false; +}
PATCH v7 15/19 gdbstub: add reverse continue support in replay mode
In reply to Pavel DovgalyukThis patch adds support of the reverse continue operation for gdbstub. Reverse continue finds the last breakpoint that would happen in normal execution from the beginning to the current moment. Implementation of the reverse continue replays the execution twice:
to find the breakpoints that were hit and to seek to the last breakpoint. Reverse continue loads the previous snapshot and tries to find the breakpoint since that moment. If there are no such breakpoints, it proceeds to the earlier snapshot, and so on. When no breakpoints or watchpoints were hit at all, execution stops at the beginning of the replay log.
--- cpus.c | 3 ++ exec.c | 1 + gdbstub.c | 10 ++++++- include/sysemu/replay.h | 6 ++++ replay/replay-debugging.c | 69 +++++++++++++++++++++++++++++++++++++++++++++ stubs/replay.c | 5 +++ 6 files changed, 93 insertions(+), 1 deletion(-)
cpus.c +3 -0
@@ -1110,6 +1110,9 @@ static void cpu_handle_guest_debug(CPUState *cpu) cpu->stopped = true; }else {if (!cpu->singlestep_enabled) { +/* Report about the breakpoint and +make a single step to skip it */ +replay_breakpoint ();cpu_single_step (cpu,SSTEP_ENABLE ); }else {cpu_single_step (cpu,0 );
exec.c +1 -0
@@ -2741,6 +2741,7 @@ static void check_watchpoint(int offset, int len, MemTxAttrs attrs, int flags)if (replay_running_debug ()) {/* Don't process the watchpoints when we are in a reverse debugging operation. */ +replay_breakpoint ();return ; }if (flags ==BP_MEM_READ ) {
gdbstub.c +9 -1
@@ -1111,6 +1111,14 @@ static int gdb_handle_packet(GDBState *s, const char *line_buf)put_packet (s,"E14" );break ; } +case 'c' : +if (replay_reverse_continue ()) { +gdb_continue (s); +return RS_IDLE ; + }else { +put_packet (s,"E14" ); +break ; + }default : goto unknown_command; }
@@ -1381,7 +1389,7 @@ static int gdb_handle_packet(GDBState *s, const char *line_buf)pstrcat (buf,sizeof (buf),";qXfer:features:read+" ); }if (replay_mode ==REPLAY_MODE_PLAY ) { -pstrcat (buf,sizeof (buf),";ReverseStep+" ); +pstrcat (buf,sizeof (buf),";ReverseStep+;ReverseContinue+" ); }put_packet (s, buf);break ;
include/sysemu/replay.h +6 -0
@@ -77,9 +77,15 @@ const char *replay_get_filename(void);Used by gdbstub for backwards debugging. Returns true on success. */ bool replay_reverse_step (void ); +/* Start searching the last breakpoint/watchpoint. +Used by gdbstub for backwards debugging. +Returns true if the process successfully started. */ +bool replay_reverse_continue (void );/* Returns true if replay module is processing reverse_continue or reverse_step request */ bool replay_running_debug (void ); +/* Called in reverse debugging mode to collect breakpoint information */ +void replay_breakpoint (void );/* Processing the instructions */
replay/replay-debugging.c +69 -0
78 lines not on this page · raw message
stubs/replay.c +5 -0
@@ -85,3 +85,8 @@ bool replay_reverse_step(void) {return false; } + +bool replay_reverse_continue (void ) +{ +return false; +}
PATCH v7 16/19 replay: describe reverse debugging in docs/replay.txt
In reply to Pavel DovgalyukThis patch updates the documentation and describes usage of the reverse debugging in QEMU+GDB.
--- docs/replay.txt | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
docs/replay.txt +33 -0
@@ -293,6 +293,39 @@ for recording and replaying must contain identical number of ports in record and replay modes, but their backends may differ. E.g., '-serial stdio' in record mode, and '-serial null' in replay mode. +Reverse debugging +----------------- + +Reverse debugging allows "executing" the program in reverse direction. +GDB remote protocol supports "reverse step" and "reverse continue" +commands. The first one steps single instruction backwards in time, +and the second one finds the last breakpoint in the past. + +Recorded executions may be used to enable reverse debugging. QEMU can't +execute the code in backwards direction, but can load a snapshot and +replay forward to find the desired position or breakpoint. + +The following GDB commands are supported: + - reverse-stepi (or rsi) - step one instruction backwards + - reverse-continue (or rc) - find last breakpoint in the past + +Reverse step loads the nearest snapshot and replays the execution until +the required instruction is met. + +Reverse continue may include several passes of examining the execution +between the snapshots. Each of the passes include the following steps: + 1. loading the snapshot + 2. replaying to examine the breakpoints + 3. if breakpoint or watchpoint was met + - loading the snaphot again + - replaying to the required breakpoint + 4. else + - proceeding to the p.1 with the earlier snapshot + +Therefore usage of the reverse debugging requires at least one snapshot +created in advance. See the "Snapshotting" section to learn about running +record/replay and creating the snapshot in these modes. + Replay log format -----------------
PATCH v7 17/19 replay: add BH oneshot event for block layer
In reply to Pavel DovgalyukReplay is capable of recording normal BH events, but sometimes there are single use callbacks scheduled with aio_bh_schedule_oneshot function. This patch enables recording and replaying such callbacks. Block layer uses these events for calling the completion function. Replaying these calls makes the execution deterministic.
--
v6:
- moved stub function to the separate file for fixing linux-user build
--- block/block-backend.c | 3 ++- include/sysemu/replay.h | 3 +++ replay/replay-events.c | 16 ++++++++++++++++ replay/replay-internal.h | 1 + replay/replay.c | 2 +- stubs/Makefile.objs | 1 + stubs/replay-user.c | 9 +++++++++ 7 files changed, 33 insertions(+), 2 deletions(-) create mode 100644 stubs/replay-user.c
block/block-backend.c +2 -1
@@ -17,6 +17,7 @@#include "block/throttle-groups.h" #include "sysemu/blockdev.h" #include "sysemu/sysemu.h" +#include "sysemu/replay.h" #include "qapi/error.h" #include "qapi/qapi-events-block.h" #include "qemu/id.h"
@@ -1379,7 +1380,7 @@ static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, int bytes, acb->has_returned = true;if (acb->rwco.ret !=NOT_DONE ) { -aio_bh_schedule_oneshot (blk_get_aio_context (blk), +replay_bh_schedule_oneshot_event (blk_get_aio_context (blk), blk_aio_complete_bh, acb); }
include/sysemu/replay.h +3 -0
@@ -157,6 +157,9 @@ bool replay_events_enabled(void);void replay_flush_events (void );/*! Adds bottom half event to the queue */ void replay_bh_schedule_event (QEMUBH *bh); +/* Adds oneshot bottom half event to the queue */ +void replay_bh_schedule_oneshot_event (AioContext *ctx, +QEMUBHFunc *cb,void *opaque);/*! Adds input event to the queue */ void replay_input_event (QemuConsole *src,InputEvent *evt);/*! Adds input sync event to the queue */
replay/replay-events.c +16 -0
@@ -37,6 +37,9 @@ static void replay_run_event(Event *event)case REPLAY_ASYNC_EVENT_BH :aio_bh_call (event->opaque);break ; +case REPLAY_ASYNC_EVENT_BH_ONESHOT : + ((QEMUBHFunc *)event->opaque)(event->opaque2); +break ;case REPLAY_ASYNC_EVENT_INPUT :qemu_input_event_send_impl (NULL , (InputEvent *)event->opaque);qapi_free_InputEvent ((InputEvent *)event->opaque);
@@ -132,6 +135,17 @@ void replay_bh_schedule_event(QEMUBH *bh) } } +void replay_bh_schedule_oneshot_event (AioContext *ctx, +QEMUBHFunc *cb,void *opaque) +{ +if (events_enabled) { +uint64_t id =replay_get_current_step (); +replay_add_event (REPLAY_ASYNC_EVENT_BH_ONESHOT , cb, opaque, id); + }else { +aio_bh_schedule_oneshot (ctx, cb, opaque); + } +} +void replay_add_input_event (struct InputEvent *event) {replay_add_event (REPLAY_ASYNC_EVENT_INPUT , event,NULL ,0 );
@@ -162,6 +176,7 @@ static void replay_save_event(Event *event, int checkpoint)/* save event-specific data */ switch (event->event_kind) {case REPLAY_ASYNC_EVENT_BH : +case REPLAY_ASYNC_EVENT_BH_ONESHOT :replay_put_qword (event->id);break ;case REPLAY_ASYNC_EVENT_INPUT :
@@ -216,6 +231,7 @@ static Event *replay_read_event(int checkpoint)/* Events that has not to be in the queue */ switch (replay_state.read_event_kind) {case REPLAY_ASYNC_EVENT_BH : +case REPLAY_ASYNC_EVENT_BH_ONESHOT :if (replay_state.read_event_id ==-1 ) { replay_state.read_event_id =replay_get_qword (); }
replay/replay-internal.h +1 -0
@@ -51,6 +51,7 @@ enum ReplayEvents {enum ReplayAsyncEventKind {REPLAY_ASYNC_EVENT_BH , +REPLAY_ASYNC_EVENT_BH_ONESHOT ,REPLAY_ASYNC_EVENT_INPUT ,REPLAY_ASYNC_EVENT_INPUT_SYNC ,REPLAY_ASYNC_EVENT_CHAR_READ ,
replay/replay.c +1 -1
@@ -22,7 +22,7 @@/* Current version of the replay mechanism. Increase it when file format changes. */ -#define REPLAY_VERSION 0xe02007 +#define REPLAY_VERSION 0xe02008/* Size of replay log header */ #define HEADER_SIZE (sizeof(uint32_t) + sizeof(uint64_t))
stubs/Makefile.objs +1 -0
@@ -24,6 +24,7 @@ stub-obj-y += monitor.o stub-obj-y += notify-event.o stub-obj-y += qtest.o stub-obj-y += replay.o +stub-obj-y += replay-user.o stub-obj-y += runstate-check.o stub-obj-y += set-fd-handler.o stub-obj-y += slirp.o
stubs/replay-user.c new file +9 -0
@@ -0,0 +1,9 @@ +#include "qemu/osdep.h" +#include "sysemu/replay.h" +#include "sysemu/sysemu.h" + +void replay_bh_schedule_oneshot_event (AioContext *ctx, +QEMUBHFunc *cb,void *opaque) +{ +aio_bh_schedule_oneshot (ctx, cb, opaque); +}
Re: [Qemu-devel] [PATCH v7 17/19] replay: add BH oneshot event for block layer
Am 10.10.2018 um 15:35 hat Pavel Dovgalyuk geschrieben:
Quoted 7 lines
Replay is capable of recording normal BH events, but sometimes there are single use callbacks scheduled with
aio_bh_schedule_oneshotfunction. This patch enables recording and replaying such callbacks. Block layer uses these events for calling the completion function. Replaying these calls makes the execution deterministic.Signed-off-by Pavel Dovgalyuk <[email protected]>
What are the rules for using aio_bh_schedule_oneshot() vs. replay_bh_schedule_oneshot_event()? You are modifying one place, but if I grep for aio_bh_schedule_oneshot(), I find many other places that use it. Why is this one place special?
Quoted 28 lines
--
v6:
- moved stub function to the separate file for fixing linux-user build--- block/block-backend.c | 3 ++- include/sysemu/replay.h | 3 +++ replay/replay-events.c | 16 ++++++++++++++++ replay/replay-internal.h | 1 + replay/replay.c | 2 +- stubs/Makefile.objs | 1 + stubs/replay-user.c | 9 +++++++++ 7 files changed, 33 insertions(+), 2 deletions(-) create mode 100644 stubs/replay-user.cblock/block-backend.c +1
-0@@ -17,6 +17,7 @@#include "block/throttle-groups.h" #include "sysemu/blockdev.h" #include "sysemu/sysemu.h" +#include "sysemu/replay.h" #include "qapi/error.h" #include "qapi/qapi-events-block.h" #include "qemu/id.h"
block/block-backend.c static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, int bytes, · line 1384
Quoted 6 lines
block/block-backend.c +1
-1@@ -1379,7 +1380,7 @@ static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, int bytes, acb->has_returned = true;if (acb->rwco.ret !=NOT_DONE ) { -aio_bh_schedule_oneshot (blk_get_aio_context (blk), +replay_bh_schedule_oneshot_event (blk_get_aio_context (blk), blk_aio_complete_bh, acb);
Indentation is off.
Kevin
Re: [Qemu-devel] [PATCH v7 17/19] replay: add BH oneshot event for block layer
Quoted 8 lines
From: Kevin Wolf [mailto:[email protected]]
Am 10.10.2018 um 15:35 hat Pavel Dovgalyuk geschrieben:Quoted 6 lines
Replay is capable of recording normal BH events, but sometimes there are single use callbacks scheduled with
aio_bh_schedule_oneshotfunction. This patch enables recording and replaying such callbacks. Block layer uses these events for calling the completion function. Replaying these calls makes the execution deterministic.
Quoted 6 lines
Quoted 1 line
Signed-off-by Pavel Dovgalyuk <[email protected]>What are the rules for using
aio_bh_schedule_oneshot()vs.replay_bh_schedule_oneshot_event()? You are modifying one place, but if I grep foraio_bh_schedule_oneshot(), I find many other places that use it. Why is this one place special?
This one is special, because it caused a record/replay bug.
I can't validate all other places, because I don't understand block layer good enough.
That's why our current strategy is fixing replay, when it is breaks. It's too hard to create the test for verifying the modification. And for the current patch there was the bug which was fixed.
The rule is the following: when the event is asynchronous and its finalization affects the guest state, then this event should be processed by the record/replay queue.
Quoted 31 lines
block/block-backend.c static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, int
Quoted 29 lines
--
v6:
- moved stub function to the separate file for fixing linux-user build--- block/block-backend.c | 3 ++- include/sysemu/replay.h | 3 +++ replay/replay-events.c | 16 ++++++++++++++++ replay/replay-internal.h | 1 + replay/replay.c | 2 +- stubs/Makefile.objs | 1 + stubs/replay-user.c | 9 +++++++++ 7 files changed, 33 insertions(+), 2 deletions(-) create mode 100644 stubs/replay-user.cblock/block-backend.c +1
-0@@ -17,6 +17,7 @@#include "block/throttle-groups.h" #include "sysemu/blockdev.h" #include "sysemu/sysemu.h" +#include "sysemu/replay.h" #include "qapi/error.h" #include "qapi/qapi-events-block.h" #include "qemu/id.h" @@ -1379,7 +1380,7 @@ static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, intbytes,
Quoted 1 line
Quoted 7 lines
block/block-backend.c static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, int bytes, · line 1384
Quoted 5 lines
acb->has_returned = true; if (acb->rwco.ret != NOT_DONE) {block/block-backend.c +1
-1-aio_bh_schedule_oneshot (blk_get_aio_context (blk), +replay_bh_schedule_oneshot_event (blk_get_aio_context (blk), blk_aio_complete_bh, acb);Indentation is off.
Thanks.
Pavel Dovgalyuk
Re: [Qemu-devel] [PATCH v7 17/19] replay: add BH oneshot event for block layer
Am 30.11.2018 um 09:21 hat Pavel Dovgalyuk geschrieben:
Quoted 15 lines
Quoted 8 lines
From: Kevin Wolf [mailto:[email protected]]
Am 10.10.2018 um 15:35 hat Pavel Dovgalyuk geschrieben:Quoted 6 lines
Replay is capable of recording normal BH events, but sometimes there are single use callbacks scheduled with
aio_bh_schedule_oneshotfunction. This patch enables recording and replaying such callbacks. Block layer uses these events for calling the completion function. Replaying these calls makes the execution deterministic.Quoted 6 lines
Quoted 1 line
Signed-off-by Pavel Dovgalyuk <[email protected]>What are the rules for using
aio_bh_schedule_oneshot()vs.replay_bh_schedule_oneshot_event()? You are modifying one place, but if I grep foraio_bh_schedule_oneshot(), I find many other places that use it. Why is this one place special?
Quoted 7 lines
This one is special, because it caused a record/replay bug. I can't validate all other places, because I don't understand block layer good enough.
That's why our current strategy is fixing replay, when it is breaks. It's too hard to create the test for verifying the modification. And for the current patch there was the bug which was fixed.
So nobody really understands the code, but we're just fixing symptoms whenever something crashes, without ever thinking about the design? And then the code will organically grow to maybe approximate what we wanted it to do?
Honestly, that's not the way to build reliable and maintainable software.
Quoted 3 lines
The rule is the following: when the event is asynchronous and its finalization affects the guest state, then this event should be processed by the record/replay queue.
Are there any BHs that can never affect the guest state?
Maybe you should really intercept this inside qemu_bh_schedule() and aio_bh_schedule_oneshot() to catch all instances. This would look more likely to address the root cause rather than twenty different special cases and forgetting the other hundred instances.
But why do you even need to record and replay BHs? Aren't they already fully deterministic if the code that schedules them is deterministic? Is this hinting at problems in a different part of the code, so that the caller isn't deterministic even though we expect it to be?
Kevin
Re: [Qemu-devel] [PATCH v7 17/19] replay: add BH oneshot event for block layer
Quoted 30 lines
From: Kevin Wolf [mailto:[email protected]]
Am 30.11.2018 um 09:21 hat Pavel Dovgalyuk geschrieben:Quoted 15 lines
Quoted 8 lines
From: Kevin Wolf [mailto:[email protected]]
Am 10.10.2018 um 15:35 hat Pavel Dovgalyuk geschrieben:Quoted 6 lines
Replay is capable of recording normal BH events, but sometimes there are single use callbacks scheduled with
aio_bh_schedule_oneshotfunction. This patch enables recording and replaying such callbacks. Block layer uses these events for calling the completion function. Replaying these calls makes the execution deterministic.Quoted 6 lines
Quoted 1 line
Signed-off-by Pavel Dovgalyuk <[email protected]>What are the rules for using
aio_bh_schedule_oneshot()vs.replay_bh_schedule_oneshot_event()? You are modifying one place, but if I grep foraio_bh_schedule_oneshot(), I find many other places that use it. Why is this one place special?Quoted 7 lines
This one is special, because it caused a record/replay bug. I can't validate all other places, because I don't understand block layer good enough.
That's why our current strategy is fixing replay, when it is breaks. It's too hard to create the test for verifying the modification. And for the current patch there was the bug which was fixed.
So nobody really understands the code, but we're just fixing symptoms whenever something crashes, without ever thinking about the design? And then the code will organically grow to maybe approximate what we wanted it to do?
Quoted 8 lines
Honestly, that's not the way to build reliable and maintainable software.
Quoted 3 lines
The rule is the following: when the event is asynchronous and its finalization affects the guest state, then this event should be processed by the record/replay queue.
Are there any BHs that can never affect the guest state?
For example, qemu_bh_schedule(qmp_dispatcher_bh) in handle_qmp_command. thread-pool.c also uses BH callbacks, but they are for creating threads, and not for modifying the replayed guest state.
Quoted 9 lines
Maybe you should really intercept this inside
qemu_bh_schedule()andaio_bh_schedule_oneshot()to catch all instances. This would look more likely to address the root cause rather than twenty different special cases and forgetting the other hundred instances.But why do you even need to record and replay BHs? Aren't they already fully deterministic if the code that schedules them is deterministic? Is this hinting at problems in a different part of the code, so that the caller isn't deterministic even though we expect it to be?
BHs may be invoked at random moments, that are not related to the executed instructions. As we replay CPU, memory, and peripheral device state, we must replay all invocations all BHs that affect them.
Replay must be done precisely, with instruction-level granularity. That is why we record the moment of BH invocation for matching it in the replay phase.
Pavel Dovgalyuk
PATCH v7 18/19 replay: init rtc after enabling the replay
In reply to Pavel DovgalyukThis patch postpones the call of 'configure_rtc' function. This call uses host clock to configure the rtc, but host clock access should be recorded when using icount record/replay mode. Therefore now rtc is configured after switching record/replay mode on.
--- vl.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
vl.c +7 -4
@@ -2932,6 +2932,7 @@ int main(int argc, char **argv, char **envp)DisplayState *ds;QemuOpts *opts, *machine_opts;QemuOpts *icount_opts =NULL , *accel_opts =NULL ; +QemuOpts *rtc_opts =NULL ;QemuOptsList *olist;int optind;const char *optarg;
@@ -3738,12 +3739,11 @@ int main(int argc, char **argv, char **envp)warn_report ("This option is ignored and will be removed soon" );break ;case QEMU_OPTION_rtc: - opts =qemu_opts_parse_noisily (qemu_find_opts ("rtc" ), optarg, - false); -if (!opts) { + rtc_opts =qemu_opts_parse_noisily (qemu_find_opts ("rtc" ), + optarg, false); +if (!rtc_opts) {exit (1 ); } -configure_rtc (opts);break ;case QEMU_OPTION_tb_size:#ifndef CONFIG_TCG
@@ -3954,6 +3954,9 @@ int main(int argc, char **argv, char **envp)loc_set_none ();replay_configure (icount_opts); +if (rtc_opts) { +configure_rtc (rtc_opts); + }if (incoming && !preconfig_exit_requested) { error_report("'preconfig' and 'incoming' options are "
Re: [Qemu-devel] [PATCH v7 18/19] replay: init rtc after enabling the replay
I guess 'configure_rtc' function is not the only one which call should be postponed. Since record/replay uses its 'hooks' at very low levels of qemu core, any option processing is subject to potential indirect dependence on effects of 'replay_configure' call. The most apparent example would be 'qemu_clock_get_*(QEMU_CLOCK_HOST)' calls, which are used widely. Furthermore, right now I submitting a patch which adds one such call in place before any option parsed yet, so it will also need to be moved to follow this patch. Even if such calls aren't involved in current version, no guarantees that they will not appear in future (like I already doing it), because there are nothing which prevents developers from doing so.
One way, how this could be enforced, is to add something like 'REPLAY_MODE_INVALID' to 'ReplayMode' enum, replace initialization of global 'replay_mode' variable with this value and add corresponding checks/assertions to 'REPLAY_CLOCK' macro, etc.
But it would be more reliable to move 'icount_opts' processing out of common processing loop (marked as 'second pass of option parsing') and place it to new pass, located before current 'second' one (like it already done for '-nouserconfig' option), and also move 'replace_configure' right after it. Probably it will also require to tune 'icount_opts' processing and record/replay initialization. Since record/replay feature is very specific, I consider it's worth to make such exception for it.
ср, 10 окт. 2018 г. в 19:32, Pavel Dovgalyuk <[email protected]>:
Quoted 42 lines
This patch postpones the call of '
configure_rtc' function. This call uses host clock to configure the rtc, but host clock access should be recorded when using icount record/replay mode. Therefore now rtc is configured after switching record/replay mode on.Signed-off-by Pavel Dovgalyuk <[email protected]>--- vl.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)vl.c +4
-4@@ -2932,6 +2932,7 @@ int main(int argc, char **argv, char **envp)DisplayState *ds;QemuOpts *opts, *machine_opts;QemuOpts *icount_opts =NULL , *accel_opts =NULL ; +QemuOpts *rtc_opts =NULL ;QemuOptsList *olist;int optind;const char *optarg;@@ -3738,12 +3739,11 @@ int main(int argc, char **argv, char **envp)warn_report ("This option is ignored and will be removed soon");break ;case QEMU_OPTION_rtc: - opts =qemu_opts_parse_noisily (qemu_find_opts ("rtc" ), optarg, - false); -if (!opts) { + rtc_opts =qemu_opts_parse_noisily (qemu_find_opts ("rtc" ), + optarg, false); +if (!rtc_opts) {exit (1 ); } -configure_rtc (opts);break ;case
QEMU_OPTION_tb_size:
#ifndefCONFIG_TCGvl.c +0
-0@@ -3954,6 +3954,9 @@ int main(int argc, char **argv, char **envp)loc_set_none ();
vl.c int main(int argc, char **argv, char **envp)
Quoted 10 lines
vl.c +3
-1replay_configure (icount_opts); +if (rtc_opts) { +configure_rtc (rtc_opts); + }if (incoming && !preconfig_exit_requested) { error_report("'preconfig' and 'incoming' options are " --
С уважением,
Артем Писаренко
PATCH v7 19/19 replay: document development rules
In reply to Pavel DovgalyukThis patch introduces docs/devel/replay.txt which describes the rules that should be followed to make virtual devices usable in record/replay mode.
--- docs/devel/replay.txt | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 docs/devel/replay.txt
docs/devel/replay.txt new file +45 -0
@@ -0,0 +1,45 @@ +Record/replay mechanism, that could be enabled through icount mode, expects +the virtual devices to satisfy the following requirements. + +The main idea behind this document is that everything that affects +the guest state during execution in icount mode should be deterministic. + +Timers +====== + +All virtual devices should use virtual clock for timers that change the guest +state. Virtual clock is deterministic, therefore such timers are deterministic +too. + +Virtual devices can also use realtime clock for the events that do not change +the guest state directly. When the clock ticking should depend on VM execution +speed, use virtual ext clock. It is not deterministic, but its speed depends +on the guest execution. This clock is used by the virtual devices (e.g., +slirp routing device) that lie outside the replayed guest. + +Bottom halves +============= + +Bottom half callbacks, that affect the guest state, should be invoked through +replay_bh_schedule_event or replay_bh_schedule_oneshot_event functions. +Their invocations are saved in record mode and synchronized with the existing +log in replay mode. + +Saving/restoring the VM state +============================= + +All fields in the device state structure (including virtual timers) +should be restored by loadvm to the same values they had before savevm. + +Avoid accessing other devices' state, because the order of saving/restoring +is not defined. It means that you should not call functions like +'update_irq' in post_load callback. Save everything explicitly to avoid +the dependencies that may make restoring the VM state non-deterministic. + +Stopping the VM +=============== + +Stopping the guest should not interfere with its state (with the exception +of the network connections, that could be broken by the remote timeouts). +VM can be stopped at any moment of replay by the user. Restarting the VM +after that stop should not break the replay by the unneeded guest state change.
Re: [Qemu-devel] [PATCH v7 19/19] replay: document development rules
Great! I'm voting with all my fingers up for such rules. But I would suggest even more generic rules which prevent breaking determinism in a more wide sense. At least, where such breakage is trivial to avoid.
Currently I'm working on modification, which extends conditions where guest execution is kept deterministic, above such narrow set like "only rtc clock=vm, no serial devices, no network, no external communication interfaces at all, etc...". Also I'm dealing with bugs (features?), which prevents advertised determinism even in such restricted conditions. I found that my work involves very similar efforts as Pavel's work. And this is extra hard work. I'm feeling like fighting with hundreds of maintainers and contributors, whose efforts are in opposite direction. They starting from qemu underlying core, encouraging asyncronous processing (aio, threads, virtio ioeventfd, etc.), and ending with particular modules or hardware models, which negligently uses any kind of non-blocking calls, callbacks, and even inappropriate QEMUClockType. Nobody cares about synchronization with vcpu thread, except Pavel, me and, possibly, 1-2 more persons in a whole world. I can understand why. Main reason is that it might greatly degradate performance of emulation, which might be avoided by introducing very high complexity. So, my words shouldn't be treated as any kind of criticism. I perfectly understand that it's a complex issue.
Key difference of record/replay is that it solves problem by hooking calls of any source of asynchrony at low level and just replaying it. In other words, it deals with end effects, whereas non-record/replay use case doesn't allow such solution and have to eliminate source of undesired asynchrony by design. As such, record/replay have strong immunity to violation of 'generic determinism' rules and even to hidden and tricky bugs in any module which affects guest state. And that's why development rules Pavel imposes are so democratic (relative to generic ones, I would like to exist).
Anyway, it's just generic idea for discussion. I know, it needs to be more specific. But, if nobody will express interest, I see no reason to continue.
P.S. Trivial example of how qemu could extend conditions for deterministic execution. Сhardev would perform writing to backend using blocking calls, thus making possible deterministic execution in use cases, where guest has only one serial port which outputs data to console and have no interaction with user. At least it would provide user with option, selecting between better performance and determinism.
ср, 10 окт. 2018 г. в 19:32, Pavel Dovgalyuk <[email protected]>:
Quoted 72 lines
This patch introduces
docs/devel/replay.txtwhich describes the rules that should be followed to make virtual devices usable in record/replay mode.Signed-off-by Pavel Dovgalyuk <[email protected]>--- docs/devel/replay.txt | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 docs/devel/replay.txtdocs/devel/replay.txt new file +1
-0@@ -0,0 +1,45 @@ +Record/replay mechanism, that could be enabled through icount mode,expects
docs/devel/replay.txt new file +9
-0+the virtual devices to satisfy the following requirements. + +The main idea behind this document is that everything that affects +the guest state during execution in icount mode should be deterministic. + +Timers +====== + +All virtual devices should use virtual clock for timers that change theguest
+state. Virtual clock is deterministic, therefore such timers are deterministicdocs/devel/replay.txt new file +3
-0+too. + +Virtual devices can also use realtime clock for the events that do notchange
+the guest state directly. When the clock ticking should depend on VM execution
+speed, use virtual ext clock. It is not deterministic, but its speed dependsdocs/devel/replay.txt new file +7
-0+on the guest execution. This clock is used by the virtual devices (e.g., +slirp routing device) that lie outside the replayed guest. + +Bottom halves +============= + +Bottom half callbacks, that affect the guest state, should be invokedthrough
docs/devel/replay.txt new file +2
-0+replay_bh_schedule_event or replay_bh_schedule_oneshot_event functions. +Their invocations are saved in record mode and synchronized with theexisting
docs/devel/replay.txt new file +9
-0+log in replay mode. + +Saving/restoring the VM state +============================= + +All fields in the device state structure (including virtual timers) +should be restored by loadvm to the same values they had before savevm. + +Avoid accessing other devices' state, because the order ofsaving/restoring
docs/devel/replay.txt new file +11
-0+is not defined. It means that you should not call functions like +'update_irq' in post_load callback. Save everything explicitly to avoid +the dependencies that may make restoring the VM state non-deterministic. + +Stopping the VM +=============== + +Stopping the guest should not interfere with its state (with the exception +of the network connections, that could be broken by the remote timeouts). +VM can be stopped at any moment of replay by the user. Restarting the VM +after that stop should not break the replay by the unneeded guest statechange.
Quoted 1 line
--
С уважением,
Артем Писаренко
Hi Pavel,
Please do not CC me in this series, this isn’t anyhow relevant for me anymore.
Regards, Alex
ср, 10 окт. 2018 г. в 16:30, Pavel Dovgalyuk <[email protected]>:
Quoted 133 lines
GDB remote protocol supports reverse debugging of the targets. It includes 'reverse step' and 'reverse continue' operations. The first one finds the previous step of the execution, and the second one is intended to stop at the last breakpoint that would happen when the program is executed normally.
Reverse debugging is possible in the replay mode, when at least one snapshot was created at the record or replay phase. QEMU can use these snapshots for travelling back in time with GDB.
Running the execution in replay mode allows using GDB reverse debugging commands:
- reverse-stepi (or rsi): Steps one instruction to the past. QEMU loads on of the prior snapshots and proceeds to the desired instruction forward. When that step is reaches, execution stops.
- reverse-continue (or rc): Runs execution "backwards". QEMU tries to find breakpoint or watchpoint by loaded prior snapshot and replaying the execution. Then QEMU loads snapshots again and replays to the latest breakpoint. When there are no breakpoints in the examined section of the execution, QEMU finds one more snapshot and tries again. After the first snapshot is processed, execution stops at this snapshot.The set of patches include the following modifications:
- gdbstub update for reverse debugging support
- functions that automatically perform reverse step and reverse continue operations
- hmp/qmp commands for manipulating the replay process
- improvement of the snapshotting for saving the execution step in the snapshot parameters
- other record/replay fixesThe patches are available in the repository:
https://github.com/ispras/qemu/tree/rr-181010v7 changes:
- rebased to the new master with upstreamed patches from the series
- several improvements in hmp/qmp commands handling (suggested by Markus
Armbruster)
- fixed record/replay with '-rtc base' option enabled
- added document with virtual hardware requirementsv6 changes:
- rebased to the new version of master
- fixed build of linux-user configurations
- added new clock for slirp and vnc timersv5 changes:
- multiple fixes of record/replay bugs appeared after QEMU core update
- changed reverse debugging to 'since 3.1'v4 changes:
- changed 'since 2.13' to 'since 3.0' in json (as suggested by Eric Blake)v3 changes:
- Fixed PS/2 bug with save/load vm, which caused failures of the replay.
- Rebased to the new code base.
- Minor fixes.v2 changes:
- documented reverse debugging
- fixed start vmstate loading in record mode
- documented qcow2 changes (as suggested by Eric Blake)
- made icount SnapshotInfo field optional (as suggested by Eric Blake)
- renamed qmp commands (as suggested by Eric Blake)
- minor changes---
Pavel Dovgalyuk (19):
block: implementbdrv_snapshot_gotofor blkreplay
replay: disable default snapshot for record/replay
replay: update docs for record/replay with block devices
replay: don't drain/flush bdrv queue while RR is working
replay: finish record/replay before closing the disks
qcow2: introduce icount field for snapshots
migration: introduce icount field for snapshots
replay: provide and accessor for rr filename
replay: introduce info hmp/qmp command
replay: introduce breakpoint at the specified step
replay: implement replay-seek command to proceed to the desired step
replay: refine replay-time module
replay: flush rr queue before loading the vmstate
gdbstub: add reverse step support in replay mode
gdbstub: add reverse continue support in replay mode
replay: describe reverse debugging indocs/replay.txt
replay: add BH oneshot event for block layer
replay: init rtc after enabling the replay
replay: document development rulesaccel/tcg/translator.c | 1 block/blkreplay.c | 8 + block/block-backend.c | 3 block/io.c | 22 +++ block/qapi.c | 17 ++ block/qcow2-snapshot.c | 9 + block/qcow2.h | 2 blockdev.c | 10 + cpus.c | 19 ++- docs/devel/replay.txt | 45 ++++++ docs/interop/qcow2.txt | 4 + docs/replay.txt | 45 ++++++ exec.c | 6 + gdbstub.c | 50 +++++++ hmp-commands-info.hx | 14 ++ hmp-commands.hx | 44 ++++++ hmp.h | 4 + include/block/snapshot.h | 1 include/sysemu/replay.h | 20 +++ migration/savevm.c | 15 +- qapi/block-core.json | 7 + qapi/block.json | 3 qapi/misc.json | 77 +++++++++++ replay/Makefile.objs | 3 replay/replay-debugging.c | 319 +++++++++++++++++++++++++++++++++++++++++++++ replay/replay-events.c | 16 ++ replay/replay-internal.h | 7 + replay/replay-time.c | 32 ++--- replay/replay.c | 26 ++++ stubs/Makefile.objs | 1 stubs/replay-user.c | 9 + stubs/replay.c | 10 + vl.c | 22 ++- 33 files changed, 818 insertions(+), 53 deletions(-) create mode 100644 docs/devel/replay.txt create mode 100644 replay/replay-debugging.c create mode 100644 stubs/replay-user.c--
Pavel Dovgalyuk
Signature · 1 line
Aleksandr Bezzubikov
Ping?
Paolo, what do you think about upstreaming this before soft freeze?
Pavel Dovgalyuk
Quoted 142 lines
-----Original Message-----
From: Pavel Dovgalyuk [mailto:[email protected]]
Sent: Wednesday, October 10, 2018 4:34 PM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v7 00/19] Fixing record/replay and adding reverse debuggingGDB remote protocol supports reverse debugging of the targets. It includes 'reverse step' and 'reverse continue' operations. The first one finds the previous step of the execution, and the second one is intended to stop at the last breakpoint that would happen when the program is executed normally.
Reverse debugging is possible in the replay mode, when at least one snapshot was created at the record or replay phase. QEMU can use these snapshots for travelling back in time with GDB.
Running the execution in replay mode allows using GDB reverse debugging commands:
- reverse-stepi (or rsi): Steps one instruction to the past. QEMU loads on of the prior snapshots and proceeds to the desired instruction forward. When that step is reaches, execution stops.
- reverse-continue (or rc): Runs execution "backwards". QEMU tries to find breakpoint or watchpoint by loaded prior snapshot and replaying the execution. Then QEMU loads snapshots again and replays to the latest breakpoint. When there are no breakpoints in the examined section of the execution, QEMU finds one more snapshot and tries again. After the first snapshot is processed, execution stops at this snapshot.The set of patches include the following modifications:
- gdbstub update for reverse debugging support
- functions that automatically perform reverse step and reverse continue operations
- hmp/qmp commands for manipulating the replay process
- improvement of the snapshotting for saving the execution step in the snapshot parameters
- other record/replay fixesThe patches are available in the repository:
https://github.com/ispras/qemu/tree/rr-181010v7 changes:
- rebased to the new master with upstreamed patches from the series
- several improvements in hmp/qmp commands handling (suggested by Markus Armbruster)
- fixed record/replay with '-rtc base' option enabled
- added document with virtual hardware requirementsv6 changes:
- rebased to the new version of master
- fixed build of linux-user configurations
- added new clock for slirp and vnc timersv5 changes:
- multiple fixes of record/replay bugs appeared after QEMU core update
- changed reverse debugging to 'since 3.1'v4 changes:
- changed 'since 2.13' to 'since 3.0' in json (as suggested by Eric Blake)v3 changes:
- Fixed PS/2 bug with save/load vm, which caused failures of the replay.
- Rebased to the new code base.
- Minor fixes.v2 changes:
- documented reverse debugging
- fixed start vmstate loading in record mode
- documented qcow2 changes (as suggested by Eric Blake)
- made icount SnapshotInfo field optional (as suggested by Eric Blake)
- renamed qmp commands (as suggested by Eric Blake)
- minor changes---
Pavel Dovgalyuk (19):
block: implementbdrv_snapshot_gotofor blkreplay
replay: disable default snapshot for record/replay
replay: update docs for record/replay with block devices
replay: don't drain/flush bdrv queue while RR is working
replay: finish record/replay before closing the disks
qcow2: introduce icount field for snapshots
migration: introduce icount field for snapshots
replay: provide and accessor for rr filename
replay: introduce info hmp/qmp command
replay: introduce breakpoint at the specified step
replay: implement replay-seek command to proceed to the desired step
replay: refine replay-time module
replay: flush rr queue before loading the vmstate
gdbstub: add reverse step support in replay mode
gdbstub: add reverse continue support in replay mode
replay: describe reverse debugging indocs/replay.txt
replay: add BH oneshot event for block layer
replay: init rtc after enabling the replay
replay: document development rulesaccel/tcg/translator.c | 1 block/blkreplay.c | 8 + block/block-backend.c | 3 block/io.c | 22 +++ block/qapi.c | 17 ++ block/qcow2-snapshot.c | 9 + block/qcow2.h | 2 blockdev.c | 10 + cpus.c | 19 ++- docs/devel/replay.txt | 45 ++++++ docs/interop/qcow2.txt | 4 + docs/replay.txt | 45 ++++++ exec.c | 6 + gdbstub.c | 50 +++++++ hmp-commands-info.hx | 14 ++ hmp-commands.hx | 44 ++++++ hmp.h | 4 + include/block/snapshot.h | 1 include/sysemu/replay.h | 20 +++ migration/savevm.c | 15 +- qapi/block-core.json | 7 + qapi/block.json | 3 qapi/misc.json | 77 +++++++++++ replay/Makefile.objs | 3 replay/replay-debugging.c | 319 +++++++++++++++++++++++++++++++++++++++++++++ replay/replay-events.c | 16 ++ replay/replay-internal.h | 7 + replay/replay-time.c | 32 ++--- replay/replay.c | 26 ++++ stubs/Makefile.objs | 1 stubs/replay-user.c | 9 + stubs/replay.c | 10 + vl.c | 22 ++- 33 files changed, 818 insertions(+), 53 deletions(-) create mode 100644 docs/devel/replay.txt create mode 100644 replay/replay-debugging.c create mode 100644 stubs/replay-user.c--
Pavel Dovgalyuk