Received: from eggs.gnu.org ([2001:4830:134:3::10]:52370) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aXoKM-0005OR-Ji for qemu-devel@nongnu.org; Mon, 22 Feb 2016 06:06:55 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aXoKH-0007sE-I8 for qemu-devel@nongnu.org; Mon, 22 Feb 2016 06:06:54 -0500 Received: from mx1.redhat.com ([209.132.183.28]:48035) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aXoKH-0007sA-AM for qemu-devel@nongnu.org; Mon, 22 Feb 2016 06:06:49 -0500 Date: Mon, 22 Feb 2016 12:06:44 +0100 From: Kevin Wolf Message-ID: <20160222110644.GD5387@noname.str.redhat.com> References: <20160215093810.GC5244@noname.str.redhat.com> <004701d167f8$5cbe70f0$163b52d0$@ru> <20160215140635.GF5244@noname.str.redhat.com> <005501d167fc$8ed75030$ac85f090$@ru> <20160215150110.GG5244@noname.str.redhat.com> <000601d16882$c9637270$5c2a5750$@ru> <20160216100208.GA4920@noname.str.redhat.com> <000a01d168ac$09929500$1cb7bf00$@ru> <20160216125453.GC4920@noname.str.redhat.com> <000601d16bad$e93f9eb0$bbbedc10$@ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <000601d16bad$e93f9eb0$bbbedc10$@ru> Subject: Re: [Qemu-devel] [PATCH 3/3] replay: introduce block devices record/replay List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Pavel Dovgalyuk Cc: edgar.iglesias@xilinx.com, peter.maydell@linaro.org, igor.rubinov@gmail.com, mark.burton@greensocs.com, real@ispras.ru, hines@cert.org, qemu-devel@nongnu.org, maria.klimushenkova@ispras.ru, stefanha@redhat.com, pbonzini@redhat.com, batuzovk@ispras.ru, alex.bennee@linaro.org, fred.konrad@greensocs.com Am 20.02.2016 um 08:11 hat Pavel Dovgalyuk geschrieben: > > From: Pavel Dovgalyuk [mailto:dovgaluk@ispras.ru] > > > From: Kevin Wolf [mailto:kwolf@redhat.com] > > > Am 16.02.2016 um 12:20 hat Pavel Dovgalyuk geschrieben: > > > > Coroutine Replay > > > > bool *done = req_replayed_list_get(reqid) // NULL > > > > co = > > > req_completed_list_get(e.reqid); // NULL > > > > > > There was no yield, this context switch is impossible to happen. Same > > > for the switch back. > > > > > > > req_completed_list_insert(reqid, qemu_coroutine_self()); > > > > qemu_coroutine_yield(); > > > > > > This is the point at which a context switch happens. The only other > > > point in my code is the qemu_coroutine_enter() in the other function. > > > > I've fixed aio_poll problem by disabling mutex lock for the replay_run_block_event() > > execution. Now virtual machine deterministically runs 4e8 instructions of Windows XP booting. Are you sure that the lock was unnecessary? Solving deadlocks by removing the lock is a rather adventurous method. I assume that you're still replaying events from low-level functions like qemu_clock_get_ns(). So even if you get rid of the hangs, the result is probably not quite right. I'm afraid this is going in a direction where my comments can't be more constructive than a simple "you're doing it wrong". > > But then one non-deterministic event happens. > > Callback after finishing coroutine may be called from different contexts. How does this happen? I'm not aware of callbacks being processed by any thread other than the I/O thread for that specific block device (unless you use dataplane, this is the main loop thread). > > apic_update_irq() function behaves differently being called from vcpu and io threads. > > In one case it sets CPU_INTERRUPT_POLL and in other - nothing happens. > > Kevin, do you have some ideas how to fix this issue? > This happens because of coroutines may be assigned to different threads. > Maybe there is some way of making this assignment more deterministic? Coroutines aren't randomly assigned to threads, but threads actively enter coroutines. To my knowledge this happens only when starting a request (either vcpu or I/O thread; consistent per device) or by a callback when some event happens (only I/O thread). I can't see any non-determinism here. Kevin > > Therefore execution becomes non-deterministic. > > In previous version of the patch I solved this problem by linking block events to the > > execution checkpoints. IO thread have its own checkpoints and vcpu - its own. > > Therefore apic callbacks are always called from the same thread in replay as in recording > > phase. > > Pavel Dovgalyuk >