Received: from eggs.gnu.org ([2001:4830:134:3::10]:33316) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aTUEu-00084V-P1 for qemu-devel@nongnu.org; Wed, 10 Feb 2016 07:51:25 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aTUEr-0005WR-HA for qemu-devel@nongnu.org; Wed, 10 Feb 2016 07:51:24 -0500 Received: from mail.ispras.ru ([83.149.199.45]:51559) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aTUEr-0005VD-3i for qemu-devel@nongnu.org; Wed, 10 Feb 2016 07:51:21 -0500 From: "Pavel Dovgalyuk" References: <20160209055506.8208.67.stgit@PASHA-ISP> <20160209055524.8208.16023.stgit@PASHA-ISP> <20160209102739.GB8554@noname.redhat.com> <001201d16330$6ce67e40$46b37ac0$@Dovgaluk@ispras.ru> <20160210114531.GA5474@noname.redhat.com> <000601d163fb$4cbcae70$e6360b50$@Dovgaluk@ispras.ru> <20160210122816.GB5474@noname.redhat.com> In-Reply-To: <20160210122816.GB5474@noname.redhat.com> Date: Wed, 10 Feb 2016 15:51:21 +0300 Message-ID: <000a01d16401$be4d31d0$3ae79570$@Dovgaluk@ispras.ru> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Content-Language: ru Subject: Re: [Qemu-devel] [PATCH 3/3] replay: introduce block devices record/replay List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: 'Kevin Wolf' Cc: edgar.iglesias@xilinx.com, peter.maydell@linaro.org, igor.rubinov@gmail.com, mark.burton@greensocs.com, real@ispras.ru, hines@cert.org, qemu-devel@nongnu.org, maria.klimushenkova@ispras.ru, stefanha@redhat.com, pbonzini@redhat.com, batuzovk@ispras.ru, alex.bennee@linaro.org, fred.konrad@greensocs.com > From: Kevin Wolf [mailto:kwolf@redhat.com] > Am 10.02.2016 um 13:05 hat Pavel Dovgalyuk geschrieben: > > > Am 09.02.2016 um 12:52 hat Pavel Dovgalyuk geschrieben: > > > > > From: Kevin Wolf [mailto:kwolf@redhat.com] > > > > > But even this doesn't feel completely right, because block drivers are > > > > > already layered and there is no need to hardcode something optional (and > > > > > rarely used) in the hot code path that could just be another layer. > > > > > > > > > > I assume that you know beforehand if you want to replay something, so > > > > > requiring you to configure your block devices with a replay driver on > > > > > top of the stack seems reasonable enough. > > > > > > > > I cannot use block drivers for this. When driver functions are used, QEMU > > > > is already used coroutines (and probably started bottom halves). > > > > Coroutines make execution non-deterministic. > > > > That's why we have to intercept blk_aio_ functions, that are called > > > > deterministically. > > > > > > What does "deterministic" mean in this context, i.e. what are your exact > > > requirements? > > > > "Deterministic" means that the replayed execution should run exactly > > the same guest instructions in the same sequence, as in recording session. > > Okay. I think with this we can do better than what you have now. > > > > I don't think that coroutines introduce anything non-deterministic per > > > se. Depending on what you mean by it, the block layer code paths in > > > block.c may contain problematic code. > > > > They are non-deterministic if we need instruction-level accuracy. > > Thread switching (and therefore callbacks and BH execution) is non-deterministic. > > Thread switching depends on an external event (the kernel scheduler > deciding to switch), so agreed, if a thread switch ever influences what > the guest sees, that would be a problem. > > Generally, however, callbacks and BHs don't involve a thread switch at > all (BHs can be invoked from a different thread in theory, but we have > very few of those cases and they shouldn't be visible for the guest). > The same is true for coroutines, which are semantically equivalent to > callbacks. > > > In two different executions these callbacks may happen at different moments of > > time (counting in number of executed instructions). > > All operations with virtual devices (including memory, interrupt controller, > > and disk drive controller) should happen at deterministic moments of time > > to be replayable. > > Right, so let's talk about what this external non-deterministic event > really is. > > I think the only thing whose timing is unknown in the block layer is the > completion of I/O requests. This non-determinism comes from the time the > I/O syscalls made by the lowest layer (usually raw-posix) take. Right. > This means that we can add logic to remove the non-determinism at the > point of our choice between raw-posix and the guest device emulation. A > block driver on top is as good as anything else. > > While recording, this block driver would just pass the request to next > lower layer (starting a request is deterministic, so it doesn't need to > be logged) and once the request completes it logs it. While replaying, > the completion of requests is delayed until we read it in the log; if we > read it in the log and the request hasn't completed yet, we do a busy > wait for it (while(!completed) aio_poll();). I tried serializing all bottom halves and worker thread callbacks in previous version of the patches. That code was much more complicated and error-prone than the current version. We had to classify all bottom halves to recorded and non-recorded (because sometimes they are used for qemu's purposes, not the guest ones). However, I don't understand yet which layer do you offer as the candidate for record/replay? What functions should be changed? I would like to investigate this way, but I don't got it yet. > This model would get rid of the bdrv_drain_all() that you call > everywhere and therefore allow concurrent requests, giving a result that > is much closer to the "normal" behaviour without replay. > > > > The block layer uses bottom halves in some cases for request completion, > > > but not until calling into the first driver (why would they be a > > > problem?). What could happen is that a request is serialised and > > > therefore delayed in some special configurations, which sounds a lot > > > like what you wanted to avoid. > > > > Drivers cannot distinguish the requests from guest CPU and from savevm/loadvm. > > First ones have to be deterministic, because they affect guest memory, > > virtual disk controller, and interrupts. > > Sure they can, these are two different callbacks. But even if they > couldn't, making more things than necessary deterministic might be > wasteful, but not really harmful. Is there any universal way to check this? Pavel Dovgalyuk