Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from ws5-mx01.kavi.com (ws5-mx01.kavi.com [34.193.7.191]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D3CB4C4332F for ; Tue, 31 Oct 2023 09:47:54 +0000 (UTC) Received: from lists.oasis-open.org (oasis.ws5.connectedcommunity.org [10.110.1.242]) by ws5-mx01.kavi.com (Postfix) with ESMTP id 4F518749ED for ; Tue, 31 Oct 2023 09:47:54 +0000 (UTC) Received: from lists.oasis-open.org (oasis-open.org [10.110.1.242]) by lists.oasis-open.org (Postfix) with ESMTP id 31410986B6B for ; Tue, 31 Oct 2023 09:47:54 +0000 (UTC) Received: from host09.ws5.connectedcommunity.org (host09.ws5.connectedcommunity.org [10.110.1.97]) by lists.oasis-open.org (Postfix) with QMQP id 1E1E2986B68; Tue, 31 Oct 2023 09:47:54 +0000 (UTC) Mailing-List: contact virtio-comment-help@lists.oasis-open.org; run by ezmlm List-ID: Sender: Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: Received: from lists.oasis-open.org (oasis-open.org [10.110.1.242]) by lists.oasis-open.org (Postfix) with ESMTP id 0EAC0986B69 for ; Tue, 31 Oct 2023 09:47:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at kavi.com X-IronPort-AV: E=McAfee;i="6600,9927,10879"; a="419363900" X-IronPort-AV: E=Sophos;i="6.03,265,1694761200"; d="scan'208";a="419363900" X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10879"; a="830992713" X-IronPort-AV: E=Sophos;i="6.03,265,1694761200"; d="scan'208";a="830992713" Message-ID: <2de53b77-ad90-48a8-b004-9b1884acc385@intel.com> Date: Tue, 31 Oct 2023 17:47:46 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: "Michael S. Tsirkin" Cc: Parav Pandit , Jason Wang , "virtio-comment@lists.oasis-open.org" , "cohuck@redhat.com" , "sburla@marvell.com" , Shahaf Shuler , Maor Gottlieb , Yishai Hadas References: <20231030131959.1386107-1-parav@nvidia.com> <20231030131959.1386107-7-parav@nvidia.com> <20231031034117-mutt-send-email-mst@kernel.org> <20231031053926-mutt-send-email-mst@kernel.org> From: "Zhu, Lingshan" In-Reply-To: <20231031053926-mutt-send-email-mst@kernel.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands On 10/31/2023 5:41 PM, Michael S. Tsirkin wrote: > On Tue, Oct 31, 2023 at 05:32:01PM +0800, Zhu, Lingshan wrote: >> >> On 10/31/2023 3:45 PM, Michael S. Tsirkin wrote: >>> On Tue, Oct 31, 2023 at 03:27:12AM +0000, Parav Pandit wrote: >>>>> From: Jason Wang >>>>> Sent: Tuesday, October 31, 2023 7:13 AM >>>>> >>>>> On Mon, Oct 30, 2023 at 9:21 PM Parav Pandit wrote: >>>>>> During a device migration flow (typically in a precopy phase of the >>>>>> live migration), a device may write to the guest memory. Some >>>>>> iommu/hypervisor may not be able to track these written pages. >>>>>> These pages to be migrated from source to destination hypervisor. >>>>>> >>>>>> A device which writes to these pages, provides the page address record >>>>>> of the to the owner device. The owner device starts write recording >>>>>> for the device and queries all the page addresses written by the >>>>>> device. >>>>>> >>>>>> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/176 >>>>>> Signed-off-by: Parav Pandit >>>>>> Signed-off-by: Satananda Burla >>>>>> --- >>>>>> changelog: >>>>>> v1->v2: >>>>>> - addressed comments from Michael >>>>>> - replaced iova with physical address >>>>>> --- >>>>>> admin-cmds-device-migration.tex | 15 +++++++++++++++ >>>>>> 1 file changed, 15 insertions(+) >>>>>> >>>>>> diff --git a/admin-cmds-device-migration.tex >>>>>> b/admin-cmds-device-migration.tex index ed911e4..2e32f2c 100644 >>>>>> --- a/admin-cmds-device-migration.tex >>>>>> +++ b/admin-cmds-device-migration.tex >>>>>> @@ -95,6 +95,21 @@ \subsubsection{Device Migration}\label{sec:Basic >>>>>> Facilities of a Virtio Device / The owner driver can discard any >>>>>> partially read or written device context when any of the device migration flow >>>>> should be aborted. >>>>>> +During the device migration flow, a passthrough device may write data >>>>>> +to the guest virtual machine's memory, a source hypervisor needs to >>>>>> +keep track of these written memory to migrate such memory to destination >>>>> hypervisor. >>>>>> +Some systems may not be able to keep track of such memory write >>>>>> +addresses at hypervisor level. In such a scenario, a device records >>>>>> +and reports these written memory addresses to the owner device. The >>>>>> +owner driver enables write recording for one or more physical address >>>>>> +ranges per device during device migration flow. The owner driver >>>>>> +periodically queries these written physical address records from the device. >>>>> I wonder how PA works in this case. Device uses untranslated requests so it can >>>>> only see IOVA. We can't mandate ATS anyhow. >>>> Michael suggested to keep the language uniform as PA as this is ultimately what the guest driver is supplying during vq creation and in posting buffers as physical address. >>> Yes the spec calls the address accessed by the device "physical >>> address". Granted, this is pointless - there is only one >>> type of address device can access. We can if we want to >>> replace that with just "address" or "memory address". >>> I don't think this ever caused confusion though and worth the >>> churn. >> But you know PA means disable IOMMU and always enable ATS on both the device >> and host > That is not how virtio uses the term. Just grep "physical address" in the spec. > This patch should be consistent. OK, I use other terms in my series, let's see whether it is better. > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/