Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D22FEC4338F for ; Tue, 10 Aug 2021 16:21:28 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7771360C51 for ; Tue, 10 Aug 2021 16:21:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7771360C51 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=xen.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.165430.302297 (Exim 4.92) (envelope-from ) id 1mDUV9-0001YD-8u; Tue, 10 Aug 2021 16:21:15 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 165430.302297; Tue, 10 Aug 2021 16:21:15 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mDUV9-0001Y6-5c; Tue, 10 Aug 2021 16:21:15 +0000 Received: by outflank-mailman (input) for mailman id 165430; Tue, 10 Aug 2021 16:21:14 +0000 Received: from mail.xenproject.org ([104.130.215.37]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mDUV8-0001Y0-JA for xen-devel@lists.xenproject.org; Tue, 10 Aug 2021 16:21:14 +0000 Received: from xenbits.xenproject.org ([104.239.192.120]) by mail.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mDUV2-0001QM-Vn; Tue, 10 Aug 2021 16:21:08 +0000 Received: from [54.239.6.184] (helo=a483e7b01a66.ant.amazon.com) by xenbits.xenproject.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1mDUV2-00030V-Oh; Tue, 10 Aug 2021 16:21:08 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=xen.org; s=20200302mail; h=Content-Transfer-Encoding:Content-Type:In-Reply-To: MIME-Version:Date:Message-ID:From:References:Cc:To:Subject; bh=NnsOM/TCeJzWZt7nRIYPEeraEMU0KBqS/XPi6ZA3Wuk=; b=cAxQOKDIu4ng52UBiHC8bZkYyV spF25GUkKwtIMRNKtnHIjia4TIcJitTvEm4t6Wv5WSK3bFTWe/wcAkgVSR1GTjYKsPd6b7c88MCoy NoFR3Dk5yMM5n3Go1VD6E4sd0GPBZk2/SDWVrDLwVLw1BaCertVbvuvpD+SvAM48xopU=; Subject: Re: [RFC PATCH] xen/memory: Introduce a hypercall to provide unallocated space To: Oleksandr , Wei Chen Cc: Penny Zheng , Stefano Stabellini , Andrew Cooper , "xen-devel@lists.xenproject.org" , Oleksandr Tyshchenko , Daniel De Graaf , "Daniel P. Smith" , Ian Jackson , Wei Liu , George Dunlap , Jan Beulich , Volodymyr Babchuk , =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= , Bertrand Marquis References: <1627489110-25633-1-git-send-email-olekstysh@gmail.com> <80fafc01-f063-d6e5-1c08-7ad64550310c@citrix.com> <4de5ed21-379e-b618-44c8-924d88b1a519@citrix.com> <6a633f4e-13e0-4a2b-cf6e-35ef90ae977c@gmail.com> <5643d414-0b76-74a4-2c37-c7a99338d547@gmail.com> <6596ad08-8398-64dd-ef62-cd7bc6f7333e@gmail.com> <1d0ea55d-2e5a-daa7-9c60-c7a1c4b48fa1@gmail.com> <0ddb516b-1990-b772-6163-3e0d3af2c635@gmail.com> From: Julien Grall Message-ID: Date: Tue, 10 Aug 2021 17:21:05 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 MIME-Version: 1.0 In-Reply-To: <0ddb516b-1990-b772-6163-3e0d3af2c635@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 8bit On 10/08/2021 12:58, Oleksandr wrote: > > On 10.08.21 09:34, Wei Chen wrote: >> Hi Oleksandr, > > Hi Wei, Julien. Hi, >>> -----Original Message----- >>> From: Oleksandr >>> Sent: 2021年8月10日 2:25 >>> To: Julien Grall >>> Cc: Stefano Stabellini ; Andrew Cooper >>> ; xen-devel@lists.xenproject.org; Oleksandr >>> Tyshchenko ; Daniel De Graaf >>> ; Daniel P. Smith ; >>> Ian Jackson ; Wei Liu ; George Dunlap >>> ; Jan Beulich ; Volodymyr >>> Babchuk ; Roger Pau Monné >>> ; Bertrand Marquis ; Wei >>> Chen >>> Subject: Re: [RFC PATCH] xen/memory: Introduce a hypercall to provide >>> unallocated space >>> >>> >>> On 09.08.21 18:42, Julien Grall wrote: >>>> Hi Oleksandr, >>> >>> Hi Julien. >>> >>> >>> Thank you for the input. >>> >>> >>>> On 07/08/2021 18:03, Oleksandr wrote: >>>>> On 06.08.21 03:30, Stefano Stabellini wrote: >>>>> >>>>> Hi Stefano >>>>> >>>>>> On Wed, 4 Aug 2021, Julien Grall wrote: >>>>>>>> +#define GUEST_SAFE_RANGE_BASE xen_mk_ullong(0xDE00000000) /* >>>>>>>> 128GB */ >>>>>>>> +#define GUEST_SAFE_RANGE_SIZE xen_mk_ullong(0x0200000000) >>>>>>>> >>>>>>>> While the possible new DT bindings has not been agreed yet, I re- >>> used >>>>>>>> existing "reg" property under the hypervisor node to pass safe >>>>>>>> range as a >>>>>>>> second region, >>>>>>>> https://elixir.bootlin.com/linux/v5.14- >>> rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10: >>>>>>> So a single region works for a guest today, but for dom0 we will >>>>>>> need multiple >>>>>>> regions because it is may be difficult to find enough contiguous >>>>>>> space for a >>>>>>> single region. >>>>>>> >>>>>>> That said, as dom0 is mapped 1:1 (including some guest mapping), >>>>>>> there is also >>>>>>> the question where to allocate the safe region. For grant table, we >>>>>>> so far >>>>>>> re-use the Xen address space because it is assumed it will space >>>>>>> will always >>>>>>> be bigger than the grant table. >>>>>>> >>>>>>> I am not sure yet where we could allocate the safe regions. >>>>>>> Stefano, do you >>>>>>> have any ideas? >>>>>> The safest choice would be the address range corresponding to memory >>>>>> (/memory) not already allocated to Dom0. >>>>>> >>>>>> For instance from my last boot logs: >>>>>> (XEN) Allocating 1:1 mappings totalling 1600MB for dom0: >>>>>> (XEN) BANK[0] 0x00000010000000-0x00000070000000 (1536MB) >>>>>> (XEN) BANK[1] 0x00000078000000-0x0000007c000000 (64MB) >>>>>> >>>>>> All the other ranges could be given as unallocated space: >>>>>> >>>>>> - 0x0 - 0x10000000 >>>>>> - 0x70000000 - 0x78000000 >>>>>> - 0x8_0000_0000 - 0x8_8000_0000 >>>>> Thank you for the ideas. >>>>> >>>>> If I got the idea correctly, yes, as these ranges represent the real >>>>> RAM, so no I/O would be in conflict with them and as the result - no >>>>> overlaps would be expected. >>>>> But, I wonder, would this work if we have IOMMU enabled for Dom0 and >>>>> need to establish 1:1 mapping for the DMA devices to work with grant >>>>> mappings... >>>>> In arm_iommu_map_page() we call guest_physmap_add_entry() with gfn = >>>>> mfn, so the question is could we end up with this new gfn replacing >>>>> the valid mapping >>>>> (with gfn allocated from the safe region)? >>>> Right, when we enable the IOMMU for dom0, Xen will add an extra >>>> mapping with GFN == MFN for foreign and grant pages. This is because >>>> Linux is not aware that whether a device is protected by an IOMMU. >>>> Therefore it is assuming it is not and will use the MFN to configure >>>> for DMA transaction. >>>> >>>> We can't remove the mapping without significant changes in Linux and >>>> Xen. I would not mandate them for this work. >>>> >>>> That said, I think it would be acceptable to have different way to >>>> find the region depending on the dom0 configuration. So we could use >>>> the RAM not used by dom0 when the IOMMU is turned off. >>> OK >>> >>> >>>>>> The second best choice would be an hole: an address range not used by >>>>>> anybody else (no reg property) and also not even mappable by a bus >>> (not >>>>>> covered by a ranges property). This is not the best choice because >>>>>> there >>>>>> can cases where physical resources appear afterwards. >>>> Are you saying that the original device-tree doesn't even describe >>>> them in any way (i.e. reserved...)? >>>> >>>>> Unfortunately, yes. >>>> So the decision where the safe region is located will be done by Xen. >>>> There is no involvement of the domain (it will discover the region >>>> from the DT). Therefore, I don't think we need to think about >>>> everything right now as we could adapt this is exact region is not >>>> part of the stable ABI. >>>> >>>> The hotplug is one I would defer because this is not supported (and >>>> quite likely not working) in Xen upstream today. >>> Sounds reasonable. >>> >>> >>>> >>>> Now regarding the case where dom0 is using the IOMMU. The assumption >>>> is Xen will be able to figure out all the regions used from the >>>> firmware table (ACPI or DT). >>>> >>>> AFAIK, this assumption would be correct for DT. However, for ACPI, I >>>> remember we were not able to find all the MMIOs region in Xen (see [1] >>>> and [2]). So even this solution would not work for ACPI. >>>> >>>> If I am not mistaken, we don't support IOMMU with ACPI yet. So we >>>> could defer the problem to when this is going to be supported. >>> Sounds reasonable. >>> >>> >>> To summarize: >>> >>> 0. Skip ACPI case for now, implement for DT case >>> >>> 1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as >>> safe ranges >>> >> Does static allocation and direct mapping driver domain can be treated >> as this case? > I am not sure I can answer this question correctly due to the limited > knowledge of these features. > > But, it feels to me that holes solution would work, at least I don't see > why not. > > > I wonder, why these can't be treated as the case #2 (where we provide > not assigned RAM), I also don't see why not, however there might be > pitfalls with direct mapped driver domain. Julien, do you have any > opinion on this? So whether the memory is statically allocated or dynamically allocated should not matter here. For direct mapped domain, then they should be treated the same way as dom0. By that I mean if the IOMMU is not enabled for the domain, then we can use thew unallocated RAM. Otherwise, we would need to find some holes. Cheers, -- Julien Grall