Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DCB86C4338F for ; Tue, 10 Aug 2021 16:49:35 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 846E460249 for ; Tue, 10 Aug 2021 16:49:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 846E460249 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.165442.302319 (Exim 4.92) (envelope-from ) id 1mDUw8-0004YF-P3; Tue, 10 Aug 2021 16:49:08 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 165442.302319; Tue, 10 Aug 2021 16:49:08 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mDUw8-0004Y8-Ly; Tue, 10 Aug 2021 16:49:08 +0000 Received: by outflank-mailman (input) for mailman id 165442; Tue, 10 Aug 2021 16:49:07 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mDUw7-0004Y2-EN for xen-devel@lists.xenproject.org; Tue, 10 Aug 2021 16:49:07 +0000 Received: from mail-lj1-x22f.google.com (unknown [2a00:1450:4864:20::22f]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 07f3066e-101b-4d18-a509-c849e7352def; Tue, 10 Aug 2021 16:49:06 +0000 (UTC) Received: by mail-lj1-x22f.google.com with SMTP id y7so19137419ljp.3 for ; Tue, 10 Aug 2021 09:49:06 -0700 (PDT) Received: from [192.168.1.7] ([212.22.223.21]) by smtp.gmail.com with ESMTPSA id y20sm363668lji.126.2021.08.10.09.49.03 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 10 Aug 2021 09:49:04 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 07f3066e-101b-4d18-a509-c849e7352def DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=y/6dCXP7pKem3jwxJV+oqakShfCnHcGFD7xZafWyuBM=; b=eIvzUjur+rcqbRVsRSFeTme7cvC7M4w4Bd0rRzL/tXvKJxLGVGQoMIYGjIGgC2wTNV Zgnz01/Jf+lolTIToAx83hZMaWB/DWRKljVLR7TpVEmAwiZktSEVpEWRkPNU9RgmGJbj P/7ZxS5HuH6WTRs9oQ9zeunLzFFflk92bBoBOlWw1kwrNv1EP+YBUZW52fDm9JOyJkpz S9JGmulK1mujVJ2EIVFCEmOO4rUSYEPVj/AWMKSRRhpJuazrx7fWDRURgO2iVQd8mgIR x9MNt0y4wNrpGXvfXLx530/5bpDLebY7Oak4PAlQHliVs3Dv3d7Fqe0wuam8+lp15w78 AbeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=y/6dCXP7pKem3jwxJV+oqakShfCnHcGFD7xZafWyuBM=; b=clq0NGkVAOimuWA7kSfoIyoFedP10jg5rjiFuMDxUvFkACKANDuGuiwCCq4+JvxUjo n7y3msoMzQnI+5gWYHwhn7DRNrpooniXPg9tj+exPC5r4c/ba20umCnO63RPe+CqBmq+ QQx2LmcID3hM0ydsPi+fcgnib+ryUGsp47sWk4Topn6rLhkerDOqo1JZxdAaJXa1djk6 yDzqLHTRZywOoVnOH5WHNSEERCJ2i5mbMkj2z6dASyk8urn/T+fQejl83E4JpBbwlB9g amkHCcGmBsBEbtI3PofHfok7klCNhZyOWvOP8lpyzvD3iW8BP7oZ1HzFjtilyLp47D0t MRMA== X-Gm-Message-State: AOAM532lMS7B3WuxEnMW/SNah0FbrxF8kiG8TqMUHQflkSwF6bEmAHtV jYPJ5aZiLlt0+3WRUSJrJkY= X-Google-Smtp-Source: ABdhPJwK3Euu6EJgaGFjkZA5D7wjR+rwskw/4N7q5o0MgxPMeu0uqD0zZRA9qWld9J+qPlyYNnHjnw== X-Received: by 2002:a2e:bc19:: with SMTP id b25mr18875081ljf.374.1628614145047; Tue, 10 Aug 2021 09:49:05 -0700 (PDT) Subject: Re: [RFC PATCH] xen/memory: Introduce a hypercall to provide unallocated space To: Julien Grall Cc: Wei Chen , Penny Zheng , Stefano Stabellini , Andrew Cooper , "xen-devel@lists.xenproject.org" , Oleksandr Tyshchenko , Daniel De Graaf , "Daniel P. Smith" , Ian Jackson , Wei Liu , George Dunlap , Jan Beulich , Volodymyr Babchuk , =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= , Bertrand Marquis References: <1627489110-25633-1-git-send-email-olekstysh@gmail.com> <80fafc01-f063-d6e5-1c08-7ad64550310c@citrix.com> <4de5ed21-379e-b618-44c8-924d88b1a519@citrix.com> <6a633f4e-13e0-4a2b-cf6e-35ef90ae977c@gmail.com> <5643d414-0b76-74a4-2c37-c7a99338d547@gmail.com> <6596ad08-8398-64dd-ef62-cd7bc6f7333e@gmail.com> <1d0ea55d-2e5a-daa7-9c60-c7a1c4b48fa1@gmail.com> <0ddb516b-1990-b772-6163-3e0d3af2c635@gmail.com> From: Oleksandr Message-ID: <005a02cb-b400-5998-5f57-3422ba7091e8@gmail.com> Date: Tue, 10 Aug 2021 19:49:03 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US On 10.08.21 19:21, Julien Grall wrote: Hi Julien > > > On 10/08/2021 12:58, Oleksandr wrote: >> >> On 10.08.21 09:34, Wei Chen wrote: >>> Hi Oleksandr, >> >> Hi Wei, Julien. > Hi, > >>>> -----Original Message----- >>>> From: Oleksandr >>>> Sent: 2021年8月10日 2:25 >>>> To: Julien Grall >>>> Cc: Stefano Stabellini ; Andrew Cooper >>>> ; xen-devel@lists.xenproject.org; Oleksandr >>>> Tyshchenko ; Daniel De Graaf >>>> ; Daniel P. Smith >>>> ; >>>> Ian Jackson ; Wei Liu ; George Dunlap >>>> ; Jan Beulich ; Volodymyr >>>> Babchuk ; Roger Pau Monné >>>> ; Bertrand Marquis >>>> ; Wei >>>> Chen >>>> Subject: Re: [RFC PATCH] xen/memory: Introduce a hypercall to provide >>>> unallocated space >>>> >>>> >>>> On 09.08.21 18:42, Julien Grall wrote: >>>>> Hi Oleksandr, >>>> >>>> Hi Julien. >>>> >>>> >>>> Thank you for the input. >>>> >>>> >>>>> On 07/08/2021 18:03, Oleksandr wrote: >>>>>> On 06.08.21 03:30, Stefano Stabellini wrote: >>>>>> >>>>>> Hi Stefano >>>>>> >>>>>>> On Wed, 4 Aug 2021, Julien Grall wrote: >>>>>>>>> +#define GUEST_SAFE_RANGE_BASE xen_mk_ullong(0xDE00000000) /* >>>>>>>>> 128GB */ >>>>>>>>> +#define GUEST_SAFE_RANGE_SIZE xen_mk_ullong(0x0200000000) >>>>>>>>> >>>>>>>>> While the possible new DT bindings has not been agreed yet, I re- >>>> used >>>>>>>>> existing "reg" property under the hypervisor node to pass safe >>>>>>>>> range as a >>>>>>>>> second region, >>>>>>>>> https://elixir.bootlin.com/linux/v5.14- >>>> rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10: >>>>>>>> So a single region works for a guest today, but for dom0 we will >>>>>>>> need multiple >>>>>>>> regions because it is may be difficult to find enough contiguous >>>>>>>> space for a >>>>>>>> single region. >>>>>>>> >>>>>>>> That said, as dom0 is mapped 1:1 (including some guest mapping), >>>>>>>> there is also >>>>>>>> the question where to allocate the safe region. For grant >>>>>>>> table, we >>>>>>>> so far >>>>>>>> re-use the Xen address space because it is assumed it will space >>>>>>>> will always >>>>>>>> be bigger than the grant table. >>>>>>>> >>>>>>>> I am not sure yet where we could allocate the safe regions. >>>>>>>> Stefano, do you >>>>>>>> have any ideas? >>>>>>> The safest choice would be the address range corresponding to >>>>>>> memory >>>>>>> (/memory) not already allocated to Dom0. >>>>>>> >>>>>>> For instance from my last boot logs: >>>>>>> (XEN) Allocating 1:1 mappings totalling 1600MB for dom0: >>>>>>> (XEN) BANK[0] 0x00000010000000-0x00000070000000 (1536MB) >>>>>>> (XEN) BANK[1] 0x00000078000000-0x0000007c000000 (64MB) >>>>>>> >>>>>>> All the other ranges could be given as unallocated space: >>>>>>> >>>>>>> - 0x0 - 0x10000000 >>>>>>> - 0x70000000 - 0x78000000 >>>>>>> - 0x8_0000_0000 - 0x8_8000_0000 >>>>>> Thank you for the ideas. >>>>>> >>>>>> If I got the idea correctly, yes, as these ranges represent the real >>>>>> RAM, so no I/O would be in conflict with them and as the result - no >>>>>> overlaps would be expected. >>>>>> But, I wonder, would this work if we have IOMMU enabled for Dom0 and >>>>>> need to establish 1:1 mapping for the DMA devices to work with grant >>>>>> mappings... >>>>>> In arm_iommu_map_page() we call guest_physmap_add_entry() with gfn = >>>>>> mfn, so the question is could we end up with this new gfn replacing >>>>>> the valid mapping >>>>>> (with gfn allocated from the safe region)? >>>>> Right, when we enable the IOMMU for dom0, Xen will add an extra >>>>> mapping with GFN == MFN for foreign and grant pages. This is because >>>>> Linux is not aware that whether a device is protected by an IOMMU. >>>>> Therefore it is assuming it is not and will use the MFN to configure >>>>> for DMA transaction. >>>>> >>>>> We can't remove the mapping without significant changes in Linux and >>>>> Xen. I would not mandate them for this work. >>>>> >>>>> That said, I think it would be acceptable to have different way to >>>>> find the region depending on the dom0 configuration. So we could use >>>>> the RAM not used by dom0 when the IOMMU is turned off. >>>> OK >>>> >>>> >>>>>>> The second best choice would be an hole: an address range not >>>>>>> used by >>>>>>> anybody else (no reg property) and also not even mappable by a bus >>>> (not >>>>>>> covered by a ranges property). This is not the best choice because >>>>>>> there >>>>>>> can cases where physical resources appear afterwards. >>>>> Are you saying that the original device-tree doesn't even describe >>>>> them in any way (i.e. reserved...)? >>>>> >>>>>> Unfortunately, yes. >>>>> So the decision where the safe region is located will be done by Xen. >>>>> There is no involvement of the domain (it will discover the region >>>>> from the DT). Therefore, I don't think we need to think about >>>>> everything right now as we could adapt this is exact region is not >>>>> part of the stable ABI. >>>>> >>>>> The hotplug is one I would defer because this is not supported (and >>>>> quite likely not working) in Xen upstream today. >>>> Sounds reasonable. >>>> >>>> >>>>> >>>>> Now regarding the case where dom0 is using the IOMMU. The assumption >>>>> is Xen will be able to figure out all the regions used from the >>>>> firmware table (ACPI or DT). >>>>> >>>>> AFAIK, this assumption would be correct for DT. However, for ACPI, I >>>>> remember we were not able to find all the MMIOs region in Xen (see >>>>> [1] >>>>> and [2]). So even this solution would not work for ACPI. >>>>> >>>>> If I am not mistaken, we don't support IOMMU with ACPI yet. So we >>>>> could defer the problem to when this is going to be supported. >>>> Sounds reasonable. >>>> >>>> >>>> To summarize: >>>> >>>> 0. Skip ACPI case for now, implement for DT case >>>> >>>> 1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as >>>> safe ranges >>>> >>> Does static allocation and direct mapping driver domain can be treated >>> as this case? >> I am not sure I can answer this question correctly due to the limited >> knowledge of these features. >> >> But, it feels to me that holes solution would work, at least I don't >> see why not. >> >> >> I wonder, why these can't be treated as the case #2 (where we provide >> not assigned RAM), I also don't see why not, however there might be >> pitfalls with direct mapped driver domain. Julien, do you have any >> opinion on this? > > So whether the memory is statically allocated or dynamically allocated > should not matter here. For direct mapped domain, then they should be > treated the same way as dom0. > > By that I mean if the IOMMU is not enabled for the domain, then we can > use thew unallocated RAM. Otherwise, we would need to find some holes. OK, thank you for the clarification. > > > Cheers, > -- Regards, Oleksandr Tyshchenko