Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E2E6EC43217 for ; Thu, 1 Dec 2022 16:17:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:From: References:Cc:To:Subject:Date:Message-ID:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=pOVeDzWFVfKtGpuOHbM12rK/hO/eXoKdx2aYcG1HhXM=; b=XycGXxwTp+PJKDw+4x3SNEgSla eO2tmwdbUG9obXgmrTiu3OUg+Yodl1naRdoY/Qd41/Pf/qLjV0xeQwtrKr2QkBIbOJfyrPTzyU4x7 mYStVhAK75/ttV9YD0f/prnaCLqL2OzgCCJDdrjyzAi8xt8dqPt84WonAwC6kUkSz9muap+7OkLlM rCAocNMJ8oTB9hmd5SeHnopSUQZvDX5gq0dmyz12++yOaeok3qk8t8EHH8Iztipe6uSwNAxtQ3nYf d+24i6RoynIIs2zfjCK0axGTCskvIteApkMzE5zfR0MZlHlBoKL1EDsTWRUGkUAOTE45rQOcu/GQm f6fwHz+Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p0mEQ-008UW6-UC; Thu, 01 Dec 2022 16:16:15 +0000 Received: from mail-ve1eur01on0630.outbound.protection.outlook.com ([2a01:111:f400:fe1f::630] helo=EUR01-VE1-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p0mEN-008UVE-SX for linux-arm-kernel@lists.infradead.org; Thu, 01 Dec 2022 16:16:13 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ywc9rwp5/7E/M+FRCS5BrXmYUMs2I4Tf6vAtEO2+KcGk6YtYOBFstvfneMGwDK0wSMHhZglLMEXiKwWvVy9tHB3UiIojk6zD4EFyTAg3k6p3c2uBqpahHsUoBnQ4VmGSd+Ym2e6XunySlLibFkzHRVqjWtpL8GSepMx4iR36k3xNJO9KsN52jhlPPW8t1nZfgcsn1gY4naas25wmX6qDj4Kf++5znfnKnrafSadwNpL4f0KmzyR+erINmqMvb2YdcOmZi0qi9eJttMnG7i9jjzz2Didm2+OEoxKtGfqTY3MZPHyCBGIVq7hwcLYRkOck0diwFnYAfLaQoiePLfpQIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QSRAxShsjgZ4Q3rY2+iywCJAc1DCiSiqHOC4VqCOq4Y=; b=PNkT9MU/2K0rozY1zauqxyPVjN5vnY4EBpWJaXUFG52BcchUdfLRaWhbORWbYFanYLmR1NNHX6GfQSf6RTa7BxiKYYfT11VWcwSSuOMot/MN7rZJbqbfoxqQAzniuqKGAAHX4JrS76lKf+9ycvEEGosctdR9Xx6JHXtUcGPHtzBCKZjk3RFBNLeMr57ccoE3wStlgRYADakV125WVpT+wLU6izsa8LsJ3piKfWcYStedfIGGHTab0pLbZGfFQKtpNmsSSsySVgUN/u2GD9qYZftnBX3g/SQAHzUO3/vzexqul+eYIUbZbQNgpSJTUKotUOxYfZqYnMc+KrQbVeXQNw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QSRAxShsjgZ4Q3rY2+iywCJAc1DCiSiqHOC4VqCOq4Y=; b=sokvdac9nP07+nulZv1zb3JF81fAkh/0AaxDu0jvgzAInf0tQJPK1f2weglfZelPN0NiQZLMZBz1yaS6NUgcOaI2PalDwkOgko3lu0Ja7InUIP4Wyjigv7QpaujJTrVw8vYMRyX74ONk8czbOCMHUw4v77G0sAjec42rwRPm1QY= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; Received: from AS8PR08MB6995.eurprd08.prod.outlook.com (2603:10a6:20b:34d::13) by PAVPR08MB9556.eurprd08.prod.outlook.com (2603:10a6:102:311::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5880.8; Thu, 1 Dec 2022 16:00:16 +0000 Received: from AS8PR08MB6995.eurprd08.prod.outlook.com ([fe80::e825:ff01:1f8:4c1f]) by AS8PR08MB6995.eurprd08.prod.outlook.com ([fe80::e825:ff01:1f8:4c1f%4]) with mapi id 15.20.5880.008; Thu, 1 Dec 2022 16:00:15 +0000 Message-ID: Date: Thu, 1 Dec 2022 16:00:13 +0000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.5.0 Subject: Re: [PATCH v2 00/19] arm64: Enable LPA2 support for 4k and 16k pages Content-Language: en-US To: Ard Biesheuvel Cc: linux-arm-kernel@lists.infradead.org, Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual , Richard Henderson References: <20221124123932.2648991-1-ardb@kernel.org> <053a9129-2424-e1fe-0e7d-b410984b7bfa@arm.com> <498f586c-49f4-3fd9-06c5-a32b7773e516@arm.com> From: Ryan Roberts In-Reply-To: X-ClientProxiedBy: LO4P123CA0343.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:18d::6) To AS8PR08MB6995.eurprd08.prod.outlook.com (2603:10a6:20b:34d::13) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS8PR08MB6995:EE_|PAVPR08MB9556:EE_ X-MS-Office365-Filtering-Correlation-Id: 49178676-2758-4cb4-d7d7-08dad3b522d1 NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: RxjoHWHKOtUXeeejvlA2sd9cwHnVnOszsJSE1+XN25UaT1Tgf2Iw9uWSGUIOZ1gbZhuaNHOjJYpxmV+PLHa8wxxq9RwvjEvsfIciBStTVz1Oer0/51fFSjY9J90xrpcB+OIhhfvFhtthdOVCiV19HkIwyNPDpAkuwkGmsHpgV6iCLXMEa/OsbwlGt5aNPgioGTBQJQjKHoJN8+ZeKclvyomGRfiPP7kzSec0yaUfzlbvaPmKUPD5yIAsPdpm8CcVFXaLUjynp8uVsfJjK0q9hVDIiGZhKUkYUID/f6RZQT7MnrII/5j5uHN9/mBzL6QByBzHCDTGxFJ3MZHSYQgPhj/Mvbxerb0PMzanQFTpQ3G0RAyy4xu0SyJWzEZdaVjuQT4gFTr5CxhofmbdTxbkNLhxAANCWRlM8qKzj3zZbRdqrox+L+VmOGQ8BIcbnOM9u8uNpiALO/wmwP+l8PO0Z8Euw07TxDhIiYj7fFjLbW4wG8n/G5da+vlB4LPu6jrljsp33GlTGS6i+gQjagkWNBhMnEf+8t7ne7Q69PPMlCainbQdPHWbZ2FV38u0+kZ8Hsah5VTcBcJ8o1XlzKpWAWrNQGTFTmaRODrcxgsgQ/X4YPtVRTwKqT2LNK+Zq56khywIzc+cs/X7OIX2P1MQ2GRMAtMTDIGFcoNdRGfCDpmVoP1s5tsUvNShXSnBDySuXXopAJvehi7laXJZEylOToU4xnoUl1LYSvDjTaLfxeY= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS8PR08MB6995.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(346002)(396003)(136003)(366004)(39860400002)(376002)(451199015)(31686004)(2906002)(6506007)(6512007)(478600001)(31696002)(86362001)(6486002)(36756003)(41300700001)(2616005)(186003)(8676002)(38100700002)(83380400001)(26005)(44832011)(4326008)(53546011)(5660300002)(8936002)(66476007)(316002)(6916009)(54906003)(66556008)(66946007)(45980500001)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TFlGNTBaZWFFVi8zdGVaWkFGT3lzTHZabEJxNW5TbDRzbXd5aVdwMlVvTURy?= =?utf-8?B?NWo2MEd6eTNaV0lLMjBQYXV4RktlTkk3cGN2TUxWem5JTlZmUEF5aUxkdkpm?= =?utf-8?B?ZDB0KytxZjN5NzlrMXFxZjhSZDhIYXNLbHdJNm1mdlRvQkY2M2lNUkFTOWxL?= =?utf-8?B?ZGVpVCtxcnR1bTZmY2tMSUNnVGZPdVhobUZpSnBEbGxGZlp1eXRIZzB0eURX?= =?utf-8?B?cG5xMEwwVFF3YmJWZkFHSVJNWVdNR0ttQ2c0TWFwRWFXMUxaOGg1ejJhVGxV?= =?utf-8?B?U001N2h6aFpxMlU0QXJ6R0tIUjBoN3ZvQ1prNlErT3k0SUpMZzFQamVKN2V4?= =?utf-8?B?UitsenpTdWVxajBsZkV0SE1OUStyaWMzQUNDY2xWVFFrcHpNSEUrYUQ3QjJP?= =?utf-8?B?RGc2czExbW1XUjNiMjU5cEg4Nk42bE1lNzlZRkgzdjVtRWNGS015MU52VkRT?= =?utf-8?B?Z3FSbWd4RGw3VzZKVVU3cDRON1JPNWdDY1BndklhMDBwbWZ3bWVTY2lWYVpP?= =?utf-8?B?bElPTVV3Y2RhZno1dVNwSnd0T01FakFHNU8xWXZjSmU1V3d6L0tzU3Y2Vy9K?= =?utf-8?B?QlFjWHZYUCttNm5aVjY4S0liZmwrNW55TGF0a1hoY3dxTmNSSFM5U2tvV3F1?= =?utf-8?B?K1pGZUx2ellRc3U3b3psV0VTZjlEMW1lZ2x3cEtnY1pOVDQ4YkZpdlJNNlAz?= =?utf-8?B?OFl4RWJwR3JOeSsveXYxTjRXNXM3b1Fta1d0WjUwSDhFWWM1N3BpQXFIZVZV?= =?utf-8?B?djJ3TTBCNnJjS2JMKzQvZStQdkxSSE9lNktCcDJPb0dMSTgvWVFpcVcrWXRP?= =?utf-8?B?V0lOZGNFb2p0YjYyU1RJUDRjZWppczNYdTBETWcwaFI3U09aUCs4eGptaGI0?= =?utf-8?B?Z2VjYzg1Z3JuTFpucU1JVXBJc1VTdXVBeE13UlFYSTNFbE9FTVc3bVZQSDJn?= =?utf-8?B?aWhscGkvS3JXL1UzOUtkVC9tYXR3QndaTHBCQUtuTWUreGZEdHFnN05MQ2Mv?= =?utf-8?B?Z0V1VVVteEY4b0N3enZUVkNsWVBBUDk4UnpkV1VFdjMycDF4OXY3T0EwR1R5?= =?utf-8?B?WUVqclJBOXJ2cjdUZ3FLSmNvR1FiT3N6K0FTUloydnZ4K2hpNHdCbU4yMzVW?= =?utf-8?B?N1hub2pIZy9TWUlvQ016RWZIcm1jUXVJK1ZkeW1vTTZmcTBpaWlUY0JiQmZZ?= =?utf-8?B?aTJNeDdMRFFOQ3dBMXZlcFQ1VjJLc3JXdFRTZFZmUDVYditzS0lqdG5WRldq?= =?utf-8?B?ZVpGMVNIMDFNSUhwaFFxdjB0bFlEYTE0UkV0UTRIRHRkNEp4OWxSYlowREM2?= =?utf-8?B?bmtaQ2RFSmVhUnZBa1I5VWNoeHJVb3V3bVQ5UkNNSHZnNzRteDU4Ulltb2dL?= =?utf-8?B?ekhSSHQ5NnFEVU9KRkUzSUpzaWZSZ0ZyeWkxWk05V0wya3RFYlkrK2llMU5W?= =?utf-8?B?ZitGclA5L3FzQTZ4TjBmRVNadWRFN3FzYlU0eng4L0o4bEp6aklCVVE5dDkx?= =?utf-8?B?ekJuaGVReWhKMHhGSmtsQmxPVjNGMFNia24wT21DN0xyYkMxQlZCQWJnMHFT?= =?utf-8?B?WWQwNHJGVDVhR0tzUWtWa2pYVGNnWVE0UnZsWjc0OGJEZ1hXaE11RVNzZlo5?= =?utf-8?B?MkdHY3RCQ2ZNL0Vzb1RudS9qdjFOZGxOUm1MMkdZUk1hcjdkU2EzZWtBa2J6?= =?utf-8?B?Z1ZVTEZtTmlOT3lrQnJGWjdqY1lIbC9RM0dRNnp1VlFpRnhDU29sRXl2K2pV?= =?utf-8?B?UE8wU3RoN3dCNEIyZjhBbmRoeGdJb3diWWdycUo1WGhqajBPQUk3Umtna0Ru?= =?utf-8?B?NDZqT3VmV2l3SHBwN2tma1dSVGVlL1F0ZHYwRThFNXdvV1V3QlZqYWRDSXdx?= =?utf-8?B?UG5Kb05idzdobkdSZGNMVkJXempGa3hHQkxrb2RFbG5pZGxFU2hISjZ5SmhJ?= =?utf-8?B?TUVYS20rOEdaM21hbTZtaC9Jd21Eb1QzeVpMT1VGYnAwaW9MdmZrMFo4NzJy?= =?utf-8?B?ZkRvQmFCbW0zRHVhdERUa09RZjJTNWJ3UkdpcTBsQ201UmlDNHluMnNzbG9z?= =?utf-8?B?TmxwMlhrT2o5VGo0Q3V3V2tSMUpmNjN6SFY0dzR1VWZlVklPQ0ZTN2F4Y0Nw?= =?utf-8?Q?VEe6EhaRWKQX8sKmUBQ/PEMJc?= X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-Network-Message-Id: 49178676-2758-4cb4-d7d7-08dad3b522d1 X-MS-Exchange-CrossTenant-AuthSource: AS8PR08MB6995.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2022 16:00:15.6346 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ekbSJww17Pl6Aww8SOQUa6miAoaqxGKV72DXWkQEdgdU7uRGph18n38du1d6qxRLM4BRBb4K+3ZVguzLFqKpNw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAVPR08MB9556 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221201_081611_939713_6A5F196D X-CRM114-Status: GOOD ( 38.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 01/12/2022 13:43, Ard Biesheuvel wrote: > [...]>>>>>> >>>>>> 2) 4 FAILING TESTS: Host kernel gets stuck initializing KVM >>>>>> >>>>>> During kernel boot, last console log is "kvm [1]: vgic interrupt IRQ9". All >>>>>> failing tests are configured for protected KVM, and are build with LPA2 >>>>>> support, running on non-LPA2 HW. >>>>>> >>>>> >>>>> I will try to reproduce this locally. >> >> It turns out the same issue is hit when running your patches without mine on >> top. The root cause in both cases is an assumption that kvm_get_parange() makes >> that ID_AA64MMFR0_EL1_PARANGE_MAX will always be 48 for 4KB and 16KB PAGE_SIZE. >> That is no longer true with your patches. It causes the VTCR_EL2 to be >> programmed incorrectly for the host stage2, then on return to the host, bang. >> > > Right. I made an attempt at replacing 'u32 level' with 's32 level' > throughout that code, along with some related changes, but I didn't > spot this issue. > >> This demonstrates that kernel stage1 support for LPA2 depends on kvm support for >> LPA2, since for protected kvm, the host stage2 needs to be able to id map the >> full physical range that the host kernel sees prior to deprivilege. So I don't >> think it's fixable in your series. I have a fix in my series for this. >> > > The reference to ID_AA64MMFR0_EL1_PARANGE_MAX should be fixable in > isolation, no? Even if it results in a KVM that cannot use 52-bit PAs > while the host can. Well that doesn't really sound like a good solution to me - the VMM can't control which physical memory it is using so it would be the luck of the draw as to whether kvm gets passed memory above 48 PA bits. So you would probably have to explicitly prevent kvm from initializing if you want to keep your kernel LPA2 patches decoupled from mine. > >> I also found another dependency problem (hit by some of the tests that were >> previously failing at the first issue) where kvm uses its page table library to >> walk a user space page table created by the kernel (see >> get_user_mapping_size()). In the case where the kernel creates an LPA2 page >> table, kvm can't walk it without my patches. I've also added a fix for this to >> my series. >> > > OK Again, I think the only way to work around this without kvm understanding LPA2 is to disable KVM when the kernel is using LPA2. > >> >>>>> >>>>>> >>>>>> 3) 42 FAILING TESTS: Guest kernel never outputs anything to console >>>>>> >>>>>> Host kernel boots fine, and we attempt to launch a guest kernel using kvmtool. >>>>>> There is no error reported, but the guest never outputs anything. Haven't >>>>>> worked out which config options are common to all failures yet. >>>>>> >>>>> >>>>> This goes a bit beyond what I am currently set up for in terms of >>>>> testing, but I'm happy to help narrow this down. >> >> I don't have a root cause for this yet. I'll try to take a loo this afternoon. >> Will keep you posted. >> OK found it. All these failures are when loading the guest in 'high' memory. My test environment is allocating a vm with 256M@2048T (so all memory above 48 bits IPA) and putting the 64KB/52VA/52PA kernel, dtb and intird there. This works fine without your changes. I guess as part of your change you have broken 64KB kernel's ability to create an ID map above 48 bits (which conforms to your clarification of the boot protocol). I guess this was intentional? It makes it really hard for me to test >48 bit IA at stage2... > >> >> Once I have all the tests passing, I'll post my series, then hopefully we can >> move it all forwards as one? >> > > That would be great, yes, although my work depends on a sizable rework > of the early boot code that has seen very little review as of yet. > > So for the time being, let's keep aligned but let's not put any eggs > in each other's baskets :-)> >> As part of my debugging, I've got a patch to sort out the tlbi code to support >> LPA2 properly - I think I raised that comment on one of the patches. Are you >> happy for me to post as part of my series? >> > > I wasn't sure where to look tbh. The generic 5-level paging stuff > seems to work fine - is this specific to KVM? There are a couple of issues that I spotted; for the range-based tlbi instructions, when LPA2 is in use (TCR_EL1.DS=1), BaseADDR must be 64KB aligned (when LPA2 is disabled it only needs to be page aligned). So there is some forward alignment required using the non-range tbli in __flush_tlb_range(). I think this would manifest as invalidating the wrong entries if left as-is once your patches are applied. The second problem is that __tlbi_level() uses level 0 as the "don't use level hint" sentinel. I think this will work just fine (if slightly suboptimal) if left as is, since the higher layers are never passing anything outside the range [0, 3] at the moment, but if that changed and -1 was passed then it would cause a bug. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel