Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 20527C433FE for ; Mon, 28 Nov 2022 14:55:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:From: References:Cc:To:Subject:Date:Message-ID:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=oV+Pqx0pZuDHJfhcUT6fWFXmsYPblg+9UrMPAvLjtMI=; b=JPpTdguE3ydpCJ6pUmiDg0dgYE /SFWBE3ICwkaYoSg3ZokcE0X0PvZRtbnSnHNFeftLpd1P1DRliPSvhCVDRGMCHv5b6R6zn28qhJl1 VVYm6/LRChLEOJEVrZvNM1+9yVY4D4/FpaAXfRBf2W4lCjBozEsVkazk4YlyWTJuafE3hBL2BVC9p Igp7nP/nkLsdUs6fvVRPkDSSD71bjgrruQK12wkBzX1CCht5KNG4eWLZR9vmmyiCr5i38XEEpjie2 QXfTKPn3x21Hs0yc5y4ePqyt7q12e7LkLDrfZ9JmQM9va9k/gA9p8fHIgbQvDdRckRCd1qSm0RCVA w5UsWPEg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ozfWq-002MEo-VD; Mon, 28 Nov 2022 14:54:41 +0000 Received: from mail-dbaeur03on20605.outbound.protection.outlook.com ([2a01:111:f400:fe1a::605] helo=EUR03-DBA-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ozfWm-002MCV-4G for linux-arm-kernel@lists.infradead.org; Mon, 28 Nov 2022 14:54:39 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AU+J6lTds4RSnBNIut6HIDtZO/VLqpP1lVBotKulr1Izm3knlQ2PJoCTxysLd95KibRul1jShfJd+eU6RNb7bB2KaMHwkOm6gEN24gaLc4TLQkZfRpQqrBsB+iBsPtRvZdcidx4LxP1Qj5NinZERbw/puUWzxh0nsRNavj6om03I+Xopi1blf31gvNOwHftjclb/5qnOmtPddKypy8HmDe0Kwq9oQ58kqVji+PrF18F6+SrV/cgvDCfUBaSnp1NIE92XeHEC3yphMOCDUZfFJvHs/8Ox4pHdIuHUPpnQaP3CL/0PWR4eqcD0jec9kG15gzu5Wh7Kaebmgc5AvFmVWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ommB9kO8eoxse2Bkl19dy/laFzpXrr6l1/xU2174EII=; b=mrJ+DtAXPYOd3qA2rS9Rf1xyexQAwCh9c/BSkODlarOGiV7dPFlGyKe5ethF6M/nW+aDnzXkPGeu3XTJVVmV8Zml66NLVtHfagQnPKwbD1+gUUHP+SHPDaXhInBzfysrDt6cjld3k04Znd07xoDahiXm4F4eAlAMNWUz8LXU8fQ+bOLXfq2RnCbkbPzeZ7gYbBm0kKxj69Ne8dt+huGDPLSoIjtEp1nCKU681rJ6AIJmOXm27SCf7D8LaCvUjEiswd6h0rQRudnTbtThlFJxfaiEJTepQX3/yvFAGeE5Nq7Dnz3DI1a9d+NBwUbHTtub5tmUg/nr6tQepTNkW0dVoA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ommB9kO8eoxse2Bkl19dy/laFzpXrr6l1/xU2174EII=; b=nRTllJEZIXMooWFj0JttSZT2liSi9MkSwJHd7qO31qfFPyhVnwPfpPAC96h8H7nNbkklQeSA5lFtbV4LLBOzjF/y3L6iQ84852V6lgRy944RhNvf2MQXFsB+3onRMp0kQNtW8JRL2QqLIY0phGgzf4FjNX5/PFQ/3yuN6wfNatg= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; Received: from AS8PR08MB6995.eurprd08.prod.outlook.com (2603:10a6:20b:34d::13) by AS8PR08MB9042.eurprd08.prod.outlook.com (2603:10a6:20b:5b0::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5880.6; Mon, 28 Nov 2022 14:54:25 +0000 Received: from AS8PR08MB6995.eurprd08.prod.outlook.com ([fe80::e825:ff01:1f8:4c1f]) by AS8PR08MB6995.eurprd08.prod.outlook.com ([fe80::e825:ff01:1f8:4c1f%4]) with mapi id 15.20.5880.008; Mon, 28 Nov 2022 14:54:25 +0000 Message-ID: Date: Mon, 28 Nov 2022 14:54:21 +0000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.5.0 Subject: Re: [PATCH v2 14/19] arm64: Enable LPA2 at boot if supported by the system Content-Language: en-US To: Ard Biesheuvel , linux-arm-kernel@lists.infradead.org Cc: Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual , Richard Henderson References: <20221124123932.2648991-1-ardb@kernel.org> <20221124123932.2648991-15-ardb@kernel.org> From: Ryan Roberts In-Reply-To: <20221124123932.2648991-15-ardb@kernel.org> X-ClientProxiedBy: LO2P265CA0249.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8a::21) To AS8PR08MB6995.eurprd08.prod.outlook.com (2603:10a6:20b:34d::13) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS8PR08MB6995:EE_|AS8PR08MB9042:EE_ X-MS-Office365-Filtering-Correlation-Id: 67c64301-c6a7-4cc5-e43e-08dad15070e1 NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: jtQBOek050+WJj/c4Aia/ObR+kw6dleAWrFtNgaYC9CVwMIecaV77N+3QxVM+Yulyg05gFTyPsjr6mSZLmOgcq5+jBp75Ost1Ggajz+L1shIH5SYYXnEXkCRUEBUulzB1Nml21/vLl85Ch5HqmmiTsAfjKt9lBnntvDfHYgoL/y4c9L517PSsbkeFnHqTtqAcmFXspVxuwrvupBP3ZFt56gPRxD3O0dRdofPTvlBrIO5bE7kXlKUCE80IOyECyWKtuaY2RnScp0afQHvdqVNOenQ1/+F/xfjK48AF5f5VavMVneq0TGVkg7noqWj+ytq0+upkZy8Z87ZYuRm3jvm7345CZfwovPTY+s4AazCD7QHrF/uOmwVXw7vn/gKwNvaeOEp9492iIwuCiSrlwbI98HQxmnvauwO50zusvrKnENSrhGxg/b2oNWzcO7yIfgmVHrG6aJ1xGueiZF29HU2bfULks1gRjm/vhc2cjdZyj1W+kQWTFuihON8s2W7YJzSpxPR/wtG6lAvDZDAwQHWwKXLo5t3eXYnNOlClXwLt8irU7bfqSWMXzUU8MPzA36TaTOOT4cao41+RcXMhSiso3MH5YyLLXmQlU+QrW+0q+3u3W7tt6gyUqygh6b9GEpjS2VQT81c4qDmjtPJw1GEGyIU7WrsZWihoVe3lGjgX3Tez85ngmSzggWVjLngrotFpD+Y9yb91jSIi1aqggb5OOcbRVMbcRkTw9hBEWZdlf8= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS8PR08MB6995.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(39850400004)(376002)(396003)(346002)(136003)(366004)(451199015)(41300700001)(6666004)(44832011)(86362001)(54906003)(26005)(4326008)(83380400001)(316002)(8676002)(66476007)(66946007)(66556008)(2906002)(6506007)(36756003)(6512007)(38100700002)(53546011)(66899015)(31686004)(478600001)(5660300002)(6486002)(186003)(2616005)(30864003)(31696002)(8936002)(45980500001)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YVVtaVhhU2pHVWE0MTFjUFNZdHBCVm9ZWnZQa1dFSlkvaGtMZkgrTkJCNkZw?= =?utf-8?B?UzVBTUNJUWNFUjVPYmJsZHplZXZZQ1JSQUsxZWVPc0lXbVcveUZmWDhBSlFj?= =?utf-8?B?NG9DY3dtVjJBdFUwZGlsQ0p0OVUrNEVadU9RU1p1WUh3d0tkcCtmN2NYYWY5?= =?utf-8?B?Vi91QllITmxLOWZRRXhOQVJ6ZHBLV2hYemViTi9jVytrMWNTSFU0bElyaFhZ?= =?utf-8?B?bTFGMDRwajFYaWhVeEN2K1ZJUnpsSTJBeEtSVGJ3SGxIUzFaaUcwcVhNcW5s?= =?utf-8?B?SUtoMXlyMlNIbGlBaWNqR2UrQ2VESzJhd3JWZHVDbW1HM3psWXhCbTMwOXBv?= =?utf-8?B?YzhWcHZZbXg3eGhIdkdmV0F6dC9nc3BHT2haSWUzN1Irb1diV2Y1d041aVNz?= =?utf-8?B?N1RZYXdxYnBRbFJwNUsvZnpQNXpSWlNtOFlGaTlkMU51c002dVhnOWhNZWJH?= =?utf-8?B?WHZxRzk3SnViYUc4RU1WdzVMMW51SVlnRThlcVp2eGZXazhJa29tY3BHYVBM?= =?utf-8?B?RXNLRWRBS1ZvS0VTdUhRYjBKclRLT2hBcENNdmFqMllTZUJnOSt2akNieFNX?= =?utf-8?B?TlRnVG52ZWF6N252VTBRdDlSaGo1c2UvVXdLRTcwdU1TZ0k5cFRUNkhWOVlU?= =?utf-8?B?Z3FUZkVadFZzR2U1VElXWHFvVzBaYWtlajZhaHUrNUVxMExUc2REZnY0UmV6?= =?utf-8?B?SEQzdkdPYWl2TWhYNFkraXk5RFlydENlNXVSdUpRbTU4Mk9wUUQyL3k2dHF5?= =?utf-8?B?eVBDWnhUSzVPYVBpMW9UOXBJUFlWN1YzdGx0TzJHVUdXUE1ROTgzTDF4WnJq?= =?utf-8?B?QjJaUCtjdHp6NnBVelVCSWNtaEI0WEZjNk1pQVlJVktIekx6TjNEK3VVUnhH?= =?utf-8?B?eXRISXl6Z09qU205Mm8vdmxObTI3K0xFRlliMWMvOUM0UWdsYmFJRTQ5QjZr?= =?utf-8?B?WXRvNGdvaVNIUGpsNFN6cWJtY0dVTHJnSjNDak1xcVNuQlJYaTBIenJjUEti?= =?utf-8?B?Smc4WjVJbCtGZGxXODNXS1ZZTGlhTWxNYWF4a0JpalFndVh2Ujk5b1oySGRj?= =?utf-8?B?TVIrRGVXUWMyK1RZOGh3aW02dVJkY0xYTVpLd2xURmFpcGx4L2swcnF4VVlX?= =?utf-8?B?M25wNnh3NlFqaCtYb2l0dm5qVzNWZGl5VVVvYVFmbFJybzZsTTlpOHhoQWNv?= =?utf-8?B?dm9qVU9DMXRCWVNiRnAydXpNMGtlcWR2TWdZa1A1aU9yU0hvZ25uSjB1TGZz?= =?utf-8?B?c25kRUtucTVnS1FXeWxpVkExekoxek5ZdFdoNXAzRlRLZnNpUWpTbVN0M2Uy?= =?utf-8?B?WmVuRjhvaWJaNHlxcjAzRUlPOXdFWFdGQitPdmpkOC8rVGJwSWt5NVl5OUJG?= =?utf-8?B?TzNYTWR2QjVMRFU4WEVnUEs3eXYwM0J0WCt1UzlwcHh2Rm8xRDB5dy9ycktX?= =?utf-8?B?R0xhcFp3STExRTRmVFRPM084K3pLS25pVE4zMVNQeVRHNUlTSkkwbnFJbTN5?= =?utf-8?B?QnluWEo4YVhzeVNEdDAxMDJabjJjM1ZjcExjSUIvZGtZYUFOdHBpbFlKZUNr?= =?utf-8?B?K0hqZ1drbER4aDVGbkVkd1ovbG1IRFkvUnJBNDdvMm1pbkNZMDIyQ2VXSkZJ?= =?utf-8?B?aitNUXp6TGhtYkk3Mlg2NHgxOXlSQkYrb0k1YjlUWCthT1ord0R4YWpMZHBJ?= =?utf-8?B?T0h2VjRXMW15NTV5TXg5c2VTNGt3WEJXK2ZhcVdPeUcybXVhS0ZtcjNJbmFr?= =?utf-8?B?YWhkaGgySFd0bDJaYzZTVEw5Si9lSzhmWHBpRDlSSkxCWEVVTTVRTmR5OHEy?= =?utf-8?B?OWlBRlZUaGZrcVpEcEoyQTJYQ2xub3M2VjhzV0hUVFdGcEtZUzIrTDQ0MTRk?= =?utf-8?B?dmdmREEvRG1GM0VVR3IraG1vTWJJVVdpZWZUS214U0hjZHd0RUcrSDRVSXlt?= =?utf-8?B?Q0c3RnZ5alZpVWViSXhueUFma204Qklpb3UveWhxd2lHQWg3U3RkekVEdmpN?= =?utf-8?B?WXladUJKUjJiTi9oQUxNMmRSM29jNkIwU0tJNmk1b0dGSCtpTWZuelV3K0xE?= =?utf-8?B?VDIwNVlJd05ZWUdtdHRsL2I5STd3eER0c1VoaVBKMkZ4WXRGdUl3QjAvZThD?= =?utf-8?Q?wZSRBOX02WwTIJSs3rwbAZyET?= X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-Network-Message-Id: 67c64301-c6a7-4cc5-e43e-08dad15070e1 X-MS-Exchange-CrossTenant-AuthSource: AS8PR08MB6995.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Nov 2022 14:54:24.9662 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: MxwbvyC4xrTINDDlUsDhHcaf+BGHv8jH2OIOvHDG3oWoXtTUpSe3ev12WJ3XDJ9VTATBVKpqrACie+iwHfTtZQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR08MB9042 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221128_065436_542706_56D59CBC X-CRM114-Status: GOOD ( 33.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 24/11/2022 12:39, Ard Biesheuvel wrote: > Update the early kernel mapping code to take 52-bit virtual addressing > into account based on the LPA2 feature. This is a bit more involved than > LVA (which is supported with 64k pages only), given that some page table > descriptor bits change meaning in this case. > > To keep the handling in asm to a minimum, the initial ID map is still > created with 48-bit virtual addressing, which implies that the kernel > image must be loaded into 48-bit addressable physical memory. This is > currently required by the boot protocol, even though we happen to > support placement outside of that for LVA/64k based configurations. > > Enabling LPA2 involves more than setting TCR.T1SZ to a lower value, > there is also a DS bit in TCR that needs to be set, and which changes > the meaning of bits [9:8] in all page table descriptors. Since we cannot > enable DS and every live page table descriptor at the same time, let's > pivot through another temporary mapping. This avoids the need to > reintroduce manipulations of the page tables with the MMU and caches > disabled. > > To permit the LPA2 feature to be overridden on the kernel command line, > which may be necessary to work around silicon errata, or to deal with > mismatched features on heterogeneous SoC designs, test for CPU feature > overrides first, and only then enable LPA2. > > Signed-off-by: Ard Biesheuvel > --- > arch/arm64/include/asm/assembler.h | 7 +- > arch/arm64/include/asm/kernel-pgtable.h | 25 +++-- > arch/arm64/include/asm/memory.h | 4 + > arch/arm64/kernel/head.S | 9 +- > arch/arm64/kernel/image-vars.h | 2 + > arch/arm64/kernel/pi/map_kernel.c | 103 +++++++++++++++++++- > arch/arm64/mm/init.c | 2 +- > arch/arm64/mm/mmu.c | 8 +- > arch/arm64/mm/proc.S | 4 + > 9 files changed, 151 insertions(+), 13 deletions(-) > > diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h > index 786bf62826a8..30eee6473cf0 100644 > --- a/arch/arm64/include/asm/assembler.h > +++ b/arch/arm64/include/asm/assembler.h > @@ -609,11 +609,16 @@ alternative_endif > * but we have to add an offset so that the TTBR1 address corresponds with the > * pgdir entry that covers the lowest 48-bit addressable VA. > * > + * Note that this trick only works for LVA/64k pages - LPA2/4k pages uses an > + * additional paging level, and on LPA2/16k pages, we would end up with a TTBR > + * address that is not 64 byte aligned, so there we reduce the number of paging > + * levels for the non-LPA2 case. > + * > * orr is used as it can cover the immediate value (and is idempotent). > * ttbr: Value of ttbr to set, modified. > */ > .macro offset_ttbr1, ttbr, tmp > -#ifdef CONFIG_ARM64_VA_BITS_52 > +#if defined(CONFIG_ARM64_VA_BITS_52) && !defined(CONFIG_ARM64_LPA2) > mrs \tmp, tcr_el1 > and \tmp, \tmp, #TCR_T1SZ_MASK > cmp \tmp, #TCR_T1SZ(VA_BITS_MIN) > diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h > index faa11e8b4a0e..2359b2af0c4c 100644 > --- a/arch/arm64/include/asm/kernel-pgtable.h > +++ b/arch/arm64/include/asm/kernel-pgtable.h > @@ -20,12 +20,16 @@ > */ > #ifdef CONFIG_ARM64_4K_PAGES > #define INIT_IDMAP_USES_PMD_MAPS 1 > -#define INIT_IDMAP_TABLE_LEVELS (CONFIG_PGTABLE_LEVELS - 1) > #else > #define INIT_IDMAP_USES_PMD_MAPS 0 > -#define INIT_IDMAP_TABLE_LEVELS (CONFIG_PGTABLE_LEVELS) > #endif > > +/* how many levels of translation are required to cover 'x' bits of VA space */ > +#define VA_LEVELS(x) (((x) - 4) / (PAGE_SHIFT - 3)) > +#define INIT_IDMAP_TABLE_LEVELS (VA_LEVELS(VA_BITS_MIN) - INIT_IDMAP_USES_PMD_MAPS) > + > +#define INIT_IDMAP_ROOT_SHIFT (VA_LEVELS(VA_BITS_MIN) * (PAGE_SHIFT - 3) + 3) > + > /* > * If KASLR is enabled, then an offset K is added to the kernel address > * space. The bottom 21 bits of this offset are zero to guarantee 2MB > @@ -52,7 +56,14 @@ > #define EARLY_ENTRIES(vstart, vend, shift, add) \ > ((((vend) - 1) >> (shift)) - ((vstart) >> (shift)) + 1 + add) > > -#define EARLY_PGDS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, PGDIR_SHIFT, add)) > +#if CONFIG_PGTABLE_LEVELS > 4 > +/* the kernel is covered entirely by the pgd_t at the top of the VA space */ > +#define EARLY_PGDS 1 > +#else > +#define EARLY_PGDS 0 > +#endif > + > +#define EARLY_P4DS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, INIT_IDMAP_ROOT_SHIFT, add)) > > #if INIT_IDMAP_TABLE_LEVELS > 3 > #define EARLY_PUDS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, PUD_SHIFT, add)) > @@ -66,11 +77,13 @@ > #define EARLY_PMDS(vstart, vend, add) (0) > #endif > > -#define EARLY_PAGES(vstart, vend, add) ( 1 /* PGDIR page */ \ > - + EARLY_PGDS((vstart), (vend), add) /* each PGDIR needs a next level page table */ \ > +#define EARLY_PAGES(vstart, vend, add) ( 1 /* PGDIR/P4D page */ \ > + + EARLY_P4DS((vstart), (vend), add) /* each P4D needs a next level page table */ \ > + EARLY_PUDS((vstart), (vend), add) /* each PUD needs a next level page table */ \ > + EARLY_PMDS((vstart), (vend), add)) /* each PMD needs a next level page table */ > -#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(KIMAGE_VADDR, _end, EARLY_KASLR) + EARLY_SEGMENT_EXTRA_PAGES)) > + > +#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(KIMAGE_VADDR, _end, EARLY_KASLR) + \ > + EARLY_SEGMENT_EXTRA_PAGES + EARLY_PGDS)) > > /* the initial ID map may need two extra pages if it needs to be extended */ > #if VA_BITS_MIN < 48 > diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h > index b3826ff2e52b..4f617e271008 100644 > --- a/arch/arm64/include/asm/memory.h > +++ b/arch/arm64/include/asm/memory.h > @@ -54,7 +54,11 @@ > #define FIXADDR_TOP (VMEMMAP_START - SZ_32M) > > #if VA_BITS > 48 > +#ifdef CONFIG_ARM64_16K_PAGES > +#define VA_BITS_MIN (47) > +#else > #define VA_BITS_MIN (48) > +#endif > #else > #define VA_BITS_MIN (VA_BITS) > #endif > diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S > index 94de42dfe97d..6be121949c06 100644 > --- a/arch/arm64/kernel/head.S > +++ b/arch/arm64/kernel/head.S > @@ -198,7 +198,7 @@ SYM_CODE_END(preserve_boot_args) > mov \tbl, \sv > .endif > .L_\@: > - compute_indices \vstart, \vend, #PGDIR_SHIFT, \istart, \iend, \count > + compute_indices \vstart, \vend, #INIT_IDMAP_ROOT_SHIFT, \istart, \iend, \count > mov \sv, \rtbl > populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp > mov \tbl, \sv > @@ -610,9 +610,16 @@ SYM_FUNC_START(__cpu_secondary_check52bitva) > alternative_if_not ARM64_HAS_LVA > ret > alternative_else_nop_endif > +#ifndef CONFIG_ARM64_LPA2 > mrs_s x0, SYS_ID_AA64MMFR2_EL1 > and x0, x0, #(0xf << ID_AA64MMFR2_EL1_VARange_SHIFT) > cbnz x0, 2f > +#else > + mrs x0, id_aa64mmfr0_el1 > + sbfx x0, x0, #ID_AA64MMFR0_EL1_TGRAN_SHIFT, 4 > + cmp x0, #ID_AA64MMFR0_EL1_TGRAN_LPA2 > + b.ge 2f > +#endif > > update_early_cpu_boot_status \ > CPU_STUCK_IN_KERNEL | CPU_STUCK_REASON_52_BIT_VA, x0, x1 > diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h > index 82bafa1f869c..f48b6f09d278 100644 > --- a/arch/arm64/kernel/image-vars.h > +++ b/arch/arm64/kernel/image-vars.h > @@ -56,6 +56,8 @@ PROVIDE(__pi_arm64_sw_feature_override = arm64_sw_feature_override); > PROVIDE(__pi_arm64_use_ng_mappings = arm64_use_ng_mappings); > PROVIDE(__pi__ctype = _ctype); > > +PROVIDE(__pi_init_idmap_pg_dir = init_idmap_pg_dir); > +PROVIDE(__pi_init_idmap_pg_end = init_idmap_pg_end); > PROVIDE(__pi_init_pg_dir = init_pg_dir); > PROVIDE(__pi_init_pg_end = init_pg_end); > PROVIDE(__pi_swapper_pg_dir = swapper_pg_dir); > diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c > index a9472ab8d901..75d643da56c8 100644 > --- a/arch/arm64/kernel/pi/map_kernel.c > +++ b/arch/arm64/kernel/pi/map_kernel.c > @@ -133,6 +133,20 @@ static bool __init arm64_early_this_cpu_has_lva(void) > ID_AA64MMFR2_EL1_VARange_SHIFT); > } > > +static bool __init arm64_early_this_cpu_has_lpa2(void) > +{ > + u64 mmfr0; > + int feat; > + > + mmfr0 = read_sysreg(id_aa64mmfr0_el1); > + mmfr0 &= ~id_aa64mmfr0_override.mask; > + mmfr0 |= id_aa64mmfr0_override.val; > + feat = cpuid_feature_extract_signed_field(mmfr0, > + ID_AA64MMFR0_EL1_TGRAN_SHIFT); > + > + return feat >= ID_AA64MMFR0_EL1_TGRAN_LPA2; > +} This fails to compile when configured for 64KB pages, since ID_AA64MMFR0_EL1_TGRAN_LPA2 is only defined for 4KB and 16KB granules (see sysreg.h). Suggest: static bool __init arm64_early_this_cpu_has_lpa2(void) { #ifdef ID_AA64MMFR0_EL1_TGRAN_LPA2 u64 mmfr0; int feat; mmfr0 = read_sysreg(id_aa64mmfr0_el1); mmfr0 &= ~id_aa64mmfr0_override.mask; mmfr0 |= id_aa64mmfr0_override.val; feat = cpuid_feature_extract_signed_field(mmfr0, ID_AA64MMFR0_EL1_TGRAN_SHIFT); return feat >= ID_AA64MMFR0_EL1_TGRAN_LPA2; #else return false; #endif } > + > static bool __init arm64_early_this_cpu_has_pac(void) > { > u64 isar1, isar2; > @@ -254,11 +268,85 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) > } > > /* Copy the root page table to its final location */ > - memcpy((void *)swapper_pg_dir + va_offset, init_pg_dir, PGD_SIZE); > + memcpy((void *)swapper_pg_dir + va_offset, init_pg_dir, PAGE_SIZE); > dsb(ishst); > idmap_cpu_replace_ttbr1(swapper_pg_dir); > } > > +static void noinline __section(".idmap.text") set_ttbr0_for_lpa2(u64 ttbr) > +{ > + u64 sctlr = read_sysreg(sctlr_el1); > + u64 tcr = read_sysreg(tcr_el1) | TCR_DS; > + > + /* Update TCR.T0SZ in case we entered with a 47-bit ID map */ > + tcr &= ~TCR_T0SZ_MASK; > + tcr |= TCR_T0SZ(48); > + > + asm(" msr sctlr_el1, %0 ;" > + " isb ;" > + " msr ttbr0_el1, %1 ;" > + " msr tcr_el1, %2 ;" > + " isb ;" > + " tlbi vmalle1 ;" > + " dsb nsh ;" > + " isb ;" > + " msr sctlr_el1, %3 ;" > + " isb ;" > + :: "r"(sctlr & ~SCTLR_ELx_M), "r"(ttbr), "r"(tcr), "r"(sctlr)); > +} > + > +static void remap_idmap_for_lpa2(void) > +{ > + extern pgd_t init_idmap_pg_dir[], init_idmap_pg_end[]; > + pgd_t *pgdp = (void *)init_pg_dir + PAGE_SIZE; > + pgprot_t text_prot = PAGE_KERNEL_ROX; > + pgprot_t data_prot = PAGE_KERNEL; > + > + /* clear the bits that change meaning once LPA2 is turned on */ > + pgprot_val(text_prot) &= ~PTE_SHARED; > + pgprot_val(data_prot) &= ~PTE_SHARED; > + > + /* > + * We have to clear bits [9:8] in all block or page descriptors in the > + * initial ID map, as otherwise they will be (mis)interpreted as > + * physical address bits once we flick the LPA2 switch (TCR.DS). Since > + * we cannot manipulate live descriptors in that way without creating > + * potential TLB conflicts, let's create another temporary ID map in a > + * LPA2 compatible fashion, and update the initial ID map while running > + * from that. > + */ > + map_segment(init_pg_dir, &pgdp, 0, _stext, __inittext_end, text_prot, > + false, 0); > + map_segment(init_pg_dir, &pgdp, 0, __initdata_begin, _end, data_prot, > + false, 0); > + dsb(ishst); > + set_ttbr0_for_lpa2((u64)init_pg_dir); > + > + /* > + * Recreate the initial ID map with the same granularity as before. > + * Don't bother with the FDT, we no longer need it after this. > + */ > + memset(init_idmap_pg_dir, 0, > + (u64)init_idmap_pg_dir - (u64)init_idmap_pg_end); > + > + pgdp = (void *)init_idmap_pg_dir + PAGE_SIZE; > + map_segment(init_idmap_pg_dir, &pgdp, 0, > + PTR_ALIGN_DOWN(&_stext[0], INIT_IDMAP_BLOCK_SIZE), > + PTR_ALIGN_DOWN(&__bss_start[0], INIT_IDMAP_BLOCK_SIZE), > + text_prot, false, 0); > + map_segment(init_idmap_pg_dir, &pgdp, 0, > + PTR_ALIGN_DOWN(&__bss_start[0], INIT_IDMAP_BLOCK_SIZE), > + PTR_ALIGN(&_end[0], INIT_IDMAP_BLOCK_SIZE), > + data_prot, false, 0); > + dsb(ishst); > + > + /* switch back to the updated initial ID map */ > + set_ttbr0_for_lpa2((u64)init_idmap_pg_dir); > + > + /* wipe the temporary ID map from memory */ > + memset(init_pg_dir, 0, (u64)init_pg_end - (u64)init_pg_dir); > +} > + > asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) > { > static char const chosen_str[] __initconst = "/chosen"; > @@ -266,6 +354,7 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) > u64 va_base, pa_base = (u64)&_text; > u64 kaslr_offset = pa_base % MIN_KIMG_ALIGN; > int root_level = 4 - CONFIG_PGTABLE_LEVELS; > + bool va52 = (VA_BITS == 52); > > /* Clear BSS and the initial page tables */ > memset(__bss_start, 0, (u64)init_pg_end - (u64)__bss_start); > @@ -295,7 +384,17 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) > arm64_use_ng_mappings = true; > } > > - if (VA_BITS == 52 && arm64_early_this_cpu_has_lva()) > + if (IS_ENABLED(CONFIG_ARM64_LPA2)) { > + if (arm64_early_this_cpu_has_lpa2()) { > + remap_idmap_for_lpa2(); > + } else { > + va52 = false; > + root_level++; > + } > + } else if (IS_ENABLED(CONFIG_ARM64_64K_PAGES)) { > + va52 &= arm64_early_this_cpu_has_lva(); > + } > + if (va52) > sysreg_clear_set(tcr_el1, TCR_T1SZ_MASK, TCR_T1SZ(VA_BITS)); > > va_base = KIMAGE_VADDR + kaslr_offset; > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c > index 4b4651ee47f2..498d327341b4 100644 > --- a/arch/arm64/mm/init.c > +++ b/arch/arm64/mm/init.c > @@ -315,7 +315,7 @@ void __init arm64_memblock_init(void) > * physical address of PAGE_OFFSET, we have to *subtract* from it. > */ > if (IS_ENABLED(CONFIG_ARM64_VA_BITS_52) && (vabits_actual != 52)) > - memstart_addr -= _PAGE_OFFSET(48) - _PAGE_OFFSET(52); > + memstart_addr -= _PAGE_OFFSET(vabits_actual) - _PAGE_OFFSET(52); > > /* > * Apply the memory limit if it was set. Since the kernel may be loaded > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c > index d089bc78e592..ba5423ff7039 100644 > --- a/arch/arm64/mm/mmu.c > +++ b/arch/arm64/mm/mmu.c > @@ -541,8 +541,12 @@ static void __init map_mem(pgd_t *pgdp) > * entries at any level are being shared between the linear region and > * the vmalloc region. Check whether this is true for the PGD level, in > * which case it is guaranteed to be true for all other levels as well. > + * (Unless we are running with support for LPA2, in which case the > + * entire reduced VA space is covered by a single pgd_t which will have > + * been populated without the PXNTable attribute by the time we get here.) > */ > - BUILD_BUG_ON(pgd_index(direct_map_end - 1) == pgd_index(direct_map_end)); > + BUILD_BUG_ON(pgd_index(direct_map_end - 1) == pgd_index(direct_map_end) && > + pgd_index(_PAGE_OFFSET(VA_BITS_MIN)) != PTRS_PER_PGD - 1); > > if (can_set_direct_map()) > flags |= NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; > @@ -726,7 +730,7 @@ static void __init create_idmap(void) > > void __init paging_init(void) > { > - idmap_t0sz = 63UL - __fls(__pa_symbol(_end) | GENMASK(VA_BITS_MIN - 1, 0)); > + idmap_t0sz = 63UL - __fls(__pa_symbol(_end) | GENMASK(vabits_actual - 1, 0)); > > map_mem(swapper_pg_dir); > > diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S > index 179e213bbe2d..d95df732b672 100644 > --- a/arch/arm64/mm/proc.S > +++ b/arch/arm64/mm/proc.S > @@ -489,7 +489,11 @@ SYM_FUNC_START(__cpu_setup) > #if VA_BITS > VA_BITS_MIN > mov x9, #64 - VA_BITS > alternative_if ARM64_HAS_LVA > + tcr_set_t0sz tcr, x9 > tcr_set_t1sz tcr, x9 > +#ifdef CONFIG_ARM64_LPA2 > + orr tcr, tcr, #TCR_DS > +#endif > alternative_else_nop_endif > #endif > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel