Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FAA9C02185 for ; Mon, 20 Jan 2025 14:14:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 781B26B0085; Mon, 20 Jan 2025 09:14:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 731F66B0088; Mon, 20 Jan 2025 09:14:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5FA186B0089; Mon, 20 Jan 2025 09:14:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4679B6B0085 for ; Mon, 20 Jan 2025 09:14:32 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id F33C31201BF for ; Mon, 20 Jan 2025 14:14:31 +0000 (UTC) X-FDA: 83028025542.13.23D5FA6 Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) by imf12.hostedemail.com (Postfix) with ESMTP id DF40B40002 for ; Mon, 20 Jan 2025 14:14:29 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=TE3i1WhX; dmarc=none; spf=pass (imf12.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.222.180 as permitted sender) smtp.mailfrom=jgg@ziepe.ca ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737382470; a=rsa-sha256; cv=none; b=mZ0cVvLWGmQSSh6LJnxkdtmzO44B868T18B/dDOBvZqGMwyOVZ0/XkwL0l2ncX1/O9LVbb pVzriVw1QK0N71EOxxHBP7BpVb/8hiq7KyjMpfcKyii10GF7Hr78+PwbEM3SOdrAzRLzXD a1xMu1NIHQ1Idi3FqvBA/ZLLPhNd6ak= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=TE3i1WhX; dmarc=none; spf=pass (imf12.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.222.180 as permitted sender) smtp.mailfrom=jgg@ziepe.ca ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737382470; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BMfPap+yf5q8mEHHYjAtu0fP0Pu5BlL7FNgE7HqGKAU=; b=obdUCGuqpouSFGj+wJjiiDw7oT26tEId8xYymGWeF9oSto9mOy419TrRp3j/1ICgI5IPzB BgRkIxZnnbqF6HMhwFLeOz2gmYRcMovtT0gbsm1OT6m25hVD1YkPHr7kRf6iYqVeVc25HU KcopCNKLCA9kh0j1J/6zyQVA7NDu7W8= Received: by mail-qk1-f180.google.com with SMTP id af79cd13be357-7b6e8814842so450800085a.0 for ; Mon, 20 Jan 2025 06:14:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1737382469; x=1737987269; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=BMfPap+yf5q8mEHHYjAtu0fP0Pu5BlL7FNgE7HqGKAU=; b=TE3i1WhXPwKZCN1lXBJ+4+ex5gpP/ktgGcFKy5FeB6KqM16qcgXG1Q7ww5UwkXhNh5 9/lEP2AMJAq1/PpEbvXOwMg8Thnr/IiisiUGP2GJUXIN11o7EE2Lij5ZgEOCsStaaEg7 /5bHcOPzwsdtJkTLZisZjRtUiHXS0BpYEg2SdfjaF4FNT3n3UYgv5DOYGUjXyNlbexCe 8asbh40X4djp+i7RagkkghqF4qVP6v2OvJ4H6u14mZTUzxR5T/airstXRtG7VBr1XBDp 3nZUyNydTBgRwIFVrX7yU/VD6L34hN30j3PIYmLZ0sPCG9wzoHadOdVPdT7Cf1CbZw/3 Tp/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737382469; x=1737987269; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=BMfPap+yf5q8mEHHYjAtu0fP0Pu5BlL7FNgE7HqGKAU=; b=g8phxA7/9UPrNavZP85IOKfyWi8G7PiG8LVhQ63LNRaumGMj8fuo5tPOah21J+TgTs 3hsI75C/IJV+58KGwyJpXrhxSOBODZzAuAj1ppxZTLWbNyEA4PUzlujRBZnbVo7aOLtf wEy3CJN4FemD4K5h6T5j6dT4u9HVsblo9BZ2DtKREuTagFOkSoA/wHtr1QDkcPINzSXd WHGbVjGs+wIcMbK+44PLWubomqp+OTie/GuuxIxIwZy8dWzqnnqk8AthJJHvoqg4W8Jb FCb+I+YwgeoXAkFgCs3Zo7rq0ulrWW00gXc3Zw8lNp/ymPY/xmQVHjNkNFTLgFi/h0yP ROZw== X-Forwarded-Encrypted: i=1; AJvYcCUS0mCbZyfGvBZ27f4YywhNbiU86MUHpqC/Xs57fOvQ2VhmOXYVlBQYrUZYQaIL0W1uUQABwndCsQ==@kvack.org X-Gm-Message-State: AOJu0YzTZBhvPjqLxfnpqyh1Ln6YjQzly6D3vI7K15ietNKYb32sMjZm ZnHShXeqDUVA6z8fE+1+mjey/VsELJAlwfHFg9doCHnV4B0kQHOYEPfM71K4dEs= X-Gm-Gg: ASbGncubuAF9VTLxnZp1fnanUkJwJc93Vvoo/TzvoBDNSBKqcGG31C6JhXCGWpkNHvj qCInjPRs2rpMqo8C230Gcm+1lUb64O2iEyLYEC1do4dmpF1+xizV0pO34YkigZS04C0NhmqW8JV juAL9Rb0OaCSaRrcKVS3i0+u00xcr6u13wJW9GPCELfthsChSGKWGu0g73lC+lefGg9untsaK5n Mo+z8Kx4mqOZvleMSmP3jCvCXKIyi3lM+w73p6QCigT+h4jApR6SYf8NtMV6AkISShqNjk65ZiC dbvnDkfq2DVP5FmjcK2x0JNDLZsxYw4+UPLUnFICPfo= X-Google-Smtp-Source: AGHT+IGeJY5OXmUZwoThfdaGhHd7hPNTAo0I4gwjygBJvBD/eV6WX3msp4ADqDUqf/IWC3xW7VbDJA== X-Received: by 2002:a05:620a:c4e:b0:7b6:7970:651c with SMTP id af79cd13be357-7be6325fa3emr2028896985a.41.1737382468957; Mon, 20 Jan 2025 06:14:28 -0800 (PST) Received: from ziepe.ca (hlfxns017vw-142-68-128-5.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.128.5]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7be6147f46fsm446601985a.35.2025.01.20.06.14.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Jan 2025 06:14:28 -0800 (PST) Received: from jgg by wakko with local (Exim 4.97) (envelope-from ) id 1tZsXr-00000003Pee-427u; Mon, 20 Jan 2025 10:14:27 -0400 Date: Mon, 20 Jan 2025 10:14:27 -0400 From: Jason Gunthorpe To: Mike Rapoport Cc: lsf-pc@lists.linux-foundation.org, Alexander Graf , "Gowans, James" , linux-mm@kvack.org, David Rientjes , Pasha Tatashin Subject: Re: [LSF/MM/BPF TOPIC] memory persistence over kexec Message-ID: <20250120141427.GK674319@ziepe.ca> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Queue-Id: DF40B40002 X-Rspamd-Server: rspam10 X-Stat-Signature: gjpxucygpqfdynf9gmcj9sx4ssdhxxyc X-HE-Tag: 1737382469-450792 X-HE-Meta: U2FsdGVkX18xMJtuZ0UeDrZt6yl3PQb9IUmnquaj9Sykf/mbUlliwhVQsL6+J+g1z0H1shDmPVHPve3/xSw/RAOTLcUpZ+iU6U7xZImizhRluBNgyHKGvUe8kKaZn2YE4g05Bh/e1g0CQW2aDoH8Sfwu/lMsJwd+OFqyrrBPHLH0v+ktamn6KTE5TOPRQ5fr0DnHrjRUpqioaO5TPb3lmuPFfhoeBuoP0cvK5rKjjK+5q2TY4CWjRto/RqDQhdQFYNAt3X47Fr9AL7wAiLW2lEnhKHnQ34HEhqyx9074SUjfs8NupvfHVhtjcdgHszj+9gLnKmHarWkMERB/f5fz/kmnm3ZaGLjCDx11HeUTtpiVtg0E6o3SntUcV6oaccSk9ymJeaZmNYAzXJDvgoHXHIW5869O40CKaSCmqrlxuZnrxh53obAXfPMqYOQSNnPMGrgREr4F7hv42H3vufXrNw8fZXDBokojH65Bdch8HeAWzOb/MsQ77X+3DGjqYF5Giho95aEgVOP9vIv0LEiNnNOgEg6eYtk1Ac/T4hb5vwjqd8trtbRZR1lzgFDKKR1nAFbiTGYaFhfCQdtlUNHbOmKnhWQxP2HO1ft2Ic7RkablGmA6x5s1GPH9UG3blOkkL9t3z80aMu95ug8mrZL8VCAW2vvpldElUK1EB6+jxr1UYiUFe61B+/DkasGEOGch5fayjvx4LsI/586KKW7gqDht+P07tO8ROQbvVDEBZxGvnKTzwz4zvEv1QphCCeAeG6gXjWdOTL3R6Wo++iaJ3a8nI8UfVtaE0dxTljHuRIypAT0wg531sZQddUl6mpOOOIMVJFfohnbD02/8yruf6lqtRJhqvQTXyL+VVQOSVesx+J5TEXOc62WdEfgkJHAeMs/OdiO7oDZNvpk3oWKQ2vabcGocKBfQZQqZC3qE0591zJb0algbxBDcPeurKIFrU1LHxIruEfSFoOTfxVC hyA1tKoj R/u8fDnqDRKYb5yQq+Zv/vdIe4Vhm9w4AlmKqgdLmwvlaaNRFZDV7OMyHrOTR+vsdjkLYxB62LybMdIx8tlwh+3p16f8PXhpuUd1U7X5HDOKzO1mOTKmDdokprn+rn931e7iPYWO2se+/JhptVA56+K6CqWzOIRB/dWyHddg0QdU5r61fauoxmyYMU5a1nq5tEorxcgjkBZ6PkDi5BI+q8uIThVpqyXXyBjacTpjXtdXIlteTQ+AdjBrDpLV8EHs0vaFuJ5zQidLY92yJ9ZaLnVMhGuPo9t0cvbWM X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Jan 20, 2025 at 09:54:15AM +0200, Mike Rapoport wrote: > Hi, > > I'd like to discuss memory persistence across kexec. > > Currently there is ongoing work on Kexec HandOver (KHO) [1] that allows > serialization and deserialization of kernel data as well as preserving > arbitrary memory ranges across kexec. > > In addition, KHO keeps a physically contiguous memory regions that are > guaranteed to not have any memory that KHO would preserve, but still can be > used by the system. The kexeced kernel bootstraps itself using those > regions and sets all handed over memory as in use. KHO users then can > recover their state from the preserved data. This includes memory > reservations, where the user can either discard or claim reservations. > > KHO can be used as the base layer for implementation of persistence-aware > memory allocator and persistent in-memory filesystem. > > Aside from status update on KHO progress there are a few topics that I would > like to discuss: > * Is it feasible and desirable to enable KHO support in tmpfs and hugetlbfs? > * Or is it better to implement yet another in-memory filesystem dedicated > for persistence? > * What is the best way to ensure that the memory we want to persist is not > scattered all over the place? There is alot of talk about taking *drivers* and having them survive kexec, meaning the driver has to put alot of its state into KHO and then get it back out again. I've been hoping for a model where a driver can be told to "go to KHO" and the KHO code can be largely contained in the driver and regulated to recording the driver state. This implies the state may be fragmented all over memory. The other direction is that the driver has to start up in some special KHO mode and KHO becomes invasive on all driver paths to use special KHO allocations. This seems like a PITA. You can see this difference just in the discussion around the iommu serialization where one idea was to have KHO be an integral (and invasive!) part of the page table operations from time zero vs some later serialization at kexec time. Regardless, I'm interested in this discussion to bring some concreteness about how drivers work.. Jason