Skip to contentSkip to conversation

arm64: Support for running as a guest in Arm CCA

PATCH Superseded Steven Price 9 participants 54 messages Jun 5, 2024 – Jun 28, 2024

Subsystems (13)
Memory management KVM MM core arm64 +9 more subsystems

v3 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v3 cover arm64: Support for running as a guest in Arm CCA

This series adds support for running Linux in a protected VM under the Arm Confidential Compute Architecture (CCA). This has been updated following the feedback from the v2 posting[1]. Thanks for the feedback! Individual patches have a change log for v3.

The biggest change from v2 is fixing set_memory_{en,de}crypted() to perform a break-before-make sequence. Note that only the virtual address supplied is flipped between shared and protected, so if e.g. a vmalloc() address is passed the linear map will still point to the (now invalid) previous IPA. Attempts to access the wrong address may trigger a Synchronous External Abort. However any code which attempts to access the 'encrypted' alias after set_memory_decrypted() is already likely to be broken on platforms that implement memory encryption, so I don't expect problems.

The ABI to the RMM from a realm (the RSI) is based on the final RMM v1.0 (EAC 5) specification[2]. Future RMM specifications will be backwards compatible so a guest using the v1.0 specification (i.e. this series) will be able to run on future versions of the RMM without modification.

Arm plans to set up a CI system to perform at a minimum boot testing of Linux as a guest within a realm.

This series is based on v6.10-rc1. It is also available as a git repository:

https://gitlab.arm.com/linux-arm/linux-cca cca-guest/v3

This series (the guest side) should be in a good state so please review with the intention that this could be merged soon. The host side (KVM changes) is likely to require some more iteration and I'll post that as a separate series shortly - note that there is no tie between the series (i.e. you can mix and match v2 and v3 postings of the host and guest).

Introduction (unchanged from v2 posting)
============
A more general introduction to Arm CCA is available on the Arm website[3], and links to the other components involved are available in the overall cover letter.

Arm Confidential Compute Architecture adds two new 'worlds' to the
architecture: Root and Realm. A new software component known as the RMM (Realm Management Monitor) runs in Realm EL2 and is trusted by both the Normal World and VMs running within Realms. This enables mutual distrust between the Realm VMs and the Normal World.

Virtual machines running within a Realm can decide on a (4k) page-by-page granularity whether to share a page with the (Normal World) host or to keep it private (protected). This protection is provided by the hardware and attempts to access a page which isn't shared by the Normal World will trigger a Granule Protection Fault.

Realm VMs can communicate with the RMM via another SMC interface known as RSI (Realm Services Interface). This series adds wrappers for the full set of RSI commands and uses them to manage the Realm IPA State (RIPAS) and to discover the configuration of the realm.

The VM running within the Realm needs to ensure that memory that is going to use is marked as 'RIPAS_RAM' (i.e. protected memory accessible only to the guest). This could be provided by the VMM (and subject to measurement to ensure it is setup correctly) or the VM can set it itself. This series includes a patch which will iterate over all described RAM and set the RIPAS. This is a relatively cheap operation, and doesn't require memory donation from the host. Instead, memory can be dynamically provided by the host on fault. An alternative would be to update booting.rst and state this as a requirement, but this would reduce the flexibility of the VMM to manage the available memory to the guest (as the initial RIPAS state is part of the guest's measurement).

Within the Realm the most-significant active bit of the IPA is used to select whether the access is to protected memory or to memory shared with the host. This series treats this bit as if it is attribute bit in the page tables and will modify it when sharing/unsharing memory with the host.

This top bit usage also necessitates that the IPA width is made more dynamic in the guest. The VMM will choose a width (and therefore which bit controls the shared flag) and the guest must be able to identify this bit to mask it out when necessary. PHYS_MASK_SHIFT/PHYS_MASK are therefore made dynamic.

To allow virtio to communicate with the host the shared buffers must be placed in memory which has this top IPA bit set. This is achieved by implementing the set_memory_{encrypted,decrypted} APIs for arm64 and forcing the use of bounce buffers. For now all device access is considered to required the memory to be shared, at this stage there is no support for real devices to be assigned to a realm guest - obviously if device assignment is added this will have to change.

Finally the GIC is (largely) emulated by the (untrusted) host. The RMM provides some management (including register save/restore) but the ITS buffers must be placed into shared memory for the host to emulate. There is likely to be future work to harden the GIC driver against a malicious host (along with any other drivers used within a Realm guest).

[1] https://lore.kernel.org/r/20240412084213.1733764-1-steven.price%40arm.com
[2] https://developer.arm.com/documentation/den0137/1-0eac5/
[3] https://www.arm.com/architecture/security-features/arm-confidential-compute-architecture

Sami Mujawar (2):
arm64: rsi: Interfaces to query attestation token
virt: arm-cca-guest: TSM_REPORT support for realms

Steven Price (5):
arm64: realm: Query IPA size from the RMM
arm64: Mark all I/O as non-secure shared
arm64: Make the PHYS_MASK_SHIFT dynamic
arm64: Enforce bounce buffers for realm DMA
arm64: realm: Support nonsecure ITS emulation shared

Suzuki K Poulose (7):
arm64: rsi: Add RSI definitions
arm64: Detect if in a realm and set RIPAS RAM
fixmap: Allow architecture overriding set_fixmap_io
arm64: Override set_fixmap_io
arm64: Enable memory encrypt for Realms
arm64: Force device mappings to be non-secure shared
efi: arm64: Map Device with Prot Shared

 arch/arm64/Kconfig                            |   3 +
 arch/arm64/include/asm/fixmap.h               |   4 +-
 arch/arm64/include/asm/io.h                   |   6 +-
 arch/arm64/include/asm/mem_encrypt.h          |  17 ++
 arch/arm64/include/asm/pgtable-hwdef.h        |   6 -
 arch/arm64/include/asm/pgtable-prot.h         |   3 +
 arch/arm64/include/asm/pgtable.h              |   7 +-
 arch/arm64/include/asm/rsi.h                  |  48 ++++
 arch/arm64/include/asm/rsi_cmds.h             | 143 ++++++++++++
 arch/arm64/include/asm/rsi_smc.h              | 142 ++++++++++++
 arch/arm64/include/asm/set_memory.h           |   3 +
 arch/arm64/kernel/Makefile                    |   3 +-
 arch/arm64/kernel/efi.c                       |   2 +-
 arch/arm64/kernel/rsi.c                       |  96 ++++++++
 arch/arm64/kernel/setup.c                     |   8 +
 arch/arm64/mm/init.c                          |  10 +-
 arch/arm64/mm/mmu.c                           |  13 ++
 arch/arm64/mm/pageattr.c                      |  65 +++++-
 drivers/irqchip/irq-gic-v3-its.c              |  90 ++++++--
 drivers/virt/coco/Kconfig                     |   2 +
 drivers/virt/coco/Makefile                    |   1 +
 drivers/virt/coco/arm-cca-guest/Kconfig       |  11 +
 drivers/virt/coco/arm-cca-guest/Makefile      |   2 +
 .../virt/coco/arm-cca-guest/arm-cca-guest.c   | 211 ++++++++++++++++++
 include/asm-generic/fixmap.h                  |   2 +
 25 files changed, 858 insertions(+), 40 deletions(-)
 create mode 100644 arch/arm64/include/asm/mem_encrypt.h
 create mode 100644 arch/arm64/include/asm/rsi.h
 create mode 100644 arch/arm64/include/asm/rsi_cmds.h
 create mode 100644 arch/arm64/include/asm/rsi_smc.h
 create mode 100644 arch/arm64/kernel/rsi.c
 create mode 100644 drivers/virt/coco/arm-cca-guest/Kconfig
 create mode 100644 drivers/virt/coco/arm-cca-guest/Makefile
 create mode 100644 drivers/virt/coco/arm-cca-guest/arm-cca-guest.c
Signature · 1 line
2.34.1
#1 lore raw

PATCH v3 1/14 arm64: rsi: Add RSI definitions

From: Suzuki K Poulose <[email protected]>

The RMM (Realm Management Monitor) provides functionality that can be accessed by a realm guest through SMC (Realm Services Interface) calls.

The SMC definitions are based on DEN0137[1] version A-eac5.

[1] https://developer.arm.com/documentation/den0137/latest

---
Changes since v2:
* Rename rsi_get_version() to rsi_request_version()
* Fix size/alignment of struct realm_config

---
 arch/arm64/include/asm/rsi_cmds.h |  47 ++++++++++
 arch/arm64/include/asm/rsi_smc.h  | 142 ++++++++++++++++++++++++++++++
 2 files changed, 189 insertions(+)
 create mode 100644 arch/arm64/include/asm/rsi_cmds.h
 create mode 100644 arch/arm64/include/asm/rsi_smc.h
arch/arm64/include/asm/rsi_cmds.h new file +47 -0
new file mode 100644
index 000000000000..ad425c5d6f1b
@@ -0,0 +1,47 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2023 ARM Ltd.
+ */
+
+#ifndef __ASM_RSI_CMDS_H
+#define __ASM_RSI_CMDS_H
+
+#include <linux/arm-smccc.h>
+
+#include <asm/rsi_smc.h>
+
+static inline void invoke_rsi_fn_smc_with_res(unsigned long function_id,
+					      unsigned long arg0,
+					      unsigned long arg1,
+					      unsigned long arg2,
+					      unsigned long arg3,
+					      struct arm_smccc_res *res)
+{
+	arm_smccc_smc(function_id, arg0, arg1, arg2, arg3, 0, 0, 0, res);
+}
+
+static inline unsigned long rsi_request_version(unsigned long req,
+						unsigned long *out_lower,
+						unsigned long *out_higher)
+{
+	struct arm_smccc_res res;
+
+	invoke_rsi_fn_smc_with_res(SMC_RSI_ABI_VERSION, req, 0, 0, 0, &res);
+
+	if (out_lower)
+		*out_lower = res.a1;
+	if (out_higher)
+		*out_higher = res.a2;
+
+	return res.a0;
+}
+
+static inline unsigned long rsi_get_realm_config(struct realm_config *cfg)
+{
+	struct arm_smccc_res res;
+
+	invoke_rsi_fn_smc_with_res(SMC_RSI_REALM_CONFIG, virt_to_phys(cfg), 0, 0, 0, &res);
+	return res.a0;
+}
+
+#endif
arch/arm64/include/asm/rsi_smc.h new file +142 -0

142 lines not on this page · raw message

Signature · 1 line
2.34.1
maintainer
#2 lore raw

Re: [PATCH v3 01/14] arm64: rsi: Add RSI definitions

Quoted 10 lines · Steven Price
arch/arm64/include/asm/rsi_cmds.h new file +7 -0
@@ -0,0 +1,47 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2023 ARM Ltd.
+ */
+
+#ifndef __ASM_RSI_CMDS_H
+#define __ASM_RSI_CMDS_H

[...]

arch/arm64/include/asm/rsi_smc.h line 7

Quoted 10 lines
arch/arm64/include/asm/rsi_smc.h new file +7 -0
@@ -0,0 +1,142 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2023 ARM Ltd.
+ */
+
+#ifndef __SMC_RSI_H_
+#define __SMC_RSI_H_

A small nitpick if you respin some patches - please make the header guards consistent. We tend to to use the top variant above, so the rsi_smc.h would be __ASM_RSI_SMC_H. The same throughout this series.

Signature · 1 line
Catalin
#3 lore raw

PATCH v3 2/14 arm64: Detect if in a realm and set RIPAS RAM

In reply to Steven Price

From: Suzuki K Poulose <[email protected]>

Detect that the VM is a realm guest by the presence of the RSI interface.

If in a realm then all memory needs to be marked as RIPAS RAM initially, the loader may or may not have done this for us. To be sure iterate over all RAM and mark it as such. Any failure is fatal as that implies the RAM regions passed to Linux are incorrect - which would mean failing later when attempting to access non-existent RAM.

---
Changes since v2:
* Use DECLARE_STATIC_KEY_FALSE rather than "extern struct static_key_false".
* Rename set_memory_range() to rsi_set_memory_range().
* Downgrade some BUG()s to WARN()s and handle the condition by propagating up the stack. Comment the remaining case that ends in a BUG() to explain why.
* Rely on the return from rsi_request_version() rather than checking the version the RMM claims to support.
* Rename the generic sounding arm64_setup_memory() to arm64_rsi_setup_memory() and move the call site to setup_arch().

---
 arch/arm64/include/asm/rsi.h      | 48 +++++++++++++++++++++
 arch/arm64/include/asm/rsi_cmds.h | 22 ++++++++++
 arch/arm64/kernel/Makefile        |  3 +-
 arch/arm64/kernel/rsi.c           | 69 +++++++++++++++++++++++++++++++
 arch/arm64/kernel/setup.c         |  8 ++++
 arch/arm64/mm/init.c              |  1 +
 6 files changed, 150 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/include/asm/rsi.h
 create mode 100644 arch/arm64/kernel/rsi.c
arch/arm64/include/asm/rsi.h new file +48 -0
new file mode 100644
index 000000000000..ce2cdb501d84
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2024 ARM Ltd.
+ */
+
+#ifndef __ASM_RSI_H_
+#define __ASM_RSI_H_
+
+#include <linux/jump_label.h>
+#include <asm/rsi_cmds.h>
+
+DECLARE_STATIC_KEY_FALSE(rsi_present);
+
+void __init arm64_rsi_init(void);
+void __init arm64_rsi_setup_memory(void);
+static inline bool is_realm_world(void)
+{
+	return static_branch_unlikely(&rsi_present);
+}
+
+static inline int rsi_set_memory_range(phys_addr_t start, phys_addr_t end,
+				       enum ripas state)
+{
+	unsigned long ret;
+	phys_addr_t top;
+
+	while (start != end) {
+		ret = rsi_set_addr_range_state(start, end, state, &top);
+		if (WARN_ON(ret || top < start || top > end))
+			return -EINVAL;
+		start = top;
+	}
+
+	return 0;
+}
+
+static inline int rsi_set_memory_range_protected(phys_addr_t start,
+						 phys_addr_t end)
+{
+	return rsi_set_memory_range(start, end, RSI_RIPAS_RAM);
+}
+
+static inline int rsi_set_memory_range_shared(phys_addr_t start,
+					      phys_addr_t end)
+{
+	return rsi_set_memory_range(start, end, RSI_RIPAS_EMPTY);
+}
+#endif
arch/arm64/include/asm/rsi_cmds.h +22 -0
index ad425c5d6f1b..ab8ad435f10e 100644
@@ -10,6 +10,11 @@
 
 #include <asm/rsi_smc.h>
 
+enum ripas {
+	RSI_RIPAS_EMPTY,
+	RSI_RIPAS_RAM,
+};
+
 static inline void invoke_rsi_fn_smc_with_res(unsigned long function_id,
 					      unsigned long arg0,
 					      unsigned long arg1,
@@ -44,4 +49,21 @@ static inline unsigned long rsi_get_realm_config(struct realm_config *cfg)
 	return res.a0;
 }
 
+static inline unsigned long rsi_set_addr_range_state(phys_addr_t start,
+						     phys_addr_t end,
+						     enum ripas state,
+						     phys_addr_t *top)
+{
+	struct arm_smccc_res res;
+
+	invoke_rsi_fn_smc_with_res(SMC_RSI_IPA_STATE_SET,
+				   start, end, state, RSI_NO_CHANGE_DESTROYED,
+				   &res);
+
+	if (top)
+		*top = res.a1;
+
+	return res.a0;
+}
+
 #endif
arch/arm64/kernel/Makefile +2 -1
index 763824963ed1..a483b916ed11 100644
@@ -33,7 +33,8 @@ obj-y			:= debug-monitors.o entry.o irq.o fpsimd.o		\
 			   return_address.o cpuinfo.o cpu_errata.o		\
 			   cpufeature.o alternative.o cacheinfo.o		\
 			   smp.o smp_spin_table.o topology.o smccc-call.o	\
-			   syscall.o proton-pack.o idle.o patching.o pi/
+			   syscall.o proton-pack.o idle.o patching.o pi/	\
+			   rsi.o
 
 obj-$(CONFIG_COMPAT)			+= sys32.o signal32.o			\
 					   sys_compat.o
arch/arm64/kernel/rsi.c new file +69 -0

69 lines not on this page · raw message

arch/arm64/kernel/setup.c +8 -0
index a096e2451044..143f87615af0 100644
@@ -43,6 +43,7 @@
 #include <asm/cpu_ops.h>
 #include <asm/kasan.h>
 #include <asm/numa.h>
+#include <asm/rsi.h>
 #include <asm/scs.h>
 #include <asm/sections.h>
 #include <asm/setup.h>
@@ -293,6 +294,11 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
 	 * cpufeature code and early parameters.
 	 */
 	jump_label_init();
+	/*
+	 * Init RSI before early param so that "earlycon" console uses the
+	 * shared alias when in a realm
+	 */
+	arm64_rsi_init();
 	parse_early_param();
 
 	dynamic_scs_init();
@@ -328,6 +334,8 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
 
 	arm64_memblock_init();
 
+	arm64_rsi_setup_memory();
+
 	paging_init();
 
 	acpi_table_upgrade();
arch/arm64/mm/init.c +1 -0
index 9b5ab6818f7f..9d8d38e3bee2 100644
@@ -41,6 +41,7 @@
 #include <asm/kvm_host.h>
 #include <asm/memory.h>
 #include <asm/numa.h>
+#include <asm/rsi.h>
 #include <asm/sections.h>
 #include <asm/setup.h>
 #include <linux/sizes.h>
Signature · 1 line
2.34.1
maintainer
#4 lore raw

Re: [PATCH v3 02/14] arm64: Detect if in a realm and set RIPAS RAM

arch/arm64/mm/init.c line 47

Quoted 12 lines · Steven Price
arch/arm64/mm/init.c +1 -0
index 9b5ab6818f7f..9d8d38e3bee2 100644
@@ -41,6 +41,7 @@
 #include <asm/kvm_host.h>
 #include <asm/memory.h>
 #include <asm/numa.h>
+#include <asm/rsi.h>
 #include <asm/sections.h>
 #include <asm/setup.h>
 #include <linux/sizes.h>

What's this random include here? It looks like a leftover from the previous version.

Signature · 1 line
Catalin
#5 lore raw

Re: [PATCH v3 02/14] arm64: Detect if in a realm and set RIPAS RAM

Quoted 17 lines · Catalin Marinas

arch/arm64/mm/init.c line 47

Quoted 12 lines · Steven Price
arch/arm64/mm/init.c +1 -0
index 9b5ab6818f7f..9d8d38e3bee2 100644
@@ -41,6 +41,7 @@
 #include <asm/kvm_host.h>
 #include <asm/memory.h>
 #include <asm/numa.h>
+#include <asm/rsi.h>
 #include <asm/sections.h>
 #include <asm/setup.h>
 #include <linux/sizes.h>

What's this random include here? It looks like a leftover from the previous version.

Whoops - indeed that shouldn't be there.

Thanks,
Steve

#6 lore raw

Re: [PATCH v3 02/14] arm64: Detect if in a realm and set RIPAS RAM

In reply to Steven Price
Quoted 15 lines · Steven Price

From: Suzuki K Poulose <[email protected]>

Detect that the VM is a realm guest by the presence of the RSI interface.

If in a realm then all memory needs to be marked as RIPAS RAM initially, the loader may or may not have done this for us. To be sure iterate over all RAM and mark it as such. Any failure is fatal as that implies the RAM regions passed to Linux are incorrect - which would mean failing later when attempting to access non-existent RAM.

Review

Quoted 6 lines
(quoted diff) +6 -0
+static bool rsi_version_matches(void)
+{
+	unsigned long ver_lower, ver_higher;
+	unsigned long ret = rsi_request_version(RSI_ABI_VERSION,
+						&ver_lower,
+						&ver_higher);

There is a regression on QEMU TCG (in emulation mode, not running under KVM):

qemu-system-aarch64 -M virt -cpu max -kernel Image -nographic

This doesn't implement EL3 or EL2, so SMC is UNDEFINED (DDI0487J.a R_HMXQS), and we end up with an undef instruction exception. So this patch would also break hardware that only implements EL1 (I don't know if it exists).

The easiest fix is to detect the SMC conduit through the PSCI node in DT. SMCCC helpers already do this, but we can't use them this early in the boot. I tested adding an early probe to the PSCI driver to check this, see attached patches.

Note that we do need to test the conduit after finding a PSCI node, because even though it doesn't implement EL2 in this configuration, QEMU still accepts PSCI HVCs in order to support SMP.

Thanks,
Jean

From 788bfd45e7ce521666a19dba99277106e4d33c80 Mon Sep 17 00:00:00 2001
From: Jean-Philippe Brucker <[email protected]>
Date: Tue, 11 Jun 2024 19:15:30 +0100
Subject: [PATCH 1/2] firmware/psci: Add psci_early_test_conduit()

Add a function to test early if PSCI is present and what conduit it uses. Because the PSCI conduit corresponds to the SMCCC one, this will let the kernel know whether it can use SMC instructions to discuss with the Realm Management Monitor (RMM), early enough to enable RAM and serial access when running in a Realm.

---
 include/linux/psci.h         |  5 +++++
 drivers/firmware/psci/psci.c | 25 +++++++++++++++++++++++++
 2 files changed, 30 insertions(+)
include/linux/psci.h +5 -0
index 4ca0060a3fc4..a1fc1703ba20 100644
@@ -45,8 +45,13 @@ struct psci_0_1_function_ids get_psci_0_1_function_ids(void);
 
 #if defined(CONFIG_ARM_PSCI_FW)
 int __init psci_dt_init(void);
+bool __init psci_early_test_conduit(enum arm_smccc_conduit conduit);
 #else
 static inline int psci_dt_init(void) { return 0; }
+static inline bool psci_early_test_conduit(enum arm_smccc_conduit conduit)
+{
+	return false;
+}
 #endif
 
 #if defined(CONFIG_ARM_PSCI_FW) && defined(CONFIG_ACPI)
drivers/firmware/psci/psci.c +25 -0
index d9629ff87861..a40dcaf17822 100644
@@ -13,6 +13,7 @@
 #include <linux/errno.h>
 #include <linux/linkage.h>
 #include <linux/of.h>
+#include <linux/of_fdt.h>
 #include <linux/pm.h>
 #include <linux/printk.h>
 #include <linux/psci.h>
@@ -767,6 +768,30 @@ int __init psci_dt_init(void)
 	return ret;
 }
 
+/*
+ * Test early if PSCI is supported, and if its conduit matches @conduit
+ */
+bool __init psci_early_test_conduit(enum arm_smccc_conduit conduit)
+{
+	int len;
+	int psci_node;
+	const char *method;
+	unsigned long dt_root;
+
+	/* DT hasn't been unflattened yet, we have to work with the flat blob */
+	dt_root = of_get_flat_dt_root();
+	psci_node = of_get_flat_dt_subnode_by_name(dt_root, "psci");
+	if (psci_node <= 0)
+		return false;
+
+	method = of_get_flat_dt_prop(psci_node, "method", &len);
+	if (!method)
+		return false;
+
+	return  (conduit == SMCCC_CONDUIT_SMC && strncmp(method, "smc", len) == 0) ||
+		(conduit == SMCCC_CONDUIT_HVC && strncmp(method, "hvc", len) == 0);
+}
+
 #ifdef CONFIG_ACPI
 /*
  * We use PSCI 0.2+ when ACPI is deployed on ARM64 and it's
Signature · 1 line
2.45.2

From fcb16e1eb494d2ce21792495955d6d3f26c319c9 Mon Sep 17 00:00:00 2001
From: Jean-Philippe Brucker <[email protected]>
Date: Wed, 12 Jun 2024 09:08:17 +0100
Subject: [PATCH 2/2] squash! arm64: Detect if in a realm and set RIPAS RAM

Before issuing an SMC, detect whether SMCCC is available through this conduit. On platforms that do not implement EL3 nor EL2, the SMC instruction is UNDEFINED. SMCCC advises probing the SMCCC availability by first looking for a PSCI node in DT (or in ACPI, but we expect a realm to boot with DT at the moment). Since RMM requires using the SMC conduit for both PSCI and RSI, we rely on the PSCI method property to ensure SMC is available. We could also check that the SMCCC version is at least 1.2 as required by RMM, but no platform requires this extra step at the moment, and we can't use the SMCCC helpers this early in boot.

---
 arch/arm64/kernel/rsi.c | 7 +++++++
 1 file changed, 7 insertions(+)
arch/arm64/kernel/rsi.c +7 -0
index 898952d135b0..21fc261a1d26 100644
@@ -7,6 +7,7 @@
 #include <linux/memblock.h>
 #include <linux/swiotlb.h>
 #include <linux/cc_platform.h>
+#include <linux/psci.h>
 
 #include <asm/rsi.h>
 
@@ -82,6 +83,12 @@ void __init arm64_rsi_setup_memory(void)
 
 void __init arm64_rsi_init(void)
 {
+	/*
+	 * If PSCI isn't using SMC, RMM isn't present. Don't try to execute an
+	 * SMC as it could be UNDEFINED.
+	 */
+	if (!psci_early_test_conduit(SMCCC_CONDUIT_SMC))
+		return;
 	if (!rsi_version_matches())
 		return;
 	if (rsi_get_realm_config(&config))
Signature · 1 line
2.45.2
#7 lore raw

Re: [PATCH v3 02/14] arm64: Detect if in a realm and set RIPAS RAM

Quoted 25 lines · Jean-Philippe Brucker
Quoted 15 lines · Steven Price

From: Suzuki K Poulose <[email protected]>

Detect that the VM is a realm guest by the presence of the RSI interface.

If in a realm then all memory needs to be marked as RIPAS RAM initially, the loader may or may not have done this for us. To be sure iterate over all RAM and mark it as such. Any failure is fatal as that implies the RAM regions passed to Linux are incorrect - which would mean failing later when attempting to access non-existent RAM.

Review

Quoted 6 lines
(quoted diff) +6 -0
+static bool rsi_version_matches(void)
+{
+	unsigned long ver_lower, ver_higher;
+	unsigned long ret = rsi_request_version(RSI_ABI_VERSION,
+						&ver_lower,
+						&ver_higher);

There is a regression on QEMU TCG (in emulation mode, not running under KVM):

Quoted 5 lines

qemu-system-aarch64 -M virt -cpu max -kernel Image -nographic

This doesn't implement EL3 or EL2, so SMC is UNDEFINED (DDI0487J.a R_HMXQS), and we end up with an undef instruction exception. So this patch would also break hardware that only implements EL1 (I don't know if it exists).

Thanks for the report, Could we not check ID_AA64PFR0_EL1.EL3 >= 0 ? I think we do this for kvm-unit-tests, we need the same here.

Suzuki

Quoted 13 lines

The easiest fix is to detect the SMC conduit through the PSCI node in DT. SMCCC helpers already do this, but we can't use them this early in the boot. I tested adding an early probe to the PSCI driver to check this, see attached patches.

Note that we do need to test the conduit after finding a PSCI node, because even though it doesn't implement EL2 in this configuration, QEMU still accepts PSCI HVCs in order to support SMP.

Thanks,
Jean

#8 lore raw

Re: [PATCH v3 02/14] arm64: Detect if in a realm and set RIPAS RAM

Quoted 26 lines · Suzuki K Poulose
Quoted 25 lines · Jean-Philippe Brucker
Quoted 15 lines · Steven Price

From: Suzuki K Poulose <[email protected]>

Detect that the VM is a realm guest by the presence of the RSI interface.

If in a realm then all memory needs to be marked as RIPAS RAM initially, the loader may or may not have done this for us. To be sure iterate over all RAM and mark it as such. Any failure is fatal as that implies the RAM regions passed to Linux are incorrect - which would mean failing later when attempting to access non-existent RAM.

Review

Quoted 6 lines
(quoted diff) +6 -0
+static bool rsi_version_matches(void)
+{
+	unsigned long ver_lower, ver_higher;
+	unsigned long ret = rsi_request_version(RSI_ABI_VERSION,
+						&ver_lower,
+						&ver_higher);

There is a regression on QEMU TCG (in emulation mode, not running under KVM):

Quoted 8 lines
Quoted 5 lines

qemu-system-aarch64 -M virt -cpu max -kernel Image -nographic

This doesn't implement EL3 or EL2, so SMC is UNDEFINED (DDI0487J.a R_HMXQS), and we end up with an undef instruction exception. So this patch would also break hardware that only implements EL1 (I don't know if it exists).

Thanks for the report, Could we not check ID_AA64PFR0_EL1.EL3 >= 0 ? I think we do this for kvm-unit-tests, we need the same here.

Good point, it also fixes this case and is simpler. It assumes RMM doesn't hide this field, but I can't think of a reason it would.

This command won't work anymore:

qemu-system-aarch64 -M virt,secure=on -cpu max -kernel Image -nographic

implements EL3 and SMC still treated as undef. QEMU has a special case for starting at EL2 in this case, but I couldn't find what this is for.

Treating SMC as undef is correct because SCR_EL3.SMD resets to an architectutally UNKNOWN value. But the architecture requires that the CPU resets to the highest implemented exception level (DDI0487J.a R_JYLQV). So in my opinion we can use the ID_AA64PFR0_EL1.EL3 field here, and breaking this particular configuration is not a problem: users shouldn't expect Linux to boot when EL3 is implemented and doesn't run a firmware.

Thanks,
Jean

Quoted 18 lines

Suzuki

Quoted 13 lines

The easiest fix is to detect the SMC conduit through the PSCI node in DT. SMCCC helpers already do this, but we can't use them this early in the boot. I tested adding an early probe to the PSCI driver to check this, see attached patches.

Note that we do need to test the conduit after finding a PSCI node, because even though it doesn't implement EL2 in this configuration, QEMU still accepts PSCI HVCs in order to support SMP.

Thanks,
Jean

#9 lore raw

Re: [PATCH v3 02/14] arm64: Detect if in a realm and set RIPAS RAM

Quoted 40 lines · Jean-Philippe Brucker
Quoted 26 lines · Suzuki K Poulose
Quoted 25 lines · Jean-Philippe Brucker
Quoted 15 lines · Steven Price

From: Suzuki K Poulose <[email protected]>

Detect that the VM is a realm guest by the presence of the RSI interface.

If in a realm then all memory needs to be marked as RIPAS RAM initially, the loader may or may not have done this for us. To be sure iterate over all RAM and mark it as such. Any failure is fatal as that implies the RAM regions passed to Linux are incorrect - which would mean failing later when attempting to access non-existent RAM.

Review

Quoted 6 lines
(quoted diff) +6 -0
+static bool rsi_version_matches(void)
+{
+ unsigned long ver_lower, ver_higher;
+ unsigned long ret = rsi_request_version(RSI_ABI_VERSION,
+                                         &ver_lower,
+                                         &ver_higher);

There is a regression on QEMU TCG (in emulation mode, not running under KVM):

Quoted 8 lines
Quoted 5 lines

qemu-system-aarch64 -M virt -cpu max -kernel Image -nographic

This doesn't implement EL3 or EL2, so SMC is UNDEFINED (DDI0487J.a R_HMXQS), and we end up with an undef instruction exception. So this patch would also break hardware that only implements EL1 (I don't know if it exists).

Thanks for the report, Could we not check ID_AA64PFR0_EL1.EL3 >= 0 ? I think we do this for kvm-unit-tests, we need the same here.

Good point, it also fixes this case and is simpler. It assumes RMM doesn't hide this field, but I can't think of a reason it would.

Quoted 6 lines

This command won't work anymore:

qemu-system-aarch64 -M virt,secure=on -cpu max -kernel Image -nographic

implements EL3 and SMC still treated as undef. QEMU has a special case for starting at EL2 in this case, but I couldn't find what this is for.

That's a bit of an odd config, because it says "emulate EL3 but never use it". QEMU's boot loader starts the kernel at EL2 because the kernel boot protocol requires that (this is more relevant on boards other than virt where EL3 is not command-line disableable). I have a feeling we've occasionally found that somebody's had some corner case reason to use it, though. (eg
https://gitlab.com/qemu-project/qemu/-/issues/1899
is from somebody who says they use this when booting Windows 11 because it asserts at boot time that EL3 is present and won't boot otherwise.)

Your underlying problem here seems to be that you don't have a way for the firmware to say "hey, SMC works, you can use it" ?

-- PMM

#10 lore raw

Re: [PATCH v3 02/14] arm64: Detect if in a realm and set RIPAS RAM

Quoted 49 lines · Peter Maydell
Quoted 40 lines · Jean-Philippe Brucker
Quoted 26 lines · Suzuki K Poulose
Quoted 25 lines · Jean-Philippe Brucker
Quoted 15 lines · Steven Price

From: Suzuki K Poulose <[email protected]>

Detect that the VM is a realm guest by the presence of the RSI interface.

If in a realm then all memory needs to be marked as RIPAS RAM initially, the loader may or may not have done this for us. To be sure iterate over all RAM and mark it as such. Any failure is fatal as that implies the RAM regions passed to Linux are incorrect - which would mean failing later when attempting to access non-existent RAM.

Review

Quoted 6 lines
(quoted diff) +6 -0
+static bool rsi_version_matches(void)
+{
+ unsigned long ver_lower, ver_higher;
+ unsigned long ret = rsi_request_version(RSI_ABI_VERSION,
+                                         &ver_lower,
+                                         &ver_higher);

There is a regression on QEMU TCG (in emulation mode, not running under KVM):

Quoted 8 lines
Quoted 5 lines

qemu-system-aarch64 -M virt -cpu max -kernel Image -nographic

This doesn't implement EL3 or EL2, so SMC is UNDEFINED (DDI0487J.a R_HMXQS), and we end up with an undef instruction exception. So this patch would also break hardware that only implements EL1 (I don't know if it exists).

Thanks for the report, Could we not check ID_AA64PFR0_EL1.EL3 >= 0 ? I think we do this for kvm-unit-tests, we need the same here.

Good point, it also fixes this case and is simpler. It assumes RMM doesn't hide this field, but I can't think of a reason it would.

Quoted 6 lines

This command won't work anymore:

qemu-system-aarch64 -M virt,secure=on -cpu max -kernel Image -nographic

implements EL3 and SMC still treated as undef. QEMU has a special case for starting at EL2 in this case, but I couldn't find what this is for.

Quoted 9 lines

That's a bit of an odd config, because it says "emulate EL3 but never use it". QEMU's boot loader starts the kernel at EL2 because the kernel boot protocol requires that (this is more relevant on boards other than virt where EL3 is not command-line disableable). I have a feeling we've occasionally found that somebody's had some corner case reason to use it, though. (eg
https://gitlab.com/qemu-project/qemu/-/issues/1899
is from somebody who says they use this when booting Windows 11 because it asserts at boot time that EL3 is present and won't boot otherwise.)

Thanks for the pointer. In this case it looks like Linux was used as reproducer and not the main use-case, though I wonder if some CIs regularly boot this particular configuration.

Quoted 3 lines

Your underlying problem here seems to be that you don't have a way for the firmware to say "hey, SMC works, you can use it" ?

We do: SMCCC recommends to look at the PSCI conduit declared in DT/ACPI. Given that RMM mandates using the SMC conduit for both PSCI and RSI calls, we could use this discovery mechanism here. The problem is that we have to discover it very early at boot, before the DT infrastructure is ready, so the implementation is a little awkward. I did post one earlier in this thread but it doesn't yet account for ACPI-only boot, which will need something similar. Testing ID_AA64PFR0_EL1.EL3 would be much simpler, but it would break this config.

Thanks,
Jean

#11 lore raw

Re: [PATCH v3 02/14] arm64: Detect if in a realm and set RIPAS RAM

In reply to Jean-Philippe Brucker

Hi,

Quoted 25 lines · Jean-Philippe Brucker
Quoted 15 lines · Steven Price

From: Suzuki K Poulose <[email protected]>

Detect that the VM is a realm guest by the presence of the RSI interface.

If in a realm then all memory needs to be marked as RIPAS RAM initially, the loader may or may not have done this for us. To be sure iterate over all RAM and mark it as such. Any failure is fatal as that implies the RAM regions passed to Linux are incorrect - which would mean failing later when attempting to access non-existent RAM.

Review

Quoted 6 lines
(quoted diff) +6 -0
+static bool rsi_version_matches(void)
+{
+	unsigned long ver_lower, ver_higher;
+	unsigned long ret = rsi_request_version(RSI_ABI_VERSION,
+						&ver_lower,
+						&ver_higher);

There is a regression on QEMU TCG (in emulation mode, not running under KVM):

Quoted 5 lines

qemu-system-aarch64 -M virt -cpu max -kernel Image -nographic

This doesn't implement EL3 or EL2, so SMC is UNDEFINED (DDI0487J.a R_HMXQS), and we end up with an undef instruction exception. So this patch would also break hardware that only implements EL1 (I don't know if it exists).

To note: i've found out the hard way this set breaks a qemu+kvm+ACPI setup as well, for roughly the same reason. I imagine we want kernels which can boot in either a realm or a normal guest.

I delayed the version check a bit and then, did enough that arm_smcccc_1_1_invoke() could replace arm_smccc_smc() in invoke_rsi_fn_smc_with_res(). Which naturally gets it booting again, the larger implications i've not considered yet.

Quoted 13 lines

The easiest fix is to detect the SMC conduit through the PSCI node in DT. SMCCC helpers already do this, but we can't use them this early in the boot. I tested adding an early probe to the PSCI driver to check this, see attached patches.

Note that we do need to test the conduit after finding a PSCI node, because even though it doesn't implement EL2 in this configuration, QEMU still accepts PSCI HVCs in order to support SMP.

Thanks,
Jean

#12 lore raw

Re: [PATCH v3 02/14] arm64: Detect if in a realm and set RIPAS RAM

In reply to Steven Price

Hi Steven

Quoted 110 lines · Steven Price

From: Suzuki K Poulose <[email protected]>

Detect that the VM is a realm guest by the presence of the RSI interface.

If in a realm then all memory needs to be marked as RIPAS RAM initially, the loader may or may not have done this for us. To be sure iterate over all RAM and mark it as such. Any failure is fatal as that implies the RAM regions passed to Linux are incorrect - which would mean failing later when attempting to access non-existent RAM.

---
Changes since v2:
* Use DECLARE_STATIC_KEY_FALSE rather than "extern struct static_key_false".
* Rename set_memory_range() to rsi_set_memory_range().
* Downgrade some BUG()s to WARN()s and handle the condition by propagating up the stack. Comment the remaining case that ends in a BUG() to explain why.
* Rely on the return from rsi_request_version() rather than checking the version the RMM claims to support.
* Rename the generic sounding arm64_setup_memory() to arm64_rsi_setup_memory() and move the call site to setup_arch().

---
  arch/arm64/include/asm/rsi.h      | 48 +++++++++++++++++++++
  arch/arm64/include/asm/rsi_cmds.h | 22 ++++++++++
  arch/arm64/kernel/Makefile        |  3 +-
  arch/arm64/kernel/rsi.c           | 69 +++++++++++++++++++++++++++++++
  arch/arm64/kernel/setup.c         |  8 ++++
  arch/arm64/mm/init.c              |  1 +
  6 files changed, 150 insertions(+), 1 deletion(-)

create mode 100644 arch/arm64/include/asm/rsi.h
create mode 100644 arch/arm64/kernel/rsi.c

arch/arm64/include/asm/rsi.h new file +48 -0
new file mode 100644
index 000000000000..ce2cdb501d84
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2024 ARM Ltd.
+ */
+
+#ifndef __ASM_RSI_H_
+#define __ASM_RSI_H_
+
+#include <linux/jump_label.h>
+#include <asm/rsi_cmds.h>
+
+DECLARE_STATIC_KEY_FALSE(rsi_present);
+
+void __init arm64_rsi_init(void);
+void __init arm64_rsi_setup_memory(void);
+static inline bool is_realm_world(void)
+{
+	return static_branch_unlikely(&rsi_present);
+}
+
+static inline int rsi_set_memory_range(phys_addr_t start, phys_addr_t end,
+				       enum ripas state)
+{
+	unsigned long ret;
+	phys_addr_t top;
+
+	while (start != end) {
+		ret = rsi_set_addr_range_state(start, end, state, &top);
+		if (WARN_ON(ret || top < start || top > end))
+			return -EINVAL;
+		start = top;
+	}
+
+	return 0;
+}
+
+static inline int rsi_set_memory_range_protected(phys_addr_t start,
+						 phys_addr_t end)
+{
+	return rsi_set_memory_range(start, end, RSI_RIPAS_RAM);
+}
+
+static inline int rsi_set_memory_range_shared(phys_addr_t start,
+					      phys_addr_t end)
+{
+	return rsi_set_memory_range(start, end, RSI_RIPAS_EMPTY);
+}
+#endif
arch/arm64/include/asm/rsi_cmds.h +5 -0
index ad425c5d6f1b..ab8ad435f10e 100644
@@ -10,6 +10,11 @@
 
  #include <asm/rsi_smc.h>
 
+enum ripas {
+	RSI_RIPAS_EMPTY,
+	RSI_RIPAS_RAM,
+};
+
  static inline void invoke_rsi_fn_smc_with_res(unsigned long function_id,
  					      unsigned long arg0,
  					      unsigned long arg1,
@@ -44,4 +49,21 @@ static inline unsigned long rsi_get_realm_config(struct realm_config *cfg)
  	return res.a0;
  }

arch/arm64/include/asm/rsi_cmds.h static inline unsigned long rsi_get_realm_config(struct realm_config *cfg) · line 60

Quoted 10 lines
arch/arm64/include/asm/rsi_cmds.h +9 -0
 
+static inline unsigned long rsi_set_addr_range_state(phys_addr_t start,
+						     phys_addr_t end,
+						     enum ripas state,
+						     phys_addr_t *top)
+{
+	struct arm_smccc_res res;
+
+	invoke_rsi_fn_smc_with_res(SMC_RSI_IPA_STATE_SET,
+				   start, end, state, RSI_NO_CHANGE_DESTROYED,

Though this is fine from the KVM as NS Host perspective, it may be unnecessarily restrictive in general for a Host implementation. We only need that RSI_NO_CHANGE_DESTROYED flag for "init all DRAM range as RAM"
(where we want to prevent a host from "destroying pages" that were
populated before activation, without consent). But in all other cases
where we do not rely on the content of the "newly" encrypted page,
we could drop the flag.

I think we need could have variants of this helper one which allows "DESTROYED" granules to be converted, which must be only used while "transitioning" a page to encrypted, where we don't rely on the contents of the page.

Something like :

rsi_set_memory_range_protected_safe() : Do not allow DESTROYED contents to be converted.

rsi_set_memory_range_protected().

Something like:

---8>---

arch/arm64/include/asm/rsi.h +18 -4
index ce2cdb501d84..dea2ed99f6d1 100644
@@ -19,13 +19,13 @@ static inline bool is_realm_world(void)
 }
 
 static inline int rsi_set_memory_range(phys_addr_t start, phys_addr_t end,
-				       enum ripas state)
+				       enum ripas state, unsigned long flags)
 {
 	unsigned long ret;
 	phys_addr_t top;
 
 	while (start != end) {
-		ret = rsi_set_addr_range_state(start, end, state, &top);
+		ret = rsi_set_addr_range_state(start, end, state, flags, &top);
 		if (WARN_ON(ret || top < start || top > end))
 			return -EINVAL;
 		start = top;
@@ -34,15 +34,29 @@ static inline int rsi_set_memory_range(phys_addr_t start, phys_addr_t end,
 	return 0;
 }
 
+/*
+ * Convert the specified range to RAM. Do not use this if you rely on the contents
+ * of a page that may already be in RAM state.
+ */
 static inline int rsi_set_memory_range_protected(phys_addr_t start,
 						 phys_addr_t end)
 {
-	return rsi_set_memory_range(start, end, RSI_RIPAS_RAM);
+	return rsi_set_memory_range(start, end, RSI_RIPAS_RAM, 0);
+}
+
+/*
+ * Convert the specified range to RAM. Do not convert any pages that may have
+ * been DESTROYED, without our permission.
+ */
+static inline int rsi_set_memory_range_protected_safe(phys_addr_t start,
+						      phys_addr_t end)
+{
+	return rsi_set_memory_range(start, end, RSI_RIPAS_RAM, RSI_NO_CHANGE_DESTROYED);
 }
 
 static inline int rsi_set_memory_range_shared(phys_addr_t start,
 					      phys_addr_t end)
 {
-	return rsi_set_memory_range(start, end, RSI_RIPAS_EMPTY);
+	return rsi_set_memory_range(start, end, RSI_RIPAS_EMPTY, 0);
 }
 #endif
arch/arm64/include/asm/rsi_cmds.h +2 -1
index ab8ad435f10e..466615ff90de 100644
@@ -52,12 +52,13 @@ static inline unsigned long rsi_get_realm_config(struct realm_config *cfg)
 static inline unsigned long rsi_set_addr_range_state(phys_addr_t start,
 						     phys_addr_t end,
 						     enum ripas state,
+						     unsigned long flags,
 						     phys_addr_t *top)
 {
 	struct arm_smccc_res res;
 
 	invoke_rsi_fn_smc_with_res(SMC_RSI_IPA_STATE_SET,
-				   start, end, state, RSI_NO_CHANGE_DESTROYED,
+				   start, end, state, flags,
 				   &res);
 
 	if (top)
arch/arm64/kernel/rsi.c +4 -3
index 3a992bdfd6bb..e6a6681524a0 100644
@@ -46,8 +46,9 @@ void __init arm64_rsi_setup_memory(void)
 		return;
 
 	/*
-	 * Iterate over the available memory ranges
-	 * and convert the state to protected memory.
+	 * Iterate over the available memory ranges and convert the state to
+	 * protected memory. We should take extra care to ensure that we DO NOT
+	 * permit any "DESTROYED" pages to be converted to "RAM".
 	 *
 	 * BUG_ON is used because if the attempt to switch the memory to
 	 * protected has failed here, then future accesses to the memory are
@@ -55,7 +56,7 @@ void __init arm64_rsi_setup_memory(void)
 	 * Bailing out early prevents the guest limping on and dieing later.
 	 */
 	for_each_mem_range(i, &start, &end) {
-		BUG_ON(rsi_set_memory_range_protected(start, end));
+		BUG_ON(rsi_set_memory_range_protected_safe(start, end));
 	}
 }
 

Kind regards

Suzuki

Quoted 143 lines
arch/arm64/include/asm/rsi_cmds.h +8 -0
+				   &res);
+
+	if (top)
+		*top = res.a1;
+
+	return res.a0;
+}
+
  #endif
arch/arm64/kernel/Makefile +2 -1
index 763824963ed1..a483b916ed11 100644
@@ -33,7 +33,8 @@ obj-y			:= debug-monitors.o entry.o irq.o fpsimd.o		\
  			   return_address.o cpuinfo.o cpu_errata.o		\
  			   cpufeature.o alternative.o cacheinfo.o		\
  			   smp.o smp_spin_table.o topology.o smccc-call.o	\
-			   syscall.o proton-pack.o idle.o patching.o pi/
+			   syscall.o proton-pack.o idle.o patching.o pi/	\
+			   rsi.o
 
  obj-$(CONFIG_COMPAT)			+= sys32.o signal32.o			\
  					   sys_compat.o
arch/arm64/kernel/rsi.c new file +69 -0
new file mode 100644
index 000000000000..3a992bdfd6bb
@@ -0,0 +1,69 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2023 ARM Ltd.
+ */
+
+#include <linux/jump_label.h>
+#include <linux/memblock.h>
+#include <asm/rsi.h>
+
+DEFINE_STATIC_KEY_FALSE_RO(rsi_present);
+EXPORT_SYMBOL(rsi_present);
+
+static bool rsi_version_matches(void)
+{
+	unsigned long ver_lower, ver_higher;
+	unsigned long ret = rsi_request_version(RSI_ABI_VERSION,
+						&ver_lower,
+						&ver_higher);
+
+	if (ret == SMCCC_RET_NOT_SUPPORTED)
+		return false;
+
+	if (ret != RSI_SUCCESS) {
+		pr_err("RME: RMM doesn't support RSI version %u.%u. Supported range: %lu.%lu-%lu.%lu\n",
+		       RSI_ABI_VERSION_MAJOR, RSI_ABI_VERSION_MINOR,
+		       RSI_ABI_VERSION_GET_MAJOR(ver_lower),
+		       RSI_ABI_VERSION_GET_MINOR(ver_lower),
+		       RSI_ABI_VERSION_GET_MAJOR(ver_higher),
+		       RSI_ABI_VERSION_GET_MINOR(ver_higher));
+		return false;
+	}
+
+	pr_info("RME: Using RSI version %lu.%lu\n",
+		RSI_ABI_VERSION_GET_MAJOR(ver_lower),
+		RSI_ABI_VERSION_GET_MINOR(ver_lower));
+
+	return true;
+}
+
+void __init arm64_rsi_setup_memory(void)
+{
+	u64 i;
+	phys_addr_t start, end;
+
+	if (!is_realm_world())
+		return;
+
+	/*
+	 * Iterate over the available memory ranges
+	 * and convert the state to protected memory.
+	 *
+	 * BUG_ON is used because if the attempt to switch the memory to
+	 * protected has failed here, then future accesses to the memory are
+	 * simply going to be reflected as a fault which we can't handle.
+	 * Bailing out early prevents the guest limping on and dieing later.
+	 */
+	for_each_mem_range(i, &start, &end) {
+		BUG_ON(rsi_set_memory_range_protected(start, end));
+	}
+}
+
+void __init arm64_rsi_init(void)
+{
+	if (!rsi_version_matches())
+		return;
+
+	static_branch_enable(&rsi_present);
+}
+
arch/arm64/kernel/setup.c +8 -0
index a096e2451044..143f87615af0 100644
@@ -43,6 +43,7 @@
  #include <asm/cpu_ops.h>
  #include <asm/kasan.h>
  #include <asm/numa.h>
+#include <asm/rsi.h>
  #include <asm/scs.h>
  #include <asm/sections.h>
  #include <asm/setup.h>
@@ -293,6 +294,11 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
  	 * cpufeature code and early parameters.
  	 */
  	jump_label_init();
+	/*
+	 * Init RSI before early param so that "earlycon" console uses the
+	 * shared alias when in a realm
+	 */
+	arm64_rsi_init();
  	parse_early_param();
 
  	dynamic_scs_init();
@@ -328,6 +334,8 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
 
  	arm64_memblock_init();
 
+	arm64_rsi_setup_memory();
+
  	paging_init();
 
  	acpi_table_upgrade();
arch/arm64/mm/init.c +1 -0
index 9b5ab6818f7f..9d8d38e3bee2 100644
@@ -41,6 +41,7 @@
  #include <asm/kvm_host.h>
  #include <asm/memory.h>
  #include <asm/numa.h>
+#include <asm/rsi.h>
  #include <asm/sections.h>
  #include <asm/setup.h>
  #include <linux/sizes.h>
#13 lore raw

PATCH v3 3/14 arm64: realm: Query IPA size from the RMM

In reply to Steven Price

The top bit of the configured IPA size is used as an attribute to control whether the address is protected or shared. Query the configuration from the RMM to assertain which bit this is.

---
Changes since v2:
* Drop unneeded extra brackets from PROT_NS_SHARED.
* Drop the explicit alignment from 'config' as struct realm_config now specifies the alignment.

---
 arch/arm64/include/asm/pgtable-prot.h | 3 +++
 arch/arm64/kernel/rsi.c               | 8 ++++++++
 2 files changed, 11 insertions(+)
arch/arm64/include/asm/pgtable-prot.h +3 -0
index b11cfb9fdd37..6c29f3b32eba 100644
@@ -70,6 +70,9 @@
 #include <asm/pgtable-types.h>
 
 extern bool arm64_use_ng_mappings;
+extern unsigned long prot_ns_shared;
+
+#define PROT_NS_SHARED		(prot_ns_shared)
 
 #define PTE_MAYBE_NG		(arm64_use_ng_mappings ? PTE_NG : 0)
 #define PMD_MAYBE_NG		(arm64_use_ng_mappings ? PMD_SECT_NG : 0)
arch/arm64/kernel/rsi.c +8 -0
index 3a992bdfd6bb..d34e05b339ae 100644
@@ -7,6 +7,11 @@
 #include <linux/memblock.h>
 #include <asm/rsi.h>
 
+struct realm_config config;
+
+unsigned long prot_ns_shared;
+EXPORT_SYMBOL(prot_ns_shared);
+
 DEFINE_STATIC_KEY_FALSE_RO(rsi_present);
 EXPORT_SYMBOL(rsi_present);
 
@@ -63,6 +68,9 @@ void __init arm64_rsi_init(void)
 {
 	if (!rsi_version_matches())
 		return;
+	if (rsi_get_realm_config(&config))
+		return;
+	prot_ns_shared = BIT(config.ipa_bits - 1);
 
 	static_branch_enable(&rsi_present);
 }
Signature · 1 line
2.34.1
#14 lore raw

PATCH v3 4/14 arm64: Mark all I/O as non-secure shared

In reply to Steven Price

All I/O is by default considered non-secure for realms. As such mark them as shared with the host.

---
 arch/arm64/include/asm/io.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
arch/arm64/include/asm/io.h +3 -3
index 4ff0ae3f6d66..0a219c03750b 100644
@@ -277,12 +277,12 @@ static inline void __const_iowrite64_copy(void __iomem *to, const void *from,
 
 #define ioremap_prot ioremap_prot
 
-#define _PAGE_IOREMAP PROT_DEVICE_nGnRE
+#define _PAGE_IOREMAP (PROT_DEVICE_nGnRE | PROT_NS_SHARED)
 
 #define ioremap_wc(addr, size)	\
-	ioremap_prot((addr), (size), PROT_NORMAL_NC)
+	ioremap_prot((addr), (size), (PROT_NORMAL_NC | PROT_NS_SHARED))
 #define ioremap_np(addr, size)	\
-	ioremap_prot((addr), (size), PROT_DEVICE_nGnRnE)
+	ioremap_prot((addr), (size), (PROT_DEVICE_nGnRnE | PROT_NS_SHARED))
 
 /*
  * io{read,write}{16,32,64}be() macros
Signature · 1 line
2.34.1
#15 lore raw

PATCH v3 5/14 fixmap: Allow architecture overriding set_fixmap_io

In reply to Steven Price

From: Suzuki K Poulose <[email protected]>

For a realm guest it will be necessary to ensure IO mappings are shared so that the VMM can emulate the device. The following patch will provide an implementation of set_fixmap_io for arm64 setting the shared bit (if in a realm).

---
 include/asm-generic/fixmap.h | 2 ++
 1 file changed, 2 insertions(+)
include/asm-generic/fixmap.h +2 -0
index 8cc7b09c1bc7..c5ce0368c1ee 100644
@@ -94,8 +94,10 @@ static inline unsigned long virt_to_fix(const unsigned long vaddr)
 /*
  * Some fixmaps are for IO
  */
+#ifndef set_fixmap_io
 #define set_fixmap_io(idx, phys) \
 	__set_fixmap(idx, phys, FIXMAP_PAGE_IO)
+#endif
 
 #define set_fixmap_offset_io(idx, phys) \
 	__set_fixmap_offset(idx, phys, FIXMAP_PAGE_IO)
Signature · 1 line
2.34.1
#16 lore raw

PATCH v3 6/14 arm64: Override set_fixmap_io

In reply to Steven Price

From: Suzuki K Poulose <[email protected]>

Override the set_fixmap_io to set shared permission for the host in case of a CC guest. For now we mark it shared unconditionally.

If/when support for device assignment and device emulation in the realm is added in the future then this will need to filter the physical address and make the decision accordingly.

---
 arch/arm64/include/asm/fixmap.h |  4 +++-
 arch/arm64/mm/mmu.c             | 13 +++++++++++++
 2 files changed, 16 insertions(+), 1 deletion(-)
arch/arm64/include/asm/fixmap.h +3 -1
index 87e307804b99..f765943b088c 100644
@@ -107,7 +107,9 @@ void __init early_fixmap_init(void);
 #define __late_set_fixmap __set_fixmap
 #define __late_clear_fixmap(idx) __set_fixmap((idx), 0, FIXMAP_PAGE_CLEAR)
 
-extern void __set_fixmap(enum fixed_addresses idx, phys_addr_t phys, pgprot_t prot);
+#define set_fixmap_io set_fixmap_io
+void set_fixmap_io(enum fixed_addresses idx, phys_addr_t phys);
+void __set_fixmap(enum fixed_addresses idx, phys_addr_t phys, pgprot_t prot);
 
 #include <asm-generic/fixmap.h>
 
arch/arm64/mm/mmu.c +13 -0
index c927e9312f10..9123df312842 100644
@@ -1192,6 +1192,19 @@ void vmemmap_free(unsigned long start, unsigned long end,
 }
 #endif /* CONFIG_MEMORY_HOTPLUG */
 
+void set_fixmap_io(enum fixed_addresses idx, phys_addr_t phys)
+{
+	pgprot_t prot = FIXMAP_PAGE_IO;
+
+	/*
+	 * For now we consider all I/O as non-secure. For future
+	 * filter the I/O base for setting appropriate permissions.
+	 */
+	prot = __pgprot(pgprot_val(prot) | PROT_NS_SHARED);
+
+	return __set_fixmap(idx, phys, prot);
+}
+
 int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t prot)
 {
 	pud_t new_pud = pfn_pud(__phys_to_pfn(phys), mk_pud_sect_prot(prot));
Signature · 1 line
2.34.1
maintainer
#17 lore raw

Re: [PATCH v3 06/14] arm64: Override set_fixmap_io

arch/arm64/mm/mmu.c void vmemmap_free(unsigned long start, unsigned long end, · line 1206

Quoted 12 lines · Steven Price
arch/arm64/mm/mmu.c +12 -0
+void set_fixmap_io(enum fixed_addresses idx, phys_addr_t phys)
+{
+	pgprot_t prot = FIXMAP_PAGE_IO;
+
+	/*
+	 * For now we consider all I/O as non-secure. For future
+	 * filter the I/O base for setting appropriate permissions.
+	 */
+	prot = __pgprot(pgprot_val(prot) | PROT_NS_SHARED);
+
+	return __set_fixmap(idx, phys, prot);
+}

In v2, Suzuki said that we want to keep this as a function rather than just adding PROT_NS_SHARED to FIXMAP_PAGE_IO in case we want to change this function in the future to allow protected MMIO.

https://lore.kernel.org/linux-arm-kernel/[email protected]/

What I don't understand is that all the other MMIO cases just statically assume unprotected/shard MMIO. Should we drop this patch here as well, adjust FIXMAP_PAGE_IO and think about protected MMIO later when we actually have to do device assignment?

Signature · 1 line
Catalin
#18 lore raw

Re: [PATCH v3 06/14] arm64: Override set_fixmap_io

Quoted 18 lines · Catalin Marinas

arch/arm64/mm/mmu.c void vmemmap_free(unsigned long start, unsigned long end, · line 1206

Quoted 12 lines · Steven Price
arch/arm64/mm/mmu.c +12 -0
+void set_fixmap_io(enum fixed_addresses idx, phys_addr_t phys)
+{
+	pgprot_t prot = FIXMAP_PAGE_IO;
+
+	/*
+	 * For now we consider all I/O as non-secure. For future
+	 * filter the I/O base for setting appropriate permissions.
+	 */
+	prot = __pgprot(pgprot_val(prot) | PROT_NS_SHARED);
+
+	return __set_fixmap(idx, phys, prot);
+}

In v2, Suzuki said that we want to keep this as a function rather than just adding PROT_NS_SHARED to FIXMAP_PAGE_IO in case we want to change this function in the future to allow protected MMIO.

Quoted 6 lines

https://lore.kernel.org/linux-arm-kernel/[email protected]/

What I don't understand is that all the other MMIO cases just statically assume unprotected/shard MMIO. Should we drop this patch here as well, adjust FIXMAP_PAGE_IO and think about protected MMIO later when we actually have to do device assignment?

I agree, there's not much point in this patch as it stands. I'll drop it (and the previous one) for the next version of the series. We can add it back if needed when protected MMIO is a thing.

Thanks,
Steve

#19 lore raw

PATCH v3 7/14 arm64: Make the PHYS_MASK_SHIFT dynamic

In reply to Steven Price

Make the PHYS_MASK_SHIFT dynamic for Realms. This is only is required for masking the PFN from a pte entry. For a realm phys_mask_shift is reduced if the RMM reports a smaller configured size for the guest.

---
v3: Drop the MAX_PHYS_MASK{,_SHIFT} definitions as they are no longer needed.

---
 arch/arm64/include/asm/pgtable-hwdef.h | 6 ------
 arch/arm64/include/asm/pgtable.h       | 5 +++++
 arch/arm64/kernel/rsi.c                | 5 +++++
 3 files changed, 10 insertions(+), 6 deletions(-)
arch/arm64/include/asm/pgtable-hwdef.h +0 -6
index 9943ff0af4c9..2e3af0693bd8 100644
@@ -203,12 +203,6 @@
  */
 #define PTE_S2_MEMATTR(t)	(_AT(pteval_t, (t)) << 2)
 
-/*
- * Highest possible physical address supported.
- */
-#define PHYS_MASK_SHIFT		(CONFIG_ARM64_PA_BITS)
-#define PHYS_MASK		((UL(1) << PHYS_MASK_SHIFT) - 1)
-
 #define TTBR_CNP_BIT		(UL(1) << 0)
 
 /*
arch/arm64/include/asm/pgtable.h +5 -0
index f8efbc128446..11d614d83317 100644
@@ -39,6 +39,11 @@
 #include <linux/sched.h>
 #include <linux/page_table_check.h>
 
+extern unsigned int phys_mask_shift;
+
+#define PHYS_MASK_SHIFT		(phys_mask_shift)
+#define PHYS_MASK		((1UL << PHYS_MASK_SHIFT) - 1)
+
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 #define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE
 
arch/arm64/kernel/rsi.c +5 -0
index d34e05b339ae..c5c03e8e341a 100644
@@ -12,6 +12,8 @@ struct realm_config config;
 unsigned long prot_ns_shared;
 EXPORT_SYMBOL(prot_ns_shared);
 
+unsigned int phys_mask_shift = CONFIG_ARM64_PA_BITS;
+
 DEFINE_STATIC_KEY_FALSE_RO(rsi_present);
 EXPORT_SYMBOL(rsi_present);
 
@@ -72,6 +74,9 @@ void __init arm64_rsi_init(void)
 		return;
 	prot_ns_shared = BIT(config.ipa_bits - 1);
 
+	if (config.ipa_bits - 1 < phys_mask_shift)
+		phys_mask_shift = config.ipa_bits - 1;
+
 	static_branch_enable(&rsi_present);
 }
 
Signature · 1 line
2.34.1
#20 lore raw

PATCH v3 8/14 arm64: Enforce bounce buffers for realm DMA

In reply to Steven Price

Within a realm guest it's not possible for a device emulated by the VMM to access arbitrary guest memory. So force the use of bounce buffers to ensure that the memory the emulated devices are accessing is in memory which is explicitly shared with the host.

---
v3: Simplify mem_init() by using a 'flags' variable.

---
 arch/arm64/kernel/rsi.c | 2 ++
 arch/arm64/mm/init.c    | 9 ++++++++-
 2 files changed, 10 insertions(+), 1 deletion(-)
arch/arm64/kernel/rsi.c +2 -0
index c5c03e8e341a..5cb42609219f 100644
@@ -5,6 +5,8 @@
 
 #include <linux/jump_label.h>
 #include <linux/memblock.h>
+#include <linux/swiotlb.h>
+
 #include <asm/rsi.h>
 
 struct realm_config config;
arch/arm64/mm/init.c +8 -1
index 9d8d38e3bee2..1d595b63da71 100644
@@ -370,8 +370,14 @@ void __init bootmem_init(void)
  */
 void __init mem_init(void)
 {
+	unsigned int flags = SWIOTLB_VERBOSE;
 	bool swiotlb = max_pfn > PFN_DOWN(arm64_dma_phys_limit);
 
+	if (is_realm_world()) {
+		swiotlb = true;
+		flags |= SWIOTLB_FORCE;
+	}
+
 	if (IS_ENABLED(CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC) && !swiotlb) {
 		/*
 		 * If no bouncing needed for ZONE_DMA, reduce the swiotlb
@@ -383,7 +389,8 @@ void __init mem_init(void)
 		swiotlb = true;
 	}
 
-	swiotlb_init(swiotlb, SWIOTLB_VERBOSE);
+	swiotlb_init(swiotlb, flags);
+	swiotlb_update_mem_attributes();
 
 	/* this will put all unused low memory onto the freelists */
 	memblock_free_all();
Signature · 1 line
2.34.1
#21 lore raw

PATCH v3 9/14 arm64: Enable memory encrypt for Realms

In reply to Steven Price

From: Suzuki K Poulose <[email protected]>

Use the memory encryption APIs to trigger a RSI call to request a transition between protected memory and shared memory (or vice versa) and updating the kernel's linear map of modified pages to flip the top bit of the IPA. This requires that block mappings are not used in the direct map for realm guests.

---
Changes since v2:
* Fix location of set_memory_{en,de}crypted() and export them.
* Break-before-make when changing the top bit of the IPA for transitioning to/from shared.

---
 arch/arm64/Kconfig                   |  3 ++
 arch/arm64/include/asm/mem_encrypt.h | 17 ++++++++
 arch/arm64/include/asm/set_memory.h  |  3 ++
 arch/arm64/kernel/rsi.c              | 12 +++++
 arch/arm64/mm/pageattr.c             | 65 ++++++++++++++++++++++++++--
 5 files changed, 97 insertions(+), 3 deletions(-)
 create mode 100644 arch/arm64/include/asm/mem_encrypt.h
arch/arm64/Kconfig +3 -0
index 5d91259ee7b5..0f1480caeeec 100644
@@ -20,6 +20,7 @@ config ARM64
 	select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2
 	select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE
 	select ARCH_HAS_CACHE_LINE_SIZE
+	select ARCH_HAS_CC_PLATFORM
 	select ARCH_HAS_CURRENT_STACK_POINTER
 	select ARCH_HAS_DEBUG_VIRTUAL
 	select ARCH_HAS_DEBUG_VM_PGTABLE
@@ -41,6 +42,8 @@ config ARM64
 	select ARCH_HAS_SETUP_DMA_OPS
 	select ARCH_HAS_SET_DIRECT_MAP
 	select ARCH_HAS_SET_MEMORY
+	select ARCH_HAS_MEM_ENCRYPT
+	select ARCH_HAS_FORCE_DMA_UNENCRYPTED
 	select ARCH_STACKWALK
 	select ARCH_HAS_STRICT_KERNEL_RWX
 	select ARCH_HAS_STRICT_MODULE_RWX
arch/arm64/include/asm/mem_encrypt.h new file +17 -0
new file mode 100644
index 000000000000..e47265cd180a
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2023 ARM Ltd.
+ */
+
+#ifndef __ASM_MEM_ENCRYPT_H
+#define __ASM_MEM_ENCRYPT_H
+
+#include <asm/rsi.h>
+
+/* All DMA must be to non-secure memory for now */
+static inline bool force_dma_unencrypted(struct device *dev)
+{
+	return is_realm_world();
+}
+
+#endif
arch/arm64/include/asm/set_memory.h +3 -0
index 0f740b781187..3b6619c04677 100644
@@ -14,4 +14,7 @@ int set_direct_map_invalid_noflush(struct page *page);
 int set_direct_map_default_noflush(struct page *page);
 bool kernel_page_present(struct page *page);
 
+int set_memory_encrypted(unsigned long addr, int numpages);
+int set_memory_decrypted(unsigned long addr, int numpages);
+
 #endif /* _ASM_ARM64_SET_MEMORY_H */
arch/arm64/kernel/rsi.c +12 -0
index 5cb42609219f..898952d135b0 100644
@@ -6,6 +6,7 @@
 #include <linux/jump_label.h>
 #include <linux/memblock.h>
 #include <linux/swiotlb.h>
+#include <linux/cc_platform.h>
 
 #include <asm/rsi.h>
 
@@ -19,6 +20,17 @@ unsigned int phys_mask_shift = CONFIG_ARM64_PA_BITS;
 DEFINE_STATIC_KEY_FALSE_RO(rsi_present);
 EXPORT_SYMBOL(rsi_present);
 
+bool cc_platform_has(enum cc_attr attr)
+{
+	switch (attr) {
+	case CC_ATTR_MEM_ENCRYPT:
+		return is_realm_world();
+	default:
+		return false;
+	}
+}
+EXPORT_SYMBOL_GPL(cc_platform_has);
+
 static bool rsi_version_matches(void)
 {
 	unsigned long ver_lower, ver_higher;
arch/arm64/mm/pageattr.c +62 -3

92 lines not on this page · raw message

Signature · 1 line
2.34.1
maintainer
#22 lore raw

Re: [PATCH v3 09/14] arm64: Enable memory encrypt for Realms

Quoted 32 lines · Steven Price
arch/arm64/mm/pageattr.c +32 -0
+static int __set_memory_encrypted(unsigned long addr,
+				  int numpages,
+				  bool encrypt)
+{
+	unsigned long set_prot = 0, clear_prot = 0;
+	phys_addr_t start, end;
+	int ret;
+
+	if (!is_realm_world())
+		return 0;
+
+	if (!__is_lm_address(addr))
+		return -EINVAL;
+
+	start = __virt_to_phys(addr);
+	end = start + numpages * PAGE_SIZE;
+
+	/*
+	 * Break the mapping before we make any changes to avoid stale TLB
+	 * entries or Synchronous External Aborts caused by RIPAS_EMPTY
+	 */
+	ret = __change_memory_common(addr, PAGE_SIZE * numpages,
+				     __pgprot(0),
+				     __pgprot(PTE_VALID));
+
+	if (encrypt) {
+		clear_prot = PROT_NS_SHARED;
+		ret = rsi_set_memory_range_protected(start, end);
+	} else {
+		set_prot = PROT_NS_SHARED;
+		ret = rsi_set_memory_range_shared(start, end);
+	}

arch/arm64/mm/pageattr.c int set_direct_map_default_noflush(struct page *page) · line 240

Quoted 10 lines
arch/arm64/mm/pageattr.c +10 -0
+
+	if (ret)
+		return ret;
+
+	set_prot |= PTE_VALID;
+
+	return __change_memory_common(addr, PAGE_SIZE * numpages,
+				      __pgprot(set_prot),
+				      __pgprot(clear_prot));
+}

This works, does break-before-make and also rejects vmalloc() ranges (for the time being).

One particular aspect I don't like is doing the TLBI twice. It's sufficient to do it when you first make the pte invalid. We could guess this in __change_memory_common() if set_mask has PTE_VALID. The call sites are restricted to this file, just add a comment. An alternative would be to add a bool flush argument to this function.

Signature · 1 line
Catalin
#23 lore raw

Re: [PATCH v3 09/14] arm64: Enable memory encrypt for Realms

Quoted 47 lines · Catalin Marinas
Quoted 32 lines · Steven Price
arch/arm64/mm/pageattr.c +32 -0
+static int __set_memory_encrypted(unsigned long addr,
+				  int numpages,
+				  bool encrypt)
+{
+	unsigned long set_prot = 0, clear_prot = 0;
+	phys_addr_t start, end;
+	int ret;
+
+	if (!is_realm_world())
+		return 0;
+
+	if (!__is_lm_address(addr))
+		return -EINVAL;
+
+	start = __virt_to_phys(addr);
+	end = start + numpages * PAGE_SIZE;
+
+	/*
+	 * Break the mapping before we make any changes to avoid stale TLB
+	 * entries or Synchronous External Aborts caused by RIPAS_EMPTY
+	 */
+	ret = __change_memory_common(addr, PAGE_SIZE * numpages,
+				     __pgprot(0),
+				     __pgprot(PTE_VALID));
+
+	if (encrypt) {
+		clear_prot = PROT_NS_SHARED;
+		ret = rsi_set_memory_range_protected(start, end);
+	} else {
+		set_prot = PROT_NS_SHARED;
+		ret = rsi_set_memory_range_shared(start, end);
+	}

arch/arm64/mm/pageattr.c int set_direct_map_default_noflush(struct page *page) · line 240

Quoted 10 lines
arch/arm64/mm/pageattr.c +10 -0
+
+	if (ret)
+		return ret;
+
+	set_prot |= PTE_VALID;
+
+	return __change_memory_common(addr, PAGE_SIZE * numpages,
+				      __pgprot(set_prot),
+				      __pgprot(clear_prot));
+}

This works, does break-before-make and also rejects vmalloc() ranges (for the time being).

Quoted 6 lines

One particular aspect I don't like is doing the TLBI twice. It's sufficient to do it when you first make the pte invalid. We could guess this in __change_memory_common() if set_mask has PTE_VALID. The call sites are restricted to this file, just add a comment. An alternative would be to add a bool flush argument to this function.

I'm always a bit scared of changing this sort of thing, but AFAICT the below should be safe:

- flush_tlb_kernel_range(start, start + size);
+ /*

+        * If the memory is being made valid without changing any other bits
+        * then a TLBI isn't required as a non-valid entry cannot be cached in
+        * the TLB.
+        */

+ if (set_mask != PTE_VALID || clear_mask)
+ flush_tlb_kernel_range(start, start + size);

It will affect users of set_memory_valid() by removing the TLBI when marking memory as valid.

I'll add this change as a separate patch so it can be reverted easily if there's something I've overlooked.

Thanks,
Steve

maintainer
#24 lore raw

Re: [PATCH v3 09/14] arm64: Enable memory encrypt for Realms

In reply to Steven Price
Quoted 22 lines · Steven Price
arch/arm64/mm/pageattr.c +22 -0
+static int __set_memory_encrypted(unsigned long addr,
+				  int numpages,
+				  bool encrypt)
+{
+	unsigned long set_prot = 0, clear_prot = 0;
+	phys_addr_t start, end;
+	int ret;
+
+	if (!is_realm_world())
+		return 0;
+
+	if (!__is_lm_address(addr))
+		return -EINVAL;
+
+	start = __virt_to_phys(addr);
+	end = start + numpages * PAGE_SIZE;
+
+	/*
+	 * Break the mapping before we make any changes to avoid stale TLB
+	 * entries or Synchronous External Aborts caused by RIPAS_EMPTY
+	 */
+	ret = __change_memory_common(addr, PAGE_SIZE * numpages,

arch/arm64/mm/pageattr.c int set_direct_map_default_noflush(struct page *page) · line 230

Quoted 10 lines
arch/arm64/mm/pageattr.c +10 -0
+				     __pgprot(0),
+				     __pgprot(PTE_VALID));
+
+	if (encrypt) {
+		clear_prot = PROT_NS_SHARED;
+		ret = rsi_set_memory_range_protected(start, end);
+	} else {
+		set_prot = PROT_NS_SHARED;
+		ret = rsi_set_memory_range_shared(start, end);
+	}

While reading Michael's replies, it occurred to me that we need check the error paths. Here for example we ignore the return code from __change_memory_common() by overriding the 'ret' variable.

I think the only other place where we don't check at all is the ITS allocation/freeing. Freeing is more interesting as I think we should not release the page back to the kernel if we did not manage to restore the original state. Better have a memory leak than data leak.

Signature · 1 line
Catalin
#25 lore raw

PATCH v3 10/14 arm64: Force device mappings to be non-secure shared

In reply to Steven Price

From: Suzuki K Poulose <[email protected]>

Device mappings (currently) need to be emulated by the VMM so must be mapped shared with the host.

---
 arch/arm64/include/asm/pgtable.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/include/asm/pgtable.h +1 -1
index 11d614d83317..c986fde262c0 100644
@@ -644,7 +644,7 @@ static inline void set_pud_at(struct mm_struct *mm, unsigned long addr,
 #define pgprot_writecombine(prot) \
 	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_NORMAL_NC) | PTE_PXN | PTE_UXN)
 #define pgprot_device(prot) \
-	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_DEVICE_nGnRE) | PTE_PXN | PTE_UXN)
+	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_DEVICE_nGnRE) | PTE_PXN | PTE_UXN | PROT_NS_SHARED)
 #define pgprot_tagged(prot) \
 	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_NORMAL_TAGGED))
 #define pgprot_mhp	pgprot_tagged
Signature · 1 line
2.34.1
#26 lore raw

RE: [PATCH v3 10/14] arm64: Force device mappings to be non-secure shared

From: Steven Price <[email protected]> Sent: Wednesday, June 5, 2024 2:30 AM

Quoted 17 lines

From: Suzuki K Poulose <[email protected]>

Device mappings (currently) need to be emulated by the VMM so must be mapped shared with the host.

---
 arch/arm64/include/asm/pgtable.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/include/asm/pgtable.h +0 -0
index 11d614d83317..c986fde262c0 100644

arch/arm64/include/asm/pgtable.h static inline void set_pud_at(struct mm_struct *mm, unsigned long addr, · line 650

Quoted 8 lines
arch/arm64/include/asm/pgtable.h +1 -1
@@ -644,7 +644,7 @@ static inline void set_pud_at(struct mm_struct *mm, unsigned long addr,
 #define pgprot_writecombine(prot) \
 	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_NORMAL_NC) | PTE_PXN | PTE_UXN)
 #define pgprot_device(prot) \
-	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_DEVICE_nGnRE) | PTE_PXN | PTE_UXN)
+	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_DEVICE_nGnRE) | PTE_PXN | PTE_UXN | PROT_NS_SHARED)
 #define pgprot_tagged(prot) \
 	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_NORMAL_TAGGED))
 #define pgprot_mhp	pgprot_tagged

In v2 of the patches, Catalin raised a question about the need for pgprot_decrypted(). What was concluded? It still looks to me like pgprot_decrypted() and prot_encrypted() are needed, by dma_direct_mmap() and remap_oldmem_pfn_range(), respectively. Also, assuming Hyper-V supports CCA at some point, the Linux guest drivers for Hyper-V need pgprot_decrypted() in hv_ringbuffer_init().

Michael

#27 lore raw

Re: [PATCH v3 10/14] arm64: Force device mappings to be non-secure shared

Quoted 25 lines · Michael Kelley

From: Steven Price <[email protected]> Sent: Wednesday, June 5, 2024 2:30 AM

Quoted 24 lines

From: Suzuki K Poulose <[email protected]>

Device mappings (currently) need to be emulated by the VMM so must be mapped shared with the host.

---
  arch/arm64/include/asm/pgtable.h | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/include/asm/pgtable.h +1 -1
index 11d614d83317..c986fde262c0 100644
@@ -644,7 +644,7 @@ static inline void set_pud_at(struct mm_struct *mm, unsigned long addr,
  #define pgprot_writecombine(prot) \
  	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_NORMAL_NC) | PTE_PXN | PTE_UXN)
  #define pgprot_device(prot) \
-	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_DEVICE_nGnRE) | PTE_PXN | PTE_UXN)
+	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_DEVICE_nGnRE) | PTE_PXN | PTE_UXN | PROT_NS_SHARED)
  #define pgprot_tagged(prot) \
  	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_NORMAL_TAGGED))
Quoted 8 lines

arch/arm64/include/asm/pgtable.h static inline void set_pud_at(struct mm_struct *mm, unsigned long addr, · line 650

Quoted 1 line
arch/arm64/include/asm/pgtable.h +0 -0
  #define pgprot_mhp	pgprot_tagged

In v2 of the patches, Catalin raised a question about the need for pgprot_decrypted(). What was concluded? It still looks to me like pgprot_decrypted() and prot_encrypted() are needed, by dma_direct_mmap() and remap_oldmem_pfn_range(), respectively. Also, assuming Hyper-V supports CCA at some point, the Linux guest drivers for Hyper-V need pgprot_decrypted() in hv_ringbuffer_init().

Right, I think we could simply do :

arch/arm64/include/asm/pgtable.h +0 -0
index c986fde262c0..1ed45893d1e6 100644
@@ -648,6 +648,10 @@ static inline void set_pud_at(struct mm_struct *mm, unsigned long addr,
 #define pgprot_tagged(prot) \
        __pgprot_modify(prot, PTE_ATTRINDX_MASK, 

PTE_ATTRINDX(MT_NORMAL_TAGGED))
#define pgprot_mhp pgprot_tagged
+
+#define pgprot_decrypted(prot) __pgprot_modify(prot, PROT_NS_SHARED, PROT_NS_SHARED) +#define pgprot_encrypted(prot) __pgprot_modify(prot, PROT_NS_SHARED, 0) +

Suzuki

maintainer
#28 lore raw

Re: [PATCH v3 10/14] arm64: Force device mappings to be non-secure shared

Quoted 43 lines · Suzuki K Poulose
Quoted 25 lines · Michael Kelley

From: Steven Price <[email protected]> Sent: Wednesday, June 5, 2024 2:30 AM

Quoted 24 lines

From: Suzuki K Poulose <[email protected]>

Device mappings (currently) need to be emulated by the VMM so must be mapped shared with the host.

---
  arch/arm64/include/asm/pgtable.h | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/include/asm/pgtable.h +1 -1
index 11d614d83317..c986fde262c0 100644
@@ -644,7 +644,7 @@ static inline void set_pud_at(struct mm_struct *mm, unsigned long addr,
  #define pgprot_writecombine(prot) \
  	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_NORMAL_NC) | PTE_PXN | PTE_UXN)
  #define pgprot_device(prot) \
-	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_DEVICE_nGnRE) | PTE_PXN | PTE_UXN)
+	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_DEVICE_nGnRE) | PTE_PXN | PTE_UXN | PROT_NS_SHARED)
  #define pgprot_tagged(prot) \
  	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_NORMAL_TAGGED))
Quoted 8 lines

arch/arm64/include/asm/pgtable.h static inline void set_pud_at(struct mm_struct *mm, unsigned long addr, · line 650

Quoted 1 line
arch/arm64/include/asm/pgtable.h +0 -0
  #define pgprot_mhp	pgprot_tagged

In v2 of the patches, Catalin raised a question about the need for pgprot_decrypted(). What was concluded? It still looks to me like pgprot_decrypted() and prot_encrypted() are needed, by dma_direct_mmap() and remap_oldmem_pfn_range(), respectively. Also, assuming Hyper-V supports CCA at some point, the Linux guest drivers for Hyper-V need pgprot_decrypted() in hv_ringbuffer_init().

Right, I think we could simply do :

arch/arm64/include/asm/pgtable.h +0 -0
index c986fde262c0..1ed45893d1e6 100644

arch/arm64/include/asm/pgtable.h static inline void set_pud_at(struct mm_struct *mm, · line 655

Quoted 8 lines
arch/arm64/include/asm/pgtable.h +3 -0
@@ -648,6 +648,10 @@ static inline void set_pud_at(struct mm_struct *mm,
 unsigned long addr,
 #define pgprot_tagged(prot) \
        __pgprot_modify(prot, PTE_ATTRINDX_MASK,
 PTE_ATTRINDX(MT_NORMAL_TAGGED))
 #define pgprot_mhp     pgprot_tagged
+
+#define pgprot_decrypted(prot) __pgprot_modify(prot, PROT_NS_SHARED, PROT_NS_SHARED)
+#define pgprot_encrypted(prot)  __pgprot_modify(prot, PROT_NS_SHARED, 0)

And maybe rewrite pgprot_device() as:

#define __pgprot_device(prot) \
__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_DEVICE_nGnRE) | PTE_PXN | PTE_UXN)
#define pgprot_device(prot) __pgprot_device(pgprot_decrypted(prot))

Signature · 1 line
Catalin
#29 lore raw

RE: [PATCH v3 10/14] arm64: Force device mappings to be non-secure shared

In reply to Suzuki K Poulose

From: Suzuki K Poulose <[email protected]> Sent: Monday, June 17, 2024 7:55 AM

Quoted 45 lines · Michael Kelley
Quoted 7 lines

From: Steven Price <[email protected]> Sent: Wednesday, June 5, 2024 2:30 AM

Quoted 6 lines

From: Suzuki K Poulose <[email protected]>

Device mappings (currently) need to be emulated by the VMM so must be mapped shared with the host.

Quoted 7 lines

Review

Quoted 7 lines
---
  arch/arm64/include/asm/pgtable.h | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0

b/arch/arm64/include/asm/pgtable.h

Quoted 11 lines
Quoted 11 lines

index 11d614d83317..c986fde262c0 100644

arch/arm64/include/asm/pgtable.h +1 -1
@@ -644,7 +644,7 @@ static inline void set_pud_at(struct mm_struct *mm, unsigned long addr,
  #define pgprot_writecombine(prot) \
  	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_NORMAL_NC) | PTE_PXN | PTE_UXN)
  #define pgprot_device(prot) \
-	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_DEVICE_nGnRE) | PTE_PXN | PTE_UXN)
+	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_DEVICE_nGnRE) | PTE_PXN | PTE_UXN | PROT_NS_SHARED)
  #define pgprot_tagged(prot) \
  	__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_NORMAL_TAGGED))
Quoted 8 lines

arch/arm64/include/asm/pgtable.h static inline void set_pud_at(struct mm_struct *mm, unsigned long addr, · line 650

Quoted 1 line
arch/arm64/include/asm/pgtable.h +0 -0
  #define pgprot_mhp	pgprot_tagged

In v2 of the patches, Catalin raised a question about the need for pgprot_decrypted(). What was concluded? It still looks to me like pgprot_decrypted() and prot_encrypted() are needed, by dma_direct_mmap() and remap_oldmem_pfn_range(), respectively. Also, assuming Hyper-V supports CCA at some point, the Linux guest drivers for Hyper-V need pgprot_decrypted() in hv_ringbuffer_init().

Right, I think we could simply do :

arch/arm64/include/asm/pgtable.h +0 -0
index c986fde262c0..1ed45893d1e6 100644

arch/arm64/include/asm/pgtable.h static inline void set_pud_at(struct mm_struct *mm, unsigned long addr, · line 655

Quoted 8 lines
arch/arm64/include/asm/pgtable.h +4 -0
@@ -648,6 +648,10 @@ static inline void set_pud_at(struct mm_struct *mm, unsigned long addr,
  #define pgprot_tagged(prot) \
         __pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_NORMAL_TAGGED))
  #define pgprot_mhp     pgprot_tagged
+
+#define pgprot_decrypted(prot) __pgprot_modify(prot, PROT_NS_SHARED, PROT_NS_SHARED)
+#define pgprot_encrypted(prot)  __pgprot_modify(prot, PROT_NS_SHARED, 0)
+
 

Yes, looks right to me.

Michael

#30 lore raw

PATCH v3 11/14 efi: arm64: Map Device with Prot Shared

In reply to Steven Price

From: Suzuki K Poulose <[email protected]>

Device mappings need to be emualted by the VMM so must be mapped shared with the host.

---
 arch/arm64/kernel/efi.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/efi.c +1 -1
index 4a92096db34e..ae1ccc8852a4 100644
@@ -34,7 +34,7 @@ static __init pteval_t create_mapping_protection(efi_memory_desc_t *md)
 	u32 type = md->type;
 
 	if (type == EFI_MEMORY_MAPPED_IO)
-		return PROT_DEVICE_nGnRE;
+		return PROT_NS_SHARED | PROT_DEVICE_nGnRE;
 
 	if (region_is_misaligned(md)) {
 		static bool __initdata code_is_misaligned;
Signature · 1 line
2.34.1
#31 lore raw

PATCH v3 12/14 arm64: realm: Support nonsecure ITS emulation shared

In reply to Steven Price

Within a realm guest the ITS is emulated by the host. This means the allocations must have been made available to the host by a call to set_memory_decrypted(). Introduce an allocation function which performs this extra call.

---
Changes since v2:
* Drop 'shared' from the new its_xxx function names as they are used for non-realm guests too.
* Don't handle the NUMA_NO_NODE case specially - alloc_pages_node() should do the right thing.
* Drop a pointless (void *) cast.

---
 drivers/irqchip/irq-gic-v3-its.c | 90 ++++++++++++++++++++++++--------
 1 file changed, 67 insertions(+), 23 deletions(-)
drivers/irqchip/irq-gic-v3-its.c +67 -23
index 40ebf1726393..ca72f830f4cc 100644
@@ -18,6 +18,7 @@
 #include <linux/irqdomain.h>
 #include <linux/list.h>
 #include <linux/log2.h>
+#include <linux/mem_encrypt.h>
 #include <linux/memblock.h>
 #include <linux/mm.h>
 #include <linux/msi.h>
@@ -27,6 +28,7 @@
 #include <linux/of_pci.h>
 #include <linux/of_platform.h>
 #include <linux/percpu.h>
+#include <linux/set_memory.h>
 #include <linux/slab.h>
 #include <linux/syscore_ops.h>
 
@@ -163,6 +165,7 @@ struct its_device {
 	struct its_node		*its;
 	struct event_lpi_map	event_map;
 	void			*itt;
+	u32			itt_order;
 	u32			nr_ites;
 	u32			device_id;
 	bool			shared;
@@ -198,6 +201,30 @@ static DEFINE_IDA(its_vpeid_ida);
 #define gic_data_rdist_rd_base()	(gic_data_rdist()->rd_base)
 #define gic_data_rdist_vlpi_base()	(gic_data_rdist_rd_base() + SZ_128K)
 
+static struct page *its_alloc_pages_node(int node, gfp_t gfp,
+					 unsigned int order)
+{
+	struct page *page;
+
+	page = alloc_pages_node(node, gfp, order);
+
+	if (page)
+		set_memory_decrypted((unsigned long)page_address(page),
+				     1 << order);
+	return page;
+}
+
+static struct page *its_alloc_pages(gfp_t gfp, unsigned int order)
+{
+	return its_alloc_pages_node(NUMA_NO_NODE, gfp, order);
+}
+
+static void its_free_pages(void *addr, unsigned int order)
+{
+	set_memory_encrypted((unsigned long)addr, 1 << order);
+	free_pages((unsigned long)addr, order);
+}
+
 /*
  * Skip ITSs that have no vLPIs mapped, unless we're on GICv4.1, as we
  * always have vSGIs mapped.
@@ -2212,7 +2239,8 @@ static struct page *its_allocate_prop_table(gfp_t gfp_flags)
 {
 	struct page *prop_page;
 
-	prop_page = alloc_pages(gfp_flags, get_order(LPI_PROPBASE_SZ));
+	prop_page = its_alloc_pages(gfp_flags,
+				    get_order(LPI_PROPBASE_SZ));
 	if (!prop_page)
 		return NULL;
 
@@ -2223,8 +2251,8 @@ static struct page *its_allocate_prop_table(gfp_t gfp_flags)
 
 static void its_free_prop_table(struct page *prop_page)
 {
-	free_pages((unsigned long)page_address(prop_page),
-		   get_order(LPI_PROPBASE_SZ));
+	its_free_pages(page_address(prop_page),
+		       get_order(LPI_PROPBASE_SZ));
 }
 
 static bool gic_check_reserved_range(phys_addr_t addr, unsigned long size)
@@ -2346,7 +2374,8 @@ static int its_setup_baser(struct its_node *its, struct its_baser *baser,
 		order = get_order(GITS_BASER_PAGES_MAX * psz);
 	}
 
-	page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO, order);
+	page = its_alloc_pages_node(its->numa_node,
+				    GFP_KERNEL | __GFP_ZERO, order);
 	if (!page)
 		return -ENOMEM;
 
@@ -2359,7 +2388,7 @@ static int its_setup_baser(struct its_node *its, struct its_baser *baser,
 		/* 52bit PA is supported only when PageSize=64K */
 		if (psz != SZ_64K) {
 			pr_err("ITS: no 52bit PA support when psz=%d\n", psz);
-			free_pages((unsigned long)base, order);
+			its_free_pages(base, order);
 			return -ENXIO;
 		}
 
@@ -2415,7 +2444,7 @@ static int its_setup_baser(struct its_node *its, struct its_baser *baser,
 		pr_err("ITS@%pa: %s doesn't stick: %llx %llx\n",
 		       &its->phys_base, its_base_type_string[type],
 		       val, tmp);
-		free_pages((unsigned long)base, order);
+		its_free_pages(base, order);
 		return -ENXIO;
 	}
 
@@ -2554,8 +2583,8 @@ static void its_free_tables(struct its_node *its)
 
 	for (i = 0; i < GITS_BASER_NR_REGS; i++) {
 		if (its->tables[i].base) {
-			free_pages((unsigned long)its->tables[i].base,
-				   its->tables[i].order);
+			its_free_pages(its->tables[i].base,
+				       its->tables[i].order);
 			its->tables[i].base = NULL;
 		}
 	}
@@ -2821,7 +2850,8 @@ static bool allocate_vpe_l2_table(int cpu, u32 id)
 
 	/* Allocate memory for 2nd level table */
 	if (!table[idx]) {
-		page = alloc_pages(GFP_KERNEL | __GFP_ZERO, get_order(psz));
+		page = its_alloc_pages(GFP_KERNEL | __GFP_ZERO,
+				       get_order(psz));
 		if (!page)
 			return false;
 
@@ -2940,7 +2970,8 @@ static int allocate_vpe_l1_table(void)
 
 	pr_debug("np = %d, npg = %lld, psz = %d, epp = %d, esz = %d\n",
 		 np, npg, psz, epp, esz);
-	page = alloc_pages(GFP_ATOMIC | __GFP_ZERO, get_order(np * PAGE_SIZE));
+	page = its_alloc_pages(GFP_ATOMIC | __GFP_ZERO,
+			       get_order(np * PAGE_SIZE));
 	if (!page)
 		return -ENOMEM;
 
@@ -2986,8 +3017,8 @@ static struct page *its_allocate_pending_table(gfp_t gfp_flags)
 {
 	struct page *pend_page;
 
-	pend_page = alloc_pages(gfp_flags | __GFP_ZERO,
-				get_order(LPI_PENDBASE_SZ));
+	pend_page = its_alloc_pages(gfp_flags | __GFP_ZERO,
+				    get_order(LPI_PENDBASE_SZ));
 	if (!pend_page)
 		return NULL;
 
@@ -2999,7 +3030,7 @@ static struct page *its_allocate_pending_table(gfp_t gfp_flags)
 
 static void its_free_pending_table(struct page *pt)
 {
-	free_pages((unsigned long)page_address(pt), get_order(LPI_PENDBASE_SZ));
+	its_free_pages(page_address(pt), get_order(LPI_PENDBASE_SZ));
 }
 
 /*
@@ -3334,8 +3365,9 @@ static bool its_alloc_table_entry(struct its_node *its,
 
 	/* Allocate memory for 2nd level table */
 	if (!table[idx]) {
-		page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO,
-					get_order(baser->psz));
+		page = its_alloc_pages_node(its->numa_node,
+					    GFP_KERNEL | __GFP_ZERO,
+					    get_order(baser->psz));
 		if (!page)
 			return false;
 
@@ -3418,7 +3450,9 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 	unsigned long *lpi_map = NULL;
 	unsigned long flags;
 	u16 *col_map = NULL;
+	struct page *page;
 	void *itt;
+	int itt_order;
 	int lpi_base;
 	int nr_lpis;
 	int nr_ites;
@@ -3430,7 +3464,6 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 	if (WARN_ON(!is_power_of_2(nvecs)))
 		nvecs = roundup_pow_of_two(nvecs);
 
-	dev = kzalloc(sizeof(*dev), GFP_KERNEL);
 	/*
 	 * Even if the device wants a single LPI, the ITT must be
 	 * sized as a power of two (and you need at least one bit...).
@@ -3438,7 +3471,16 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 	nr_ites = max(2, nvecs);
 	sz = nr_ites * (FIELD_GET(GITS_TYPER_ITT_ENTRY_SIZE, its->typer) + 1);
 	sz = max(sz, ITS_ITT_ALIGN) + ITS_ITT_ALIGN - 1;
-	itt = kzalloc_node(sz, GFP_KERNEL, its->numa_node);
+	itt_order = get_order(sz);
+	page = its_alloc_pages_node(its->numa_node,
+				    GFP_KERNEL | __GFP_ZERO,
+				    itt_order);
+	if (!page)
+		return NULL;
+	itt = page_address(page);
+
+	dev = kzalloc(sizeof(*dev), GFP_KERNEL);
+
 	if (alloc_lpis) {
 		lpi_map = its_lpi_alloc(nvecs, &lpi_base, &nr_lpis);
 		if (lpi_map)
@@ -3450,9 +3492,9 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 		lpi_base = 0;
 	}
 
-	if (!dev || !itt ||  !col_map || (!lpi_map && alloc_lpis)) {
+	if (!dev || !col_map || (!lpi_map && alloc_lpis)) {
 		kfree(dev);
-		kfree(itt);
+		its_free_pages(itt, itt_order);
 		bitmap_free(lpi_map);
 		kfree(col_map);
 		return NULL;
@@ -3462,6 +3504,7 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 
 	dev->its = its;
 	dev->itt = itt;
+	dev->itt_order = itt_order;
 	dev->nr_ites = nr_ites;
 	dev->event_map.lpi_map = lpi_map;
 	dev->event_map.col_map = col_map;
@@ -3489,7 +3532,7 @@ static void its_free_device(struct its_device *its_dev)
 	list_del(&its_dev->entry);
 	raw_spin_unlock_irqrestore(&its_dev->its->lock, flags);
 	kfree(its_dev->event_map.col_map);
-	kfree(its_dev->itt);
+	its_free_pages(its_dev->itt, its_dev->itt_order);
 	kfree(its_dev);
 }
 
@@ -5131,8 +5174,9 @@ static int __init its_probe_one(struct its_node *its)
 		}
 	}
 
-	page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO,
-				get_order(ITS_CMD_QUEUE_SZ));
+	page = its_alloc_pages_node(its->numa_node,
+				    GFP_KERNEL | __GFP_ZERO,
+				    get_order(ITS_CMD_QUEUE_SZ));
 	if (!page) {
 		err = -ENOMEM;
 		goto out_unmap_sgir;
@@ -5196,7 +5240,7 @@ static int __init its_probe_one(struct its_node *its)
 out_free_tables:
 	its_free_tables(its);
 out_free_cmd:
-	free_pages((unsigned long)its->cmd_base, get_order(ITS_CMD_QUEUE_SZ));
+	its_free_pages(its->cmd_base, get_order(ITS_CMD_QUEUE_SZ));
 out_unmap_sgir:
 	if (its->sgir_base)
 		iounmap(its->sgir_base);
Signature · 1 line
2.34.1
maintainer
#32 lore raw

Re: [PATCH v3 12/14] arm64: realm: Support nonsecure ITS emulation shared

The subject line is... odd. I'd expect something like:

"irqchip/gic-v3-its: Share ITS tables with a non-trusted hypervisor"

because nothing here should be CCA specific.

Quoted 5 lines · Steven Price

Within a realm guest the ITS is emulated by the host. This means the allocations must have been made available to the host by a call to set_memory_decrypted(). Introduce an allocation function which performs this extra call.

This doesn't mention that this patch radically changes the allocation of some tables.

Quoted 47 lines

---
Changes since v2:
* Drop 'shared' from the new its_xxx function names as they are used for non-realm guests too.
* Don't handle the NUMA_NO_NODE case specially - alloc_pages_node() should do the right thing.
* Drop a pointless (void *) cast.

---
 drivers/irqchip/irq-gic-v3-its.c | 90 ++++++++++++++++++++++++--------
 1 file changed, 67 insertions(+), 23 deletions(-)
drivers/irqchip/irq-gic-v3-its.c +3 -0
index 40ebf1726393..ca72f830f4cc 100644
@@ -18,6 +18,7 @@
 #include <linux/irqdomain.h>
 #include <linux/list.h>
 #include <linux/log2.h>
+#include <linux/mem_encrypt.h>
 #include <linux/memblock.h>
 #include <linux/mm.h>
 #include <linux/msi.h>
@@ -27,6 +28,7 @@
 #include <linux/of_pci.h>
 #include <linux/of_platform.h>
 #include <linux/percpu.h>
+#include <linux/set_memory.h>
 #include <linux/slab.h>
 #include <linux/syscore_ops.h>
 
@@ -163,6 +165,7 @@ struct its_device {
 	struct its_node		*its;
 	struct event_lpi_map	event_map;
 	void			*itt;
+	u32			itt_order;
 	u32			nr_ites;
 	u32			device_id;
 	bool			shared;
@@ -198,6 +201,30 @@ static DEFINE_IDA(its_vpeid_ida);
 #define gic_data_rdist_rd_base()	(gic_data_rdist()->rd_base)
 #define gic_data_rdist_vlpi_base()	(gic_data_rdist_rd_base() + SZ_128K)
 

drivers/irqchip/irq-gic-v3-its.c static DEFINE_IDA(its_vpeid_ida); · line 213

Quoted 10 lines
drivers/irqchip/irq-gic-v3-its.c +10 -0
+static struct page *its_alloc_pages_node(int node, gfp_t gfp,
+					 unsigned int order)
+{
+	struct page *page;
+
+	page = alloc_pages_node(node, gfp, order);
+
+	if (page)
+		set_memory_decrypted((unsigned long)page_address(page),
+				     1 << order);

Please use BIT(order).

Quoted 148 lines
drivers/irqchip/irq-gic-v3-its.c +36 -16
+	return page;
+}
+
+static struct page *its_alloc_pages(gfp_t gfp, unsigned int order)
+{
+	return its_alloc_pages_node(NUMA_NO_NODE, gfp, order);
+}
+
+static void its_free_pages(void *addr, unsigned int order)
+{
+	set_memory_encrypted((unsigned long)addr, 1 << order);
+	free_pages((unsigned long)addr, order);
+}
+
 /*
  * Skip ITSs that have no vLPIs mapped, unless we're on GICv4.1, as we
  * always have vSGIs mapped.
@@ -2212,7 +2239,8 @@ static struct page *its_allocate_prop_table(gfp_t gfp_flags)
 {
 	struct page *prop_page;
 
-	prop_page = alloc_pages(gfp_flags, get_order(LPI_PROPBASE_SZ));
+	prop_page = its_alloc_pages(gfp_flags,
+				    get_order(LPI_PROPBASE_SZ));
 	if (!prop_page)
 		return NULL;
 
@@ -2223,8 +2251,8 @@ static struct page *its_allocate_prop_table(gfp_t gfp_flags)
 
 static void its_free_prop_table(struct page *prop_page)
 {
-	free_pages((unsigned long)page_address(prop_page),
-		   get_order(LPI_PROPBASE_SZ));
+	its_free_pages(page_address(prop_page),
+		       get_order(LPI_PROPBASE_SZ));
 }
 
 static bool gic_check_reserved_range(phys_addr_t addr, unsigned long size)
@@ -2346,7 +2374,8 @@ static int its_setup_baser(struct its_node *its, struct its_baser *baser,
 		order = get_order(GITS_BASER_PAGES_MAX * psz);
 	}
 
-	page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO, order);
+	page = its_alloc_pages_node(its->numa_node,
+				    GFP_KERNEL | __GFP_ZERO, order);
 	if (!page)
 		return -ENOMEM;
 
@@ -2359,7 +2388,7 @@ static int its_setup_baser(struct its_node *its, struct its_baser *baser,
 		/* 52bit PA is supported only when PageSize=64K */
 		if (psz != SZ_64K) {
 			pr_err("ITS: no 52bit PA support when psz=%d\n", psz);
-			free_pages((unsigned long)base, order);
+			its_free_pages(base, order);
 			return -ENXIO;
 		}
 
@@ -2415,7 +2444,7 @@ static int its_setup_baser(struct its_node *its, struct its_baser *baser,
 		pr_err("ITS@%pa: %s doesn't stick: %llx %llx\n",
 		       &its->phys_base, its_base_type_string[type],
 		       val, tmp);
-		free_pages((unsigned long)base, order);
+		its_free_pages(base, order);
 		return -ENXIO;
 	}
 
@@ -2554,8 +2583,8 @@ static void its_free_tables(struct its_node *its)
 
 	for (i = 0; i < GITS_BASER_NR_REGS; i++) {
 		if (its->tables[i].base) {
-			free_pages((unsigned long)its->tables[i].base,
-				   its->tables[i].order);
+			its_free_pages(its->tables[i].base,
+				       its->tables[i].order);
 			its->tables[i].base = NULL;
 		}
 	}
@@ -2821,7 +2850,8 @@ static bool allocate_vpe_l2_table(int cpu, u32 id)
 
 	/* Allocate memory for 2nd level table */
 	if (!table[idx]) {
-		page = alloc_pages(GFP_KERNEL | __GFP_ZERO, get_order(psz));
+		page = its_alloc_pages(GFP_KERNEL | __GFP_ZERO,
+				       get_order(psz));
 		if (!page)
 			return false;
 
@@ -2940,7 +2970,8 @@ static int allocate_vpe_l1_table(void)
 
 	pr_debug("np = %d, npg = %lld, psz = %d, epp = %d, esz = %d\n",
 		 np, npg, psz, epp, esz);
-	page = alloc_pages(GFP_ATOMIC | __GFP_ZERO, get_order(np * PAGE_SIZE));
+	page = its_alloc_pages(GFP_ATOMIC | __GFP_ZERO,
+			       get_order(np * PAGE_SIZE));
 	if (!page)
 		return -ENOMEM;
 
@@ -2986,8 +3017,8 @@ static struct page *its_allocate_pending_table(gfp_t gfp_flags)
 {
 	struct page *pend_page;
 
-	pend_page = alloc_pages(gfp_flags | __GFP_ZERO,
-				get_order(LPI_PENDBASE_SZ));
+	pend_page = its_alloc_pages(gfp_flags | __GFP_ZERO,
+				    get_order(LPI_PENDBASE_SZ));
 	if (!pend_page)
 		return NULL;
 
@@ -2999,7 +3030,7 @@ static struct page *its_allocate_pending_table(gfp_t gfp_flags)
 
 static void its_free_pending_table(struct page *pt)
 {
-	free_pages((unsigned long)page_address(pt), get_order(LPI_PENDBASE_SZ));
+	its_free_pages(page_address(pt), get_order(LPI_PENDBASE_SZ));
 }
 
 /*
@@ -3334,8 +3365,9 @@ static bool its_alloc_table_entry(struct its_node *its,
 
 	/* Allocate memory for 2nd level table */
 	if (!table[idx]) {
-		page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO,
-					get_order(baser->psz));
+		page = its_alloc_pages_node(its->numa_node,
+					    GFP_KERNEL | __GFP_ZERO,
+					    get_order(baser->psz));
 		if (!page)
 			return false;
 
@@ -3418,7 +3450,9 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 	unsigned long *lpi_map = NULL;
 	unsigned long flags;
 	u16 *col_map = NULL;
+	struct page *page;
 	void *itt;
+	int itt_order;
 	int lpi_base;
 	int nr_lpis;
 	int nr_ites;
@@ -3430,7 +3464,6 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 	if (WARN_ON(!is_power_of_2(nvecs)))
 		nvecs = roundup_pow_of_two(nvecs);
 
-	dev = kzalloc(sizeof(*dev), GFP_KERNEL);
 	/*
 	 * Even if the device wants a single LPI, the ITT must be
 	 * sized as a power of two (and you need at least one bit...).

drivers/irqchip/irq-gic-v3-its.c static struct its_device *its_create_device(struct its_node *its, u32 dev_id, · line 3477

Quoted 8 lines
drivers/irqchip/irq-gic-v3-its.c +4 -1
@@ -3438,7 +3471,16 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 	nr_ites = max(2, nvecs);
 	sz = nr_ites * (FIELD_GET(GITS_TYPER_ITT_ENTRY_SIZE, its->typer) + 1);
 	sz = max(sz, ITS_ITT_ALIGN) + ITS_ITT_ALIGN - 1;
-	itt = kzalloc_node(sz, GFP_KERNEL, its->numa_node);
+	itt_order = get_order(sz);
+	page = its_alloc_pages_node(its->numa_node,
+				    GFP_KERNEL | __GFP_ZERO,
+				    itt_order);

So we go from an allocation that was so far measured in *bytes* to something that is now at least a page. Per device. This seems a bit excessive to me, specially when it isn't conditioned on anything and is now imposed on all platforms, including the non-CCA systems (which are exactly 100% of the machines).

Another thing is that if we go with page alignment, then the 256 byte alignment can obviously be removed everywhere (hint: MAPD needs to change).

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#33 lore raw

Re: [PATCH v3 12/14] arm64: realm: Support nonsecure ITS emulation shared

Hi Marc,

Quoted 5 lines · Marc Zyngier

The subject line is... odd. I'd expect something like:

"irqchip/gic-v3-its: Share ITS tables with a non-trusted hypervisor"

because nothing here should be CCA specific.

Good point - that's a much better subject.

Quoted 10 lines
Quoted 5 lines · Steven Price

Within a realm guest the ITS is emulated by the host. This means the allocations must have been made available to the host by a call to set_memory_decrypted(). Introduce an allocation function which performs this extra call.

This doesn't mention that this patch radically changes the allocation of some tables.

I guess that depends on your definition of radical, see below.

Quoted 47 lines
Quoted 47 lines

---
Changes since v2:
* Drop 'shared' from the new its_xxx function names as they are used for non-realm guests too.
* Don't handle the NUMA_NO_NODE case specially - alloc_pages_node() should do the right thing.
* Drop a pointless (void *) cast.

---
 drivers/irqchip/irq-gic-v3-its.c | 90 ++++++++++++++++++++++++--------
 1 file changed, 67 insertions(+), 23 deletions(-)
drivers/irqchip/irq-gic-v3-its.c +3 -0
index 40ebf1726393..ca72f830f4cc 100644
@@ -18,6 +18,7 @@
 #include <linux/irqdomain.h>
 #include <linux/list.h>
 #include <linux/log2.h>
+#include <linux/mem_encrypt.h>
 #include <linux/memblock.h>
 #include <linux/mm.h>
 #include <linux/msi.h>
@@ -27,6 +28,7 @@
 #include <linux/of_pci.h>
 #include <linux/of_platform.h>
 #include <linux/percpu.h>
+#include <linux/set_memory.h>
 #include <linux/slab.h>
 #include <linux/syscore_ops.h>
 
@@ -163,6 +165,7 @@ struct its_device {
 	struct its_node		*its;
 	struct event_lpi_map	event_map;
 	void			*itt;
+	u32			itt_order;
 	u32			nr_ites;
 	u32			device_id;
 	bool			shared;
@@ -198,6 +201,30 @@ static DEFINE_IDA(its_vpeid_ida);
 #define gic_data_rdist_rd_base()	(gic_data_rdist()->rd_base)
 #define gic_data_rdist_vlpi_base()	(gic_data_rdist_rd_base() + SZ_128K)
 
Quoted 12 lines

drivers/irqchip/irq-gic-v3-its.c static DEFINE_IDA(its_vpeid_ida); · line 213

Quoted 10 lines
drivers/irqchip/irq-gic-v3-its.c +10 -0
+static struct page *its_alloc_pages_node(int node, gfp_t gfp,
+					 unsigned int order)
+{
+	struct page *page;
+
+	page = alloc_pages_node(node, gfp, order);
+
+	if (page)
+		set_memory_decrypted((unsigned long)page_address(page),
+				     1 << order);

Please use BIT(order).

Sure.

Quoted 154 lines
Quoted 154 lines
drivers/irqchip/irq-gic-v3-its.c +38 -17
+	return page;
+}
+
+static struct page *its_alloc_pages(gfp_t gfp, unsigned int order)
+{
+	return its_alloc_pages_node(NUMA_NO_NODE, gfp, order);
+}
+
+static void its_free_pages(void *addr, unsigned int order)
+{
+	set_memory_encrypted((unsigned long)addr, 1 << order);
+	free_pages((unsigned long)addr, order);
+}
+
 /*
  * Skip ITSs that have no vLPIs mapped, unless we're on GICv4.1, as we
  * always have vSGIs mapped.
@@ -2212,7 +2239,8 @@ static struct page *its_allocate_prop_table(gfp_t gfp_flags)
 {
 	struct page *prop_page;
 
-	prop_page = alloc_pages(gfp_flags, get_order(LPI_PROPBASE_SZ));
+	prop_page = its_alloc_pages(gfp_flags,
+				    get_order(LPI_PROPBASE_SZ));
 	if (!prop_page)
 		return NULL;
 
@@ -2223,8 +2251,8 @@ static struct page *its_allocate_prop_table(gfp_t gfp_flags)
 
 static void its_free_prop_table(struct page *prop_page)
 {
-	free_pages((unsigned long)page_address(prop_page),
-		   get_order(LPI_PROPBASE_SZ));
+	its_free_pages(page_address(prop_page),
+		       get_order(LPI_PROPBASE_SZ));
 }
 
 static bool gic_check_reserved_range(phys_addr_t addr, unsigned long size)
@@ -2346,7 +2374,8 @@ static int its_setup_baser(struct its_node *its, struct its_baser *baser,
 		order = get_order(GITS_BASER_PAGES_MAX * psz);
 	}
 
-	page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO, order);
+	page = its_alloc_pages_node(its->numa_node,
+				    GFP_KERNEL | __GFP_ZERO, order);
 	if (!page)
 		return -ENOMEM;
 
@@ -2359,7 +2388,7 @@ static int its_setup_baser(struct its_node *its, struct its_baser *baser,
 		/* 52bit PA is supported only when PageSize=64K */
 		if (psz != SZ_64K) {
 			pr_err("ITS: no 52bit PA support when psz=%d\n", psz);
-			free_pages((unsigned long)base, order);
+			its_free_pages(base, order);
 			return -ENXIO;
 		}
 
@@ -2415,7 +2444,7 @@ static int its_setup_baser(struct its_node *its, struct its_baser *baser,
 		pr_err("ITS@%pa: %s doesn't stick: %llx %llx\n",
 		       &its->phys_base, its_base_type_string[type],
 		       val, tmp);
-		free_pages((unsigned long)base, order);
+		its_free_pages(base, order);
 		return -ENXIO;
 	}
 
@@ -2554,8 +2583,8 @@ static void its_free_tables(struct its_node *its)
 
 	for (i = 0; i < GITS_BASER_NR_REGS; i++) {
 		if (its->tables[i].base) {
-			free_pages((unsigned long)its->tables[i].base,
-				   its->tables[i].order);
+			its_free_pages(its->tables[i].base,
+				       its->tables[i].order);
 			its->tables[i].base = NULL;
 		}
 	}
@@ -2821,7 +2850,8 @@ static bool allocate_vpe_l2_table(int cpu, u32 id)
 
 	/* Allocate memory for 2nd level table */
 	if (!table[idx]) {
-		page = alloc_pages(GFP_KERNEL | __GFP_ZERO, get_order(psz));
+		page = its_alloc_pages(GFP_KERNEL | __GFP_ZERO,
+				       get_order(psz));
 		if (!page)
 			return false;
 
@@ -2940,7 +2970,8 @@ static int allocate_vpe_l1_table(void)
 
 	pr_debug("np = %d, npg = %lld, psz = %d, epp = %d, esz = %d\n",
 		 np, npg, psz, epp, esz);
-	page = alloc_pages(GFP_ATOMIC | __GFP_ZERO, get_order(np * PAGE_SIZE));
+	page = its_alloc_pages(GFP_ATOMIC | __GFP_ZERO,
+			       get_order(np * PAGE_SIZE));
 	if (!page)
 		return -ENOMEM;
 
@@ -2986,8 +3017,8 @@ static struct page *its_allocate_pending_table(gfp_t gfp_flags)
 {
 	struct page *pend_page;
 
-	pend_page = alloc_pages(gfp_flags | __GFP_ZERO,
-				get_order(LPI_PENDBASE_SZ));
+	pend_page = its_alloc_pages(gfp_flags | __GFP_ZERO,
+				    get_order(LPI_PENDBASE_SZ));
 	if (!pend_page)
 		return NULL;
 
@@ -2999,7 +3030,7 @@ static struct page *its_allocate_pending_table(gfp_t gfp_flags)
 
 static void its_free_pending_table(struct page *pt)
 {
-	free_pages((unsigned long)page_address(pt), get_order(LPI_PENDBASE_SZ));
+	its_free_pages(page_address(pt), get_order(LPI_PENDBASE_SZ));
 }
 
 /*
@@ -3334,8 +3365,9 @@ static bool its_alloc_table_entry(struct its_node *its,
 
 	/* Allocate memory for 2nd level table */
 	if (!table[idx]) {
-		page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO,
-					get_order(baser->psz));
+		page = its_alloc_pages_node(its->numa_node,
+					    GFP_KERNEL | __GFP_ZERO,
+					    get_order(baser->psz));
 		if (!page)
 			return false;
 
@@ -3418,7 +3450,9 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 	unsigned long *lpi_map = NULL;
 	unsigned long flags;
 	u16 *col_map = NULL;
+	struct page *page;
 	void *itt;
+	int itt_order;
 	int lpi_base;
 	int nr_lpis;
 	int nr_ites;
@@ -3430,7 +3464,6 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 	if (WARN_ON(!is_power_of_2(nvecs)))
 		nvecs = roundup_pow_of_two(nvecs);
 
-	dev = kzalloc(sizeof(*dev), GFP_KERNEL);
 	/*
 	 * Even if the device wants a single LPI, the ITT must be
 	 * sized as a power of two (and you need at least one bit...).
@@ -3438,7 +3471,16 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 	nr_ites = max(2, nvecs);
 	sz = nr_ites * (FIELD_GET(GITS_TYPER_ITT_ENTRY_SIZE, its->typer) + 1);
 	sz = max(sz, ITS_ITT_ALIGN) + ITS_ITT_ALIGN - 1;
-	itt = kzalloc_node(sz, GFP_KERNEL, its->numa_node);
+	itt_order = get_order(sz);
+	page = its_alloc_pages_node(its->numa_node,
Quoted 8 lines

drivers/irqchip/irq-gic-v3-its.c static struct its_device *its_create_device(struct its_node *its, u32 dev_id, · line 3477

Quoted 2 lines
drivers/irqchip/irq-gic-v3-its.c +2 -0
+				    GFP_KERNEL | __GFP_ZERO,
+				    itt_order);

So we go from an allocation that was so far measured in *bytes* to something that is now at least a page. Per device. This seems a bit excessive to me, specially when it isn't conditioned on anything and is now imposed on all platforms, including the non-CCA systems (which are exactly 100% of the machines).

Catalin asked about this in v2:
https://lore.kernel.org/lkml/[email protected]/

To be honest, I don't have a great handle on how much memory is being wasted here. Within the realm guest I was testing this is rounding up an otherwise 511 byte allocation to a 4k page, and there are 3 of them. Which seems reasonable from a realm guest perspective.

I can see two options to improve here:

1. Add a !is_realm_world() check and return to the previous behaviour when not running in a realm. It's ugly, and doesn't deal with any other potential future memory encryption. cc_platform_has(CC_ATTR_MEM_ENCRYPT) might be preferable? But this means no impact to non-realm guests.

2. Use a special (global) memory allocator that does the set_memory_decrypted() dance on the pages that it allocates but allows packing the allocations. I'm not aware of an existing kernel API for this, so it's potentially quite a bit of code. The benefit is that it reduces memory consumption in a realm guest, although fragmentation still means we're likely to see a (small) growth.

Any thoughts on what you think would be best?

Quoted 3 lines

Another thing is that if we go with page alignment, then the 256 byte alignment can obviously be removed everywhere (hint: MAPD needs to change).

Ah, good point - I'll need to look into that, my GIC-foo isn't quite up to speed on that.

Thanks,

Steve

maintainer
#34 lore raw

Re: [PATCH v3 12/14] arm64: realm: Support nonsecure ITS emulation shared

Quoted 20 lines · Steven Price

Hi Marc,

Quoted 5 lines · Marc Zyngier

The subject line is... odd. I'd expect something like:

"irqchip/gic-v3-its: Share ITS tables with a non-trusted hypervisor"

because nothing here should be CCA specific.

Good point - that's a much better subject.

Quoted 8 lines
Quoted 5 lines · Steven Price

Within a realm guest the ITS is emulated by the host. This means the allocations must have been made available to the host by a call to set_memory_decrypted(). Introduce an allocation function which performs this extra call.

Quoted 4 lines
Quoted 2 lines

This doesn't mention that this patch radically changes the allocation of some tables.

I guess that depends on your definition of radical, see below.

It's election time, I'm all about making bold statements!

[...]

Quoted 46 lines
Quoted 37 lines
Quoted 37 lines
drivers/irqchip/irq-gic-v3-its.c +7 -4
@@ -3334,8 +3365,9 @@ static bool its_alloc_table_entry(struct its_node *its,
 
 	/* Allocate memory for 2nd level table */
 	if (!table[idx]) {
-		page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO,
-					get_order(baser->psz));
+		page = its_alloc_pages_node(its->numa_node,
+					    GFP_KERNEL | __GFP_ZERO,
+					    get_order(baser->psz));
 		if (!page)
 			return false;
 
@@ -3418,7 +3450,9 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 	unsigned long *lpi_map = NULL;
 	unsigned long flags;
 	u16 *col_map = NULL;
+	struct page *page;
 	void *itt;
+	int itt_order;
 	int lpi_base;
 	int nr_lpis;
 	int nr_ites;
@@ -3430,7 +3464,6 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 	if (WARN_ON(!is_power_of_2(nvecs)))
 		nvecs = roundup_pow_of_two(nvecs);
 
-	dev = kzalloc(sizeof(*dev), GFP_KERNEL);
 	/*
 	 * Even if the device wants a single LPI, the ITT must be
 	 * sized as a power of two (and you need at least one bit...).
@@ -3438,7 +3471,16 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 	nr_ites = max(2, nvecs);
 	sz = nr_ites * (FIELD_GET(GITS_TYPER_ITT_ENTRY_SIZE, its->typer) + 1);
 	sz = max(sz, ITS_ITT_ALIGN) + ITS_ITT_ALIGN - 1;
-	itt = kzalloc_node(sz, GFP_KERNEL, its->numa_node);
+	itt_order = get_order(sz);
+	page = its_alloc_pages_node(its->numa_node,
Quoted 8 lines

drivers/irqchip/irq-gic-v3-its.c static struct its_device *its_create_device(struct its_node *its, u32 dev_id, · line 3477

Quoted 2 lines
drivers/irqchip/irq-gic-v3-its.c +2 -0
+				    GFP_KERNEL | __GFP_ZERO,
+				    itt_order);

So we go from an allocation that was so far measured in *bytes* to something that is now at least a page. Per device. This seems a bit excessive to me, specially when it isn't conditioned on anything and is now imposed on all platforms, including the non-CCA systems (which are exactly 100% of the machines).

Quoted 7 lines

Catalin asked about this in v2:
https://lore.kernel.org/lkml/[email protected]/

To be honest, I don't have a great handle on how much memory is being wasted here. Within the realm guest I was testing this is rounding up an otherwise 511 byte allocation to a 4k page, and there are 3 of them. Which seems reasonable from a realm guest perspective.

And not that reasonable on a smaller system, such as my own router VM that has a whole lot of devices and very little memory. Not to mention that while CCA is stuck with 4k pages (duh!), the world is moving towards larger pages, meaning that this is wasting even more memory.

Quoted 7 lines

I can see two options to improve here:

1. Add a !is_realm_world() check and return to the previous behaviour when not running in a realm. It's ugly, and doesn't deal with any other potential future memory encryption. cc_platform_has(CC_ATTR_MEM_ENCRYPT) might be preferable? But this means no impact to non-realm guests.

No, this is way too ugly, and doesn't help with things like pKVM.

Quoted 9 lines

2. Use a special (global) memory allocator that does the set_memory_decrypted() dance on the pages that it allocates but allows packing the allocations. I'm not aware of an existing kernel API for this, so it's potentially quite a bit of code. The benefit is that it reduces memory consumption in a realm guest, although fragmentation still means we're likely to see a (small) growth.

Any thoughts on what you think would be best?

I would expect that something similar to kmem_cache could be of help, only with the ability to deal with variable object sizes (in this
case: minimum of 256 bytes, in increments defined by the implementation, and with a 256 byte alignment).

I don't think the ITS is particularly special here, and we should come up with something that is generic enough to support sharing of non-page-sized objects.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
maintainer
#35 lore raw

Re: [PATCH v3 12/14] arm64: realm: Support nonsecure ITS emulation shared

Quoted 9 lines · Marc Zyngier
Quoted 7 lines · Steven Price

2. Use a special (global) memory allocator that does the set_memory_decrypted() dance on the pages that it allocates but allows packing the allocations. I'm not aware of an existing kernel API for this, so it's potentially quite a bit of code. The benefit is that it reduces memory consumption in a realm guest, although fragmentation still means we're likely to see a (small) growth.

Quoted 6 lines
Quoted 1 line

Any thoughts on what you think would be best?

I would expect that something similar to kmem_cache could be of help, only with the ability to deal with variable object sizes (in this
case: minimum of 256 bytes, in increments defined by the implementation, and with a 256 byte alignment).

Hmm, that's doable but not that easy to make generic. We'd need a new class of kmalloc-* caches (e.g. kmalloc-decrypted-*) which use only decrypted pages together with a GFP_DECRYPTED flag or something to get the slab allocator to go for these pages and avoid merging with other caches. It would actually be the page allocator parsing this gfp flag, probably in post_alloc_hook() to set the page decrypted and re-encrypt it in free_pages_prepare(). A slight problem here is that free_pages() doesn't get the gfp flag, so we'd need to store some bit in the page flags. Maybe the flag is not that bad, do we have something like for page_to_phys() to give us the high IPA address for decrypted pages?

Similarly if we go for a kmem_cache (or a few for multiple sizes). One can specify a constructor which could set the memory decrypted but there's no destructor (and also the constructor is per object, not per page, so we'd need some refcounting).

Another approach contained within the driver is to use mempool_create() with our own _alloc_fn/_free_fn that sets the memory decrypted/encrypted accordingly, though sub-page allocations need additional tracking. Also that's fairly similar to kmem_cache, fixed size.

Yet another option would be to wire it somehow in the DMA API but the minimum allocation is already a page size, so we don't gain anything.

What gets somewhat closer to what we need is gen_pool. It can track different sizes, we just need to populate the chunks as needed. I don't think this would work as a generic allocator but may be good enough within the ITS code.

If there's a need for such generic allocations in other parts of the kernel, my preference would be something around kmalloc caches and a new GFP flag (first option; subject to the selling it to the mm folk). But that's more of a separate prototyping effort that may or may not succeed.

Anyway, we could do some hacking around gen_pool as a temporary solution (maybe as a set of patches on top of this series to be easier to revert) and start investigating a proper decrypted page allocator in parallel. We just need to find a victim that has the page allocator fresh in mind (Ryan or Alexandru ;)).

Signature · 1 line
Catalin
#36 lore raw

Re: [PATCH v3 12/14] arm64: realm: Support nonsecure ITS emulation shared

Quoted 52 lines · Catalin Marinas
Quoted 9 lines · Marc Zyngier
Quoted 7 lines · Steven Price

2. Use a special (global) memory allocator that does the set_memory_decrypted() dance on the pages that it allocates but allows packing the allocations. I'm not aware of an existing kernel API for this, so it's potentially quite a bit of code. The benefit is that it reduces memory consumption in a realm guest, although fragmentation still means we're likely to see a (small) growth.

Quoted 6 lines
Quoted 1 line

Any thoughts on what you think would be best?

I would expect that something similar to kmem_cache could be of help, only with the ability to deal with variable object sizes (in this
case: minimum of 256 bytes, in increments defined by the implementation, and with a 256 byte alignment).

Hmm, that's doable but not that easy to make generic. We'd need a new class of kmalloc-* caches (e.g. kmalloc-decrypted-*) which use only decrypted pages together with a GFP_DECRYPTED flag or something to get the slab allocator to go for these pages and avoid merging with other caches. It would actually be the page allocator parsing this gfp flag, probably in post_alloc_hook() to set the page decrypted and re-encrypt it in free_pages_prepare(). A slight problem here is that free_pages() doesn't get the gfp flag, so we'd need to store some bit in the page flags. Maybe the flag is not that bad, do we have something like for page_to_phys() to give us the high IPA address for decrypted pages?

Similarly if we go for a kmem_cache (or a few for multiple sizes). One can specify a constructor which could set the memory decrypted but there's no destructor (and also the constructor is per object, not per page, so we'd need some refcounting).

Another approach contained within the driver is to use mempool_create() with our own _alloc_fn/_free_fn that sets the memory decrypted/encrypted accordingly, though sub-page allocations need additional tracking. Also that's fairly similar to kmem_cache, fixed size.

Yet another option would be to wire it somehow in the DMA API but the minimum allocation is already a page size, so we don't gain anything.

What gets somewhat closer to what we need is gen_pool. It can track different sizes, we just need to populate the chunks as needed. I don't think this would work as a generic allocator but may be good enough within the ITS code.

If there's a need for such generic allocations in other parts of the kernel, my preference would be something around kmalloc caches and a new GFP flag (first option; subject to the selling it to the mm folk). But that's more of a separate prototyping effort that may or may not succeed.

Quoted 5 lines

Anyway, we could do some hacking around gen_pool as a temporary solution (maybe as a set of patches on top of this series to be easier to revert) and start investigating a proper decrypted page allocator in parallel. We just need to find a victim that has the page allocator fresh in mind (Ryan or Alexandru ;)).

Thanks for the suggestions Catalin. I had a go at implementing something with gen_pool - the below (very lightly tested) hack seems to work. This is on top of the current series.

I *think* it should also be safe to drop the whole alignment part with this custom allocator, which could actually save memory. But I haven't quite got my head around that yet.

Steve

----8<-----

drivers/irqchip/irq-gic-v3-its.c +56 -14
index ca72f830f4cc..e78634d4d22c 100644
@@ -12,6 +12,7 @@
 #include <linux/crash_dump.h>
 #include <linux/delay.h>
 #include <linux/efi.h>
+#include <linux/genalloc.h>
 #include <linux/interrupt.h>
 #include <linux/iommu.h>
 #include <linux/iopoll.h>
@@ -165,7 +166,7 @@ struct its_device {
 	struct its_node		*its;
 	struct event_lpi_map	event_map;
 	void			*itt;
-	u32			itt_order;
+	u32			itt_sz;
 	u32			nr_ites;
 	u32			device_id;
 	bool			shared;
@@ -225,6 +226,50 @@ static void its_free_pages(void *addr, unsigned int order)
 	free_pages((unsigned long)addr, order);
 }
 
+static struct gen_pool *itt_pool;
+
+static void *itt_alloc_pool(int node, int size)
+{
+	unsigned long addr;
+	struct page *page;
+
+	if (size >= PAGE_SIZE) {
+		page = its_alloc_pages_node(node,
+					    GFP_KERNEL | __GFP_ZERO,
+					    get_order(size));
+
+		return page_address(page);
+	}
+
+	do {
+		addr = gen_pool_alloc(itt_pool, size);
+		if (addr)
+			break;
+
+		page = its_alloc_pages_node(node, GFP_KERNEL | __GFP_ZERO, 1);
+		if (!page)
+			break;
+
+		gen_pool_add(itt_pool, (unsigned long)page_address(page),
+			     PAGE_SIZE, node);
+	} while (!addr);
+
+	return (void*)addr;
+}
+
+static void itt_free_pool(void *addr, int size)
+{
+	if (!addr)
+		return;
+
+	if (size >= PAGE_SIZE) {
+		its_free_pages(addr, get_order(size));
+		return;
+	}
+
+	gen_pool_free(itt_pool, (unsigned long)addr, size);
+}
+
 /*
  * Skip ITSs that have no vLPIs mapped, unless we're on GICv4.1, as we
  * always have vSGIs mapped.
@@ -3450,9 +3495,7 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 	unsigned long *lpi_map = NULL;
 	unsigned long flags;
 	u16 *col_map = NULL;
-	struct page *page;
 	void *itt;
-	int itt_order;
 	int lpi_base;
 	int nr_lpis;
 	int nr_ites;
@@ -3471,13 +3514,8 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 	nr_ites = max(2, nvecs);
 	sz = nr_ites * (FIELD_GET(GITS_TYPER_ITT_ENTRY_SIZE, its->typer) + 1);
 	sz = max(sz, ITS_ITT_ALIGN) + ITS_ITT_ALIGN - 1;
-	itt_order = get_order(sz);
-	page = its_alloc_pages_node(its->numa_node,
-				    GFP_KERNEL | __GFP_ZERO,
-				    itt_order);
-	if (!page)
-		return NULL;
-	itt = page_address(page);
+
+	itt = itt_alloc_pool(its->numa_node, sz);
 
 	dev = kzalloc(sizeof(*dev), GFP_KERNEL);
 
@@ -3492,9 +3530,9 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 		lpi_base = 0;
 	}
 
-	if (!dev || !col_map || (!lpi_map && alloc_lpis)) {
+	if (!dev || !itt || !col_map || (!lpi_map && alloc_lpis)) {
 		kfree(dev);
-		its_free_pages(itt, itt_order);
+		itt_free_pool(itt, sz);
 		bitmap_free(lpi_map);
 		kfree(col_map);
 		return NULL;
@@ -3504,7 +3542,7 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 
 	dev->its = its;
 	dev->itt = itt;
-	dev->itt_order = itt_order;
+	dev->itt_sz = sz;
 	dev->nr_ites = nr_ites;
 	dev->event_map.lpi_map = lpi_map;
 	dev->event_map.col_map = col_map;
@@ -3532,7 +3570,7 @@ static void its_free_device(struct its_device *its_dev)
 	list_del(&its_dev->entry);
 	raw_spin_unlock_irqrestore(&its_dev->its->lock, flags);
 	kfree(its_dev->event_map.col_map);
-	its_free_pages(its_dev->itt, its_dev->itt_order);
+	itt_free_pool(its_dev->itt, its_dev->itt_sz);
 	kfree(its_dev);
 }
 
@@ -5722,6 +5760,10 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
 	bool has_v4_1 = false;
 	int err;
 
+	itt_pool = gen_pool_create(get_order(ITS_ITT_ALIGN), -1);
+	if (!itt_pool)
+		return -ENOMEM;
+
 	gic_rdists = rdists;
 
 	its_parent = parent_domain;
maintainer
#37 lore raw

Re: [PATCH v3 12/14] arm64: realm: Support nonsecure ITS emulation shared

Quoted 7 lines · Steven Price
Quoted 5 lines · Catalin Marinas

Anyway, we could do some hacking around gen_pool as a temporary solution (maybe as a set of patches on top of this series to be easier to revert) and start investigating a proper decrypted page allocator in parallel. We just need to find a victim that has the page allocator fresh in mind (Ryan or Alexandru ;)).

Quoted 7 lines

Thanks for the suggestions Catalin. I had a go at implementing something with gen_pool - the below (very lightly tested) hack seems to work. This is on top of the current series.

I *think* it should also be safe to drop the whole alignment part with this custom allocator, which could actually save memory. But I haven't quite got my head around that yet.

Thanks Steven. It doesn't look too complex and it solves the memory wasting. We don't actually free the pages from gen_pool but I don't think it matters much, the memory would get reused if devices are removed and re-added.

Signature · 1 line
Catalin
maintainer
#38 lore raw

Re: [PATCH v3 12/14] arm64: realm: Support nonsecure ITS emulation shared

In reply to Catalin Marinas
Quoted 17 lines · Catalin Marinas
Quoted 9 lines · Marc Zyngier
Quoted 7 lines · Steven Price

2. Use a special (global) memory allocator that does the set_memory_decrypted() dance on the pages that it allocates but allows packing the allocations. I'm not aware of an existing kernel API for this, so it's potentially quite a bit of code. The benefit is that it reduces memory consumption in a realm guest, although fragmentation still means we're likely to see a (small) growth.

Quoted 6 lines
Quoted 1 line

Any thoughts on what you think would be best?

I would expect that something similar to kmem_cache could be of help, only with the ability to deal with variable object sizes (in this
case: minimum of 256 bytes, in increments defined by the implementation, and with a 256 byte alignment).

Quoted 10 lines

Hmm, that's doable but not that easy to make generic. We'd need a new class of kmalloc-* caches (e.g. kmalloc-decrypted-*) which use only decrypted pages together with a GFP_DECRYPTED flag or something to get the slab allocator to go for these pages and avoid merging with other caches. It would actually be the page allocator parsing this gfp flag, probably in post_alloc_hook() to set the page decrypted and re-encrypt it in free_pages_prepare(). A slight problem here is that free_pages() doesn't get the gfp flag, so we'd need to store some bit in the page flags. Maybe the flag is not that bad, do we have something like for page_to_phys() to give us the high IPA address for decrypted pages?

I had a go at this generic approach, Friday afternoon fun. Not tested with the CCA patches, just my own fake set_memory_*() functionality on top of 6.10-rc2. I also mindlessly added __GFP_DECRYPTED to the ITS code following the changes in this patch. I noticed that allocate_vpe_l2_table() doesn't use shared pages (I either missed it or it's not needed).

If we want to go this way, I can tidy up the diff below, split it into a proper series and add what's missing. What I can't tell is how a driver writer knows when to pass __GFP_DECRYPTED. There's also a theoretical overlap with __GFP_DMA, it can't handle both. The DMA flag takes priority currently but I realised it probably needs to be the other way around, we wouldn't have dma mask restrictions for emulated devices.

--------------------8<----------------------------------

drivers/irqchip/irq-gic-v3-its.c +12 -8
index 40ebf1726393..b6627e839e62 100644
@@ -2212,7 +2212,8 @@ static struct page *its_allocate_prop_table(gfp_t gfp_flags)
 {
 	struct page *prop_page;
 
-	prop_page = alloc_pages(gfp_flags, get_order(LPI_PROPBASE_SZ));
+	prop_page = alloc_pages(gfp_flags | __GFP_DECRYPTED,
+				get_order(LPI_PROPBASE_SZ));
 	if (!prop_page)
 		return NULL;
 
@@ -2346,7 +2347,8 @@ static int its_setup_baser(struct its_node *its, struct its_baser *baser,
 		order = get_order(GITS_BASER_PAGES_MAX * psz);
 	}
 
-	page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO, order);
+	page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO |
+				__GFP_DECRYPTED, order);
 	if (!page)
 		return -ENOMEM;
 
@@ -2940,7 +2942,8 @@ static int allocate_vpe_l1_table(void)
 
 	pr_debug("np = %d, npg = %lld, psz = %d, epp = %d, esz = %d\n",
 		 np, npg, psz, epp, esz);
-	page = alloc_pages(GFP_ATOMIC | __GFP_ZERO, get_order(np * PAGE_SIZE));
+	page = alloc_pages(GFP_ATOMIC | __GFP_ZERO | __GFP_DECRYPTED,
+			   get_order(np * PAGE_SIZE));
 	if (!page)
 		return -ENOMEM;
 
@@ -2986,7 +2989,7 @@ static struct page *its_allocate_pending_table(gfp_t gfp_flags)
 {
 	struct page *pend_page;
 
-	pend_page = alloc_pages(gfp_flags | __GFP_ZERO,
+	pend_page = alloc_pages(gfp_flags | __GFP_ZERO | __GFP_DECRYPTED,
 				get_order(LPI_PENDBASE_SZ));
 	if (!pend_page)
 		return NULL;
@@ -3334,7 +3337,8 @@ static bool its_alloc_table_entry(struct its_node *its,
 
 	/* Allocate memory for 2nd level table */
 	if (!table[idx]) {
-		page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO,
+		page = alloc_pages_node(its->numa_node, GFP_KERNEL |
+					__GFP_ZERO | __GFP_DECRYPTED,
 					get_order(baser->psz));
 		if (!page)
 			return false;
@@ -3438,7 +3442,7 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
 	nr_ites = max(2, nvecs);
 	sz = nr_ites * (FIELD_GET(GITS_TYPER_ITT_ENTRY_SIZE, its->typer) + 1);
 	sz = max(sz, ITS_ITT_ALIGN) + ITS_ITT_ALIGN - 1;
-	itt = kzalloc_node(sz, GFP_KERNEL, its->numa_node);
+	itt = kzalloc_node(sz, GFP_KERNEL | __GFP_DECRYPTED, its->numa_node);
 	if (alloc_lpis) {
 		lpi_map = its_lpi_alloc(nvecs, &lpi_base, &nr_lpis);
 		if (lpi_map)
@@ -5131,8 +5135,8 @@ static int __init its_probe_one(struct its_node *its)
 		}
 	}
 
-	page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO,
-				get_order(ITS_CMD_QUEUE_SZ));
+	page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO |
+				__GFP_DECRYPTED, get_order(ITS_CMD_QUEUE_SZ));
 	if (!page) {
 		err = -ENOMEM;
 		goto out_unmap_sgir;
include/linux/gfp_types.h +10 -0
index 313be4ad79fd..573989664639 100644
@@ -57,6 +57,9 @@ enum {
 #endif
 #ifdef CONFIG_SLAB_OBJ_EXT
 	___GFP_NO_OBJ_EXT_BIT,
+#endif
+#ifdef CONFIG_ARCH_HAS_MEM_ENCRYPT
+	___GFP_DECRYPTED_BIT,
 #endif
 	___GFP_LAST_BIT
 };
@@ -103,6 +106,11 @@ enum {
 #else
 #define ___GFP_NO_OBJ_EXT       0
 #endif
+#ifdef CONFIG_ARCH_HAS_MEM_ENCRYPT
+#define ___GFP_DECRYPTED	BIT(___GFP_DECRYPTED_BIT)
+#else
+#define ___GFP_DECRYPTED	0
+#endif
 
 /*
  * Physical address zone modifiers (see linux/mmzone.h - low four bits)
@@ -117,6 +125,8 @@ enum {
 #define __GFP_MOVABLE	((__force gfp_t)___GFP_MOVABLE)  /* ZONE_MOVABLE allowed */
 #define GFP_ZONEMASK	(__GFP_DMA|__GFP_HIGHMEM|__GFP_DMA32|__GFP_MOVABLE)
 
+#define __GFP_DECRYPTED	((__force gfp_t)___GFP_DECRYPTED)
+
 /**
  * DOC: Page mobility and placement hints
  *
include/linux/page-flags.h +12 -0
index 104078afe0b1..705707052274 100644
@@ -127,6 +127,9 @@ enum pageflags {
 #ifdef CONFIG_MEMORY_FAILURE
 	PG_hwpoison,		/* hardware poisoned page. Don't touch */
 #endif
+#ifdef CONFIG_ARCH_HAS_MEM_ENCRYPT
+	PG_decrypted,
+#endif
 #if defined(CONFIG_PAGE_IDLE_FLAG) && defined(CONFIG_64BIT)
 	PG_young,
 	PG_idle,
@@ -626,6 +629,15 @@ PAGEFLAG_FALSE(HWPoison, hwpoison)
 #define __PG_HWPOISON 0
 #endif
 
+#ifdef CONFIG_ARCH_HAS_MEM_ENCRYPT
+PAGEFLAG(Decrypted, decrypted, PF_HEAD)
+#define __PG_DECRYPTED	(1UL << PG_decrypted)
+#else
+PAGEFLAG_FALSE(Decrypted, decrypted)
+#define __PG_DECRYPTED	0
+#endif
+
+
 #ifdef CONFIG_PAGE_IDLE_FLAG
 #ifdef CONFIG_64BIT
 FOLIO_TEST_FLAG(young, FOLIO_HEAD_PAGE)
include/linux/slab.h +9 -3
index 7247e217e21b..f7a2cf624c35 100644
@@ -422,6 +422,9 @@ enum kmalloc_cache_type {
 #endif
 #ifdef CONFIG_MEMCG_KMEM
 	KMALLOC_CGROUP,
+#endif
+#ifdef CONFIG_ARCH_HAS_MEM_ENCRYPT
+	KMALLOC_DECRYPTED,
 #endif
 	NR_KMALLOC_TYPES
 };
@@ -433,7 +436,7 @@ kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1];
  * Define gfp bits that should not be set for KMALLOC_NORMAL.
  */
 #define KMALLOC_NOT_NORMAL_BITS					\
-	(__GFP_RECLAIMABLE |					\
+	(__GFP_RECLAIMABLE | __GFP_DECRYPTED |			\
 	(IS_ENABLED(CONFIG_ZONE_DMA)   ? __GFP_DMA : 0) |	\
 	(IS_ENABLED(CONFIG_MEMCG_KMEM) ? __GFP_ACCOUNT : 0))
 
@@ -458,11 +461,14 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigne
 	 * At least one of the flags has to be set. Their priorities in
 	 * decreasing order are:
 	 *  1) __GFP_DMA
-	 *  2) __GFP_RECLAIMABLE
-	 *  3) __GFP_ACCOUNT
+	 *  2) __GFP_DECRYPTED
+	 *  3) __GFP_RECLAIMABLE
+	 *  4) __GFP_ACCOUNT
 	 */
 	if (IS_ENABLED(CONFIG_ZONE_DMA) && (flags & __GFP_DMA))
 		return KMALLOC_DMA;
+	if (IS_ENABLED(CONFIG_ARCH_HAS_MEM_ENCRYPT) && (flags & __GFP_DECRYPTED))
+		return KMALLOC_DECRYPTED;
 	if (!IS_ENABLED(CONFIG_MEMCG_KMEM) || (flags & __GFP_RECLAIMABLE))
 		return KMALLOC_RECLAIM;
 	else
include/trace/events/mmflags.h +7 -0
index e46d6e82765e..a0879155f892 100644
@@ -83,6 +83,12 @@
 #define IF_HAVE_PG_HWPOISON(_name)
 #endif
 
+#ifdef CONFIG_ARCH_HAS_MEM_ENCRYPT
+#define IF_HAVE_PG_DECRYPTED(_name) ,{1UL << PG_##_name, __stringify(_name)}
+#else
+#define IF_HAVE_PG_DECRYPTED(_name)
+#endif
+
 #if defined(CONFIG_PAGE_IDLE_FLAG) && defined(CONFIG_64BIT)
 #define IF_HAVE_PG_IDLE(_name) ,{1UL << PG_##_name, __stringify(_name)}
 #else
@@ -121,6 +127,7 @@
 IF_HAVE_PG_MLOCK(mlocked)						\
 IF_HAVE_PG_UNCACHED(uncached)						\
 IF_HAVE_PG_HWPOISON(hwpoison)						\
+IF_HAVE_PG_DECRYPTED(decrypted)						\
 IF_HAVE_PG_IDLE(idle)							\
 IF_HAVE_PG_IDLE(young)							\
 IF_HAVE_PG_ARCH_X(arch_2)						\
mm/page_alloc.c +14 -0
index 2e22ce5675ca..c93ae50ec402 100644
@@ -47,6 +47,7 @@
 #include <linux/sched/mm.h>
 #include <linux/page_owner.h>
 #include <linux/page_table_check.h>
+#include <linux/set_memory.h>
 #include <linux/memcontrol.h>
 #include <linux/ftrace.h>
 #include <linux/lockdep.h>
@@ -1051,6 +1052,12 @@ __always_inline bool free_pages_prepare(struct page *page,
 		return false;
 	}
 
+	if (unlikely(PageDecrypted(page))) {
+		set_memory_encrypted((unsigned long)page_address(page),
+				     1 << order);
+		ClearPageDecrypted(page);
+	}
+
 	VM_BUG_ON_PAGE(compound && compound_order(page) != order, page);
 
 	/*
@@ -1415,6 +1422,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order,
 	bool init = !want_init_on_free() && want_init_on_alloc(gfp_flags) &&
 			!should_skip_init(gfp_flags);
 	bool zero_tags = init && (gfp_flags & __GFP_ZEROTAGS);
+	bool decrypted = true; //gfp_flags & __GFP_DECRYPTED;
 	int i;
 
 	set_page_private(page, 0);
@@ -1465,6 +1473,12 @@ inline void post_alloc_hook(struct page *page, unsigned int order,
 	if (init)
 		kernel_init_pages(page, 1 << order);
 
+	if (decrypted) {
+		set_memory_decrypted((unsigned long)page_address(page),
+				     1 << order);
+		SetPageDecrypted(page);
+	}
+
 	set_page_owner(page, order, gfp_flags);
 	page_table_check_alloc(page, order);
 	pgalloc_tag_add(page, current, 1 << order);
mm/slab_common.c +10 -0
index 1560a1546bb1..de9c8c674aa1 100644
@@ -737,6 +737,12 @@ EXPORT_SYMBOL(kmalloc_size_roundup);
 #define KMALLOC_RCL_NAME(sz)
 #endif
 
+#ifdef CONFIG_ARCH_HAS_MEM_ENCRYPT
+#define KMALLOC_DECRYPTED_NAME(sz)	.name[KMALLOC_DECRYPTED] = "kmalloc-decrypted-" #sz,
+#else
+#define KMALLOC_DECRYPTED_NAME(sz)
+#endif
+
 #ifdef CONFIG_RANDOM_KMALLOC_CACHES
 #define __KMALLOC_RANDOM_CONCAT(a, b) a ## b
 #define KMALLOC_RANDOM_NAME(N, sz) __KMALLOC_RANDOM_CONCAT(KMA_RAND_, N)(sz)
@@ -765,6 +771,7 @@ EXPORT_SYMBOL(kmalloc_size_roundup);
 	KMALLOC_RCL_NAME(__short_size)				\
 	KMALLOC_CGROUP_NAME(__short_size)			\
 	KMALLOC_DMA_NAME(__short_size)				\
+	KMALLOC_DECRYPTED_NAME(__short_size)			\
 	KMALLOC_RANDOM_NAME(RANDOM_KMALLOC_CACHES_NR, __short_size)	\
 	.size = __size,						\
 }
@@ -889,6 +896,9 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type)
 	if (IS_ENABLED(CONFIG_MEMCG_KMEM) && (type == KMALLOC_NORMAL))
 		flags |= SLAB_NO_MERGE;
 
+	if (IS_ENABLED(CONFIG_ARCH_HAS_MEM_ENCRYPT) && (type == KMALLOC_DECRYPTED))
+		flags |= SLAB_NO_MERGE;
+
 	if (minalign > ARCH_KMALLOC_MINALIGN) {
 		aligned_size = ALIGN(aligned_size, minalign);
 		aligned_idx = __kmalloc_index(aligned_size, false);
#39 lore raw

RE: [PATCH v3 12/14] arm64: realm: Support nonsecure ITS emulation shared

From: Catalin Marinas <[email protected]> Sent: Friday, June 7, 2024 10:55 AM

Quoted 37 lines · Catalin Marinas
Quoted 17 lines · Marc Zyngier
Quoted 9 lines
Quoted 7 lines · Steven Price

2. Use a special (global) memory allocator that does the set_memory_decrypted() dance on the pages that it allocates but allows packing the allocations. I'm not aware of an existing kernel API for this, so it's potentially quite a bit of code. The benefit is that it reduces memory consumption in a realm guest, although fragmentation still means we're likely to see a (small) growth.

Quoted 6 lines
Quoted 1 line

Any thoughts on what you think would be best?

I would expect that something similar to kmem_cache could be of help, only with the ability to deal with variable object sizes (in this
case: minimum of 256 bytes, in increments defined by the implementation, and with a 256 byte alignment).

Quoted 10 lines

Hmm, that's doable but not that easy to make generic. We'd need a new class of kmalloc-* caches (e.g. kmalloc-decrypted-*) which use only decrypted pages together with a GFP_DECRYPTED flag or something to get the slab allocator to go for these pages and avoid merging with other caches. It would actually be the page allocator parsing this gfp flag, probably in post_alloc_hook() to set the page decrypted and re-encrypt it in free_pages_prepare(). A slight problem here is that free_pages() doesn't get the gfp flag, so we'd need to store some bit in the page flags. Maybe the flag is not that bad, do we have something like for page_to_phys() to give us the high IPA address for decrypted pages?

I had a go at this generic approach, Friday afternoon fun. Not tested with the CCA patches, just my own fake set_memory_*() functionality on top of 6.10-rc2. I also mindlessly added __GFP_DECRYPTED to the ITS code following the changes in this patch. I noticed that allocate_vpe_l2_table() doesn't use shared pages (I either missed it or it's not needed).

Quoted 6 lines

If we want to go this way, I can tidy up the diff below, split it into a proper series and add what's missing. What I can't tell is how a driver writer knows when to pass __GFP_DECRYPTED. There's also a theoretical overlap with __GFP_DMA, it can't handle both. The DMA flag takes priority currently but I realised it probably needs to be the other way around, we wouldn't have dma mask restrictions for emulated devices.

As someone who has spent time in Linux code on the x86 side that manages memory shared between the guest and host, a GFP_DECRYPTED flag on the memory allocation calls seems very useful. Here are my general comments, some of which are probably obvious, but I thought I'd write them down anyway.

1) The impetus in this case is to efficiently allow sub-page allocations. But on x86, there's also an issue in that the x86 direct map uses large page mappings, which must be split if any 4K page in the large page is decrypted. Ideally, we could cluster such decrypted pages into the same large page(s) vs. just grabbing any 4K page, and reduce the number of splits. I haven't looked at how feasible that is to implement in the context of the existing allocator code, so this is aspirational.

2) GFP_DECRYPTED should work for the three memory allocation interfaces and their variants: alloc_pages(), kmalloc(), and vmalloc().

3) The current paradigm is to allocate memory and call set_memory_decrypted(). Then do the reverse when freeing memory. Using GFP_DECRYPTED on the allocation, and having the re-encryption done automatically when freeing certainly simplifies the call sites. The error handling at the call sites is additional messiness that gets much simpler.

4) GFP_DECRYPTED should be silently ignored when not running in a CoCo VM. Drivers that need decrypted memory in a CoCo VM always set this flag, and work normally as if the flag isn't set when not in a CoCo VM. This is how set_memory_decrypted()/encrypted() work today. Drivers call them in all cases, and they are no-ops when not in a CoCo VM.

5) The implementation of GFP_DECRYPTED must correctly handle errors from set_memory_decrypted()/encrypted() as I've described separately on this thread.

Michael

Quoted 301 lines

--------------------8<----------------------------------

drivers/irqchip/irq-gic-v3-its.c +7 -4
index 40ebf1726393..b6627e839e62 100644
@@ -2212,7 +2212,8 @@ static struct page *its_allocate_prop_table(gfp_t gfp_flags)
 {
 	struct page *prop_page;
 
-	prop_page = alloc_pages(gfp_flags, get_order(LPI_PROPBASE_SZ));
+	prop_page = alloc_pages(gfp_flags | __GFP_DECRYPTED,
+				get_order(LPI_PROPBASE_SZ));
 	if (!prop_page)
 		return NULL;
 
@@ -2346,7 +2347,8 @@ static int its_setup_baser(struct its_node *its, struct
 its_baser *baser,
 		order = get_order(GITS_BASER_PAGES_MAX * psz);
 	}
 
-	page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO, order);
+	page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO |
+				__GFP_DECRYPTED, order);
 	if (!page)
 		return -ENOMEM;
@@ -2940,7 +2942,8 @@ static int allocate_vpe_l1_table(void)
 
 	pr_debug("np = %d, npg = %lld, psz = %d, epp = %d, esz = %d\n",
 		 np, npg, psz, epp, esz);
-	page = alloc_pages(GFP_ATOMIC | __GFP_ZERO, get_order(np * PAGE_SIZE));
+	page = alloc_pages(GFP_ATOMIC | __GFP_ZERO | __GFP_DECRYPTED,
+			   get_order(np * PAGE_SIZE));
 	if (!page)
 		return -ENOMEM;
 
@@ -2986,7 +2989,7 @@ static struct page *its_allocate_pending_table(gfp_t
 gfp_flags)
 {
 	struct page *pend_page;
 
-	pend_page = alloc_pages(gfp_flags | __GFP_ZERO,
+	pend_page = alloc_pages(gfp_flags | __GFP_ZERO | __GFP_DECRYPTED,
 				get_order(LPI_PENDBASE_SZ));
 	if (!pend_page)
drivers/irqchip/irq-gic-v3-its.c +3 -2
 		return NULL;
@@ -3334,7 +3337,8 @@ static bool its_alloc_table_entry(struct its_node *its,
 
 	/* Allocate memory for 2nd level table */
 	if (!table[idx]) {
-		page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO,
+		page = alloc_pages_node(its->numa_node, GFP_KERNEL |
+					__GFP_ZERO | __GFP_DECRYPTED,
 					get_order(baser->psz));
 		if (!page)
 			return false;
@@ -3438,7 +3442,7 @@ static struct its_device *its_create_device(struct its_node
 *its, u32 dev_id,
 	nr_ites = max(2, nvecs);
 	sz = nr_ites * (FIELD_GET(GITS_TYPER_ITT_ENTRY_SIZE, its->typer) + 1);
 	sz = max(sz, ITS_ITT_ALIGN) + ITS_ITT_ALIGN - 1;
-	itt = kzalloc_node(sz, GFP_KERNEL, its->numa_node);
+	itt = kzalloc_node(sz, GFP_KERNEL | __GFP_DECRYPTED, its->numa_node);
 	if (alloc_lpis) {
 		lpi_map = its_lpi_alloc(nvecs, &lpi_base, &nr_lpis);
drivers/irqchip/irq-gic-v3-its.c +2 -2
 		if (lpi_map)
@@ -5131,8 +5135,8 @@ static int __init its_probe_one(struct its_node *its)
 		}
 	}
 
-	page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO,
-				get_order(ITS_CMD_QUEUE_SZ));
+	page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO |
+				__GFP_DECRYPTED, get_order(ITS_CMD_QUEUE_SZ));
 	if (!page) {
 		err = -ENOMEM;
 		goto out_unmap_sgir;
include/linux/gfp_types.h +8 -0
index 313be4ad79fd..573989664639 100644
@@ -57,6 +57,9 @@ enum {
 #endif
 #ifdef CONFIG_SLAB_OBJ_EXT
 	___GFP_NO_OBJ_EXT_BIT,
+#endif
+#ifdef CONFIG_ARCH_HAS_MEM_ENCRYPT
+	___GFP_DECRYPTED_BIT,
 #endif
 	___GFP_LAST_BIT
 };
@@ -103,6 +106,11 @@ enum {
 #else
 #define ___GFP_NO_OBJ_EXT       0
 #endif
+#ifdef CONFIG_ARCH_HAS_MEM_ENCRYPT
+#define ___GFP_DECRYPTED	BIT(___GFP_DECRYPTED_BIT)
+#else
+#define ___GFP_DECRYPTED	0
+#endif
 
 /*
  * Physical address zone modifiers (see linux/mmzone.h - low four bits)
@@ -117,6 +125,8 @@ enum {
 #define __GFP_MOVABLE	((__force gfp_t)___GFP_MOVABLE)  /*

ZONE_MOVABLE allowed */

 #define GFP_ZONEMASK
	(__GFP_DMA|__GFP_HIGHMEM|__GFP_DMA32|__GFP_MOVABLE)
include/linux/gfp_types.h +2 -0
+#define __GFP_DECRYPTED	((__force gfp_t)___GFP_DECRYPTED)
+
 /**
  * DOC: Page mobility and placement hints
  *
include/linux/page-flags.h +12 -0
index 104078afe0b1..705707052274 100644
@@ -127,6 +127,9 @@ enum pageflags {
 #ifdef CONFIG_MEMORY_FAILURE
 	PG_hwpoison,		/* hardware poisoned page. Don't touch */
 #endif
+#ifdef CONFIG_ARCH_HAS_MEM_ENCRYPT
+	PG_decrypted,
+#endif
 #if defined(CONFIG_PAGE_IDLE_FLAG) && defined(CONFIG_64BIT)
 	PG_young,
 	PG_idle,
@@ -626,6 +629,15 @@ PAGEFLAG_FALSE(HWPoison, hwpoison)
 #define __PG_HWPOISON 0
 #endif
 
+#ifdef CONFIG_ARCH_HAS_MEM_ENCRYPT
+PAGEFLAG(Decrypted, decrypted, PF_HEAD)
+#define __PG_DECRYPTED	(1UL << PG_decrypted)
+#else
+PAGEFLAG_FALSE(Decrypted, decrypted)
+#define __PG_DECRYPTED	0
+#endif
+
+
 #ifdef CONFIG_PAGE_IDLE_FLAG
 #ifdef CONFIG_64BIT
 FOLIO_TEST_FLAG(young, FOLIO_HEAD_PAGE)
include/linux/slab.h +5 -1
index 7247e217e21b..f7a2cf624c35 100644
@@ -422,6 +422,9 @@ enum kmalloc_cache_type {
 #endif
 #ifdef CONFIG_MEMCG_KMEM
 	KMALLOC_CGROUP,
+#endif
+#ifdef CONFIG_ARCH_HAS_MEM_ENCRYPT
+	KMALLOC_DECRYPTED,
 #endif
 	NR_KMALLOC_TYPES
 };
@@ -433,7 +436,7 @@ kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH
+ 1];
  * Define gfp bits that should not be set for KMALLOC_NORMAL.
  */
 #define KMALLOC_NOT_NORMAL_BITS					\
-	(__GFP_RECLAIMABLE |					\
+	(__GFP_RECLAIMABLE | __GFP_DECRYPTED |			\
 	(IS_ENABLED(CONFIG_ZONE_DMA)   ? __GFP_DMA : 0) |	\
 	(IS_ENABLED(CONFIG_MEMCG_KMEM) ? __GFP_ACCOUNT : 0))
@@ -458,11 +461,14 @@ static __always_inline enum kmalloc_cache_type

kmalloc_type(gfp_t flags, unsigne

include/linux/slab.h +4 -2
 	 * At least one of the flags has to be set. Their priorities in
 	 * decreasing order are:
 	 *  1) __GFP_DMA
-	 *  2) __GFP_RECLAIMABLE
-	 *  3) __GFP_ACCOUNT
+	 *  2) __GFP_DECRYPTED
+	 *  3) __GFP_RECLAIMABLE
+	 *  4) __GFP_ACCOUNT
 	 */
 	if (IS_ENABLED(CONFIG_ZONE_DMA) && (flags & __GFP_DMA))
 		return KMALLOC_DMA;
+	if (IS_ENABLED(CONFIG_ARCH_HAS_MEM_ENCRYPT) && (flags &

__GFP_DECRYPTED))

include/linux/slab.h +1 -0
+		return KMALLOC_DECRYPTED;
 	if (!IS_ENABLED(CONFIG_MEMCG_KMEM) || (flags & __GFP_RECLAIMABLE))
 		return KMALLOC_RECLAIM;
 	else
include/trace/events/mmflags.h +7 -0
index e46d6e82765e..a0879155f892 100644
@@ -83,6 +83,12 @@
 #define IF_HAVE_PG_HWPOISON(_name)
 #endif
 
+#ifdef CONFIG_ARCH_HAS_MEM_ENCRYPT
+#define IF_HAVE_PG_DECRYPTED(_name) ,{1UL << PG_##_name, __stringify(_name)}
+#else
+#define IF_HAVE_PG_DECRYPTED(_name)
+#endif
+
 #if defined(CONFIG_PAGE_IDLE_FLAG) && defined(CONFIG_64BIT)
 #define IF_HAVE_PG_IDLE(_name) ,{1UL << PG_##_name, __stringify(_name)}
 #else
@@ -121,6 +127,7 @@
 IF_HAVE_PG_MLOCK(mlocked)						\
 IF_HAVE_PG_UNCACHED(uncached)						\
 IF_HAVE_PG_HWPOISON(hwpoison)						\
+IF_HAVE_PG_DECRYPTED(decrypted)						\
 IF_HAVE_PG_IDLE(idle)							\
 IF_HAVE_PG_IDLE(young)							\
 IF_HAVE_PG_ARCH_X(arch_2)						\
mm/page_alloc.c +7 -0
index 2e22ce5675ca..c93ae50ec402 100644
@@ -47,6 +47,7 @@
 #include <linux/sched/mm.h>
 #include <linux/page_owner.h>
 #include <linux/page_table_check.h>
+#include <linux/set_memory.h>
 #include <linux/memcontrol.h>
 #include <linux/ftrace.h>
 #include <linux/lockdep.h>
@@ -1051,6 +1052,12 @@ __always_inline bool free_pages_prepare(struct page
 *page,
 		return false;
 	}
 
+	if (unlikely(PageDecrypted(page))) {
+		set_memory_encrypted((unsigned long)page_address(page),
+				     1 << order);
+		ClearPageDecrypted(page);
+	}
+
 	VM_BUG_ON_PAGE(compound && compound_order(page) != order, page);
 
mm/page_alloc.c +1 -0
 	/*
@@ -1415,6 +1422,7 @@ inline void post_alloc_hook(struct page *page, unsigned int
 order,
 	bool init = !want_init_on_free() && want_init_on_alloc(gfp_flags) &&
 			!should_skip_init(gfp_flags);
 	bool zero_tags = init && (gfp_flags & __GFP_ZEROTAGS);
+	bool decrypted = true; //gfp_flags & __GFP_DECRYPTED;
 	int i;
 
mm/page_alloc.c +6 -0
 	set_page_private(page, 0);
@@ -1465,6 +1473,12 @@ inline void post_alloc_hook(struct page *page, unsigned int
 order,
 	if (init)
 		kernel_init_pages(page, 1 << order);
 
+	if (decrypted) {
+		set_memory_decrypted((unsigned long)page_address(page),
+				     1 << order);
+		SetPageDecrypted(page);
+	}
+
 	set_page_owner(page, order, gfp_flags);
 	page_table_check_alloc(page, order);
mm/page_alloc.c +0 -0
 	pgalloc_tag_add(page, current, 1 << order);
mm/slab_common.c +6 -0
index 1560a1546bb1..de9c8c674aa1 100644
@@ -737,6 +737,12 @@ EXPORT_SYMBOL(kmalloc_size_roundup);
 #define KMALLOC_RCL_NAME(sz)
 #endif
 
+#ifdef CONFIG_ARCH_HAS_MEM_ENCRYPT
+#define KMALLOC_DECRYPTED_NAME(sz)	.name[KMALLOC_DECRYPTED] =
 "kmalloc-decrypted-" #sz,
+#else
+#define KMALLOC_DECRYPTED_NAME(sz)
+#endif
+
 #ifdef CONFIG_RANDOM_KMALLOC_CACHES
 #define __KMALLOC_RANDOM_CONCAT(a, b) a ## b

#define KMALLOC_RANDOM_NAME(N, sz)
__KMALLOC_RANDOM_CONCAT(KMA_RAND_, N)(sz)

mm/slab_common.c +1 -0
@@ -765,6 +771,7 @@ EXPORT_SYMBOL(kmalloc_size_roundup);
 	KMALLOC_RCL_NAME(__short_size)				\
 	KMALLOC_CGROUP_NAME(__short_size)			\
 	KMALLOC_DMA_NAME(__short_size)				\
+	KMALLOC_DECRYPTED_NAME(__short_size)			\
 	KMALLOC_RANDOM_NAME(RANDOM_KMALLOC_CACHES_NR, __short_size)
 	\
 	.size = __size,						\

}

mm/slab_common.c +3 -0
@@ -889,6 +896,9 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type)
 	if (IS_ENABLED(CONFIG_MEMCG_KMEM) && (type == KMALLOC_NORMAL))
 		flags |= SLAB_NO_MERGE;
 
+	if (IS_ENABLED(CONFIG_ARCH_HAS_MEM_ENCRYPT) && (type ==
 KMALLOC_DECRYPTED))
+		flags |= SLAB_NO_MERGE;
+
 	if (minalign > ARCH_KMALLOC_MINALIGN) {
 		aligned_size = ALIGN(aligned_size, minalign);

aligned_idx = __kmalloc_index(aligned_size, false);

maintainer
#40 lore raw

Re: [PATCH v3 12/14] arm64: realm: Support nonsecure ITS emulation shared

Hi Michael,

All good points below, thanks.

Quoted 6 lines · Michael Kelley

As someone who has spent time in Linux code on the x86 side that manages memory shared between the guest and host, a GFP_DECRYPTED flag on the memory allocation calls seems very useful. Here are my general comments, some of which are probably obvious, but I thought I'd write them down anyway.

Quoted 7 lines

1) The impetus in this case is to efficiently allow sub-page allocations. But on x86, there's also an issue in that the x86 direct map uses large page mappings, which must be split if any 4K page in the large page is decrypted. Ideally, we could cluster such decrypted pages into the same large page(s) vs. just grabbing any 4K page, and reduce the number of splits. I haven't looked at how feasible that is to implement in the context of the existing allocator code, so this is aspirational.

It might be doable, though it's a lot more intrusive than just a GFP flag. However, such memory cannot be used for any other things than sharing data with the host.

Quoted 2 lines

2) GFP_DECRYPTED should work for the three memory allocation interfaces and their variants: alloc_pages(), kmalloc(), and vmalloc().

Yes. There are two main aspects: (1) the page allocation + linear map change and (2) additional mapping attributes like in the vmalloc() case.

Quoted 7 lines

3) The current paradigm is to allocate memory and call set_memory_decrypted(). Then do the reverse when freeing memory. Using GFP_DECRYPTED on the allocation, and having the re-encryption done automatically when freeing certainly simplifies the call sites. The error handling at the call sites is additional messiness that gets much simpler.

Quoted 6 lines

4) GFP_DECRYPTED should be silently ignored when not running in a CoCo VM. Drivers that need decrypted memory in a CoCo VM always set this flag, and work normally as if the flag isn't set when not in a CoCo VM. This is how set_memory_decrypted()/encrypted() work today. Drivers call them in all cases, and they are no-ops when not in a CoCo VM.

Yes, this should be straightforward. Maybe simplified further to avoid creating additional kmalloc() caches if these functions are not supported, though we seem to be missing some generic arch_has_mem_encrypt() API.

Quoted 3 lines

5) The implementation of GFP_DECRYPTED must correctly handle errors from set_memory_decrypted()/encrypted() as I've described separately on this thread.

Good point, I completely missed this aspect.

I'll prepare some patches and post them next week to get feedback from the mm folk.

Thanks.

Signature · 1 line
Catalin
#41 lore raw

RE: [PATCH v3 12/14] arm64: realm: Support nonsecure ITS emulation shared

In reply to Steven Price

From: Steven Price <[email protected]> Sent: Wednesday, June 5, 2024 2:30 AM

Quoted 52 lines

Within a realm guest the ITS is emulated by the host. This means the allocations must have been made available to the host by a call to set_memory_decrypted(). Introduce an allocation function which performs this extra call.

---
Changes since v2:
* Drop 'shared' from the new its_xxx function names as they are used for non-realm guests too.
* Don't handle the NUMA_NO_NODE case specially - alloc_pages_node() should do the right thing.
* Drop a pointless (void *) cast.

---
 drivers/irqchip/irq-gic-v3-its.c | 90 ++++++++++++++++++++++++--------
 1 file changed, 67 insertions(+), 23 deletions(-)
drivers/irqchip/irq-gic-v3-its.c +3 -0
index 40ebf1726393..ca72f830f4cc 100644
@@ -18,6 +18,7 @@
 #include <linux/irqdomain.h>
 #include <linux/list.h>
 #include <linux/log2.h>
+#include <linux/mem_encrypt.h>
 #include <linux/memblock.h>
 #include <linux/mm.h>
 #include <linux/msi.h>
@@ -27,6 +28,7 @@
 #include <linux/of_pci.h>
 #include <linux/of_platform.h>
 #include <linux/percpu.h>
+#include <linux/set_memory.h>
 #include <linux/slab.h>
 #include <linux/syscore_ops.h>
 
@@ -163,6 +165,7 @@ struct its_device {
 	struct its_node		*its;
 	struct event_lpi_map	event_map;
 	void			*itt;
+	u32			itt_order;
 	u32			nr_ites;
 	u32			device_id;
 	bool			shared;
@@ -198,6 +201,30 @@ static DEFINE_IDA(its_vpeid_ida);
 #define gic_data_rdist_rd_base()	(gic_data_rdist()->rd_base)
 #define gic_data_rdist_vlpi_base()	(gic_data_rdist_rd_base() + SZ_128K)
 

drivers/irqchip/irq-gic-v3-its.c static DEFINE_IDA(its_vpeid_ida); · line 213

Quoted 10 lines
drivers/irqchip/irq-gic-v3-its.c +10 -0
+static struct page *its_alloc_pages_node(int node, gfp_t gfp,
+					 unsigned int order)
+{
+	struct page *page;
+
+	page = alloc_pages_node(node, gfp, order);
+
+	if (page)
+		set_memory_decrypted((unsigned long)page_address(page),
+				     1 << order);

There's been considerable discussion on the x86 side about what to do when set_memory_decrypted() or set_memory_encrypted() fails. The conclusions are:

1) set_memory_decrypted()/encrypted() *could* fail due to a compromised/malicious host, due to a bug somewhere in the software stack, or due to resource constraints (x86 might need to split a large page mapping, and need to allocate additional page table pages, which could fail).

2) The guest memory that was the target of such a failed call could be left in an indeterminate state that the guest could not reliably undo or correct. The guest's view of the page's decrypted/encrypted state might not match the host's view. Therefore, any such guest memory must be leaked rather than being used or put back on the free list.

3) After some discussion, we decided not to panic in such a case. Instead, set_memory_decrypted()/encrypted() generates a WARN, as well as returns a failure result. The most security conscious users could set panic_on_warn=1 in their VMs, and thereby stop further operation if there any indication that the transition between encrypted and decrypt is suspect. The caller of these functions also can take explicit action in the case of a failure.

It seems like the same guidelines should apply here. On the x86 side we've also cleaned up cases where the return value isn't checked, like here and the use of set_memory_encrypted() below.

Michael

Quoted 228 lines
drivers/irqchip/irq-gic-v3-its.c +31 -13
+	return page;
+}
+
+static struct page *its_alloc_pages(gfp_t gfp, unsigned int order)
+{
+	return its_alloc_pages_node(NUMA_NO_NODE, gfp, order);
+}
+
+static void its_free_pages(void *addr, unsigned int order)
+{
+	set_memory_encrypted((unsigned long)addr, 1 << order);
+	free_pages((unsigned long)addr, order);
+}
+
 /*
  * Skip ITSs that have no vLPIs mapped, unless we're on GICv4.1, as we
  * always have vSGIs mapped.
@@ -2212,7 +2239,8 @@ static struct page *its_allocate_prop_table(gfp_t gfp_flags)
 {
 	struct page *prop_page;
 
-	prop_page = alloc_pages(gfp_flags, get_order(LPI_PROPBASE_SZ));
+	prop_page = its_alloc_pages(gfp_flags,
+				    get_order(LPI_PROPBASE_SZ));
 	if (!prop_page)
 		return NULL;
 
@@ -2223,8 +2251,8 @@ static struct page *its_allocate_prop_table(gfp_t gfp_flags)
 
 static void its_free_prop_table(struct page *prop_page)
 {
-	free_pages((unsigned long)page_address(prop_page),
-		   get_order(LPI_PROPBASE_SZ));
+	its_free_pages(page_address(prop_page),
+		       get_order(LPI_PROPBASE_SZ));
 }
 
 static bool gic_check_reserved_range(phys_addr_t addr, unsigned long size)
@@ -2346,7 +2374,8 @@ static int its_setup_baser(struct its_node *its, struct
 its_baser *baser,
 		order = get_order(GITS_BASER_PAGES_MAX * psz);
 	}
 
-	page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO, order);
+	page = its_alloc_pages_node(its->numa_node,
+				    GFP_KERNEL | __GFP_ZERO, order);
 	if (!page)
 		return -ENOMEM;
@@ -2359,7 +2388,7 @@ static int its_setup_baser(struct its_node *its, struct
 its_baser *baser,
 		/* 52bit PA is supported only when PageSize=64K */
 		if (psz != SZ_64K) {
 			pr_err("ITS: no 52bit PA support when psz=%d\n", psz);
-			free_pages((unsigned long)base, order);
+			its_free_pages(base, order);
 			return -ENXIO;
 		}
@@ -2415,7 +2444,7 @@ static int its_setup_baser(struct its_node *its, struct
 its_baser *baser,
 		pr_err("ITS@%pa: %s doesn't stick: %llx %llx\n",
 		       &its->phys_base, its_base_type_string[type],
 		       val, tmp);
-		free_pages((unsigned long)base, order);
+		its_free_pages(base, order);
 		return -ENXIO;
 	}
@@ -2554,8 +2583,8 @@ static void its_free_tables(struct its_node *its)
 
 	for (i = 0; i < GITS_BASER_NR_REGS; i++) {
 		if (its->tables[i].base) {
-			free_pages((unsigned long)its->tables[i].base,
-				   its->tables[i].order);
+			its_free_pages(its->tables[i].base,
+				       its->tables[i].order);
 			its->tables[i].base = NULL;
 		}
 	}
@@ -2821,7 +2850,8 @@ static bool allocate_vpe_l2_table(int cpu, u32 id)
 
 	/* Allocate memory for 2nd level table */
 	if (!table[idx]) {
-		page = alloc_pages(GFP_KERNEL | __GFP_ZERO, get_order(psz));
+		page = its_alloc_pages(GFP_KERNEL | __GFP_ZERO,
+				       get_order(psz));
 		if (!page)
 			return false;
 
@@ -2940,7 +2970,8 @@ static int allocate_vpe_l1_table(void)
 
 	pr_debug("np = %d, npg = %lld, psz = %d, epp = %d, esz = %d\n",
 		 np, npg, psz, epp, esz);
-	page = alloc_pages(GFP_ATOMIC | __GFP_ZERO, get_order(np * PAGE_SIZE));
+	page = its_alloc_pages(GFP_ATOMIC | __GFP_ZERO,
+			       get_order(np * PAGE_SIZE));
 	if (!page)
 		return -ENOMEM;
 
@@ -2986,8 +3017,8 @@ static struct page *its_allocate_pending_table(gfp_t
 gfp_flags)
 {
 	struct page *pend_page;
 
-	pend_page = alloc_pages(gfp_flags | __GFP_ZERO,
-				get_order(LPI_PENDBASE_SZ));
+	pend_page = its_alloc_pages(gfp_flags | __GFP_ZERO,
+				    get_order(LPI_PENDBASE_SZ));
 	if (!pend_page)
 		return NULL;
@@ -2999,7 +3030,7 @@ static struct page *its_allocate_pending_table(gfp_t
 gfp_flags)
 
 static void its_free_pending_table(struct page *pt)
 {
-	free_pages((unsigned long)page_address(pt), get_order(LPI_PENDBASE_SZ));
+	its_free_pages(page_address(pt), get_order(LPI_PENDBASE_SZ));
 }
 

/*

drivers/irqchip/irq-gic-v3-its.c +5 -2
@@ -3334,8 +3365,9 @@ static bool its_alloc_table_entry(struct its_node *its,
 
 	/* Allocate memory for 2nd level table */
 	if (!table[idx]) {
-		page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO,
-					get_order(baser->psz));
+		page = its_alloc_pages_node(its->numa_node,
+					    GFP_KERNEL | __GFP_ZERO,
+					    get_order(baser->psz));
 		if (!page)
 			return false;
 
@@ -3418,7 +3450,9 @@ static struct its_device *its_create_device(struct its_node
 *its, u32 dev_id,
 	unsigned long *lpi_map = NULL;
 	unsigned long flags;
 	u16 *col_map = NULL;
+	struct page *page;
 	void *itt;
+	int itt_order;
 	int lpi_base;
 	int nr_lpis;
drivers/irqchip/irq-gic-v3-its.c +0 -1
 	int nr_ites;
@@ -3430,7 +3464,6 @@ static struct its_device *its_create_device(struct its_node
 *its, u32 dev_id,
 	if (WARN_ON(!is_power_of_2(nvecs)))
 		nvecs = roundup_pow_of_two(nvecs);
 
-	dev = kzalloc(sizeof(*dev), GFP_KERNEL);
 	/*
 	 * Even if the device wants a single LPI, the ITT must be
drivers/irqchip/irq-gic-v3-its.c +10 -1
 	 * sized as a power of two (and you need at least one bit...).
@@ -3438,7 +3471,16 @@ static struct its_device *its_create_device(struct its_node
 *its, u32 dev_id,
 	nr_ites = max(2, nvecs);
 	sz = nr_ites * (FIELD_GET(GITS_TYPER_ITT_ENTRY_SIZE, its->typer) + 1);
 	sz = max(sz, ITS_ITT_ALIGN) + ITS_ITT_ALIGN - 1;
-	itt = kzalloc_node(sz, GFP_KERNEL, its->numa_node);
+	itt_order = get_order(sz);
+	page = its_alloc_pages_node(its->numa_node,
+				    GFP_KERNEL | __GFP_ZERO,
+				    itt_order);
+	if (!page)
+		return NULL;
+	itt = page_address(page);
+
+	dev = kzalloc(sizeof(*dev), GFP_KERNEL);
+
 	if (alloc_lpis) {
 		lpi_map = its_lpi_alloc(nvecs, &lpi_base, &nr_lpis);
drivers/irqchip/irq-gic-v3-its.c +2 -2
 		if (lpi_map)
@@ -3450,9 +3492,9 @@ static struct its_device *its_create_device(struct its_node
 *its, u32 dev_id,
 		lpi_base = 0;
 	}
 
-	if (!dev || !itt ||  !col_map || (!lpi_map && alloc_lpis)) {
+	if (!dev || !col_map || (!lpi_map && alloc_lpis)) {
 		kfree(dev);
-		kfree(itt);
+		its_free_pages(itt, itt_order);
 		bitmap_free(lpi_map);
 		kfree(col_map);
drivers/irqchip/irq-gic-v3-its.c +1 -0
 		return NULL;
@@ -3462,6 +3504,7 @@ static struct its_device *its_create_device(struct its_node
 *its, u32 dev_id,
 
 	dev->its = its;
 	dev->itt = itt;
+	dev->itt_order = itt_order;
 	dev->nr_ites = nr_ites;
 	dev->event_map.lpi_map = lpi_map;
drivers/irqchip/irq-gic-v3-its.c +5 -5
 	dev->event_map.col_map = col_map;
@@ -3489,7 +3532,7 @@ static void its_free_device(struct its_device *its_dev)
 	list_del(&its_dev->entry);
 	raw_spin_unlock_irqrestore(&its_dev->its->lock, flags);
 	kfree(its_dev->event_map.col_map);
-	kfree(its_dev->itt);
+	its_free_pages(its_dev->itt, its_dev->itt_order);
 	kfree(its_dev);
 }
 
@@ -5131,8 +5174,9 @@ static int __init its_probe_one(struct its_node *its)
 		}
 	}
 
-	page = alloc_pages_node(its->numa_node, GFP_KERNEL | __GFP_ZERO,
-				get_order(ITS_CMD_QUEUE_SZ));
+	page = its_alloc_pages_node(its->numa_node,
+				    GFP_KERNEL | __GFP_ZERO,
+				    get_order(ITS_CMD_QUEUE_SZ));
 	if (!page) {
 		err = -ENOMEM;
 		goto out_unmap_sgir;
@@ -5196,7 +5240,7 @@ static int __init its_probe_one(struct its_node *its)
 out_free_tables:
 	its_free_tables(its);
 out_free_cmd:
-	free_pages((unsigned long)its->cmd_base, get_order(ITS_CMD_QUEUE_SZ));
+	its_free_pages(its->cmd_base, get_order(ITS_CMD_QUEUE_SZ));
 out_unmap_sgir:
 	if (its->sgir_base)
 		iounmap(its->sgir_base);
--

2.34.1

#42 lore raw

Re: [PATCH v3 12/14] arm64: realm: Support nonsecure ITS emulation shared

Quoted 89 lines · Michael Kelley

From: Steven Price <[email protected]> Sent: Wednesday, June 5, 2024 2:30 AM

Quoted 52 lines

Within a realm guest the ITS is emulated by the host. This means the allocations must have been made available to the host by a call to set_memory_decrypted(). Introduce an allocation function which performs this extra call.

---
Changes since v2:
* Drop 'shared' from the new its_xxx function names as they are used for non-realm guests too.
* Don't handle the NUMA_NO_NODE case specially - alloc_pages_node() should do the right thing.
* Drop a pointless (void *) cast.

---
 drivers/irqchip/irq-gic-v3-its.c | 90 ++++++++++++++++++++++++--------
 1 file changed, 67 insertions(+), 23 deletions(-)
drivers/irqchip/irq-gic-v3-its.c +3 -0
index 40ebf1726393..ca72f830f4cc 100644
@@ -18,6 +18,7 @@
 #include <linux/irqdomain.h>
 #include <linux/list.h>
 #include <linux/log2.h>
+#include <linux/mem_encrypt.h>
 #include <linux/memblock.h>
 #include <linux/mm.h>
 #include <linux/msi.h>
@@ -27,6 +28,7 @@
 #include <linux/of_pci.h>
 #include <linux/of_platform.h>
 #include <linux/percpu.h>
+#include <linux/set_memory.h>
 #include <linux/slab.h>
 #include <linux/syscore_ops.h>
 
@@ -163,6 +165,7 @@ struct its_device {
 	struct its_node		*its;
 	struct event_lpi_map	event_map;
 	void			*itt;
+	u32			itt_order;
 	u32			nr_ites;
 	u32			device_id;
 	bool			shared;
@@ -198,6 +201,30 @@ static DEFINE_IDA(its_vpeid_ida);
 #define gic_data_rdist_rd_base()	(gic_data_rdist()->rd_base)
 #define gic_data_rdist_vlpi_base()	(gic_data_rdist_rd_base() + SZ_128K)
 

drivers/irqchip/irq-gic-v3-its.c static DEFINE_IDA(its_vpeid_ida); · line 213

Quoted 10 lines
drivers/irqchip/irq-gic-v3-its.c +10 -0
+static struct page *its_alloc_pages_node(int node, gfp_t gfp,
+					 unsigned int order)
+{
+	struct page *page;
+
+	page = alloc_pages_node(node, gfp, order);
+
+	if (page)
+		set_memory_decrypted((unsigned long)page_address(page),
+				     1 << order);

There's been considerable discussion on the x86 side about what to do when set_memory_decrypted() or set_memory_encrypted() fails. The conclusions are:

1) set_memory_decrypted()/encrypted() *could* fail due to a compromised/malicious host, due to a bug somewhere in the software stack, or due to resource constraints (x86 might need to split a large page mapping, and need to allocate additional page table pages, which could fail).

2) The guest memory that was the target of such a failed call could be left in an indeterminate state that the guest could not reliably undo or correct. The guest's view of the page's decrypted/encrypted state might not match the host's view. Therefore, any such guest memory must be leaked rather than being used or put back on the free list.

3) After some discussion, we decided not to panic in such a case. Instead, set_memory_decrypted()/encrypted() generates a WARN, as well as returns a failure result. The most security conscious users could set panic_on_warn=1 in their VMs, and thereby stop further operation if there any indication that the transition between encrypted and decrypt is suspect. The caller of these functions also can take explicit action in the case of a failure.

Quoted 3 lines

It seems like the same guidelines should apply here. On the x86 side we've also cleaned up cases where the return value isn't checked, like here and the use of set_memory_encrypted() below.

Very good points - this code was lacking error handling. I think you are also right that the correct situation when set_memory_{en,de}crypted() fails is to WARN() and leak the page. It's something that shouldn't happen with a well behaving host and it's unclear how to safely recover the page - so leaking the page is the safest result. And the WARN() approach gives the user the option as to whether this is fatal via panic_on_warn.

Thanks,
Steve

#43 lore raw

PATCH v3 13/14 arm64: rsi: Interfaces to query attestation token

In reply to Steven Price

From: Sami Mujawar <[email protected]>

Add interfaces to query the attestation token using the RSI calls.

---
v3: Prefix GRANULE_xxx defines with RSI_.

---
 arch/arm64/include/asm/rsi_cmds.h | 74 +++++++++++++++++++++++++++++++
 1 file changed, 74 insertions(+)
arch/arm64/include/asm/rsi_cmds.h +74 -0
index ab8ad435f10e..ca0ea5929ecc 100644
@@ -10,6 +10,9 @@
 
 #include <asm/rsi_smc.h>
 
+#define RSI_GRANULE_SHIFT		12
+#define RSI_GRANULE_SIZE		(_AC(1, UL) << RSI_GRANULE_SHIFT)
+
 enum ripas {
 	RSI_RIPAS_EMPTY,
 	RSI_RIPAS_RAM,
@@ -66,4 +69,75 @@ static inline unsigned long rsi_set_addr_range_state(phys_addr_t start,
 	return res.a0;
 }
 
+/**
+ * rsi_attestation_token_init - Initialise the operation to retrieve an
+ * attestation token.
+ *
+ * @challenge:	The challenge data to be used in the attestation token
+ *		generation.
+ * @size:	Size of the challenge data in bytes.
+ *
+ * Initialises the attestation token generation and returns an upper bound
+ * on the attestation token size that can be used to allocate an adequate
+ * buffer. The caller is expected to subsequently call
+ * rsi_attestation_token_continue() to retrieve the attestation token data on
+ * the same CPU.
+ *
+ * Returns:
+ *  On success, returns the upper limit of the attestation report size.
+ *  Otherwise, -EINVAL
+ */
+static inline unsigned long
+rsi_attestation_token_init(const u8 *challenge, unsigned long size)
+{
+	struct arm_smccc_1_2_regs regs = { 0 };
+
+	/* The challenge must be at least 32bytes and at most 64bytes */
+	if (!challenge || size < 32 || size > 64)
+		return -EINVAL;
+
+	regs.a0 = SMC_RSI_ATTESTATION_TOKEN_INIT;
+	memcpy(&regs.a1, challenge, size);
+	arm_smccc_1_2_smc(&regs, &regs);
+
+	if (regs.a0 == RSI_SUCCESS)
+		return regs.a1;
+
+	return -EINVAL;
+}
+
+/**
+ * rsi_attestation_token_continue - Continue the operation to retrieve an
+ * attestation token.
+ *
+ * @granule: {I}PA of the Granule to which the token will be written.
+ * @offset:  Offset within Granule to start of buffer in bytes.
+ * @size:    The size of the buffer.
+ * @len:     The number of bytes written to the buffer.
+ *
+ * Retrieves up to a RSI_GRANULE_SIZE worth of token data per call. The caller
+ * is expected to call rsi_attestation_token_init() before calling this
+ * function to retrieve the attestation token.
+ *
+ * Return:
+ * * %RSI_SUCCESS     - Attestation token retrieved successfully.
+ * * %RSI_INCOMPLETE  - Token generation is not complete.
+ * * %RSI_ERROR_INPUT - A parameter was not valid.
+ * * %RSI_ERROR_STATE - Attestation not in progress.
+ */
+static inline int rsi_attestation_token_continue(phys_addr_t granule,
+						 unsigned long offset,
+						 unsigned long size,
+						 unsigned long *len)
+{
+	struct arm_smccc_res res;
+
+	arm_smccc_1_1_invoke(SMC_RSI_ATTESTATION_TOKEN_CONTINUE,
+			     granule, offset, size, 0, &res);
+
+	if (len)
+		*len = res.a1;
+	return res.a0;
+}
+
 #endif
Signature · 1 line
2.34.1
#44 lore raw

PATCH v3 14/14 virt: arm-cca-guest: TSM_REPORT support for realms

In reply to Steven Price

From: Sami Mujawar <[email protected]>

Introduce an arm-cca-guest driver that registers with the configfs-tsm module to provide user interfaces for retrieving an attestation token.

When a new report is requested the arm-cca-guest driver invokes the appropriate RSI interfaces to query an attestation token.

The steps to retrieve an attestation token are as follows:
1. Mount the configfs filesystem if not already mounted mount -t configfs none /sys/kernel/config
2. Generate an attestation token report=/sys/kernel/config/tsm/report/report0 mkdir $report
dd if=/dev/urandom bs=64 count=1 > $report/inblob hexdump -C $report/outblob
rmdir $report

---
v3: Minor improvements to comments and adapt to the renaming of GRANULE_SIZE to RSI_GRANULE_SIZE.

---
 drivers/virt/coco/Kconfig                     |   2 +
 drivers/virt/coco/Makefile                    |   1 +
 drivers/virt/coco/arm-cca-guest/Kconfig       |  11 +
 drivers/virt/coco/arm-cca-guest/Makefile      |   2 +
 .../virt/coco/arm-cca-guest/arm-cca-guest.c   | 211 ++++++++++++++++++
 5 files changed, 227 insertions(+)
 create mode 100644 drivers/virt/coco/arm-cca-guest/Kconfig
 create mode 100644 drivers/virt/coco/arm-cca-guest/Makefile
 create mode 100644 drivers/virt/coco/arm-cca-guest/arm-cca-guest.c
drivers/virt/coco/Kconfig +2 -0
index 87d142c1f932..4fb69804b622 100644
@@ -12,3 +12,5 @@ source "drivers/virt/coco/efi_secret/Kconfig"
 source "drivers/virt/coco/sev-guest/Kconfig"
 
 source "drivers/virt/coco/tdx-guest/Kconfig"
+
+source "drivers/virt/coco/arm-cca-guest/Kconfig"
drivers/virt/coco/Makefile +1 -0
index 18c1aba5edb7..a6228a1bf992 100644
@@ -6,3 +6,4 @@ obj-$(CONFIG_TSM_REPORTS)	+= tsm.o
 obj-$(CONFIG_EFI_SECRET)	+= efi_secret/
 obj-$(CONFIG_SEV_GUEST)		+= sev-guest/
 obj-$(CONFIG_INTEL_TDX_GUEST)	+= tdx-guest/
+obj-$(CONFIG_ARM_CCA_GUEST)	+= arm-cca-guest/
drivers/virt/coco/arm-cca-guest/Kconfig new file +11 -0
new file mode 100644
index 000000000000..9dd27c3ee215
@@ -0,0 +1,11 @@
+config ARM_CCA_GUEST
+	tristate "Arm CCA Guest driver"
+	depends on ARM64
+	default m
+	select TSM_REPORTS
+	help
+	  The driver provides userspace interface to request and
+	  attestation report from the Realm Management Monitor(RMM).
+
+	  If you choose 'M' here, this module will be called
+	  arm-cca-guest.
drivers/virt/coco/arm-cca-guest/Makefile new file +2 -0
new file mode 100644
index 000000000000..69eeba08e98a
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_ARM_CCA_GUEST) += arm-cca-guest.o
drivers/virt/coco/arm-cca-guest/arm-cca-guest.c new file +211 -0

211 lines not on this page · raw message

Signature · 1 line
2.34.1
Tested-by on the v3 series
#45 lore raw
In reply to Steven Price

Hi Steven,

Quoted 95 lines · Steven Price

This series adds support for running Linux in a protected VM under the Arm Confidential Compute Architecture (CCA). This has been updated following the feedback from the v2 posting[1]. Thanks for the feedback! Individual patches have a change log for v3.

The biggest change from v2 is fixing set_memory_{en,de}crypted() to perform a break-before-make sequence. Note that only the virtual address supplied is flipped between shared and protected, so if e.g. a vmalloc() address is passed the linear map will still point to the (now invalid) previous IPA. Attempts to access the wrong address may trigger a Synchronous External Abort. However any code which attempts to access the 'encrypted' alias after set_memory_decrypted() is already likely to be broken on platforms that implement memory encryption, so I don't expect problems.

The ABI to the RMM from a realm (the RSI) is based on the final RMM v1.0 (EAC 5) specification[2]. Future RMM specifications will be backwards compatible so a guest using the v1.0 specification (i.e. this series) will be able to run on future versions of the RMM without modification.

Arm plans to set up a CI system to perform at a minimum boot testing of Linux as a guest within a realm.

This series is based on v6.10-rc1. It is also available as a git repository:

https://gitlab.arm.com/linux-arm/linux-cca cca-guest/v3

This series (the guest side) should be in a good state so please review with the intention that this could be merged soon. The host side (KVM changes) is likely to require some more iteration and I'll post that as a separate series shortly - note that there is no tie between the series (i.e. you can mix and match v2 and v3 postings of the host and guest).

Introduction (unchanged from v2 posting)
============
A more general introduction to Arm CCA is available on the Arm website[3], and links to the other components involved are available in the overall cover letter.

Arm Confidential Compute Architecture adds two new 'worlds' to the
architecture: Root and Realm. A new software component known as the RMM (Realm Management Monitor) runs in Realm EL2 and is trusted by both the Normal World and VMs running within Realms. This enables mutual distrust between the Realm VMs and the Normal World.

Virtual machines running within a Realm can decide on a (4k) page-by-page granularity whether to share a page with the (Normal World) host or to keep it private (protected). This protection is provided by the hardware and attempts to access a page which isn't shared by the Normal World will trigger a Granule Protection Fault.

Realm VMs can communicate with the RMM via another SMC interface known as RSI (Realm Services Interface). This series adds wrappers for the full set of RSI commands and uses them to manage the Realm IPA State (RIPAS) and to discover the configuration of the realm.

The VM running within the Realm needs to ensure that memory that is going to use is marked as 'RIPAS_RAM' (i.e. protected memory accessible only to the guest). This could be provided by the VMM (and subject to measurement to ensure it is setup correctly) or the VM can set it itself. This series includes a patch which will iterate over all described RAM and set the RIPAS. This is a relatively cheap operation, and doesn't require memory donation from the host. Instead, memory can be dynamically provided by the host on fault. An alternative would be to update booting.rst and state this as a requirement, but this would reduce the flexibility of the VMM to manage the available memory to the guest (as the initial RIPAS state is part of the guest's measurement).

Within the Realm the most-significant active bit of the IPA is used to select whether the access is to protected memory or to memory shared with the host. This series treats this bit as if it is attribute bit in the page tables and will modify it when sharing/unsharing memory with the host.

This top bit usage also necessitates that the IPA width is made more dynamic in the guest. The VMM will choose a width (and therefore which bit controls the shared flag) and the guest must be able to identify this bit to mask it out when necessary. PHYS_MASK_SHIFT/PHYS_MASK are therefore made dynamic.

To allow virtio to communicate with the host the shared buffers must be placed in memory which has this top IPA bit set. This is achieved by implementing the set_memory_{encrypted,decrypted} APIs for arm64 and forcing the use of bounce buffers. For now all device access is considered to required the memory to be shared, at this stage there is no support for real devices to be assigned to a realm guest - obviously if device assignment is added this will have to change.

Finally the GIC is (largely) emulated by the (untrusted) host. The RMM provides some management (including register save/restore) but the ITS buffers must be placed into shared memory for the host to emulate. There is likely to be future work to harden the GIC driver against a malicious host (along with any other drivers used within a Realm guest).

Quoted 4 lines

[1] https://lore.kernel.org/r/20240412084213.1733764-1-steven.price%40arm.com
[2] https://developer.arm.com/documentation/den0137/1-0eac5/
[3] https://www.arm.com/architecture/security-features/arm-confidential-compute-architecture

The v3 guest built with clang booted fine on FVP backed by v2 host kernel.

Thanks,
Itaru.

Quoted 58 lines

Sami Mujawar (2):
arm64: rsi: Interfaces to query attestation token
virt: arm-cca-guest: TSM_REPORT support for realms

Steven Price (5):
arm64: realm: Query IPA size from the RMM
arm64: Mark all I/O as non-secure shared
arm64: Make the PHYS_MASK_SHIFT dynamic
arm64: Enforce bounce buffers for realm DMA
arm64: realm: Support nonsecure ITS emulation shared

Suzuki K Poulose (7):
arm64: rsi: Add RSI definitions
arm64: Detect if in a realm and set RIPAS RAM
fixmap: Allow architecture overriding set_fixmap_io
arm64: Override set_fixmap_io
arm64: Enable memory encrypt for Realms
arm64: Force device mappings to be non-secure shared
efi: arm64: Map Device with Prot Shared

 arch/arm64/Kconfig                            |   3 +
 arch/arm64/include/asm/fixmap.h               |   4 +-
 arch/arm64/include/asm/io.h                   |   6 +-
 arch/arm64/include/asm/mem_encrypt.h          |  17 ++
 arch/arm64/include/asm/pgtable-hwdef.h        |   6 -
 arch/arm64/include/asm/pgtable-prot.h         |   3 +
 arch/arm64/include/asm/pgtable.h              |   7 +-
 arch/arm64/include/asm/rsi.h                  |  48 ++++
 arch/arm64/include/asm/rsi_cmds.h             | 143 ++++++++++++
 arch/arm64/include/asm/rsi_smc.h              | 142 ++++++++++++
 arch/arm64/include/asm/set_memory.h           |   3 +
 arch/arm64/kernel/Makefile                    |   3 +-
 arch/arm64/kernel/efi.c                       |   2 +-
 arch/arm64/kernel/rsi.c                       |  96 ++++++++
 arch/arm64/kernel/setup.c                     |   8 +
 arch/arm64/mm/init.c                          |  10 +-
 arch/arm64/mm/mmu.c                           |  13 ++
 arch/arm64/mm/pageattr.c                      |  65 +++++-
 drivers/irqchip/irq-gic-v3-its.c              |  90 ++++++--
 drivers/virt/coco/Kconfig                     |   2 +
 drivers/virt/coco/Makefile                    |   1 +
 drivers/virt/coco/arm-cca-guest/Kconfig       |  11 +
 drivers/virt/coco/arm-cca-guest/Makefile      |   2 +
 .../virt/coco/arm-cca-guest/arm-cca-guest.c   | 211 ++++++++++++++++++
 include/asm-generic/fixmap.h                  |   2 +
 25 files changed, 858 insertions(+), 40 deletions(-)
 create mode 100644 arch/arm64/include/asm/mem_encrypt.h
 create mode 100644 arch/arm64/include/asm/rsi.h
 create mode 100644 arch/arm64/include/asm/rsi_cmds.h
 create mode 100644 arch/arm64/include/asm/rsi_smc.h
 create mode 100644 arch/arm64/kernel/rsi.c
 create mode 100644 drivers/virt/coco/arm-cca-guest/Kconfig
 create mode 100644 drivers/virt/coco/arm-cca-guest/Makefile
 create mode 100644 drivers/virt/coco/arm-cca-guest/arm-cca-guest.c
Signature · 1 line
2.34.1
#46 lore raw
Quoted 97 lines · Itaru Kitayama

Hi Steven,

Quoted 95 lines · Steven Price

This series adds support for running Linux in a protected VM under the Arm Confidential Compute Architecture (CCA). This has been updated following the feedback from the v2 posting[1]. Thanks for the feedback! Individual patches have a change log for v3.

The biggest change from v2 is fixing set_memory_{en,de}crypted() to perform a break-before-make sequence. Note that only the virtual address supplied is flipped between shared and protected, so if e.g. a vmalloc() address is passed the linear map will still point to the (now invalid) previous IPA. Attempts to access the wrong address may trigger a Synchronous External Abort. However any code which attempts to access the 'encrypted' alias after set_memory_decrypted() is already likely to be broken on platforms that implement memory encryption, so I don't expect problems.

The ABI to the RMM from a realm (the RSI) is based on the final RMM v1.0 (EAC 5) specification[2]. Future RMM specifications will be backwards compatible so a guest using the v1.0 specification (i.e. this series) will be able to run on future versions of the RMM without modification.

Arm plans to set up a CI system to perform at a minimum boot testing of Linux as a guest within a realm.

This series is based on v6.10-rc1. It is also available as a git repository:

https://gitlab.arm.com/linux-arm/linux-cca cca-guest/v3

This series (the guest side) should be in a good state so please review with the intention that this could be merged soon. The host side (KVM changes) is likely to require some more iteration and I'll post that as a separate series shortly - note that there is no tie between the series (i.e. you can mix and match v2 and v3 postings of the host and guest).

Introduction (unchanged from v2 posting)
============
A more general introduction to Arm CCA is available on the Arm website[3], and links to the other components involved are available in the overall cover letter.

Arm Confidential Compute Architecture adds two new 'worlds' to the
architecture: Root and Realm. A new software component known as the RMM (Realm Management Monitor) runs in Realm EL2 and is trusted by both the Normal World and VMs running within Realms. This enables mutual distrust between the Realm VMs and the Normal World.

Virtual machines running within a Realm can decide on a (4k) page-by-page granularity whether to share a page with the (Normal World) host or to keep it private (protected). This protection is provided by the hardware and attempts to access a page which isn't shared by the Normal World will trigger a Granule Protection Fault.

Realm VMs can communicate with the RMM via another SMC interface known as RSI (Realm Services Interface). This series adds wrappers for the full set of RSI commands and uses them to manage the Realm IPA State (RIPAS) and to discover the configuration of the realm.

The VM running within the Realm needs to ensure that memory that is going to use is marked as 'RIPAS_RAM' (i.e. protected memory accessible only to the guest). This could be provided by the VMM (and subject to measurement to ensure it is setup correctly) or the VM can set it itself. This series includes a patch which will iterate over all described RAM and set the RIPAS. This is a relatively cheap operation, and doesn't require memory donation from the host. Instead, memory can be dynamically provided by the host on fault. An alternative would be to update booting.rst and state this as a requirement, but this would reduce the flexibility of the VMM to manage the available memory to the guest (as the initial RIPAS state is part of the guest's measurement).

Within the Realm the most-significant active bit of the IPA is used to select whether the access is to protected memory or to memory shared with the host. This series treats this bit as if it is attribute bit in the page tables and will modify it when sharing/unsharing memory with the host.

This top bit usage also necessitates that the IPA width is made more dynamic in the guest. The VMM will choose a width (and therefore which bit controls the shared flag) and the guest must be able to identify this bit to mask it out when necessary. PHYS_MASK_SHIFT/PHYS_MASK are therefore made dynamic.

To allow virtio to communicate with the host the shared buffers must be placed in memory which has this top IPA bit set. This is achieved by implementing the set_memory_{encrypted,decrypted} APIs for arm64 and forcing the use of bounce buffers. For now all device access is considered to required the memory to be shared, at this stage there is no support for real devices to be assigned to a realm guest - obviously if device assignment is added this will have to change.

Finally the GIC is (largely) emulated by the (untrusted) host. The RMM provides some management (including register save/restore) but the ITS buffers must be placed into shared memory for the host to emulate. There is likely to be future work to harden the GIC driver against a malicious host (along with any other drivers used within a Realm guest).

Quoted 8 lines
Quoted 4 lines

[1] https://lore.kernel.org/r/20240412084213.1733764-1-steven.price%40arm.com
[2] https://developer.arm.com/documentation/den0137/1-0eac5/
[3] https://www.arm.com/architecture/security-features/arm-confidential-compute-architecture

The v3 guest built with clang booted fine on FVP backed by v2 host kernel.

Thanks for testing!

Steve

#47 lore raw
In reply to Steven Price

From: Steven Price <[email protected]> Sent: Wednesday, June 5, 2024 2:30 AM

Quoted 6 lines

This series adds support for running Linux in a protected VM under the Arm Confidential Compute Architecture (CCA). This has been updated following the feedback from the v2 posting[1]. Thanks for the feedback! Individual patches have a change log for v3.

Quoted 9 lines

The biggest change from v2 is fixing set_memory_{en,de}crypted() to perform a break-before-make sequence. Note that only the virtual address supplied is flipped between shared and protected, so if e.g. a vmalloc() address is passed the linear map will still point to the (now invalid) previous IPA. Attempts to access the wrong address may trigger a Synchronous External Abort. However any code which attempts to access the 'encrypted' alias after set_memory_decrypted() is already likely to be broken on platforms that implement memory encryption, so I don't expect problems.

In the case of a vmalloc() address, load_unaligned_zeropad() could still make an access to the underlying pages through the linear address. In CoCo guests on x86, both the vmalloc PTE and the linear map PTE are flipped, so the load_unaligned_zeropad() problem can occur only during the transition between decrypted and encrypted. But even then, the exception handlers have code to fixup this case and allow everything to proceed normally.

I haven't looked at the code in your patches, but do you handle that case, or somehow prevent it?

Thanks,

Michael

maintainer
#48 lore raw
Quoted 24 lines · Michael Kelley

From: Steven Price <[email protected]> Sent: Wednesday, June 5, 2024 2:30 AM

Quoted 5 lines

This series adds support for running Linux in a protected VM under the Arm Confidential Compute Architecture (CCA). This has been updated following the feedback from the v2 posting[1]. Thanks for the feedback! Individual patches have a change log for v3.

Quoted 9 lines

The biggest change from v2 is fixing set_memory_{en,de}crypted() to perform a break-before-make sequence. Note that only the virtual address supplied is flipped between shared and protected, so if e.g. a vmalloc() address is passed the linear map will still point to the (now invalid) previous IPA. Attempts to access the wrong address may trigger a Synchronous External Abort. However any code which attempts to access the 'encrypted' alias after set_memory_decrypted() is already likely to be broken on platforms that implement memory encryption, so I don't expect problems.

In the case of a vmalloc() address, load_unaligned_zeropad() could still make an access to the underlying pages through the linear address. In CoCo guests on x86, both the vmalloc PTE and the linear map PTE are flipped, so the load_unaligned_zeropad() problem can occur only during the transition between decrypted and encrypted. But even then, the exception handlers have code to fixup this case and allow everything to proceed normally.

Quoted 2 lines

I haven't looked at the code in your patches, but do you handle that case, or somehow prevent it?

If we can guarantee that only full a vm_struct area is changed at a time, the vmap guard page would prevent this issue (not sure we can though). Otherwise I think we either change the set_memory_*() code to deal with the other mappings or we handle the exception.

We also have potential user mappings, do we need to do anything about them?

Signature · 1 line
Catalin
#49 lore raw

From: Catalin Marinas <[email protected]> Sent: Friday, June 7, 2024 8:13 AM

Quoted 26 lines · Michael Kelley
Quoted 24 lines

From: Steven Price <[email protected]> Sent: Wednesday, June 5, 2024 2:30 AM

Quoted 5 lines

This series adds support for running Linux in a protected VM under the Arm Confidential Compute Architecture (CCA). This has been updated following the feedback from the v2 posting[1]. Thanks for the feedback! Individual patches have a change log for v3.

Quoted 9 lines

The biggest change from v2 is fixing set_memory_{en,de}crypted() to perform a break-before-make sequence. Note that only the virtual address supplied is flipped between shared and protected, so if e.g. a vmalloc() address is passed the linear map will still point to the (now invalid) previous IPA. Attempts to access the wrong address may trigger a Synchronous External Abort. However any code which attempts to access the 'encrypted' alias after set_memory_decrypted() is already likely to be broken on platforms that implement memory encryption, so I don't expect problems.

In the case of a vmalloc() address, load_unaligned_zeropad() could still make an access to the underlying pages through the linear address. In CoCo guests on x86, both the vmalloc PTE and the linear map PTE are flipped, so the load_unaligned_zeropad() problem can occur only during the transition between decrypted and encrypted. But even then, the exception handlers have code to fixup this case and allow everything to proceed normally.

Quoted 7 lines
Quoted 2 lines

I haven't looked at the code in your patches, but do you handle that case, or somehow prevent it?

If we can guarantee that only full a vm_struct area is changed at a time, the vmap guard page would prevent this issue (not sure we can though). Otherwise I think we either change the set_memory_*() code to deal with the other mappings or we handle the exception.

I don't think the vmap guard pages help. The vmalloc() memory consists of individual pages that are scattered throughout the direct map. The stray reference from load_unaligned_zeropad() will originate in a kmalloc'ed memory page that precedes one of these scattered individual pages, and will use a direct map kernel vaddr. So the guard page in vmalloc space don't come into play. At least in the Hyper-V use case, an entire vmalloc allocation *is* flipped as a unit, so the guard pages do prevent a stray reference from load_unaligned_zeropad() that originates in vmalloc space. At one point I looked to see if load_unaligned_zeropad() is ever used on vmalloc addresses. I think the answer was "no", making the guard page question moot, but I'm not sure. :-(

Another thought: The use of load_unaligned_zeropad() is conditional on CONFIG_DCACHE_WORD_ACCESS. There are #ifdef'ed alternate implementations that don't use load_unaligned_zeropad() if it is not enabled. I looked at just disabling it in CoCo VMs, but I don't know the performance impact. I speculated that the benefits were more noticeable in processors from a decade or more ago, and perhaps less so now, but never did any measurements. There was also a snag in that x86-only code has a usage of load_unaligned_zeropad() without an alternate implementation, so I never went fully down that path. But arm64 would probably "just work" if it were disabled.

Quoted 3 lines

We also have potential user mappings, do we need to do anything about them?

I'm unclear on the scenario here. Would memory with a user mapping ever be flipped between decrypted and encrypted while the user mapping existed? I don't recall being concerned about user mappings, so maybe had ruled out that scenario. On x86, flipping between decrypted and encrypted may effectively change the contents of the memory, so doing a flip while mapped into user space seems problematic. But maybe I'm missing something.

Michael

maintainer
#50 lore raw
Quoted 18 lines · Michael Kelley

From: Catalin Marinas <[email protected]> Sent: Friday, June 7, 2024 8:13 AM

Quoted 9 lines · Michael Kelley
Quoted 8 lines

In the case of a vmalloc() address, load_unaligned_zeropad() could still make an access to the underlying pages through the linear address. In CoCo guests on x86, both the vmalloc PTE and the linear map PTE are flipped, so the load_unaligned_zeropad() problem can occur only during the transition between decrypted and encrypted. But even then, the exception handlers have code to fixup this case and allow everything to proceed normally.

Quoted 7 lines
Quoted 2 lines

I haven't looked at the code in your patches, but do you handle that case, or somehow prevent it?

If we can guarantee that only full a vm_struct area is changed at a time, the vmap guard page would prevent this issue (not sure we can though). Otherwise I think we either change the set_memory_*() code to deal with the other mappings or we handle the exception.

Quoted 11 lines

I don't think the vmap guard pages help. The vmalloc() memory consists of individual pages that are scattered throughout the direct map. The stray reference from load_unaligned_zeropad() will originate in a kmalloc'ed memory page that precedes one of these scattered individual pages, and will use a direct map kernel vaddr. So the guard page in vmalloc space don't come into play. At least in the Hyper-V use case, an entire vmalloc allocation *is* flipped as a unit, so the guard pages do prevent a stray reference from load_unaligned_zeropad() that originates in vmalloc space. At one point I looked to see if load_unaligned_zeropad() is ever used on vmalloc addresses. I think the answer was "no", making the guard page question moot, but I'm not sure. :-(

My point was about load_unaligned_zeropad() originating in the vmalloc space. What I had in mind is changing the underlying linear map via set_memory_*() while we have live vmalloc() mappings. But I forgot about the case you mentioned in a previous thread: set_memory_*() being called on vmalloc()'ed memory directly:

https://lore.kernel.org/r/SN6PR02MB41578D7BFEDE33BD2E8246EFD4E92@SN6PR02MB4157.namprd02.prod.outlook.com/

I wonder whether something like __GFP_DECRYPTED could be used to get shared memory from the allocation time and avoid having to change the vmalloc() ranges. This way functions like netvsc_init_buf() would get decrypted memory from the start and vmbus_establish_gpadl() would not need to call set_memory_decrypted() on a vmalloc() address.

Quoted 10 lines

Another thought: The use of load_unaligned_zeropad() is conditional on CONFIG_DCACHE_WORD_ACCESS. There are #ifdef'ed alternate implementations that don't use load_unaligned_zeropad() if it is not enabled. I looked at just disabling it in CoCo VMs, but I don't know the performance impact. I speculated that the benefits were more noticeable in processors from a decade or more ago, and perhaps less so now, but never did any measurements. There was also a snag in that x86-only code has a usage of load_unaligned_zeropad() without an alternate implementation, so I never went fully down that path. But arm64 would probably "just work" if it were disabled.

We shouldn't penalise the performance, especially as I expect a single image to run both as a guest or a host. However, I think now the linear map is handled correctly since we make the PTE invalid before making the page shared and this would force the fault path through the one that safely handles load_unaligned_zeropad(). Steven's patches also currently reject non-linear-map addresses, I guess this would be a separate addition.

Quoted 6 lines
Quoted 2 lines

We also have potential user mappings, do we need to do anything about them?

I'm unclear on the scenario here. Would memory with a user mapping ever be flipped between decrypted and encrypted while the user mapping existed?

Maybe it doesn't matter. Do we expect it the underlying pages to be flipped while live mappings other than the linear map exist? I assume not, one would first allocate and configure the memory in the kernel before some remap_pfn_range() to user with the appropriate pgprot.

Signature · 1 line
Catalin
#51 lore raw

From: Catalin Marinas <[email protected]> Sent: Monday, June 10, 2024 3:34 AM

Quoted 32 lines · Michael Kelley
Quoted 18 lines

From: Catalin Marinas <[email protected]> Sent: Friday, June 7, 2024 8:13 AM

Quoted 9 lines · Michael Kelley
Quoted 8 lines

In the case of a vmalloc() address, load_unaligned_zeropad() could still make an access to the underlying pages through the linear address. In CoCo guests on x86, both the vmalloc PTE and the linear map PTE are flipped, so the load_unaligned_zeropad() problem can occur only during the transition between decrypted and encrypted. But even then, the exception handlers have code to fixup this case and allow everything to proceed normally.

Quoted 7 lines
Quoted 2 lines

I haven't looked at the code in your patches, but do you handle that case, or somehow prevent it?

If we can guarantee that only full a vm_struct area is changed at a time, the vmap guard page would prevent this issue (not sure we can though). Otherwise I think we either change the set_memory_*() code to deal with the other mappings or we handle the exception.

Quoted 11 lines

I don't think the vmap guard pages help. The vmalloc() memory consists of individual pages that are scattered throughout the direct map. The stray reference from load_unaligned_zeropad() will originate in a kmalloc'ed memory page that precedes one of these scattered individual pages, and will use a direct map kernel vaddr. So the guard page in vmalloc space don't come into play. At least in the Hyper-V use case, an entire vmalloc allocation *is* flipped as a unit, so the guard pages do prevent a stray reference from load_unaligned_zeropad() that originates in vmalloc space. At one point I looked to see if load_unaligned_zeropad() is ever used on vmalloc addresses. I think the answer was "no", making the guard page question moot, but I'm not sure. :-(

Quoted 8 lines

My point was about load_unaligned_zeropad() originating in the vmalloc space. What I had in mind is changing the underlying linear map via set_memory_*() while we have live vmalloc() mappings. But I forgot about the case you mentioned in a previous thread: set_memory_*() being called on vmalloc()'ed memory directly:

https://lore.kernel.org/all/SN6PR02MB41578D7BFEDE33BD2E8246EFD4E92@SN6PR02MB4157.namprd02.prod.outlook.com/

OK, right. You and I were thinking about different cases.

Quoted 5 lines

I wonder whether something like __GFP_DECRYPTED could be used to get shared memory from the allocation time and avoid having to change the vmalloc() ranges. This way functions like netvsc_init_buf() would get decrypted memory from the start and vmbus_establish_gpadl() would not need to call set_memory_decrypted() on a vmalloc() address.

I would not have any conceptual objections to such an approach. But I'm certainly not an expert in that area so I'm not sure what it would take to make that work for vmalloc(). I presume that __GFP_DECRYPTED should also work for kmalloc()?

I've seen the separate discussion about a designated pool of decrypted memory, to avoid always allocating a new page and decrypting when a smaller allocation is sufficient. If such a pool could also work for page size or larger allocations, it would have the additional benefit of concentrating decrypted allocations in fewer 2 Meg large pages vs. scattering wherever and forcing the break-up of more large page mappings in the direct map.

I'll note that netvsc devices can be added or removed from a running VM. The vmalloc() memory allocated by netvsc_init_buf() can be freed, and/or additional calls to netvsc_init_buf() can be made at any time -- they aren't limited to initial Linux boot. So the mechanism for getting decrypted memory at allocation time must be reasonably dynamic.

Quoted 12 lines
Quoted 11 lines

Another thought: The use of load_unaligned_zeropad() is conditional on CONFIG_DCACHE_WORD_ACCESS. There are #ifdef'ed alternate implementations that don't use load_unaligned_zeropad() if it is not enabled. I looked at just disabling it in CoCo VMs, but I don't know the performance impact. I speculated that the benefits were more noticeable in processors from a decade or more ago, and perhaps less so now, but never did any measurements. There was also a snag in that x86-only code has a usage of load_unaligned_zeropad() without an alternate implementation, so I never went fully down that path. But arm64 would probably "just work" if it were disabled.

Quoted 7 lines

We shouldn't penalise the performance, especially as I expect a single image to run both as a guest or a host. However, I think now the linear map is handled correctly since we make the PTE invalid before making the page shared and this would force the fault path through the one that safely handles load_unaligned_zeropad(). Steven's patches also currently reject non-linear-map addresses, I guess this would be a separate addition.

Rejecting vmalloc() addresses may work for the moment -- I don't know when CCA guests might be tried on Hyper-V. The original SEV-SNP and TDX work started that way as well. :-) Handling the vmalloc() case was added later, though I think on x86 the machinery to also flip all the alias PTEs was already mostly or completely in place, probably for other reasons. So fixing the vmalloc() case was more about not assuming that the underlying physical address range is contiguous. Instead, each page must be processed independently, which was straightforward.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

We also have potential user mappings, do we need to do anything about them?

Quoted 8 lines
Quoted 3 lines

I'm unclear on the scenario here. Would memory with a user mapping ever be flipped between decrypted and encrypted while the user mapping existed?

Maybe it doesn't matter. Do we expect it the underlying pages to be flipped while live mappings other than the linear map exist? I assume not, one would first allocate and configure the memory in the kernel before some remap_pfn_range() to user with the appropriate pgprot.

Yes, for user space mappings I also assume not.

Michael

maintainer
#52 lore raw
Quoted 12 lines · Michael Kelley

From: Catalin Marinas <[email protected]> Sent: Monday, June 10, 2024 3:34 AM

Quoted 5 lines

I wonder whether something like __GFP_DECRYPTED could be used to get shared memory from the allocation time and avoid having to change the vmalloc() ranges. This way functions like netvsc_init_buf() would get decrypted memory from the start and vmbus_establish_gpadl() would not need to call set_memory_decrypted() on a vmalloc() address.

I would not have any conceptual objections to such an approach. But I'm certainly not an expert in that area so I'm not sure what it would take to make that work for vmalloc(). I presume that __GFP_DECRYPTED should also work for kmalloc()?

Quoted 6 lines

I've seen the separate discussion about a designated pool of decrypted memory, to avoid always allocating a new page and decrypting when a smaller allocation is sufficient. If such a pool could also work for page size or larger allocations, it would have the additional benefit of concentrating decrypted allocations in fewer 2 Meg large pages vs. scattering wherever and forcing the break-up of more large page mappings in the direct map.

Yeah, my quick, not fully tested hack here:

https://lore.kernel.org/linux-arm-kernel/[email protected]/

It's the underlying page allocator that gives back decrypted pages when the flag is passed, so it should work for alloc_pages() and friends. The kmalloc() changes only ensure that we have separate caches for this memory and they are not merged. It needs some more work on kmem_cache, maybe introducing a SLAB_DECRYPTED flag as well as not to rely on the GFP flag.

For vmalloc(), we'd need a pgprot_decrypted() macro to ensure the decrypted pages are marked with the appropriate attributes (arch specific), otherwise it's fairly easy to wire up if alloc_pages() gives back decrypted memory.

Quoted 5 lines

I'll note that netvsc devices can be added or removed from a running VM. The vmalloc() memory allocated by netvsc_init_buf() can be freed, and/or additional calls to netvsc_init_buf() can be made at any time -- they aren't limited to initial Linux boot. So the mechanism for getting decrypted memory at allocation time must be reasonably dynamic.

I think the above should work. But, of course, we'd have to get this past the mm maintainers, it's likely that I missed something.

Quoted 8 lines

Rejecting vmalloc() addresses may work for the moment -- I don't know when CCA guests might be tried on Hyper-V. The original SEV-SNP and TDX work started that way as well. :-) Handling the vmalloc() case was added later, though I think on x86 the machinery to also flip all the alias PTEs was already mostly or completely in place, probably for other reasons. So fixing the vmalloc() case was more about not assuming that the underlying physical address range is contiguous. Instead, each page must be processed independently, which was straightforward.

There may be a slight performance impact but I guess that's not on a critical path. Walking the page tables and changing the vmalloc ptes should be fine but for each page, we'd have to break the linear map, flush the TLBs, re-create the linear map. Those TLBs may become a bottleneck, especially on hardware with lots of CPUs and the microarchitecture. Note that even with a __GFP_DECRYPTED attribute, we'd still need to go for individual pages in the linear map.

Signature · 1 line
Catalin
#53 lore raw

From: Catalin Marinas <[email protected]> Sent: Monday, June 10, 2024 10:46 AM

Quoted 37 lines · Michael Kelley
Quoted 12 lines

From: Catalin Marinas <[email protected]> Sent: Monday, June 10, 2024 3:34 AM

Quoted 5 lines

I wonder whether something like __GFP_DECRYPTED could be used to get shared memory from the allocation time and avoid having to change the vmalloc() ranges. This way functions like netvsc_init_buf() would get decrypted memory from the start and vmbus_establish_gpadl() would not need to call set_memory_decrypted() on a vmalloc() address.

I would not have any conceptual objections to such an approach. But I'm certainly not an expert in that area so I'm not sure what it would take to make that work for vmalloc(). I presume that __GFP_DECRYPTED should also work for kmalloc()?

Quoted 6 lines

I've seen the separate discussion about a designated pool of decrypted memory, to avoid always allocating a new page and decrypting when a smaller allocation is sufficient. If such a pool could also work for page size or larger allocations, it would have the additional benefit of concentrating decrypted allocations in fewer 2 Meg large pages vs. scattering wherever and forcing the break-up of more large page mappings in the direct map.

Yeah, my quick, not fully tested hack here:

https://lore.kernel.org/linux-arm-kernel/[email protected]/

It's the underlying page allocator that gives back decrypted pages when the flag is passed, so it should work for alloc_pages() and friends. The kmalloc() changes only ensure that we have separate caches for this memory and they are not merged. It needs some more work on kmem_cache, maybe introducing a SLAB_DECRYPTED flag as well as not to rely on the GFP flag.

For vmalloc(), we'd need a pgprot_decrypted() macro to ensure the decrypted pages are marked with the appropriate attributes (arch specific), otherwise it's fairly easy to wire up if alloc_pages() gives back decrypted memory.

Quoted 8 lines
Quoted 5 lines

I'll note that netvsc devices can be added or removed from a running VM. The vmalloc() memory allocated by netvsc_init_buf() can be freed, and/or additional calls to netvsc_init_buf() can be made at any time -- they aren't limited to initial Linux boot. So the mechanism for getting decrypted memory at allocation time must be reasonably dynamic.

I think the above should work. But, of course, we'd have to get this past the mm maintainers, it's likely that I missed something.

Having thought about this a few days, I like the model of telling the memory allocators to decrypt/re-encrypt the memory, instead of the caller having to explicitly do set_memory_decrypted()/encrypted(). I'll add some further comments to the thread with your initial implementation.

Quoted 10 lines
Quoted 9 lines

Rejecting vmalloc() addresses may work for the moment -- I don't know when CCA guests might be tried on Hyper-V. The original SEV-SNP and TDX work started that way as well. :-) Handling the vmalloc() case was added later, though I think on x86 the machinery to also flip all the alias PTEs was already mostly or completely in place, probably for other reasons. So fixing the vmalloc() case was more about not assuming that the underlying physical address range is contiguous. Instead, each page must be processed independently, which was straightforward.

Quoted 7 lines

There may be a slight performance impact but I guess that's not on a critical path. Walking the page tables and changing the vmalloc ptes should be fine but for each page, we'd have to break the linear map, flush the TLBs, re-create the linear map. Those TLBs may become a bottleneck, especially on hardware with lots of CPUs and the microarchitecture. Note that even with a __GFP_DECRYPTED attribute, we'd still need to go for individual pages in the linear map.

Agreed. While synthetic devices can come-and-go anytime, it's pretty rare in the grand scheme of things. I guess we would have to try it on a system with high CPU count, but even if the code needed to "pace itself" to avoid hammering the TLBs too hard, that would be OK.

Michael