Skip to contentSkip to conversation

KVM protected memory extension

RFC In review Kirill A. Shutemov 18 participants 64 messages Oct 20, 2020 – Oct 26, 2020

Subsystems (18)
Memory management x86 KVM PowerPC +14 more subsystems
lore
Share & tools
lore
#0 lore raw

v2 cover KVM protected memory extension

== Background / Problem ==

There are a number of hardware features (MKTME, SEV) which protect guest memory from some unauthorized host access. The patchset proposes a purely software feature that mitigates some of the same host-side read-only attacks.

== What does this set mitigate? ==

- Host kernel ”accidental” access to guest data (think speculation)

- Host kernel induced access to guest data (write(fd, &guest_data_ptr, len))

- Host userspace access to guest data (compromised qemu)

- Guest privilege escalation via compromised QEMU device emulation

== What does this set NOT mitigate? ==

- Full host kernel compromise. Kernel will just map the pages again.

- Hardware attacks

The second RFC revision addresses /most/ of the feedback.

I still didn't found a good solution to reboot and kexec. Unprotect all the memory on such operations defeat the goal of the feature. Clearing up most of the memory before unprotecting what is required for reboot (or kexec) is tedious and error-prone.
Maybe we should just declare them unsupported?

== Series Overview ==

The hardware features protect guest data by encrypting it and then ensuring that only the right guest can decrypt it. This has the side-effect of making the kernel direct map and userspace mapping (QEMU et al) useless. But, this teaches us something very useful:
neither the kernel or userspace mappings are really necessary for normal guest operations.

Instead of using encryption, this series simply unmaps the memory. One advantage compared to allowing access to ciphertext is that it allows bad accesses to be caught instead of simply reading garbage.

Protection from physical attacks needs to be provided by some other means. On Intel platforms, (single-key) Total Memory Encryption (TME) provides mitigation against physical attacks, such as DIMM interposers sniffing memory bus traffic.

The patchset modifies both host and guest kernel. The guest OS must enable the feature via hypercall and mark any memory range that has to be shared with the host: DMA regions, bounce buffers, etc. SEV does this marking via a bit in the guest’s page table while this approach uses a hypercall.

For removing the userspace mapping, use a trick similar to what NUMA balancing does: convert memory that belongs to KVM memory slots to
PROT_NONE: all existing entries converted to PROT_NONE with mprotect() and the newly faulted in pages get PROT_NONE from the updated vm_page_prot. The new VMA flag -- VM_KVM_PROTECTED -- indicates that the pages in the VMA must be treated in a special way in the GUP and fault paths. The flag allows GUP to return the page even though it is mapped with PROT_NONE, but only if the new GUP flag -- FOLL_KVM -- is specified. Any userspace access to the memory would result in SIGBUS. Any GUP access without FOLL_KVM would result in -EFAULT.

Removing userspace mapping of the guest memory from QEMU process can help to address some guest privilege escalation attacks. Consider the case when unprivileged guest user exploits bug in a QEMU device emulation to gain access to data it cannot normally have access within the guest.

Any anonymous page faulted into the VM_KVM_PROTECTED VMA gets removed from the direct mapping with kernel_map_pages(). Note that kernel_map_pages() only flushes local TLB. I think it's a reasonable compromise between security and perfromance.

Zapping the PTE would bring the page back to the direct mapping after clearing. At least for now, we don't remove file-backed pages from the direct mapping. File-backed pages could be accessed via read/write syscalls. It adds complexity.

Occasionally, host kernel has to access guest memory that was not made shared by the guest. For instance, it happens for instruction emulation. Normally, it's done via copy_to/from_user() which would fail with -EFAULT now. We introduced a new pair of helpers: copy_to/from_guest(). The new helpers acquire the page via GUP, map it into kernel address space with kmap_atomic()-style mechanism and only then copy the data.

For some instruction emulation copying is not good enough: cmpxchg emulation has to have direct access to the guest memory. __kvm_map_gfn() is modified to accommodate the case.

The patchset is on top of v5.9

Kirill A. Shutemov (16):
x86/mm: Move force_dma_unencrypted() to common code x86/kvm: Introduce KVM memory protection feature
x86/kvm: Make DMA pages shared
x86/kvm: Use bounce buffers for KVM memory protection x86/kvm: Make VirtIO use DMA API in KVM guest x86/kvmclock: Share hvclock memory with the host x86/realmode: Share trampoline area if KVM memory protection enabled
KVM: Use GUP instead of copy_from/to_user() to access guest memory
KVM: mm: Introduce VM_KVM_PROTECTED
KVM: x86: Use GUP for page walk instead of __get_user()
KVM: Protected memory extension
KVM: x86: Enabled protected memory extension
KVM: Rework copy_to/from_guest() to avoid direct mapping
KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()
KVM: Unmap protected pages from direct mapping
mm: Do not use zero page for VM_KVM_PROTECTED VMAs

 arch/powerpc/kvm/book3s_64_mmu_hv.c    |   2 +-
 arch/powerpc/kvm/book3s_64_mmu_radix.c |   2 +-
 arch/s390/include/asm/pgtable.h        |   2 +-
 arch/x86/Kconfig                       |  11 +-
 arch/x86/include/asm/cpufeatures.h     |   1 +
 arch/x86/include/asm/io.h              |   6 +-
 arch/x86/include/asm/kvm_para.h        |   5 +
 arch/x86/include/asm/pgtable_types.h   |   1 +
 arch/x86/include/uapi/asm/kvm_para.h   |   3 +-
 arch/x86/kernel/kvm.c                  |  20 +++
 arch/x86/kernel/kvmclock.c             |   2 +-
 arch/x86/kernel/pci-swiotlb.c          |   3 +-
 arch/x86/kvm/Kconfig                   |   1 +
 arch/x86/kvm/cpuid.c                   |   3 +-
 arch/x86/kvm/mmu/mmu.c                 |   6 +-
 arch/x86/kvm/mmu/paging_tmpl.h         |  10 +-
 arch/x86/kvm/x86.c                     |   9 +
 arch/x86/mm/Makefile                   |   2 +
 arch/x86/mm/ioremap.c                  |  16 +-
 arch/x86/mm/mem_encrypt.c              |  51 ------
 arch/x86/mm/mem_encrypt_common.c       |  62 +++++++
 arch/x86/mm/pat/set_memory.c           |   7 +
 arch/x86/realmode/init.c               |   7 +-
 drivers/virtio/virtio_ring.c           |   4 +
 include/linux/kvm_host.h               |  11 +-
 include/linux/kvm_types.h              |   1 +
 include/linux/mm.h                     |  21 ++-
 include/uapi/linux/kvm_para.h          |   5 +-
 mm/gup.c                               |  20 ++-
 mm/huge_memory.c                       |  31 +++-
 mm/ksm.c                               |   2 +
 mm/memory.c                            |  18 +-
 mm/mmap.c                              |   3 +
 mm/rmap.c                              |   4 +
 virt/kvm/Kconfig                       |   3 +
 virt/kvm/async_pf.c                    |   2 +-
 virt/kvm/kvm_main.c                    | 238 +++++++++++++++++++++----
 virt/lib/Makefile                      |   1 +
 virt/lib/mem_protected.c               | 193 ++++++++++++++++++++
 39 files changed, 666 insertions(+), 123 deletions(-)
 create mode 100644 arch/x86/mm/mem_encrypt_common.c
 create mode 100644 virt/lib/mem_protected.c
Signature · 1 line
2.26.2
#1 lore raw

RFC v2 1/16 x86/mm: Move force_dma_unencrypted() to common code

force_dma_unencrypted() has to return true for KVM guest with the memory protected enabled. Move it out of AMD SME code.

Introduce new config option X86_MEM_ENCRYPT_COMMON that has to be selected by all x86 memory encryption features.

This is preparation for the following patches.

---
 arch/x86/Kconfig                 |  8 +++++--
 arch/x86/include/asm/io.h        |  4 +++-
 arch/x86/mm/Makefile             |  2 ++
 arch/x86/mm/mem_encrypt.c        | 30 -------------------------
 arch/x86/mm/mem_encrypt_common.c | 38 ++++++++++++++++++++++++++++++++
 5 files changed, 49 insertions(+), 33 deletions(-)
 create mode 100644 arch/x86/mm/mem_encrypt_common.c
arch/x86/Kconfig +6 -2
index 7101ac64bb20..619ebf40e457 100644
@@ -1514,13 +1514,17 @@ config X86_CPA_STATISTICS
 	  helps to determine the effectiveness of preserving large and huge
 	  page mappings when mapping protections are changed.
 
+config X86_MEM_ENCRYPT_COMMON
+	select ARCH_HAS_FORCE_DMA_UNENCRYPTED
+	select DYNAMIC_PHYSICAL_MASK
+	def_bool n
+
 config AMD_MEM_ENCRYPT
 	bool "AMD Secure Memory Encryption (SME) support"
 	depends on X86_64 && CPU_SUP_AMD
 	select DMA_COHERENT_POOL
-	select DYNAMIC_PHYSICAL_MASK
 	select ARCH_USE_MEMREMAP_PROT
-	select ARCH_HAS_FORCE_DMA_UNENCRYPTED
+	select X86_MEM_ENCRYPT_COMMON
 	help
 	  Say yes to enable support for the encryption of system memory.
 	  This requires an AMD processor that supports Secure Memory
arch/x86/include/asm/io.h +3 -1
index e1aa17a468a8..c58d52fd7bf2 100644
@@ -256,10 +256,12 @@ static inline void slow_down_io(void)
 
 #endif
 
-#ifdef CONFIG_AMD_MEM_ENCRYPT
 #include <linux/jump_label.h>
 
 extern struct static_key_false sev_enable_key;
+
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+
 static inline bool sev_key_active(void)
 {
 	return static_branch_unlikely(&sev_enable_key);
arch/x86/mm/Makefile +2 -0
index 5864219221ca..b31cb52bf1bd 100644
@@ -52,6 +52,8 @@ obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS)	+= pkeys.o
 obj-$(CONFIG_RANDOMIZE_MEMORY)			+= kaslr.o
 obj-$(CONFIG_PAGE_TABLE_ISOLATION)		+= pti.o
 
+obj-$(CONFIG_X86_MEM_ENCRYPT_COMMON)	+= mem_encrypt_common.o
+
 obj-$(CONFIG_AMD_MEM_ENCRYPT)	+= mem_encrypt.o
 obj-$(CONFIG_AMD_MEM_ENCRYPT)	+= mem_encrypt_identity.o
 obj-$(CONFIG_AMD_MEM_ENCRYPT)	+= mem_encrypt_boot.o
arch/x86/mm/mem_encrypt.c +0 -30
index 9f1177edc2e7..4dbdc9dac36b 100644
@@ -15,10 +15,6 @@
 #include <linux/dma-direct.h>
 #include <linux/swiotlb.h>
 #include <linux/mem_encrypt.h>
-#include <linux/device.h>
-#include <linux/kernel.h>
-#include <linux/bitops.h>
-#include <linux/dma-mapping.h>
 
 #include <asm/tlbflush.h>
 #include <asm/fixmap.h>
@@ -350,32 +346,6 @@ bool sev_active(void)
 	return sme_me_mask && sev_enabled;
 }
 
-/* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */
-bool force_dma_unencrypted(struct device *dev)
-{
-	/*
-	 * For SEV, all DMA must be to unencrypted addresses.
-	 */
-	if (sev_active())
-		return true;
-
-	/*
-	 * For SME, all DMA must be to unencrypted addresses if the
-	 * device does not support DMA to addresses that include the
-	 * encryption mask.
-	 */
-	if (sme_active()) {
-		u64 dma_enc_mask = DMA_BIT_MASK(__ffs64(sme_me_mask));
-		u64 dma_dev_mask = min_not_zero(dev->coherent_dma_mask,
-						dev->bus_dma_limit);
-
-		if (dma_dev_mask <= dma_enc_mask)
-			return true;
-	}
-
-	return false;
-}
-
 void __init mem_encrypt_free_decrypted_mem(void)
 {
 	unsigned long vaddr, vaddr_end, npages;
arch/x86/mm/mem_encrypt_common.c new file +38 -0
new file mode 100644
index 000000000000..964e04152417
@@ -0,0 +1,38 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * AMD Memory Encryption Support
+ *
+ * Copyright (C) 2016 Advanced Micro Devices, Inc.
+ *
+ * Author: Tom Lendacky <[email protected]>
+ */
+
+#include <linux/mm.h>
+#include <linux/mem_encrypt.h>
+#include <linux/dma-mapping.h>
+
+/* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */
+bool force_dma_unencrypted(struct device *dev)
+{
+	/*
+	 * For SEV, all DMA must be to unencrypted/shared addresses.
+	 */
+	if (sev_active())
+		return true;
+
+	/*
+	 * For SME, all DMA must be to unencrypted addresses if the
+	 * device does not support DMA to addresses that include the
+	 * encryption mask.
+	 */
+	if (sme_active()) {
+		u64 dma_enc_mask = DMA_BIT_MASK(__ffs64(sme_me_mask));
+		u64 dma_dev_mask = min_not_zero(dev->coherent_dma_mask,
+						dev->bus_dma_limit);
+
+		if (dma_dev_mask <= dma_enc_mask)
+			return true;
+	}
+
+	return false;
+}
Signature · 1 line
2.26.2
#2 lore raw

RFC v2 2/16 x86/kvm: Introduce KVM memory protection feature

In reply to Kirill A. Shutemov

Provide basic helpers, KVM_FEATURE, CPUID flag and a hypercall.

Host side doesn't provide the feature yet, so it is a dead code for now.

---
 arch/x86/include/asm/cpufeatures.h   |  1 +
 arch/x86/include/asm/kvm_para.h      |  5 +++++
 arch/x86/include/uapi/asm/kvm_para.h |  3 ++-
 arch/x86/kernel/kvm.c                | 18 ++++++++++++++++++
 include/uapi/linux/kvm_para.h        |  3 ++-
 5 files changed, 28 insertions(+), 2 deletions(-)
arch/x86/include/asm/cpufeatures.h +1 -0
index 2901d5df4366..a72157137764 100644
@@ -236,6 +236,7 @@
 #define X86_FEATURE_EPT_AD		( 8*32+17) /* Intel Extended Page Table access-dirty bit */
 #define X86_FEATURE_VMCALL		( 8*32+18) /* "" Hypervisor supports the VMCALL instruction */
 #define X86_FEATURE_VMW_VMMCALL		( 8*32+19) /* "" VMware prefers VMMCALL hypercall instruction */
+#define X86_FEATURE_KVM_MEM_PROTECTED	( 8*32+20) /* KVM memory protection extenstion */
 
 /* Intel-defined CPU features, CPUID level 0x00000007:0 (EBX), word 9 */
 #define X86_FEATURE_FSGSBASE		( 9*32+ 0) /* RDFSBASE, WRFSBASE, RDGSBASE, WRGSBASE instructions*/
arch/x86/include/asm/kvm_para.h +5 -0
index 338119852512..74aea18f3130 100644
@@ -11,11 +11,16 @@ extern void kvmclock_init(void);
 
 #ifdef CONFIG_KVM_GUEST
 bool kvm_check_and_clear_guest_paused(void);
+bool kvm_mem_protected(void);
 #else
 static inline bool kvm_check_and_clear_guest_paused(void)
 {
 	return false;
 }
+static inline bool kvm_mem_protected(void)
+{
+	return false;
+}
 #endif /* CONFIG_KVM_GUEST */
 
 #define KVM_HYPERCALL \
arch/x86/include/uapi/asm/kvm_para.h +2 -1
index 812e9b4c1114..defbfc630a9f 100644
@@ -28,10 +28,11 @@
 #define KVM_FEATURE_PV_UNHALT		7
 #define KVM_FEATURE_PV_TLB_FLUSH	9
 #define KVM_FEATURE_ASYNC_PF_VMEXIT	10
-#define KVM_FEATURE_PV_SEND_IPI	11
+#define KVM_FEATURE_PV_SEND_IPI		11
 #define KVM_FEATURE_POLL_CONTROL	12
 #define KVM_FEATURE_PV_SCHED_YIELD	13
 #define KVM_FEATURE_ASYNC_PF_INT	14
+#define KVM_FEATURE_MEM_PROTECTED	15
 
 #define KVM_HINTS_REALTIME      0
 
arch/x86/kernel/kvm.c +18 -0
index 9663ba31347c..2c1f8952b92a 100644
@@ -37,6 +37,13 @@
 #include <asm/tlb.h>
 #include <asm/cpuidle_haltpoll.h>
 
+static bool mem_protected;
+
+bool kvm_mem_protected(void)
+{
+	return mem_protected;
+}
+
 DEFINE_STATIC_KEY_FALSE(kvm_async_pf_enabled);
 
 static int kvmapf = 1;
@@ -742,6 +749,17 @@ static void __init kvm_init_platform(void)
 {
 	kvmclock_init();
 	x86_platform.apic_post_init = kvm_apic_init;
+
+	if (kvm_para_has_feature(KVM_FEATURE_MEM_PROTECTED)) {
+		if (kvm_hypercall0(KVM_HC_ENABLE_MEM_PROTECTED)) {
+			pr_err("Failed to enable KVM memory protection\n");
+			return;
+		}
+
+		pr_info("KVM memory protection enabled\n");
+		mem_protected = true;
+		setup_force_cpu_cap(X86_FEATURE_KVM_MEM_PROTECTED);
+	}
 }
 
 const __initconst struct hypervisor_x86 x86_hyper_kvm = {
include/uapi/linux/kvm_para.h +2 -1
index 8b86609849b9..1a216f32e572 100644
@@ -27,8 +27,9 @@
 #define KVM_HC_MIPS_EXIT_VM		7
 #define KVM_HC_MIPS_CONSOLE_OUTPUT	8
 #define KVM_HC_CLOCK_PAIRING		9
-#define KVM_HC_SEND_IPI		10
+#define KVM_HC_SEND_IPI			10
 #define KVM_HC_SCHED_YIELD		11
+#define KVM_HC_ENABLE_MEM_PROTECTED	12
 
 /*
  * hypercalls use architecture specific
Signature · 1 line
2.26.2
#3 lore raw

RFC v2 3/16 x86/kvm: Make DMA pages shared

In reply to Kirill A. Shutemov

Make force_dma_unencrypted() return true for KVM to get DMA pages mapped as shared.

__set_memory_enc_dec() now informs the host via hypercall if the state of the page has changed from shared to private or back.

---
 arch/x86/Kconfig                 | 1 +
 arch/x86/mm/mem_encrypt_common.c | 5 +++--
 arch/x86/mm/pat/set_memory.c     | 7 +++++++
 include/uapi/linux/kvm_para.h    | 2 ++
 4 files changed, 13 insertions(+), 2 deletions(-)
arch/x86/Kconfig +1 -0
index 619ebf40e457..cd272e3babbc 100644
@@ -805,6 +805,7 @@ config KVM_GUEST
 	select PARAVIRT_CLOCK
 	select ARCH_CPUIDLE_HALTPOLL
 	select X86_HV_CALLBACK_VECTOR
+	select X86_MEM_ENCRYPT_COMMON
 	default y
 	help
 	  This option enables various optimizations for running under the KVM
arch/x86/mm/mem_encrypt_common.c +3 -2
index 964e04152417..a878e7f246d5 100644
@@ -10,14 +10,15 @@
 #include <linux/mm.h>
 #include <linux/mem_encrypt.h>
 #include <linux/dma-mapping.h>
+#include <asm/kvm_para.h>
 
 /* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */
 bool force_dma_unencrypted(struct device *dev)
 {
 	/*
-	 * For SEV, all DMA must be to unencrypted/shared addresses.
+	 * For SEV and KVM, all DMA must be to unencrypted/shared addresses.
 	 */
-	if (sev_active())
+	if (sev_active() || kvm_mem_protected())
 		return true;
 
 	/*
arch/x86/mm/pat/set_memory.c +7 -0
index d1b2a889f035..4c49303126c9 100644
@@ -16,6 +16,7 @@
 #include <linux/pci.h>
 #include <linux/vmalloc.h>
 #include <linux/libnvdimm.h>
+#include <linux/kvm_para.h>
 
 #include <asm/e820/api.h>
 #include <asm/processor.h>
@@ -1977,6 +1978,12 @@ static int __set_memory_enc_dec(unsigned long addr, int numpages, bool enc)
 	struct cpa_data cpa;
 	int ret;
 
+	if (kvm_mem_protected()) {
+		unsigned long gfn = __pa(addr) >> PAGE_SHIFT;
+		int call = enc ? KVM_HC_MEM_UNSHARE : KVM_HC_MEM_SHARE;
+		return kvm_hypercall2(call, gfn, numpages);
+	}
+
 	/* Nothing to do if memory encryption is not active */
 	if (!mem_encrypt_active())
 		return 0;
include/uapi/linux/kvm_para.h +2 -0
index 1a216f32e572..c6d8c988e330 100644
@@ -30,6 +30,8 @@
 #define KVM_HC_SEND_IPI			10
 #define KVM_HC_SCHED_YIELD		11
 #define KVM_HC_ENABLE_MEM_PROTECTED	12
+#define KVM_HC_MEM_SHARE		13
+#define KVM_HC_MEM_UNSHARE		14
 
 /*
  * hypercalls use architecture specific
Signature · 1 line
2.26.2
#4 lore raw

RFC v2 4/16 x86/kvm: Use bounce buffers for KVM memory protection

In reply to Kirill A. Shutemov

Mirror SEV, use SWIOTLB always if KVM memory protection is enabled.

---
 arch/x86/Kconfig                 |  1 +
 arch/x86/kernel/kvm.c            |  2 ++
 arch/x86/kernel/pci-swiotlb.c    |  3 ++-
 arch/x86/mm/mem_encrypt.c        | 21 ---------------------
 arch/x86/mm/mem_encrypt_common.c | 23 +++++++++++++++++++++++
 5 files changed, 28 insertions(+), 22 deletions(-)
arch/x86/Kconfig +1 -0
index cd272e3babbc..b22b95517437 100644
@@ -806,6 +806,7 @@ config KVM_GUEST
 	select ARCH_CPUIDLE_HALTPOLL
 	select X86_HV_CALLBACK_VECTOR
 	select X86_MEM_ENCRYPT_COMMON
+	select SWIOTLB
 	default y
 	help
 	  This option enables various optimizations for running under the KVM
arch/x86/kernel/kvm.c +2 -0
index 2c1f8952b92a..30bb3d2d6ccd 100644
@@ -26,6 +26,7 @@
 #include <linux/kprobes.h>
 #include <linux/nmi.h>
 #include <linux/swait.h>
+#include <linux/swiotlb.h>
 #include <asm/timer.h>
 #include <asm/cpu.h>
 #include <asm/traps.h>
@@ -759,6 +760,7 @@ static void __init kvm_init_platform(void)
 		pr_info("KVM memory protection enabled\n");
 		mem_protected = true;
 		setup_force_cpu_cap(X86_FEATURE_KVM_MEM_PROTECTED);
+		swiotlb_force = SWIOTLB_FORCE;
 	}
 }
 
arch/x86/kernel/pci-swiotlb.c +2 -1
index c2cfa5e7c152..814060a6ceb0 100644
@@ -13,6 +13,7 @@
 #include <asm/dma.h>
 #include <asm/xen/swiotlb-xen.h>
 #include <asm/iommu_table.h>
+#include <asm/kvm_para.h>
 
 int swiotlb __read_mostly;
 
@@ -49,7 +50,7 @@ int __init pci_swiotlb_detect_4gb(void)
 	 * buffers are allocated and used for devices that do not support
 	 * the addressing range required for the encryption mask.
 	 */
-	if (sme_active())
+	if (sme_active() || kvm_mem_protected())
 		swiotlb = 1;
 
 	return swiotlb;
arch/x86/mm/mem_encrypt.c +0 -21
index 4dbdc9dac36b..5de64e068b0a 100644
@@ -369,24 +369,3 @@ void __init mem_encrypt_free_decrypted_mem(void)
 
 	free_init_pages("unused decrypted", vaddr, vaddr_end);
 }
-
-/* Architecture __weak replacement functions */
-void __init mem_encrypt_init(void)
-{
-	if (!sme_me_mask)
-		return;
-
-	/* Call into SWIOTLB to update the SWIOTLB DMA buffers */
-	swiotlb_update_mem_attributes();
-
-	/*
-	 * With SEV, we need to unroll the rep string I/O instructions.
-	 */
-	if (sev_active())
-		static_branch_enable(&sev_enable_key);
-
-	pr_info("AMD %s active\n",
-		sev_active() ? "Secure Encrypted Virtualization (SEV)"
-			     : "Secure Memory Encryption (SME)");
-}
-
arch/x86/mm/mem_encrypt_common.c +23 -0
index a878e7f246d5..7900f3788010 100644
@@ -37,3 +37,26 @@ bool force_dma_unencrypted(struct device *dev)
 
 	return false;
 }
+
+void __init mem_encrypt_init(void)
+{
+	if (!sme_me_mask && !kvm_mem_protected())
+		return;
+
+	/* Call into SWIOTLB to update the SWIOTLB DMA buffers */
+	swiotlb_update_mem_attributes();
+
+	/*
+	 * With SEV, we need to unroll the rep string I/O instructions.
+	 */
+	if (sev_active())
+		static_branch_enable(&sev_enable_key);
+
+	if (sme_me_mask) {
+		pr_info("AMD %s active\n",
+			sev_active() ? "Secure Encrypted Virtualization (SEV)"
+			: "Secure Memory Encryption (SME)");
+	} else {
+		pr_info("KVM memory protection enabled\n");
+	}
+}
Signature · 1 line
2.26.2
#5 lore raw

Re: [RFCv2 04/16] x86/kvm: Use bounce buffers for KVM memory protection

[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]

Hi "Kirill,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on tip/x86/core]
[also build test WARNING on kvm/linux-next hnaz-linux-mm/master v5.9] [cannot apply to next-20201016]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 238c91115cd05c71447ea071624a4c9fe661f970
config: x86_64-alldefconfig (attached as .config)
compiler: gcc-9 (Debian 9.3.0-15) 9.3.0
reproduce (this is a W=1 build):
        # https://github.com/0day-ci/linux/commit/0bdb6c3e82c3106d720106932629f0e3a121fb76
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130
        git checkout 0bdb6c3e82c3106d720106932629f0e3a121fb76
        # save the attached .config to linux build tree
        make W=1 ARCH=x86_64 

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>):

arch/x86/mm/mem_encrypt_common.c:16:6: warning: no previous prototype for 'force_dma_unencrypted' [-Wmissing-prototypes]
16 | bool force_dma_unencrypted(struct device *dev)
| ^~~~~~~~~~~~~~~~~~~~~

>> arch/x86/mm/mem_encrypt_common.c:41:13: warning: no previous prototype for 'mem_encrypt_init' [-Wmissing-prototypes]

41 | void __init mem_encrypt_init(void)
| ^~~~~~~~~~~~~~~~

vim +/mem_encrypt_init +41 arch/x86/mm/mem_encrypt_common.c

14
15 /* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */

  > 16	bool force_dma_unencrypted(struct device *dev)
    17	{
    18		/*
    19		 * For SEV and KVM, all DMA must be to unencrypted/shared addresses.
    20		 */
    21		if (sev_active() || kvm_mem_protected())
    22			return true;
    23	
    24		/*
    25		 * For SME, all DMA must be to unencrypted addresses if the
    26		 * device does not support DMA to addresses that include the
    27		 * encryption mask.
    28		 */
    29		if (sme_active()) {
    30			u64 dma_enc_mask = DMA_BIT_MASK(__ffs64(sme_me_mask));
    31			u64 dma_dev_mask = min_not_zero(dev->coherent_dma_mask,
    32							dev->bus_dma_limit);
    33	
    34			if (dma_dev_mask <= dma_enc_mask)
    35				return true;
    36		}
    37	
    38		return false;
    39	}
    40	
  > 41	void __init mem_encrypt_init(void)

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#6 lore raw

Re: [RFCv2 04/16] x86/kvm: Use bounce buffers for KVM memory protection

In reply to Kirill A. Shutemov
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]

Hi "Kirill,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on tip/x86/core]
[also build test WARNING on kvm/linux-next hnaz-linux-mm/master v5.9] [cannot apply to next-20201016]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 238c91115cd05c71447ea071624a4c9fe661f970
config: x86_64-randconfig-a001-20201020 (attached as .config)
compiler: clang version 12.0.0 (https://github.com/llvm/llvm-project ea693a162786d933863ab079648d4261ac0ead47)

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install x86_64 cross compiling tool for clang build
# apt-get install binutils-x86-64-linux-gnu
# https://github.com/0day-ci/linux/commit/0bdb6c3e82c3106d720106932629f0e3a121fb76
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130 git checkout 0bdb6c3e82c3106d720106932629f0e3a121fb76
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=x86_64

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>):

arch/x86/mm/mem_encrypt_common.c:16:6: warning: no previous prototype for function 'force_dma_unencrypted' [-Wmissing-prototypes] bool force_dma_unencrypted(struct device *dev)

        ^
   arch/x86/mm/mem_encrypt_common.c:16:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
   bool force_dma_unencrypted(struct device *dev)
   ^
   static 
>> arch/x86/mm/mem_encrypt_common.c:41:13: warning: no previous prototype for function 'mem_encrypt_init' [-Wmissing-prototypes]
   void __init mem_encrypt_init(void)
               ^

arch/x86/mm/mem_encrypt_common.c:41:1: note: declare 'static' if the function is not intended to be used outside of this translation unit void __init mem_encrypt_init(void)
^
static
2 warnings generated.

vim +/mem_encrypt_init +41 arch/x86/mm/mem_encrypt_common.c

    40	
  > 41	void __init mem_encrypt_init(void)

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#7 lore raw

RFC v2 5/16 x86/kvm: Make VirtIO use DMA API in KVM guest

In reply to Kirill A. Shutemov

VirtIO for KVM is a primary way to provide IO. All memory that used for communication with the host has to be marked as shared.

The easiest way to archive that is to use DMA API that already knows how to deal with shared memory.

---
 drivers/virtio/virtio_ring.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/virtio/virtio_ring.c +4 -0
index becc77697960..ace733845d5d 100644
@@ -12,6 +12,7 @@
 #include <linux/hrtimer.h>
 #include <linux/dma-mapping.h>
 #include <xen/xen.h>
+#include <asm/kvm_para.h>
 
 #ifdef DEBUG
 /* For development, we want to crash whenever the ring is screwed. */
@@ -255,6 +256,9 @@ static bool vring_use_dma_api(struct virtio_device *vdev)
 	if (xen_domain())
 		return true;
 
+	if (kvm_mem_protected())
+		return true;
+
 	return false;
 }
 
Signature · 1 line
2.26.2
#8 lore raw

Re: [RFCv2 05/16] x86/kvm: Make VirtIO use DMA API in KVM guest

NAK. Any virtio implementation that needs special DMA OPS treatment needs to set the VIRTIO_F_ACCESS_PLATFORM bit. The only reason the Xen hack existst is because it slipped in a long time ago and we can't fix that any more.

#9 lore raw

Re: [RFCv2 05/16] x86/kvm: Make VirtIO use DMA API in KVM guest

Quoted 4 lines · Christoph Hellwig

NAK. Any virtio implementation that needs special DMA OPS treatment needs to set the VIRTIO_F_ACCESS_PLATFORM bit. The only reason the Xen hack existst is because it slipped in a long time ago and we can't fix that any more.

Thanks. Will fix.

Signature · 1 line
 Kirill A. Shutemov
#10 lore raw

Re: [RFCv2 05/16] x86/kvm: Make VirtIO use DMA API in KVM guest

In reply to Kirill A. Shutemov
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]

Hi "Kirill,

[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/x86/core]
[also build test ERROR on kvm/linux-next hnaz-linux-mm/master v5.9] [cannot apply to next-20201016]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 238c91115cd05c71447ea071624a4c9fe661f970
config: xtensa-allyesconfig (attached as .config)
compiler: xtensa-linux-gcc (GCC) 9.3.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/e3df08a4af6c321d5942dcee585916ccf5e59ef6
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130 git checkout e3df08a4af6c321d5942dcee585916ccf5e59ef6
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=xtensa

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>):

drivers/virtio/virtio_ring.c: In function 'vring_use_dma_api':

>> drivers/virtio/virtio_ring.c:259:6: error: implicit declaration of function 'kvm_mem_protected' [-Werror=implicit-function-declaration]
     259 |  if (kvm_mem_protected())
         |      ^~~~~~~~~~~~~~~~~

cc1: some warnings being treated as errors

vim +/kvm_mem_protected +259 drivers/virtio/virtio_ring.c

215
216 /*
217 * Modern virtio devices have feature bits to specify whether they need a
218 * quirk and bypass the IOMMU. If not there, just use the DMA API.
219 *
220 * If there, the interaction between virtio and DMA API is messy.
221 *
222 * On most systems with virtio, physical addresses match bus addresses,
223 * and it doesn't particularly matter whether we use the DMA API.
224 *
225 * On some systems, including Xen and any system with a physical device
226 * that speaks virtio behind a physical IOMMU, we must use the DMA API
227 * for virtio DMA to work at all.
228 *
229 * On other systems, including SPARC and PPC64, virtio-pci devices are
230 * enumerated as though they are behind an IOMMU, but the virtio host
231 * ignores the IOMMU, so we must either pretend that the IOMMU isn't
232 * there or somehow map everything as the identity.
233 *
234 * For the time being, we preserve historic behavior and bypass the DMA
235 * API.
236 *
237 * TODO: install a per-device DMA ops structure that does the right thing
238 * taking into account all the above quirks, and use the DMA API
239 * unconditionally on data path.
240 */
241
242 static bool vring_use_dma_api(struct virtio_device *vdev)
243 {
244 if (!virtio_has_dma_quirk(vdev))
245 return true;
246
247 /* Otherwise, we are left to guess. */
248 /*
249 * In theory, it's possible to have a buggy QEMU-supposed
250 * emulated Q35 IOMMU and Xen enabled at the same time. On
251 * such a configuration, virtio has never worked and will
252 * not work without an even larger kludge. Instead, enable
253 * the DMA API if we're a Xen guest, which at least allows
254 * all of the sensible Xen configurations to work correctly.
255 */
256 if (xen_domain())
257 return true;
258

 > 259		if (kvm_mem_protected())

260 return true;
261
262 return false;
263 }
264

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#11 lore raw

Re: [RFCv2 05/16] x86/kvm: Make VirtIO use DMA API in KVM guest

In reply to Kirill A. Shutemov
Quoted 24 lines · On Tue, 20 Oct 2020 09:18:48 +0300 "Kirill A. Shutemov" <[email protected]>…

VirtIO for KVM is a primary way to provide IO. All memory that used for communication with the host has to be marked as shared.

The easiest way to archive that is to use DMA API that already knows how to deal with shared memory.

---
 drivers/virtio/virtio_ring.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/virtio/virtio_ring.c +1 -0
index becc77697960..ace733845d5d 100644
@@ -12,6 +12,7 @@
 #include <linux/hrtimer.h>
 #include <linux/dma-mapping.h>
 #include <xen/xen.h>
+#include <asm/kvm_para.h>
 
 #ifdef DEBUG
 /* For development, we want to crash whenever the ring is screwed. */

drivers/virtio/virtio_ring.c static bool vring_use_dma_api(struct virtio_device *vdev) · line 261

Quoted 6 lines
drivers/virtio/virtio_ring.c +3 -0
@@ -255,6 +256,9 @@ static bool vring_use_dma_api(struct virtio_device *vdev)
 	if (xen_domain())
 		return true;
 
+	if (kvm_mem_protected())
+		return true;
+

I guess it does not matter because Christophs comment, but this breaks the build for s390, because there is no kvm_mem_protected() for s390.

Regards,
Halil

Quoted 3 lines
drivers/virtio/virtio_ring.c +0 -0
 	return false;
 }
 
#12 lore raw

RFC v2 6/16 x86/kvmclock: Share hvclock memory with the host

In reply to Kirill A. Shutemov

hvclock is shared between the guest and the hypervisor. It has to be accessible by host.

---
 arch/x86/kernel/kvmclock.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/kvmclock.c +1 -1
index 34b18f6eeb2c..ac6c2abe0d0f 100644
@@ -253,7 +253,7 @@ static void __init kvmclock_init_mem(void)
 	 * hvclock is shared between the guest and the hypervisor, must
 	 * be mapped decrypted.
 	 */
-	if (sev_active()) {
+	if (sev_active() || kvm_mem_protected()) {
 		r = set_memory_decrypted((unsigned long) hvclock_mem,
 					 1UL << order);
 		if (r) {
Signature · 1 line
2.26.2
#13 lore raw

RFC v2 7/16 x86/realmode: Share trampoline area if KVM memory protection enabled

In reply to Kirill A. Shutemov

If KVM memory protection is active, the trampoline area will need to be in shared memory.

---
 arch/x86/realmode/init.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
arch/x86/realmode/init.c +4 -3
index 1ed1208931e0..7392940a7f96 100644
@@ -9,6 +9,7 @@
 #include <asm/realmode.h>
 #include <asm/tlbflush.h>
 #include <asm/crash.h>
+#include <asm/kvm_para.h>
 
 struct real_mode_header *real_mode_header;
 u32 *trampoline_cr4_features;
@@ -55,11 +56,11 @@ static void __init setup_real_mode(void)
 	base = (unsigned char *)real_mode_header;
 
 	/*
-	 * If SME is active, the trampoline area will need to be in
-	 * decrypted memory in order to bring up other processors
+	 * If SME or KVM memory protection is active, the trampoline area will
+	 * need to be in decrypted memory in order to bring up other processors
 	 * successfully. This is not needed for SEV.
 	 */
-	if (sme_active())
+	if (sme_active() || kvm_mem_protected())
 		set_memory_decrypted((unsigned long)base, size >> PAGE_SHIFT);
 
 	memcpy(base, real_mode_blob, size);
Signature · 1 line
2.26.2
#14 lore raw

RFC v2 8/16 KVM: Use GUP instead of copy_from/to_user() to access guest memory

In reply to Kirill A. Shutemov

New helpers copy_from_guest()/copy_to_guest() to be used if KVM memory protection feature is enabled.

---
 include/linux/kvm_host.h |  4 ++
 virt/kvm/kvm_main.c      | 90 +++++++++++++++++++++++++++++++---------
 2 files changed, 75 insertions(+), 19 deletions(-)
include/linux/kvm_host.h +4 -0
index 05e3c2fb3ef7..380a64613880 100644
@@ -504,6 +504,7 @@ struct kvm {
 	struct srcu_struct irq_srcu;
 	pid_t userspace_pid;
 	unsigned int max_halt_poll_ns;
+	bool mem_protected;
 };
 
 #define kvm_err(fmt, ...) \
@@ -728,6 +729,9 @@ void kvm_set_pfn_dirty(kvm_pfn_t pfn);
 void kvm_set_pfn_accessed(kvm_pfn_t pfn);
 void kvm_get_pfn(kvm_pfn_t pfn);
 
+int copy_from_guest(void *data, unsigned long hva, int len, bool protected);
+int copy_to_guest(unsigned long hva, const void *data, int len, bool protected);
+
 void kvm_release_pfn(kvm_pfn_t pfn, bool dirty, struct gfn_to_pfn_cache *cache);
 int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,
 			int len);
virt/kvm/kvm_main.c +71 -19

157 lines not on this page · raw message

Signature · 1 line
2.26.2
reviewer
#15 lore raw

Re: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory

Quoted 47 lines · Kirill A. Shutemov

New helpers copy_from_guest()/copy_to_guest() to be used if KVM memory protection feature is enabled.

---
  include/linux/kvm_host.h |  4 ++
  virt/kvm/kvm_main.c      | 90 +++++++++++++++++++++++++++++++---------
  2 files changed, 75 insertions(+), 19 deletions(-)
include/linux/kvm_host.h +4 -0
index 05e3c2fb3ef7..380a64613880 100644
@@ -504,6 +504,7 @@ struct kvm {
  	struct srcu_struct irq_srcu;
  	pid_t userspace_pid;
  	unsigned int max_halt_poll_ns;
+	bool mem_protected;
  };
 
  #define kvm_err(fmt, ...) \
@@ -728,6 +729,9 @@ void kvm_set_pfn_dirty(kvm_pfn_t pfn);
  void kvm_set_pfn_accessed(kvm_pfn_t pfn);
  void kvm_get_pfn(kvm_pfn_t pfn);
 
+int copy_from_guest(void *data, unsigned long hva, int len, bool protected);
+int copy_to_guest(unsigned long hva, const void *data, int len, bool protected);
+
  void kvm_release_pfn(kvm_pfn_t pfn, bool dirty, struct gfn_to_pfn_cache *cache);
  int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,
  			int len);
virt/kvm/kvm_main.c +8 -0
index cf88233b819a..a9884cb8c867 100644
@@ -2313,19 +2313,70 @@ static int next_segment(unsigned long len, int offset)
  		return len;
  }
 
+int copy_from_guest(void *data, unsigned long hva, int len, bool protected)
+{
+	int offset = offset_in_page(hva);
+	struct page *page;
+	int npages, seg;
+
+	if (!protected)
+		return __copy_from_user(data, (void __user *)hva, len);

virt/kvm/kvm_main.c static int next_segment(unsigned long len, int offset) · line 2333

Quoted 10 lines
virt/kvm/kvm_main.c +10 -0
+
+	might_fault();
+	kasan_check_write(data, len);
+	check_object_size(data, len, false);
+
+	while ((seg = next_segment(len, offset)) != 0) {
+		npages = get_user_pages_unlocked(hva, 1, &page, 0);
+		if (npages != 1)
+			return -EFAULT;
+		memcpy(data, page_address(page) + offset, seg);

Hi Kirill!

OK, so the copy_from_guest() is a read-only case for gup, which I think is safe from a gup/pup + filesystem point of view, but see below about copy_to_guest()...

Quoted 14 lines
virt/kvm/kvm_main.c +14 -0
+		put_page(page);
+		len -= seg;
+		hva += seg;
+		offset = 0;
+	}
+
+	return 0;
+}
+
+int copy_to_guest(unsigned long hva, const void *data, int len, bool protected)
+{
+	int offset = offset_in_page(hva);
+	struct page *page;
+	int npages, seg;

virt/kvm/kvm_main.c static int next_segment(unsigned long len, int offset) · line 2357

Quoted 10 lines
virt/kvm/kvm_main.c +10 -0
+
+	if (!protected)
+		return __copy_to_user((void __user *)hva, data, len);
+
+	might_fault();
+	kasan_check_read(data, len);
+	check_object_size(data, len, true);
+
+	while ((seg = next_segment(len, offset)) != 0) {
+		npages = get_user_pages_unlocked(hva, 1, &page, FOLL_WRITE);

Should copy_to_guest() use pin_user_pages_unlocked() instead of gup_unlocked? We wrote a "Case 5" in Documentation/core-api/pin_user_pages.rst, just for this situation, I think:

CASE 5: Pinning in order to write to the data within the page
-------------------------------------------------------------
Even though neither DMA nor Direct IO is involved, just a simple case of "pin, write to a page's data, unpin" can cause a problem. Case 5 may be considered a superset of Case 1, plus Case 2, plus anything that invokes that pattern. In other words, if the code is neither Case 1 nor Case 2, it may still require FOLL_PIN, for patterns like this:

Correct (uses FOLL_PIN calls):
pin_user_pages()
write to the data within the pages
unpin_user_pages()

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
Quoted 122 lines
virt/kvm/kvm_main.c +29 -19
+		if (npages != 1)
+			return -EFAULT;
+		memcpy(page_address(page) + offset, data, seg);
+		put_page(page);
+		len -= seg;
+		hva += seg;
+		offset = 0;
+	}
+
+	return 0;
+}
+
  static int __kvm_read_guest_page(struct kvm_memory_slot *slot, gfn_t gfn,
-				 void *data, int offset, int len)
+				 void *data, int offset, int len,
+				 bool protected)
  {
-	int r;
  	unsigned long addr;
 
  	addr = gfn_to_hva_memslot_prot(slot, gfn, NULL);
  	if (kvm_is_error_hva(addr))
  		return -EFAULT;
-	r = __copy_from_user(data, (void __user *)addr + offset, len);
-	if (r)
-		return -EFAULT;
-	return 0;
+	return copy_from_guest(data, addr + offset, len, protected);
  }
 
  int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,
@@ -2333,7 +2384,8 @@ int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,
  {
  	struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn);
 
-	return __kvm_read_guest_page(slot, gfn, data, offset, len);
+	return __kvm_read_guest_page(slot, gfn, data, offset, len,
+				     kvm->mem_protected);
  }
  EXPORT_SYMBOL_GPL(kvm_read_guest_page);
 
@@ -2342,7 +2394,8 @@ int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data,
  {
  	struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
 
-	return __kvm_read_guest_page(slot, gfn, data, offset, len);
+	return __kvm_read_guest_page(slot, gfn, data, offset, len,
+				     vcpu->kvm->mem_protected);
  }
  EXPORT_SYMBOL_GPL(kvm_vcpu_read_guest_page);
 
@@ -2415,7 +2468,8 @@ int kvm_vcpu_read_guest_atomic(struct kvm_vcpu *vcpu, gpa_t gpa,
  EXPORT_SYMBOL_GPL(kvm_vcpu_read_guest_atomic);
 
  static int __kvm_write_guest_page(struct kvm_memory_slot *memslot, gfn_t gfn,
-			          const void *data, int offset, int len)
+			          const void *data, int offset, int len,
+				  bool protected)
  {
  	int r;
  	unsigned long addr;
@@ -2423,7 +2477,8 @@ static int __kvm_write_guest_page(struct kvm_memory_slot *memslot, gfn_t gfn,
  	addr = gfn_to_hva_memslot(memslot, gfn);
  	if (kvm_is_error_hva(addr))
  		return -EFAULT;
-	r = __copy_to_user((void __user *)addr + offset, data, len);
+
+	r = copy_to_guest(addr + offset, data, len, protected);
  	if (r)
  		return -EFAULT;
  	mark_page_dirty_in_slot(memslot, gfn);
@@ -2435,7 +2490,8 @@ int kvm_write_guest_page(struct kvm *kvm, gfn_t gfn,
  {
  	struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn);
 
-	return __kvm_write_guest_page(slot, gfn, data, offset, len);
+	return __kvm_write_guest_page(slot, gfn, data, offset, len,
+				      kvm->mem_protected);
  }
  EXPORT_SYMBOL_GPL(kvm_write_guest_page);
 
@@ -2444,7 +2500,8 @@ int kvm_vcpu_write_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn,
  {
  	struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
 
-	return __kvm_write_guest_page(slot, gfn, data, offset, len);
+	return __kvm_write_guest_page(slot, gfn, data, offset, len,
+				      vcpu->kvm->mem_protected);
  }
  EXPORT_SYMBOL_GPL(kvm_vcpu_write_guest_page);
 
@@ -2560,7 +2617,7 @@ int kvm_write_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
  	if (unlikely(!ghc->memslot))
  		return kvm_write_guest(kvm, gpa, data, len);
 
-	r = __copy_to_user((void __user *)ghc->hva + offset, data, len);
+	r = copy_to_guest(ghc->hva + offset, data, len, kvm->mem_protected);
  	if (r)
  		return -EFAULT;
  	mark_page_dirty_in_slot(ghc->memslot, gpa >> PAGE_SHIFT);
@@ -2581,7 +2638,6 @@ int kvm_read_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
  				 unsigned long len)
  {
  	struct kvm_memslots *slots = kvm_memslots(kvm);
-	int r;
  	gpa_t gpa = ghc->gpa + offset;
 
  	BUG_ON(len + offset > ghc->len);
@@ -2597,11 +2653,7 @@ int kvm_read_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
  	if (unlikely(!ghc->memslot))
  		return kvm_read_guest(kvm, gpa, data, len);
 
-	r = __copy_from_user(data, (void __user *)ghc->hva + offset, len);
-	if (r)
-		return -EFAULT;
-
-	return 0;
+	return copy_from_guest(data, ghc->hva + offset, len, kvm->mem_protected);
  }
  EXPORT_SYMBOL_GPL(kvm_read_guest_offset_cached);
 
#16 lore raw

Re: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory

Quoted 100 lines · John Hubbard
Quoted 44 lines · Kirill A. Shutemov

New helpers copy_from_guest()/copy_to_guest() to be used if KVM memory protection feature is enabled.

---
  include/linux/kvm_host.h |  4 ++
  virt/kvm/kvm_main.c      | 90 +++++++++++++++++++++++++++++++---------
  2 files changed, 75 insertions(+), 19 deletions(-)
include/linux/kvm_host.h +4 -0
index 05e3c2fb3ef7..380a64613880 100644
@@ -504,6 +504,7 @@ struct kvm {
  	struct srcu_struct irq_srcu;
  	pid_t userspace_pid;
  	unsigned int max_halt_poll_ns;
+	bool mem_protected;
  };
  #define kvm_err(fmt, ...) \
@@ -728,6 +729,9 @@ void kvm_set_pfn_dirty(kvm_pfn_t pfn);
  void kvm_set_pfn_accessed(kvm_pfn_t pfn);
  void kvm_get_pfn(kvm_pfn_t pfn);
+int copy_from_guest(void *data, unsigned long hva, int len, bool protected);
+int copy_to_guest(unsigned long hva, const void *data, int len, bool protected);
+
  void kvm_release_pfn(kvm_pfn_t pfn, bool dirty, struct gfn_to_pfn_cache *cache);
  int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,
  			int len);
virt/kvm/kvm_main.c +8 -0
index cf88233b819a..a9884cb8c867 100644
@@ -2313,19 +2313,70 @@ static int next_segment(unsigned long len, int offset)
  		return len;
  }
+int copy_from_guest(void *data, unsigned long hva, int len, bool protected)
+{
+	int offset = offset_in_page(hva);
+	struct page *page;
+	int npages, seg;
+
+	if (!protected)
+		return __copy_from_user(data, (void __user *)hva, len);

virt/kvm/kvm_main.c static int next_segment(unsigned long len, int offset) · line 2332

Quoted 10 lines
virt/kvm/kvm_main.c +10 -0
+
+	might_fault();
+	kasan_check_write(data, len);
+	check_object_size(data, len, false);
+
+	while ((seg = next_segment(len, offset)) != 0) {
+		npages = get_user_pages_unlocked(hva, 1, &page, 0);
+		if (npages != 1)
+			return -EFAULT;
+		memcpy(data, page_address(page) + offset, seg);

Hi Kirill!

OK, so the copy_from_guest() is a read-only case for gup, which I think is safe from a gup/pup + filesystem point of view, but see below about copy_to_guest()...

Quoted 14 lines
virt/kvm/kvm_main.c +14 -0
+		put_page(page);
+		len -= seg;
+		hva += seg;
+		offset = 0;
+	}
+
+	return 0;
+}
+
+int copy_to_guest(unsigned long hva, const void *data, int len, bool protected)
+{
+	int offset = offset_in_page(hva);
+	struct page *page;
+	int npages, seg;

virt/kvm/kvm_main.c static int next_segment(unsigned long len, int offset) · line 2357

Quoted 10 lines
virt/kvm/kvm_main.c +10 -0
+
+	if (!protected)
+		return __copy_to_user((void __user *)hva, data, len);
+
+	might_fault();
+	kasan_check_read(data, len);
+	check_object_size(data, len, true);
+
+	while ((seg = next_segment(len, offset)) != 0) {
+		npages = get_user_pages_unlocked(hva, 1, &page, FOLL_WRITE);

Should copy_to_guest() use pin_user_pages_unlocked() instead of gup_unlocked? We wrote a "Case 5" in Documentation/core-api/pin_user_pages.rst, just for this situation, I think:

CASE 5: Pinning in order to write to the data within the page
-------------------------------------------------------------
Even though neither DMA nor Direct IO is involved, just a simple case of "pin, write to a page's data, unpin" can cause a problem. Case 5 may be considered a superset of Case 1, plus Case 2, plus anything that invokes that pattern. In other words, if the code is neither Case 1 nor Case 2, it may still require FOLL_PIN, for patterns like this:

Quoted 4 lines

Correct (uses FOLL_PIN calls):
pin_user_pages()
write to the data within the pages
unpin_user_pages()

Right. I didn't internalize changes in GUP interface yet. Will update.

Signature · 1 line
 Kirill A. Shutemov
#17 lore raw

Re: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory

In reply to John Hubbard
Quoted 13 lines · John Hubbard

Should copy_to_guest() use pin_user_pages_unlocked() instead of gup_unlocked? We wrote a "Case 5" in Documentation/core-api/pin_user_pages.rst, just for this situation, I think:

CASE 5: Pinning in order to write to the data within the page
-------------------------------------------------------------
Even though neither DMA nor Direct IO is involved, just a simple case of "pin, write to a page's data, unpin" can cause a problem. Case 5 may be considered a superset of Case 1, plus Case 2, plus anything that invokes that pattern. In other words, if the code is neither Case 1 nor Case 2, it may still require FOLL_PIN, for patterns like this:

Quoted 4 lines

Correct (uses FOLL_PIN calls):
pin_user_pages()
write to the data within the pages
unpin_user_pages()

Case 5 is crap though. That bug should have been fixed by getting the locking right. ie:

	get_user_pages_fast();
	lock_page();
	kmap();
	set_bit();
	kunmap();
	set_page_dirty()
	unlock_page();

I should have vetoed that patch at the time, but I was busy with other things.

reviewer
#18 lore raw

Re: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory

Quoted 30 lines · Matthew Wilcox
Quoted 13 lines · John Hubbard

Should copy_to_guest() use pin_user_pages_unlocked() instead of gup_unlocked? We wrote a "Case 5" in Documentation/core-api/pin_user_pages.rst, just for this situation, I think:

CASE 5: Pinning in order to write to the data within the page
-------------------------------------------------------------
Even though neither DMA nor Direct IO is involved, just a simple case of "pin, write to a page's data, unpin" can cause a problem. Case 5 may be considered a superset of Case 1, plus Case 2, plus anything that invokes that pattern. In other words, if the code is neither Case 1 nor Case 2, it may still require FOLL_PIN, for patterns like this:

Quoted 4 lines

Correct (uses FOLL_PIN calls):
pin_user_pages()
write to the data within the pages
unpin_user_pages()

Case 5 is crap though. That bug should have been fixed by getting the locking right. ie:

	get_user_pages_fast();
	lock_page();
	kmap();
	set_bit();
	kunmap();
	set_page_dirty()
	unlock_page();
Quoted 2 lines

I should have vetoed that patch at the time, but I was busy with other things.

It does seem like lock_page() is better, for now at least, because it forces the kind of synchronization with file system writeback that is still yet to be implemented for pin_user_pages().

Long term though, Case 5 provides an alternative way to do this pattern--without using lock_page(). Also, note that Case 5, *in general*, need not be done page-at-a-time, unlike the lock_page() approach. Therefore, Case 5 might potentially help at some call sites, either for deadlock avoidance or performance improvements.

In other words, once the other half of the pin_user_pages() plan is implemented, either of these approaches should work.

Or, are you thinking that there is never a situation in which Case 5 is valid?

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#19 lore raw

Re: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory

Quoted 43 lines · John Hubbard
Quoted 30 lines · Matthew Wilcox
Quoted 13 lines · John Hubbard

Should copy_to_guest() use pin_user_pages_unlocked() instead of gup_unlocked? We wrote a "Case 5" in Documentation/core-api/pin_user_pages.rst, just for this situation, I think:

CASE 5: Pinning in order to write to the data within the page
-------------------------------------------------------------
Even though neither DMA nor Direct IO is involved, just a simple case of "pin, write to a page's data, unpin" can cause a problem. Case 5 may be considered a superset of Case 1, plus Case 2, plus anything that invokes that pattern. In other words, if the code is neither Case 1 nor Case 2, it may still require FOLL_PIN, for patterns like this:

Quoted 4 lines

Correct (uses FOLL_PIN calls):
pin_user_pages()
write to the data within the pages
unpin_user_pages()

Case 5 is crap though. That bug should have been fixed by getting the locking right. ie:

	get_user_pages_fast();
	lock_page();
	kmap();
	set_bit();
	kunmap();
	set_page_dirty()
	unlock_page();
Quoted 1 line

I should have vetoed that patch at the time, but I was busy with other things.

It does seem like lock_page() is better, for now at least, because it forces the kind of synchronization with file system writeback that is still yet to be implemented for pin_user_pages().

Long term though, Case 5 provides an alternative way to do this pattern--without using lock_page(). Also, note that Case 5, *in general*, need not be done page-at-a-time, unlike the lock_page() approach. Therefore, Case 5 might potentially help at some call sites, either for deadlock avoidance or performance improvements.

Quoted 5 lines

In other words, once the other half of the pin_user_pages() plan is implemented, either of these approaches should work.

Or, are you thinking that there is never a situation in which Case 5 is valid?

I don't think the page pinning approach is ever valid. For file mappings, the infiniband registration should take a lease on the inode, blocking truncation. DAX shouldn't be using struct pages at all, so there shouldn't be anything there to pin.

It's been five years since DAX was merged, and page pinning still doesn't work. How much longer before the people who are pushing it realise that it's fundamentally flawed?

reviewer
#20 lore raw

Re: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory

Quoted 44 lines · Matthew Wilcox
Quoted 43 lines · John Hubbard
Quoted 30 lines · Matthew Wilcox
Quoted 13 lines · John Hubbard

Should copy_to_guest() use pin_user_pages_unlocked() instead of gup_unlocked? We wrote a "Case 5" in Documentation/core-api/pin_user_pages.rst, just for this situation, I think:

CASE 5: Pinning in order to write to the data within the page
-------------------------------------------------------------
Even though neither DMA nor Direct IO is involved, just a simple case of "pin, write to a page's data, unpin" can cause a problem. Case 5 may be considered a superset of Case 1, plus Case 2, plus anything that invokes that pattern. In other words, if the code is neither Case 1 nor Case 2, it may still require FOLL_PIN, for patterns like this:

Quoted 4 lines

Correct (uses FOLL_PIN calls):
pin_user_pages()
write to the data within the pages
unpin_user_pages()

Case 5 is crap though. That bug should have been fixed by getting the locking right. ie:

	get_user_pages_fast();
	lock_page();
	kmap();
	set_bit();
	kunmap();
	set_page_dirty()
	unlock_page();
Quoted 1 line

I should have vetoed that patch at the time, but I was busy with other things.

It does seem like lock_page() is better, for now at least, because it forces the kind of synchronization with file system writeback that is still yet to be implemented for pin_user_pages().

Long term though, Case 5 provides an alternative way to do this pattern--without using lock_page(). Also, note that Case 5, *in general*, need not be done page-at-a-time, unlike the lock_page() approach. Therefore, Case 5 might potentially help at some call sites, either for deadlock avoidance or performance improvements.

Quoted 7 lines
Quoted 5 lines

In other words, once the other half of the pin_user_pages() plan is implemented, either of these approaches should work.

Or, are you thinking that there is never a situation in which Case 5 is valid?

I don't think the page pinning approach is ever valid. For file

Could you qualify that? Surely you don't mean that the entire pin_user_pages story is a waste of time--I would have expected you to make more noise earlier if you thought that, yes?

Quoted 8 lines

mappings, the infiniband registration should take a lease on the inode, blocking truncation. DAX shouldn't be using struct pages at all, so there shouldn't be anything there to pin.

It's been five years since DAX was merged, and page pinning still doesn't work. How much longer before the people who are pushing it realise that it's fundamentally flawed?

Is this a separate rant about *only* DAX, or is general RDMA in your sights too? :)

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#21 lore raw

Re: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory

Quoted 6 lines · John Hubbard
Quoted 1 line · Matthew Wilcox

I don't think the page pinning approach is ever valid. For file

Could you qualify that? Surely you don't mean that the entire pin_user_pages story is a waste of time--I would have expected you to make more noise earlier if you thought that, yes?

I do think page pinning is the wrong approach for everything. I did say so at the time, and I continue to say so when the opportunity presents itself. But shouting about it constantly only annoys people, so I don't generally bother. I have other things to work on, and they're productive, so I don't need to spend my time arguing.

Quoted 6 lines
Quoted 3 lines

It's been five years since DAX was merged, and page pinning still doesn't work. How much longer before the people who are pushing it realise that it's fundamentally flawed?

Is this a separate rant about *only* DAX, or is general RDMA in your sights too? :)

This is a case where it's not RDMA's _fault_ that there's no good API for it to do what it needs to do. There's a lot of work needed to wean Linux device drivers off their assumption that there's a struct page for every byte of memory.

#22 lore raw

Re: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory

Quoted 11 lines · Matthew Wilcox
Quoted 6 lines
Quoted 3 lines

It's been five years since DAX was merged, and page pinning still doesn't work. How much longer before the people who are pushing it realise that it's fundamentally flawed?

Is this a separate rant about *only* DAX, or is general RDMA in your sights too? :)

This is a case where it's not RDMA's _fault_ that there's no good API for it to do what it needs to do. There's a lot of work needed to wean Linux device drivers off their assumption that there's a struct page for every byte of memory.

People who care seem to have just given up and are using RDMA ODP, so I'm not optimistic this DAX issue will ever be solved. I've also almost removed all the struct page references from this flow in RDMA, so if there is some way that helps it is certainly doable.

Regardless of DAX the pinning indication is now being used during fork() for some good reasons, and seems to make sense in other use cases. It just doesn't seem like a way to solve the DAX issue.

More or less it seems to mean that pages pinned cannot be write protected and more broadly the kernel should not change the PTEs for those pages independently of the application. ie the more agressive COW on fork() caused data corruption regressions...

I wonder if the point here is that some page owners can't/won't support DMA pinning and should just be blocked completely for them.

I'd much rather have write access pin_user_pages() just fail than oops the kernel on ext4 owned VMAs, for instance.

Jason

reviewer
#23 lore raw

Re: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory

In reply to Matthew Wilcox
Quoted 9 lines · Matthew Wilcox
Quoted 6 lines · John Hubbard
Quoted 1 line · Matthew Wilcox

I don't think the page pinning approach is ever valid. For file

Could you qualify that? Surely you don't mean that the entire pin_user_pages story is a waste of time--I would have expected you to make more noise earlier if you thought that, yes?

I do think page pinning is the wrong approach for everything. I did say

Not *everything*, just "pinning for DMA", right? Because I don't recall any viable solutions for Direct IO that avoided gup/pup!

Also, back to Case 5: I *could* create a small patchset to change over the very few Case 5 call sites to use "gup, lock_page(), write to page...etc", instead of pup. And also, update pin_user_pages.rst to recommend that approach in similar situations. After all, it's not really a long-term DMA pin, which is really what pin_user_pages*() is intended for.

Would that be something you'd like to see happen? It's certainly easy enough to fix that up. And your retroactive NAK is sufficient motivation to do so.

Quoted 4 lines

so at the time, and I continue to say so when the opportunity presents itself. But shouting about it constantly only annoys people, so I don't generally bother. I have other things to work on, and they're productive, so I don't need to spend my time arguing.

Sure. As a practical matter, I've assumed that page pinning is not going to go away any time soon, so I want it to work properly while it's here. But if there is a viable way to eventually replace dma-pinning with something better, then let's keep thinking about it. I'm glad to help in that area.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#24 lore raw

Re: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory

In reply to Kirill A. Shutemov
Quoted 51 lines · Kirill A. Shutemov

New helpers copy_from_guest()/copy_to_guest() to be used if KVM memory protection feature is enabled.

---
 include/linux/kvm_host.h |  4 ++
 virt/kvm/kvm_main.c      | 90 +++++++++++++++++++++++++++++++---------
 2 files changed, 75 insertions(+), 19 deletions(-)
include/linux/kvm_host.h +4 -0
index 05e3c2fb3ef7..380a64613880 100644
@@ -504,6 +504,7 @@ struct kvm {
 	struct srcu_struct irq_srcu;
 	pid_t userspace_pid;
 	unsigned int max_halt_poll_ns;
+	bool mem_protected;
 };
 
 #define kvm_err(fmt, ...) \
@@ -728,6 +729,9 @@ void kvm_set_pfn_dirty(kvm_pfn_t pfn);
 void kvm_set_pfn_accessed(kvm_pfn_t pfn);
 void kvm_get_pfn(kvm_pfn_t pfn);
 
+int copy_from_guest(void *data, unsigned long hva, int len, bool protected);
+int copy_to_guest(unsigned long hva, const void *data, int len, bool protected);
+
 void kvm_release_pfn(kvm_pfn_t pfn, bool dirty, struct gfn_to_pfn_cache *cache);
 int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,
 			int len);
virt/kvm/kvm_main.c +12 -0
index cf88233b819a..a9884cb8c867 100644
@@ -2313,19 +2313,70 @@ static int next_segment(unsigned long len, int offset)
 		return len;
 }
 
+int copy_from_guest(void *data, unsigned long hva, int len, bool protected)
+{
+	int offset = offset_in_page(hva);
+	struct page *page;
+	int npages, seg;
+
+	if (!protected)
+		return __copy_from_user(data, (void __user *)hva, len);
+
+	might_fault();
+	kasan_check_write(data, len);
+	check_object_size(data, len, false);

virt/kvm/kvm_main.c static int next_segment(unsigned long len, int offset) · line 2337

Quoted 10 lines
virt/kvm/kvm_main.c +10 -0
+
+	while ((seg = next_segment(len, offset)) != 0) {
+		npages = get_user_pages_unlocked(hva, 1, &page, 0);
+		if (npages != 1)
+			return -EFAULT;
+		memcpy(data, page_address(page) + offset, seg);
+		put_page(page);
+		len -= seg;
+		hva += seg;
+		offset = 0;

Why is data not updated on each iteration?

Quoted 17 lines
virt/kvm/kvm_main.c +17 -0
+	}
+
+	return 0;
+}
+
+int copy_to_guest(unsigned long hva, const void *data, int len, bool protected)
+{
+	int offset = offset_in_page(hva);
+	struct page *page;
+	int npages, seg;
+
+	if (!protected)
+		return __copy_to_user((void __user *)hva, data, len);
+
+	might_fault();
+	kasan_check_read(data, len);
+	check_object_size(data, len, true);

virt/kvm/kvm_main.c static int next_segment(unsigned long len, int offset) · line 2364

Quoted 10 lines
virt/kvm/kvm_main.c +10 -0
+
+	while ((seg = next_segment(len, offset)) != 0) {
+		npages = get_user_pages_unlocked(hva, 1, &page, FOLL_WRITE);
+		if (npages != 1)
+			return -EFAULT;
+		memcpy(page_address(page) + offset, data, seg);
+		put_page(page);
+		len -= seg;
+		hva += seg;
+		offset = 0;

Same question? Doesn't this result in *data being copied to multiple pages?

Ira

Quoted 118 lines
virt/kvm/kvm_main.c +22 -19
+	}
+
+	return 0;
+}
+
 static int __kvm_read_guest_page(struct kvm_memory_slot *slot, gfn_t gfn,
-				 void *data, int offset, int len)
+				 void *data, int offset, int len,
+				 bool protected)
 {
-	int r;
 	unsigned long addr;
 
 	addr = gfn_to_hva_memslot_prot(slot, gfn, NULL);
 	if (kvm_is_error_hva(addr))
 		return -EFAULT;
-	r = __copy_from_user(data, (void __user *)addr + offset, len);
-	if (r)
-		return -EFAULT;
-	return 0;
+	return copy_from_guest(data, addr + offset, len, protected);
 }
 
 int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,
@@ -2333,7 +2384,8 @@ int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,
 {
 	struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn);
 
-	return __kvm_read_guest_page(slot, gfn, data, offset, len);
+	return __kvm_read_guest_page(slot, gfn, data, offset, len,
+				     kvm->mem_protected);
 }
 EXPORT_SYMBOL_GPL(kvm_read_guest_page);
 
@@ -2342,7 +2394,8 @@ int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data,
 {
 	struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
 
-	return __kvm_read_guest_page(slot, gfn, data, offset, len);
+	return __kvm_read_guest_page(slot, gfn, data, offset, len,
+				     vcpu->kvm->mem_protected);
 }
 EXPORT_SYMBOL_GPL(kvm_vcpu_read_guest_page);
 
@@ -2415,7 +2468,8 @@ int kvm_vcpu_read_guest_atomic(struct kvm_vcpu *vcpu, gpa_t gpa,
 EXPORT_SYMBOL_GPL(kvm_vcpu_read_guest_atomic);
 
 static int __kvm_write_guest_page(struct kvm_memory_slot *memslot, gfn_t gfn,
-			          const void *data, int offset, int len)
+			          const void *data, int offset, int len,
+				  bool protected)
 {
 	int r;
 	unsigned long addr;
@@ -2423,7 +2477,8 @@ static int __kvm_write_guest_page(struct kvm_memory_slot *memslot, gfn_t gfn,
 	addr = gfn_to_hva_memslot(memslot, gfn);
 	if (kvm_is_error_hva(addr))
 		return -EFAULT;
-	r = __copy_to_user((void __user *)addr + offset, data, len);
+
+	r = copy_to_guest(addr + offset, data, len, protected);
 	if (r)
 		return -EFAULT;
 	mark_page_dirty_in_slot(memslot, gfn);
@@ -2435,7 +2490,8 @@ int kvm_write_guest_page(struct kvm *kvm, gfn_t gfn,
 {
 	struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn);
 
-	return __kvm_write_guest_page(slot, gfn, data, offset, len);
+	return __kvm_write_guest_page(slot, gfn, data, offset, len,
+				      kvm->mem_protected);
 }
 EXPORT_SYMBOL_GPL(kvm_write_guest_page);
 
@@ -2444,7 +2500,8 @@ int kvm_vcpu_write_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn,
 {
 	struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
 
-	return __kvm_write_guest_page(slot, gfn, data, offset, len);
+	return __kvm_write_guest_page(slot, gfn, data, offset, len,
+				      vcpu->kvm->mem_protected);
 }
 EXPORT_SYMBOL_GPL(kvm_vcpu_write_guest_page);
 
@@ -2560,7 +2617,7 @@ int kvm_write_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 	if (unlikely(!ghc->memslot))
 		return kvm_write_guest(kvm, gpa, data, len);
 
-	r = __copy_to_user((void __user *)ghc->hva + offset, data, len);
+	r = copy_to_guest(ghc->hva + offset, data, len, kvm->mem_protected);
 	if (r)
 		return -EFAULT;
 	mark_page_dirty_in_slot(ghc->memslot, gpa >> PAGE_SHIFT);
@@ -2581,7 +2638,6 @@ int kvm_read_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 				 unsigned long len)
 {
 	struct kvm_memslots *slots = kvm_memslots(kvm);
-	int r;
 	gpa_t gpa = ghc->gpa + offset;
 
 	BUG_ON(len + offset > ghc->len);
@@ -2597,11 +2653,7 @@ int kvm_read_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 	if (unlikely(!ghc->memslot))
 		return kvm_read_guest(kvm, gpa, data, len);
 
-	r = __copy_from_user(data, (void __user *)ghc->hva + offset, len);
-	if (r)
-		return -EFAULT;
-
-	return 0;
+	return copy_from_guest(data, ghc->hva + offset, len, kvm->mem_protected);
 }
 EXPORT_SYMBOL_GPL(kvm_read_guest_offset_cached);
 
Signature · 1 line
2.26.2
#25 lore raw

Re: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory

Quoted 56 lines · Ira Weiny
Quoted 55 lines · Kirill A. Shutemov

New helpers copy_from_guest()/copy_to_guest() to be used if KVM memory protection feature is enabled.

---
 include/linux/kvm_host.h |  4 ++
 virt/kvm/kvm_main.c      | 90 +++++++++++++++++++++++++++++++---------
 2 files changed, 75 insertions(+), 19 deletions(-)
include/linux/kvm_host.h +4 -0
index 05e3c2fb3ef7..380a64613880 100644
@@ -504,6 +504,7 @@ struct kvm {
 	struct srcu_struct irq_srcu;
 	pid_t userspace_pid;
 	unsigned int max_halt_poll_ns;
+	bool mem_protected;
 };
 
 #define kvm_err(fmt, ...) \
@@ -728,6 +729,9 @@ void kvm_set_pfn_dirty(kvm_pfn_t pfn);
 void kvm_set_pfn_accessed(kvm_pfn_t pfn);
 void kvm_get_pfn(kvm_pfn_t pfn);
 
+int copy_from_guest(void *data, unsigned long hva, int len, bool protected);
+int copy_to_guest(unsigned long hva, const void *data, int len, bool protected);
+
 void kvm_release_pfn(kvm_pfn_t pfn, bool dirty, struct gfn_to_pfn_cache *cache);
 int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,
 			int len);
virt/kvm/kvm_main.c +16 -0
index cf88233b819a..a9884cb8c867 100644
@@ -2313,19 +2313,70 @@ static int next_segment(unsigned long len, int offset)
 		return len;
 }
 
+int copy_from_guest(void *data, unsigned long hva, int len, bool protected)
+{
+	int offset = offset_in_page(hva);
+	struct page *page;
+	int npages, seg;
+
+	if (!protected)
+		return __copy_from_user(data, (void __user *)hva, len);
+
+	might_fault();
+	kasan_check_write(data, len);
+	check_object_size(data, len, false);
+
+	while ((seg = next_segment(len, offset)) != 0) {
+		npages = get_user_pages_unlocked(hva, 1, &page, 0);
+		if (npages != 1)
Quoted 8 lines

virt/kvm/kvm_main.c static int next_segment(unsigned long len, int offset) · line 2337

Quoted 6 lines
virt/kvm/kvm_main.c +6 -0
+			return -EFAULT;
+		memcpy(data, page_address(page) + offset, seg);
+		put_page(page);
+		len -= seg;
+		hva += seg;
+		offset = 0;

Why is data not updated on each iteration?

Ouch. I guess no caller actually steps over page boundary. Will fix.

Signature · 1 line
 Kirill A. Shutemov
#26 lore raw

RFC v2 9/16 KVM: mm: Introduce VM_KVM_PROTECTED

In reply to Kirill A. Shutemov

The new VMA flag that indicate a VMA that is not accessible to userspace but usable by kernel with GUP if FOLL_KVM is specified.

The FOLL_KVM is only used in the KVM code. The code has to know how to deal with such pages.

---
 include/linux/mm.h  |  8 ++++++++
 mm/gup.c            | 20 ++++++++++++++++----
 mm/huge_memory.c    | 20 ++++++++++++++++----
 mm/memory.c         |  3 +++
 mm/mmap.c           |  3 +++
 virt/kvm/async_pf.c |  2 +-
 virt/kvm/kvm_main.c |  9 +++++----
 7 files changed, 52 insertions(+), 13 deletions(-)
include/linux/mm.h +8 -0
index 16b799a0522c..c8d8cdcbc425 100644
@@ -342,6 +342,8 @@ extern unsigned int kobjsize(const void *objp);
 # define VM_MAPPED_COPY	VM_ARCH_1	/* T if mapped copy of data (nommu mmap) */
 #endif
 
+#define VM_KVM_PROTECTED 0
+
 #ifndef VM_GROWSUP
 # define VM_GROWSUP	VM_NONE
 #endif
@@ -658,6 +660,11 @@ static inline bool vma_is_accessible(struct vm_area_struct *vma)
 	return vma->vm_flags & VM_ACCESS_FLAGS;
 }
 
+static inline bool vma_is_kvm_protected(struct vm_area_struct *vma)
+{
+	return vma->vm_flags & VM_KVM_PROTECTED;
+}
+
 #ifdef CONFIG_SHMEM
 /*
  * The vma_is_shmem is not inline because it is used only by slow
@@ -2766,6 +2773,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,
 #define FOLL_SPLIT_PMD	0x20000	/* split huge pmd before returning */
 #define FOLL_PIN	0x40000	/* pages must be released via unpin_user_page */
 #define FOLL_FAST_ONLY	0x80000	/* gup_fast: prevent fall-back to slow gup */
+#define FOLL_KVM	0x100000 /* access to VM_KVM_PROTECTED VMAs */
 
 /*
  * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each
mm/gup.c +16 -4
index e869c634cc9a..accf6db0c06f 100644
@@ -384,10 +384,19 @@ static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address,
  * FOLL_FORCE can write to even unwritable pte's, but only
  * after we've gone through a COW cycle and they are dirty.
  */
-static inline bool can_follow_write_pte(pte_t pte, unsigned int flags)
+static inline bool can_follow_write_pte(struct vm_area_struct *vma,
+					pte_t pte, unsigned int flags)
 {
-	return pte_write(pte) ||
-		((flags & FOLL_FORCE) && (flags & FOLL_COW) && pte_dirty(pte));
+	if (pte_write(pte))
+		return true;
+
+	if ((flags & FOLL_FORCE) && (flags & FOLL_COW) && pte_dirty(pte))
+		return true;
+
+	if (!vma_is_kvm_protected(vma) || !(vma->vm_flags & VM_WRITE))
+		return false;
+
+	return (vma->vm_flags & VM_SHARED) || page_mapcount(pte_page(pte)) == 1;
 }
 
 static struct page *follow_page_pte(struct vm_area_struct *vma,
@@ -430,7 +439,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,
 	}
 	if ((flags & FOLL_NUMA) && pte_protnone(pte))
 		goto no_page;
-	if ((flags & FOLL_WRITE) && !can_follow_write_pte(pte, flags)) {
+	if ((flags & FOLL_WRITE) && !can_follow_write_pte(vma, pte, flags)) {
 		pte_unmap_unlock(ptep, ptl);
 		return NULL;
 	}
@@ -750,6 +759,9 @@ static struct page *follow_page_mask(struct vm_area_struct *vma,
 
 	ctx->page_mask = 0;
 
+	if (vma_is_kvm_protected(vma) && (flags & FOLL_KVM))
+		flags &= ~FOLL_NUMA;
+
 	/* make this handle hugepd */
 	page = follow_huge_addr(mm, address, flags & FOLL_WRITE);
 	if (!IS_ERR(page)) {
mm/huge_memory.c +16 -4
index da397779a6d4..ec8cf9a40cfd 100644
@@ -1322,10 +1322,19 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd)
  * FOLL_FORCE can write to even unwritable pmd's, but only
  * after we've gone through a COW cycle and they are dirty.
  */
-static inline bool can_follow_write_pmd(pmd_t pmd, unsigned int flags)
+static inline bool can_follow_write_pmd(struct vm_area_struct *vma,
+					pmd_t pmd, unsigned int flags)
 {
-	return pmd_write(pmd) ||
-	       ((flags & FOLL_FORCE) && (flags & FOLL_COW) && pmd_dirty(pmd));
+	if (pmd_write(pmd))
+		return true;
+
+	if ((flags & FOLL_FORCE) && (flags & FOLL_COW) && pmd_dirty(pmd))
+		return true;
+
+	if (!vma_is_kvm_protected(vma) || !(vma->vm_flags & VM_WRITE))
+		return false;
+
+	return (vma->vm_flags & VM_SHARED) || page_mapcount(pmd_page(pmd)) == 1;
 }
 
 struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
@@ -1338,7 +1347,7 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
 
 	assert_spin_locked(pmd_lockptr(mm, pmd));
 
-	if (flags & FOLL_WRITE && !can_follow_write_pmd(*pmd, flags))
+	if (flags & FOLL_WRITE && !can_follow_write_pmd(vma, *pmd, flags))
 		goto out;
 
 	/* Avoid dumping huge zero page */
@@ -1412,6 +1421,9 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t pmd)
 	bool was_writable;
 	int flags = 0;
 
+	if (vma_is_kvm_protected(vma))
+		return VM_FAULT_SIGBUS;
+
 	vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
 	if (unlikely(!pmd_same(pmd, *vmf->pmd)))
 		goto out_unlock;
mm/memory.c +3 -0
index eeae590e526a..2c9756b4e52f 100644
@@ -4165,6 +4165,9 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 	bool was_writable = pte_savedwrite(vmf->orig_pte);
 	int flags = 0;
 
+	if (vma_is_kvm_protected(vma))
+		return VM_FAULT_SIGBUS;
+
 	/*
 	 * The "pte" at this point cannot be used safely without
 	 * validation through pte_unmap_same(). It's of NUMA type but
mm/mmap.c +3 -0
index bdd19f5b994e..be699f688b6c 100644
@@ -112,6 +112,9 @@ pgprot_t vm_get_page_prot(unsigned long vm_flags)
 				(VM_READ|VM_WRITE|VM_EXEC|VM_SHARED)]) |
 			pgprot_val(arch_vm_get_page_prot(vm_flags)));
 
+	if (vm_flags & VM_KVM_PROTECTED)
+		ret = PAGE_NONE;
+
 	return arch_filter_pgprot(ret);
 }
 EXPORT_SYMBOL(vm_get_page_prot);
virt/kvm/async_pf.c +1 -1
index dd777688d14a..85a2f99f6e9b 100644
@@ -61,7 +61,7 @@ static void async_pf_execute(struct work_struct *work)
 	 * access remotely.
 	 */
 	mmap_read_lock(mm);
-	get_user_pages_remote(mm, addr, 1, FOLL_WRITE, NULL, NULL,
+	get_user_pages_remote(mm, addr, 1, FOLL_WRITE | FOLL_KVM, NULL, NULL,
 			&locked);
 	if (locked)
 		mmap_read_unlock(mm);
virt/kvm/kvm_main.c +5 -4
index a9884cb8c867..125db5a73e10 100644
@@ -1794,7 +1794,7 @@ unsigned long kvm_vcpu_gfn_to_hva_prot(struct kvm_vcpu *vcpu, gfn_t gfn, bool *w
 
 static inline int check_user_page_hwpoison(unsigned long addr)
 {
-	int rc, flags = FOLL_HWPOISON | FOLL_WRITE;
+	int rc, flags = FOLL_HWPOISON | FOLL_WRITE | FOLL_KVM;
 
 	rc = get_user_pages(addr, 1, flags, NULL, NULL);
 	return rc == -EHWPOISON;
@@ -1836,7 +1836,7 @@ static bool hva_to_pfn_fast(unsigned long addr, bool write_fault,
 static int hva_to_pfn_slow(unsigned long addr, bool *async, bool write_fault,
 			   bool *writable, kvm_pfn_t *pfn)
 {
-	unsigned int flags = FOLL_HWPOISON;
+	unsigned int flags = FOLL_HWPOISON | FOLL_KVM;
 	struct page *page;
 	int npages = 0;
 
@@ -2327,7 +2327,7 @@ int copy_from_guest(void *data, unsigned long hva, int len, bool protected)
 	check_object_size(data, len, false);
 
 	while ((seg = next_segment(len, offset)) != 0) {
-		npages = get_user_pages_unlocked(hva, 1, &page, 0);
+		npages = get_user_pages_unlocked(hva, 1, &page, FOLL_KVM);
 		if (npages != 1)
 			return -EFAULT;
 		memcpy(data, page_address(page) + offset, seg);
@@ -2354,7 +2354,8 @@ int copy_to_guest(unsigned long hva, const void *data, int len, bool protected)
 	check_object_size(data, len, true);
 
 	while ((seg = next_segment(len, offset)) != 0) {
-		npages = get_user_pages_unlocked(hva, 1, &page, FOLL_WRITE);
+		npages = get_user_pages_unlocked(hva, 1, &page,
+						 FOLL_WRITE | FOLL_KVM);
 		if (npages != 1)
 			return -EFAULT;
 		memcpy(page_address(page) + offset, data, seg);
Signature · 1 line
2.26.2
#27 lore raw

Re: [RFCv2 09/16] KVM: mm: Introduce VM_KVM_PROTECTED

Quoted 8 lines · Kirill A. Shutemov
 include/linux/mm.h  |  8 ++++++++
 mm/gup.c            | 20 ++++++++++++++++----
 mm/huge_memory.c    | 20 ++++++++++++++++----
 mm/memory.c         |  3 +++
 mm/mmap.c           |  3 +++
 virt/kvm/async_pf.c |  2 +-
 virt/kvm/kvm_main.c |  9 +++++----
 7 files changed, 52 insertions(+), 13 deletions(-)

There is another get_user_pages() in
paging_tmpl.h:FNAME(cmpxchg_gpte).

Also, did you leave off FOLL_KVM in gfn_to_page_many_atomic() on purpose? It looks like its only used for pte prefetch which I guess is not required.

#28 lore raw

Re: [RFCv2 09/16] KVM: mm: Introduce VM_KVM_PROTECTED

Quoted 10 lines · Rick P
Quoted 8 lines · Kirill A. Shutemov
 include/linux/mm.h  |  8 ++++++++
 mm/gup.c            | 20 ++++++++++++++++----
 mm/huge_memory.c    | 20 ++++++++++++++++----
 mm/memory.c         |  3 +++
 mm/mmap.c           |  3 +++
 virt/kvm/async_pf.c |  2 +-
 virt/kvm/kvm_main.c |  9 +++++----
 7 files changed, 52 insertions(+), 13 deletions(-)
Quoted 2 lines

There is another get_user_pages() in
paging_tmpl.h:FNAME(cmpxchg_gpte).

Right, I will look into it. I wounder why I don't step onto it.

Quoted 3 lines

Also, did you leave off FOLL_KVM in gfn_to_page_many_atomic() on purpose? It looks like its only used for pte prefetch which I guess is not required.

FOLL_FAST_ONLY is going to fail anyway. The caller has to handle it gracefully.

Signature · 1 line
 Kirill A. Shutemov
#29 lore raw

RFC v2 10/16 KVM: x86: Use GUP for page walk instead of __get_user()

In reply to Kirill A. Shutemov

The user mapping doesn't have the page mapping for protected memory.

---
 arch/x86/kvm/mmu/paging_tmpl.h | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/paging_tmpl.h +8 -2
index 4dd6b1e5b8cf..258a6361b9b2 100644
@@ -397,8 +397,14 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,
 			goto error;
 
 		ptep_user = (pt_element_t __user *)((void *)host_addr + offset);
-		if (unlikely(__get_user(pte, ptep_user)))
-			goto error;
+		if (vcpu->kvm->mem_protected) {
+			if (copy_from_guest(&pte, host_addr + offset,
+					    sizeof(pte), true))
+				goto error;
+		} else {
+			if (unlikely(__get_user(pte, ptep_user)))
+				goto error;
+		}
 		walker->ptep_user[walker->level - 1] = ptep_user;
 
 		trace_kvm_mmu_paging_element(pte, walker->level);
Signature · 1 line
2.26.2
#30 lore raw

RFC v2 11/16 KVM: Protected memory extension

In reply to Kirill A. Shutemov

Add infrastructure that handles protected memory extension.

Arch-specific code has to provide hypercalls and define non-zero VM_KVM_PROTECTED.

---
 include/linux/kvm_host.h |  4 +++
 virt/kvm/Kconfig         |  3 ++
 virt/kvm/kvm_main.c      | 68 ++++++++++++++++++++++++++++++++++++++
 virt/lib/Makefile        |  1 +
 virt/lib/mem_protected.c | 71 ++++++++++++++++++++++++++++++++++++++++
 5 files changed, 147 insertions(+)
 create mode 100644 virt/lib/mem_protected.c
include/linux/kvm_host.h +4 -0
index 380a64613880..6655e8da4555 100644
@@ -701,6 +701,10 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm);
 void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
 				   struct kvm_memory_slot *slot);
 
+int kvm_protect_all_memory(struct kvm *kvm);
+int kvm_protect_memory(struct kvm *kvm,
+		       unsigned long gfn, unsigned long npages, bool protect);
+
 int gfn_to_page_many_atomic(struct kvm_memory_slot *slot, gfn_t gfn,
 			    struct page **pages, int nr_pages);
 
virt/kvm/Kconfig +3 -0
index 1c37ccd5d402..50d7422386aa 100644
@@ -63,3 +63,6 @@ config HAVE_KVM_NO_POLL
 
 config KVM_XFER_TO_GUEST_WORK
        bool
+
+config HAVE_KVM_PROTECTED_MEMORY
+       bool
virt/kvm/kvm_main.c +68 -0

86 lines not on this page · raw message

virt/lib/Makefile +1 -0
index bd7f9a78bb6b..d6e50510801f 100644
@@ -1,2 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0-only
 obj-$(CONFIG_IRQ_BYPASS_MANAGER) += irqbypass.o
+obj-$(CONFIG_HAVE_KVM_PROTECTED_MEMORY) += mem_protected.o
virt/lib/mem_protected.c new file +71 -0

71 lines not on this page · raw message

Signature · 1 line
2.26.2
maintainer
#31 lore raw

Re: [RFCv2 11/16] KVM: Protected memory extension

Quoted 54 lines · Kirill A. Shutemov
virt/lib/mem_protected.c new file +54 -0
+int __kvm_protect_memory(unsigned long start, unsigned long end, bool protect)
+{
+	struct mm_struct *mm = current->mm;
+	struct vm_area_struct *vma, *prev;
+	int ret;
+
+	if (mmap_write_lock_killable(mm))
+		return -EINTR;
+
+	ret = -ENOMEM;
+	vma = find_vma(current->mm, start);
+	if (!vma)
+		goto out;
+
+	ret = -EINVAL;
+	if (vma->vm_start > start)
+		goto out;
+
+	if (start > vma->vm_start)
+		prev = vma;
+	else
+		prev = vma->vm_prev;
+
+	ret = 0;
+	while (true) {
+		unsigned long newflags, tmp;
+
+		tmp = vma->vm_end;
+		if (tmp > end)
+			tmp = end;
+
+		newflags = vma->vm_flags;
+		if (protect)
+			newflags |= VM_KVM_PROTECTED;
+		else
+			newflags &= ~VM_KVM_PROTECTED;
+
+		/* The VMA has been handled as part of other memslot */
+		if (newflags == vma->vm_flags)
+			goto next;
+
+		ret = mprotect_fixup(vma, &prev, start, tmp, newflags);
+		if (ret)
+			goto out;
+
+next:
+		start = tmp;
+		if (start < prev->vm_end)
+			start = prev->vm_end;
+
+		if (start >= end)
+			goto out;
+
+		vma = prev->vm_next;

virt/lib/mem_protected.c line 71

Quoted 10 lines
virt/lib/mem_protected.c new file +10 -0
+		if (!vma || vma->vm_start != start) {
+			ret = -ENOMEM;
+			goto out;
+		}
+	}
+out:
+	mmap_write_unlock(mm);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(__kvm_protect_memory);

Since migration will be disabled after this; should the above not (at the very least) force compaction before proceeding to lock the pages in?

#32 lore raw

Re: [RFCv2 11/16] KVM: Protected memory extension

Quoted 60 lines · Peter Zijlstra
Quoted 59 lines · Kirill A. Shutemov
virt/lib/mem_protected.c new file +59 -0
+int __kvm_protect_memory(unsigned long start, unsigned long end, bool protect)
+{
+	struct mm_struct *mm = current->mm;
+	struct vm_area_struct *vma, *prev;
+	int ret;
+
+	if (mmap_write_lock_killable(mm))
+		return -EINTR;
+
+	ret = -ENOMEM;
+	vma = find_vma(current->mm, start);
+	if (!vma)
+		goto out;
+
+	ret = -EINVAL;
+	if (vma->vm_start > start)
+		goto out;
+
+	if (start > vma->vm_start)
+		prev = vma;
+	else
+		prev = vma->vm_prev;
+
+	ret = 0;
+	while (true) {
+		unsigned long newflags, tmp;
+
+		tmp = vma->vm_end;
+		if (tmp > end)
+			tmp = end;
+
+		newflags = vma->vm_flags;
+		if (protect)
+			newflags |= VM_KVM_PROTECTED;
+		else
+			newflags &= ~VM_KVM_PROTECTED;
+
+		/* The VMA has been handled as part of other memslot */
+		if (newflags == vma->vm_flags)
+			goto next;
+
+		ret = mprotect_fixup(vma, &prev, start, tmp, newflags);
+		if (ret)
+			goto out;
+
+next:
+		start = tmp;
+		if (start < prev->vm_end)
+			start = prev->vm_end;
+
+		if (start >= end)
+			goto out;
+
+		vma = prev->vm_next;
+		if (!vma || vma->vm_start != start) {
+			ret = -ENOMEM;
+			goto out;
+		}
+	}
Quoted 8 lines

virt/lib/mem_protected.c line 71

Quoted 5 lines
virt/lib/mem_protected.c new file +5 -0
+out:
+	mmap_write_unlock(mm);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(__kvm_protect_memory);

Since migration will be disabled after this; should the above not (at the very least) force compaction before proceeding to lock the pages in?

Migration has to be implemented instead, before we hit upstream.

BTW, VMs with direct device assignment pins all guest memory today. So it's not something new in the virtualization world.

Signature · 1 line
 Kirill A. Shutemov
#33 lore raw

RFC v2 12/16 KVM: x86: Enabled protected memory extension

In reply to Kirill A. Shutemov

Wire up hypercalls for the feature and define VM_KVM_PROTECTED.

---
 arch/x86/Kconfig     | 1 +
 arch/x86/kvm/Kconfig | 1 +
 arch/x86/kvm/cpuid.c | 3 ++-
 arch/x86/kvm/x86.c   | 9 +++++++++
 include/linux/mm.h   | 6 ++++++
 5 files changed, 19 insertions(+), 1 deletion(-)
arch/x86/Kconfig +1 -0
index b22b95517437..0bcbdadb97d6 100644
@@ -807,6 +807,7 @@ config KVM_GUEST
 	select X86_HV_CALLBACK_VECTOR
 	select X86_MEM_ENCRYPT_COMMON
 	select SWIOTLB
+	select ARCH_USES_HIGH_VMA_FLAGS
 	default y
 	help
 	  This option enables various optimizations for running under the KVM
arch/x86/kvm/Kconfig +1 -0
index fbd5bd7a945a..2ea77c2a8029 100644
@@ -46,6 +46,7 @@ config KVM
 	select KVM_GENERIC_DIRTYLOG_READ_PROTECT
 	select KVM_VFIO
 	select SRCU
+	select HAVE_KVM_PROTECTED_MEMORY
 	help
 	  Support hosting fully virtualized guest machines using hardware
 	  virtualization extensions.  You will need a fairly recent
arch/x86/kvm/cpuid.c +2 -1
index 3fd6eec202d7..eed33db032fb 100644
@@ -746,7 +746,8 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
 			     (1 << KVM_FEATURE_PV_SEND_IPI) |
 			     (1 << KVM_FEATURE_POLL_CONTROL) |
 			     (1 << KVM_FEATURE_PV_SCHED_YIELD) |
-			     (1 << KVM_FEATURE_ASYNC_PF_INT);
+			     (1 << KVM_FEATURE_ASYNC_PF_INT) |
+			     (1 << KVM_FEATURE_MEM_PROTECTED);
 
 		if (sched_info_on())
 			entry->eax |= (1 << KVM_FEATURE_STEAL_TIME);
arch/x86/kvm/x86.c +9 -0
index ce856e0ece84..e89ff39204f0 100644
@@ -7752,6 +7752,15 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
 		kvm_sched_yield(vcpu->kvm, a0);
 		ret = 0;
 		break;
+	case KVM_HC_ENABLE_MEM_PROTECTED:
+		ret = kvm_protect_all_memory(vcpu->kvm);
+		break;
+	case KVM_HC_MEM_SHARE:
+		ret = kvm_protect_memory(vcpu->kvm, a0, a1, false);
+		break;
+	case KVM_HC_MEM_UNSHARE:
+		ret = kvm_protect_memory(vcpu->kvm, a0, a1, true);
+		break;
 	default:
 		ret = -KVM_ENOSYS;
 		break;
include/linux/mm.h +6 -0
index c8d8cdcbc425..ee274d27e764 100644
@@ -304,11 +304,13 @@ extern unsigned int kobjsize(const void *objp);
 #define VM_HIGH_ARCH_BIT_2	34	/* bit only usable on 64-bit architectures */
 #define VM_HIGH_ARCH_BIT_3	35	/* bit only usable on 64-bit architectures */
 #define VM_HIGH_ARCH_BIT_4	36	/* bit only usable on 64-bit architectures */
+#define VM_HIGH_ARCH_BIT_5	37	/* bit only usable on 64-bit architectures */
 #define VM_HIGH_ARCH_0	BIT(VM_HIGH_ARCH_BIT_0)
 #define VM_HIGH_ARCH_1	BIT(VM_HIGH_ARCH_BIT_1)
 #define VM_HIGH_ARCH_2	BIT(VM_HIGH_ARCH_BIT_2)
 #define VM_HIGH_ARCH_3	BIT(VM_HIGH_ARCH_BIT_3)
 #define VM_HIGH_ARCH_4	BIT(VM_HIGH_ARCH_BIT_4)
+#define VM_HIGH_ARCH_5	BIT(VM_HIGH_ARCH_BIT_5)
 #endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */
 
 #ifdef CONFIG_ARCH_HAS_PKEYS
@@ -342,7 +344,11 @@ extern unsigned int kobjsize(const void *objp);
 # define VM_MAPPED_COPY	VM_ARCH_1	/* T if mapped copy of data (nommu mmap) */
 #endif
 
+#if defined(CONFIG_X86_64) && defined(CONFIG_KVM)
+#define VM_KVM_PROTECTED VM_HIGH_ARCH_5
+#else
 #define VM_KVM_PROTECTED 0
+#endif
 
 #ifndef VM_GROWSUP
 # define VM_GROWSUP	VM_NONE
Signature · 1 line
2.26.2
#34 lore raw

Re: [RFCv2 12/16] KVM: x86: Enabled protected memory extension

[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]

Hi "Kirill,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on tip/x86/core]
[also build test WARNING on v5.9]
[cannot apply to kvm/linux-next hnaz-linux-mm/master next-20201016] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 238c91115cd05c71447ea071624a4c9fe661f970
config: i386-allyesconfig (attached as .config)
compiler: gcc-9 (Debian 9.3.0-15) 9.3.0
reproduce (this is a W=1 build):
        # https://github.com/0day-ci/linux/commit/7f6be13aae00ba3b3bd2c6ed70add1acd854c563
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130
        git checkout 7f6be13aae00ba3b3bd2c6ed70add1acd854c563
        # save the attached .config to linux build tree
        make W=1 ARCH=i386 

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>):

>> virt/lib/mem_protected.c:8:5: warning: no previous prototype for '__kvm_protect_memory' [-Wmissing-prototypes]

8 | int __kvm_protect_memory(unsigned long start, unsigned long end, bool protect)
| ^~~~~~~~~~~~~~~~~~~~

vim +/__kvm_protect_memory +8 virt/lib/mem_protected.c

63d2c458104bc29 Kirill A. Shutemov 2020-10-20 7
63d2c458104bc29 Kirill A. Shutemov 2020-10-20 @8 int __kvm_protect_memory(unsigned long start, unsigned long end, bool protect)

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#35 lore raw

RFC v2 13/16 KVM: Rework copy_to/from_guest() to avoid direct mapping

In reply to Kirill A. Shutemov

We are going unmap guest pages from direct mapping and cannot rely on it for guest memory access. Use temporary kmap_atomic()-style mapping to access guest memory.

---
 virt/kvm/kvm_main.c      |  27 ++++++++++-
 virt/lib/mem_protected.c | 101 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 126 insertions(+), 2 deletions(-)
virt/kvm/kvm_main.c +25 -2
index 4c008c7b4974..9b569b78874a 100644
@@ -51,6 +51,7 @@
 #include <linux/io.h>
 #include <linux/lockdep.h>
 #include <linux/kthread.h>
+#include <linux/pagewalk.h>
 
 #include <asm/processor.h>
 #include <asm/ioctl.h>
@@ -154,6 +155,12 @@ static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm);
 static unsigned long long kvm_createvm_count;
 static unsigned long long kvm_active_vms;
 
+void *kvm_map_page_atomic(struct page *page);
+void kvm_unmap_page_atomic(void *vaddr);
+
+int kvm_init_protected_memory(void);
+void kvm_exit_protected_memory(void);
+
 int __kvm_protect_memory(unsigned long start, unsigned long end, bool protect);
 
 __weak void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm,
@@ -2329,6 +2336,7 @@ int copy_from_guest(void *data, unsigned long hva, int len, bool protected)
 	int offset = offset_in_page(hva);
 	struct page *page;
 	int npages, seg;
+	void *vaddr;
 
 	if (!protected)
 		return __copy_from_user(data, (void __user *)hva, len);
@@ -2341,7 +2349,11 @@ int copy_from_guest(void *data, unsigned long hva, int len, bool protected)
 		npages = get_user_pages_unlocked(hva, 1, &page, FOLL_KVM);
 		if (npages != 1)
 			return -EFAULT;
-		memcpy(data, page_address(page) + offset, seg);
+
+		vaddr = kvm_map_page_atomic(page);
+		memcpy(data, vaddr + offset, seg);
+		kvm_unmap_page_atomic(vaddr);
+
 		put_page(page);
 		len -= seg;
 		hva += seg;
@@ -2356,6 +2368,7 @@ int copy_to_guest(unsigned long hva, const void *data, int len, bool protected)
 	int offset = offset_in_page(hva);
 	struct page *page;
 	int npages, seg;
+	void *vaddr;
 
 	if (!protected)
 		return __copy_to_user((void __user *)hva, data, len);
@@ -2369,7 +2382,11 @@ int copy_to_guest(unsigned long hva, const void *data, int len, bool protected)
 						 FOLL_WRITE | FOLL_KVM);
 		if (npages != 1)
 			return -EFAULT;
-		memcpy(page_address(page) + offset, data, seg);
+
+		vaddr = kvm_map_page_atomic(page);
+		memcpy(vaddr + offset, data, seg);
+		kvm_unmap_page_atomic(vaddr);
+
 		put_page(page);
 		len -= seg;
 		hva += seg;
@@ -4945,6 +4962,10 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
 	if (r)
 		goto out_free;
 
+	if (IS_ENABLED(CONFIG_HAVE_KVM_PROTECTED_MEMORY) &&
+	    kvm_init_protected_memory())
+		goto out_unreg;
+
 	kvm_chardev_ops.owner = module;
 	kvm_vm_fops.owner = module;
 	kvm_vcpu_fops.owner = module;
@@ -4968,6 +4989,7 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
 	return 0;
 
 out_unreg:
+	kvm_exit_protected_memory();
 	kvm_async_pf_deinit();
 out_free:
 	kmem_cache_destroy(kvm_vcpu_cache);
@@ -4989,6 +5011,7 @@ EXPORT_SYMBOL_GPL(kvm_init);
 
 void kvm_exit(void)
 {
+	kvm_exit_protected_memory();
 	debugfs_remove_recursive(kvm_debugfs_dir);
 	misc_deregister(&kvm_dev);
 	kmem_cache_destroy(kvm_vcpu_cache);
virt/lib/mem_protected.c +101 -0

113 lines not on this page · raw message

Signature · 1 line
2.26.2
#36 lore raw

Re: [RFCv2 13/16] KVM: Rework copy_to/from_guest() to avoid direct mapping

[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]

Hi "Kirill,

[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/x86/core]
[also build test ERROR on v5.9]
[cannot apply to kvm/linux-next hnaz-linux-mm/master next-20201016] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 238c91115cd05c71447ea071624a4c9fe661f970
config: i386-allyesconfig (attached as .config)
compiler: gcc-9 (Debian 9.3.0-15) 9.3.0
reproduce (this is a W=1 build):
        # https://github.com/0day-ci/linux/commit/1af02c27a796b91d58cd8b6d50d7e7b9b5564ce4
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130
        git checkout 1af02c27a796b91d58cd8b6d50d7e7b9b5564ce4
        # save the attached .config to linux build tree
        make W=1 ARCH=i386 

If you fix the issue, kindly add following tag as appropriate

All error/warnings (new ones prefixed by >>):

>> virt/lib/mem_protected.c:11:7: warning: no previous prototype for 'kvm_map_page_atomic' [-Wmissing-prototypes]
      11 | void *kvm_map_page_atomic(struct page *page)
         |       ^~~~~~~~~~~~~~~~~~~
>> virt/lib/mem_protected.c:24:6: warning: no previous prototype for 'kvm_unmap_page_atomic' [-Wmissing-prototypes]
      24 | void kvm_unmap_page_atomic(void *vaddr)
         |      ^~~~~~~~~~~~~~~~~~~~~
>> virt/lib/mem_protected.c:33:5: warning: no previous prototype for 'kvm_init_protected_memory' [-Wmissing-prototypes]
      33 | int kvm_init_protected_memory(void)
         |     ^~~~~~~~~~~~~~~~~~~~~~~~~
>> virt/lib/mem_protected.c:51:6: warning: no previous prototype for 'kvm_exit_protected_memory' [-Wmissing-prototypes]
      51 | void kvm_exit_protected_memory(void)
         |      ^~~~~~~~~~~~~~~~~~~~~~~~~
   In file included from include/linux/mm.h:33,
                    from include/linux/kvm_host.h:14,
                    from virt/lib/mem_protected.c:1:
   virt/lib/mem_protected.c: In function 'adjust_direct_mapping_pte_range':
>> include/linux/pgtable.h:79:12: error: implicit declaration of function 'kmap_atomic'; did you mean 'in_atomic'? [-Werror=implicit-function-declaration]
      79 |  ((pte_t *)kmap_atomic(pmd_page(*(dir))) +  \
         |            ^~~~~~~~~~~
   virt/lib/mem_protected.c:79:8: note: in expansion of macro 'pte_offset_map'
      79 |  pte = pte_offset_map(pmd, addr);
         |        ^~~~~~~~~~~~~~

virt/lib/mem_protected.c: At top level:
virt/lib/mem_protected.c:102:5: warning: no previous prototype for '__kvm_protect_memory' [-Wmissing-prototypes]
102 | int __kvm_protect_memory(unsigned long start, unsigned long end, bool protect)
| ^~~~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors

vim +/kvm_map_page_atomic +11 virt/lib/mem_protected.c

    10	
  > 11	void *kvm_map_page_atomic(struct page *page)
    12	{
    13		pte_t *pte;
    14		void *vaddr;
    15	
    16		preempt_disable();
    17		pte = guest_map_ptes[smp_processor_id()];
    18		vaddr = guest_map_area->addr + smp_processor_id() * PAGE_SIZE;
    19		set_pte(pte, mk_pte(page, PAGE_KERNEL));
    20		return vaddr;
    21	}
    22	EXPORT_SYMBOL_GPL(kvm_map_page_atomic);
    23	
  > 24	void kvm_unmap_page_atomic(void *vaddr)
    25	{
    26		pte_t *pte = guest_map_ptes[smp_processor_id()];
    27		set_pte(pte, __pte(0));
    28		flush_tlb_one_kernel((unsigned long)vaddr);
    29		preempt_enable();
    30	}
    31	EXPORT_SYMBOL_GPL(kvm_unmap_page_atomic);
    32	
  > 33	int kvm_init_protected_memory(void)
    34	{
    35		guest_map_ptes = kmalloc_array(num_possible_cpus(),
    36					       sizeof(pte_t *), GFP_KERNEL);
    37		if (!guest_map_ptes)
    38			return -ENOMEM;
    39	
    40		guest_map_area = alloc_vm_area(PAGE_SIZE * num_possible_cpus(),
    41					       guest_map_ptes);
    42		if (!guest_map_ptes) {
    43			kfree(guest_map_ptes);
    44			return -ENOMEM;
    45		}
    46	
    47		return 0;
    48	}
    49	EXPORT_SYMBOL_GPL(kvm_init_protected_memory);
    50	
  > 51	void kvm_exit_protected_memory(void)
    52	{
    53		if (guest_map_area)
    54			free_vm_area(guest_map_area);
    55		if (guest_map_ptes)
    56			kfree(guest_map_ptes);
    57	}
    58	EXPORT_SYMBOL_GPL(kvm_exit_protected_memory);
    59	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#37 lore raw

RFC v2 14/16 KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()

In reply to Kirill A. Shutemov

We cannot access protected pages directly. Use ioremap() to create a temporary mapping of the page. The mapping is destroyed on __kvm_unmap_gfn().

The new interface gfn_to_pfn_memslot_protected() is used to detect if the page is protected.

ioremap_cache_force() is a hack to bypass IORES_MAP_SYSTEM_RAM check in the x86 ioremap code. We need a better solution.

---
 arch/powerpc/kvm/book3s_64_mmu_hv.c    |  2 +-
 arch/powerpc/kvm/book3s_64_mmu_radix.c |  2 +-
 arch/x86/include/asm/io.h              |  2 +
 arch/x86/include/asm/pgtable_types.h   |  1 +
 arch/x86/kvm/mmu/mmu.c                 |  6 ++-
 arch/x86/mm/ioremap.c                  | 16 ++++++--
 include/linux/kvm_host.h               |  3 +-
 include/linux/kvm_types.h              |  1 +
 virt/kvm/kvm_main.c                    | 52 +++++++++++++++++++-------
 9 files changed, 63 insertions(+), 22 deletions(-)
arch/powerpc/kvm/book3s_64_mmu_hv.c +1 -1
index 38ea396a23d6..8e06cd3f759c 100644
@@ -590,7 +590,7 @@ int kvmppc_book3s_hv_page_fault(struct kvm_vcpu *vcpu,
 	} else {
 		/* Call KVM generic code to do the slow-path check */
 		pfn = __gfn_to_pfn_memslot(memslot, gfn, false, NULL,
-					   writing, &write_ok);
+					   writing, &write_ok, NULL);
 		if (is_error_noslot_pfn(pfn))
 			return -EFAULT;
 		page = NULL;
arch/powerpc/kvm/book3s_64_mmu_radix.c +1 -1
index 22a677b18695..6fd4e3f9b66a 100644
@@ -822,7 +822,7 @@ int kvmppc_book3s_instantiate_page(struct kvm_vcpu *vcpu,
 
 		/* Call KVM generic code to do the slow-path check */
 		pfn = __gfn_to_pfn_memslot(memslot, gfn, false, NULL,
-					   writing, upgrade_p);
+					   writing, upgrade_p, NULL);
 		if (is_error_noslot_pfn(pfn))
 			return -EFAULT;
 		page = NULL;
arch/x86/include/asm/io.h +2 -0
index c58d52fd7bf2..a3e1bfad1026 100644
@@ -184,6 +184,8 @@ extern void __iomem *ioremap_uc(resource_size_t offset, unsigned long size);
 #define ioremap_uc ioremap_uc
 extern void __iomem *ioremap_cache(resource_size_t offset, unsigned long size);
 #define ioremap_cache ioremap_cache
+extern void __iomem *ioremap_cache_force(resource_size_t offset, unsigned long size);
+#define ioremap_cache_force ioremap_cache_force
 extern void __iomem *ioremap_prot(resource_size_t offset, unsigned long size, unsigned long prot_val);
 #define ioremap_prot ioremap_prot
 extern void __iomem *ioremap_encrypted(resource_size_t phys_addr, unsigned long size);
arch/x86/include/asm/pgtable_types.h +1 -0
index 816b31c68550..4c16a9583786 100644
@@ -147,6 +147,7 @@ enum page_cache_mode {
 	_PAGE_CACHE_MODE_UC       = 3,
 	_PAGE_CACHE_MODE_WT       = 4,
 	_PAGE_CACHE_MODE_WP       = 5,
+	_PAGE_CACHE_MODE_WB_FORCE = 6,
 
 	_PAGE_CACHE_MODE_NUM      = 8
 };
arch/x86/kvm/mmu/mmu.c +4 -2
index 71aa3da2a0b7..162cb285b87b 100644
@@ -4058,7 +4058,8 @@ static bool try_async_pf(struct kvm_vcpu *vcpu, bool prefault, gfn_t gfn,
 	}
 
 	async = false;
-	*pfn = __gfn_to_pfn_memslot(slot, gfn, false, &async, write, writable);
+	*pfn = __gfn_to_pfn_memslot(slot, gfn, false, &async, write, writable,
+				    NULL);
 	if (!async)
 		return false; /* *pfn has correct page already */
 
@@ -4072,7 +4073,8 @@ static bool try_async_pf(struct kvm_vcpu *vcpu, bool prefault, gfn_t gfn,
 			return true;
 	}
 
-	*pfn = __gfn_to_pfn_memslot(slot, gfn, false, NULL, write, writable);
+	*pfn = __gfn_to_pfn_memslot(slot, gfn, false, NULL, write, writable,
+				    NULL);
 	return false;
 }
 
arch/x86/mm/ioremap.c +13 -3
index 9e5ccc56f8e0..4409785e294c 100644
@@ -202,9 +202,12 @@ __ioremap_caller(resource_size_t phys_addr, unsigned long size,
 	__ioremap_check_mem(phys_addr, size, &io_desc);
 
 	/*
-	 * Don't allow anybody to remap normal RAM that we're using..
+	 * Don't allow anybody to remap normal RAM that we're using, unless
+	 * _PAGE_CACHE_MODE_WB_FORCE is used.
 	 */
-	if (io_desc.flags & IORES_MAP_SYSTEM_RAM) {
+	if (pcm == _PAGE_CACHE_MODE_WB_FORCE) {
+	    pcm = _PAGE_CACHE_MODE_WB;
+	} else if (io_desc.flags & IORES_MAP_SYSTEM_RAM) {
 		WARN_ONCE(1, "ioremap on RAM at %pa - %pa\n",
 			  &phys_addr, &last_addr);
 		return NULL;
@@ -419,6 +422,13 @@ void __iomem *ioremap_cache(resource_size_t phys_addr, unsigned long size)
 }
 EXPORT_SYMBOL(ioremap_cache);
 
+void __iomem *ioremap_cache_force(resource_size_t phys_addr, unsigned long size)
+{
+	return __ioremap_caller(phys_addr, size, _PAGE_CACHE_MODE_WB_FORCE,
+				__builtin_return_address(0), false);
+}
+EXPORT_SYMBOL(ioremap_cache_force);
+
 void __iomem *ioremap_prot(resource_size_t phys_addr, unsigned long size,
 				unsigned long prot_val)
 {
@@ -467,7 +477,7 @@ void iounmap(volatile void __iomem *addr)
 	p = find_vm_area((void __force *)addr);
 
 	if (!p) {
-		printk(KERN_ERR "iounmap: bad address %p\n", addr);
+		printk(KERN_ERR "iounmap: bad address %px\n", addr);
 		dump_stack();
 		return;
 	}
include/linux/kvm_host.h +2 -1
index 6655e8da4555..0d5f3885747b 100644
@@ -238,6 +238,7 @@ struct kvm_host_map {
 	void *hva;
 	kvm_pfn_t pfn;
 	kvm_pfn_t gfn;
+	bool protected;
 };
 
 /*
@@ -725,7 +726,7 @@ kvm_pfn_t gfn_to_pfn_memslot(struct kvm_memory_slot *slot, gfn_t gfn);
 kvm_pfn_t gfn_to_pfn_memslot_atomic(struct kvm_memory_slot *slot, gfn_t gfn);
 kvm_pfn_t __gfn_to_pfn_memslot(struct kvm_memory_slot *slot, gfn_t gfn,
 			       bool atomic, bool *async, bool write_fault,
-			       bool *writable);
+			       bool *writable, bool *protected);
 
 void kvm_release_pfn_clean(kvm_pfn_t pfn);
 void kvm_release_pfn_dirty(kvm_pfn_t pfn);
include/linux/kvm_types.h +1 -0
index a7580f69dda0..0a8c6426b4f4 100644
@@ -58,6 +58,7 @@ struct gfn_to_pfn_cache {
 	gfn_t gfn;
 	kvm_pfn_t pfn;
 	bool dirty;
+	bool protected;
 };
 
 #ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE
virt/kvm/kvm_main.c +38 -14

144 lines not on this page · raw message

Signature · 1 line
2.26.2
#38 lore raw

Re: [RFCv2 14/16] KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()

[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]

Hi "Kirill,

[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/x86/core]
[also build test ERROR on v5.9]
[cannot apply to kvm/linux-next hnaz-linux-mm/master next-20201016] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 238c91115cd05c71447ea071624a4c9fe661f970
config: s390-randconfig-r011-20201020 (attached as .config)
compiler: s390-linux-gcc (GCC) 9.3.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/47745b863bc9c5221c7b7f10bb0e1456a7c1416a
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130 git checkout 47745b863bc9c5221c7b7f10bb0e1456a7c1416a
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=s390

If you fix the issue, kindly add following tag as appropriate

All error/warnings (new ones prefixed by >>):

arch/s390/kvm/../../../virt/kvm/kvm_main.c: In function '__kvm_map_gfn':

>> arch/s390/kvm/../../../virt/kvm/kvm_main.c:2193:9: error: implicit declaration of function 'ioremap_cache_force' [-Werror=implicit-function-declaration]
    2193 |   hva = ioremap_cache_force(pfn_to_hpa(pfn), PAGE_SIZE);
         |         ^~~~~~~~~~~~~~~~~~~
>> arch/s390/kvm/../../../virt/kvm/kvm_main.c:2193:7: warning: assignment to 'void *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
    2193 |   hva = ioremap_cache_force(pfn_to_hpa(pfn), PAGE_SIZE);
         |       ^

cc1: some warnings being treated as errors

vim +/ioremap_cache_force +2193 arch/s390/kvm/../../../virt/kvm/kvm_main.c

2157
2158 static int __kvm_map_gfn(struct kvm_memslots *slots, gfn_t gfn,
2159 struct kvm_host_map *map,
2160 struct gfn_to_pfn_cache *cache,
2161 bool atomic)
2162 {
2163 kvm_pfn_t pfn;
2164 void *hva = NULL;
2165 struct page *page = KVM_UNMAPPED_PAGE;
2166 struct kvm_memory_slot *slot = __gfn_to_memslot(slots, gfn);
2167 bool protected;
2168 u64 gen = slots->generation;
2169
2170 if (!map)
2171 return -EINVAL;
2172
2173 if (cache) {
2174 if (!cache->pfn || cache->gfn != gfn ||
2175 cache->generation != gen) {
2176 if (atomic)
2177 return -EAGAIN;
2178 kvm_cache_gfn_to_pfn(slot, gfn, cache, gen);
2179 }
2180 pfn = cache->pfn;
2181 protected = cache->protected;
2182 } else {
2183 if (atomic)
2184 return -EAGAIN;
2185 pfn = gfn_to_pfn_memslot_protected(slot, gfn, &protected);
2186 }
2187 if (is_error_noslot_pfn(pfn))
2188 return -EINVAL;
2189
2190 if (protected) {
2191 if (atomic)
2192 return -EAGAIN;

> 2193			hva = ioremap_cache_force(pfn_to_hpa(pfn), PAGE_SIZE);

2194 } else if (pfn_valid(pfn)) {
2195 page = pfn_to_page(pfn);
2196 if (atomic)
2197 hva = kmap_atomic(page);
2198 else
2199 hva = kmap(page);
2200 #ifdef CONFIG_HAS_IOMEM
2201 } else if (!atomic) {
2202 hva = memremap(pfn_to_hpa(pfn), PAGE_SIZE, MEMREMAP_WB);
2203 } else {
2204 return -EINVAL;
2205 #endif
2206 }
2207
2208 if (!hva)
2209 return -EFAULT;
2210
2211 map->page = page;
2212 map->hva = hva;
2213 map->pfn = pfn;
2214 map->gfn = gfn;
2215 map->protected = protected;
2216
2217 return 0;
2218 }
2219

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#39 lore raw

Re: [RFCv2 14/16] KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()

In reply to Kirill A. Shutemov
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]

Hi "Kirill,

[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/x86/core]
[also build test ERROR on v5.9]
[cannot apply to kvm/linux-next hnaz-linux-mm/master next-20201016] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 238c91115cd05c71447ea071624a4c9fe661f970
config: mips-randconfig-r031-20201020 (attached as .config)
compiler: mipsel-linux-gcc (GCC) 9.3.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/47745b863bc9c5221c7b7f10bb0e1456a7c1416a
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130 git checkout 47745b863bc9c5221c7b7f10bb0e1456a7c1416a
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=mips

If you fix the issue, kindly add following tag as appropriate

All error/warnings (new ones prefixed by >>):

arch/mips/kvm/../../../virt/kvm/kvm_main.c: In function '__kvm_map_gfn':

>> arch/mips/kvm/../../../virt/kvm/kvm_main.c:2193:9: error: implicit declaration of function 'ioremap_cache_force'; did you mean 'ioremap_cache'? [-Werror=implicit-function-declaration]
    2193 |   hva = ioremap_cache_force(pfn_to_hpa(pfn), PAGE_SIZE);
         |         ^~~~~~~~~~~~~~~~~~~
         |         ioremap_cache
>> arch/mips/kvm/../../../virt/kvm/kvm_main.c:2193:7: warning: assignment to 'void *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
    2193 |   hva = ioremap_cache_force(pfn_to_hpa(pfn), PAGE_SIZE);
         |       ^

cc1: some warnings being treated as errors

vim +2193 arch/mips/kvm/../../../virt/kvm/kvm_main.c

2157
2158 static int __kvm_map_gfn(struct kvm_memslots *slots, gfn_t gfn,
2159 struct kvm_host_map *map,
2160 struct gfn_to_pfn_cache *cache,
2161 bool atomic)
2162 {
2163 kvm_pfn_t pfn;
2164 void *hva = NULL;
2165 struct page *page = KVM_UNMAPPED_PAGE;
2166 struct kvm_memory_slot *slot = __gfn_to_memslot(slots, gfn);
2167 bool protected;
2168 u64 gen = slots->generation;
2169
2170 if (!map)
2171 return -EINVAL;
2172
2173 if (cache) {
2174 if (!cache->pfn || cache->gfn != gfn ||
2175 cache->generation != gen) {
2176 if (atomic)
2177 return -EAGAIN;
2178 kvm_cache_gfn_to_pfn(slot, gfn, cache, gen);
2179 }
2180 pfn = cache->pfn;
2181 protected = cache->protected;
2182 } else {
2183 if (atomic)
2184 return -EAGAIN;
2185 pfn = gfn_to_pfn_memslot_protected(slot, gfn, &protected);
2186 }
2187 if (is_error_noslot_pfn(pfn))
2188 return -EINVAL;
2189
2190 if (protected) {
2191 if (atomic)
2192 return -EAGAIN;

> 2193			hva = ioremap_cache_force(pfn_to_hpa(pfn), PAGE_SIZE);

2194 } else if (pfn_valid(pfn)) {
2195 page = pfn_to_page(pfn);
2196 if (atomic)
2197 hva = kmap_atomic(page);
2198 else
2199 hva = kmap(page);
2200 #ifdef CONFIG_HAS_IOMEM
2201 } else if (!atomic) {
2202 hva = memremap(pfn_to_hpa(pfn), PAGE_SIZE, MEMREMAP_WB);
2203 } else {
2204 return -EINVAL;
2205 #endif
2206 }
2207
2208 if (!hva)
2209 return -EFAULT;
2210
2211 map->page = page;
2212 map->hva = hva;
2213 map->pfn = pfn;
2214 map->gfn = gfn;
2215 map->protected = protected;
2216
2217 return 0;
2218 }
2219

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#40 lore raw

Re: [RFCv2 14/16] KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()

In reply to Kirill A. Shutemov
Quoted 156 lines · Kirill A. Shutemov

We cannot access protected pages directly. Use ioremap() to create a temporary mapping of the page. The mapping is destroyed on __kvm_unmap_gfn().

The new interface gfn_to_pfn_memslot_protected() is used to detect if the page is protected.

ioremap_cache_force() is a hack to bypass IORES_MAP_SYSTEM_RAM check in
the x86 ioremap code. We need a better solution.

---
 arch/powerpc/kvm/book3s_64_mmu_hv.c    |  2 +-
 arch/powerpc/kvm/book3s_64_mmu_radix.c |  2 +-
 arch/x86/include/asm/io.h              |  2 +
 arch/x86/include/asm/pgtable_types.h   |  1 +
 arch/x86/kvm/mmu/mmu.c                 |  6 ++-
 arch/x86/mm/ioremap.c                  | 16 ++++++--
 include/linux/kvm_host.h               |  3 +-
 include/linux/kvm_types.h              |  1 +
 virt/kvm/kvm_main.c                    | 52 +++++++++++++++++++-----
--
 9 files changed, 63 insertions(+), 22 deletions(-)
arch/powerpc/kvm/book3s_64_mmu_hv.c +1 -1
index 38ea396a23d6..8e06cd3f759c 100644
@@ -590,7 +590,7 @@ int kvmppc_book3s_hv_page_fault(struct kvm_vcpu
 *vcpu,
 	} else {
 		/* Call KVM generic code to do the slow-path check */
 		pfn = __gfn_to_pfn_memslot(memslot, gfn, false, NULL,
-					   writing, &write_ok);
+					   writing, &write_ok, NULL);
 		if (is_error_noslot_pfn(pfn))
 			return -EFAULT;
arch/powerpc/kvm/book3s_64_mmu_hv.c +0 -0
 		page = NULL;
arch/powerpc/kvm/book3s_64_mmu_radix.c +1 -1
index 22a677b18695..6fd4e3f9b66a 100644
@@ -822,7 +822,7 @@ int kvmppc_book3s_instantiate_page(struct
 kvm_vcpu *vcpu,
 
 		/* Call KVM generic code to do the slow-path check */
 		pfn = __gfn_to_pfn_memslot(memslot, gfn, false, NULL,
-					   writing, upgrade_p);
+					   writing, upgrade_p, NULL);
 		if (is_error_noslot_pfn(pfn))
 			return -EFAULT;
arch/powerpc/kvm/book3s_64_mmu_radix.c +0 -0
 		page = NULL;
arch/x86/include/asm/io.h +2 -0
index c58d52fd7bf2..a3e1bfad1026 100644
@@ -184,6 +184,8 @@ extern void __iomem *ioremap_uc(resource_size_t
 offset, unsigned long size);
 #define ioremap_uc ioremap_uc
 extern void __iomem *ioremap_cache(resource_size_t offset, unsigned
 long size);
 #define ioremap_cache ioremap_cache
+extern void __iomem *ioremap_cache_force(resource_size_t offset,
 unsigned long size);
+#define ioremap_cache_force ioremap_cache_force

extern void __iomem *ioremap_prot(resource_size_t offset, unsigned
long size, unsigned long prot_val);
#define ioremap_prot ioremap_prot
extern void __iomem *ioremap_encrypted(resource_size_t phys_addr,
unsigned long size);

arch/x86/include/asm/pgtable_types.h +1 -0
index 816b31c68550..4c16a9583786 100644
@@ -147,6 +147,7 @@ enum page_cache_mode {
 	_PAGE_CACHE_MODE_UC       = 3,
 	_PAGE_CACHE_MODE_WT       = 4,
 	_PAGE_CACHE_MODE_WP       = 5,
+	_PAGE_CACHE_MODE_WB_FORCE = 6,
 
 	_PAGE_CACHE_MODE_NUM      = 8
 };
arch/x86/kvm/mmu/mmu.c +2 -1
index 71aa3da2a0b7..162cb285b87b 100644
@@ -4058,7 +4058,8 @@ static bool try_async_pf(struct kvm_vcpu *vcpu,
 bool prefault, gfn_t gfn,
 	}
 
 	async = false;
-	*pfn = __gfn_to_pfn_memslot(slot, gfn, false, &async, write,
 writable);
+	*pfn = __gfn_to_pfn_memslot(slot, gfn, false, &async, write,
 writable,
+				    NULL);
arch/x86/kvm/mmu/mmu.c +2 -1
 	if (!async)
 		return false; /* *pfn has correct page already */
@@ -4072,7 +4073,8 @@ static bool try_async_pf(struct kvm_vcpu *vcpu,
 bool prefault, gfn_t gfn,
 			return true;
 	}
 
-	*pfn = __gfn_to_pfn_memslot(slot, gfn, false, NULL, write,
 writable);
+	*pfn = __gfn_to_pfn_memslot(slot, gfn, false, NULL, write,
 writable,
+				    NULL);
arch/x86/kvm/mmu/mmu.c +0 -0
 	return false;
 }
arch/x86/mm/ioremap.c +1 -1
index 9e5ccc56f8e0..4409785e294c 100644
@@ -202,9 +202,12 @@ __ioremap_caller(resource_size_t phys_addr,
 unsigned long size,
 	__ioremap_check_mem(phys_addr, size, &io_desc);
 
 	/*
-	 * Don't allow anybody to remap normal RAM that we're using..
+	 * Don't allow anybody to remap normal RAM that we're using,

unless

arch/x86/mm/ioremap.c +11 -1
+	 * _PAGE_CACHE_MODE_WB_FORCE is used.
 	 */
-	if (io_desc.flags & IORES_MAP_SYSTEM_RAM) {
+	if (pcm == _PAGE_CACHE_MODE_WB_FORCE) {
+	    pcm = _PAGE_CACHE_MODE_WB;
+	} else if (io_desc.flags & IORES_MAP_SYSTEM_RAM) {
 		WARN_ONCE(1, "ioremap on RAM at %pa - %pa\n",
 			  &phys_addr, &last_addr);
 		return NULL;
@@ -419,6 +422,13 @@ void __iomem *ioremap_cache(resource_size_t
 phys_addr, unsigned long size)
 }
 EXPORT_SYMBOL(ioremap_cache);
 
+void __iomem *ioremap_cache_force(resource_size_t phys_addr,
 unsigned long size)
+{
+	return __ioremap_caller(phys_addr, size,
 _PAGE_CACHE_MODE_WB_FORCE,
+				__builtin_return_address(0), false);
+}
+EXPORT_SYMBOL(ioremap_cache_force);
+

void __iomem *ioremap_prot(resource_size_t phys_addr, unsigned long
size,

arch/x86/mm/ioremap.c +0 -0
 				unsigned long prot_val)
 {

arch/x86/mm/ioremap.c void iounmap(volatile void __iomem *addr) · line 480

Quoted 5 lines
arch/x86/mm/ioremap.c +1 -1
@@ -467,7 +477,7 @@ void iounmap(volatile void __iomem *addr)
 	p = find_vm_area((void __force *)addr);
 
 	if (!p) {
-		printk(KERN_ERR "iounmap: bad address %p\n", addr);
+		printk(KERN_ERR "iounmap: bad address %px\n", addr);

Unintentional?

Quoted 208 lines
arch/x86/mm/ioremap.c +0 -0
 		dump_stack();
 		return;
 	}
include/linux/kvm_host.h +2 -1
index 6655e8da4555..0d5f3885747b 100644
@@ -238,6 +238,7 @@ struct kvm_host_map {
 	void *hva;
 	kvm_pfn_t pfn;
 	kvm_pfn_t gfn;
+	bool protected;
 };
 
 /*
@@ -725,7 +726,7 @@ kvm_pfn_t gfn_to_pfn_memslot(struct
 kvm_memory_slot *slot, gfn_t gfn);
 kvm_pfn_t gfn_to_pfn_memslot_atomic(struct kvm_memory_slot *slot,
 gfn_t gfn);
 kvm_pfn_t __gfn_to_pfn_memslot(struct kvm_memory_slot *slot, gfn_t
 gfn,
 			       bool atomic, bool *async, bool
 write_fault,
-			       bool *writable);
+			       bool *writable, bool *protected);
 
 void kvm_release_pfn_clean(kvm_pfn_t pfn);
 void kvm_release_pfn_dirty(kvm_pfn_t pfn);
include/linux/kvm_types.h +1 -0
index a7580f69dda0..0a8c6426b4f4 100644
@@ -58,6 +58,7 @@ struct gfn_to_pfn_cache {
 	gfn_t gfn;
 	kvm_pfn_t pfn;
 	bool dirty;
+	bool protected;
 };
 
 #ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE
virt/kvm/kvm_main.c +2 -1
index 9b569b78874a..7c2c764c28c5 100644
@@ -1852,9 +1852,10 @@ static bool hva_to_pfn_fast(unsigned long
 addr, bool write_fault,
  * 1 indicates success, -errno is returned if error is detected.
  */
 static int hva_to_pfn_slow(unsigned long addr, bool *async, bool
 write_fault,
-			   bool *writable, kvm_pfn_t *pfn)
+			   bool *writable, bool *protected, kvm_pfn_t
 *pfn)
 {
 	unsigned int flags = FOLL_HWPOISON | FOLL_KVM;
+	struct vm_area_struct *vma;
virt/kvm/kvm_main.c +8 -2
 	struct page *page;
 	int npages = 0;
@@ -1868,9 +1869,15 @@ static int hva_to_pfn_slow(unsigned long addr,
 bool *async, bool write_fault,
 	if (async)
 		flags |= FOLL_NOWAIT;
 
-	npages = get_user_pages_unlocked(addr, 1, &page, flags);
-	if (npages != 1)
+	mmap_read_lock(current->mm);
+	npages = get_user_pages(addr, 1, flags, &page, &vma);
+	if (npages != 1) {
+		mmap_read_unlock(current->mm);
 		return npages;
+	}
+	if (protected)
+		*protected = vma_is_kvm_protected(vma);
+	mmap_read_unlock(current->mm);
 
 	/* map read fault as writable if possible */
virt/kvm/kvm_main.c +1 -1
 	if (unlikely(!write_fault) && writable) {
@@ -1961,7 +1968,7 @@ static int hva_to_pfn_remapped(struct
 vm_area_struct *vma,
  *     whether the mapping is writable.
  */
 static kvm_pfn_t hva_to_pfn(unsigned long addr, bool atomic, bool
 *async,
-			bool write_fault, bool *writable)
+			bool write_fault, bool *writable, bool
 *protected)

{

virt/kvm/kvm_main.c +2 -1
 	struct vm_area_struct *vma;
 	kvm_pfn_t pfn = 0;
@@ -1976,7 +1983,8 @@ static kvm_pfn_t hva_to_pfn(unsigned long addr,
 bool atomic, bool *async,
 	if (atomic)
 		return KVM_PFN_ERR_FAULT;
 
-	npages = hva_to_pfn_slow(addr, async, write_fault, writable,
 &pfn);
+	npages = hva_to_pfn_slow(addr, async, write_fault, writable,
 protected,
+				 &pfn);
virt/kvm/kvm_main.c +1 -1
 	if (npages == 1)
 		return pfn;
@@ -2010,7 +2018,7 @@ static kvm_pfn_t hva_to_pfn(unsigned long addr,
 bool atomic, bool *async,
 
 kvm_pfn_t __gfn_to_pfn_memslot(struct kvm_memory_slot *slot, gfn_t
 gfn,
 			       bool atomic, bool *async, bool
 write_fault,
-			       bool *writable)
+			       bool *writable, bool *protected)

{
unsigned long addr = __gfn_to_hva_many(slot, gfn, NULL,
write_fault);

virt/kvm/kvm_main.c +5 -3
@@ -2033,7 +2041,7 @@ kvm_pfn_t __gfn_to_pfn_memslot(struct
 kvm_memory_slot *slot, gfn_t gfn,
 	}
 
 	return hva_to_pfn(addr, atomic, async, write_fault,
-			  writable);
+			  writable, protected);
 }
 EXPORT_SYMBOL_GPL(__gfn_to_pfn_memslot);
@@ -2041,19 +2049,26 @@ kvm_pfn_t gfn_to_pfn_prot(struct kvm *kvm,
 gfn_t gfn, bool write_fault,
 		      bool *writable)
 {
 	return __gfn_to_pfn_memslot(gfn_to_memslot(kvm, gfn), gfn,
 false, NULL,
-				    write_fault, writable);
+				    write_fault, writable, NULL);
 }
 EXPORT_SYMBOL_GPL(gfn_to_pfn_prot);
 
 kvm_pfn_t gfn_to_pfn_memslot(struct kvm_memory_slot *slot, gfn_t
 gfn)
 {
-	return __gfn_to_pfn_memslot(slot, gfn, false, NULL, true,
 NULL);
+	return __gfn_to_pfn_memslot(slot, gfn, false, NULL, true, NULL,
 NULL);
 }
 EXPORT_SYMBOL_GPL(gfn_to_pfn_memslot);
 
+static kvm_pfn_t gfn_to_pfn_memslot_protected(struct kvm_memory_slot
 *slot,
+					      gfn_t gfn, bool

*protected)

virt/kvm/kvm_main.c +5 -0
+{
+	return __gfn_to_pfn_memslot(slot, gfn, false, NULL, true, NULL,
+				    protected);
+}
+
 kvm_pfn_t gfn_to_pfn_memslot_atomic(struct kvm_memory_slot *slot,

gfn_t gfn)
{

virt/kvm/kvm_main.c +1 -1
-	return __gfn_to_pfn_memslot(slot, gfn, true, NULL, true, NULL);
+	return __gfn_to_pfn_memslot(slot, gfn, true, NULL, true, NULL,

NULL);
}
EXPORT_SYMBOL_GPL(gfn_to_pfn_memslot_atomic);

virt/kvm/kvm_main.c +1 -1
@@ -2134,7 +2149,7 @@ static void kvm_cache_gfn_to_pfn(struct
 kvm_memory_slot *slot, gfn_t gfn,
 {
 	kvm_release_pfn(cache->pfn, cache->dirty, cache);
 
-	cache->pfn = gfn_to_pfn_memslot(slot, gfn);
+	cache->pfn = gfn_to_pfn_memslot_protected(slot, gfn, &cache-
 >protected);
 	cache->gfn = gfn;
virt/kvm/kvm_main.c +1 -0
 	cache->dirty = false;
 	cache->generation = gen;
@@ -2149,6 +2164,7 @@ static int __kvm_map_gfn(struct kvm_memslots
 *slots, gfn_t gfn,
 	void *hva = NULL;
 	struct page *page = KVM_UNMAPPED_PAGE;
 	struct kvm_memory_slot *slot = __gfn_to_memslot(slots, gfn);
+	bool protected;
 	u64 gen = slots->generation;
 
virt/kvm/kvm_main.c +7 -2
 	if (!map)
@@ -2162,15 +2178,20 @@ static int __kvm_map_gfn(struct kvm_memslots
 *slots, gfn_t gfn,
 			kvm_cache_gfn_to_pfn(slot, gfn, cache, gen);
 		}
 		pfn = cache->pfn;
+		protected = cache->protected;
 	} else {
 		if (atomic)
 			return -EAGAIN;
-		pfn = gfn_to_pfn_memslot(slot, gfn);
+		pfn = gfn_to_pfn_memslot_protected(slot, gfn,
 &protected);
 	}
 	if (is_error_noslot_pfn(pfn))
 		return -EINVAL;
 
-	if (pfn_valid(pfn)) {
+	if (protected) {
+		if (atomic)
+			return -EAGAIN;
+		hva = ioremap_cache_force(pfn_to_hpa(pfn), PAGE_SIZE);
+	} else if (pfn_valid(pfn)) {
 		page = pfn_to_page(pfn);

virt/kvm/kvm_main.c

Quoted 2 lines
virt/kvm/kvm_main.c +0 -0
 		if (atomic)
 			hva = kmap_atomic(page);

I think the page could have got unmapped since the gup via the hypercall on another CPU. It could be an avenue for the guest to crash the host.

Quoted 21 lines
virt/kvm/kvm_main.c +1 -0
@@ -2191,6 +2212,7 @@ static int __kvm_map_gfn(struct kvm_memslots
 *slots, gfn_t gfn,
 	map->hva = hva;
 	map->pfn = pfn;
 	map->gfn = gfn;
+	map->protected = protected;
 
 	return 0;

}

virt/kvm/kvm_main.c +3 -1
@@ -2221,7 +2243,9 @@ static void __kvm_unmap_gfn(struct
 kvm_memory_slot *memslot,
 	if (!map->hva)
 		return;
 
-	if (map->page != KVM_UNMAPPED_PAGE) {
+	if (map->protected) {
+		iounmap(map->hva);
+	} else if (map->page != KVM_UNMAPPED_PAGE) {
 		if (atomic)
 			kunmap_atomic(map->hva);

else

#41 lore raw

Re: [RFCv2 14/16] KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()

Quoted 9 lines · Rick P

arch/x86/mm/ioremap.c void iounmap(volatile void __iomem *addr) · line 480

Quoted 6 lines · Kirill A. Shutemov
arch/x86/mm/ioremap.c +1 -1
@@ -467,7 +477,7 @@ void iounmap(volatile void __iomem *addr)
 	p = find_vm_area((void __force *)addr);
 
 	if (!p) {
-		printk(KERN_ERR "iounmap: bad address %p\n", addr);
+		printk(KERN_ERR "iounmap: bad address %px\n", addr);

Unintentional?

Yep. Will fix.

Quoted 16 lines
Quoted 16 lines
virt/kvm/kvm_main.c +2 -1
@@ -2162,15 +2178,20 @@ static int __kvm_map_gfn(struct kvm_memslots
 *slots, gfn_t gfn,
 			kvm_cache_gfn_to_pfn(slot, gfn, cache, gen);
 		}
 		pfn = cache->pfn;
+		protected = cache->protected;
 	} else {
 		if (atomic)
 			return -EAGAIN;
-		pfn = gfn_to_pfn_memslot(slot, gfn);
+		pfn = gfn_to_pfn_memslot_protected(slot, gfn,
 &protected);
 	}
 	if (is_error_noslot_pfn(pfn))
 		return -EINVAL;
 
Quoted 13 lines

Review

Quoted 9 lines
virt/kvm/kvm_main.c +5 -1
-	if (pfn_valid(pfn)) {
+	if (protected) {
+		if (atomic)
+			return -EAGAIN;
+		hva = ioremap_cache_force(pfn_to_hpa(pfn), PAGE_SIZE);
+	} else if (pfn_valid(pfn)) {
 		page = pfn_to_page(pfn);
(quoted diff) +0 -0
 		if (atomic)
 			hva = kmap_atomic(page);

I think the page could have got unmapped since the gup via the hypercall on another CPU. It could be an avenue for the guest to crash the host.

Hm.. I'm not sure I follow. Could you elaborate on what scenario you have in mind?

Signature · 1 line
 Kirill A. Shutemov
#42 lore raw

Re: [RFCv2 14/16] KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()

Quoted 8 lines · Kirill A. Shutemov
Quoted 4 lines

I think the page could have got unmapped since the gup via the hypercall on another CPU. It could be an avenue for the guest to crash
the host.

Hm.. I'm not sure I follow. Could you elaborate on what scenario you have
in mind?

Kind of similar scenario as the userspace triggered oops. My understanding is that the protected status was gathered along with the gup, but after the mm gets unlocked, nothing stops the page transitioning to unmapped(?). At which point kmap() from a previous gup with !protected, would go down the regular kmap() route and return an address to an unmapped page.

So the guest kernel could start with a page mapped as shared via the hypercall. Then trigger one of the PV MSR's that kmap() on CPU0. On CPU1, after the gup on CPU0, it could transitioned the page to private/unmapped via the hypercall. So the hva_to_pfn() would find !protected, but by the time the kmap() happened the page would have been unmapped. Am I missing something?

#43 lore raw

Re: [RFCv2 14/16] KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()

Quoted 17 lines · Rick P
Quoted 8 lines · Kirill A. Shutemov
Quoted 4 lines

I think the page could have got unmapped since the gup via the hypercall on another CPU. It could be an avenue for the guest to crash
the host.

Hm.. I'm not sure I follow. Could you elaborate on what scenario you have
in mind?

Kind of similar scenario as the userspace triggered oops. My understanding is that the protected status was gathered along with the gup, but after the mm gets unlocked, nothing stops the page transitioning to unmapped(?). At which point kmap() from a previous gup with !protected, would go down the regular kmap() route and return an address to an unmapped page.

Quoted 6 lines

So the guest kernel could start with a page mapped as shared via the hypercall. Then trigger one of the PV MSR's that kmap() on CPU0. On CPU1, after the gup on CPU0, it could transitioned the page to private/unmapped via the hypercall. So the hva_to_pfn() would find !protected, but by the time the kmap() happened the page would have been unmapped. Am I missing something?

We need to fail protection enabling if a page is pinned. That's the only option I see. But it might be pain to debug.

Signature · 1 line
 Kirill A. Shutemov
#44 lore raw

Re: [RFCv2 14/16] KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()

In reply to Kirill A. Shutemov
Quoted 11 lines · On Tue, 20 Oct 2020 09:18:57 +0300 "Kirill A. Shutemov" <[email protected]>…

We cannot access protected pages directly. Use ioremap() to create a temporary mapping of the page. The mapping is destroyed on __kvm_unmap_gfn().

The new interface gfn_to_pfn_memslot_protected() is used to detect if the page is protected.

ioremap_cache_force() is a hack to bypass IORES_MAP_SYSTEM_RAM check in the x86 ioremap code. We need a better solution.

Quoted 12 lines
---
 arch/powerpc/kvm/book3s_64_mmu_hv.c    |  2 +-
 arch/powerpc/kvm/book3s_64_mmu_radix.c |  2 +-
 arch/x86/include/asm/io.h              |  2 +
 arch/x86/include/asm/pgtable_types.h   |  1 +
 arch/x86/kvm/mmu/mmu.c                 |  6 ++-
 arch/x86/mm/ioremap.c                  | 16 ++++++--
 include/linux/kvm_host.h               |  3 +-
 include/linux/kvm_types.h              |  1 +
 virt/kvm/kvm_main.c                    | 52 +++++++++++++++++++-------
 9 files changed, 63 insertions(+), 22 deletions(-)

You declare ioremap_cache_force() arch/x86/include/asm/io.h in and define it in arch/x86/mm/ioremap.c which is architecture specific code, but use it in __kvm_map_gfn() in virt/kvm/kvm_main.c which is common code.

Thus your series breaks the build for the s390 architecture. Have you tried to (cross) compile for s390?

Regards,
Halil

#45 lore raw

Re: [RFCv2 14/16] KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()

Quoted 27 lines · Halil Pasic
Quoted 11 lines · On Tue, 20 Oct 2020 09:18:57 +0300 "Kirill A. Shutemov" <[email protected]>…

We cannot access protected pages directly. Use ioremap() to create a temporary mapping of the page. The mapping is destroyed on __kvm_unmap_gfn().

The new interface gfn_to_pfn_memslot_protected() is used to detect if the page is protected.

ioremap_cache_force() is a hack to bypass IORES_MAP_SYSTEM_RAM check in the x86 ioremap code. We need a better solution.

Quoted 12 lines
---
 arch/powerpc/kvm/book3s_64_mmu_hv.c    |  2 +-
 arch/powerpc/kvm/book3s_64_mmu_radix.c |  2 +-
 arch/x86/include/asm/io.h              |  2 +
 arch/x86/include/asm/pgtable_types.h   |  1 +
 arch/x86/kvm/mmu/mmu.c                 |  6 ++-
 arch/x86/mm/ioremap.c                  | 16 ++++++--
 include/linux/kvm_host.h               |  3 +-
 include/linux/kvm_types.h              |  1 +
 virt/kvm/kvm_main.c                    | 52 +++++++++++++++++++-------
 9 files changed, 63 insertions(+), 22 deletions(-)
Quoted 7 lines

You declare ioremap_cache_force() arch/x86/include/asm/io.h in and define it in arch/x86/mm/ioremap.c which is architecture specific code, but use it in __kvm_map_gfn() in virt/kvm/kvm_main.c which is common code.

Thus your series breaks the build for the s390 architecture. Have you tried to (cross) compile for s390?

Obviously not. I've got reports already from 0day and going to fix it.

Thanks.

Signature · 1 line
 Kirill A. Shutemov
#46 lore raw

RFC v2 15/16 KVM: Unmap protected pages from direct mapping

In reply to Kirill A. Shutemov

If the protected memory feature enabled, unmap guest memory from kernel's direct mappings.

Migration and KSM is disabled for protected memory as it would require a special treatment.

---
 include/linux/mm.h       |  3 +++
 mm/huge_memory.c         |  8 ++++++++
 mm/ksm.c                 |  2 ++
 mm/memory.c              | 12 ++++++++++++
 mm/rmap.c                |  4 ++++
 virt/lib/mem_protected.c | 21 +++++++++++++++++++++
 6 files changed, 50 insertions(+)
include/linux/mm.h +3 -0
index ee274d27e764..74efc51e63f0 100644
@@ -671,6 +671,9 @@ static inline bool vma_is_kvm_protected(struct vm_area_struct *vma)
 	return vma->vm_flags & VM_KVM_PROTECTED;
 }
 
+void kvm_map_page(struct page *page, int nr_pages);
+void kvm_unmap_page(struct page *page, int nr_pages);
+
 #ifdef CONFIG_SHMEM
 /*
  * The vma_is_shmem is not inline because it is used only by slow
mm/huge_memory.c +8 -0
index ec8cf9a40cfd..40974656cb43 100644
@@ -627,6 +627,10 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,
 		spin_unlock(vmf->ptl);
 		count_vm_event(THP_FAULT_ALLOC);
 		count_memcg_event_mm(vma->vm_mm, THP_FAULT_ALLOC);
+
+		/* Unmap page from direct mapping */
+		if (vma_is_kvm_protected(vma))
+			kvm_unmap_page(page, HPAGE_PMD_NR);
 	}
 
 	return 0;
@@ -1689,6 +1693,10 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 			page_remove_rmap(page, true);
 			VM_BUG_ON_PAGE(page_mapcount(page) < 0, page);
 			VM_BUG_ON_PAGE(!PageHead(page), page);
+
+			/* Map the page back to the direct mapping */
+			if (vma_is_kvm_protected(vma))
+				kvm_map_page(page, HPAGE_PMD_NR);
 		} else if (thp_migration_supported()) {
 			swp_entry_t entry;
 
mm/ksm.c +2 -0
index 9afccc36dbd2..c720e271448f 100644
@@ -528,6 +528,8 @@ static struct vm_area_struct *find_mergeable_vma(struct mm_struct *mm,
 		return NULL;
 	if (!(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma)
 		return NULL;
+	if (vma_is_kvm_protected(vma))
+		return NULL;
 	return vma;
 }
 
mm/memory.c +12 -0
index 2c9756b4e52f..e28bd5f902a7 100644
@@ -1245,6 +1245,11 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
 				    likely(!(vma->vm_flags & VM_SEQ_READ)))
 					mark_page_accessed(page);
 			}
+
+			/* Map the page back to the direct mapping */
+			if (vma_is_anonymous(vma) && vma_is_kvm_protected(vma))
+				kvm_map_page(page, 1);
+
 			rss[mm_counter(page)]--;
 			page_remove_rmap(page, false);
 			if (unlikely(page_mapcount(page) < 0))
@@ -3466,6 +3471,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 	struct page *page;
 	vm_fault_t ret = 0;
 	pte_t entry;
+	bool set = false;
 
 	/* File mapping without ->vm_ops ? */
 	if (vma->vm_flags & VM_SHARED)
@@ -3554,6 +3560,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 	inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
 	page_add_new_anon_rmap(page, vma, vmf->address, false);
 	lru_cache_add_inactive_or_unevictable(page, vma);
+	set = true;
 setpte:
 	set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
 
@@ -3561,6 +3568,11 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 	update_mmu_cache(vma, vmf->address, vmf->pte);
 unlock:
 	pte_unmap_unlock(vmf->pte, vmf->ptl);
+
+	/* Unmap page from direct mapping */
+	if (vma_is_kvm_protected(vma) && set)
+		kvm_unmap_page(page, 1);
+
 	return ret;
 release:
 	put_page(page);
mm/rmap.c +4 -0
index 9425260774a1..247548d6d24b 100644
@@ -1725,6 +1725,10 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
 
 static bool invalid_migration_vma(struct vm_area_struct *vma, void *arg)
 {
+	/* TODO */
+	if (vma_is_kvm_protected(vma))
+		return true;
+
 	return vma_is_temporary_stack(vma);
 }
 
virt/lib/mem_protected.c +21 -0
index 1dfe82534242..9d2ef99285e5 100644
@@ -30,6 +30,27 @@ void kvm_unmap_page_atomic(void *vaddr)
 }
 EXPORT_SYMBOL_GPL(kvm_unmap_page_atomic);
 
+void kvm_map_page(struct page *page, int nr_pages)
+{
+	int i;
+
+	/* Clear page before returning it to the direct mapping */
+	for (i = 0; i < nr_pages; i++) {
+		void *p = kvm_map_page_atomic(page + i);
+		memset(p, 0, PAGE_SIZE);
+		kvm_unmap_page_atomic(p);
+	}
+
+	kernel_map_pages(page, nr_pages, 1);
+}
+EXPORT_SYMBOL_GPL(kvm_map_page);
+
+void kvm_unmap_page(struct page *page, int nr_pages)
+{
+	kernel_map_pages(page, nr_pages, 0);
+}
+EXPORT_SYMBOL_GPL(kvm_unmap_page);
+
 int kvm_init_protected_memory(void)
 {
 	guest_map_ptes = kmalloc_array(num_possible_cpus(),
Signature · 1 line
2.26.2
maintainer
#47 lore raw

Re: [RFCv2 15/16] KVM: Unmap protected pages from direct mapping

Quoted 5 lines · Kirill A. Shutemov

If the protected memory feature enabled, unmap guest memory from kernel's direct mappings.

Migration and KSM is disabled for protected memory as it would require a special treatment.

How isn't disabling migration a problem? Are we going to back allocation by CMA instead?

#48 lore raw

Re: [RFCv2 15/16] KVM: Unmap protected pages from direct mapping

In reply to Kirill A. Shutemov
Quoted 2 lines · Kirill A. Shutemov

If the protected memory feature enabled, unmap guest memory from kernel's direct mappings.

Gah, ugly. I guess this also defeats compaction, swapping, ... oh gosh. As if all of the encrypted VM implementations didn't bring us enough ugliness already (SEV extensions also don't support reboots, but can at least kexec() IIRC).

Something similar is done with secretmem [1]. And people don't seem to like fragmenting the direct mapping (including me).

[1] https://lkml.kernel.org/r/[email protected]

Signature · 3 lines
Thanks,

David / dhildenb
#49 lore raw

Re: [RFCv2 15/16] KVM: Unmap protected pages from direct mapping

Quoted 9 lines · David Hildenbrand
Quoted 2 lines · Kirill A. Shutemov

If the protected memory feature enabled, unmap guest memory from kernel's direct mappings.

Gah, ugly. I guess this also defeats compaction, swapping, ... oh gosh. As if all of the encrypted VM implementations didn't bring us enough ugliness already (SEV extensions also don't support reboots, but can at least kexec() IIRC).

Quoted 5 lines

Something similar is done with secretmem [1]. And people don't seem to like fragmenting the direct mapping (including me).

[1] https://lkml.kernel.org/r/[email protected]

I just thought "hey, we might have to replace pud/pmd mappings by page tables when calling kernel_map_pages", this can fail with -ENOMEM, why isn't there proper error handling.

Then I dived into __kernel_map_pages() which states:

"The return value is ignored as the calls cannot fail. Large pages for identity mappings are not used at boot time and hence no memory allocations during large page split."

I am probably missing something important, but how is calling kernel_map_pages() safe *after* booting?! I know we use it for debug_pagealloc(), but using it in a production-ready feature feels completely irresponsible. What am I missing?

Signature · 3 lines
Thanks,

David / dhildenb
#50 lore raw

Re: [RFCv2 15/16] KVM: Unmap protected pages from direct mapping

Quoted 33 lines · David Hildenbrand
Quoted 12 lines · David Hildenbrand
Quoted 2 lines · Kirill A. Shutemov

If the protected memory feature enabled, unmap guest memory from kernel's direct mappings.

Gah, ugly. I guess this also defeats compaction, swapping, ... oh gosh.
As if all of the encrypted VM implementations didn't bring us enough
ugliness already (SEV extensions also don't support reboots, but can at
least kexec() IIRC).

Quoted 6 lines

Something similar is done with secretmem [1]. And people don't seem to
like fragmenting the direct mapping (including me).

[1] https://lkml.kernel.org/r/[email protected]

I just thought "hey, we might have to replace pud/pmd mappings by page
tables when calling kernel_map_pages", this can fail with -ENOMEM, why
isn't there proper error handling.

Then I dived into __kernel_map_pages() which states:

"The return value is ignored as the calls cannot fail. Large pages for
identity mappings are not used at boot time and hence no memory allocations during large page split."

Quoted 5 lines

I am probably missing something important, but how is calling kernel_map_pages() safe *after* booting?! I know we use it for debug_pagealloc(), but using it in a production-ready feature feels completely irresponsible. What am I missing?

My understanding was that DEBUG_PAGEALLOC maps the direct map at 4k post-boot as well so that kernel_map_pages() can't fail for it, and to avoid having to take locks for splits in interrupts. I think you are on to something though. That function is essentially a set_memory_() functions on x86 at least, and many callers do not check the error codes. Kees Cook has been pushing to fix this actually:
https://github.com/KSPP/linux/issues/7

As long as a page has been broken to 4k, it should be able to be re- mapped without failure (like debug page alloc relies on). If an initial call to restrict permissions needs and fails to break a page, its remap does not need to succeed either before freeing the page, since the page never got set with a lower permission. That's my understanding from x86's cpa at least. The problem becomes that the page silently doesn't have its expected protections during use. Unchecked set_memory_() caching property change failures might result in functional problems though.

So there is some background if you wanted it, but yea, I agree this feature should handle if the unmap failed. Probably return an error on the IOCTL and maybe the hypercall. kernel_map_pages() also doesn't do a shootdown though, so this direct map protection is more in the best effort category in its current state.

For unmapping causing direct map fragmentation. The two techniques floating around, besides breaking indiscriminately, seem to be:
1. Group and cache unmapped pages to minimize the damage (like secret mem)
2. Somehow repair them back to large pages when reset RW (as Kirill had tried earlier this year in another series)

I had imagined this usage would want something like that eventually, but neither helps with the other limitations you mentioned (migration, etc).

#51 lore raw

Re: [RFCv2 15/16] KVM: Unmap protected pages from direct mapping

In reply to David Hildenbrand
Quoted 8 lines · David Hildenbrand
Quoted 2 lines · Kirill A. Shutemov

If the protected memory feature enabled, unmap guest memory from kernel's direct mappings.

Gah, ugly. I guess this also defeats compaction, swapping, ... oh gosh. As if all of the encrypted VM implementations didn't bring us enough ugliness already (SEV extensions also don't support reboots, but can at least kexec() IIRC).

SEV does support reboot. SEV-ES using Qemu doesn't support reboot because of the way Qemu resets the vCPU state. If Qemu could relaunch the guest through the SEV APIs to reset the vCPU state, then a "reboot" would be possible.

SEV does support kexec, SEV-ES does not at the moment.

Thanks,
Tom

Quoted 6 lines

Something similar is done with secretmem [1]. And people don't seem to like fragmenting the direct mapping (including me).

[1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml.kernel.org%2Fr%2F20200924132904.1391-1-rppt%40kernel.org&amp;data=04%7C01%7Cthomas.lendacky%40amd.com%7Cb98a5033da37432131b508d874f25194%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637387931403890525%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=BzC%2FeIyOau7BORuUY%2BaiRzYZ%2BOAHANvBDcmV9hpkrts%3D&amp;reserved=0

#52 lore raw

Re: [RFCv2 15/16] KVM: Unmap protected pages from direct mapping

In reply to Kirill A. Shutemov
Quoted 7 lines · Kirill A. Shutemov

If the protected memory feature enabled, unmap guest memory from kernel's direct mappings.

Migration and KSM is disabled for protected memory as it would require a
special treatment.

So do we care about this scenario where a malicious userspace causes a kernel oops? I'm not sure if it's prevented somehow.

CPU0 (exercising other kernel functionality) CPU1
mark page shared
page = get_user_pages(!FOLL_KVM)
mark page private
kmap(page)
access unmapped page and oops

reviewer
#53 lore raw

Re: [RFCv2 15/16] KVM: Unmap protected pages from direct mapping

In reply to Kirill A. Shutemov
Quoted 21 lines · Kirill A. Shutemov

If the protected memory feature enabled, unmap guest memory from kernel's direct mappings.

Migration and KSM is disabled for protected memory as it would require a special treatment.

---
 include/linux/mm.h       |  3 +++
 mm/huge_memory.c         |  8 ++++++++
 mm/ksm.c                 |  2 ++
 mm/memory.c              | 12 ++++++++++++
 mm/rmap.c                |  4 ++++
 virt/lib/mem_protected.c | 21 +++++++++++++++++++++
 6 files changed, 50 insertions(+)
include/linux/mm.h +0 -0
index ee274d27e764..74efc51e63f0 100644

include/linux/mm.h static inline bool vma_is_kvm_protected(struct vm_area_struct *vma) · line 675

Quoted 5 lines
include/linux/mm.h +2 -0
@@ -671,6 +671,9 @@ static inline bool vma_is_kvm_protected(struct vm_area_struct *vma)
 	return vma->vm_flags & VM_KVM_PROTECTED;
 }
 
+void kvm_map_page(struct page *page, int nr_pages);
+void kvm_unmap_page(struct page *page, int nr_pages);

This still does not seem right ;-)

And I still think that map/unmap primitives shoud be a part of the generic mm rather than exported by KVM.

Quoted 138 lines
include/linux/mm.h +1 -0
+
 #ifdef CONFIG_SHMEM
 /*
  * The vma_is_shmem is not inline because it is used only by slow
mm/huge_memory.c +8 -0
index ec8cf9a40cfd..40974656cb43 100644
@@ -627,6 +627,10 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,
 		spin_unlock(vmf->ptl);
 		count_vm_event(THP_FAULT_ALLOC);
 		count_memcg_event_mm(vma->vm_mm, THP_FAULT_ALLOC);
+
+		/* Unmap page from direct mapping */
+		if (vma_is_kvm_protected(vma))
+			kvm_unmap_page(page, HPAGE_PMD_NR);
 	}
 
 	return 0;
@@ -1689,6 +1693,10 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 			page_remove_rmap(page, true);
 			VM_BUG_ON_PAGE(page_mapcount(page) < 0, page);
 			VM_BUG_ON_PAGE(!PageHead(page), page);
+
+			/* Map the page back to the direct mapping */
+			if (vma_is_kvm_protected(vma))
+				kvm_map_page(page, HPAGE_PMD_NR);
 		} else if (thp_migration_supported()) {
 			swp_entry_t entry;
 
mm/ksm.c +2 -0
index 9afccc36dbd2..c720e271448f 100644
@@ -528,6 +528,8 @@ static struct vm_area_struct *find_mergeable_vma(struct mm_struct *mm,
 		return NULL;
 	if (!(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma)
 		return NULL;
+	if (vma_is_kvm_protected(vma))
+		return NULL;
 	return vma;
 }
 
mm/memory.c +12 -0
index 2c9756b4e52f..e28bd5f902a7 100644
@@ -1245,6 +1245,11 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
 				    likely(!(vma->vm_flags & VM_SEQ_READ)))
 					mark_page_accessed(page);
 			}
+
+			/* Map the page back to the direct mapping */
+			if (vma_is_anonymous(vma) && vma_is_kvm_protected(vma))
+				kvm_map_page(page, 1);
+
 			rss[mm_counter(page)]--;
 			page_remove_rmap(page, false);
 			if (unlikely(page_mapcount(page) < 0))
@@ -3466,6 +3471,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 	struct page *page;
 	vm_fault_t ret = 0;
 	pte_t entry;
+	bool set = false;
 
 	/* File mapping without ->vm_ops ? */
 	if (vma->vm_flags & VM_SHARED)
@@ -3554,6 +3560,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 	inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
 	page_add_new_anon_rmap(page, vma, vmf->address, false);
 	lru_cache_add_inactive_or_unevictable(page, vma);
+	set = true;
 setpte:
 	set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
 
@@ -3561,6 +3568,11 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 	update_mmu_cache(vma, vmf->address, vmf->pte);
 unlock:
 	pte_unmap_unlock(vmf->pte, vmf->ptl);
+
+	/* Unmap page from direct mapping */
+	if (vma_is_kvm_protected(vma) && set)
+		kvm_unmap_page(page, 1);
+
 	return ret;
 release:
 	put_page(page);
mm/rmap.c +4 -0
index 9425260774a1..247548d6d24b 100644
@@ -1725,6 +1725,10 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
 
 static bool invalid_migration_vma(struct vm_area_struct *vma, void *arg)
 {
+	/* TODO */
+	if (vma_is_kvm_protected(vma))
+		return true;
+
 	return vma_is_temporary_stack(vma);
 }
 
virt/lib/mem_protected.c +21 -0
index 1dfe82534242..9d2ef99285e5 100644
@@ -30,6 +30,27 @@ void kvm_unmap_page_atomic(void *vaddr)
 }
 EXPORT_SYMBOL_GPL(kvm_unmap_page_atomic);
 
+void kvm_map_page(struct page *page, int nr_pages)
+{
+	int i;
+
+	/* Clear page before returning it to the direct mapping */
+	for (i = 0; i < nr_pages; i++) {
+		void *p = kvm_map_page_atomic(page + i);
+		memset(p, 0, PAGE_SIZE);
+		kvm_unmap_page_atomic(p);
+	}
+
+	kernel_map_pages(page, nr_pages, 1);
+}
+EXPORT_SYMBOL_GPL(kvm_map_page);
+
+void kvm_unmap_page(struct page *page, int nr_pages)
+{
+	kernel_map_pages(page, nr_pages, 0);
+}
+EXPORT_SYMBOL_GPL(kvm_unmap_page);
+
 int kvm_init_protected_memory(void)
 {
 	guest_map_ptes = kmalloc_array(num_possible_cpus(),
Signature · 1 line
2.26.2
Signature · 2 lines
Sincerely yours,
Mike.
#54 lore raw

Re: [RFCv2 15/16] KVM: Unmap protected pages from direct mapping

Quoted 25 lines · Mike Rapoport
Quoted 24 lines · Kirill A. Shutemov

If the protected memory feature enabled, unmap guest memory from kernel's direct mappings.

Migration and KSM is disabled for protected memory as it would require a special treatment.

---
 include/linux/mm.h       |  3 +++
 mm/huge_memory.c         |  8 ++++++++
 mm/ksm.c                 |  2 ++
 mm/memory.c              | 12 ++++++++++++
 mm/rmap.c                |  4 ++++
 virt/lib/mem_protected.c | 21 +++++++++++++++++++++
 6 files changed, 50 insertions(+)
include/linux/mm.h +0 -0
index ee274d27e764..74efc51e63f0 100644
@@ -671,6 +671,9 @@ static inline bool vma_is_kvm_protected(struct vm_area_struct *vma)
 	return vma->vm_flags & VM_KVM_PROTECTED;
 }
 
Quoted 7 lines

include/linux/mm.h static inline bool vma_is_kvm_protected(struct vm_area_struct *vma) · line 675

Quoted 2 lines
include/linux/mm.h +2 -0
+void kvm_map_page(struct page *page, int nr_pages);
+void kvm_unmap_page(struct page *page, int nr_pages);

This still does not seem right ;-)

And I still think that map/unmap primitives shoud be a part of the generic mm rather than exported by KVM.

Ya, and going a step further, I suspect it will be cleaner in the long run if the kernel does not automatically map or unmap when converting between private and shared/public memory. Conversions will be rare in a well behaved guest, so exiting to userspace and forcing userspace to do the unmap->map would not be a performance bottleneck. In theory, userspace could also maintain separate pools for private vs. public mappings, though I doubt any VMM will do that in practice.

#55 lore raw

RFC v2 16/16 mm: Do not use zero page for VM_KVM_PROTECTED VMAs

In reply to Kirill A. Shutemov

Presence of zero pages in the mapping would disclose content of the mapping. Don't use them if KVM memory protection is enabled.

---
 arch/s390/include/asm/pgtable.h | 2 +-
 include/linux/mm.h              | 4 ++--
 mm/huge_memory.c                | 3 +--
 mm/memory.c                     | 3 +--
 4 files changed, 5 insertions(+), 7 deletions(-)
arch/s390/include/asm/pgtable.h +1 -1
index b55561cc8786..72ca3b3f04cb 100644
@@ -543,7 +543,7 @@ static inline int mm_alloc_pgste(struct mm_struct *mm)
  * In the case that a guest uses storage keys
  * faults should no longer be backed by zero pages
  */
-#define mm_forbids_zeropage mm_has_pgste
+#define vma_forbids_zeropage(vma) mm_has_pgste(vma->vm_mm)
 static inline int mm_uses_skeys(struct mm_struct *mm)
 {
 #ifdef CONFIG_PGSTE
include/linux/mm.h +2 -2
index 74efc51e63f0..ee713b7c2819 100644
@@ -130,8 +130,8 @@ extern int mmap_rnd_compat_bits __read_mostly;
  * s390 does this to prevent multiplexing of hardware bits
  * related to the physical page in case of virtualization.
  */
-#ifndef mm_forbids_zeropage
-#define mm_forbids_zeropage(X)	(0)
+#ifndef vma_forbids_zeropage
+#define vma_forbids_zeropage(vma) vma_is_kvm_protected(vma)
 #endif
 
 /*
mm/huge_memory.c +1 -2
index 40974656cb43..383614b24c4f 100644
@@ -709,8 +709,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)
 		return VM_FAULT_OOM;
 	if (unlikely(khugepaged_enter(vma, vma->vm_flags)))
 		return VM_FAULT_OOM;
-	if (!(vmf->flags & FAULT_FLAG_WRITE) &&
-			!mm_forbids_zeropage(vma->vm_mm) &&
+	if (!(vmf->flags & FAULT_FLAG_WRITE) && !vma_forbids_zeropage(vma) &&
 			transparent_hugepage_use_zero_page()) {
 		pgtable_t pgtable;
 		struct page *zero_page;
mm/memory.c +1 -2
index e28bd5f902a7..9907ffe00490 100644
@@ -3495,8 +3495,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 		return 0;
 
 	/* Use the zero-page for reads */
-	if (!(vmf->flags & FAULT_FLAG_WRITE) &&
-			!mm_forbids_zeropage(vma->vm_mm)) {
+	if (!(vmf->flags & FAULT_FLAG_WRITE) && !vma_forbids_zeropage(vma)) {
 		entry = pte_mkspecial(pfn_pte(my_zero_pfn(vmf->address),
 						vma->vm_page_prot));
 		vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
Signature · 1 line
2.26.2
#56 lore raw
In reply to Kirill A. Shutemov
Quoted 25 lines · Kirill A. Shutemov

== Background / Problem ==

There are a number of hardware features (MKTME, SEV) which protect guest memory from some unauthorized host access. The patchset proposes a purely software feature that mitigates some of the same host-side read-only attacks.

== What does this set mitigate? ==

- Host kernel ”accidental” access to guest data (think speculation)

- Host kernel induced access to guest data (write(fd, &guest_data_ptr, len))

- Host userspace access to guest data (compromised qemu)

- Guest privilege escalation via compromised QEMU device emulation

== What does this set NOT mitigate? ==

- Full host kernel compromise. Kernel will just map the pages again.

- Hardware attacks

Quoted 7 lines

The second RFC revision addresses /most/ of the feedback.

I still didn't found a good solution to reboot and kexec. Unprotect all the memory on such operations defeat the goal of the feature. Clearing up most of the memory before unprotecting what is required for reboot (or kexec) is tedious and error-prone.
Maybe we should just declare them unsupported?

Making reboot unsupported is a hard sell. Could you please elaborate on why you think that "unprotect all" hypercall (or rather a single hypercall supporting both protecting/unprotecting) defeats the purpose of the feature?

(Leaving kexec aside for a while) Yes, it is not easy for a guest to clean up *all* its memory upon reboot, however:
- It may only clean up the most sensitive parts. This should probably be done even without this new feature and even on bare metal (think about next boot target being malicious).
- The attack window shrinks significantly. "Speculative" bugs require time to exploit and it will only remain open until it boots up again (few seconds).

Quoted 123 lines

== Series Overview ==

The hardware features protect guest data by encrypting it and then ensuring that only the right guest can decrypt it. This has the side-effect of making the kernel direct map and userspace mapping (QEMU et al) useless. But, this teaches us something very useful:
neither the kernel or userspace mappings are really necessary for normal guest operations.

Instead of using encryption, this series simply unmaps the memory. One advantage compared to allowing access to ciphertext is that it allows bad accesses to be caught instead of simply reading garbage.

Protection from physical attacks needs to be provided by some other means. On Intel platforms, (single-key) Total Memory Encryption (TME) provides mitigation against physical attacks, such as DIMM interposers sniffing memory bus traffic.

The patchset modifies both host and guest kernel. The guest OS must enable the feature via hypercall and mark any memory range that has to be shared with the host: DMA regions, bounce buffers, etc. SEV does this marking via a bit in the guest’s page table while this approach uses a hypercall.

For removing the userspace mapping, use a trick similar to what NUMA balancing does: convert memory that belongs to KVM memory slots to
PROT_NONE: all existing entries converted to PROT_NONE with mprotect() and the newly faulted in pages get PROT_NONE from the updated vm_page_prot. The new VMA flag -- VM_KVM_PROTECTED -- indicates that the pages in the VMA must be treated in a special way in the GUP and fault paths. The flag allows GUP to return the page even though it is mapped with PROT_NONE, but only if the new GUP flag -- FOLL_KVM -- is specified. Any userspace access to the memory would result in SIGBUS. Any GUP access without FOLL_KVM would result in -EFAULT.

Removing userspace mapping of the guest memory from QEMU process can help to address some guest privilege escalation attacks. Consider the case when unprivileged guest user exploits bug in a QEMU device emulation to gain access to data it cannot normally have access within the guest.

Any anonymous page faulted into the VM_KVM_PROTECTED VMA gets removed from the direct mapping with kernel_map_pages(). Note that kernel_map_pages() only flushes local TLB. I think it's a reasonable compromise between security and perfromance.

Zapping the PTE would bring the page back to the direct mapping after clearing. At least for now, we don't remove file-backed pages from the direct mapping. File-backed pages could be accessed via read/write syscalls. It adds complexity.

Occasionally, host kernel has to access guest memory that was not made shared by the guest. For instance, it happens for instruction emulation. Normally, it's done via copy_to/from_user() which would fail with -EFAULT now. We introduced a new pair of helpers: copy_to/from_guest(). The new helpers acquire the page via GUP, map it into kernel address space with kmap_atomic()-style mechanism and only then copy the data.

For some instruction emulation copying is not good enough: cmpxchg emulation has to have direct access to the guest memory. __kvm_map_gfn() is modified to accommodate the case.

The patchset is on top of v5.9

Kirill A. Shutemov (16):
x86/mm: Move force_dma_unencrypted() to common code x86/kvm: Introduce KVM memory protection feature
x86/kvm: Make DMA pages shared
x86/kvm: Use bounce buffers for KVM memory protection x86/kvm: Make VirtIO use DMA API in KVM guest x86/kvmclock: Share hvclock memory with the host x86/realmode: Share trampoline area if KVM memory protection enabled
KVM: Use GUP instead of copy_from/to_user() to access guest memory
KVM: mm: Introduce VM_KVM_PROTECTED
KVM: x86: Use GUP for page walk instead of __get_user()
KVM: Protected memory extension
KVM: x86: Enabled protected memory extension
KVM: Rework copy_to/from_guest() to avoid direct mapping
KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()
KVM: Unmap protected pages from direct mapping
mm: Do not use zero page for VM_KVM_PROTECTED VMAs

 arch/powerpc/kvm/book3s_64_mmu_hv.c    |   2 +-
 arch/powerpc/kvm/book3s_64_mmu_radix.c |   2 +-
 arch/s390/include/asm/pgtable.h        |   2 +-
 arch/x86/Kconfig                       |  11 +-
 arch/x86/include/asm/cpufeatures.h     |   1 +
 arch/x86/include/asm/io.h              |   6 +-
 arch/x86/include/asm/kvm_para.h        |   5 +
 arch/x86/include/asm/pgtable_types.h   |   1 +
 arch/x86/include/uapi/asm/kvm_para.h   |   3 +-
 arch/x86/kernel/kvm.c                  |  20 +++
 arch/x86/kernel/kvmclock.c             |   2 +-
 arch/x86/kernel/pci-swiotlb.c          |   3 +-
 arch/x86/kvm/Kconfig                   |   1 +
 arch/x86/kvm/cpuid.c                   |   3 +-
 arch/x86/kvm/mmu/mmu.c                 |   6 +-
 arch/x86/kvm/mmu/paging_tmpl.h         |  10 +-
 arch/x86/kvm/x86.c                     |   9 +
 arch/x86/mm/Makefile                   |   2 +
 arch/x86/mm/ioremap.c                  |  16 +-
 arch/x86/mm/mem_encrypt.c              |  51 ------
 arch/x86/mm/mem_encrypt_common.c       |  62 +++++++
 arch/x86/mm/pat/set_memory.c           |   7 +
 arch/x86/realmode/init.c               |   7 +-
 drivers/virtio/virtio_ring.c           |   4 +
 include/linux/kvm_host.h               |  11 +-
 include/linux/kvm_types.h              |   1 +
 include/linux/mm.h                     |  21 ++-
 include/uapi/linux/kvm_para.h          |   5 +-
 mm/gup.c                               |  20 ++-
 mm/huge_memory.c                       |  31 +++-
 mm/ksm.c                               |   2 +
 mm/memory.c                            |  18 +-
 mm/mmap.c                              |   3 +
 mm/rmap.c                              |   4 +
 virt/kvm/Kconfig                       |   3 +
 virt/kvm/async_pf.c                    |   2 +-
 virt/kvm/kvm_main.c                    | 238 +++++++++++++++++++++----
 virt/lib/Makefile                      |   1 +
 virt/lib/mem_protected.c               | 193 ++++++++++++++++++++
 39 files changed, 666 insertions(+), 123 deletions(-)
 create mode 100644 arch/x86/mm/mem_encrypt_common.c
 create mode 100644 virt/lib/mem_protected.c
Signature · 1 line
Vitaly
#57 lore raw
Quoted 35 lines · Vitaly Kuznetsov
Quoted 25 lines · Kirill A. Shutemov

== Background / Problem ==

There are a number of hardware features (MKTME, SEV) which protect guest memory from some unauthorized host access. The patchset proposes a purely software feature that mitigates some of the same host-side read-only attacks.

== What does this set mitigate? ==

- Host kernel ”accidental” access to guest data (think speculation)

- Host kernel induced access to guest data (write(fd, &guest_data_ptr, len))

- Host userspace access to guest data (compromised qemu)

- Guest privilege escalation via compromised QEMU device emulation

== What does this set NOT mitigate? ==

- Full host kernel compromise. Kernel will just map the pages again.

- Hardware attacks

Quoted 7 lines

The second RFC revision addresses /most/ of the feedback.

I still didn't found a good solution to reboot and kexec. Unprotect all the memory on such operations defeat the goal of the feature. Clearing up most of the memory before unprotecting what is required for reboot (or kexec) is tedious and error-prone.
Maybe we should just declare them unsupported?

Quoted 4 lines

Making reboot unsupported is a hard sell. Could you please elaborate on why you think that "unprotect all" hypercall (or rather a single hypercall supporting both protecting/unprotecting) defeats the purpose of the feature?

If guest has some data that it prefers not to leak to the host and use the feature for the purpose, share all the memory to get through reboot is a very weak point.

Quoted 8 lines

clean up *all* its memory upon reboot, however:
- It may only clean up the most sensitive parts. This should probably be done even without this new feature and even on bare metal (think about next boot target being malicious).
- The attack window shrinks significantly. "Speculative" bugs require time to exploit and it will only remain open until it boots up again (few seconds).

Maybe it would be cleaner to handle reboot in userspace? If we got the VM rebooted, just reconstruct it from scratch as if it would be new boot.

Signature · 1 line
 Kirill A. Shutemov
#58 lore raw
Quoted 36 lines · Kirill A. Shutemov
Quoted 35 lines · Vitaly Kuznetsov
Quoted 32 lines · Kirill A. Shutemov

== Background / Problem ==

There are a number of hardware features (MKTME, SEV) which protect guest memory from some unauthorized host access. The patchset proposes a purely software feature that mitigates some of the same host-side read-only attacks.

== What does this set mitigate? ==

- Host kernel ”accidental” access to guest data (think speculation)

- Host kernel induced access to guest data (write(fd, &guest_data_ptr, len))

- Host userspace access to guest data (compromised qemu)

- Guest privilege escalation via compromised QEMU device emulation

== What does this set NOT mitigate? ==

- Full host kernel compromise. Kernel will just map the pages again.

- Hardware attacks

The second RFC revision addresses /most/ of the feedback.

I still didn't found a good solution to reboot and kexec. Unprotect all the memory on such operations defeat the goal of the feature. Clearing up most of the memory before unprotecting what is required for reboot (or kexec) is tedious and error-prone.
Maybe we should just declare them unsupported?

Quoted 9 lines
Quoted 4 lines

Making reboot unsupported is a hard sell. Could you please elaborate on why you think that "unprotect all" hypercall (or rather a single hypercall supporting both protecting/unprotecting) defeats the purpose of the feature?

If guest has some data that it prefers not to leak to the host and use the feature for the purpose, share all the memory to get through reboot is a very weak point.

My point that if it knows that there's something sensitive in its memory it should clean it up even today without your feature before rebooting to an unknown target.

Quoted 11 lines
Quoted 8 lines

clean up *all* its memory upon reboot, however:
- It may only clean up the most sensitive parts. This should probably be done even without this new feature and even on bare metal (think about next boot target being malicious).
- The attack window shrinks significantly. "Speculative" bugs require time to exploit and it will only remain open until it boots up again (few seconds).

Maybe it would be cleaner to handle reboot in userspace? If we got the VM rebooted, just reconstruct it from scratch as if it would be new boot.

We are definitely not trying to protect against malicious KVM so maybe we can do the cleanup there (when protection was enabled) so we can unprotect everything without risk of a leak?

Signature · 1 line
Vitaly
#59 lore raw
Quoted 43 lines · Vitaly Kuznetsov
Quoted 41 lines · Kirill A. Shutemov
Quoted 39 lines · Vitaly Kuznetsov
Quoted 25 lines · Kirill A. Shutemov

== Background / Problem ==

There are a number of hardware features (MKTME, SEV) which protect guest memory from some unauthorized host access. The patchset proposes a purely software feature that mitigates some of the same host-side read-only attacks.

== What does this set mitigate? ==

- Host kernel ”accidental” access to guest data (think speculation)

- Host kernel induced access to guest data (write(fd, &guest_data_ptr, len))

- Host userspace access to guest data (compromised qemu)

- Guest privilege escalation via compromised QEMU device emulation

== What does this set NOT mitigate? ==

- Full host kernel compromise. Kernel will just map the pages again.

- Hardware attacks

Quoted 7 lines

The second RFC revision addresses /most/ of the feedback.

I still didn't found a good solution to reboot and kexec. Unprotect all the memory on such operations defeat the goal of the feature. Clearing up most of the memory before unprotecting what is required for reboot (or kexec) is tedious and error-prone.
Maybe we should just declare them unsupported?

Making reboot unsupported is a hard sell. Could you please elaborate on why you think that "unprotect all" hypercall (or rather a single hypercall supporting both protecting/unprotecting) defeats the purpose of the feature?

Quoted 8 lines
Quoted 4 lines

If guest has some data that it prefers not to leak to the host and use the feature for the purpose, share all the memory to get through reboot is a very weak point.

My point that if it knows that there's something sensitive in its memory it should clean it up even today without your feature before rebooting to an unknown target.

It's unrealistic to expect everybody to do the right thing.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

clean up *all* its memory upon reboot, however:
- It may only clean up the most sensitive parts. This should probably be done even without this new feature and even on bare metal (think about next boot target being malicious).
- The attack window shrinks significantly. "Speculative" bugs require time to exploit and it will only remain open until it boots up again (few seconds).

Quoted 6 lines
Quoted 2 lines

Maybe it would be cleaner to handle reboot in userspace? If we got the VM rebooted, just reconstruct it from scratch as if it would be new boot.

We are definitely not trying to protect against malicious KVM so maybe we can do the cleanup there (when protection was enabled) so we can unprotect everything without risk of a leak?

Do you have any particular codepath in mind? I didn't find anything suitable so far.

Signature · 1 line
 Kirill A. Shutemov
#60 lore raw
Quoted 6 lines · Kirill A. Shutemov
Quoted 5 lines · Vitaly Kuznetsov
Quoted 2 lines · Kirill A. Shutemov

Maybe it would be cleaner to handle reboot in userspace? If we got the VM rebooted, just reconstruct it from scratch as if it would be new boot.

Quoted 6 lines
Quoted 3 lines

We are definitely not trying to protect against malicious KVM so maybe we can do the cleanup there (when protection was enabled) so we can unprotect everything without risk of a leak?

Do you have any particular codepath in mind? I didn't find anything suitable so far.

I didn't put much thought in it but e.g. on x86, what if we put this to kvm_vcpu_reset() under 'if (kvm_vcpu_is_bsp())' condition?

The main problem I see is that we can't clean up *all* memory, e.g. firmware related stuff should stay intact and this contraducts your KVM_HC_ENABLE_MEM_PROTECTED which protects everything. We can, probably, get rid of it leaving KVM_HC_MEM_SHARE/KVM_HC_MEM_UNSHARE only shifting responsibility to define what can be cleaned up on the guest kernel (stating in the doc that all protected memory will get whiped out on reboot).

Signature · 1 line
Vitaly
#61 lore raw
In reply to Kirill A. Shutemov
Quoted 13 lines · Kirill A. Shutemov

For removing the userspace mapping, use a trick similar to what NUMA balancing does: convert memory that belongs to KVM memory slots to
PROT_NONE: all existing entries converted to PROT_NONE with mprotect() and the newly faulted in pages get PROT_NONE from the updated vm_page_prot. The new VMA flag -- VM_KVM_PROTECTED -- indicates that the pages in the VMA must be treated in a special way in the GUP and fault paths. The flag allows GUP to return the page even though it is mapped with PROT_NONE, but only if the new GUP flag -- FOLL_KVM -- is specified. Any userspace access to the memory would result in SIGBUS. Any GUP access without FOLL_KVM would result in -EFAULT.

I definitely like the direction this patchset is going in, and I think that allowing KVM guests to have memory that is inaccessible to QEMU is a great idea.

I do wonder, though: do we really want to do this with these PROT_NONE tricks, or should we actually come up with a way to have KVM guest map memory that isn't mapped into QEMU's mm_struct at all? As an example of the latter, I mean something a bit like this:

https://lkml.kernel.org/r/CALCETrUSUp_7svg8EHNTk3nQ0x9sdzMCU=h8G-Sy6=SODq5GHg@mail.gmail.com

I don't mean to say that this is a requirement of any kind of protected memory like this, but I do think we should understand the tradeoffs, in terms of what a full implementation looks like, the effort and time frames involved, and the maintenance burden of supporting whatever gets merged going forward.

#62 lore raw
Quoted 23 lines · Andy Lutomirski
Quoted 13 lines · Kirill A. Shutemov

For removing the userspace mapping, use a trick similar to what NUMA balancing does: convert memory that belongs to KVM memory slots to
PROT_NONE: all existing entries converted to PROT_NONE with mprotect() and the newly faulted in pages get PROT_NONE from the updated vm_page_prot. The new VMA flag -- VM_KVM_PROTECTED -- indicates that the pages in the VMA must be treated in a special way in the GUP and fault paths. The flag allows GUP to return the page even though it is mapped with PROT_NONE, but only if the new GUP flag -- FOLL_KVM -- is specified. Any userspace access to the memory would result in SIGBUS. Any GUP access without FOLL_KVM would result in -EFAULT.

I definitely like the direction this patchset is going in, and I think that allowing KVM guests to have memory that is inaccessible to QEMU is a great idea.

I do wonder, though: do we really want to do this with these PROT_NONE tricks, or should we actually come up with a way to have KVM guest map memory that isn't mapped into QEMU's mm_struct at all? As an example of the latter, I mean something a bit like this:

Quoted 7 lines

https://lkml.kernel.org/r/CALCETrUSUp_7svg8EHNTk3nQ0x9sdzMCU=h8G-Sy6=SODq5GHg@mail.gmail.com

I don't mean to say that this is a requirement of any kind of protected memory like this, but I do think we should understand the tradeoffs, in terms of what a full implementation looks like, the effort and time frames involved, and the maintenance burden of supporting whatever gets merged going forward.

I considered the PROT_NONE trick neat. Complete removing of the mapping from QEMU would require more changes into KVM and I'm not really familiar with it.

About tradeoffs: the trick interferes with AutoNUMA. I didn't put much thought into how we can get it work together. Need to look into it.

Do you see other tradeoffs?

Signature · 1 line
 Kirill A. Shutemov
maintainer
#63 lore raw
Quoted 33 lines · Kirill A. Shutemov
Quoted 23 lines · Andy Lutomirski
Quoted 13 lines · Kirill A. Shutemov

For removing the userspace mapping, use a trick similar to what NUMA balancing does: convert memory that belongs to KVM memory slots to
PROT_NONE: all existing entries converted to PROT_NONE with mprotect() and the newly faulted in pages get PROT_NONE from the updated vm_page_prot. The new VMA flag -- VM_KVM_PROTECTED -- indicates that the pages in the VMA must be treated in a special way in the GUP and fault paths. The flag allows GUP to return the page even though it is mapped with PROT_NONE, but only if the new GUP flag -- FOLL_KVM -- is specified. Any userspace access to the memory would result in SIGBUS. Any GUP access without FOLL_KVM would result in -EFAULT.

I definitely like the direction this patchset is going in, and I think that allowing KVM guests to have memory that is inaccessible to QEMU is a great idea.

I do wonder, though: do we really want to do this with these PROT_NONE tricks, or should we actually come up with a way to have KVM guest map memory that isn't mapped into QEMU's mm_struct at all? As an example of the latter, I mean something a bit like this:

Quoted 7 lines

https://lkml.kernel.org/r/CALCETrUSUp_7svg8EHNTk3nQ0x9sdzMCU=h8G-Sy6=SODq5GHg@mail.gmail.com

I don't mean to say that this is a requirement of any kind of protected memory like this, but I do think we should understand the tradeoffs, in terms of what a full implementation looks like, the effort and time frames involved, and the maintenance burden of supporting whatever gets merged going forward.

Quoted 3 lines

I considered the PROT_NONE trick neat. Complete removing of the mapping from QEMU would require more changes into KVM and I'm not really familiar with it.

I think it's neat. The big tradeoff I'm concerned about is that it will likely become ABI once it lands. That is, if this series lands, then we will always have to support the case in which QEMU has a special non-present mapping that is nonetheless reflected as present in a guest. This is a bizarre state of affairs, it may become obsolete if a better API ever shows up, and it might end up placing constraints on the Linux VM that we don't love going forward.

I don't think my proposal in the referenced thread above is that crazy or that difficult to implement. The basic idea is to have a way to create an mm_struct that is not loaded in CR3 anywhere. Instead, KVM will reference it, much as it currently references QEMU's mm_struct, to mirror mappings into the guest. This means it would be safe to have "protected" memory mapped into the special mm_struct because nothing other than KVM will ever reference the PTEs. But I think that someone who really understands the KVM memory mapping code should chime in.

Quoted 8 lines

About tradeoffs: the trick interferes with AutoNUMA. I didn't put much thought into how we can get it work together. Need to look into it.

Do you see other tradeoffs?

--
Kirill A. Shutemov