KVM protected memory extension
Subsystems (18)
v2 cover KVM protected memory extension
== Background / Problem ==
There are a number of hardware features (MKTME, SEV) which protect guest memory from some unauthorized host access. The patchset proposes a purely software feature that mitigates some of the same host-side read-only attacks.
== What does this set mitigate? ==
- Host kernel ”accidental” access to guest data (think speculation)
- Host kernel induced access to guest data (write(fd, &guest_data_ptr, len))
- Host userspace access to guest data (compromised qemu)
- Guest privilege escalation via compromised QEMU device emulation
== What does this set NOT mitigate? ==
- Full host kernel compromise. Kernel will just map the pages again.
- Hardware attacks
The second RFC revision addresses /most/ of the feedback.
I still didn't found a good solution to reboot and kexec. Unprotect all the memory on such operations defeat the goal of the feature. Clearing up most of the memory before unprotecting what is required for reboot (or kexec) is tedious and error-prone.
Maybe we should just declare them unsupported?
== Series Overview ==
The hardware features protect guest data by encrypting it and then ensuring that only the right guest can decrypt it. This has the side-effect of making the kernel direct map and userspace mapping (QEMU et al) useless. But, this teaches us something very useful:
neither the kernel or userspace mappings are really necessary for normal guest operations.
Instead of using encryption, this series simply unmaps the memory. One advantage compared to allowing access to ciphertext is that it allows bad accesses to be caught instead of simply reading garbage.
Protection from physical attacks needs to be provided by some other means. On Intel platforms, (single-key) Total Memory Encryption (TME) provides mitigation against physical attacks, such as DIMM interposers sniffing memory bus traffic.
The patchset modifies both host and guest kernel. The guest OS must enable the feature via hypercall and mark any memory range that has to be shared with the host: DMA regions, bounce buffers, etc. SEV does this marking via a bit in the guest’s page table while this approach uses a hypercall.
For removing the userspace mapping, use a trick similar to what NUMA balancing does: convert memory that belongs to KVM memory slots toPROT_NONE: all existing entries converted to PROT_NONE with mprotect() and the newly faulted in pages get PROT_NONE from the updated vm_page_prot. The new VMA flag -- VM_KVM_PROTECTED -- indicates that the pages in the VMA must be treated in a special way in the GUP and fault paths. The flag allows GUP to return the page even though it is mapped with PROT_NONE, but only if the new GUP flag -- FOLL_KVM -- is specified. Any userspace access to the memory would result in SIGBUS. Any GUP access without FOLL_KVM would result in -EFAULT.
Removing userspace mapping of the guest memory from QEMU process can help to address some guest privilege escalation attacks. Consider the case when unprivileged guest user exploits bug in a QEMU device emulation to gain access to data it cannot normally have access within the guest.
Any anonymous page faulted into the VM_KVM_PROTECTED VMA gets removed from the direct mapping with kernel_map_pages(). Note that kernel_map_pages() only flushes local TLB. I think it's a reasonable compromise between security and perfromance.
Zapping the PTE would bring the page back to the direct mapping after clearing. At least for now, we don't remove file-backed pages from the direct mapping. File-backed pages could be accessed via read/write syscalls. It adds complexity.
Occasionally, host kernel has to access guest memory that was not made shared by the guest. For instance, it happens for instruction emulation. Normally, it's done via copy_to/from_user() which would fail with -EFAULT now. We introduced a new pair of helpers: copy_to/from_guest(). The new helpers acquire the page via GUP, map it into kernel address space with kmap_atomic()-style mechanism and only then copy the data.
For some instruction emulation copying is not good enough: cmpxchg emulation has to have direct access to the guest memory. __kvm_map_gfn() is modified to accommodate the case.
The patchset is on top of v5.9
Kirill A. Shutemov (16):
x86/mm: Move force_dma_unencrypted() to common code x86/kvm: Introduce KVM memory protection feature
x86/kvm: Make DMA pages shared
x86/kvm: Use bounce buffers for KVM memory protection x86/kvm: Make VirtIO use DMA API in KVM guest x86/kvmclock: Share hvclock memory with the host x86/realmode: Share trampoline area if KVM memory protection enabled
KVM: Use GUP instead of copy_from/to_user() to access guest memory
KVM: mm: Introduce VM_KVM_PROTECTED
KVM: x86: Use GUP for page walk instead of __get_user()
KVM: Protected memory extension
KVM: x86: Enabled protected memory extension
KVM: Rework copy_to/from_guest() to avoid direct mapping
KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()
KVM: Unmap protected pages from direct mapping
mm: Do not use zero page for VM_KVM_PROTECTED VMAs
arch/powerpc/kvm/book3s_64_mmu_hv.c | 2 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 2 +- arch/s390/include/asm/pgtable.h | 2 +- arch/x86/Kconfig | 11 +- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/io.h | 6 +- arch/x86/include/asm/kvm_para.h | 5 + arch/x86/include/asm/pgtable_types.h | 1 + arch/x86/include/uapi/asm/kvm_para.h | 3 +- arch/x86/kernel/kvm.c | 20 +++ arch/x86/kernel/kvmclock.c | 2 +- arch/x86/kernel/pci-swiotlb.c | 3 +- arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/cpuid.c | 3 +- arch/x86/kvm/mmu/mmu.c | 6 +- arch/x86/kvm/mmu/paging_tmpl.h | 10 +- arch/x86/kvm/x86.c | 9 + arch/x86/mm/Makefile | 2 + arch/x86/mm/ioremap.c | 16 +- arch/x86/mm/mem_encrypt.c | 51 ------ arch/x86/mm/mem_encrypt_common.c | 62 +++++++ arch/x86/mm/pat/set_memory.c | 7 + arch/x86/realmode/init.c | 7 +- drivers/virtio/virtio_ring.c | 4 + include/linux/kvm_host.h | 11 +- include/linux/kvm_types.h | 1 + include/linux/mm.h | 21 ++- include/uapi/linux/kvm_para.h | 5 +- mm/gup.c | 20 ++- mm/huge_memory.c | 31 +++- mm/ksm.c | 2 + mm/memory.c | 18 +- mm/mmap.c | 3 + mm/rmap.c | 4 + virt/kvm/Kconfig | 3 + virt/kvm/async_pf.c | 2 +- virt/kvm/kvm_main.c | 238 +++++++++++++++++++++---- virt/lib/Makefile | 1 + virt/lib/mem_protected.c | 193 ++++++++++++++++++++ 39 files changed, 666 insertions(+), 123 deletions(-) create mode 100644 arch/x86/mm/mem_encrypt_common.c create mode 100644 virt/lib/mem_protected.c
Signature · 1 line
2.26.2
RFC v2 1/16 x86/mm: Move force_dma_unencrypted() to common code
force_dma_unencrypted() has to return true for KVM guest with the memory protected enabled. Move it out of AMD SME code.
Introduce new config option X86_MEM_ENCRYPT_COMMON that has to be selected by all x86 memory encryption features.
This is preparation for the following patches.
--- arch/x86/Kconfig | 8 +++++-- arch/x86/include/asm/io.h | 4 +++- arch/x86/mm/Makefile | 2 ++ arch/x86/mm/mem_encrypt.c | 30 ------------------------- arch/x86/mm/mem_encrypt_common.c | 38 ++++++++++++++++++++++++++++++++ 5 files changed, 49 insertions(+), 33 deletions(-) create mode 100644 arch/x86/mm/mem_encrypt_common.c
arch/x86/Kconfig +6 -2
@@ -1514,13 +1514,17 @@ config X86_CPA_STATISTICS helps to determine the effectiveness of preserving large and huge page mappings when mapping protections are changed. +config X86_MEM_ENCRYPT_COMMON + select ARCH_HAS_FORCE_DMA_UNENCRYPTED + select DYNAMIC_PHYSICAL_MASK + def_bool n + config AMD_MEM_ENCRYPT bool "AMD Secure Memory Encryption (SME) support" depends on X86_64 && CPU_SUP_AMD select DMA_COHERENT_POOL - select DYNAMIC_PHYSICAL_MASK select ARCH_USE_MEMREMAP_PROT - select ARCH_HAS_FORCE_DMA_UNENCRYPTED + select X86_MEM_ENCRYPT_COMMON help Say yes to enable support for the encryption of system memory. This requires an AMD processor that supports Secure Memory
arch/x86/include/asm/io.h +3 -1
@@ -256,10 +256,12 @@ static inline void slow_down_io(void)#endif -#ifdef CONFIG_AMD_MEM_ENCRYPT #include <linux/jump_label.h> extern struct static_key_false sev_enable_key; + +#ifdef CONFIG_AMD_MEM_ENCRYPT +static inline bool sev_key_active (void ) {return static_branch_unlikely (&sev_enable_key);
arch/x86/mm/Makefile +2 -0
@@ -52,6 +52,8 @@ obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) += pkeys.o obj-$(CONFIG_RANDOMIZE_MEMORY) += kaslr.o obj-$(CONFIG_PAGE_TABLE_ISOLATION) += pti.o +obj-$(CONFIG_X86_MEM_ENCRYPT_COMMON) += mem_encrypt_common.o + obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_identity.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_boot.o
arch/x86/mm/mem_encrypt.c +0 -30
@@ -15,10 +15,6 @@#include <linux/dma-direct.h> #include <linux/swiotlb.h> #include <linux/mem_encrypt.h> -#include <linux/device.h> -#include <linux/kernel.h> -#include <linux/bitops.h> -#include <linux/dma-mapping.h> #include <asm/tlbflush.h> #include <asm/fixmap.h>
@@ -350,32 +346,6 @@ bool sev_active(void)return sme_me_mask && sev_enabled; } -/* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */ -bool force_dma_unencrypted (struct device *dev) -{ -/* -* For SEV, all DMA must be to unencrypted addresses. -*/ -if (sev_active ()) -return true; - -/* -* For SME, all DMA must be to unencrypted addresses if the -* device does not support DMA to addresses that include the -* encryption mask. -*/ -if (sme_active ()) { -u64 dma_enc_mask =DMA_BIT_MASK (__ffs64 (sme_me_mask)); -u64 dma_dev_mask =min_not_zero (dev->coherent_dma_mask, - dev->bus_dma_limit); - -if (dma_dev_mask <= dma_enc_mask) -return true; - } - -return false; -} -void __init mem_encrypt_free_decrypted_mem (void ) {unsigned long vaddr, vaddr_end, npages;
arch/x86/mm/mem_encrypt_common.c new file +38 -0
@@ -0,0 +1,38 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* AMD Memory Encryption Support +* +* Copyright (C) 2016 Advanced Micro Devices, Inc. +* +* Author: Tom Lendacky <[email protected]> +*/ + +#include <linux/mm.h> +#include <linux/mem_encrypt.h> +#include <linux/dma-mapping.h> + +/* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */ +bool force_dma_unencrypted (struct device *dev) +{ +/* +* For SEV, all DMA must be to unencrypted/shared addresses. +*/ +if (sev_active ()) +return true; + +/* +* For SME, all DMA must be to unencrypted addresses if the +* device does not support DMA to addresses that include the +* encryption mask. +*/ +if (sme_active ()) { +u64 dma_enc_mask =DMA_BIT_MASK (__ffs64 (sme_me_mask)); +u64 dma_dev_mask =min_not_zero (dev->coherent_dma_mask, + dev->bus_dma_limit); + +if (dma_dev_mask <= dma_enc_mask) +return true; + } + +return false; +}
Signature · 1 line
2.26.2
RFC v2 2/16 x86/kvm: Introduce KVM memory protection feature
In reply to Kirill A. ShutemovProvide basic helpers, KVM_FEATURE, CPUID flag and a hypercall.
Host side doesn't provide the feature yet, so it is a dead code for now.
--- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/kvm_para.h | 5 +++++ arch/x86/include/uapi/asm/kvm_para.h | 3 ++- arch/x86/kernel/kvm.c | 18 ++++++++++++++++++ include/uapi/linux/kvm_para.h | 3 ++- 5 files changed, 28 insertions(+), 2 deletions(-)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -236,6 +236,7 @@#define X86_FEATURE_EPT_AD ( 8*32+17)/* Intel Extended Page Table access-dirty bit */ #define X86_FEATURE_VMCALL ( 8*32+18)/* "" Hypervisor supports the VMCALL instruction */ #define X86_FEATURE_VMW_VMMCALL ( 8*32+19)/* "" VMware prefers VMMCALL hypercall instruction */ +#define X86_FEATURE_KVM_MEM_PROTECTED ( 8*32+20)/* KVM memory protection extenstion */ /* Intel-defined CPU features, CPUID level 0x00000007:0 (EBX), word 9 */ #define X86_FEATURE_FSGSBASE ( 9*32+ 0)/* RDFSBASE, WRFSBASE, RDGSBASE, WRGSBASE instructions*/
arch/x86/include/asm/kvm_para.h +5 -0
@@ -11,11 +11,16 @@ extern void kvmclock_init(void);#ifdef CONFIG_KVM_GUEST bool kvm_check_and_clear_guest_paused (void ); +bool kvm_mem_protected (void );#else static inline bool kvm_check_and_clear_guest_paused (void ) {return false; } +static inline bool kvm_mem_protected (void ) +{ +return false; +}#endif /* CONFIG_KVM_GUEST */ #define KVM_HYPERCALL \
arch/x86/include/uapi/asm/kvm_para.h +2 -1
@@ -28,10 +28,11 @@#define KVM_FEATURE_PV_UNHALT 7#define KVM_FEATURE_PV_TLB_FLUSH 9#define KVM_FEATURE_ASYNC_PF_VMEXIT 10 -#define KVM_FEATURE_PV_SEND_IPI 11 +#define KVM_FEATURE_PV_SEND_IPI 11#define KVM_FEATURE_POLL_CONTROL 12#define KVM_FEATURE_PV_SCHED_YIELD 13#define KVM_FEATURE_ASYNC_PF_INT 14 +#define KVM_FEATURE_MEM_PROTECTED 15#define KVM_HINTS_REALTIME 0
arch/x86/kernel/kvm.c +18 -0
@@ -37,6 +37,13 @@#include <asm/tlb.h> #include <asm/cpuidle_haltpoll.h> +static bool mem_protected; + +bool kvm_mem_protected (void ) +{ +return mem_protected; +} +DEFINE_STATIC_KEY_FALSE (kvm_async_pf_enabled);static int kvmapf =1 ;
@@ -742,6 +749,17 @@ static void __init kvm_init_platform(void) {kvmclock_init (); x86_platform.apic_post_init = kvm_apic_init; + +if (kvm_para_has_feature (KVM_FEATURE_MEM_PROTECTED )) { +if (kvm_hypercall0 (KVM_HC_ENABLE_MEM_PROTECTED )) { +pr_err ("Failed to enable KVM memory protection\n" ); +return ; + } + +pr_info ("KVM memory protection enabled\n" ); + mem_protected = true; +setup_force_cpu_cap (X86_FEATURE_KVM_MEM_PROTECTED ); + } }const __initconst struct hypervisor_x86 x86_hyper_kvm = {
include/uapi/linux/kvm_para.h +2 -1
@@ -27,8 +27,9 @@#define KVM_HC_MIPS_EXIT_VM 7#define KVM_HC_MIPS_CONSOLE_OUTPUT 8#define KVM_HC_CLOCK_PAIRING 9 -#define KVM_HC_SEND_IPI 10 +#define KVM_HC_SEND_IPI 10#define KVM_HC_SCHED_YIELD 11 +#define KVM_HC_ENABLE_MEM_PROTECTED 12/* * hypercalls use architecture specific
Signature · 1 line
2.26.2
RFC v2 3/16 x86/kvm: Make DMA pages shared
In reply to Kirill A. ShutemovMake force_dma_unencrypted() return true for KVM to get DMA pages mapped as shared.
__set_memory_enc_dec() now informs the host via hypercall if the state of the page has changed from shared to private or back.
--- arch/x86/Kconfig | 1 + arch/x86/mm/mem_encrypt_common.c | 5 +++-- arch/x86/mm/pat/set_memory.c | 7 +++++++ include/uapi/linux/kvm_para.h | 2 ++ 4 files changed, 13 insertions(+), 2 deletions(-)
arch/x86/Kconfig +1 -0
@@ -805,6 +805,7 @@ config KVM_GUEST select PARAVIRT_CLOCK select ARCH_CPUIDLE_HALTPOLL select X86_HV_CALLBACK_VECTOR + select X86_MEM_ENCRYPT_COMMON default y help This option enables various optimizations for running under the KVM
arch/x86/mm/mem_encrypt_common.c +3 -2
@@ -10,14 +10,15 @@#include <linux/mm.h> #include <linux/mem_encrypt.h> #include <linux/dma-mapping.h> +#include <asm/kvm_para.h> /* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */ bool force_dma_unencrypted (struct device *dev) {/* -* For SEV, all DMA must be to unencrypted/shared addresses. +* For SEV and KVM, all DMA must be to unencrypted/shared addresses. */ -if (sev_active ()) +if (sev_active () ||kvm_mem_protected ())return true;/*
arch/x86/mm/pat/set_memory.c +7 -0
@@ -16,6 +16,7 @@#include <linux/pci.h> #include <linux/vmalloc.h> #include <linux/libnvdimm.h> +#include <linux/kvm_para.h> #include <asm/e820/api.h> #include <asm/processor.h>
@@ -1977,6 +1978,12 @@ static int __set_memory_enc_dec(unsigned long addr, int numpages, bool enc)struct cpa_data cpa;int ret; +if (kvm_mem_protected ()) { +unsigned long gfn =__pa (addr) >>PAGE_SHIFT ; +int call = enc ?KVM_HC_MEM_UNSHARE :KVM_HC_MEM_SHARE ; +return kvm_hypercall2 (call, gfn, numpages); + } +/* Nothing to do if memory encryption is not active */ if (!mem_encrypt_active ())return 0 ;
include/uapi/linux/kvm_para.h +2 -0
@@ -30,6 +30,8 @@#define KVM_HC_SEND_IPI 10#define KVM_HC_SCHED_YIELD 11#define KVM_HC_ENABLE_MEM_PROTECTED 12 +#define KVM_HC_MEM_SHARE 13 +#define KVM_HC_MEM_UNSHARE 14/* * hypercalls use architecture specific
Signature · 1 line
2.26.2
RFC v2 4/16 x86/kvm: Use bounce buffers for KVM memory protection
In reply to Kirill A. ShutemovMirror SEV, use SWIOTLB always if KVM memory protection is enabled.
--- arch/x86/Kconfig | 1 + arch/x86/kernel/kvm.c | 2 ++ arch/x86/kernel/pci-swiotlb.c | 3 ++- arch/x86/mm/mem_encrypt.c | 21 --------------------- arch/x86/mm/mem_encrypt_common.c | 23 +++++++++++++++++++++++ 5 files changed, 28 insertions(+), 22 deletions(-)
arch/x86/Kconfig +1 -0
@@ -806,6 +806,7 @@ config KVM_GUEST select ARCH_CPUIDLE_HALTPOLL select X86_HV_CALLBACK_VECTOR select X86_MEM_ENCRYPT_COMMON + select SWIOTLB default y help This option enables various optimizations for running under the KVM
arch/x86/kernel/kvm.c +2 -0
@@ -26,6 +26,7 @@#include <linux/kprobes.h> #include <linux/nmi.h> #include <linux/swait.h> +#include <linux/swiotlb.h> #include <asm/timer.h> #include <asm/cpu.h> #include <asm/traps.h>
@@ -759,6 +760,7 @@ static void __init kvm_init_platform(void)pr_info ("KVM memory protection enabled\n" ); mem_protected = true;setup_force_cpu_cap (X86_FEATURE_KVM_MEM_PROTECTED ); + swiotlb_force =SWIOTLB_FORCE ; } }
arch/x86/kernel/pci-swiotlb.c +2 -1
@@ -13,6 +13,7 @@#include <asm/dma.h> #include <asm/xen/swiotlb-xen.h> #include <asm/iommu_table.h> +#include <asm/kvm_para.h> int swiotlb__read_mostly ;
@@ -49,7 +50,7 @@ int __init pci_swiotlb_detect_4gb(void)* buffers are allocated and used for devices that do not support * the addressing range required for the encryption mask. */ -if (sme_active ()) +if (sme_active () ||kvm_mem_protected ()) swiotlb =1 ;return swiotlb;
arch/x86/mm/mem_encrypt.c +0 -21
@@ -369,24 +369,3 @@ void __init mem_encrypt_free_decrypted_mem(void)free_init_pages ("unused decrypted" , vaddr, vaddr_end); } - -/* Architecture __weak replacement functions */ -void __init mem_encrypt_init (void ) -{ -if (!sme_me_mask) -return ; - -/* Call into SWIOTLB to update the SWIOTLB DMA buffers */ -swiotlb_update_mem_attributes (); - -/* -* With SEV, we need to unroll the rep string I/O instructions. -*/ -if (sev_active ()) -static_branch_enable (&sev_enable_key); - -pr_info ("AMD %s active\n" , -sev_active () ?"Secure Encrypted Virtualization (SEV)" - :"Secure Memory Encryption (SME)" ); -} -
arch/x86/mm/mem_encrypt_common.c +23 -0
@@ -37,3 +37,26 @@ bool force_dma_unencrypted(struct device *dev)return false; } + +void __init mem_encrypt_init (void ) +{ +if (!sme_me_mask && !kvm_mem_protected ()) +return ; + +/* Call into SWIOTLB to update the SWIOTLB DMA buffers */ +swiotlb_update_mem_attributes (); + +/* +* With SEV, we need to unroll the rep string I/O instructions. +*/ +if (sev_active ()) +static_branch_enable (&sev_enable_key); + +if (sme_me_mask) { +pr_info ("AMD %s active\n" , +sev_active () ?"Secure Encrypted Virtualization (SEV)" + :"Secure Memory Encryption (SME)" ); + }else { +pr_info ("KVM memory protection enabled\n" ); + } +}
Signature · 1 line
2.26.2
kernel test robot
build botRe: [RFCv2 04/16] x86/kvm: Use bounce buffers for KVM memory protection
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi "Kirill,
[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on tip/x86/core]
[also build test WARNING on kvm/linux-next hnaz-linux-mm/master v5.9] [cannot apply to next-20201016]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130 base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 238c91115cd05c71447ea071624a4c9fe661f970 config: x86_64-alldefconfig (attached as .config) compiler: gcc-9 (Debian 9.3.0-15) 9.3.0 reproduce (this is a W=1 build): # https://github.com/0day-ci/linux/commit/0bdb6c3e82c3106d720106932629f0e3a121fb76 git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130 git checkout 0bdb6c3e82c3106d720106932629f0e3a121fb76 # save the attached .config to linux build tree make W=1 ARCH=x86_64
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
arch/x86/mm/mem_encrypt_common.c:16:6: warning: no previous prototype for 'force_dma_unencrypted' [-Wmissing-prototypes]
16 | bool force_dma_unencrypted(struct device *dev)
| ^~~~~~~~~~~~~~~~~~~~~
>> arch/x86/mm/mem_encrypt_common.c:41:13: warning: no previous prototype for 'mem_encrypt_init' [-Wmissing-prototypes]
41 | void __init mem_encrypt_init(void)
| ^~~~~~~~~~~~~~~~
vim +/mem_encrypt_init +41 arch/x86/mm/mem_encrypt_common.c
14
15 /* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */
> 16 bool force_dma_unencrypted(struct device *dev)
17 {
18 /*
19 * For SEV and KVM, all DMA must be to unencrypted/shared addresses.
20 */
21 if (sev_active() || kvm_mem_protected())
22 return true;
23
24 /*
25 * For SME, all DMA must be to unencrypted addresses if the
26 * device does not support DMA to addresses that include the
27 * encryption mask.
28 */
29 if (sme_active()) {
30 u64 dma_enc_mask = DMA_BIT_MASK(__ffs64(sme_me_mask));
31 u64 dma_dev_mask = min_not_zero(dev->coherent_dma_mask,
32 dev->bus_dma_limit);
33
34 if (dma_dev_mask <= dma_enc_mask)
35 return true;
36 }
37
38 return false;
39 }
40
> 41 void __init mem_encrypt_init(void)---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kernel test robot
build botRe: [RFCv2 04/16] x86/kvm: Use bounce buffers for KVM memory protection
In reply to Kirill A. Shutemov[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi "Kirill,
[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on tip/x86/core]
[also build test WARNING on kvm/linux-next hnaz-linux-mm/master v5.9] [cannot apply to next-20201016]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130 base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 238c91115cd05c71447ea071624a4c9fe661f970 config: x86_64-randconfig-a001-20201020 (attached as .config) compiler: clang version 12.0.0 (https://github.com/llvm/llvm-project ea693a162786d933863ab079648d4261ac0ead47)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install x86_64 cross compiling tool for clang build
# apt-get install binutils-x86-64-linux-gnu
# https://github.com/0day-ci/linux/commit/0bdb6c3e82c3106d720106932629f0e3a121fb76
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130 git checkout 0bdb6c3e82c3106d720106932629f0e3a121fb76
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=x86_64
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
arch/x86/mm/mem_encrypt_common.c:16:6: warning: no previous prototype for function 'force_dma_unencrypted' [-Wmissing-prototypes] bool force_dma_unencrypted(struct device *dev)
^
arch/x86/mm/mem_encrypt_common.c:16:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
bool force_dma_unencrypted(struct device *dev)
^
static
>> arch/x86/mm/mem_encrypt_common.c:41:13: warning: no previous prototype for function 'mem_encrypt_init' [-Wmissing-prototypes]
void __init mem_encrypt_init(void)
^ arch/x86/mm/mem_encrypt_common.c:41:1: note: declare 'static' if the function is not intended to be used outside of this translation unit void __init mem_encrypt_init(void)
^
static
2 warnings generated.
vim +/mem_encrypt_init +41 arch/x86/mm/mem_encrypt_common.c
40 > 41 void __init mem_encrypt_init(void)
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
RFC v2 5/16 x86/kvm: Make VirtIO use DMA API in KVM guest
In reply to Kirill A. ShutemovVirtIO for KVM is a primary way to provide IO. All memory that used for communication with the host has to be marked as shared.
The easiest way to archive that is to use DMA API that already knows how to deal with shared memory.
--- drivers/virtio/virtio_ring.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/virtio/virtio_ring.c +4 -0
@@ -12,6 +12,7 @@#include <linux/hrtimer.h> #include <linux/dma-mapping.h> #include <xen/xen.h> +#include <asm/kvm_para.h> #ifdef DEBUG /* For development, we want to crash whenever the ring is screwed. */
@@ -255,6 +256,9 @@ static bool vring_use_dma_api(struct virtio_device *vdev)if (xen_domain ())return true; +if (kvm_mem_protected ()) +return true; +return false; }
Signature · 1 line
2.26.2
Re: [RFCv2 05/16] x86/kvm: Make VirtIO use DMA API in KVM guest
NAK. Any virtio implementation that needs special DMA OPS treatment needs to set the VIRTIO_F_ACCESS_PLATFORM bit. The only reason the Xen hack existst is because it slipped in a long time ago and we can't fix that any more.
Re: [RFCv2 05/16] x86/kvm: Make VirtIO use DMA API in KVM guest
Quoted 4 lines · Christoph Hellwig
NAK. Any virtio implementation that needs special DMA OPS treatment needs to set the
VIRTIO_F_ACCESS_PLATFORMbit. The only reason the Xen hack existst is because it slipped in a long time ago and we can't fix that any more.
Thanks. Will fix.
Signature · 1 line
Kirill A. Shutemov
kernel test robot
build botRe: [RFCv2 05/16] x86/kvm: Make VirtIO use DMA API in KVM guest
In reply to Kirill A. Shutemov[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi "Kirill,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/x86/core]
[also build test ERROR on kvm/linux-next hnaz-linux-mm/master v5.9] [cannot apply to next-20201016]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130 base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 238c91115cd05c71447ea071624a4c9fe661f970 config: xtensa-allyesconfig (attached as .config) compiler: xtensa-linux-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/e3df08a4af6c321d5942dcee585916ccf5e59ef6
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130 git checkout e3df08a4af6c321d5942dcee585916ccf5e59ef6
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=xtensa
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
drivers/virtio/virtio_ring.c: In function 'vring_use_dma_api':
>> drivers/virtio/virtio_ring.c:259:6: error: implicit declaration of function 'kvm_mem_protected' [-Werror=implicit-function-declaration]
259 | if (kvm_mem_protected())
| ^~~~~~~~~~~~~~~~~cc1: some warnings being treated as errors
vim +/kvm_mem_protected +259 drivers/virtio/virtio_ring.c
215
216 /*
217 * Modern virtio devices have feature bits to specify whether they need a
218 * quirk and bypass the IOMMU. If not there, just use the DMA API.
219 *
220 * If there, the interaction between virtio and DMA API is messy.
221 *
222 * On most systems with virtio, physical addresses match bus addresses,
223 * and it doesn't particularly matter whether we use the DMA API.
224 *
225 * On some systems, including Xen and any system with a physical device
226 * that speaks virtio behind a physical IOMMU, we must use the DMA API
227 * for virtio DMA to work at all.
228 *
229 * On other systems, including SPARC and PPC64, virtio-pci devices are
230 * enumerated as though they are behind an IOMMU, but the virtio host
231 * ignores the IOMMU, so we must either pretend that the IOMMU isn't
232 * there or somehow map everything as the identity.
233 *
234 * For the time being, we preserve historic behavior and bypass the DMA
235 * API.
236 *
237 * TODO: install a per-device DMA ops structure that does the right thing
238 * taking into account all the above quirks, and use the DMA API
239 * unconditionally on data path.
240 */
241
242 static bool vring_use_dma_api(struct virtio_device *vdev)
243 {
244 if (!virtio_has_dma_quirk(vdev))
245 return true;
246
247 /* Otherwise, we are left to guess. */
248 /*
249 * In theory, it's possible to have a buggy QEMU-supposed
250 * emulated Q35 IOMMU and Xen enabled at the same time. On
251 * such a configuration, virtio has never worked and will
252 * not work without an even larger kludge. Instead, enable
253 * the DMA API if we're a Xen guest, which at least allows
254 * all of the sensible Xen configurations to work correctly.
255 */
256 if (xen_domain())
257 return true;
258
> 259 if (kvm_mem_protected())
260 return true;
261
262 return false;
263 }
264
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Re: [RFCv2 05/16] x86/kvm: Make VirtIO use DMA API in KVM guest
In reply to Kirill A. ShutemovQuoted 24 lines · On Tue, 20 Oct 2020 09:18:48 +0300 "Kirill A. Shutemov" <[email protected]>…
VirtIO for KVM is a primary way to provide IO. All memory that used for communication with the host has to be marked as shared.
The easiest way to archive that is to use DMA API that already knows how to deal with shared memory.
Signed-off-by Kirill A. Shutemov <[email protected]>--- drivers/virtio/virtio_ring.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/virtio/virtio_ring.c +1
-0@@ -12,6 +12,7 @@#include <linux/hrtimer.h> #include <linux/dma-mapping.h> #include <xen/xen.h> +#include <asm/kvm_para.h> #ifdef DEBUG /* For development, we want to crash whenever the ring is screwed. */
drivers/virtio/virtio_ring.c static bool vring_use_dma_api(struct virtio_device *vdev) · line 261
Quoted 6 lines
drivers/virtio/virtio_ring.c +3
-0@@ -255,6 +256,9 @@ static bool vring_use_dma_api(struct virtio_device *vdev)if (xen_domain ())return true; +if (kvm_mem_protected ()) +return true; +
I guess it does not matter because Christophs comment, but this breaks the build for s390, because there is no kvm_mem_protected() for s390.
Regards,
Halil
Quoted 3 lines
drivers/virtio/virtio_ring.c +0
-0return false; }
RFC v2 6/16 x86/kvmclock: Share hvclock memory with the host
In reply to Kirill A. Shutemovhvclock is shared between the guest and the hypervisor. It has to be accessible by host.
--- arch/x86/kernel/kvmclock.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/kvmclock.c +1 -1
@@ -253,7 +253,7 @@ static void __init kvmclock_init_mem(void)* hvclock is shared between the guest and the hypervisor, must * be mapped decrypted. */ -if (sev_active ()) { +if (sev_active () ||kvm_mem_protected ()) { r =set_memory_decrypted ((unsigned long ) hvclock_mem,1UL << order);if (r) {
Signature · 1 line
2.26.2
RFC v2 7/16 x86/realmode: Share trampoline area if KVM memory protection enabled
In reply to Kirill A. ShutemovIf KVM memory protection is active, the trampoline area will need to be in shared memory.
--- arch/x86/realmode/init.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
arch/x86/realmode/init.c +4 -3
@@ -9,6 +9,7 @@#include <asm/realmode.h> #include <asm/tlbflush.h> #include <asm/crash.h> +#include <asm/kvm_para.h> struct real_mode_header *real_mode_header;u32 *trampoline_cr4_features;
@@ -55,11 +56,11 @@ static void __init setup_real_mode(void) base = (unsigned char *)real_mode_header;/* -* If SME is active, the trampoline area will need to be in -* decrypted memory in order to bring up other processors +* If SME or KVM memory protection is active, the trampoline area will +* need to be in decrypted memory in order to bring up other processors * successfully. This is not needed for SEV. */ -if (sme_active ()) +if (sme_active () ||kvm_mem_protected ())set_memory_decrypted ((unsigned long )base, size >>PAGE_SHIFT );memcpy (base, real_mode_blob, size);
Signature · 1 line
2.26.2
RFC v2 8/16 KVM: Use GUP instead of copy_from/to_user() to access guest memory
In reply to Kirill A. ShutemovNew helpers copy_from_guest()/copy_to_guest() to be used if KVM memory protection feature is enabled.
--- include/linux/kvm_host.h | 4 ++ virt/kvm/kvm_main.c | 90 +++++++++++++++++++++++++++++++--------- 2 files changed, 75 insertions(+), 19 deletions(-)
include/linux/kvm_host.h +4 -0
@@ -504,6 +504,7 @@ struct kvm {struct srcu_struct irq_srcu;pid_t userspace_pid;unsigned int max_halt_poll_ns; +bool mem_protected; };#define kvm_err (fmt, ...) \
@@ -728,6 +729,9 @@ void kvm_set_pfn_dirty(kvm_pfn_t pfn);void kvm_set_pfn_accessed (kvm_pfn_t pfn);void kvm_get_pfn (kvm_pfn_t pfn); +int copy_from_guest (void *data,unsigned long hva,int len,bool protected); +int copy_to_guest (unsigned long hva,const void *data,int len,bool protected); +void kvm_release_pfn (kvm_pfn_t pfn,bool dirty,struct gfn_to_pfn_cache *cache);int kvm_read_guest_page (struct kvm *kvm,gfn_t gfn,void *data,int offset,int len);
virt/kvm/kvm_main.c +71 -19
157 lines not on this page · raw message
Signature · 1 line
2.26.2
John Hubbard
reviewerRe: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory
Quoted 47 lines · Kirill A. Shutemov
New helpers copy_from_guest()/copy_to_guest() to be used if KVM memory protection feature is enabled.
Signed-off-by Kirill A. Shutemov <[email protected]>--- include/linux/kvm_host.h | 4 ++ virt/kvm/kvm_main.c | 90 +++++++++++++++++++++++++++++++--------- 2 files changed, 75 insertions(+), 19 deletions(-)include/linux/kvm_host.h +4
-0@@ -504,6 +504,7 @@ struct kvm {struct srcu_struct irq_srcu;pid_t userspace_pid;unsigned int max_halt_poll_ns; +bool mem_protected; };#define kvm_err (fmt, ...) \@@ -728,6 +729,9 @@ void kvm_set_pfn_dirty(kvm_pfn_t pfn);void kvm_set_pfn_accessed (kvm_pfn_t pfn);void kvm_get_pfn (kvm_pfn_t pfn); +int copy_from_guest (void *data,unsigned long hva,int len,bool protected); +int copy_to_guest (unsigned long hva,const void *data,int len,bool protected); +void kvm_release_pfn (kvm_pfn_t pfn,bool dirty,struct gfn_to_pfn_cache *cache);int kvm_read_guest_page (struct kvm *kvm,gfn_t gfn,void *data,int offset,int len);virt/kvm/kvm_main.c +8
-0@@ -2313,19 +2313,70 @@ static int next_segment(unsigned long len, int offset)return len; } +int copy_from_guest (void *data,unsigned long hva,int len,bool protected) +{ +int offset =offset_in_page (hva); +struct page *page; +int npages, seg; + +if (!protected) +return __copy_from_user (data, (void __user *)hva, len);
virt/kvm/kvm_main.c static int next_segment(unsigned long len, int offset) · line 2333
Quoted 10 lines
virt/kvm/kvm_main.c +10
-0+ +might_fault (); +kasan_check_write (data, len); +check_object_size (data, len, false); + +while ((seg =next_segment (len, offset)) !=0 ) { + npages =get_user_pages_unlocked (hva,1 , &page,0 ); +if (npages !=1 ) +return -EFAULT ; +memcpy (data,page_address (page) + offset, seg);
Hi Kirill!
OK, so the copy_from_guest() is a read-only case for gup, which I think is safe from a gup/pup + filesystem point of view, but see below about copy_to_guest()...
Quoted 14 lines
virt/kvm/kvm_main.c +14
-0+put_page (page); + len -= seg; + hva += seg; + offset =0 ; + } + +return 0 ; +} + +int copy_to_guest (unsigned long hva,const void *data,int len,bool protected) +{ +int offset =offset_in_page (hva); +struct page *page; +int npages, seg;
virt/kvm/kvm_main.c static int next_segment(unsigned long len, int offset) · line 2357
Quoted 10 lines
virt/kvm/kvm_main.c +10
-0+ +if (!protected) +return __copy_to_user ((void __user *)hva, data, len); + +might_fault (); +kasan_check_read (data, len); +check_object_size (data, len, true); + +while ((seg =next_segment (len, offset)) !=0 ) { + npages =get_user_pages_unlocked (hva,1 , &page,FOLL_WRITE );
Should copy_to_guest() use pin_user_pages_unlocked() instead of gup_unlocked? We wrote a "Case 5" in Documentation/core-api/pin_user_pages.rst, just for this situation, I think:
CASE 5: Pinning in order to write to the data within the page
-------------------------------------------------------------
Even though neither DMA nor Direct IO is involved, just a simple case of "pin, write to a page's data, unpin" can cause a problem. Case 5 may be considered a superset of Case 1, plus Case 2, plus anything that invokes that pattern. In other words, if the code is neither Case 1 nor Case 2, it may still require FOLL_PIN, for patterns like this:
Correct (uses FOLL_PIN calls):
pin_user_pages()
write to the data within the pages
unpin_user_pages()
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Quoted 122 lines
virt/kvm/kvm_main.c +29
-19+if (npages !=1 ) +return -EFAULT ; +memcpy (page_address (page) + offset, data, seg); +put_page (page); + len -= seg; + hva += seg; + offset =0 ; + } + +return 0 ; +} +static int __kvm_read_guest_page (struct kvm_memory_slot *slot,gfn_t gfn, -void *data,int offset,int len) +void *data,int offset,int len, +bool protected) { -int r;unsigned long addr; addr =gfn_to_hva_memslot_prot (slot, gfn,NULL );if (kvm_is_error_hva (addr))return -EFAULT ; - r =__copy_from_user (data, (void __user *)addr + offset, len); -if (r) -return -EFAULT ; -return 0 ; +return copy_from_guest (data, addr + offset, len, protected); } int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,@@ -2333,7 +2384,8 @@ int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset, {struct kvm_memory_slot *slot =gfn_to_memslot (kvm, gfn); -return __kvm_read_guest_page (slot, gfn, data, offset, len); +return __kvm_read_guest_page (slot, gfn, data, offset, len, + kvm->mem_protected); }EXPORT_SYMBOL_GPL (kvm_read_guest_page);@@ -2342,7 +2394,8 @@ int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data, {struct kvm_memory_slot *slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn); -return __kvm_read_guest_page (slot, gfn, data, offset, len); +return __kvm_read_guest_page (slot, gfn, data, offset, len, + vcpu->kvm->mem_protected); }EXPORT_SYMBOL_GPL (kvm_vcpu_read_guest_page);@@ -2415,7 +2468,8 @@ int kvm_vcpu_read_guest_atomic(struct kvm_vcpu *vcpu, gpa_t gpa,EXPORT_SYMBOL_GPL (kvm_vcpu_read_guest_atomic);static int __kvm_write_guest_page (struct kvm_memory_slot *memslot,gfn_t gfn, -const void *data,int offset,int len) +const void *data,int offset,int len, +bool protected) {int r;unsigned long addr;@@ -2423,7 +2477,8 @@ static int __kvm_write_guest_page(struct kvm_memory_slot *memslot, gfn_t gfn, addr =gfn_to_hva_memslot (memslot, gfn);if (kvm_is_error_hva (addr))return -EFAULT ; - r =__copy_to_user ((void __user *)addr + offset, data, len); + + r =copy_to_guest (addr + offset, data, len, protected);if (r)return -EFAULT ;mark_page_dirty_in_slot (memslot, gfn);@@ -2435,7 +2490,8 @@ int kvm_write_guest_page(struct kvm *kvm, gfn_t gfn, {struct kvm_memory_slot *slot =gfn_to_memslot (kvm, gfn); -return __kvm_write_guest_page (slot, gfn, data, offset, len); +return __kvm_write_guest_page (slot, gfn, data, offset, len, + kvm->mem_protected); }EXPORT_SYMBOL_GPL (kvm_write_guest_page);@@ -2444,7 +2500,8 @@ int kvm_vcpu_write_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, {struct kvm_memory_slot *slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn); -return __kvm_write_guest_page (slot, gfn, data, offset, len); +return __kvm_write_guest_page (slot, gfn, data, offset, len, + vcpu->kvm->mem_protected); }EXPORT_SYMBOL_GPL (kvm_vcpu_write_guest_page);@@ -2560,7 +2617,7 @@ int kvm_write_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,if (unlikely (!ghc->memslot))return kvm_write_guest (kvm, gpa, data, len); - r =__copy_to_user ((void __user *)ghc->hva + offset, data, len); + r =copy_to_guest (ghc->hva + offset, data, len, kvm->mem_protected);if (r)return -EFAULT ;mark_page_dirty_in_slot (ghc->memslot, gpa >>PAGE_SHIFT );@@ -2581,7 +2638,6 @@ int kvm_read_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc, unsigned long len) {struct kvm_memslots *slots =kvm_memslots (kvm); -int r;gpa_t gpa = ghc->gpa + offset;BUG_ON (len + offset > ghc->len);@@ -2597,11 +2653,7 @@ int kvm_read_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,if (unlikely (!ghc->memslot))return kvm_read_guest (kvm, gpa, data, len); - r =__copy_from_user (data, (void __user *)ghc->hva + offset, len); -if (r) -return -EFAULT ; - -return 0 ; +return copy_from_guest (data, ghc->hva + offset, len, kvm->mem_protected); }EXPORT_SYMBOL_GPL (kvm_read_guest_offset_cached);
Re: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory
Quoted 100 lines · John Hubbard
Quoted 44 lines · Kirill A. Shutemov
New helpers copy_from_guest()/copy_to_guest() to be used if KVM memory protection feature is enabled.
Signed-off-by Kirill A. Shutemov <[email protected]>--- include/linux/kvm_host.h | 4 ++ virt/kvm/kvm_main.c | 90 +++++++++++++++++++++++++++++++--------- 2 files changed, 75 insertions(+), 19 deletions(-)include/linux/kvm_host.h +4
-0@@ -504,6 +504,7 @@ struct kvm {struct srcu_struct irq_srcu;pid_t userspace_pid;unsigned int max_halt_poll_ns; +bool mem_protected; };#define kvm_err (fmt, ...) \@@ -728,6 +729,9 @@ void kvm_set_pfn_dirty(kvm_pfn_t pfn);void kvm_set_pfn_accessed (kvm_pfn_t pfn);void kvm_get_pfn (kvm_pfn_t pfn); +int copy_from_guest (void *data,unsigned long hva,int len,bool protected); +int copy_to_guest (unsigned long hva,const void *data,int len,bool protected); +void kvm_release_pfn (kvm_pfn_t pfn,bool dirty,struct gfn_to_pfn_cache *cache);int kvm_read_guest_page (struct kvm *kvm,gfn_t gfn,void *data,int offset,int len);virt/kvm/kvm_main.c +8
-0@@ -2313,19 +2313,70 @@ static int next_segment(unsigned long len, int offset)return len; } +int copy_from_guest (void *data,unsigned long hva,int len,bool protected) +{ +int offset =offset_in_page (hva); +struct page *page; +int npages, seg; + +if (!protected) +return __copy_from_user (data, (void __user *)hva, len);virt/kvm/kvm_main.c static int next_segment(unsigned long len, int offset) · line 2332
Quoted 10 lines
virt/kvm/kvm_main.c +10
-0+ +might_fault (); +kasan_check_write (data, len); +check_object_size (data, len, false); + +while ((seg =next_segment (len, offset)) !=0 ) { + npages =get_user_pages_unlocked (hva,1 , &page,0 ); +if (npages !=1 ) +return -EFAULT ; +memcpy (data,page_address (page) + offset, seg);Hi Kirill!
OK, so the
copy_from_guest()is a read-only case for gup, which I think is safe from a gup/pup + filesystem point of view, but see below aboutcopy_to_guest()...Quoted 14 lines
virt/kvm/kvm_main.c +14
-0+put_page (page); + len -= seg; + hva += seg; + offset =0 ; + } + +return 0 ; +} + +int copy_to_guest (unsigned long hva,const void *data,int len,bool protected) +{ +int offset =offset_in_page (hva); +struct page *page; +int npages, seg;virt/kvm/kvm_main.c static int next_segment(unsigned long len, int offset) · line 2357
Quoted 10 lines
virt/kvm/kvm_main.c +10
-0+ +if (!protected) +return __copy_to_user ((void __user *)hva, data, len); + +might_fault (); +kasan_check_read (data, len); +check_object_size (data, len, true); + +while ((seg =next_segment (len, offset)) !=0 ) { + npages =get_user_pages_unlocked (hva,1 , &page,FOLL_WRITE );Should
copy_to_guest()usepin_user_pages_unlocked()instead ofgup_unlocked? We wrote a "Case 5" inDocumentation/core-api/pin_user_pages.rst, just for this situation, I think:CASE 5: Pinning in order to write to the data within the page
-------------------------------------------------------------
Even though neither DMA nor Direct IO is involved, just a simple case of "pin, write to a page's data, unpin" can cause a problem. Case 5 may be considered a superset of Case 1, plus Case 2, plus anything that invokes that pattern. In other words, if the code is neither Case 1 nor Case 2, it may still requireFOLL_PIN, for patterns like this:
Quoted 4 lines
Correct (uses
FOLL_PINcalls):
pin_user_pages()
write to the data within the pages
unpin_user_pages()
Right. I didn't internalize changes in GUP interface yet. Will update.
Signature · 1 line
Kirill A. Shutemov
Re: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory
In reply to John HubbardQuoted 13 lines · John Hubbard
Should
copy_to_guest()usepin_user_pages_unlocked()instead ofgup_unlocked? We wrote a "Case 5" inDocumentation/core-api/pin_user_pages.rst, just for this situation, I think:CASE 5: Pinning in order to write to the data within the page
-------------------------------------------------------------
Even though neither DMA nor Direct IO is involved, just a simple case of "pin, write to a page's data, unpin" can cause a problem. Case 5 may be considered a superset of Case 1, plus Case 2, plus anything that invokes that pattern. In other words, if the code is neither Case 1 nor Case 2, it may still requireFOLL_PIN, for patterns like this:
Quoted 4 lines
Correct (uses
FOLL_PINcalls):
pin_user_pages()
write to the data within the pages
unpin_user_pages()
Case 5 is crap though. That bug should have been fixed by getting the locking right. ie:
get_user_pages_fast(); lock_page(); kmap(); set_bit(); kunmap(); set_page_dirty() unlock_page();
I should have vetoed that patch at the time, but I was busy with other things.
John Hubbard
reviewerRe: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory
Quoted 30 lines · Matthew Wilcox
Quoted 13 lines · John Hubbard
Should
copy_to_guest()usepin_user_pages_unlocked()instead ofgup_unlocked? We wrote a "Case 5" inDocumentation/core-api/pin_user_pages.rst, just for this situation, I think:CASE 5: Pinning in order to write to the data within the page
-------------------------------------------------------------
Even though neither DMA nor Direct IO is involved, just a simple case of "pin, write to a page's data, unpin" can cause a problem. Case 5 may be considered a superset of Case 1, plus Case 2, plus anything that invokes that pattern. In other words, if the code is neither Case 1 nor Case 2, it may still requireFOLL_PIN, for patterns like this:Quoted 4 lines
Correct (uses
FOLL_PINcalls):
pin_user_pages()
write to the data within the pages
unpin_user_pages()Case 5 is crap though. That bug should have been fixed by getting the locking right. ie:
get_user_pages_fast(); lock_page(); kmap(); set_bit(); kunmap(); set_page_dirty() unlock_page();
Quoted 2 lines
I should have vetoed that patch at the time, but I was busy with other things.
It does seem like lock_page() is better, for now at least, because it forces the kind of synchronization with file system writeback that is still yet to be implemented for pin_user_pages().
Long term though, Case 5 provides an alternative way to do this pattern--without using lock_page(). Also, note that Case 5, *in general*, need not be done page-at-a-time, unlike the lock_page() approach. Therefore, Case 5 might potentially help at some call sites, either for deadlock avoidance or performance improvements.
In other words, once the other half of the pin_user_pages() plan is implemented, either of these approaches should work.
Or, are you thinking that there is never a situation in which Case 5 is valid?
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Re: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory
Quoted 43 lines · John Hubbard
Quoted 30 lines · Matthew Wilcox
Quoted 13 lines · John Hubbard
Should
copy_to_guest()usepin_user_pages_unlocked()instead ofgup_unlocked? We wrote a "Case 5" inDocumentation/core-api/pin_user_pages.rst, just for this situation, I think:CASE 5: Pinning in order to write to the data within the page
-------------------------------------------------------------
Even though neither DMA nor Direct IO is involved, just a simple case of "pin, write to a page's data, unpin" can cause a problem. Case 5 may be considered a superset of Case 1, plus Case 2, plus anything that invokes that pattern. In other words, if the code is neither Case 1 nor Case 2, it may still requireFOLL_PIN, for patterns like this:Quoted 4 lines
Correct (uses
FOLL_PINcalls):
pin_user_pages()
write to the data within the pages
unpin_user_pages()Case 5 is crap though. That bug should have been fixed by getting the locking right. ie:
get_user_pages_fast(); lock_page(); kmap(); set_bit(); kunmap(); set_page_dirty() unlock_page();Quoted 1 line
I should have vetoed that patch at the time, but I was busy with other things.
It does seem like
lock_page()is better, for now at least, because it forces the kind of synchronization with file system writeback that is still yet to be implemented forpin_user_pages().Long term though, Case 5 provides an alternative way to do this pattern--without using
lock_page(). Also, note that Case 5, *in general*, need not be done page-at-a-time, unlike thelock_page()approach. Therefore, Case 5 might potentially help at some call sites, either for deadlock avoidance or performance improvements.
Quoted 5 lines
In other words, once the other half of the
pin_user_pages()plan is implemented, either of these approaches should work.Or, are you thinking that there is never a situation in which Case 5 is valid?
I don't think the page pinning approach is ever valid. For file mappings, the infiniband registration should take a lease on the inode, blocking truncation. DAX shouldn't be using struct pages at all, so there shouldn't be anything there to pin.
It's been five years since DAX was merged, and page pinning still doesn't work. How much longer before the people who are pushing it realise that it's fundamentally flawed?
John Hubbard
reviewerRe: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory
Quoted 44 lines · Matthew Wilcox
Quoted 43 lines · John Hubbard
Quoted 30 lines · Matthew Wilcox
Quoted 13 lines · John Hubbard
Should
copy_to_guest()usepin_user_pages_unlocked()instead ofgup_unlocked? We wrote a "Case 5" inDocumentation/core-api/pin_user_pages.rst, just for this situation, I think:CASE 5: Pinning in order to write to the data within the page
-------------------------------------------------------------
Even though neither DMA nor Direct IO is involved, just a simple case of "pin, write to a page's data, unpin" can cause a problem. Case 5 may be considered a superset of Case 1, plus Case 2, plus anything that invokes that pattern. In other words, if the code is neither Case 1 nor Case 2, it may still requireFOLL_PIN, for patterns like this:Quoted 4 lines
Correct (uses
FOLL_PINcalls):
pin_user_pages()
write to the data within the pages
unpin_user_pages()Case 5 is crap though. That bug should have been fixed by getting the locking right. ie:
get_user_pages_fast(); lock_page(); kmap(); set_bit(); kunmap(); set_page_dirty() unlock_page();Quoted 1 line
I should have vetoed that patch at the time, but I was busy with other things.
It does seem like
lock_page()is better, for now at least, because it forces the kind of synchronization with file system writeback that is still yet to be implemented forpin_user_pages().Long term though, Case 5 provides an alternative way to do this pattern--without using
lock_page(). Also, note that Case 5, *in general*, need not be done page-at-a-time, unlike thelock_page()approach. Therefore, Case 5 might potentially help at some call sites, either for deadlock avoidance or performance improvements.
Quoted 7 lines
Quoted 5 lines
In other words, once the other half of the
pin_user_pages()plan is implemented, either of these approaches should work.Or, are you thinking that there is never a situation in which Case 5 is valid?
I don't think the page pinning approach is ever valid. For file
Could you qualify that? Surely you don't mean that the entire pin_user_pages story is a waste of time--I would have expected you to make more noise earlier if you thought that, yes?
Quoted 8 lines
mappings, the infiniband registration should take a lease on the inode, blocking truncation. DAX shouldn't be using struct pages at all, so there shouldn't be anything there to pin.
It's been five years since DAX was merged, and page pinning still doesn't work. How much longer before the people who are pushing it realise that it's fundamentally flawed?
Is this a separate rant about *only* DAX, or is general RDMA in your sights too? :)
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Re: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory
Quoted 6 lines · John Hubbard
Quoted 1 line · Matthew Wilcox
I don't think the page pinning approach is ever valid. For file
Could you qualify that? Surely you don't mean that the entire
pin_user_pagesstory is a waste of time--I would have expected you to make more noise earlier if you thought that, yes?
I do think page pinning is the wrong approach for everything. I did say so at the time, and I continue to say so when the opportunity presents itself. But shouting about it constantly only annoys people, so I don't generally bother. I have other things to work on, and they're productive, so I don't need to spend my time arguing.
Quoted 6 lines
Quoted 3 lines
It's been five years since DAX was merged, and page pinning still doesn't work. How much longer before the people who are pushing it realise that it's fundamentally flawed?
Is this a separate rant about *only* DAX, or is general RDMA in your sights too? :)
This is a case where it's not RDMA's _fault_ that there's no good API for it to do what it needs to do. There's a lot of work needed to wean Linux device drivers off their assumption that there's a struct page for every byte of memory.
Re: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory
Quoted 11 lines · Matthew Wilcox
Quoted 6 lines
Quoted 3 lines
It's been five years since DAX was merged, and page pinning still doesn't work. How much longer before the people who are pushing it realise that it's fundamentally flawed?
Is this a separate rant about *only* DAX, or is general RDMA in your sights too? :)
This is a case where it's not RDMA's _fault_ that there's no good API for it to do what it needs to do. There's a lot of work needed to wean Linux device drivers off their assumption that there's a struct page for every byte of memory.
People who care seem to have just given up and are using RDMA ODP, so I'm not optimistic this DAX issue will ever be solved. I've also almost removed all the struct page references from this flow in RDMA, so if there is some way that helps it is certainly doable.
Regardless of DAX the pinning indication is now being used during fork() for some good reasons, and seems to make sense in other use cases. It just doesn't seem like a way to solve the DAX issue.
More or less it seems to mean that pages pinned cannot be write protected and more broadly the kernel should not change the PTEs for those pages independently of the application. ie the more agressive COW on fork() caused data corruption regressions...
I wonder if the point here is that some page owners can't/won't support DMA pinning and should just be blocked completely for them.
I'd much rather have write access pin_user_pages() just fail than oops the kernel on ext4 owned VMAs, for instance.
Jason
John Hubbard
reviewerRe: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory
In reply to Matthew WilcoxQuoted 9 lines · Matthew Wilcox
Quoted 6 lines · John Hubbard
Quoted 1 line · Matthew Wilcox
I don't think the page pinning approach is ever valid. For file
Could you qualify that? Surely you don't mean that the entire
pin_user_pagesstory is a waste of time--I would have expected you to make more noise earlier if you thought that, yes?I do think page pinning is the wrong approach for everything. I did say
Not *everything*, just "pinning for DMA", right? Because I don't recall any viable solutions for Direct IO that avoided gup/pup!
Also, back to Case 5: I *could* create a small patchset to change over the very few Case 5 call sites to use "gup, lock_page(), write to page...etc", instead of pup. And also, update pin_user_pages.rst to recommend that approach in similar situations. After all, it's not really a long-term DMA pin, which is really what pin_user_pages*() is intended for.
Would that be something you'd like to see happen? It's certainly easy enough to fix that up. And your retroactive NAK is sufficient motivation to do so.
Quoted 4 lines
so at the time, and I continue to say so when the opportunity presents itself. But shouting about it constantly only annoys people, so I don't generally bother. I have other things to work on, and they're productive, so I don't need to spend my time arguing.
Sure. As a practical matter, I've assumed that page pinning is not going to go away any time soon, so I want it to work properly while it's here. But if there is a viable way to eventually replace dma-pinning with something better, then let's keep thinking about it. I'm glad to help in that area.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Re: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory
In reply to Kirill A. ShutemovQuoted 51 lines · Kirill A. Shutemov
New helpers copy_from_guest()/copy_to_guest() to be used if KVM memory protection feature is enabled.
Signed-off-by Kirill A. Shutemov <[email protected]>--- include/linux/kvm_host.h | 4 ++ virt/kvm/kvm_main.c | 90 +++++++++++++++++++++++++++++++--------- 2 files changed, 75 insertions(+), 19 deletions(-)include/linux/kvm_host.h +4
-0@@ -504,6 +504,7 @@ struct kvm {struct srcu_struct irq_srcu;pid_t userspace_pid;unsigned int max_halt_poll_ns; +bool mem_protected; };#define kvm_err (fmt, ...) \@@ -728,6 +729,9 @@ void kvm_set_pfn_dirty(kvm_pfn_t pfn);void kvm_set_pfn_accessed (kvm_pfn_t pfn);void kvm_get_pfn (kvm_pfn_t pfn); +int copy_from_guest (void *data,unsigned long hva,int len,bool protected); +int copy_to_guest (unsigned long hva,const void *data,int len,bool protected); +void kvm_release_pfn (kvm_pfn_t pfn,bool dirty,struct gfn_to_pfn_cache *cache);int kvm_read_guest_page (struct kvm *kvm,gfn_t gfn,void *data,int offset,int len);virt/kvm/kvm_main.c +12
-0@@ -2313,19 +2313,70 @@ static int next_segment(unsigned long len, int offset)return len; } +int copy_from_guest (void *data,unsigned long hva,int len,bool protected) +{ +int offset =offset_in_page (hva); +struct page *page; +int npages, seg; + +if (!protected) +return __copy_from_user (data, (void __user *)hva, len); + +might_fault (); +kasan_check_write (data, len); +check_object_size (data, len, false);
virt/kvm/kvm_main.c static int next_segment(unsigned long len, int offset) · line 2337
Quoted 10 lines
virt/kvm/kvm_main.c +10
-0+ +while ((seg =next_segment (len, offset)) !=0 ) { + npages =get_user_pages_unlocked (hva,1 , &page,0 ); +if (npages !=1 ) +return -EFAULT ; +memcpy (data,page_address (page) + offset, seg); +put_page (page); + len -= seg; + hva += seg; + offset =0 ;
Why is data not updated on each iteration?
Quoted 17 lines
virt/kvm/kvm_main.c +17
-0+ } + +return 0 ; +} + +int copy_to_guest (unsigned long hva,const void *data,int len,bool protected) +{ +int offset =offset_in_page (hva); +struct page *page; +int npages, seg; + +if (!protected) +return __copy_to_user ((void __user *)hva, data, len); + +might_fault (); +kasan_check_read (data, len); +check_object_size (data, len, true);
virt/kvm/kvm_main.c static int next_segment(unsigned long len, int offset) · line 2364
Quoted 10 lines
virt/kvm/kvm_main.c +10
-0+ +while ((seg =next_segment (len, offset)) !=0 ) { + npages =get_user_pages_unlocked (hva,1 , &page,FOLL_WRITE ); +if (npages !=1 ) +return -EFAULT ; +memcpy (page_address (page) + offset, data, seg); +put_page (page); + len -= seg; + hva += seg; + offset =0 ;
Same question? Doesn't this result in *data being copied to multiple pages?
Ira
Quoted 118 lines
virt/kvm/kvm_main.c +22
-19+ } + +return 0 ; +} +static int __kvm_read_guest_page (struct kvm_memory_slot *slot,gfn_t gfn, -void *data,int offset,int len) +void *data,int offset,int len, +bool protected) { -int r;unsigned long addr; addr =gfn_to_hva_memslot_prot (slot, gfn,NULL );if (kvm_is_error_hva (addr))return -EFAULT ; - r =__copy_from_user (data, (void __user *)addr + offset, len); -if (r) -return -EFAULT ; -return 0 ; +return copy_from_guest (data, addr + offset, len, protected); } int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,@@ -2333,7 +2384,8 @@ int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset, {struct kvm_memory_slot *slot =gfn_to_memslot (kvm, gfn); -return __kvm_read_guest_page (slot, gfn, data, offset, len); +return __kvm_read_guest_page (slot, gfn, data, offset, len, + kvm->mem_protected); }EXPORT_SYMBOL_GPL (kvm_read_guest_page);@@ -2342,7 +2394,8 @@ int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data, {struct kvm_memory_slot *slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn); -return __kvm_read_guest_page (slot, gfn, data, offset, len); +return __kvm_read_guest_page (slot, gfn, data, offset, len, + vcpu->kvm->mem_protected); }EXPORT_SYMBOL_GPL (kvm_vcpu_read_guest_page);@@ -2415,7 +2468,8 @@ int kvm_vcpu_read_guest_atomic(struct kvm_vcpu *vcpu, gpa_t gpa,EXPORT_SYMBOL_GPL (kvm_vcpu_read_guest_atomic);static int __kvm_write_guest_page (struct kvm_memory_slot *memslot,gfn_t gfn, -const void *data,int offset,int len) +const void *data,int offset,int len, +bool protected) {int r;unsigned long addr;@@ -2423,7 +2477,8 @@ static int __kvm_write_guest_page(struct kvm_memory_slot *memslot, gfn_t gfn, addr =gfn_to_hva_memslot (memslot, gfn);if (kvm_is_error_hva (addr))return -EFAULT ; - r =__copy_to_user ((void __user *)addr + offset, data, len); + + r =copy_to_guest (addr + offset, data, len, protected);if (r)return -EFAULT ;mark_page_dirty_in_slot (memslot, gfn);@@ -2435,7 +2490,8 @@ int kvm_write_guest_page(struct kvm *kvm, gfn_t gfn, {struct kvm_memory_slot *slot =gfn_to_memslot (kvm, gfn); -return __kvm_write_guest_page (slot, gfn, data, offset, len); +return __kvm_write_guest_page (slot, gfn, data, offset, len, + kvm->mem_protected); }EXPORT_SYMBOL_GPL (kvm_write_guest_page);@@ -2444,7 +2500,8 @@ int kvm_vcpu_write_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, {struct kvm_memory_slot *slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn); -return __kvm_write_guest_page (slot, gfn, data, offset, len); +return __kvm_write_guest_page (slot, gfn, data, offset, len, + vcpu->kvm->mem_protected); }EXPORT_SYMBOL_GPL (kvm_vcpu_write_guest_page);@@ -2560,7 +2617,7 @@ int kvm_write_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,if (unlikely (!ghc->memslot))return kvm_write_guest (kvm, gpa, data, len); - r =__copy_to_user ((void __user *)ghc->hva + offset, data, len); + r =copy_to_guest (ghc->hva + offset, data, len, kvm->mem_protected);if (r)return -EFAULT ;mark_page_dirty_in_slot (ghc->memslot, gpa >>PAGE_SHIFT );@@ -2581,7 +2638,6 @@ int kvm_read_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc, unsigned long len) {struct kvm_memslots *slots =kvm_memslots (kvm); -int r;gpa_t gpa = ghc->gpa + offset;BUG_ON (len + offset > ghc->len);@@ -2597,11 +2653,7 @@ int kvm_read_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,if (unlikely (!ghc->memslot))return kvm_read_guest (kvm, gpa, data, len); - r =__copy_from_user (data, (void __user *)ghc->hva + offset, len); -if (r) -return -EFAULT ; - -return 0 ; +return copy_from_guest (data, ghc->hva + offset, len, kvm->mem_protected); }EXPORT_SYMBOL_GPL (kvm_read_guest_offset_cached);Signature · 1 line
2.26.2
Re: [RFCv2 08/16] KVM: Use GUP instead of copy_from/to_user() to access guest memory
Quoted 56 lines · Ira Weiny
Quoted 55 lines · Kirill A. Shutemov
New helpers copy_from_guest()/copy_to_guest() to be used if KVM memory protection feature is enabled.
Signed-off-by Kirill A. Shutemov <[email protected]>--- include/linux/kvm_host.h | 4 ++ virt/kvm/kvm_main.c | 90 +++++++++++++++++++++++++++++++--------- 2 files changed, 75 insertions(+), 19 deletions(-)include/linux/kvm_host.h +4
-0@@ -504,6 +504,7 @@ struct kvm {struct srcu_struct irq_srcu;pid_t userspace_pid;unsigned int max_halt_poll_ns; +bool mem_protected; };#define kvm_err (fmt, ...) \@@ -728,6 +729,9 @@ void kvm_set_pfn_dirty(kvm_pfn_t pfn);void kvm_set_pfn_accessed (kvm_pfn_t pfn);void kvm_get_pfn (kvm_pfn_t pfn); +int copy_from_guest (void *data,unsigned long hva,int len,bool protected); +int copy_to_guest (unsigned long hva,const void *data,int len,bool protected); +void kvm_release_pfn (kvm_pfn_t pfn,bool dirty,struct gfn_to_pfn_cache *cache);int kvm_read_guest_page (struct kvm *kvm,gfn_t gfn,void *data,int offset,int len);virt/kvm/kvm_main.c +16
-0@@ -2313,19 +2313,70 @@ static int next_segment(unsigned long len, int offset)return len; } +int copy_from_guest (void *data,unsigned long hva,int len,bool protected) +{ +int offset =offset_in_page (hva); +struct page *page; +int npages, seg; + +if (!protected) +return __copy_from_user (data, (void __user *)hva, len); + +might_fault (); +kasan_check_write (data, len); +check_object_size (data, len, false); + +while ((seg =next_segment (len, offset)) !=0 ) { + npages =get_user_pages_unlocked (hva,1 , &page,0 ); +if (npages !=1 )
Quoted 8 lines
virt/kvm/kvm_main.c static int next_segment(unsigned long len, int offset) · line 2337
Quoted 6 lines
virt/kvm/kvm_main.c +6
-0+return -EFAULT ; +memcpy (data,page_address (page) + offset, seg); +put_page (page); + len -= seg; + hva += seg; + offset =0 ;Why is data not updated on each iteration?
Ouch. I guess no caller actually steps over page boundary. Will fix.
Signature · 1 line
Kirill A. Shutemov
RFC v2 9/16 KVM: mm: Introduce VM_KVM_PROTECTED
In reply to Kirill A. ShutemovThe new VMA flag that indicate a VMA that is not accessible to userspace but usable by kernel with GUP if FOLL_KVM is specified.
The FOLL_KVM is only used in the KVM code. The code has to know how to deal with such pages.
--- include/linux/mm.h | 8 ++++++++ mm/gup.c | 20 ++++++++++++++++---- mm/huge_memory.c | 20 ++++++++++++++++---- mm/memory.c | 3 +++ mm/mmap.c | 3 +++ virt/kvm/async_pf.c | 2 +- virt/kvm/kvm_main.c | 9 +++++---- 7 files changed, 52 insertions(+), 13 deletions(-)
include/linux/mm.h +8 -0
@@ -342,6 +342,8 @@ extern unsigned int kobjsize(const void *objp);# define VM_MAPPED_COPY VM_ARCH_1/* T if mapped copy of data (nommu mmap) */ #endif +#define VM_KVM_PROTECTED 0 +#ifndef VM_GROWSUP # define VM_GROWSUP VM_NONE#endif
@@ -658,6 +660,11 @@ static inline bool vma_is_accessible(struct vm_area_struct *vma)return vma->vm_flags &VM_ACCESS_FLAGS ; } +static inline bool vma_is_kvm_protected (struct vm_area_struct *vma) +{ +return vma->vm_flags &VM_KVM_PROTECTED ; +} +#ifdef CONFIG_SHMEM /* * The vma_is_shmem is not inline because it is used only by slow
@@ -2766,6 +2773,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,#define FOLL_SPLIT_PMD 0x20000/* split huge pmd before returning */ #define FOLL_PIN 0x40000/* pages must be released via unpin_user_page */ #define FOLL_FAST_ONLY 0x80000/* gup_fast: prevent fall-back to slow gup */ +#define FOLL_KVM 0x100000/* access to VM_KVM_PROTECTED VMAs */ /* * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each
mm/gup.c +16 -4
@@ -384,10 +384,19 @@ static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address,* FOLL_FORCE can write to even unwritable pte's, but only * after we've gone through a COW cycle and they are dirty. */ -static inline bool can_follow_write_pte (pte_t pte,unsigned int flags) +static inline bool can_follow_write_pte (struct vm_area_struct *vma, +pte_t pte,unsigned int flags) { -return pte_write (pte) || - ((flags &FOLL_FORCE ) && (flags &FOLL_COW ) &&pte_dirty (pte)); +if (pte_write (pte)) +return true; + +if ((flags &FOLL_FORCE ) && (flags &FOLL_COW ) &&pte_dirty (pte)) +return true; + +if (!vma_is_kvm_protected (vma) || !(vma->vm_flags &VM_WRITE )) +return false; + +return (vma->vm_flags &VM_SHARED ) ||page_mapcount (pte_page (pte)) ==1 ; } static struct page *follow_page_pte(struct vm_area_struct *vma,
@@ -430,7 +439,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, }if ((flags &FOLL_NUMA ) &&pte_protnone (pte)) goto no_page; -if ((flags &FOLL_WRITE ) && !can_follow_write_pte (pte, flags)) { +if ((flags &FOLL_WRITE ) && !can_follow_write_pte (vma, pte, flags)) {pte_unmap_unlock (ptep, ptl);return NULL ; }
@@ -750,6 +759,9 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, ctx->page_mask =0 ; +if (vma_is_kvm_protected (vma) && (flags &FOLL_KVM )) + flags &= ~FOLL_NUMA ; +/* make this handle hugepd */ page =follow_huge_addr (mm, address, flags &FOLL_WRITE );if (!IS_ERR (page)) {
mm/huge_memory.c +16 -4
@@ -1322,10 +1322,19 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd)* FOLL_FORCE can write to even unwritable pmd's, but only * after we've gone through a COW cycle and they are dirty. */ -static inline bool can_follow_write_pmd (pmd_t pmd,unsigned int flags) +static inline bool can_follow_write_pmd (struct vm_area_struct *vma, +pmd_t pmd,unsigned int flags) { -return pmd_write (pmd) || - ((flags &FOLL_FORCE ) && (flags &FOLL_COW ) &&pmd_dirty (pmd)); +if (pmd_write (pmd)) +return true; + +if ((flags &FOLL_FORCE ) && (flags &FOLL_COW ) &&pmd_dirty (pmd)) +return true; + +if (!vma_is_kvm_protected (vma) || !(vma->vm_flags &VM_WRITE )) +return false; + +return (vma->vm_flags &VM_SHARED ) ||page_mapcount (pmd_page (pmd)) ==1 ; } struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
@@ -1338,7 +1347,7 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,assert_spin_locked (pmd_lockptr (mm, pmd)); -if (flags &FOLL_WRITE && !can_follow_write_pmd (*pmd, flags)) +if (flags &FOLL_WRITE && !can_follow_write_pmd (vma, *pmd, flags)) goto out;/* Avoid dumping huge zero page */
@@ -1412,6 +1421,9 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t pmd)bool was_writable;int flags =0 ; +if (vma_is_kvm_protected (vma)) +return VM_FAULT_SIGBUS ; + vmf->ptl =pmd_lock (vma->vm_mm, vmf->pmd);if (unlikely (!pmd_same (pmd, *vmf->pmd))) goto out_unlock;
mm/memory.c +3 -0
@@ -4165,6 +4165,9 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)bool was_writable =pte_savedwrite (vmf->orig_pte);int flags =0 ; +if (vma_is_kvm_protected (vma)) +return VM_FAULT_SIGBUS ; +/* * The "pte" at this point cannot be used safely without * validation through pte_unmap_same(). It's of NUMA type but
mm/mmap.c +3 -0
@@ -112,6 +112,9 @@ pgprot_t vm_get_page_prot(unsigned long vm_flags) (VM_READ|VM_WRITE|VM_EXEC|VM_SHARED)]) | pgprot_val(arch_vm_get_page_prot(vm_flags))); +if (vm_flags &VM_KVM_PROTECTED ) + ret =PAGE_NONE ; +return arch_filter_pgprot (ret); }EXPORT_SYMBOL (vm_get_page_prot);
virt/kvm/async_pf.c +1 -1
@@ -61,7 +61,7 @@ static void async_pf_execute(struct work_struct *work)* access remotely. */ mmap_read_lock (mm); -get_user_pages_remote (mm, addr,1 ,FOLL_WRITE ,NULL ,NULL , +get_user_pages_remote (mm, addr,1 ,FOLL_WRITE |FOLL_KVM ,NULL ,NULL , &locked);if (locked)mmap_read_unlock (mm);
virt/kvm/kvm_main.c +5 -4
@@ -1794,7 +1794,7 @@ unsigned long kvm_vcpu_gfn_to_hva_prot(struct kvm_vcpu *vcpu, gfn_t gfn, bool *wstatic inline int check_user_page_hwpoison (unsigned long addr) { -int rc, flags =FOLL_HWPOISON |FOLL_WRITE ; +int rc, flags =FOLL_HWPOISON |FOLL_WRITE |FOLL_KVM ; rc =get_user_pages (addr,1 , flags,NULL ,NULL );return rc == -EHWPOISON ;
@@ -1836,7 +1836,7 @@ static bool hva_to_pfn_fast(unsigned long addr, bool write_fault,static int hva_to_pfn_slow (unsigned long addr,bool *async,bool write_fault,bool *writable,kvm_pfn_t *pfn) { -unsigned int flags =FOLL_HWPOISON ; +unsigned int flags =FOLL_HWPOISON |FOLL_KVM ;struct page *page;int npages =0 ;
@@ -2327,7 +2327,7 @@ int copy_from_guest(void *data, unsigned long hva, int len, bool protected)check_object_size (data, len, false);while ((seg =next_segment (len, offset)) !=0 ) { - npages =get_user_pages_unlocked (hva,1 , &page,0 ); + npages =get_user_pages_unlocked (hva,1 , &page,FOLL_KVM );if (npages !=1 )return -EFAULT ;memcpy (data,page_address (page) + offset, seg);
@@ -2354,7 +2354,8 @@ int copy_to_guest(unsigned long hva, const void *data, int len, bool protected)check_object_size (data, len, true);while ((seg =next_segment (len, offset)) !=0 ) { - npages =get_user_pages_unlocked (hva,1 , &page,FOLL_WRITE ); + npages =get_user_pages_unlocked (hva,1 , &page, +FOLL_WRITE |FOLL_KVM );if (npages !=1 )return -EFAULT ;memcpy (page_address (page) + offset, data, seg);
Signature · 1 line
2.26.2
Re: [RFCv2 09/16] KVM: mm: Introduce VM_KVM_PROTECTED
Quoted 8 lines · Kirill A. Shutemov
include/linux/mm.h | 8 ++++++++ mm/gup.c | 20 ++++++++++++++++---- mm/huge_memory.c | 20 ++++++++++++++++---- mm/memory.c | 3 +++ mm/mmap.c | 3 +++ virt/kvm/async_pf.c | 2 +- virt/kvm/kvm_main.c | 9 +++++---- 7 files changed, 52 insertions(+), 13 deletions(-)
There is another get_user_pages() in
paging_tmpl.h:FNAME(cmpxchg_gpte).
Also, did you leave off FOLL_KVM in gfn_to_page_many_atomic() on purpose? It looks like its only used for pte prefetch which I guess is not required.
Re: [RFCv2 09/16] KVM: mm: Introduce VM_KVM_PROTECTED
Quoted 10 lines · Rick P
Quoted 8 lines · Kirill A. Shutemov
include/linux/mm.h | 8 ++++++++ mm/gup.c | 20 ++++++++++++++++---- mm/huge_memory.c | 20 ++++++++++++++++---- mm/memory.c | 3 +++ mm/mmap.c | 3 +++ virt/kvm/async_pf.c | 2 +- virt/kvm/kvm_main.c | 9 +++++---- 7 files changed, 52 insertions(+), 13 deletions(-)
Quoted 2 lines
There is another
get_user_pages()in
paging_tmpl.h:FNAME(cmpxchg_gpte).
Right, I will look into it. I wounder why I don't step onto it.
Quoted 3 lines
Also, did you leave off
FOLL_KVMingfn_to_page_many_atomic()on purpose? It looks like its only used for pte prefetch which I guess is not required.
FOLL_FAST_ONLY is going to fail anyway. The caller has to handle it gracefully.
Signature · 1 line
Kirill A. Shutemov
RFC v2 10/16 KVM: x86: Use GUP for page walk instead of __get_user()
In reply to Kirill A. ShutemovThe user mapping doesn't have the page mapping for protected memory.
--- arch/x86/kvm/mmu/paging_tmpl.h | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/paging_tmpl.h +8 -2
@@ -397,8 +397,14 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, goto error; ptep_user = (pt_element_t __user *)((void *)host_addr + offset); -if (unlikely (__get_user (pte, ptep_user))) - goto error; +if (vcpu->kvm->mem_protected) { +if (copy_from_guest (&pte, host_addr + offset, +sizeof (pte), true)) + goto error; + }else { +if (unlikely (__get_user (pte, ptep_user))) + goto error; + } walker->ptep_user[walker->level -1 ] = ptep_user;trace_kvm_mmu_paging_element (pte, walker->level);
Signature · 1 line
2.26.2
RFC v2 11/16 KVM: Protected memory extension
In reply to Kirill A. ShutemovAdd infrastructure that handles protected memory extension.
Arch-specific code has to provide hypercalls and define non-zero VM_KVM_PROTECTED.
--- include/linux/kvm_host.h | 4 +++ virt/kvm/Kconfig | 3 ++ virt/kvm/kvm_main.c | 68 ++++++++++++++++++++++++++++++++++++++ virt/lib/Makefile | 1 + virt/lib/mem_protected.c | 71 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 147 insertions(+) create mode 100644 virt/lib/mem_protected.c
include/linux/kvm_host.h +4 -0
@@ -701,6 +701,10 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm);void kvm_arch_flush_shadow_memslot (struct kvm *kvm,struct kvm_memory_slot *slot); +int kvm_protect_all_memory (struct kvm *kvm); +int kvm_protect_memory (struct kvm *kvm, +unsigned long gfn,unsigned long npages,bool protect); +int gfn_to_page_many_atomic (struct kvm_memory_slot *slot,gfn_t gfn,struct page **pages,int nr_pages);
virt/kvm/Kconfig +3 -0
@@ -63,3 +63,6 @@ config HAVE_KVM_NO_POLL config KVM_XFER_TO_GUEST_WORK bool + +config HAVE_KVM_PROTECTED_MEMORY + bool
virt/kvm/kvm_main.c +68 -0
86 lines not on this page · raw message
virt/lib/Makefile +1 -0
@@ -1,2 +1,3 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_IRQ_BYPASS_MANAGER) += irqbypass.o +obj-$(CONFIG_HAVE_KVM_PROTECTED_MEMORY) += mem_protected.o
virt/lib/mem_protected.c new file +71 -0
71 lines not on this page · raw message
Signature · 1 line
2.26.2
Peter Zijlstra
maintainerRe: [RFCv2 11/16] KVM: Protected memory extension
Quoted 54 lines · Kirill A. Shutemov
virt/lib/mem_protected.c new file +54
-0+int __kvm_protect_memory (unsigned long start,unsigned long end,bool protect) +{ +struct mm_struct *mm = current->mm; +struct vm_area_struct *vma, *prev; +int ret; + +if (mmap_write_lock_killable (mm)) +return -EINTR ; + + ret = -ENOMEM ; + vma =find_vma (current->mm, start); +if (!vma) + goto out; + + ret = -EINVAL ; +if (vma->vm_start > start) + goto out; + +if (start > vma->vm_start) + prev = vma; +else + prev = vma->vm_prev; + + ret =0 ; +while (true) { +unsigned long newflags, tmp; + + tmp = vma->vm_end; +if (tmp > end) + tmp = end; + + newflags = vma->vm_flags; +if (protect) + newflags |=VM_KVM_PROTECTED ; +else + newflags &= ~VM_KVM_PROTECTED ; + +/* The VMA has been handled as part of other memslot */ +if (newflags == vma->vm_flags) + goto next; + + ret =mprotect_fixup (vma, &prev, start, tmp, newflags); +if (ret) + goto out; + +next: + start = tmp; +if (start < prev->vm_end) + start = prev->vm_end; + +if (start >= end) + goto out; + + vma = prev->vm_next;
virt/lib/mem_protected.c line 71
Quoted 10 lines
virt/lib/mem_protected.c new file +10
-0+if (!vma || vma->vm_start != start) { + ret = -ENOMEM ; + goto out; + } + } +out: +mmap_write_unlock (mm); +return ret; +} +EXPORT_SYMBOL_GPL (__kvm_protect_memory);
Since migration will be disabled after this; should the above not (at the very least) force compaction before proceeding to lock the pages in?
Re: [RFCv2 11/16] KVM: Protected memory extension
Quoted 60 lines · Peter Zijlstra
Quoted 59 lines · Kirill A. Shutemov
virt/lib/mem_protected.c new file +59
-0+int __kvm_protect_memory (unsigned long start,unsigned long end,bool protect) +{ +struct mm_struct *mm = current->mm; +struct vm_area_struct *vma, *prev; +int ret; + +if (mmap_write_lock_killable (mm)) +return -EINTR ; + + ret = -ENOMEM ; + vma =find_vma (current->mm, start); +if (!vma) + goto out; + + ret = -EINVAL ; +if (vma->vm_start > start) + goto out; + +if (start > vma->vm_start) + prev = vma; +else + prev = vma->vm_prev; + + ret =0 ; +while (true) { +unsigned long newflags, tmp; + + tmp = vma->vm_end; +if (tmp > end) + tmp = end; + + newflags = vma->vm_flags; +if (protect) + newflags |=VM_KVM_PROTECTED ; +else + newflags &= ~VM_KVM_PROTECTED ; + +/* The VMA has been handled as part of other memslot */ +if (newflags == vma->vm_flags) + goto next; + + ret =mprotect_fixup (vma, &prev, start, tmp, newflags); +if (ret) + goto out; + +next: + start = tmp; +if (start < prev->vm_end) + start = prev->vm_end; + +if (start >= end) + goto out; + + vma = prev->vm_next; +if (!vma || vma->vm_start != start) { + ret = -ENOMEM ; + goto out; + } + }
Quoted 8 lines
virt/lib/mem_protected.c line 71
Quoted 5 lines
virt/lib/mem_protected.c new file +5
-0+out: +mmap_write_unlock (mm); +return ret; +} +EXPORT_SYMBOL_GPL (__kvm_protect_memory);Since migration will be disabled after this; should the above not (at the very least) force compaction before proceeding to lock the pages in?
Migration has to be implemented instead, before we hit upstream.
BTW, VMs with direct device assignment pins all guest memory today. So it's not something new in the virtualization world.
Signature · 1 line
Kirill A. Shutemov
RFC v2 12/16 KVM: x86: Enabled protected memory extension
In reply to Kirill A. ShutemovWire up hypercalls for the feature and define VM_KVM_PROTECTED.
--- arch/x86/Kconfig | 1 + arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/cpuid.c | 3 ++- arch/x86/kvm/x86.c | 9 +++++++++ include/linux/mm.h | 6 ++++++ 5 files changed, 19 insertions(+), 1 deletion(-)
arch/x86/Kconfig +1 -0
@@ -807,6 +807,7 @@ config KVM_GUEST select X86_HV_CALLBACK_VECTOR select X86_MEM_ENCRYPT_COMMON select SWIOTLB + select ARCH_USES_HIGH_VMA_FLAGS default y help This option enables various optimizations for running under the KVM
arch/x86/kvm/Kconfig +1 -0
@@ -46,6 +46,7 @@ config KVM select KVM_GENERIC_DIRTYLOG_READ_PROTECT select KVM_VFIO select SRCU + select HAVE_KVM_PROTECTED_MEMORY help Support hosting fully virtualized guest machines using hardware virtualization extensions. You will need a fairly recent
arch/x86/kvm/cpuid.c +2 -1
@@ -746,7 +746,8 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) (1 <<KVM_FEATURE_PV_SEND_IPI ) | (1 <<KVM_FEATURE_POLL_CONTROL ) | (1 <<KVM_FEATURE_PV_SCHED_YIELD ) | - (1 <<KVM_FEATURE_ASYNC_PF_INT ); + (1 <<KVM_FEATURE_ASYNC_PF_INT ) | + (1 <<KVM_FEATURE_MEM_PROTECTED );if (sched_info_on ()) entry->eax |= (1 <<KVM_FEATURE_STEAL_TIME );
arch/x86/kvm/x86.c +9 -0
@@ -7752,6 +7752,15 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)kvm_sched_yield (vcpu->kvm, a0); ret =0 ;break ; +case KVM_HC_ENABLE_MEM_PROTECTED : + ret =kvm_protect_all_memory (vcpu->kvm); +break ; +case KVM_HC_MEM_SHARE : + ret =kvm_protect_memory (vcpu->kvm, a0, a1, false); +break ; +case KVM_HC_MEM_UNSHARE : + ret =kvm_protect_memory (vcpu->kvm, a0, a1, true); +break ;default : ret = -KVM_ENOSYS ;break ;
include/linux/mm.h +6 -0
@@ -304,11 +304,13 @@ extern unsigned int kobjsize(const void *objp);#define VM_HIGH_ARCH_BIT_2 34/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_3 35/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_4 36/* bit only usable on 64-bit architectures */ +#define VM_HIGH_ARCH_BIT_5 37/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_0 BIT(VM_HIGH_ARCH_BIT_0)#define VM_HIGH_ARCH_1 BIT(VM_HIGH_ARCH_BIT_1)#define VM_HIGH_ARCH_2 BIT(VM_HIGH_ARCH_BIT_2)#define VM_HIGH_ARCH_3 BIT(VM_HIGH_ARCH_BIT_3)#define VM_HIGH_ARCH_4 BIT(VM_HIGH_ARCH_BIT_4) +#define VM_HIGH_ARCH_5 BIT(VM_HIGH_ARCH_BIT_5)#endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */ #ifdef CONFIG_ARCH_HAS_PKEYS
@@ -342,7 +344,11 @@ extern unsigned int kobjsize(const void *objp);# define VM_MAPPED_COPY VM_ARCH_1/* T if mapped copy of data (nommu mmap) */ #endif +#if defined(CONFIG_X86_64 ) && defined(CONFIG_KVM ) +#define VM_KVM_PROTECTED VM_HIGH_ARCH_5 +#else #define VM_KVM_PROTECTED 0 +#endif #ifndef VM_GROWSUP # define VM_GROWSUP VM_NONE
Signature · 1 line
2.26.2
kernel test robot
build botRe: [RFCv2 12/16] KVM: x86: Enabled protected memory extension
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi "Kirill,
[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on tip/x86/core]
[also build test WARNING on v5.9]
[cannot apply to kvm/linux-next hnaz-linux-mm/master next-20201016] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130 base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 238c91115cd05c71447ea071624a4c9fe661f970 config: i386-allyesconfig (attached as .config) compiler: gcc-9 (Debian 9.3.0-15) 9.3.0 reproduce (this is a W=1 build): # https://github.com/0day-ci/linux/commit/7f6be13aae00ba3b3bd2c6ed70add1acd854c563 git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130 git checkout 7f6be13aae00ba3b3bd2c6ed70add1acd854c563 # save the attached .config to linux build tree make W=1 ARCH=i386
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
>> virt/lib/mem_protected.c:8:5: warning: no previous prototype for '__kvm_protect_memory' [-Wmissing-prototypes]
8 | int __kvm_protect_memory(unsigned long start, unsigned long end, bool protect)
| ^~~~~~~~~~~~~~~~~~~~
vim +/__kvm_protect_memory +8 virt/lib/mem_protected.c
63d2c458104bc29 Kirill A. Shutemov 2020-10-20 763d2c458104bc29 Kirill A. Shutemov 2020-10-20 @8 int __kvm_protect_memory(unsigned long start, unsigned long end, bool protect)
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
RFC v2 13/16 KVM: Rework copy_to/from_guest() to avoid direct mapping
In reply to Kirill A. ShutemovWe are going unmap guest pages from direct mapping and cannot rely on it for guest memory access. Use temporary kmap_atomic()-style mapping to access guest memory.
--- virt/kvm/kvm_main.c | 27 ++++++++++- virt/lib/mem_protected.c | 101 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 126 insertions(+), 2 deletions(-)
virt/kvm/kvm_main.c +25 -2
@@ -51,6 +51,7 @@#include <linux/io.h> #include <linux/lockdep.h> #include <linux/kthread.h> +#include <linux/pagewalk.h> #include <asm/processor.h> #include <asm/ioctl.h>
@@ -154,6 +155,12 @@ static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm);static unsigned long long kvm_createvm_count;static unsigned long long kvm_active_vms; +void *kvm_map_page_atomic (struct page *page); +void kvm_unmap_page_atomic (void *vaddr); + +int kvm_init_protected_memory (void ); +void kvm_exit_protected_memory (void ); +int __kvm_protect_memory (unsigned long start,unsigned long end,bool protect); __weak void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm,
@@ -2329,6 +2336,7 @@ int copy_from_guest(void *data, unsigned long hva, int len, bool protected)int offset =offset_in_page (hva);struct page *page;int npages, seg; +void *vaddr;if (!protected)return __copy_from_user (data, (void __user *)hva, len);
@@ -2341,7 +2349,11 @@ int copy_from_guest(void *data, unsigned long hva, int len, bool protected) npages =get_user_pages_unlocked (hva,1 , &page,FOLL_KVM );if (npages !=1 )return -EFAULT ; -memcpy (data,page_address (page) + offset, seg); + + vaddr =kvm_map_page_atomic (page); +memcpy (data, vaddr + offset, seg); +kvm_unmap_page_atomic (vaddr); +put_page (page); len -= seg; hva += seg;
@@ -2356,6 +2368,7 @@ int copy_to_guest(unsigned long hva, const void *data, int len, bool protected)int offset =offset_in_page (hva);struct page *page;int npages, seg; +void *vaddr;if (!protected)return __copy_to_user ((void __user *)hva, data, len);
@@ -2369,7 +2382,11 @@ int copy_to_guest(unsigned long hva, const void *data, int len, bool protected) FOLL_WRITE | FOLL_KVM);if (npages !=1 )return -EFAULT ; -memcpy (page_address (page) + offset, data, seg); + + vaddr =kvm_map_page_atomic (page); +memcpy (vaddr + offset, data, seg); +kvm_unmap_page_atomic (vaddr); +put_page (page); len -= seg; hva += seg;
@@ -4945,6 +4962,10 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,if (r) goto out_free; +if (IS_ENABLED (CONFIG_HAVE_KVM_PROTECTED_MEMORY ) && +kvm_init_protected_memory ()) + goto out_unreg; + kvm_chardev_ops.owner = module; kvm_vm_fops.owner = module; kvm_vcpu_fops.owner = module;
@@ -4968,6 +4989,7 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,return 0 ; out_unreg: +kvm_exit_protected_memory ();kvm_async_pf_deinit (); out_free:kmem_cache_destroy (kvm_vcpu_cache);
@@ -4989,6 +5011,7 @@ EXPORT_SYMBOL_GPL(kvm_init);void kvm_exit (void ) { +kvm_exit_protected_memory ();debugfs_remove_recursive (kvm_debugfs_dir);misc_deregister (&kvm_dev);kmem_cache_destroy (kvm_vcpu_cache);
virt/lib/mem_protected.c +101 -0
113 lines not on this page · raw message
Signature · 1 line
2.26.2
kernel test robot
build botRe: [RFCv2 13/16] KVM: Rework copy_to/from_guest() to avoid direct mapping
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi "Kirill,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/x86/core]
[also build test ERROR on v5.9]
[cannot apply to kvm/linux-next hnaz-linux-mm/master next-20201016] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130 base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 238c91115cd05c71447ea071624a4c9fe661f970 config: i386-allyesconfig (attached as .config) compiler: gcc-9 (Debian 9.3.0-15) 9.3.0 reproduce (this is a W=1 build): # https://github.com/0day-ci/linux/commit/1af02c27a796b91d58cd8b6d50d7e7b9b5564ce4 git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130 git checkout 1af02c27a796b91d58cd8b6d50d7e7b9b5564ce4 # save the attached .config to linux build tree make W=1 ARCH=i386
If you fix the issue, kindly add following tag as appropriate
All error/warnings (new ones prefixed by >>):
>> virt/lib/mem_protected.c:11:7: warning: no previous prototype for 'kvm_map_page_atomic' [-Wmissing-prototypes]
11 | void *kvm_map_page_atomic(struct page *page)
| ^~~~~~~~~~~~~~~~~~~
>> virt/lib/mem_protected.c:24:6: warning: no previous prototype for 'kvm_unmap_page_atomic' [-Wmissing-prototypes]
24 | void kvm_unmap_page_atomic(void *vaddr)
| ^~~~~~~~~~~~~~~~~~~~~
>> virt/lib/mem_protected.c:33:5: warning: no previous prototype for 'kvm_init_protected_memory' [-Wmissing-prototypes]
33 | int kvm_init_protected_memory(void)
| ^~~~~~~~~~~~~~~~~~~~~~~~~
>> virt/lib/mem_protected.c:51:6: warning: no previous prototype for 'kvm_exit_protected_memory' [-Wmissing-prototypes]
51 | void kvm_exit_protected_memory(void)
| ^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from include/linux/mm.h:33,
from include/linux/kvm_host.h:14,
from virt/lib/mem_protected.c:1:
virt/lib/mem_protected.c: In function 'adjust_direct_mapping_pte_range':
>> include/linux/pgtable.h:79:12: error: implicit declaration of function 'kmap_atomic'; did you mean 'in_atomic'? [-Werror=implicit-function-declaration]
79 | ((pte_t *)kmap_atomic(pmd_page(*(dir))) + \
| ^~~~~~~~~~~
virt/lib/mem_protected.c:79:8: note: in expansion of macro 'pte_offset_map'
79 | pte = pte_offset_map(pmd, addr);
| ^~~~~~~~~~~~~~ virt/lib/mem_protected.c: At top level:
virt/lib/mem_protected.c:102:5: warning: no previous prototype for '__kvm_protect_memory' [-Wmissing-prototypes]
102 | int __kvm_protect_memory(unsigned long start, unsigned long end, bool protect)
| ^~~~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
vim +/kvm_map_page_atomic +11 virt/lib/mem_protected.c
10
> 11 void *kvm_map_page_atomic(struct page *page)
12 {
13 pte_t *pte;
14 void *vaddr;
15
16 preempt_disable();
17 pte = guest_map_ptes[smp_processor_id()];
18 vaddr = guest_map_area->addr + smp_processor_id() * PAGE_SIZE;
19 set_pte(pte, mk_pte(page, PAGE_KERNEL));
20 return vaddr;
21 }
22 EXPORT_SYMBOL_GPL(kvm_map_page_atomic);
23
> 24 void kvm_unmap_page_atomic(void *vaddr)
25 {
26 pte_t *pte = guest_map_ptes[smp_processor_id()];
27 set_pte(pte, __pte(0));
28 flush_tlb_one_kernel((unsigned long)vaddr);
29 preempt_enable();
30 }
31 EXPORT_SYMBOL_GPL(kvm_unmap_page_atomic);
32
> 33 int kvm_init_protected_memory(void)
34 {
35 guest_map_ptes = kmalloc_array(num_possible_cpus(),
36 sizeof(pte_t *), GFP_KERNEL);
37 if (!guest_map_ptes)
38 return -ENOMEM;
39
40 guest_map_area = alloc_vm_area(PAGE_SIZE * num_possible_cpus(),
41 guest_map_ptes);
42 if (!guest_map_ptes) {
43 kfree(guest_map_ptes);
44 return -ENOMEM;
45 }
46
47 return 0;
48 }
49 EXPORT_SYMBOL_GPL(kvm_init_protected_memory);
50
> 51 void kvm_exit_protected_memory(void)
52 {
53 if (guest_map_area)
54 free_vm_area(guest_map_area);
55 if (guest_map_ptes)
56 kfree(guest_map_ptes);
57 }
58 EXPORT_SYMBOL_GPL(kvm_exit_protected_memory);
59 ---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
RFC v2 14/16 KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()
In reply to Kirill A. ShutemovWe cannot access protected pages directly. Use ioremap() to create a temporary mapping of the page. The mapping is destroyed on __kvm_unmap_gfn().
The new interface gfn_to_pfn_memslot_protected() is used to detect if the page is protected.
ioremap_cache_force() is a hack to bypass IORES_MAP_SYSTEM_RAM check in the x86 ioremap code. We need a better solution.
--- arch/powerpc/kvm/book3s_64_mmu_hv.c | 2 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 2 +- arch/x86/include/asm/io.h | 2 + arch/x86/include/asm/pgtable_types.h | 1 + arch/x86/kvm/mmu/mmu.c | 6 ++- arch/x86/mm/ioremap.c | 16 ++++++-- include/linux/kvm_host.h | 3 +- include/linux/kvm_types.h | 1 + virt/kvm/kvm_main.c | 52 +++++++++++++++++++------- 9 files changed, 63 insertions(+), 22 deletions(-)
arch/powerpc/kvm/book3s_64_mmu_hv.c +1 -1
@@ -590,7 +590,7 @@ int kvmppc_book3s_hv_page_fault(struct kvm_vcpu *vcpu, }else {/* Call KVM generic code to do the slow-path check */ pfn =__gfn_to_pfn_memslot (memslot, gfn, false,NULL , - writing, &write_ok); + writing, &write_ok,NULL );if (is_error_noslot_pfn (pfn))return -EFAULT ; page =NULL ;
arch/powerpc/kvm/book3s_64_mmu_radix.c +1 -1
@@ -822,7 +822,7 @@ int kvmppc_book3s_instantiate_page(struct kvm_vcpu *vcpu,/* Call KVM generic code to do the slow-path check */ pfn =__gfn_to_pfn_memslot (memslot, gfn, false,NULL , - writing, upgrade_p); + writing, upgrade_p,NULL );if (is_error_noslot_pfn (pfn))return -EFAULT ; page =NULL ;
arch/x86/include/asm/io.h +2 -0
@@ -184,6 +184,8 @@ extern void __iomem *ioremap_uc(resource_size_t offset, unsigned long size);#define ioremap_uc ioremap_ucextern void __iomem *ioremap_cache (resource_size_t offset,unsigned long size);#define ioremap_cache ioremap_cache +extern void __iomem *ioremap_cache_force (resource_size_t offset,unsigned long size); +#define ioremap_cache_force ioremap_cache_forceextern void __iomem *ioremap_prot (resource_size_t offset,unsigned long size,unsigned long prot_val);#define ioremap_prot ioremap_protextern void __iomem *ioremap_encrypted (resource_size_t phys_addr,unsigned long size);
arch/x86/include/asm/pgtable_types.h +1 -0
@@ -147,6 +147,7 @@ enum page_cache_mode { _PAGE_CACHE_MODE_UC =3 , _PAGE_CACHE_MODE_WT =4 , _PAGE_CACHE_MODE_WP =5 , + _PAGE_CACHE_MODE_WB_FORCE =6 , _PAGE_CACHE_MODE_NUM = 8 };
arch/x86/kvm/mmu/mmu.c +4 -2
@@ -4058,7 +4058,8 @@ static bool try_async_pf(struct kvm_vcpu *vcpu, bool prefault, gfn_t gfn, } async = false; - *pfn =__gfn_to_pfn_memslot (slot, gfn, false, &async, write, writable); + *pfn =__gfn_to_pfn_memslot (slot, gfn, false, &async, write, writable, +NULL );if (!async)return false;/* *pfn has correct page already */
@@ -4072,7 +4073,8 @@ static bool try_async_pf(struct kvm_vcpu *vcpu, bool prefault, gfn_t gfn,return true; } - *pfn =__gfn_to_pfn_memslot (slot, gfn, false,NULL , write, writable); + *pfn =__gfn_to_pfn_memslot (slot, gfn, false,NULL , write, writable, +NULL );return false; }
arch/x86/mm/ioremap.c +13 -3
@@ -202,9 +202,12 @@ __ioremap_caller(resource_size_t phys_addr, unsigned long size,__ioremap_check_mem (phys_addr, size, &io_desc);/* -* Don't allow anybody to remap normal RAM that we're using.. +* Don't allow anybody to remap normal RAM that we're using, unless +* _PAGE_CACHE_MODE_WB_FORCE is used. */ -if (io_desc.flags &IORES_MAP_SYSTEM_RAM ) { +if (pcm == _PAGE_CACHE_MODE_WB_FORCE) { + pcm = _PAGE_CACHE_MODE_WB; + }else if (io_desc.flags &IORES_MAP_SYSTEM_RAM ) {WARN_ONCE (1 ,"ioremap on RAM at %pa - %pa\n" , &phys_addr, &last_addr);return NULL ;
@@ -419,6 +422,13 @@ void __iomem *ioremap_cache(resource_size_t phys_addr, unsigned long size) }EXPORT_SYMBOL (ioremap_cache); +void __iomem *ioremap_cache_force (resource_size_t phys_addr,unsigned long size) +{ +return __ioremap_caller (phys_addr, size, _PAGE_CACHE_MODE_WB_FORCE, +__builtin_return_address (0 ), false); +} +EXPORT_SYMBOL (ioremap_cache_force); +void __iomem *ioremap_prot (resource_size_t phys_addr,unsigned long size,unsigned long prot_val) {
@@ -467,7 +477,7 @@ void iounmap(volatile void __iomem *addr) p =find_vm_area ((void __force *)addr);if (!p) { -printk (KERN_ERR "iounmap: bad address %p\n" , addr); +printk (KERN_ERR "iounmap: bad address %px\n" , addr);dump_stack ();return ; }
include/linux/kvm_host.h +2 -1
@@ -238,6 +238,7 @@ struct kvm_host_map {void *hva;kvm_pfn_t pfn;kvm_pfn_t gfn; +bool protected; };/*
@@ -725,7 +726,7 @@ kvm_pfn_t gfn_to_pfn_memslot(struct kvm_memory_slot *slot, gfn_t gfn);kvm_pfn_t gfn_to_pfn_memslot_atomic (struct kvm_memory_slot *slot,gfn_t gfn);kvm_pfn_t __gfn_to_pfn_memslot (struct kvm_memory_slot *slot,gfn_t gfn,bool atomic,bool *async,bool write_fault, -bool *writable); +bool *writable,bool *protected);void kvm_release_pfn_clean (kvm_pfn_t pfn);void kvm_release_pfn_dirty (kvm_pfn_t pfn);
include/linux/kvm_types.h +1 -0
@@ -58,6 +58,7 @@ struct gfn_to_pfn_cache {gfn_t gfn;kvm_pfn_t pfn;bool dirty; +bool protected; };#ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE
virt/kvm/kvm_main.c +38 -14
144 lines not on this page · raw message
Signature · 1 line
2.26.2
kernel test robot
build botRe: [RFCv2 14/16] KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi "Kirill,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/x86/core]
[also build test ERROR on v5.9]
[cannot apply to kvm/linux-next hnaz-linux-mm/master next-20201016] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130 base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 238c91115cd05c71447ea071624a4c9fe661f970 config: s390-randconfig-r011-20201020 (attached as .config) compiler: s390-linux-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/47745b863bc9c5221c7b7f10bb0e1456a7c1416a
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130 git checkout 47745b863bc9c5221c7b7f10bb0e1456a7c1416a
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=s390
If you fix the issue, kindly add following tag as appropriate
All error/warnings (new ones prefixed by >>):
arch/s390/kvm/../../../virt/kvm/kvm_main.c: In function '__kvm_map_gfn':
>> arch/s390/kvm/../../../virt/kvm/kvm_main.c:2193:9: error: implicit declaration of function 'ioremap_cache_force' [-Werror=implicit-function-declaration]
2193 | hva = ioremap_cache_force(pfn_to_hpa(pfn), PAGE_SIZE);
| ^~~~~~~~~~~~~~~~~~~
>> arch/s390/kvm/../../../virt/kvm/kvm_main.c:2193:7: warning: assignment to 'void *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
2193 | hva = ioremap_cache_force(pfn_to_hpa(pfn), PAGE_SIZE);
| ^cc1: some warnings being treated as errors
vim +/ioremap_cache_force +2193 arch/s390/kvm/../../../virt/kvm/kvm_main.c
2157
2158 static int __kvm_map_gfn(struct kvm_memslots *slots, gfn_t gfn,
2159 struct kvm_host_map *map,
2160 struct gfn_to_pfn_cache *cache,
2161 bool atomic)
2162 {
2163 kvm_pfn_t pfn;
2164 void *hva = NULL;
2165 struct page *page = KVM_UNMAPPED_PAGE;
2166 struct kvm_memory_slot *slot = __gfn_to_memslot(slots, gfn);
2167 bool protected;
2168 u64 gen = slots->generation;
2169
2170 if (!map)
2171 return -EINVAL;
2172
2173 if (cache) {
2174 if (!cache->pfn || cache->gfn != gfn ||
2175 cache->generation != gen) {
2176 if (atomic)
2177 return -EAGAIN;
2178 kvm_cache_gfn_to_pfn(slot, gfn, cache, gen);
2179 }
2180 pfn = cache->pfn;
2181 protected = cache->protected;
2182 } else {
2183 if (atomic)
2184 return -EAGAIN;
2185 pfn = gfn_to_pfn_memslot_protected(slot, gfn, &protected);
2186 }
2187 if (is_error_noslot_pfn(pfn))
2188 return -EINVAL;
2189
2190 if (protected) {
2191 if (atomic)
2192 return -EAGAIN;
> 2193 hva = ioremap_cache_force(pfn_to_hpa(pfn), PAGE_SIZE);
2194 } else if (pfn_valid(pfn)) {
2195 page = pfn_to_page(pfn);
2196 if (atomic)
2197 hva = kmap_atomic(page);
2198 else
2199 hva = kmap(page);
2200 #ifdef CONFIG_HAS_IOMEM
2201 } else if (!atomic) {
2202 hva = memremap(pfn_to_hpa(pfn), PAGE_SIZE, MEMREMAP_WB);
2203 } else {
2204 return -EINVAL;
2205 #endif
2206 }
2207
2208 if (!hva)
2209 return -EFAULT;
2210
2211 map->page = page;
2212 map->hva = hva;
2213 map->pfn = pfn;
2214 map->gfn = gfn;
2215 map->protected = protected;
2216
2217 return 0;
2218 }
2219
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kernel test robot
build botRe: [RFCv2 14/16] KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()
In reply to Kirill A. Shutemov[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi "Kirill,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/x86/core]
[also build test ERROR on v5.9]
[cannot apply to kvm/linux-next hnaz-linux-mm/master next-20201016] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130 base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 238c91115cd05c71447ea071624a4c9fe661f970 config: mips-randconfig-r031-20201020 (attached as .config) compiler: mipsel-linux-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/47745b863bc9c5221c7b7f10bb0e1456a7c1416a
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Kirill-A-Shutemov/KVM-protected-memory-extension/20201020-142130 git checkout 47745b863bc9c5221c7b7f10bb0e1456a7c1416a
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=mips
If you fix the issue, kindly add following tag as appropriate
All error/warnings (new ones prefixed by >>):
arch/mips/kvm/../../../virt/kvm/kvm_main.c: In function '__kvm_map_gfn':
>> arch/mips/kvm/../../../virt/kvm/kvm_main.c:2193:9: error: implicit declaration of function 'ioremap_cache_force'; did you mean 'ioremap_cache'? [-Werror=implicit-function-declaration]
2193 | hva = ioremap_cache_force(pfn_to_hpa(pfn), PAGE_SIZE);
| ^~~~~~~~~~~~~~~~~~~
| ioremap_cache
>> arch/mips/kvm/../../../virt/kvm/kvm_main.c:2193:7: warning: assignment to 'void *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
2193 | hva = ioremap_cache_force(pfn_to_hpa(pfn), PAGE_SIZE);
| ^cc1: some warnings being treated as errors
vim +2193 arch/mips/kvm/../../../virt/kvm/kvm_main.c
2157
2158 static int __kvm_map_gfn(struct kvm_memslots *slots, gfn_t gfn,
2159 struct kvm_host_map *map,
2160 struct gfn_to_pfn_cache *cache,
2161 bool atomic)
2162 {
2163 kvm_pfn_t pfn;
2164 void *hva = NULL;
2165 struct page *page = KVM_UNMAPPED_PAGE;
2166 struct kvm_memory_slot *slot = __gfn_to_memslot(slots, gfn);
2167 bool protected;
2168 u64 gen = slots->generation;
2169
2170 if (!map)
2171 return -EINVAL;
2172
2173 if (cache) {
2174 if (!cache->pfn || cache->gfn != gfn ||
2175 cache->generation != gen) {
2176 if (atomic)
2177 return -EAGAIN;
2178 kvm_cache_gfn_to_pfn(slot, gfn, cache, gen);
2179 }
2180 pfn = cache->pfn;
2181 protected = cache->protected;
2182 } else {
2183 if (atomic)
2184 return -EAGAIN;
2185 pfn = gfn_to_pfn_memslot_protected(slot, gfn, &protected);
2186 }
2187 if (is_error_noslot_pfn(pfn))
2188 return -EINVAL;
2189
2190 if (protected) {
2191 if (atomic)
2192 return -EAGAIN;
> 2193 hva = ioremap_cache_force(pfn_to_hpa(pfn), PAGE_SIZE);
2194 } else if (pfn_valid(pfn)) {
2195 page = pfn_to_page(pfn);
2196 if (atomic)
2197 hva = kmap_atomic(page);
2198 else
2199 hva = kmap(page);
2200 #ifdef CONFIG_HAS_IOMEM
2201 } else if (!atomic) {
2202 hva = memremap(pfn_to_hpa(pfn), PAGE_SIZE, MEMREMAP_WB);
2203 } else {
2204 return -EINVAL;
2205 #endif
2206 }
2207
2208 if (!hva)
2209 return -EFAULT;
2210
2211 map->page = page;
2212 map->hva = hva;
2213 map->pfn = pfn;
2214 map->gfn = gfn;
2215 map->protected = protected;
2216
2217 return 0;
2218 }
2219
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Re: [RFCv2 14/16] KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()
In reply to Kirill A. ShutemovQuoted 156 lines · Kirill A. Shutemov
We cannot access protected pages directly. Use
ioremap()to create a temporary mapping of the page. The mapping is destroyed on__kvm_unmap_gfn().The new interface
gfn_to_pfn_memslot_protected()is used to detect if the page is protected.
ioremap_cache_force()is a hack to bypassIORES_MAP_SYSTEM_RAMcheck in
the x86 ioremap code. We need a better solution.Signed-off-by Kirill A. Shutemov <[email protected]>--- arch/powerpc/kvm/book3s_64_mmu_hv.c | 2 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 2 +- arch/x86/include/asm/io.h | 2 + arch/x86/include/asm/pgtable_types.h | 1 + arch/x86/kvm/mmu/mmu.c | 6 ++- arch/x86/mm/ioremap.c | 16 ++++++-- include/linux/kvm_host.h | 3 +- include/linux/kvm_types.h | 1 + virt/kvm/kvm_main.c | 52 +++++++++++++++++++----- -- 9 files changed, 63 insertions(+), 22 deletions(-)arch/powerpc/kvm/book3s_64_mmu_hv.c +1
-1@@ -590,7 +590,7 @@ int kvmppc_book3s_hv_page_fault(struct kvm_vcpu *vcpu, }else {/* Call KVM generic code to do the slow-path check */ pfn =__gfn_to_pfn_memslot (memslot, gfn, false,NULL , - writing, &write_ok); + writing, &write_ok,NULL );if (is_error_noslot_pfn (pfn))return -EFAULT ;arch/powerpc/kvm/book3s_64_mmu_hv.c +0
-0page =NULL ;arch/powerpc/kvm/book3s_64_mmu_radix.c +1
-1@@ -822,7 +822,7 @@ int kvmppc_book3s_instantiate_page(structkvm_vcpu *vcpu,/* Call KVM generic code to do the slow-path check */ pfn =__gfn_to_pfn_memslot (memslot, gfn, false,NULL , - writing, upgrade_p); + writing, upgrade_p,NULL );if (is_error_noslot_pfn (pfn))return -EFAULT ;arch/powerpc/kvm/book3s_64_mmu_radix.c +0
-0page =NULL ;arch/x86/include/asm/io.h +2
-0@@ -184,6 +184,8 @@ extern void __iomem *ioremap_uc(resource_size_t offset, unsigned long size);#define ioremap_uc ioremap_ucextern void __iomem *ioremap_cache (resource_size_t offset,unsigned long size);#define ioremap_cache ioremap_cache +extern void __iomem *ioremap_cache_force (resource_size_t offset,unsigned long size); +#define ioremap_cache_force ioremap_cache_forceextern void
__iomem*ioremap_prot(resource_size_t offset, unsigned
long size, unsigned longprot_val);
#defineioremap_protioremap_prot
extern void__iomem*ioremap_encrypted(resource_size_tphys_addr,
unsigned long size);arch/x86/include/asm/pgtable_types.h +1
-0@@ -147,6 +147,7 @@ enum page_cache_mode { _PAGE_CACHE_MODE_UC =3 , _PAGE_CACHE_MODE_WT =4 , _PAGE_CACHE_MODE_WP =5 , + _PAGE_CACHE_MODE_WB_FORCE =6 , _PAGE_CACHE_MODE_NUM = 8 };arch/x86/kvm/mmu/mmu.c +2
-1@@ -4058,7 +4058,8 @@ static bool try_async_pf(struct kvm_vcpu *vcpu, bool prefault, gfn_t gfn, } async = false; - *pfn =__gfn_to_pfn_memslot (slot, gfn, false, &async, write, writable); + *pfn =__gfn_to_pfn_memslot (slot, gfn, false, &async, write, writable, +NULL );arch/x86/kvm/mmu/mmu.c +2
-1if (!async)return false;/* *pfn has correct page already */ @@ -4072,7 +4073,8 @@ static bool try_async_pf(struct kvm_vcpu *vcpu, bool prefault, gfn_t gfn, return true; } - *pfn =__gfn_to_pfn_memslot (slot, gfn, false,NULL , write, writable); + *pfn =__gfn_to_pfn_memslot (slot, gfn, false,NULL , write, writable, +NULL );arch/x86/kvm/mmu/mmu.c +0
-0return false; }arch/x86/mm/ioremap.c +1
-1@@ -202,9 +202,12 @@ __ioremap_caller(resource_size_t phys_addr,unsigned long size, __ioremap_check_mem(phys_addr, size, &io_desc);/* -* Don't allow anybody to remap normal RAM that we're using.. +* Don't allow anybody to remap normal RAM that we're using, unless
arch/x86/mm/ioremap.c +11
-1+* _PAGE_CACHE_MODE_WB_FORCE is used. */ -if (io_desc.flags &IORES_MAP_SYSTEM_RAM ) { +if (pcm == _PAGE_CACHE_MODE_WB_FORCE) { + pcm = _PAGE_CACHE_MODE_WB; + }else if (io_desc.flags &IORES_MAP_SYSTEM_RAM ) {WARN_ONCE (1 ,"ioremap on RAM at %pa - %pa\n" , &phys_addr, &last_addr);return NULL ;@@ -419,6 +422,13 @@ void __iomem *ioremap_cache(resource_size_t phys_addr, unsigned long size) }EXPORT_SYMBOL (ioremap_cache); +void __iomem *ioremap_cache_force (resource_size_t phys_addr,unsigned long size) +{ +return __ioremap_caller (phys_addr, size, _PAGE_CACHE_MODE_WB_FORCE, +__builtin_return_address (0 ), false); +} +EXPORT_SYMBOL (ioremap_cache_force); +void
__iomem*ioremap_prot(resource_size_tphys_addr, unsigned long
size,arch/x86/mm/ioremap.c +0
-0unsigned long prot_val) {
arch/x86/mm/ioremap.c void iounmap(volatile void __iomem *addr) · line 480
Quoted 5 lines
arch/x86/mm/ioremap.c +1
-1@@ -467,7 +477,7 @@ void iounmap(volatile void __iomem *addr) p =find_vm_area ((void __force *)addr);if (!p) { -printk (KERN_ERR "iounmap: bad address %p\n" , addr); +printk (KERN_ERR "iounmap: bad address %px\n" , addr);
Unintentional?
Quoted 208 lines
arch/x86/mm/ioremap.c +0
-0dump_stack ();return ; }include/linux/kvm_host.h +2
-1@@ -238,6 +238,7 @@ struct kvm_host_map {void *hva;kvm_pfn_t pfn;kvm_pfn_t gfn; +bool protected; };/* @@ -725,7 +726,7 @@ kvm_pfn_t gfn_to_pfn_memslot(struct kvm_memory_slot *slot, gfn_t gfn);kvm_pfn_t gfn_to_pfn_memslot_atomic (struct kvm_memory_slot *slot,gfn_t gfn);kvm_pfn_t __gfn_to_pfn_memslot (struct kvm_memory_slot *slot,gfn_t gfn,bool atomic,bool *async,bool write_fault, -bool *writable); +bool *writable,bool *protected);void kvm_release_pfn_clean (kvm_pfn_t pfn);void kvm_release_pfn_dirty (kvm_pfn_t pfn);include/linux/kvm_types.h +1
-0@@ -58,6 +58,7 @@ struct gfn_to_pfn_cache {gfn_t gfn;kvm_pfn_t pfn;bool dirty; +bool protected; };#ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE virt/kvm/kvm_main.c +2
-1@@ -1852,9 +1852,10 @@ static bool hva_to_pfn_fast(unsigned longaddr, bool write_fault, * 1 indicates success, -errno is returned if error is detected. */ static int hva_to_pfn_slow (unsigned long addr,bool *async,bool write_fault, -bool *writable,kvm_pfn_t *pfn) +bool *writable,bool *protected,kvm_pfn_t *pfn) {unsigned int flags =FOLL_HWPOISON |FOLL_KVM ; +struct vm_area_struct *vma;virt/kvm/kvm_main.c +8
-2struct page *page;int npages =0 ;@@ -1868,9 +1869,15 @@ static int hva_to_pfn_slow(unsigned long addr, bool *async, bool write_fault, if (async) flags |= FOLL_NOWAIT; - npages =get_user_pages_unlocked (addr,1 , &page, flags); -if (npages !=1 ) +mmap_read_lock (current->mm); + npages =get_user_pages (addr,1 , flags, &page, &vma); +if (npages !=1 ) { +mmap_read_unlock (current->mm);return npages; + } +if (protected) + *protected =vma_is_kvm_protected (vma); +mmap_read_unlock (current->mm);/* map read fault as writable if possible */ virt/kvm/kvm_main.c +1
-1if (unlikely (!write_fault) && writable) {@@ -1961,7 +1968,7 @@ static int hva_to_pfn_remapped(structvm_area_struct *vma, * whether the mapping is writable. */ static kvm_pfn_t hva_to_pfn (unsigned long addr,bool atomic,bool *async, - bool write_fault, bool *writable) +bool write_fault,bool *writable,bool *protected){
virt/kvm/kvm_main.c +2
-1struct vm_area_struct *vma;kvm_pfn_t pfn =0 ;@@ -1976,7 +1983,8 @@ static kvm_pfn_t hva_to_pfn(unsigned long addr, bool atomic, bool *async, if (atomic) return KVM_PFN_ERR_FAULT; - npages =hva_to_pfn_slow (addr, async, write_fault, writable, &pfn); + npages =hva_to_pfn_slow (addr, async, write_fault, writable, protected, + &pfn);virt/kvm/kvm_main.c +1
-1if (npages ==1 )return pfn;@@ -2010,7 +2018,7 @@ static kvm_pfn_t hva_to_pfn(unsigned long addr, bool atomic, bool *async, kvm_pfn_t __gfn_to_pfn_memslot(struct kvm_memory_slot *slot, gfn_t gfn,bool atomic,bool *async,bool write_fault, -bool *writable) +bool *writable,bool *protected){
unsigned long addr = __gfn_to_hva_many(slot, gfn, NULL,write_fault);virt/kvm/kvm_main.c +5
-3@@ -2033,7 +2041,7 @@ kvm_pfn_t __gfn_to_pfn_memslot(struct kvm_memory_slot *slot, gfn_t gfn, } return hva_to_pfn(addr, atomic, async, write_fault, -writable ); +writable ,protected ); }EXPORT_SYMBOL_GPL (__gfn_to_pfn_memslot);@@ -2041,19 +2049,26 @@ kvm_pfn_t gfn_to_pfn_prot(struct kvm *kvm, gfn_t gfn, bool write_fault, bool *writable) { return __gfn_to_pfn_memslot(gfn_to_memslot(kvm, gfn), gfn,false ,NULL , -write_fault ,writable ); +write_fault ,writable ,NULL ); }EXPORT_SYMBOL_GPL (gfn_to_pfn_prot);kvm_pfn_t gfn_to_pfn_memslot (struct kvm_memory_slot *slot,gfn_t gfn) { -return __gfn_to_pfn_memslot (slot, gfn, false,NULL , true, NULL); +return __gfn_to_pfn_memslot (slot, gfn, false,NULL , true,NULL ,NULL ); }EXPORT_SYMBOL_GPL (gfn_to_pfn_memslot); +static kvm_pfn_t gfn_to_pfn_memslot_protected (struct kvm_memory_slot *slot, + gfn_t gfn, bool*protected)
virt/kvm/kvm_main.c +5
-0+{ +return __gfn_to_pfn_memslot (slot, gfn, false,NULL , true,NULL , + protected); +} + kvm_pfn_t gfn_to_pfn_memslot_atomic(struct kvm_memory_slot *slot,
gfn_tgfn)
{virt/kvm/kvm_main.c +1
-1-return __gfn_to_pfn_memslot (slot, gfn, true,NULL , true,NULL ); + return __gfn_to_pfn_memslot(slot, gfn, true, NULL, true, NULL,NULL);
}
EXPORT_SYMBOL_GPL(gfn_to_pfn_memslot_atomic);virt/kvm/kvm_main.c +1
-1@@ -2134,7 +2149,7 @@ static void kvm_cache_gfn_to_pfn(struct kvm_memory_slot *slot, gfn_t gfn, { kvm_release_pfn(cache->pfn, cache->dirty, cache); - cache->pfn =gfn_to_pfn_memslot (slot, gfn); + cache->pfn =gfn_to_pfn_memslot_protected (slot, gfn, &cache- >protected); cache->gfn = gfn;virt/kvm/kvm_main.c +1
-0cache->dirty = false; cache->generation = gen;@@ -2149,6 +2164,7 @@ static int __kvm_map_gfn(struct kvm_memslots *slots, gfn_t gfn, void *hva =NULL ;struct page *page =KVM_UNMAPPED_PAGE ;struct kvm_memory_slot *slot =__gfn_to_memslot (slots, gfn); +bool protected;u64 gen = slots->generation;virt/kvm/kvm_main.c +7
-2if (!map)@@ -2162,15 +2178,20 @@ static int __kvm_map_gfn(struct kvm_memslots *slots, gfn_t gfn,kvm_cache_gfn_to_pfn (slot, gfn, cache, gen); } pfn = cache->pfn; + protected = cache->protected; }else {if (atomic)return -EAGAIN ; - pfn =gfn_to_pfn_memslot (slot, gfn); + pfn =gfn_to_pfn_memslot_protected (slot, gfn, &protected); }if (is_error_noslot_pfn (pfn))return -EINVAL ; -if (pfn_valid (pfn)) { +if (protected) { +if (atomic) +return -EAGAIN ; + hva =ioremap_cache_force (pfn_to_hpa (pfn),PAGE_SIZE ); + }else if (pfn_valid (pfn)) { page =pfn_to_page (pfn);
virt/kvm/kvm_main.c
Quoted 2 lines
virt/kvm/kvm_main.c +0
-0if (atomic) hva =kmap_atomic (page);
I think the page could have got unmapped since the gup via the hypercall on another CPU. It could be an avenue for the guest to crash the host.
Quoted 21 lines
virt/kvm/kvm_main.c +1
-0@@ -2191,6 +2212,7 @@ static int __kvm_map_gfn(struct kvm_memslots *slots, gfn_t gfn, map->hva = hva; map->pfn = pfn; map->gfn = gfn; + map->protected = protected;return 0 ;}
virt/kvm/kvm_main.c +3
-1@@ -2221,7 +2243,9 @@ static void __kvm_unmap_gfn(struct kvm_memory_slot *memslot, if (!map->hva) return; -if (map->page !=KVM_UNMAPPED_PAGE ) { +if (map->protected) { +iounmap (map->hva); + }else if (map->page !=KVM_UNMAPPED_PAGE ) {if (atomic)kunmap_atomic (map->hva);else
Re: [RFCv2 14/16] KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()
Quoted 9 lines · Rick P
arch/x86/mm/ioremap.c void iounmap(volatile void __iomem *addr) · line 480
Quoted 6 lines · Kirill A. Shutemov
arch/x86/mm/ioremap.c +1
-1@@ -467,7 +477,7 @@ void iounmap(volatile void __iomem *addr) p =find_vm_area ((void __force *)addr);if (!p) { -printk (KERN_ERR "iounmap: bad address %p\n" , addr); +printk (KERN_ERR "iounmap: bad address %px\n" , addr);Unintentional?
Yep. Will fix.
Quoted 16 lines
Quoted 16 lines
virt/kvm/kvm_main.c +2
-1@@ -2162,15 +2178,20 @@ static int __kvm_map_gfn(struct kvm_memslots *slots, gfn_t gfn,kvm_cache_gfn_to_pfn (slot, gfn, cache, gen); } pfn = cache->pfn; + protected = cache->protected; }else {if (atomic)return -EAGAIN ; - pfn =gfn_to_pfn_memslot (slot, gfn); + pfn =gfn_to_pfn_memslot_protected (slot, gfn, &protected); }if (is_error_noslot_pfn (pfn))return -EINVAL ;
Quoted 13 lines
Review
Quoted 9 lines
virt/kvm/kvm_main.c +5
-1-if (pfn_valid (pfn)) { +if (protected) { +if (atomic) +return -EAGAIN ; + hva =ioremap_cache_force (pfn_to_hpa (pfn),PAGE_SIZE ); + }else if (pfn_valid (pfn)) { page =pfn_to_page (pfn);(quoted diff) +0
-0if (atomic) hva = kmap_atomic(page);I think the page could have got unmapped since the gup via the hypercall on another CPU. It could be an avenue for the guest to crash the host.
Hm.. I'm not sure I follow. Could you elaborate on what scenario you have in mind?
Signature · 1 line
Kirill A. Shutemov
Re: [RFCv2 14/16] KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()
Quoted 8 lines · Kirill A. Shutemov
Quoted 4 lines
I think the page could have got unmapped since the gup via the hypercall on another CPU. It could be an avenue for the guest to crash
the host.Hm.. I'm not sure I follow. Could you elaborate on what scenario you have
in mind?
Kind of similar scenario as the userspace triggered oops. My understanding is that the protected status was gathered along with the gup, but after the mm gets unlocked, nothing stops the page transitioning to unmapped(?). At which point kmap() from a previous gup with !protected, would go down the regular kmap() route and return an address to an unmapped page.
So the guest kernel could start with a page mapped as shared via the hypercall. Then trigger one of the PV MSR's that kmap() on CPU0. On CPU1, after the gup on CPU0, it could transitioned the page to private/unmapped via the hypercall. So the hva_to_pfn() would find !protected, but by the time the kmap() happened the page would have been unmapped. Am I missing something?
Re: [RFCv2 14/16] KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()
Quoted 17 lines · Rick P
Quoted 8 lines · Kirill A. Shutemov
Quoted 4 lines
I think the page could have got unmapped since the gup via the hypercall on another CPU. It could be an avenue for the guest to crash
the host.Hm.. I'm not sure I follow. Could you elaborate on what scenario you have
in mind?Kind of similar scenario as the userspace triggered oops. My understanding is that the protected status was gathered along with the gup, but after the mm gets unlocked, nothing stops the page transitioning to unmapped(?). At which point
kmap()from a previous gup with !protected, would go down the regularkmap()route and return an address to an unmapped page.
Quoted 6 lines
So the guest kernel could start with a page mapped as shared via the hypercall. Then trigger one of the PV MSR's that
kmap()on CPU0. On CPU1, after the gup on CPU0, it could transitioned the page to private/unmapped via the hypercall. So thehva_to_pfn()would find !protected, but by the time thekmap()happened the page would have been unmapped. Am I missing something?
We need to fail protection enabling if a page is pinned. That's the only option I see. But it might be pain to debug.
Signature · 1 line
Kirill A. Shutemov
Re: [RFCv2 14/16] KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()
In reply to Kirill A. ShutemovQuoted 11 lines · On Tue, 20 Oct 2020 09:18:57 +0300 "Kirill A. Shutemov" <[email protected]>…
We cannot access protected pages directly. Use
ioremap()to create a temporary mapping of the page. The mapping is destroyed on__kvm_unmap_gfn().The new interface
gfn_to_pfn_memslot_protected()is used to detect if the page is protected.
ioremap_cache_force()is a hack to bypassIORES_MAP_SYSTEM_RAMcheck in the x86 ioremap code. We need a better solution.Signed-off-by Kirill A. Shutemov <[email protected]>
Quoted 12 lines
--- arch/powerpc/kvm/book3s_64_mmu_hv.c | 2 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 2 +- arch/x86/include/asm/io.h | 2 + arch/x86/include/asm/pgtable_types.h | 1 + arch/x86/kvm/mmu/mmu.c | 6 ++- arch/x86/mm/ioremap.c | 16 ++++++-- include/linux/kvm_host.h | 3 +- include/linux/kvm_types.h | 1 + virt/kvm/kvm_main.c | 52 +++++++++++++++++++------- 9 files changed, 63 insertions(+), 22 deletions(-)
You declare ioremap_cache_force() arch/x86/include/asm/io.h in and define it in arch/x86/mm/ioremap.c which is architecture specific code, but use it in __kvm_map_gfn() in virt/kvm/kvm_main.c which is common code.
Thus your series breaks the build for the s390 architecture. Have you tried to (cross) compile for s390?
Regards,
Halil
Re: [RFCv2 14/16] KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()
Quoted 27 lines · Halil Pasic
Quoted 11 lines · On Tue, 20 Oct 2020 09:18:57 +0300 "Kirill A. Shutemov" <[email protected]>…
We cannot access protected pages directly. Use
ioremap()to create a temporary mapping of the page. The mapping is destroyed on__kvm_unmap_gfn().The new interface
gfn_to_pfn_memslot_protected()is used to detect if the page is protected.
ioremap_cache_force()is a hack to bypassIORES_MAP_SYSTEM_RAMcheck in the x86 ioremap code. We need a better solution.Signed-off-by Kirill A. Shutemov <[email protected]>Quoted 12 lines
--- arch/powerpc/kvm/book3s_64_mmu_hv.c | 2 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 2 +- arch/x86/include/asm/io.h | 2 + arch/x86/include/asm/pgtable_types.h | 1 + arch/x86/kvm/mmu/mmu.c | 6 ++- arch/x86/mm/ioremap.c | 16 ++++++-- include/linux/kvm_host.h | 3 +- include/linux/kvm_types.h | 1 + virt/kvm/kvm_main.c | 52 +++++++++++++++++++------- 9 files changed, 63 insertions(+), 22 deletions(-)
Quoted 7 lines
You declare
ioremap_cache_force()arch/x86/include/asm/io.hin and define it inarch/x86/mm/ioremap.cwhich is architecture specific code, but use it in__kvm_map_gfn()invirt/kvm/kvm_main.cwhich is common code.Thus your series breaks the build for the s390 architecture. Have you tried to (cross) compile for s390?
Obviously not. I've got reports already from 0day and going to fix it.
Thanks.
Signature · 1 line
Kirill A. Shutemov
RFC v2 15/16 KVM: Unmap protected pages from direct mapping
In reply to Kirill A. ShutemovIf the protected memory feature enabled, unmap guest memory from kernel's direct mappings.
Migration and KSM is disabled for protected memory as it would require a special treatment.
--- include/linux/mm.h | 3 +++ mm/huge_memory.c | 8 ++++++++ mm/ksm.c | 2 ++ mm/memory.c | 12 ++++++++++++ mm/rmap.c | 4 ++++ virt/lib/mem_protected.c | 21 +++++++++++++++++++++ 6 files changed, 50 insertions(+)
include/linux/mm.h +3 -0
@@ -671,6 +671,9 @@ static inline bool vma_is_kvm_protected(struct vm_area_struct *vma)return vma->vm_flags &VM_KVM_PROTECTED ; } +void kvm_map_page (struct page *page,int nr_pages); +void kvm_unmap_page (struct page *page,int nr_pages); +#ifdef CONFIG_SHMEM /* * The vma_is_shmem is not inline because it is used only by slow
mm/huge_memory.c +8 -0
@@ -627,6 +627,10 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,spin_unlock (vmf->ptl);count_vm_event (THP_FAULT_ALLOC );count_memcg_event_mm (vma->vm_mm,THP_FAULT_ALLOC ); + +/* Unmap page from direct mapping */ +if (vma_is_kvm_protected (vma)) +kvm_unmap_page (page,HPAGE_PMD_NR ); }return 0 ;
@@ -1689,6 +1693,10 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,page_remove_rmap (page, true);VM_BUG_ON_PAGE (page_mapcount (page) <0 , page);VM_BUG_ON_PAGE (!PageHead (page), page); + +/* Map the page back to the direct mapping */ +if (vma_is_kvm_protected (vma)) +kvm_map_page (page,HPAGE_PMD_NR ); }else if (thp_migration_supported ()) {swp_entry_t entry;
mm/ksm.c +2 -0
@@ -528,6 +528,8 @@ static struct vm_area_struct *find_mergeable_vma(struct mm_struct *mm,return NULL ;if (!(vma->vm_flags &VM_MERGEABLE ) || !vma->anon_vma)return NULL ; +if (vma_is_kvm_protected (vma)) +return NULL ;return vma; }
mm/memory.c +12 -0
@@ -1245,6 +1245,11 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, likely(!(vma->vm_flags & VM_SEQ_READ)))mark_page_accessed (page); } + +/* Map the page back to the direct mapping */ +if (vma_is_anonymous (vma) &&vma_is_kvm_protected (vma)) +kvm_map_page (page,1 ); + rss[mm_counter (page)]--;page_remove_rmap (page, false);if (unlikely (page_mapcount (page) <0 ))
@@ -3466,6 +3471,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)struct page *page;vm_fault_t ret =0 ;pte_t entry; +bool set = false;/* File mapping without ->vm_ops ? */ if (vma->vm_flags &VM_SHARED )
@@ -3554,6 +3560,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)inc_mm_counter_fast (vma->vm_mm,MM_ANONPAGES );page_add_new_anon_rmap (page, vma, vmf->address, false);lru_cache_add_inactive_or_unevictable (page, vma); + set = true; setpte:set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry);
@@ -3561,6 +3568,11 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)update_mmu_cache (vma, vmf->address, vmf->pte); unlock:pte_unmap_unlock (vmf->pte, vmf->ptl); + +/* Unmap page from direct mapping */ +if (vma_is_kvm_protected (vma) && set) +kvm_unmap_page (page,1 ); +return ret; release:put_page (page);
mm/rmap.c +4 -0
@@ -1725,6 +1725,10 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,static bool invalid_migration_vma (struct vm_area_struct *vma,void *arg) { +/* TODO */ +if (vma_is_kvm_protected (vma)) +return true; +return vma_is_temporary_stack (vma); }
virt/lib/mem_protected.c +21 -0
@@ -30,6 +30,27 @@ void kvm_unmap_page_atomic(void *vaddr) }EXPORT_SYMBOL_GPL (kvm_unmap_page_atomic); +void kvm_map_page (struct page *page,int nr_pages) +{ +int i; + +/* Clear page before returning it to the direct mapping */ +for (i =0 ; i < nr_pages; i++) { +void *p =kvm_map_page_atomic (page + i); +memset (p,0 ,PAGE_SIZE ); +kvm_unmap_page_atomic (p); + } + +kernel_map_pages (page, nr_pages,1 ); +} +EXPORT_SYMBOL_GPL (kvm_map_page); + +void kvm_unmap_page (struct page *page,int nr_pages) +{ +kernel_map_pages (page, nr_pages,0 ); +} +EXPORT_SYMBOL_GPL (kvm_unmap_page); +int kvm_init_protected_memory (void ) { guest_map_ptes = kmalloc_array(num_possible_cpus(),
Signature · 1 line
2.26.2
Peter Zijlstra
maintainerRe: [RFCv2 15/16] KVM: Unmap protected pages from direct mapping
Quoted 5 lines · Kirill A. Shutemov
If the protected memory feature enabled, unmap guest memory from kernel's direct mappings.
Migration and KSM is disabled for protected memory as it would require a special treatment.
How isn't disabling migration a problem? Are we going to back allocation by CMA instead?
Re: [RFCv2 15/16] KVM: Unmap protected pages from direct mapping
In reply to Kirill A. ShutemovQuoted 2 lines · Kirill A. Shutemov
If the protected memory feature enabled, unmap guest memory from kernel's direct mappings.
Gah, ugly. I guess this also defeats compaction, swapping, ... oh gosh. As if all of the encrypted VM implementations didn't bring us enough ugliness already (SEV extensions also don't support reboots, but can at least kexec() IIRC).
Something similar is done with secretmem [1]. And people don't seem to like fragmenting the direct mapping (including me).
[1] https://lkml.kernel.org/r/[email protected]
Signature · 3 lines
Thanks, David / dhildenb
Re: [RFCv2 15/16] KVM: Unmap protected pages from direct mapping
Quoted 9 lines · David Hildenbrand
Quoted 2 lines · Kirill A. Shutemov
If the protected memory feature enabled, unmap guest memory from kernel's direct mappings.
Gah, ugly. I guess this also defeats compaction, swapping, ... oh gosh. As if all of the encrypted VM implementations didn't bring us enough ugliness already (SEV extensions also don't support reboots, but can at least
kexec()IIRC).
Quoted 5 lines
Something similar is done with secretmem [1]. And people don't seem to like fragmenting the direct mapping (including me).
I just thought "hey, we might have to replace pud/pmd mappings by page tables when calling kernel_map_pages", this can fail with -ENOMEM, why isn't there proper error handling.
Then I dived into __kernel_map_pages() which states:
"The return value is ignored as the calls cannot fail. Large pages for identity mappings are not used at boot time and hence no memory allocations during large page split."
I am probably missing something important, but how is calling kernel_map_pages() safe *after* booting?! I know we use it for debug_pagealloc(), but using it in a production-ready feature feels completely irresponsible. What am I missing?
Signature · 3 lines
Thanks, David / dhildenb
Re: [RFCv2 15/16] KVM: Unmap protected pages from direct mapping
Quoted 33 lines · David Hildenbrand
Quoted 12 lines · David Hildenbrand
Quoted 2 lines · Kirill A. Shutemov
If the protected memory feature enabled, unmap guest memory from kernel's direct mappings.
Gah, ugly. I guess this also defeats compaction, swapping, ... oh gosh.
As if all of the encrypted VM implementations didn't bring us enough
ugliness already (SEV extensions also don't support reboots, but can at
leastkexec()IIRC).Quoted 6 lines
Something similar is done with secretmem [1]. And people don't seem to
like fragmenting the direct mapping (including me).I just thought "hey, we might have to replace pud/pmd mappings by page
tables when callingkernel_map_pages", this can fail with-ENOMEM, why
isn't there proper error handling.Then I dived into
__kernel_map_pages()which states:"The return value is ignored as the calls cannot fail. Large pages for
identity mappings are not used at boot time and hence no memory allocations during large page split."
Quoted 5 lines
I am probably missing something important, but how is calling
kernel_map_pages()safe *after* booting?! I know we use it fordebug_pagealloc(), but using it in a production-ready feature feels completely irresponsible. What am I missing?
My understanding was that DEBUG_PAGEALLOC maps the direct map at 4k post-boot as well so that kernel_map_pages() can't fail for it, and to avoid having to take locks for splits in interrupts. I think you are on to something though. That function is essentially a set_memory_() functions on x86 at least, and many callers do not check the error codes. Kees Cook has been pushing to fix this actually:
https://github.com/KSPP/linux/issues/7
As long as a page has been broken to 4k, it should be able to be re- mapped without failure (like debug page alloc relies on). If an initial call to restrict permissions needs and fails to break a page, its remap does not need to succeed either before freeing the page, since the page never got set with a lower permission. That's my understanding from x86's cpa at least. The problem becomes that the page silently doesn't have its expected protections during use. Unchecked set_memory_() caching property change failures might result in functional problems though.
So there is some background if you wanted it, but yea, I agree this feature should handle if the unmap failed. Probably return an error on the IOCTL and maybe the hypercall. kernel_map_pages() also doesn't do a shootdown though, so this direct map protection is more in the best effort category in its current state.
For unmapping causing direct map fragmentation. The two techniques floating around, besides breaking indiscriminately, seem to be:
1. Group and cache unmapped pages to minimize the damage (like secret mem)
2. Somehow repair them back to large pages when reset RW (as Kirill had tried earlier this year in another series)
I had imagined this usage would want something like that eventually, but neither helps with the other limitations you mentioned (migration, etc).
Re: [RFCv2 15/16] KVM: Unmap protected pages from direct mapping
In reply to David HildenbrandQuoted 8 lines · David Hildenbrand
Quoted 2 lines · Kirill A. Shutemov
If the protected memory feature enabled, unmap guest memory from kernel's direct mappings.
Gah, ugly. I guess this also defeats compaction, swapping, ... oh gosh. As if all of the encrypted VM implementations didn't bring us enough ugliness already (SEV extensions also don't support reboots, but can at least
kexec()IIRC).
SEV does support reboot. SEV-ES using Qemu doesn't support reboot because of the way Qemu resets the vCPU state. If Qemu could relaunch the guest through the SEV APIs to reset the vCPU state, then a "reboot" would be possible.
SEV does support kexec, SEV-ES does not at the moment.
Thanks,
Tom
Quoted 6 lines
Something similar is done with secretmem [1]. And people don't seem to like fragmenting the direct mapping (including me).
Re: [RFCv2 15/16] KVM: Unmap protected pages from direct mapping
In reply to Kirill A. ShutemovQuoted 7 lines · Kirill A. Shutemov
If the protected memory feature enabled, unmap guest memory from kernel's direct mappings.
Migration and KSM is disabled for protected memory as it would require a
special treatment.
So do we care about this scenario where a malicious userspace causes a kernel oops? I'm not sure if it's prevented somehow.
CPU0 (exercising other kernel functionality) CPU1
mark page shared
page = get_user_pages(!FOLL_KVM)
mark page private
kmap(page)
access unmapped page and oops
Mike Rapoport
reviewerRe: [RFCv2 15/16] KVM: Unmap protected pages from direct mapping
In reply to Kirill A. ShutemovQuoted 21 lines · Kirill A. Shutemov
If the protected memory feature enabled, unmap guest memory from kernel's direct mappings.
Migration and KSM is disabled for protected memory as it would require a special treatment.
Signed-off-by Kirill A. Shutemov <[email protected]>--- include/linux/mm.h | 3 +++ mm/huge_memory.c | 8 ++++++++ mm/ksm.c | 2 ++ mm/memory.c | 12 ++++++++++++ mm/rmap.c | 4 ++++ virt/lib/mem_protected.c | 21 +++++++++++++++++++++ 6 files changed, 50 insertions(+)include/linux/mm.h +0
-0
include/linux/mm.h static inline bool vma_is_kvm_protected(struct vm_area_struct *vma) · line 675
Quoted 5 lines
include/linux/mm.h +2
-0@@ -671,6 +671,9 @@ static inline bool vma_is_kvm_protected(struct vm_area_struct *vma)return vma->vm_flags &VM_KVM_PROTECTED ; } +void kvm_map_page (struct page *page,int nr_pages); +void kvm_unmap_page (struct page *page,int nr_pages);
This still does not seem right ;-)
And I still think that map/unmap primitives shoud be a part of the generic mm rather than exported by KVM.
Quoted 138 lines
include/linux/mm.h +1
-0+#ifdef CONFIG_SHMEM /* * The vma_is_shmem is not inline because it is used only by slow mm/huge_memory.c +8
-0@@ -627,6 +627,10 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,spin_unlock (vmf->ptl);count_vm_event (THP_FAULT_ALLOC );count_memcg_event_mm (vma->vm_mm,THP_FAULT_ALLOC ); + +/* Unmap page from direct mapping */ +if (vma_is_kvm_protected (vma)) +kvm_unmap_page (page,HPAGE_PMD_NR ); }return 0 ;@@ -1689,6 +1693,10 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,page_remove_rmap (page, true);VM_BUG_ON_PAGE (page_mapcount (page) <0 , page);VM_BUG_ON_PAGE (!PageHead (page), page); + +/* Map the page back to the direct mapping */ +if (vma_is_kvm_protected (vma)) +kvm_map_page (page,HPAGE_PMD_NR ); }else if (thp_migration_supported ()) {swp_entry_t entry;mm/ksm.c +2
-0@@ -528,6 +528,8 @@ static struct vm_area_struct *find_mergeable_vma(struct mm_struct *mm,return NULL ;if (!(vma->vm_flags &VM_MERGEABLE ) || !vma->anon_vma)return NULL ; +if (vma_is_kvm_protected (vma)) +return NULL ;return vma; }mm/memory.c +12
-0@@ -1245,6 +1245,11 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, likely(!(vma->vm_flags & VM_SEQ_READ)))mark_page_accessed (page); } + +/* Map the page back to the direct mapping */ +if (vma_is_anonymous (vma) &&vma_is_kvm_protected (vma)) +kvm_map_page (page,1 ); + rss[mm_counter (page)]--;page_remove_rmap (page, false);if (unlikely (page_mapcount (page) <0 ))@@ -3466,6 +3471,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)struct page *page;vm_fault_t ret =0 ;pte_t entry; +bool set = false;/* File mapping without ->vm_ops ? */ if (vma->vm_flags &VM_SHARED )@@ -3554,6 +3560,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)inc_mm_counter_fast (vma->vm_mm,MM_ANONPAGES );page_add_new_anon_rmap (page, vma, vmf->address, false);lru_cache_add_inactive_or_unevictable (page, vma); + set = true; setpte:set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry);@@ -3561,6 +3568,11 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)update_mmu_cache (vma, vmf->address, vmf->pte); unlock:pte_unmap_unlock (vmf->pte, vmf->ptl); + +/* Unmap page from direct mapping */ +if (vma_is_kvm_protected (vma) && set) +kvm_unmap_page (page,1 ); +return ret; release:put_page (page);mm/rmap.c +4
-0@@ -1725,6 +1725,10 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,static bool invalid_migration_vma (struct vm_area_struct *vma,void *arg) { +/* TODO */ +if (vma_is_kvm_protected (vma)) +return true; +return vma_is_temporary_stack (vma); }virt/lib/mem_protected.c +21
-0@@ -30,6 +30,27 @@ void kvm_unmap_page_atomic(void *vaddr) }EXPORT_SYMBOL_GPL (kvm_unmap_page_atomic); +void kvm_map_page (struct page *page,int nr_pages) +{ +int i; + +/* Clear page before returning it to the direct mapping */ +for (i =0 ; i < nr_pages; i++) { +void *p =kvm_map_page_atomic (page + i); +memset (p,0 ,PAGE_SIZE ); +kvm_unmap_page_atomic (p); + } + +kernel_map_pages (page, nr_pages,1 ); +} +EXPORT_SYMBOL_GPL (kvm_map_page); + +void kvm_unmap_page (struct page *page,int nr_pages) +{ +kernel_map_pages (page, nr_pages,0 ); +} +EXPORT_SYMBOL_GPL (kvm_unmap_page); +int kvm_init_protected_memory (void ) { guest_map_ptes = kmalloc_array(num_possible_cpus(),Signature · 1 line
2.26.2
Signature · 2 lines
Sincerely yours, Mike.
Re: [RFCv2 15/16] KVM: Unmap protected pages from direct mapping
Quoted 25 lines · Mike Rapoport
Quoted 24 lines · Kirill A. Shutemov
If the protected memory feature enabled, unmap guest memory from kernel's direct mappings.
Migration and KSM is disabled for protected memory as it would require a special treatment.
Signed-off-by Kirill A. Shutemov <[email protected]>--- include/linux/mm.h | 3 +++ mm/huge_memory.c | 8 ++++++++ mm/ksm.c | 2 ++ mm/memory.c | 12 ++++++++++++ mm/rmap.c | 4 ++++ virt/lib/mem_protected.c | 21 +++++++++++++++++++++ 6 files changed, 50 insertions(+)include/linux/mm.h +0
-0@@ -671,6 +671,9 @@ static inline bool vma_is_kvm_protected(struct vm_area_struct *vma)return vma->vm_flags &VM_KVM_PROTECTED ; }
Quoted 7 lines
include/linux/mm.h static inline bool vma_is_kvm_protected(struct vm_area_struct *vma) · line 675
Quoted 2 lines
include/linux/mm.h +2
-0+void kvm_map_page (struct page *page,int nr_pages); +void kvm_unmap_page (struct page *page,int nr_pages);This still does not seem right ;-)
And I still think that map/unmap primitives shoud be a part of the generic mm rather than exported by KVM.
Ya, and going a step further, I suspect it will be cleaner in the long run if the kernel does not automatically map or unmap when converting between private and shared/public memory. Conversions will be rare in a well behaved guest, so exiting to userspace and forcing userspace to do the unmap->map would not be a performance bottleneck. In theory, userspace could also maintain separate pools for private vs. public mappings, though I doubt any VMM will do that in practice.
RFC v2 16/16 mm: Do not use zero page for VM_KVM_PROTECTED VMAs
In reply to Kirill A. ShutemovPresence of zero pages in the mapping would disclose content of the mapping. Don't use them if KVM memory protection is enabled.
--- arch/s390/include/asm/pgtable.h | 2 +- include/linux/mm.h | 4 ++-- mm/huge_memory.c | 3 +-- mm/memory.c | 3 +-- 4 files changed, 5 insertions(+), 7 deletions(-)
arch/s390/include/asm/pgtable.h +1 -1
@@ -543,7 +543,7 @@ static inline int mm_alloc_pgste(struct mm_struct *mm)* In the case that a guest uses storage keys * faults should no longer be backed by zero pages */ -#define mm_forbids_zeropage mm_has_pgste +#define vma_forbids_zeropage (vma) mm_has_pgste(vma->vm_mm)static inline int mm_uses_skeys (struct mm_struct *mm) {#ifdef CONFIG_PGSTE
include/linux/mm.h +2 -2
@@ -130,8 +130,8 @@ extern int mmap_rnd_compat_bits __read_mostly;* s390 does this to prevent multiplexing of hardware bits * related to the physical page in case of virtualization. */ -#ifndef mm_forbids_zeropage -#define mm_forbids_zeropage (X ) (0) +#ifndef vma_forbids_zeropage +#define vma_forbids_zeropage (vma) vma_is_kvm_protected(vma)#endif /*
mm/huge_memory.c +1 -2
@@ -709,8 +709,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)return VM_FAULT_OOM ;if (unlikely (khugepaged_enter (vma, vma->vm_flags)))return VM_FAULT_OOM ; -if (!(vmf->flags &FAULT_FLAG_WRITE ) && - !mm_forbids_zeropage (vma->vm_mm) && +if (!(vmf->flags &FAULT_FLAG_WRITE ) && !vma_forbids_zeropage (vma) &&transparent_hugepage_use_zero_page ()) {pgtable_t pgtable;struct page *zero_page;
mm/memory.c +1 -2
@@ -3495,8 +3495,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)return 0 ;/* Use the zero-page for reads */ -if (!(vmf->flags &FAULT_FLAG_WRITE ) && - !mm_forbids_zeropage (vma->vm_mm)) { +if (!(vmf->flags &FAULT_FLAG_WRITE ) && !vma_forbids_zeropage (vma)) { entry =pte_mkspecial (pfn_pte (my_zero_pfn (vmf->address), vma->vm_page_prot)); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
Signature · 1 line
2.26.2
Vitaly Kuznetsov
reviewerQuoted 25 lines · Kirill A. Shutemov
== Background / Problem ==
There are a number of hardware features (MKTME, SEV) which protect guest memory from some unauthorized host access. The patchset proposes a purely software feature that mitigates some of the same host-side read-only attacks.
== What does this set mitigate? ==
- Host kernel ”accidental” access to guest data (think speculation)
- Host kernel induced access to guest data (
write(fd, &guest_data_ptr, len))- Host userspace access to guest data (compromised qemu)
- Guest privilege escalation via compromised QEMU device emulation
== What does this set NOT mitigate? ==
- Full host kernel compromise. Kernel will just map the pages again.
- Hardware attacks
Quoted 7 lines
The second RFC revision addresses /most/ of the feedback.
I still didn't found a good solution to reboot and kexec. Unprotect all the memory on such operations defeat the goal of the feature. Clearing up most of the memory before unprotecting what is required for reboot (or kexec) is tedious and error-prone.
Maybe we should just declare them unsupported?
Making reboot unsupported is a hard sell. Could you please elaborate on why you think that "unprotect all" hypercall (or rather a single hypercall supporting both protecting/unprotecting) defeats the purpose of the feature?
(Leaving kexec aside for a while) Yes, it is not easy for a guest to clean up *all* its memory upon reboot, however:
- It may only clean up the most sensitive parts. This should probably be done even without this new feature and even on bare metal (think about next boot target being malicious).
- The attack window shrinks significantly. "Speculative" bugs require time to exploit and it will only remain open until it boots up again (few seconds).
Quoted 123 lines
== Series Overview ==
The hardware features protect guest data by encrypting it and then ensuring that only the right guest can decrypt it. This has the side-effect of making the kernel direct map and userspace mapping (QEMU et al) useless. But, this teaches us something very useful:
neither the kernel or userspace mappings are really necessary for normal guest operations.Instead of using encryption, this series simply unmaps the memory. One advantage compared to allowing access to ciphertext is that it allows bad accesses to be caught instead of simply reading garbage.
Protection from physical attacks needs to be provided by some other means. On Intel platforms, (single-key) Total Memory Encryption (TME) provides mitigation against physical attacks, such as DIMM interposers sniffing memory bus traffic.
The patchset modifies both host and guest kernel. The guest OS must enable the feature via hypercall and mark any memory range that has to be shared with the host: DMA regions, bounce buffers, etc. SEV does this marking via a bit in the guest’s page table while this approach uses a hypercall.
For removing the userspace mapping, use a trick similar to what NUMA balancing does: convert memory that belongs to KVM memory slots to
PROT_NONE: all existing entries converted toPROT_NONEwithmprotect()and the newly faulted in pages getPROT_NONEfrom the updatedvm_page_prot. The new VMA flag --VM_KVM_PROTECTED-- indicates that the pages in the VMA must be treated in a special way in the GUP and fault paths. The flag allows GUP to return the page even though it is mapped withPROT_NONE, but only if the new GUP flag --FOLL_KVM-- is specified. Any userspace access to the memory would result in SIGBUS. Any GUP access withoutFOLL_KVMwould result in-EFAULT.Removing userspace mapping of the guest memory from QEMU process can help to address some guest privilege escalation attacks. Consider the case when unprivileged guest user exploits bug in a QEMU device emulation to gain access to data it cannot normally have access within the guest.
Any anonymous page faulted into the
VM_KVM_PROTECTEDVMA gets removed from the direct mapping withkernel_map_pages(). Note thatkernel_map_pages()only flushes local TLB. I think it's a reasonable compromise between security and perfromance.Zapping the PTE would bring the page back to the direct mapping after clearing. At least for now, we don't remove file-backed pages from the direct mapping. File-backed pages could be accessed via read/write syscalls. It adds complexity.
Occasionally, host kernel has to access guest memory that was not made shared by the guest. For instance, it happens for instruction emulation. Normally, it's done via copy_to/from_user() which would fail with
-EFAULTnow. We introduced a new pair of helpers: copy_to/from_guest(). The new helpers acquire the page via GUP, map it into kernel address space with kmap_atomic()-style mechanism and only then copy the data.For some instruction emulation copying is not good enough: cmpxchg emulation has to have direct access to the guest memory.
__kvm_map_gfn()is modified to accommodate the case.The patchset is on top of v5.9
Kirill A. Shutemov (16):
x86/mm: Moveforce_dma_unencrypted()to common code x86/kvm: Introduce KVM memory protection feature
x86/kvm: Make DMA pages shared
x86/kvm: Use bounce buffers for KVM memory protection x86/kvm: Make VirtIO use DMA API in KVM guest x86/kvmclock: Share hvclock memory with the host x86/realmode: Share trampoline area if KVM memory protection enabled
KVM: Use GUP instead of copy_from/to_user() to access guest memory
KVM: mm: IntroduceVM_KVM_PROTECTED
KVM: x86: Use GUP for page walk instead of__get_user()
KVM: Protected memory extension
KVM: x86: Enabled protected memory extension
KVM: Rework copy_to/from_guest() to avoid direct mapping
KVM: Handle protected memory in __kvm_map_gfn()/__kvm_unmap_gfn()
KVM: Unmap protected pages from direct mapping
mm: Do not use zero page forVM_KVM_PROTECTEDVMAsarch/powerpc/kvm/book3s_64_mmu_hv.c | 2 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 2 +- arch/s390/include/asm/pgtable.h | 2 +- arch/x86/Kconfig | 11 +- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/io.h | 6 +- arch/x86/include/asm/kvm_para.h | 5 + arch/x86/include/asm/pgtable_types.h | 1 + arch/x86/include/uapi/asm/kvm_para.h | 3 +- arch/x86/kernel/kvm.c | 20 +++ arch/x86/kernel/kvmclock.c | 2 +- arch/x86/kernel/pci-swiotlb.c | 3 +- arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/cpuid.c | 3 +- arch/x86/kvm/mmu/mmu.c | 6 +- arch/x86/kvm/mmu/paging_tmpl.h | 10 +- arch/x86/kvm/x86.c | 9 + arch/x86/mm/Makefile | 2 + arch/x86/mm/ioremap.c | 16 +- arch/x86/mm/mem_encrypt.c | 51 ------ arch/x86/mm/mem_encrypt_common.c | 62 +++++++ arch/x86/mm/pat/set_memory.c | 7 + arch/x86/realmode/init.c | 7 +- drivers/virtio/virtio_ring.c | 4 + include/linux/kvm_host.h | 11 +- include/linux/kvm_types.h | 1 + include/linux/mm.h | 21 ++- include/uapi/linux/kvm_para.h | 5 +- mm/gup.c | 20 ++- mm/huge_memory.c | 31 +++- mm/ksm.c | 2 + mm/memory.c | 18 +- mm/mmap.c | 3 + mm/rmap.c | 4 + virt/kvm/Kconfig | 3 + virt/kvm/async_pf.c | 2 +- virt/kvm/kvm_main.c | 238 +++++++++++++++++++++---- virt/lib/Makefile | 1 + virt/lib/mem_protected.c | 193 ++++++++++++++++++++ 39 files changed, 666 insertions(+), 123 deletions(-) create mode 100644 arch/x86/mm/mem_encrypt_common.c create mode 100644 virt/lib/mem_protected.c
Signature · 1 line
Vitaly
Quoted 35 lines · Vitaly Kuznetsov
Quoted 25 lines · Kirill A. Shutemov
== Background / Problem ==
There are a number of hardware features (MKTME, SEV) which protect guest memory from some unauthorized host access. The patchset proposes a purely software feature that mitigates some of the same host-side read-only attacks.
== What does this set mitigate? ==
- Host kernel ”accidental” access to guest data (think speculation)
- Host kernel induced access to guest data (
write(fd, &guest_data_ptr, len))- Host userspace access to guest data (compromised qemu)
- Guest privilege escalation via compromised QEMU device emulation
== What does this set NOT mitigate? ==
- Full host kernel compromise. Kernel will just map the pages again.
- Hardware attacks
Quoted 7 lines
The second RFC revision addresses /most/ of the feedback.
I still didn't found a good solution to reboot and kexec. Unprotect all the memory on such operations defeat the goal of the feature. Clearing up most of the memory before unprotecting what is required for reboot (or kexec) is tedious and error-prone.
Maybe we should just declare them unsupported?
Quoted 4 lines
Making reboot unsupported is a hard sell. Could you please elaborate on why you think that "unprotect all" hypercall (or rather a single hypercall supporting both protecting/unprotecting) defeats the purpose of the feature?
If guest has some data that it prefers not to leak to the host and use the feature for the purpose, share all the memory to get through reboot is a very weak point.
Quoted 8 lines
clean up *all* its memory upon reboot, however:
- It may only clean up the most sensitive parts. This should probably be done even without this new feature and even on bare metal (think about next boot target being malicious).
- The attack window shrinks significantly. "Speculative" bugs require time to exploit and it will only remain open until it boots up again (few seconds).
Maybe it would be cleaner to handle reboot in userspace? If we got the VM rebooted, just reconstruct it from scratch as if it would be new boot.
Signature · 1 line
Kirill A. Shutemov
Vitaly Kuznetsov
reviewerQuoted 36 lines · Kirill A. Shutemov
Quoted 35 lines · Vitaly Kuznetsov
Quoted 32 lines · Kirill A. Shutemov
== Background / Problem ==
There are a number of hardware features (MKTME, SEV) which protect guest memory from some unauthorized host access. The patchset proposes a purely software feature that mitigates some of the same host-side read-only attacks.
== What does this set mitigate? ==
- Host kernel ”accidental” access to guest data (think speculation)
- Host kernel induced access to guest data (
write(fd, &guest_data_ptr, len))- Host userspace access to guest data (compromised qemu)
- Guest privilege escalation via compromised QEMU device emulation
== What does this set NOT mitigate? ==
- Full host kernel compromise. Kernel will just map the pages again.
- Hardware attacks
The second RFC revision addresses /most/ of the feedback.
I still didn't found a good solution to reboot and kexec. Unprotect all the memory on such operations defeat the goal of the feature. Clearing up most of the memory before unprotecting what is required for reboot (or kexec) is tedious and error-prone.
Maybe we should just declare them unsupported?
Quoted 9 lines
Quoted 4 lines
Making reboot unsupported is a hard sell. Could you please elaborate on why you think that "unprotect all" hypercall (or rather a single hypercall supporting both protecting/unprotecting) defeats the purpose of the feature?
If guest has some data that it prefers not to leak to the host and use the feature for the purpose, share all the memory to get through reboot is a very weak point.
My point that if it knows that there's something sensitive in its memory it should clean it up even today without your feature before rebooting to an unknown target.
Quoted 11 lines
Quoted 8 lines
clean up *all* its memory upon reboot, however:
- It may only clean up the most sensitive parts. This should probably be done even without this new feature and even on bare metal (think about next boot target being malicious).
- The attack window shrinks significantly. "Speculative" bugs require time to exploit and it will only remain open until it boots up again (few seconds).Maybe it would be cleaner to handle reboot in userspace? If we got the VM rebooted, just reconstruct it from scratch as if it would be new boot.
We are definitely not trying to protect against malicious KVM so maybe we can do the cleanup there (when protection was enabled) so we can unprotect everything without risk of a leak?
Signature · 1 line
Vitaly
Quoted 43 lines · Vitaly Kuznetsov
Quoted 41 lines · Kirill A. Shutemov
Quoted 39 lines · Vitaly Kuznetsov
Quoted 25 lines · Kirill A. Shutemov
== Background / Problem ==
There are a number of hardware features (MKTME, SEV) which protect guest memory from some unauthorized host access. The patchset proposes a purely software feature that mitigates some of the same host-side read-only attacks.
== What does this set mitigate? ==
- Host kernel ”accidental” access to guest data (think speculation)
- Host kernel induced access to guest data (
write(fd, &guest_data_ptr, len))- Host userspace access to guest data (compromised qemu)
- Guest privilege escalation via compromised QEMU device emulation
== What does this set NOT mitigate? ==
- Full host kernel compromise. Kernel will just map the pages again.
- Hardware attacks
Quoted 7 lines
The second RFC revision addresses /most/ of the feedback.
I still didn't found a good solution to reboot and kexec. Unprotect all the memory on such operations defeat the goal of the feature. Clearing up most of the memory before unprotecting what is required for reboot (or kexec) is tedious and error-prone.
Maybe we should just declare them unsupported?Making reboot unsupported is a hard sell. Could you please elaborate on why you think that "unprotect all" hypercall (or rather a single hypercall supporting both protecting/unprotecting) defeats the purpose of the feature?
Quoted 8 lines
Quoted 4 lines
If guest has some data that it prefers not to leak to the host and use the feature for the purpose, share all the memory to get through reboot is a very weak point.
My point that if it knows that there's something sensitive in its memory it should clean it up even today without your feature before rebooting to an unknown target.
It's unrealistic to expect everybody to do the right thing.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
clean up *all* its memory upon reboot, however:
- It may only clean up the most sensitive parts. This should probably be done even without this new feature and even on bare metal (think about next boot target being malicious).
- The attack window shrinks significantly. "Speculative" bugs require time to exploit and it will only remain open until it boots up again (few seconds).
Quoted 6 lines
Quoted 2 lines
Maybe it would be cleaner to handle reboot in userspace? If we got the VM rebooted, just reconstruct it from scratch as if it would be new boot.
We are definitely not trying to protect against malicious KVM so maybe we can do the cleanup there (when protection was enabled) so we can unprotect everything without risk of a leak?
Do you have any particular codepath in mind? I didn't find anything suitable so far.
Signature · 1 line
Kirill A. Shutemov
Vitaly Kuznetsov
reviewerQuoted 6 lines · Kirill A. Shutemov
Quoted 5 lines · Vitaly Kuznetsov
Quoted 2 lines · Kirill A. Shutemov
Maybe it would be cleaner to handle reboot in userspace? If we got the VM rebooted, just reconstruct it from scratch as if it would be new boot.
Quoted 6 lines
Quoted 3 lines
We are definitely not trying to protect against malicious KVM so maybe we can do the cleanup there (when protection was enabled) so we can unprotect everything without risk of a leak?
Do you have any particular codepath in mind? I didn't find anything suitable so far.
I didn't put much thought in it but e.g. on x86, what if we put this to kvm_vcpu_reset() under 'if (kvm_vcpu_is_bsp())' condition?
The main problem I see is that we can't clean up *all* memory, e.g. firmware related stuff should stay intact and this contraducts your KVM_HC_ENABLE_MEM_PROTECTED which protects everything. We can, probably, get rid of it leaving KVM_HC_MEM_SHARE/KVM_HC_MEM_UNSHARE only shifting responsibility to define what can be cleaned up on the guest kernel (stating in the doc that all protected memory will get whiped out on reboot).
Signature · 1 line
Vitaly
Quoted 13 lines · Kirill A. Shutemov
For removing the userspace mapping, use a trick similar to what NUMA balancing does: convert memory that belongs to KVM memory slots to
PROT_NONE: all existing entries converted toPROT_NONEwithmprotect()and the newly faulted in pages getPROT_NONEfrom the updatedvm_page_prot. The new VMA flag --VM_KVM_PROTECTED-- indicates that the pages in the VMA must be treated in a special way in the GUP and fault paths. The flag allows GUP to return the page even though it is mapped withPROT_NONE, but only if the new GUP flag --FOLL_KVM-- is specified. Any userspace access to the memory would result in SIGBUS. Any GUP access withoutFOLL_KVMwould result in-EFAULT.
I definitely like the direction this patchset is going in, and I think that allowing KVM guests to have memory that is inaccessible to QEMU is a great idea.
I do wonder, though: do we really want to do this with these PROT_NONE tricks, or should we actually come up with a way to have KVM guest map memory that isn't mapped into QEMU's mm_struct at all? As an example of the latter, I mean something a bit like this:
https://lkml.kernel.org/r/CALCETrUSUp_7svg8EHNTk3nQ0x9sdzMCU=h8G-Sy6=SODq5GHg@mail.gmail.com
I don't mean to say that this is a requirement of any kind of protected memory like this, but I do think we should understand the tradeoffs, in terms of what a full implementation looks like, the effort and time frames involved, and the maintenance burden of supporting whatever gets merged going forward.
Quoted 23 lines · Andy Lutomirski
Quoted 13 lines · Kirill A. Shutemov
For removing the userspace mapping, use a trick similar to what NUMA balancing does: convert memory that belongs to KVM memory slots to
PROT_NONE: all existing entries converted toPROT_NONEwithmprotect()and the newly faulted in pages getPROT_NONEfrom the updatedvm_page_prot. The new VMA flag --VM_KVM_PROTECTED-- indicates that the pages in the VMA must be treated in a special way in the GUP and fault paths. The flag allows GUP to return the page even though it is mapped withPROT_NONE, but only if the new GUP flag --FOLL_KVM-- is specified. Any userspace access to the memory would result in SIGBUS. Any GUP access withoutFOLL_KVMwould result in-EFAULT.I definitely like the direction this patchset is going in, and I think that allowing KVM guests to have memory that is inaccessible to QEMU is a great idea.
I do wonder, though: do we really want to do this with these
PROT_NONEtricks, or should we actually come up with a way to have KVM guest map memory that isn't mapped into QEMU'smm_structat all? As an example of the latter, I mean something a bit like this:
Quoted 7 lines
https://lkml.kernel.org/r/CALCETrUSUp_7svg8EHNTk3nQ0x9sdzMCU=h8G-Sy6=SODq5GHg@mail.gmail.com
I don't mean to say that this is a requirement of any kind of protected memory like this, but I do think we should understand the tradeoffs, in terms of what a full implementation looks like, the effort and time frames involved, and the maintenance burden of supporting whatever gets merged going forward.
I considered the PROT_NONE trick neat. Complete removing of the mapping from QEMU would require more changes into KVM and I'm not really familiar with it.
About tradeoffs: the trick interferes with AutoNUMA. I didn't put much thought into how we can get it work together. Need to look into it.
Do you see other tradeoffs?
Signature · 1 line
Kirill A. Shutemov
Andy Lutomirski
maintainerQuoted 33 lines · Kirill A. Shutemov
Quoted 23 lines · Andy Lutomirski
Quoted 13 lines · Kirill A. Shutemov
For removing the userspace mapping, use a trick similar to what NUMA balancing does: convert memory that belongs to KVM memory slots to
PROT_NONE: all existing entries converted toPROT_NONEwithmprotect()and the newly faulted in pages getPROT_NONEfrom the updatedvm_page_prot. The new VMA flag --VM_KVM_PROTECTED-- indicates that the pages in the VMA must be treated in a special way in the GUP and fault paths. The flag allows GUP to return the page even though it is mapped withPROT_NONE, but only if the new GUP flag --FOLL_KVM-- is specified. Any userspace access to the memory would result in SIGBUS. Any GUP access withoutFOLL_KVMwould result in-EFAULT.I definitely like the direction this patchset is going in, and I think that allowing KVM guests to have memory that is inaccessible to QEMU is a great idea.
I do wonder, though: do we really want to do this with these
PROT_NONEtricks, or should we actually come up with a way to have KVM guest map memory that isn't mapped into QEMU'smm_structat all? As an example of the latter, I mean something a bit like this:Quoted 7 lines
https://lkml.kernel.org/r/CALCETrUSUp_7svg8EHNTk3nQ0x9sdzMCU=h8G-Sy6=SODq5GHg@mail.gmail.com
I don't mean to say that this is a requirement of any kind of protected memory like this, but I do think we should understand the tradeoffs, in terms of what a full implementation looks like, the effort and time frames involved, and the maintenance burden of supporting whatever gets merged going forward.
Quoted 3 lines
I considered the
PROT_NONEtrick neat. Complete removing of the mapping from QEMU would require more changes into KVM and I'm not really familiar with it.
I think it's neat. The big tradeoff I'm concerned about is that it will likely become ABI once it lands. That is, if this series lands, then we will always have to support the case in which QEMU has a special non-present mapping that is nonetheless reflected as present in a guest. This is a bizarre state of affairs, it may become obsolete if a better API ever shows up, and it might end up placing constraints on the Linux VM that we don't love going forward.
I don't think my proposal in the referenced thread above is that crazy or that difficult to implement. The basic idea is to have a way to create an mm_struct that is not loaded in CR3 anywhere. Instead, KVM will reference it, much as it currently references QEMU's mm_struct, to mirror mappings into the guest. This means it would be safe to have "protected" memory mapped into the special mm_struct because nothing other than KVM will ever reference the PTEs. But I think that someone who really understands the KVM memory mapping code should chime in.
Quoted 8 lines
About tradeoffs: the trick interferes with AutoNUMA. I didn't put much thought into how we can get it work together. Need to look into it.
Do you see other tradeoffs?
--
Kirill A. Shutemov