Linear Address Masking enabling
Subsystems (28)
v6 cover Linear Address Masking enabling
Linear Address Masking[1] (LAM) modifies the checking that is applied to 64-bit linear addresses, allowing software to use of the untranslated address bits for metadata.
The patchset brings support for LAM for userspace addresses. Only LAM_U57 at this time.
Please review and consider applying.
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git lam
v6:
- Rebased onto v6.0-rc1
- LAM_U48 excluded from the patchet. Still available in the git tree;
- Add ARCH_GET_MAX_TAG_BITS (and tests for it);
- Fix build without CONFIG_DEBUG_VM;
- Update comments;
- Reviewed/Tested-by from Alexander;
v5:
- Do not use switch_mm() in enable_lam_func()
- Use mb()/READ_ONCE() pair on LAM enabling;
- Add self-test by Weihong Zhang;
- Add comments;
v4:
- Fix untagged_addr() for LAM_U48;
- Remove no-threads restriction on LAM enabling;
- Fix mm_struct access from /proc/$PID/arch_status
- Fix LAM handling in initialize_tlbstate_and_flush()
- Pack tlb_state better;
- Comments and commit messages;
v3:
- Rebased onto v5.19-rc1
- Per-process enabling;
- API overhaul (again);
- Avoid branches and costly computations in the fast path;
- LAM_U48 is in optional patch.
v2:
- Rebased onto v5.18-rc1
- New arch_prctl(2)-based API
- Expose status of LAM (or other thread features) in /proc/$PID/arch_status
[1] ISE, Chapter 10. https://cdrdv2.intel.com/v1/dl/getContent/671368
Kirill A. Shutemov (7):
x86/mm: Fix CR3_ADDR_MASK
x86: CPUID and CR3/CR4 flags for Linear Address Masking
mm: Pass down mm_struct to untagged_addr()
x86/mm: Handle LAM on context switch
x86/uaccess: Provide untagged_addr() and remove tags before address
check
x86/mm: Provide arch_prctl() interface for LAM
x86: Expose untagging mask in /proc/$PID/arch_status
Weihong Zhang (4):
selftests/x86/lam: Add malloc and tag-bits test cases for
linear-address masking
selftests/x86/lam: Add mmap and SYSCALL test cases for linear-address
masking
selftests/x86/lam: Add io_uring test cases for linear-address masking selftests/x86/lam: Add inherit test cases for linear-address masking
arch/arm64/include/asm/memory.h | 4 +- arch/arm64/include/asm/signal.h | 2 +- arch/arm64/include/asm/uaccess.h | 4 +- arch/arm64/kernel/hw_breakpoint.c | 2 +- arch/arm64/kernel/traps.c | 4 +- arch/arm64/mm/fault.c | 10 +- arch/sparc/include/asm/pgtable_64.h | 2 +- arch/sparc/include/asm/uaccess_64.h | 2 + arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/mmu.h | 6 + arch/x86/include/asm/mmu_context.h | 45 + arch/x86/include/asm/processor-flags.h | 4 +- arch/x86/include/asm/tlbflush.h | 35 + arch/x86/include/asm/uaccess.h | 42 +- arch/x86/include/uapi/asm/prctl.h | 4 + arch/x86/include/uapi/asm/processor-flags.h | 6 + arch/x86/kernel/Makefile | 2 + arch/x86/kernel/fpu/xstate.c | 47 - arch/x86/kernel/proc.c | 60 ++ arch/x86/kernel/process.c | 3 + arch/x86/kernel/process_64.c | 65 +- arch/x86/mm/tlb.c | 43 +- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 2 +- drivers/gpu/drm/radeon/radeon_gem.c | 2 +- drivers/infiniband/hw/mlx4/mr.c | 2 +- drivers/media/common/videobuf2/frame_vector.c | 2 +- drivers/media/v4l2-core/videobuf-dma-contig.c | 2 +- .../staging/media/atomisp/pci/hmm/hmm_bo.c | 2 +- drivers/tee/tee_shm.c | 2 +- drivers/vfio/vfio_iommu_type1.c | 2 +- fs/proc/task_mmu.c | 2 +- include/linux/mm.h | 11 - include/linux/uaccess.h | 15 + lib/strncpy_from_user.c | 2 +- lib/strnlen_user.c | 2 +- mm/gup.c | 6 +- mm/madvise.c | 2 +- mm/mempolicy.c | 6 +- mm/migrate.c | 2 +- mm/mincore.c | 2 +- mm/mlock.c | 4 +- mm/mmap.c | 2 +- mm/mprotect.c | 2 +- mm/mremap.c | 2 +- mm/msync.c | 2 +- tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/lam.c | 900 ++++++++++++++++++ virt/kvm/kvm_main.c | 2 +- 49 files changed, 1256 insertions(+), 121 deletions(-) create mode 100644 arch/x86/kernel/proc.c create mode 100644 tools/testing/selftests/x86/lam.c
Signature · 1 line
2.35.1
PATCH v6 1/11 x86/mm: Fix CR3_ADDR_MASK
The mask must not include bits above physical address mask. These bits are reserved and can be used for other things. Bits 61 and 62 are used for Linear Address Masking.
--- arch/x86/include/asm/processor-flags.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/processor-flags.h +1 -1
@@ -35,7 +35,7 @@*/ #ifdef CONFIG_X86_64 /* Mask off the address space ID and SME encryption bits. */ -#define CR3_ADDR_MASK __sme_clr(0x7FFFFFFFFFFFF000ull) +#define CR3_ADDR_MASK __sme_clr(PHYSICAL_PAGE_MASK)#define CR3_PCID_MASK 0xFFFull#define CR3_NOFLUSH BIT_ULL(63)
Signature · 1 line
2.35.1
PATCH v6 2/11 x86: CPUID and CR3/CR4 flags for Linear Address Masking
In reply to Kirill A. ShutemovEnumerate Linear Address Masking and provide defines for CR3 and CR4 flags.
--- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/processor-flags.h | 2 ++ arch/x86/include/uapi/asm/processor-flags.h | 6 ++++++ 3 files changed, 9 insertions(+)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -308,6 +308,7 @@/* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4)/* AVX VNNI instructions */ #define X86_FEATURE_AVX512_BF16 (12*32+ 5)/* AVX512 BFLOAT16 instructions */ +#define X86_FEATURE_LAM (12*32+26)/* Linear Address Masking */ /* AMD-defined CPU features, CPUID level 0x80000008 (EBX), word 13 */ #define X86_FEATURE_CLZERO (13*32+ 0)/* CLZERO instruction */
arch/x86/include/asm/processor-flags.h +2 -0
@@ -28,6 +28,8 @@* On systems with SME, one bit (in a variable position!) is stolen to indicate * that the top-level paging structure is encrypted. * +* On systemms with LAM, bits 61 and 62 are used to indicate LAM mode. +* * All of the remaining bits indicate the physical address of the top-level * paging structure. *
arch/x86/include/uapi/asm/processor-flags.h +6 -0
@@ -82,6 +82,10 @@#define X86_CR3_PCID_BITS 12#define X86_CR3_PCID_MASK (_AC((1UL << X86_CR3_PCID_BITS) - 1, UL)) +#define X86_CR3_LAM_U57_BIT 61/* Activate LAM for userspace, 62:57 bits masked */ +#define X86_CR3_LAM_U57 _BITULL(X86_CR3_LAM_U57_BIT) +#define X86_CR3_LAM_U48_BIT 62/* Activate LAM for userspace, 62:48 bits masked */ +#define X86_CR3_LAM_U48 _BITULL(X86_CR3_LAM_U48_BIT)#define X86_CR3_PCID_NOFLUSH_BIT 63/* Preserve old PCID */ #define X86_CR3_PCID_NOFLUSH _BITULL(X86_CR3_PCID_NOFLUSH_BIT)
@@ -132,6 +136,8 @@#define X86_CR4_PKE _BITUL(X86_CR4_PKE_BIT)#define X86_CR4_CET_BIT 23/* enable Control-flow Enforcement Technology */ #define X86_CR4_CET _BITUL(X86_CR4_CET_BIT) +#define X86_CR4_LAM_SUP_BIT 28/* LAM for supervisor pointers */ +#define X86_CR4_LAM_SUP _BITUL(X86_CR4_LAM_SUP_BIT)/* * x86-64 Task Priority Register, CR8
Signature · 1 line
2.35.1
PATCH v6 3/11 mm: Pass down mm_struct to untagged_addr()
In reply to Kirill A. ShutemovIntel Linear Address Masking (LAM) brings per-mm untagging rules. Pass down mm_struct to the untagging helper. It will help to apply untagging policy correctly.
In most cases, current->mm is the one to use, but there are some exceptions, such as get_user_page_remote().
Move dummy implementation of untagged_addr() from <linux/mm.h> to <linux/uaccess.h>. <asm/uaccess.h> can override the implementation. Moving the dummy header outside <linux/mm.h> helps to avoid header hell if you need to defer mm_struct within the helper.
--- arch/arm64/include/asm/memory.h | 4 ++-- arch/arm64/include/asm/signal.h | 2 +- arch/arm64/include/asm/uaccess.h | 4 ++-- arch/arm64/kernel/hw_breakpoint.c | 2 +- arch/arm64/kernel/traps.c | 4 ++-- arch/arm64/mm/fault.c | 10 +++++----- arch/sparc/include/asm/pgtable_64.h | 2 +- arch/sparc/include/asm/uaccess_64.h | 2 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 2 +- drivers/gpu/drm/radeon/radeon_gem.c | 2 +- drivers/infiniband/hw/mlx4/mr.c | 2 +- drivers/media/common/videobuf2/frame_vector.c | 2 +- drivers/media/v4l2-core/videobuf-dma-contig.c | 2 +- drivers/staging/media/atomisp/pci/hmm/hmm_bo.c | 2 +- drivers/tee/tee_shm.c | 2 +- drivers/vfio/vfio_iommu_type1.c | 2 +- fs/proc/task_mmu.c | 2 +- include/linux/mm.h | 11 ----------- include/linux/uaccess.h | 15 +++++++++++++++ lib/strncpy_from_user.c | 2 +- lib/strnlen_user.c | 2 +- mm/gup.c | 6 +++--- mm/madvise.c | 2 +- mm/mempolicy.c | 6 +++--- mm/migrate.c | 2 +- mm/mincore.c | 2 +- mm/mlock.c | 4 ++-- mm/mmap.c | 2 +- mm/mprotect.c | 2 +- mm/mremap.c | 2 +- mm/msync.c | 2 +- virt/kvm/kvm_main.c | 2 +- 33 files changed, 59 insertions(+), 53 deletions(-)
arch/arm64/include/asm/memory.h +2 -2
@@ -227,8 +227,8 @@ static inline unsigned long kaslr_offset(void)#define __untagged_addr (addr) \ ((__force __typeof__(addr))sign_extend64((__force u64)(addr), 55)) -#define untagged_addr (addr) ({ \ - u64 __addr = (__force u64)(addr); \ +#define untagged_addr (mm, addr) ({ \ + u64 __addr = (__force u64)(addr); \ __addr &= __untagged_addr(__addr); \ (__force __typeof__(addr))__addr; \ })
arch/arm64/include/asm/signal.h +1 -1
@@ -18,7 +18,7 @@ static inline void __user *arch_untagged_si_addr(void __user *addr,if (sig ==SIGTRAP && si_code ==TRAP_BRKPT )return addr; -return untagged_addr (addr); +return untagged_addr (current->mm, addr); }#define arch_untagged_si_addr arch_untagged_si_addr
arch/arm64/include/asm/uaccess.h +2 -2
@@ -44,7 +44,7 @@ static inline int access_ok(const void __user *addr, unsigned long size)*/ if (IS_ENABLED (CONFIG_ARM64_TAGGED_ADDR_ABI ) && (current->flags &PF_KTHREAD ||test_thread_flag (TIF_TAGGED_ADDR ))) - addr =untagged_addr (addr); + addr =untagged_addr (current->mm, addr);return likely (__access_ok (addr, size)); }
@@ -217,7 +217,7 @@ static inline void __user *__uaccess_mask_ptr(const void __user *ptr) " csel %0, %1, xzr, eq\n" : "=&r" (safe_ptr) : "r" (ptr), "r" (TASK_SIZE_MAX - 1), - "r" (untagged_addr(ptr)) + "r" (untagged_addr(current->mm, ptr)) : "cc");csdb ();
arch/arm64/kernel/hw_breakpoint.c +1 -1
@@ -715,7 +715,7 @@ static u64 get_distance_from_watchpoint(unsigned long addr, u64 val,u64 wp_low, wp_high;u32 lens, lene; - addr =untagged_addr (addr); + addr =untagged_addr (current->mm, addr); lens =__ffs (ctrl->len); lene =__fls (ctrl->len);
arch/arm64/kernel/traps.c +2 -2
@@ -476,7 +476,7 @@ void arm64_notify_segfault(unsigned long addr)int code;mmap_read_lock (current->mm); -if (find_vma (current->mm,untagged_addr (addr)) ==NULL ) +if (find_vma (current->mm,untagged_addr (current->mm, addr)) ==NULL ) code =SEGV_MAPERR ;else code =SEGV_ACCERR ;
@@ -540,7 +540,7 @@ static void user_cache_maint_handler(unsigned long esr, struct pt_regs *regs)int ret =0 ; tagged_address =pt_regs_read_reg (regs, rt); - address =untagged_addr (tagged_address); + address =untagged_addr (current->mm, tagged_address);switch (crm) {case ESR_ELx_SYS64_ISS_CRM_DC_CVAU:/* DC CVAU, gets promoted */
arch/arm64/mm/fault.c +5 -5
@@ -454,7 +454,7 @@ static void set_thread_esr(unsigned long address, unsigned long esr)static void do_bad_area (unsigned long far,unsigned long esr,struct pt_regs *regs) { -unsigned long addr =untagged_addr (far); +unsigned long addr =untagged_addr (current->mm, far);/* * If we are in kernel mode at this point, we have no context to
@@ -524,7 +524,7 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr,vm_fault_t fault;unsigned long vm_flags;unsigned int mm_flags =FAULT_FLAG_DEFAULT ; -unsigned long addr =untagged_addr (far); +unsigned long addr =untagged_addr (mm, far);if (kprobe_page_fault (regs, esr))return 0 ;
@@ -679,7 +679,7 @@ static int __kprobes do_translation_fault(unsigned long far,unsigned long esr, struct pt_regs *regs) { - unsigned long addr = untagged_addr(far); + unsigned long addr = untagged_addr(current->mm, far);if (is_ttbr0_addr (addr))return do_page_fault (far, esr, regs);
@@ -723,7 +723,7 @@ static int do_sea(unsigned long far, unsigned long esr, struct pt_regs *regs)* UNKNOWN for synchronous external aborts. Mask them out now * so that userspace doesn't see them. */ - siaddr =untagged_addr (far); + siaddr =untagged_addr (current->mm, far); }arm64_notify_die (inf->name, regs, inf->sig, inf->code, siaddr, esr);
@@ -813,7 +813,7 @@ static const struct fault_info fault_info[] = {void do_mem_abort (unsigned long far,unsigned long esr,struct pt_regs *regs) {const struct fault_info *inf =esr_to_fault_info (esr); -unsigned long addr =untagged_addr (far); +unsigned long addr =untagged_addr (current->mm, far);if (!inf->fn (far, esr, regs))return ;
arch/sparc/include/asm/pgtable_64.h +1 -1
@@ -1052,7 +1052,7 @@ static inline unsigned long __untagged_addr(unsigned long start)return start; } -#define untagged_addr (addr) \ +#define untagged_addr (mm, addr) \ ((__typeof__(addr))(__untagged_addr((unsigned long)(addr))))static inline bool pte_access_permitted (pte_t pte,bool write)
arch/sparc/include/asm/uaccess_64.h +2 -0
@@ -8,8 +8,10 @@#include <linux/compiler.h> #include <linux/string.h> +#include <linux/mm_types.h> #include <asm/asi.h> #include <asm/spitfire.h> +#include <asm/pgtable.h> #include <asm/processor.h> #include <asm-generic/access_ok.h>
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1 -1
@@ -1653,7 +1653,7 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(if (flags &KFD_IOC_ALLOC_MEM_FLAGS_USERPTR ) {if (!offset || !*offset)return -EINVAL ; - user_addr =untagged_addr (*offset); + user_addr =untagged_addr (current->mm, *offset); }else if (flags & (KFD_IOC_ALLOC_MEM_FLAGS_DOORBELL |KFD_IOC_ALLOC_MEM_FLAGS_MMIO_REMAP )) { bo_type = ttm_bo_type_sg;
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1 -1
@@ -382,7 +382,7 @@ int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data,uint32_t handle;int r; - args->addr =untagged_addr (args->addr); + args->addr =untagged_addr (current->mm, args->addr);if (offset_in_page (args->addr | args->size))return -EINVAL ;
drivers/gpu/drm/radeon/radeon_gem.c +1 -1
@@ -371,7 +371,7 @@ int radeon_gem_userptr_ioctl(struct drm_device *dev, void *data,uint32_t handle;int r; - args->addr =untagged_addr (args->addr); + args->addr =untagged_addr (current->mm, args->addr);if (offset_in_page (args->addr | args->size))return -EINVAL ;
drivers/infiniband/hw/mlx4/mr.c +1 -1
@@ -379,7 +379,7 @@ static struct ib_umem *mlx4_get_umem_mr(struct ib_device *device, u64 start,* again */ if (!ib_access_writable (access_flags)) { -unsigned long untagged_start =untagged_addr (start); +unsigned long untagged_start =untagged_addr (current->mm, start);struct vm_area_struct *vma;mmap_read_lock (current->mm);
drivers/media/common/videobuf2/frame_vector.c +1 -1
@@ -47,7 +47,7 @@ int get_vaddr_frames(unsigned long start, unsigned int nr_frames,if (WARN_ON_ONCE (nr_frames > vec->nr_allocated)) nr_frames = vec->nr_allocated; - start =untagged_addr (start); + start =untagged_addr (mm, start); ret = pin_user_pages_fast(start, nr_frames, FOLL_FORCE | FOLL_WRITE | FOLL_LONGTERM,
drivers/media/v4l2-core/videobuf-dma-contig.c +1 -1
@@ -157,8 +157,8 @@ static void videobuf_dma_contig_user_put(struct videobuf_dma_contig_memory *mem)static int videobuf_dma_contig_user_get (struct videobuf_dma_contig_memory *mem,struct videobuf_buffer *vb) { -unsigned long untagged_baddr =untagged_addr (vb->baddr);struct mm_struct *mm = current->mm; +unsigned long untagged_baddr =untagged_addr (mm, vb->baddr);struct vm_area_struct *vma;unsigned long prev_pfn, this_pfn;unsigned long pages_done, user_address;
drivers/staging/media/atomisp/pci/hmm/hmm_bo.c +1 -1
@@ -794,7 +794,7 @@ static int alloc_user_pages(struct hmm_buffer_object *bo,* and map to user space */ - userptr =untagged_addr (userptr); + userptr =untagged_addr (current->mm, userptr);if (vma->vm_flags & (VM_IO |VM_PFNMAP )) { page_nr = pin_user_pages((unsigned long)userptr, bo->pgnr,
drivers/tee/tee_shm.c +1 -1
@@ -261,7 +261,7 @@ register_shm_helper(struct tee_context *ctx, unsigned long addr, shm->flags = flags; shm->ctx = ctx; shm->id = id; - addr =untagged_addr (addr); + addr =untagged_addr (current->mm, addr); start =rounddown (addr,PAGE_SIZE ); shm->offset = addr - start; shm->size = length;
drivers/vfio/vfio_iommu_type1.c +1 -1
@@ -562,7 +562,7 @@ static int vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr, goto done; } - vaddr =untagged_addr (vaddr); + vaddr =untagged_addr (mm, vaddr); retry: vma =vma_lookup (mm, vaddr);
fs/proc/task_mmu.c +1 -1
@@ -1662,7 +1662,7 @@ static ssize_t pagemap_read(struct file *file, char __user *buf,/* watch out for wraparound */ start_vaddr = end_vaddr;if (svpfn <= (ULONG_MAX >>PAGE_SHIFT )) - start_vaddr =untagged_addr (svpfn <<PAGE_SHIFT ); + start_vaddr =untagged_addr (mm, svpfn <<PAGE_SHIFT );/* Ensure the address is inside the task */ if (start_vaddr > mm->task_size)
include/linux/mm.h +0 -11
@@ -95,17 +95,6 @@ extern int mmap_rnd_compat_bits __read_mostly;#include <asm/page.h> #include <asm/processor.h> -/* -* Architectures that support memory tagging (assigning tags to memory regions, -* embedding these tags into addresses that point to these memory regions, and -* checking that the memory and the pointer tags match on memory accesses) -* redefine this macro to strip tags from pointers. -* It's defined as noop for architectures that don't support memory tagging. -*/ -#ifndef untagged_addr -#define untagged_addr (addr) (addr) -#endif -#ifndef __pa_symbol#define __pa_symbol (x) __pa(RELOC_HIDE((unsigned long)(x), 0))#endif
include/linux/uaccess.h +15 -0
@@ -10,6 +10,21 @@#include <asm/uaccess.h> +/* +* Architectures that support memory tagging (assigning tags to memory regions, +* embedding these tags into addresses that point to these memory regions, and +* checking that the memory and the pointer tags match on memory accesses) +* redefine this macro to strip tags from pointers. +* +* Passing down mm_struct allows to define untagging rules on per-process +* basis. +* +* It's defined as noop for architectures that don't support memory tagging. +*/ +#ifndef untagged_addr +#define untagged_addr (mm, addr) (addr) +#endif +/* * Architectures should provide two primitives (raw_copy_{to,from}_user()) * and get rid of their private instances of copy_{to,from}_user() and
lib/strncpy_from_user.c +1 -1
@@ -121,7 +121,7 @@ long strncpy_from_user(char *dst, const char __user *src, long count)return 0 ; max_addr =TASK_SIZE_MAX ; - src_addr = (unsigned long )untagged_addr (src); + src_addr = (unsigned long )untagged_addr (current->mm, src);if (likely (src_addr < max_addr)) {unsigned long max = max_addr - src_addr;long retval;
lib/strnlen_user.c +1 -1
@@ -97,7 +97,7 @@ long strnlen_user(const char __user *str, long count)return 0 ; max_addr =TASK_SIZE_MAX ; - src_addr = (unsigned long )untagged_addr (str); + src_addr = (unsigned long )untagged_addr (current->mm, str);if (likely (src_addr < max_addr)) {unsigned long max = max_addr - src_addr;long retval;
mm/gup.c +3 -3
@@ -1125,7 +1125,7 @@ static long __get_user_pages(struct mm_struct *mm,if (!nr_pages)return 0 ; - start =untagged_addr (start); + start =untagged_addr (mm, start);VM_BUG_ON (!!pages != !!(gup_flags & (FOLL_GET |FOLL_PIN )));
@@ -1307,7 +1307,7 @@ int fixup_user_fault(struct mm_struct *mm,struct vm_area_struct *vma;vm_fault_t ret; - address =untagged_addr (address); + address =untagged_addr (mm, address);if (unlocked) fault_flags |=FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_KILLABLE ;
@@ -2935,7 +2935,7 @@ static int internal_get_user_pages_fast(unsigned long start,if (!(gup_flags &FOLL_FAST_ONLY ))might_lock_read (¤t->mm->mmap_lock); - start =untagged_addr (start) &PAGE_MASK ; + start =untagged_addr (current->mm, start) &PAGE_MASK ; len = nr_pages <<PAGE_SHIFT ;if (check_add_overflow (start, len, &end))return 0 ;
mm/madvise.c +1 -1
@@ -1373,7 +1373,7 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behsize_t len;struct blk_plug plug; - start =untagged_addr (start); + start =untagged_addr (mm, start);if (!madvise_behavior_valid (behavior))return -EINVAL ;
mm/mempolicy.c +3 -3
@@ -1456,7 +1456,7 @@ static long kernel_mbind(unsigned long start, unsigned long len,int lmode = mode;int err; - start =untagged_addr (start); + start =untagged_addr (current->mm, start); err =sanitize_mpol_flags (&lmode, &mode_flags);if (err)return err;
@@ -1479,7 +1479,7 @@ SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned long, leunsigned long end;int err = -ENOENT ; - start =untagged_addr (start); + start =untagged_addr (mm, start);if (start & ~PAGE_MASK )return -EINVAL ;/*
@@ -1682,7 +1682,7 @@ static int kernel_get_mempolicy(int __user *policy,if (nmask !=NULL && maxnode < nr_node_ids)return -EINVAL ; - addr =untagged_addr (addr); + addr =untagged_addr (current->mm, addr); err =do_get_mempolicy (&pval, &nodes, addr, flags);
mm/migrate.c +1 -1
@@ -1768,7 +1768,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, goto out_flush;if (get_user (node, nodes + i)) goto out_flush; - addr = (unsigned long )untagged_addr (p); + addr = (unsigned long )untagged_addr (mm, p); err = -ENODEV ;if (node <0 || node >=MAX_NUMNODES )
mm/mincore.c +1 -1
@@ -236,7 +236,7 @@ SYSCALL_DEFINE3(mincore, unsigned long, start, size_t, len,unsigned long pages;unsigned char *tmp; - start =untagged_addr (start); + start =untagged_addr (current->mm, start);/* Check the start address: needs to be page-aligned.. */ if (start & ~PAGE_MASK )
mm/mlock.c +2 -2
@@ -571,7 +571,7 @@ static __must_check int do_mlock(unsigned long start, size_t len, vm_flags_t flaunsigned long lock_limit;int error = -ENOMEM ; - start =untagged_addr (start); + start =untagged_addr (current->mm, start);if (!can_do_mlock ())return -EPERM ;
@@ -634,7 +634,7 @@ SYSCALL_DEFINE2(munlock, unsigned long, start, size_t, len) {int ret; - start =untagged_addr (start); + start =untagged_addr (current->mm, start); len =PAGE_ALIGN (len + (offset_in_page (start))); start &=PAGE_MASK ;
mm/mmap.c +1 -1
@@ -2877,7 +2877,7 @@ EXPORT_SYMBOL(vm_munmap); SYSCALL_DEFINE2(munmap, unsigned long, addr, size_t, len) { - addr =untagged_addr (addr); + addr =untagged_addr (current->mm, addr);return __vm_munmap (addr, len, true); }
mm/mprotect.c +1 -1
@@ -669,7 +669,7 @@ static int do_mprotect_pkey(unsigned long start, size_t len, (prot &PROT_READ );struct mmu_gather tlb; - start =untagged_addr (start); + start =untagged_addr (current->mm, start); prot &= ~(PROT_GROWSDOWN |PROT_GROWSUP );if (grows == (PROT_GROWSDOWN |PROT_GROWSUP ))/* can't be both */
mm/mremap.c +1 -1
@@ -906,7 +906,7 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len,* * See Documentation/arm64/tagged-address-abi.rst for more information. */ - addr =untagged_addr (addr); + addr =untagged_addr (mm, addr);if (flags & ~(MREMAP_FIXED |MREMAP_MAYMOVE |MREMAP_DONTUNMAP ))return ret;
mm/msync.c +1 -1
@@ -37,7 +37,7 @@ SYSCALL_DEFINE3(msync, unsigned long, start, size_t, len, int, flags)int unmapped_error =0 ;int error = -EINVAL ; - start =untagged_addr (start); + start =untagged_addr (mm, start);if (flags & ~(MS_ASYNC |MS_INVALIDATE |MS_SYNC )) goto out;
virt/kvm/kvm_main.c +1 -1
@@ -1943,7 +1943,7 @@ int __kvm_set_memory_region(struct kvm *kvm,return -EINVAL ;/* We can read the guest memory with __xxx_user() later on. */ if ((mem->userspace_addr & (PAGE_SIZE -1 )) || - (mem->userspace_addr !=untagged_addr (mem->userspace_addr)) || + (mem->userspace_addr !=untagged_addr (kvm->mm, mem->userspace_addr)) || !access_ok ((void __user *)(unsigned long )mem->userspace_addr, mem->memory_size))return -EINVAL ;
Signature · 1 line
2.35.1
PATCH v6 4/11 x86/mm: Handle LAM on context switch
In reply to Kirill A. ShutemovLinear Address Masking mode for userspace pointers encoded in CR3 bits. The mode is selected per-thread. Add new thread features indicate that the thread has Linear Address Masking enabled.
switch_mm_irqs_off() now respects these flags and constructs CR3 accordingly.
The active LAM mode gets recorded in the tlb_state.
--- arch/x86/include/asm/mmu.h | 3 +++ arch/x86/include/asm/mmu_context.h | 24 +++++++++++++++++ arch/x86/include/asm/tlbflush.h | 35 ++++++++++++++++++++++++ arch/x86/mm/tlb.c | 43 +++++++++++++++++++----------- 4 files changed, 90 insertions(+), 15 deletions(-)
arch/x86/include/asm/mmu.h +3 -0
@@ -40,6 +40,9 @@ typedef struct {#ifdef CONFIG_X86_64 unsigned short flags; + +/* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ +unsigned long lam_cr3_mask;#endif struct mutex lock;
arch/x86/include/asm/mmu_context.h +24 -0
@@ -91,6 +91,29 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next) }#endif +#ifdef CONFIG_X86_64 +static inline unsigned long mm_lam_cr3_mask (struct mm_struct *mm) +{ +return mm->context.lam_cr3_mask; +} + +static inline void dup_lam (struct mm_struct *oldmm,struct mm_struct *mm) +{ + mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; +} + +#else + +static inline unsigned long mm_lam_cr3_mask (struct mm_struct *mm) +{ +return 0 ; +} + +static inline void dup_lam (struct mm_struct *oldmm,struct mm_struct *mm) +{ +} +#endif +#define enter_lazy_tlb enter_lazy_tlbextern void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk);
@@ -168,6 +191,7 @@ static inline int arch_dup_mmap(struct mm_struct *oldmm, struct mm_struct *mm) {arch_dup_pkeys (oldmm, mm);paravirt_arch_dup_mmap (oldmm, mm); +dup_lam (oldmm, mm);return ldt_dup_context (oldmm, mm); }
arch/x86/include/asm/tlbflush.h +35 -0
@@ -101,6 +101,16 @@ struct tlb_state {*/ bool invalidate_other; +#ifdef CONFIG_X86_64 +/* +* Active LAM mode. +* +* X86_CR3_LAM_U57/U48 shifted right by X86_CR3_LAM_U57_BIT or 0 if LAM +* disabled. +*/ +u8 lam; +#endif +/* * Mask that contains TLB_NR_DYN_ASIDS+1 bits to indicate * the corresponding user PCID needs a flush next time we
@@ -357,6 +367,30 @@ static inline bool huge_pmd_needs_flush(pmd_t oldpmd, pmd_t newpmd) }#define huge_pmd_needs_flush huge_pmd_needs_flush +#ifdef CONFIG_X86_64 +static inline unsigned long tlbstate_lam_cr3_mask (void ) +{ +unsigned long lam =this_cpu_read (cpu_tlbstate.lam); + +return lam <<X86_CR3_LAM_U57_BIT ; +} + +static inline void set_tlbstate_cr3_lam_mask (unsigned long mask) +{ +this_cpu_write (cpu_tlbstate.lam, mask >>X86_CR3_LAM_U57_BIT ); +} + +#else + +static inline unsigned long tlbstate_lam_cr3_mask (void ) +{ +return 0 ; +} + +static inline void set_tlbstate_cr3_lam_mask (u64 mask) +{ +} +#endif #endif /* !MODULE */ static inline void __native_tlb_flush_global (unsigned long cr4)
@@ -364,4 +398,5 @@ static inline void __native_tlb_flush_global(unsigned long cr4)native_write_cr4 (cr4 ^X86_CR4_PGE ); native_write_cr4(cr4); } +#endif /* _ASM_X86_TLBFLUSH_H */
arch/x86/mm/tlb.c +28 -15
129 lines not on this page · raw message
Signature · 1 line
2.35.1
Peter Zijlstra
maintainerRe: [PATCHv6 04/11] x86/mm: Handle LAM on context switch
arch/x86/mm/tlb.c void flush_tlb_kernel_range(unsigned long start, unsigned long end) · line 1088
Quoted 10 lines · Kirill A. Shutemov
arch/x86/mm/tlb.c +4
-2@@ -1071,8 +1082,10 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end)*/ unsigned long __get_current_cr3_fast (void ) { -unsigned long cr3 =build_cr3 (this_cpu_read (cpu_tlbstate.loaded_mm)->pgd, -this_cpu_read (cpu_tlbstate.loaded_mm_asid)); +unsigned long cr3 = +build_cr3 (this_cpu_read (cpu_tlbstate.loaded_mm)->pgd, +this_cpu_read (cpu_tlbstate.loaded_mm_asid), +tlbstate_lam_cr3_mask ());
My editor auto indents this as:
unsigned long cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, this_cpu_read(cpu_tlbstate.loaded_mm_asid), tlbstate_lam_cr3_mask());
which (to me) is *much* clearer.
Peter Zijlstra
maintainerRe: [PATCHv6 04/11] x86/mm: Handle LAM on context switch
In reply to Kirill A. ShutemovQuoted 28 lines · Kirill A. Shutemov
arch/x86/mm/tlb.c +5
-4@@ -154,17 +154,18 @@ static inline u16 user_pcid(u16 asid)return ret; } -static inline unsigned long build_cr3 (pgd_t *pgd,u16 asid) +static inline unsigned long build_cr3 (pgd_t *pgd,u16 asid,unsigned long lam) {if (static_cpu_has (X86_FEATURE_PCID )) { -return __sme_pa (pgd) |kern_pcid (asid); +return __sme_pa (pgd) |kern_pcid (asid) | lam; }else {VM_WARN_ON_ONCE (asid !=0 ); -return __sme_pa (pgd); +return __sme_pa (pgd) | lam; } } -static inline unsigned long build_cr3_noflush (pgd_t *pgd,u16 asid) +static inline unsigned long build_cr3_noflush (pgd_t *pgd,u16 asid, +unsigned long lam) {VM_WARN_ON_ONCE (asid >MAX_ASID_AVAILABLE );/*
arch/x86/mm/tlb.c static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid) · line 178
Quoted 6 lines
arch/x86/mm/tlb.c +1
-1@@ -173,7 +174,7 @@ static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid)* boot because all CPU's the have same capabilities: */ VM_WARN_ON_ONCE (!boot_cpu_has (X86_FEATURE_PCID )); -return __sme_pa (pgd) |kern_pcid (asid) |CR3_NOFLUSH ; +return __sme_pa (pgd) |kern_pcid (asid) | lam |CR3_NOFLUSH ; }
Looking at this; I wonder if we want something like this:
arch/x86/mm/tlb.c +2 -2
@@ -157,6 +157,7 @@ static inline u16 user_pcid(u16 asid)static inline unsigned long build_cr3 (pgd_t *pgd,u16 asid,unsigned long lam) {if (static_cpu_has (X86_FEATURE_PCID )) { +VM_WARN_ON_ONCE (asid >MAX_ASID_AVAILABLE );return __sme_pa (pgd) |kern_pcid (asid) | lam; }else {VM_WARN_ON_ONCE (asid !=0 );
@@ -167,14 +168,13 @@ static inline unsigned long build_cr3(pgstatic inline unsigned long build_cr3_noflush (pgd_t *pgd,u16 asid,unsigned long lam) { -VM_WARN_ON_ONCE (asid >MAX_ASID_AVAILABLE );/* * Use boot_cpu_has() instead of this_cpu_has() as this function * might be called during early boot. This should work even after * boot because all CPU's the have same capabilities: */ VM_WARN_ON_ONCE (!boot_cpu_has (X86_FEATURE_PCID )); -return __sme_pa (pgd) |kern_pcid (asid) | lam |CR3_NOFLUSH ; +return build_cr3 (pgd, asid, lam) |CR3_NOFLUSH ; }/*
Re: [PATCHv6 04/11] x86/mm: Handle LAM on context switch
Quoted 52 lines · Peter Zijlstra
Quoted 28 lines · Kirill A. Shutemov
arch/x86/mm/tlb.c +5
-4@@ -154,17 +154,18 @@ static inline u16 user_pcid(u16 asid)return ret; } -static inline unsigned long build_cr3 (pgd_t *pgd,u16 asid) +static inline unsigned long build_cr3 (pgd_t *pgd,u16 asid,unsigned long lam) {if (static_cpu_has (X86_FEATURE_PCID )) { -return __sme_pa (pgd) |kern_pcid (asid); +return __sme_pa (pgd) |kern_pcid (asid) | lam; }else {VM_WARN_ON_ONCE (asid !=0 ); -return __sme_pa (pgd); +return __sme_pa (pgd) | lam; } } -static inline unsigned long build_cr3_noflush (pgd_t *pgd,u16 asid) +static inline unsigned long build_cr3_noflush (pgd_t *pgd,u16 asid, +unsigned long lam) {VM_WARN_ON_ONCE (asid >MAX_ASID_AVAILABLE );/* arch/x86/mm/tlb.c static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid) · line 178
Quoted 6 lines
arch/x86/mm/tlb.c +1
-1@@ -173,7 +174,7 @@ static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid)* boot because all CPU's the have same capabilities: */ VM_WARN_ON_ONCE (!boot_cpu_has (X86_FEATURE_PCID )); -return __sme_pa (pgd) |kern_pcid (asid) |CR3_NOFLUSH ; +return __sme_pa (pgd) |kern_pcid (asid) | lam |CR3_NOFLUSH ; }Looking at this; I wonder if we want something like this:
arch/x86/mm/tlb.c +1
-0@@ -157,6 +157,7 @@ static inline u16 user_pcid(u16 asid)static inline unsigned long build_cr3 (pgd_t *pgd,u16 asid,unsigned long lam) {if (static_cpu_has (X86_FEATURE_PCID )) { +VM_WARN_ON_ONCE (asid >MAX_ASID_AVAILABLE );return __sme_pa (pgd) |kern_pcid (asid) | lam; }else {VM_WARN_ON_ONCE (asid !=0 );@@ -167,14 +168,13 @@ static inline unsigned long build_cr3(pgstatic inline unsigned long build_cr3_noflush (pgd_t *pgd,u16 asid,unsigned long lam) {
arch/x86/mm/tlb.c static inline unsigned long build_cr3(pg · line 178
Quoted 10 lines
arch/x86/mm/tlb.c +1
-2-VM_WARN_ON_ONCE (asid >MAX_ASID_AVAILABLE );/* * Use boot_cpu_has() instead of this_cpu_has() as this function * might be called during early boot. This should work even after * boot because all CPU's the have same capabilities: */ VM_WARN_ON_ONCE (!boot_cpu_has (X86_FEATURE_PCID )); -return __sme_pa (pgd) |kern_pcid (asid) | lam |CR3_NOFLUSH ; +return build_cr3 (pgd, asid, lam) |CR3_NOFLUSH ; }
Looks sane, but seems unrelated to the patch. Is it okay to fold it anyway?
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Peter Zijlstra
maintainerRe: [PATCHv6 04/11] x86/mm: Handle LAM on context switch
Quoted 58 lines · Kirill A. Shutemov
Quoted 57 lines · Peter Zijlstra
Quoted 28 lines · Kirill A. Shutemov
arch/x86/mm/tlb.c +5
-4@@ -154,17 +154,18 @@ static inline u16 user_pcid(u16 asid)return ret; } -static inline unsigned long build_cr3 (pgd_t *pgd,u16 asid) +static inline unsigned long build_cr3 (pgd_t *pgd,u16 asid,unsigned long lam) {if (static_cpu_has (X86_FEATURE_PCID )) { -return __sme_pa (pgd) |kern_pcid (asid); +return __sme_pa (pgd) |kern_pcid (asid) | lam; }else {VM_WARN_ON_ONCE (asid !=0 ); -return __sme_pa (pgd); +return __sme_pa (pgd) | lam; } } -static inline unsigned long build_cr3_noflush (pgd_t *pgd,u16 asid) +static inline unsigned long build_cr3_noflush (pgd_t *pgd,u16 asid, +unsigned long lam) {VM_WARN_ON_ONCE (asid >MAX_ASID_AVAILABLE );/* arch/x86/mm/tlb.c static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid) · line 178
Quoted 6 lines
arch/x86/mm/tlb.c +1
-1@@ -173,7 +174,7 @@ static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid)* boot because all CPU's the have same capabilities: */ VM_WARN_ON_ONCE (!boot_cpu_has (X86_FEATURE_PCID )); -return __sme_pa (pgd) |kern_pcid (asid) |CR3_NOFLUSH ; +return __sme_pa (pgd) |kern_pcid (asid) | lam |CR3_NOFLUSH ; }Looking at this; I wonder if we want something like this:
arch/x86/mm/tlb.c +1
-1@@ -157,6 +157,7 @@ static inline u16 user_pcid(u16 asid)static inline unsigned long build_cr3 (pgd_t *pgd,u16 asid,unsigned long lam) {if (static_cpu_has (X86_FEATURE_PCID )) { +VM_WARN_ON_ONCE (asid >MAX_ASID_AVAILABLE );return __sme_pa (pgd) |kern_pcid (asid) | lam; }else {VM_WARN_ON_ONCE (asid !=0 );@@ -167,14 +168,13 @@ static inline unsigned long build_cr3(pgstatic inline unsigned long build_cr3_noflush (pgd_t *pgd,u16 asid,unsigned long lam) { -VM_WARN_ON_ONCE (asid >MAX_ASID_AVAILABLE );/* * Use boot_cpu_has() instead of this_cpu_has() as this function * might be called during early boot. This should work even after * boot because all CPU's the have same capabilities:
Quoted 8 lines
arch/x86/mm/tlb.c static inline unsigned long build_cr3(pg · line 178
Quoted 5 lines
arch/x86/mm/tlb.c +1
-1*/ VM_WARN_ON_ONCE (!boot_cpu_has (X86_FEATURE_PCID )); -return __sme_pa (pgd) |kern_pcid (asid) | lam |CR3_NOFLUSH ; +return build_cr3 (pgd, asid, lam) |CR3_NOFLUSH ; }Looks sane, but seems unrelated to the patch. Is it okay to fold it anyway?
Related in so far as that it reduces the number of sites where we have the actual CR3 'computation' (which is how I arrived at the thing).
Arguably we could even do something like:
static inline unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam)
{
unsigned long cr3 = __sme_pa(pgd) | lam;
if (static_cpu_has(X86_FEATURE_PCID)) {
VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE);
cr |= kern_pcid(asid);
} else {
VM_WARN_ON_ONCE(asid != 0);
}
return cr3;
}But perhaps that's pushing things a little.
IMO fine to fold.
PATCH v6 4/11 x86/mm: Handle LAM on context switch
In reply to Kirill A. ShutemovLinear Address Masking mode for userspace pointers encoded in CR3 bits. The mode is selected per-process and stored in mm_context_t.
switch_mm_irqs_off() now respects selected LAM mode and constructs CR3 accordingly.
The active LAM mode gets recorded in the tlb_state.
---
v6.1
- Drop redundant smb_mb() in prctl_enable_tagged_addr();
- Cleanup code around build_cr3();
- Fix commit message;
--- arch/x86/include/asm/mmu.h | 3 ++ arch/x86/include/asm/mmu_context.h | 24 +++++++++++++++ arch/x86/include/asm/tlbflush.h | 35 ++++++++++++++++++++++ arch/x86/mm/tlb.c | 48 ++++++++++++++++++++---------- 4 files changed, 94 insertions(+), 16 deletions(-)
arch/x86/include/asm/mmu.h +3 -0
@@ -40,6 +40,9 @@ typedef struct {#ifdef CONFIG_X86_64 unsigned short flags; + +/* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ +unsigned long lam_cr3_mask;#endif struct mutex lock;
arch/x86/include/asm/mmu_context.h +24 -0
@@ -91,6 +91,29 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next) }#endif +#ifdef CONFIG_X86_64 +static inline unsigned long mm_lam_cr3_mask (struct mm_struct *mm) +{ +return mm->context.lam_cr3_mask; +} + +static inline void dup_lam (struct mm_struct *oldmm,struct mm_struct *mm) +{ + mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; +} + +#else + +static inline unsigned long mm_lam_cr3_mask (struct mm_struct *mm) +{ +return 0 ; +} + +static inline void dup_lam (struct mm_struct *oldmm,struct mm_struct *mm) +{ +} +#endif +#define enter_lazy_tlb enter_lazy_tlbextern void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk);
@@ -168,6 +191,7 @@ static inline int arch_dup_mmap(struct mm_struct *oldmm, struct mm_struct *mm) {arch_dup_pkeys (oldmm, mm);paravirt_arch_dup_mmap (oldmm, mm); +dup_lam (oldmm, mm);return ldt_dup_context (oldmm, mm); }
arch/x86/include/asm/tlbflush.h +35 -0
@@ -101,6 +101,16 @@ struct tlb_state {*/ bool invalidate_other; +#ifdef CONFIG_X86_64 +/* +* Active LAM mode. +* +* X86_CR3_LAM_U57/U48 shifted right by X86_CR3_LAM_U57_BIT or 0 if LAM +* disabled. +*/ +u8 lam; +#endif +/* * Mask that contains TLB_NR_DYN_ASIDS+1 bits to indicate * the corresponding user PCID needs a flush next time we
@@ -357,6 +367,30 @@ static inline bool huge_pmd_needs_flush(pmd_t oldpmd, pmd_t newpmd) }#define huge_pmd_needs_flush huge_pmd_needs_flush +#ifdef CONFIG_X86_64 +static inline unsigned long tlbstate_lam_cr3_mask (void ) +{ +unsigned long lam =this_cpu_read (cpu_tlbstate.lam); + +return lam <<X86_CR3_LAM_U57_BIT ; +} + +static inline void set_tlbstate_cr3_lam_mask (unsigned long mask) +{ +this_cpu_write (cpu_tlbstate.lam, mask >>X86_CR3_LAM_U57_BIT ); +} + +#else + +static inline unsigned long tlbstate_lam_cr3_mask (void ) +{ +return 0 ; +} + +static inline void set_tlbstate_cr3_lam_mask (u64 mask) +{ +} +#endif #endif /* !MODULE */ static inline void __native_tlb_flush_global (unsigned long cr4)
@@ -364,4 +398,5 @@ static inline void __native_tlb_flush_global(unsigned long cr4)native_write_cr4 (cr4 ^X86_CR4_PGE ); native_write_cr4(cr4); } +#endif /* _ASM_X86_TLBFLUSH_H */
arch/x86/mm/tlb.c +32 -16
135 lines not on this page · raw message
Signature · 1 line
2.35.1
PATCH v6 5/11 x86/uaccess: Provide untagged_addr() and remove tags before address check
In reply to Kirill A. Shutemovuntagged_addr() is a helper used by the core-mm to strip tag bits and get the address to the canonical shape. In only handles userspace addresses. The untagging mask is stored in mmu_context and will be set on enabling LAM for the process.
The tags must not be included into check whether it's okay to access the userspace address.
Strip tags in access_ok().
get_user() and put_user() don't use access_ok(), but check access against TASK_SIZE directly in assembly. Strip tags, before calling into the assembly helper.
--- arch/x86/include/asm/mmu.h | 3 +++ arch/x86/include/asm/mmu_context.h | 11 ++++++++ arch/x86/include/asm/uaccess.h | 42 +++++++++++++++++++++++++++--- arch/x86/kernel/process.c | 3 +++ 4 files changed, 56 insertions(+), 3 deletions(-)
arch/x86/include/asm/mmu.h +3 -0
@@ -43,6 +43,9 @@ typedef struct {/* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ unsigned long lam_cr3_mask; + +/* Significant bits of the virtual address. Excludes tag bits. */ +u64 untag_mask;#endif struct mutex lock;
arch/x86/include/asm/mmu_context.h +11 -0
@@ -100,6 +100,12 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm)static inline void dup_lam (struct mm_struct *oldmm,struct mm_struct *mm) { mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; + mm->context.untag_mask = oldmm->context.untag_mask; +} + +static inline void mm_reset_untag_mask (struct mm_struct *mm) +{ + mm->context.untag_mask =-1UL ; }#else
@@ -112,6 +118,10 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm)static inline void dup_lam (struct mm_struct *oldmm,struct mm_struct *mm) { } + +static inline void mm_reset_untag_mask (struct mm_struct *mm) +{ +}#endif #define enter_lazy_tlb enter_lazy_tlb
@@ -138,6 +148,7 @@ static inline int init_new_context(struct task_struct *tsk, mm->context.execute_only_pkey = -1; }#endif +mm_reset_untag_mask (mm);init_new_context_ldt (mm);return 0 ; }
arch/x86/include/asm/uaccess.h +39 -3
72 lines not on this page · raw message
arch/x86/kernel/process.c +3 -0
@@ -47,6 +47,7 @@#include <asm/frame.h> #include <asm/unwind.h> #include <asm/tdx.h> +#include <asm/mmu_context.h> #include "process.h"
@@ -367,6 +368,8 @@ void arch_setup_new_exec(void)task_clear_spec_ssb_noexec (current);speculation_ctrl_update (read_thread_flags ()); } + +mm_reset_untag_mask (current->mm); }#ifdef CONFIG_X86_IOPL_IOPERM
Signature · 1 line
2.35.1
PATCH v6 6/11 x86/mm: Provide arch_prctl() interface for LAM
In reply to Kirill A. ShutemovAdd a couple of arch_prctl() handles:
- ARCH_ENABLE_TAGGED_ADDR enabled LAM. The argument is required number of tag bits. It is rounded up to the nearest LAM mode that can provide it. For now only LAM_U57 is supported, with 6 tag bits.
- ARCH_GET_UNTAG_MASK returns untag mask. It can indicates where tag bits located in the address.
- ARCH_GET_MAX_TAG_BITS returns the maximum tag bits user can request. Zero if LAM is not supported.
--- arch/x86/include/uapi/asm/prctl.h | 4 ++ arch/x86/kernel/process_64.c | 65 ++++++++++++++++++++++++++++++- 2 files changed, 68 insertions(+), 1 deletion(-)
arch/x86/include/uapi/asm/prctl.h +4 -0
@@ -20,4 +20,8 @@#define ARCH_MAP_VDSO_32 0x2002#define ARCH_MAP_VDSO_64 0x2003 +#define ARCH_GET_UNTAG_MASK 0x4001 +#define ARCH_ENABLE_TAGGED_ADDR 0x4002 +#define ARCH_GET_MAX_TAG_BITS 0x4003 +#endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/process_64.c +64 -1
77 lines not on this page · raw message
Signature · 1 line
2.35.1
Peter Zijlstra
maintainerRe: [PATCHv6 06/11] x86/mm: Provide arch_prctl() interface for LAM
Quoted 37 lines · Kirill A. Shutemov
arch/x86/kernel/process_64.c +37
-0+static void enable_lam_func (void *mm) +{ +struct mm_struct *loaded_mm =this_cpu_read (cpu_tlbstate.loaded_mm); +unsigned long lam_mask; +unsigned long cr3; + +if (loaded_mm != mm) +return ; + + lam_mask =READ_ONCE (loaded_mm->context.lam_cr3_mask); + +/* Update CR3 to get LAM active on the CPU */ + cr3 =__read_cr3 (); + cr3 &= ~(X86_CR3_LAM_U48 |X86_CR3_LAM_U57 ); + cr3 |= lam_mask; +write_cr3 (cr3); +set_tlbstate_cr3_lam_mask (lam_mask); +} + +static int prctl_enable_tagged_addr (struct mm_struct *mm,unsigned long nr_bits) +{ +int ret =0 ; + +if (!cpu_feature_enabled (X86_FEATURE_LAM )) +return -ENODEV ; + +mutex_lock (&mm->context.lock); + +/* Already enabled? */ +if (mm->context.lam_cr3_mask) { + ret = -EBUSY ; + goto out; + } + +if (!nr_bits) { + ret = -EINVAL ; + goto out;
arch/x86/kernel/process_64.c static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) · line 791
Quoted 10 lines
arch/x86/kernel/process_64.c +10
-0+ }else if (nr_bits <=6 ) { + mm->context.lam_cr3_mask =X86_CR3_LAM_U57 ; + mm->context.untag_mask = ~GENMASK (62 ,57 ); + }else { + ret = -EINVAL ; + goto out; + } + +/* Make lam_cr3_mask and untag_mask visible on other CPUs */ +smp_mb ();
smp_mb() doesn't make visible -- it is about ordering, what does it order against that the below on_each_cpu_mask() doesn't already take care of?
Quoted 6 lines
arch/x86/kernel/process_64.c +6
-0+ +on_each_cpu_mask (mm_cpumask (mm), enable_lam_func, mm, true); +out: +mutex_unlock (&mm->context.lock); +return ret; +}
Re: [PATCHv6 06/11] x86/mm: Provide arch_prctl() interface for LAM
Quoted 45 lines · Peter Zijlstra
Quoted 43 lines · Kirill A. Shutemov
arch/x86/kernel/process_64.c +43
-0+static void enable_lam_func (void *mm) +{ +struct mm_struct *loaded_mm =this_cpu_read (cpu_tlbstate.loaded_mm); +unsigned long lam_mask; +unsigned long cr3; + +if (loaded_mm != mm) +return ; + + lam_mask =READ_ONCE (loaded_mm->context.lam_cr3_mask); + +/* Update CR3 to get LAM active on the CPU */ + cr3 =__read_cr3 (); + cr3 &= ~(X86_CR3_LAM_U48 |X86_CR3_LAM_U57 ); + cr3 |= lam_mask; +write_cr3 (cr3); +set_tlbstate_cr3_lam_mask (lam_mask); +} + +static int prctl_enable_tagged_addr (struct mm_struct *mm,unsigned long nr_bits) +{ +int ret =0 ; + +if (!cpu_feature_enabled (X86_FEATURE_LAM )) +return -ENODEV ; + +mutex_lock (&mm->context.lock); + +/* Already enabled? */ +if (mm->context.lam_cr3_mask) { + ret = -EBUSY ; + goto out; + } + +if (!nr_bits) { + ret = -EINVAL ; + goto out; + }else if (nr_bits <=6 ) { + mm->context.lam_cr3_mask =X86_CR3_LAM_U57 ; + mm->context.untag_mask = ~GENMASK (62 ,57 ); + }else { + ret = -EINVAL ; + goto out;
Quoted 8 lines
arch/x86/kernel/process_64.c static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) · line 791
Quoted 4 lines
arch/x86/kernel/process_64.c +4
-0+ } + +/* Make lam_cr3_mask and untag_mask visible on other CPUs */ +smp_mb ();
smp_mb()doesn't make visible -- it is about ordering, what does it order against that the belowon_each_cpu_mask()doesn't already take care of?
You are right it is redundant. I will drop it.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
PATCH v6 6/11 x86/mm: Provide arch_prctl() interface for LAM
In reply to Kirill A. ShutemovAdd a couple of arch_prctl() handles:
- ARCH_ENABLE_TAGGED_ADDR enabled LAM. The argument is required number of tag bits. It is rounded up to the nearest LAM mode that can provide it. For now only LAM_U57 is supported, with 6 tag bits.
- ARCH_GET_UNTAG_MASK returns untag mask. It can indicates where tag bits located in the address.
- ARCH_GET_MAX_TAG_BITS returns the maximum tag bits user can request. Zero if LAM is not supported.
---
v6.1:
- Drop redundant smb_mb() in prctl_enable_tagged_addr();
--- arch/x86/include/uapi/asm/prctl.h | 4 ++ arch/x86/kernel/process_64.c | 62 ++++++++++++++++++++++++++++++- 2 files changed, 65 insertions(+), 1 deletion(-)
arch/x86/include/uapi/asm/prctl.h +4 -0
@@ -20,4 +20,8 @@#define ARCH_MAP_VDSO_32 0x2002#define ARCH_MAP_VDSO_64 0x2003 +#define ARCH_GET_UNTAG_MASK 0x4001 +#define ARCH_ENABLE_TAGGED_ADDR 0x4002 +#define ARCH_GET_MAX_TAG_BITS 0x4003 +#endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/process_64.c +61 -1
74 lines not on this page · raw message
Signature · 1 line
2.35.1
Re: [PATCHv6 06/11] x86/mm: Provide arch_prctl() interface for LAM
In reply to Kirill A. ShutemovQuoted 7 lines · Kirill A. Shutemov
Add a couple of
arch_prctl()handles:-
ARCH_ENABLE_TAGGED_ADDRenabled LAM. The argument is required number of tag bits. It is rounded up to the nearest LAM mode that can provide it. For now onlyLAM_U57is supported, with 6 tag bits.
Quoted 7 lines
-
ARCH_GET_UNTAG_MASKreturns untag mask. It can indicates where tag bits located in the address.-
ARCH_GET_MAX_TAG_BITSreturns the maximum tag bits user can request. Zero if LAM is not supported.Signed-off-by Kirill A. Shutemov <[email protected]>
(with a nit, see below)
Quoted 9 lines
arch/x86/kernel/process_64.c +9
-0+static int prctl_enable_tagged_addr (struct mm_struct *mm,unsigned long nr_bits) +{ +int ret =0 ; + +if (!cpu_feature_enabled (X86_FEATURE_LAM )) +return -ENODEV ; + +mutex_lock (&mm->context.lock); +
arch/x86/kernel/process_64.c static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) · line 782
Quoted 10 lines
arch/x86/kernel/process_64.c +10
-0+/* Already enabled? */ +if (mm->context.lam_cr3_mask) { + ret = -EBUSY ; + goto out; + } + +if (!nr_bits) { + ret = -EINVAL ; + goto out; + }else if (nr_bits <=6 ) {
Can you please make this 6 a #define?
Quoted 1 line
+ return
put_user(6, (unsigned long __user *)arg2);
... and use it at least here (could also express masks in terms of this number, but maybe it's enough to just declare them in the same header next to each other).
Alexander Potapenko
Tested-byRe: [PATCHv6 06/11] x86/mm: Provide arch_prctl() interface for LAM
Quoted 10 lines · Alexander Potapenko
Quoted 7 lines · Kirill A. Shutemov
Add a couple of
arch_prctl()handles:-
ARCH_ENABLE_TAGGED_ADDRenabled LAM. The argument is required number of tag bits. It is rounded up to the nearest LAM mode that can provide it. For now onlyLAM_U57is supported, with 6 tag bits.
Quoted 8 lines
Quoted 7 lines
-
ARCH_GET_UNTAG_MASKreturns untag mask. It can indicates where tag bits located in the address.-
ARCH_GET_MAX_TAG_BITSreturns the maximum tag bits user can request. Zero if LAM is not supported.Signed-off-by Kirill A. Shutemov <[email protected]>Reviewed-by Alexander Potapenko <[email protected]>
And also:
PATCH v6 7/11 x86: Expose untagging mask in /proc/$PID/arch_status
In reply to Kirill A. ShutemovAdd a line in /proc/$PID/arch_status to report untag_mask. It can be used to find out LAM status of the process from the outside. It is useful for debuggers.
--- arch/x86/include/asm/mmu_context.h | 10 +++++ arch/x86/kernel/Makefile | 2 + arch/x86/kernel/fpu/xstate.c | 47 ----------------------- arch/x86/kernel/proc.c | 60 ++++++++++++++++++++++++++++++ 4 files changed, 72 insertions(+), 47 deletions(-) create mode 100644 arch/x86/kernel/proc.c
arch/x86/include/asm/mmu_context.h +10 -0
@@ -103,6 +103,11 @@ static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) mm->context.untag_mask = oldmm->context.untag_mask; } +static inline unsigned long mm_untag_mask (struct mm_struct *mm) +{ +return mm->context.untag_mask; +} +static inline void mm_reset_untag_mask (struct mm_struct *mm) { mm->context.untag_mask =-1UL ;
@@ -119,6 +124,11 @@ static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { } +static inline unsigned long mm_untag_mask (struct mm_struct *mm) +{ +return -1UL ; +} +static inline void mm_reset_untag_mask (struct mm_struct *mm) { }
arch/x86/kernel/Makefile +2 -0
@@ -139,6 +139,8 @@ obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += sev.o +obj-$(CONFIG_PROC_FS) += proc.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y)
arch/x86/kernel/fpu/xstate.c +0 -47
56 lines not on this page · raw message
arch/x86/kernel/proc.c new file +60 -0
60 lines not on this page · raw message
Signature · 1 line
2.35.1
PATCH v6 8/11 selftests/x86/lam: Add malloc and tag-bits test cases for linear-address masking
In reply to Kirill A. ShutemovFrom: Weihong Zhang <[email protected]>
LAM is supported only in 64-bit mode and applies only addresses used for data accesses. In 64-bit mode, linear address have 64 bits. LAM is applied to 64-bit linear address and allow software to use high bits for metadata. LAM supports configurations that differ regarding which pointer bits are masked and can be used for metadata.
LAM includes following mode:
- LAM_U57, pointer bits in positions 62:57 are masked (LAM width 6), allows bits 62:57 of a user pointer to be used as metadata.
There are two arch_prctls:ARCH_ENABLE_TAGGED_ADDR: enable LAM mode, mask high bits of a user pointer.ARCH_GET_UNTAG_MASK: get current untagged mask.ARCH_GET_MAX_TAG_BITS: the maximum tag bits user can request. zero if LAM is not supported.
The LAM mode is for pre-process, a process has only one chance to set LAM mode. But there is no API to disable LAM mode. So all of test cases are run under child process.
Functions of this test:
MALLOC
- LAM_U57 masks bits 57:62 of a user pointer. Process on user space can dereference such pointers.
- Disable LAM, dereference a pointer with metadata above 48 bit or 57 bit lead to trigger SIGSEGV.
TAG_BITS
- Max tag bits of LAM_U57 is 6.
--- tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/lam.c | 317 +++++++++++++++++++++++++++ 2 files changed, 318 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/x86/lam.c
tools/testing/selftests/x86/Makefile +1 -1
@@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \ - corrupt_xstate_header amx + corrupt_xstate_header amx lam # Some selftests require 32bit support enabled also on 64bit systems TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall
tools/testing/selftests/x86/lam.c new file +317 -0
317 lines not on this page · raw message
Signature · 1 line
2.35.1
RE: [PATCHv6 08/11] selftests/x86/lam: Add malloc and tag-bits test cases for linear-address masking
Quoted 156 lines
-----Original Message-----
From: Kirill A. Shutemov <[email protected]>
Sent: Monday, August 15, 2022 12:18
To: Dave Hansen <[email protected]>; Lutomirski, Andy <[email protected]>; Peter Zijlstra <[email protected]>
Cc: [email protected]; Kostya Serebryany <[email protected]>; Andrey Ryabinin <[email protected]>; Andrey Konovalov <[email protected]>; Alexander Potapenko <[email protected]>; Taras Madan <[email protected]>; Dmitry Vyukov <[email protected]>; H . J . Lu <[email protected]>; Andi Kleen <[email protected]>; Edgecombe, Rick P <[email protected]>; [email protected]; linux- [email protected]; Zhang, Weihong <[email protected]>; Kirill A . Shutemov <[email protected]>
Subject: [PATCHv6 08/11] selftests/x86/lam: Add malloc and tag-bits test cases for linear-address maskingFrom: Weihong Zhang <[email protected]>
LAM is supported only in 64-bit mode and applies only addresses used for data accesses. In 64-bit mode, linear address have 64 bits. LAM is applied to 64-bit linear address and allow software to use high bits for metadata. LAM supports configurations that differ regarding which pointer bits are masked and can be used for metadata.
LAM includes following mode:
-
LAM_U57, pointer bits in positions 62:57 are masked (LAM width 6), allows bits 62:57 of a user pointer to be used as metadata.There are two
arch_prctls:ARCH_ENABLE_TAGGED_ADDR: enable LAM mode, mask high bits of a user pointer.ARCH_GET_UNTAG_MASK: get current untagged mask.ARCH_GET_MAX_TAG_BITS: the maximum tag bits user can request. zero if LAM is not supported.The LAM mode is for pre-process, a process has only one chance to set LAM mode.
But there is no API to disable LAM mode. So all of test cases are run under child process.Functions of this test:
MALLOC
-
LAM_U57masks bits 57:62 of a user pointer. Process on user space can dereference such pointers.- Disable LAM, dereference a pointer with metadata above 48 bit or 57 bit lead to trigger SIGSEGV.
TAG_BITS- Max tag bits of
LAM_U57is 6.Signed-off-by Weihong Zhang <[email protected]>Signed-off-by Kirill A. Shutemov <[email protected]>--- tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/lam.c | 317 +++++++++++++++++++++++++++ 2 files changed, 318 insertions(+), 1 deletion(-) create mode 100644
tools/testing/selftests/x86/lam.ctools/testing/selftests/x86/Makefile +1
-1@@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \ - corrupt_xstate_header amx + corrupt_xstate_header amx lam # Some selftests require 32bit support enabled also on 64bit systems
TARGETS_C_32BIT_NEEDED:=ldt_gdtptrace_syscalltools/testing/selftests/x86/lam.c new file +72
-0@@ -0,0 +1,317 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <sys/syscall.h> +#include <time.h> +#include <signal.h> +#include <setjmp.h> +#include <sys/mman.h> +#include <sys/wait.h> +#include <inttypes.h> + +#include "../kselftest.h" + +#ifndef __x86_64__ +# error This test is 64-bit only +#endif + +/* LAM modes, these definitions were copied from kernel code */ +#define LAM_NONE 0 +#define LAM_U57_BITS 6 +/* arch prctl for LAM */ +#define ARCH_GET_UNTAG_MASK 0x4001 +#define ARCH_ENABLE_TAGGED_ADDR 0x4002 +#define ARCH_GET_MAX_TAG_BITS 0x4003 + +/* Specified test function bits */ +#define FUNC_MALLOC 0x1 +#define FUNC_BITS 0x2 + +#define TEST_MASK 0x3 + +#define MALLOC_LEN 32 + +struct testcases { +unsigned int later; +int expected;/* 2: SIGSEGV Error; 1: other errors */ +unsigned long lam; +uint64_t addr; +int (*test_func)(struct testcases *test); +const char *msg; +}; + +int tests_cnt; +jmp_buf segv_env; + +static void segv_handler (int sig) +{ +ksft_print_msg ("Get segmentation fault(%d)." , sig); +siglongjmp (segv_env,1 ); +} + +static inline int cpu_has_lam (void ) +{ +unsigned int cpuinfo[4 ]; + +__cpuid_count (0x7 ,1 , cpuinfo[0 ], cpuinfo[1 ], cpuinfo[2 ], cpuinfo[3 ]); + +return (cpuinfo[0 ] & (1 <<26 )); +} + +/* +* Set tagged address and read back untag mask. +* check if the untagged mask is expected. +*/ +static int set_lam (unsigned long lam) +{ +int ret =0 ; +uint64_t ptr =0 ; + +if (lam !=LAM_U57_BITS && lam !=LAM_NONE ) +return -1 ;
tools/testing/selftests/x86/lam.c line 82
Quoted 10 lines
tools/testing/selftests/x86/lam.c new file +10
-0+ +/* Skip check return */ +syscall (SYS_arch_prctl,ARCH_ENABLE_TAGGED_ADDR , lam); + +/* Get untagged mask */ +syscall (SYS_arch_prctl,ARCH_GET_UNTAG_MASK , &ptr); + +/* Check mask returned is expected */ +if (lam ==LAM_U57_BITS ) + ret = (ptr != ~(0x3fULL <<57 ));
[Hu, Robert]
Any special reason not "bool ret"?
Quoted 5 lines
tools/testing/selftests/x86/lam.c new file +5
-0+else if (lam ==LAM_NONE ) + ret = (ptr !=-1ULL ); + +return ret; +}
tools/testing/selftests/x86/lam.c line 98
Quoted 10 lines
tools/testing/selftests/x86/lam.c new file +10
-0+ +static unsigned long get_default_tag_bits (void ) { +pid_t pid; +int lam =LAM_NONE ; +int ret =0 ; + + pid =fork (); +if (pid <0 ) { +perror ("Fork failed." ); + ret =1 ;
[Hu, Robert]
In this fault case, at last you "return lam", which inited as 0, not your intended "ret".
tools/testing/selftests/x86/lam.c line 100
Quoted 2 lines
tools/testing/selftests/x86/lam.c new file +2
-0+ }else if (pid ==0 ) { +/* Set LAM mode in parent process */
[Hu, Robert]
pid == 0 is child process?
Quoted 5 lines
tools/testing/selftests/x86/lam.c new file +5
-0+if (set_lam (LAM_U57_BITS ) ==0 ) + lam =LAM_U57_BITS ; +else + lam =LAM_NONE ; +exit (lam);
tools/testing/selftests/x86/lam.c
Quoted 10 lines
tools/testing/selftests/x86/lam.c new file +10
-0+ }else { +wait (&ret); + lam =WEXITSTATUS (ret); + } + +return lam; +} + +/* According to LAM mode, set metadata in high bits */ static uint64_t +get_metadata (uint64_t src,unsigned long lam) {
[Hu, Robert]
This function looks like not "get metadata", but "embed metadata" to canonical address. Naming looks confusing. Perhaps embed_metadata()?
tools/testing/selftests/x86/lam.c
Quoted 9 lines
tools/testing/selftests/x86/lam.c new file +9
-0+uint64_t metadata; + +srand (time (NULL )); +/* Get a random value as metadata */ + metadata =rand (); + +switch (lam) { +case LAM_U57_BITS :/* Set metadata in bits 62:57 */ + metadata = (src & ~(0x3fULL << 57)) | ((metadata & 0x3f) <<
[Hu, Robert]
Looks like "0x3fULL << 57" is frequently used across LAM selftests, why not name a macro for it?
Quoted 17 lines
57);
tools/testing/selftests/x86/lam.c new file +16
-0+break ; +default : + metadata = src; +break ; + } + +return metadata; +} + +/* +* Set metadata in user pointer, compare new pointer with original pointer. +* both pointers should point to the same address. +*/ +static int handle_lam_test (void *src,unsigned int lam) { +char *ptr; +
tools/testing/selftests/x86/lam.c
Quoted 10 lines
tools/testing/selftests/x86/lam.c new file +10
-0+strcpy ((char *)src,"USER POINTER" ); + + ptr = (char *)get_metadata ((uint64_t )src, lam); +if (src == ptr) +return 0 ; + +/* Copy a string into the pointer with metadata */ +strcpy ((char *)ptr,"METADATA POINTER" ); + +return (!!strcmp ((char *)src, (char *)ptr)); }
[Hu, Robert]
Why "!!" here? now that function return type is int and strcmp() returns int.
Quoted 164 lines
tools/testing/selftests/x86/lam.c new file +93
-0+ + +int handle_max_bits (struct testcases *test) { +unsigned long exp_bits =get_default_tag_bits (); +unsigned long bits =0 ; + +if (exp_bits !=LAM_NONE ) + exp_bits =LAM_U57_BITS ; + +/* Get LAM max tag bits */ +if (syscall (SYS_arch_prctl,ARCH_GET_MAX_TAG_BITS , &bits) ==-1 ) +return 1 ; + +return (exp_bits != bits); +} + +/* +* Test lam feature through dereference pointer get from malloc. +* @return 0: Pass test. 1: Get failure during test 2: Get SIGSEGV */ +static int handle_malloc (struct testcases *test) { +char *ptr =NULL ; +int ret =0 ; + +if (test->later ==0 && test->lam !=0 ) +if (set_lam (test->lam) ==-1 ) +return 1 ; + + ptr = (char *)malloc (MALLOC_LEN ); +if (ptr ==NULL ) { +perror ("malloc() failure\n" ); +return 1 ; + } + +/* Set signal handler */ +if (sigsetjmp (segv_env,1 ) ==0 ) { +signal (SIGSEGV , segv_handler); + ret =handle_lam_test (ptr, test->lam); + }else { + ret =2 ; + } + +if (test->later !=0 && test->lam !=0 ) +if (set_lam (test->lam) ==-1 && ret ==0 ) + ret =1 ; + +free (ptr); + +return ret; +} + +static int fork_test (struct testcases *test) { +int ret, child_ret; +pid_t pid; + + pid =fork (); +if (pid <0 ) { +perror ("Fork failed." ); + ret =1 ; + }else if (pid ==0 ) { + ret = test->test_func (test); +exit (ret); + }else { +wait (&child_ret); + ret =WEXITSTATUS (child_ret); + } + +return ret; +} + +static void run_test (struct testcases *test,int count) { +int i, ret =0 ; + +for (i =0 ; i < count; i++) { +struct testcases *t = test + i; + +/* fork a process to run test case */ + ret =fork_test (t); +if (ret !=0 ) + ret = (t->expected == ret); +else + ret = !(t->expected); + + tests_cnt++; +ksft_test_result (ret, t->msg); + } +} + +static struct testcases malloc_cases[] = { + { + .later =0 , + .lam =LAM_U57_BITS , + .test_func = handle_malloc, + .msg = "MALLOC: LAM_U57. Dereferencing pointer withmetadata\n",
tools/testing/selftests/x86/lam.c new file +7
-0+ }, + { + .later = 1, + .expected = 2, + .lam = LAM_U57_BITS, + .test_func = handle_malloc, + .msg = "MALLOC:[Negtive] Disable LAM. Dereferencing pointerwith metadata.\n",
tools/testing/selftests/x86/lam.c new file +15
-0+ }, +}; + + +static struct testcases bits_cases[] = { + { + .test_func = handle_max_bits, + .msg ="BITS: Check default tag bits\n" , + }, +}; + +static void cmd_help (void ) +{ +printf ("usage: lam [-h] [-t test list]\n" ); + printf("\t-t test list: run tests specified in the test list, default:0x%x\n",
TEST_MASK);tools/testing/selftests/x86/lam.c new file +44
-1+printf ("\t\t0x1:malloc; 0x2:max_bits;\n" ); +printf ("\t-h: help\n" ); +} + +int main (int argc,char **argv) +{ +int c =0 ; +unsigned int tests =TEST_MASK ; + + tests_cnt =0 ; + +if (!cpu_has_lam ()) { +ksft_print_msg ("Unsupported LAM feature!\n" ); +return -1 ; + } + +while ((c =getopt (argc, argv,"ht:" )) !=-1 ) { +switch (c) { +case 't' : + tests =strtoul (optarg,NULL ,16 ); +if (!(tests &TEST_MASK )) { +ksft_print_msg ("Invalid argument!\n" ); +return -1 ; + } +break ; +case 'h' : +cmd_help (); +return 0 ; +default : +ksft_print_msg ("Invalid argument\n" ); +return -1 ; + } + } + +if (tests &FUNC_MALLOC ) +run_test (malloc_cases,ARRAY_SIZE (malloc_cases)); + +if (tests &FUNC_BITS ) +run_test (bits_cases,ARRAY_SIZE (bits_cases)); + +ksft_set_plan (tests_cnt); + +return ksft_exit_pass (); +} --2.35.1
RE: [PATCHv6 08/11] selftests/x86/lam: Add malloc and tag-bits test cases for linear-address masking
Quoted 187 lines
-----Original Message-----
From: Hu, Robert <[email protected]>
Sent: Friday, August 19, 2022 1:17 PM
To: Kirill A. Shutemov <[email protected]>; Dave Hansen <[email protected]>; Lutomirski, Andy <[email protected]>; Peter Zijlstra <[email protected]>
Cc: [email protected]; Kostya Serebryany <[email protected]>; Andrey Ryabinin <[email protected]>; Andrey Konovalov <[email protected]>; Alexander Potapenko <[email protected]>; Taras Madan <[email protected]>; Dmitry Vyukov <[email protected]>; H . J . Lu <[email protected]>; Andi Kleen <[email protected]>; Edgecombe, Rick P <[email protected]>; [email protected]; linux- [email protected]; Zhang, Weihong <[email protected]>
Subject: RE: [PATCHv6 08/11] selftests/x86/lam: Add malloc and tag-bits test cases for linear-address maskingQuoted 8 lines
-----Original Message-----
From: Kirill A. Shutemov <[email protected]>
Sent: Monday, August 15, 2022 12:18
To: Dave Hansen <[email protected]>; Lutomirski, Andy <[email protected]>; Peter Zijlstra <[email protected]>
Cc: [email protected]; Kostya Serebryany <[email protected]>; Andrey Ryabinin <[email protected]>; Andrey Konovalov <[email protected]>; Alexander Potapenko <[email protected]>;Taras
Quoted 1 line
Madan <[email protected]>; Dmitry Vyukov
<[email protected]>; H .
Quoted 3 lines
J . Lu <[email protected]>; Andi Kleen <[email protected]>; Edgecombe, Rick P <[email protected]>; [email protected]; linux- [email protected]; Zhang, Weihong
<[email protected]>; Kirill A .
Quoted 9 lines
Shutemov <[email protected]>
Subject: [PATCHv6 08/11] selftests/x86/lam: Add malloc and tag-bits test cases for linear-address maskingFrom: Weihong Zhang <[email protected]>
LAM is supported only in 64-bit mode and applies only addresses used for data accesses. In 64-bit mode, linear address have 64 bits. LAM is applied to 64-bit linear address and allow software to use high bits for
metadata.
Quoted 10 lines
LAM supports configurations that differ regarding which pointer bits are masked and can be used for metadata.
LAM includes following mode:
-
LAM_U57, pointer bits in positions 62:57 are masked (LAM width 6), allows bits 62:57 of a user pointer to be used as metadata.There are two
arch_prctls:ARCH_ENABLE_TAGGED_ADDR: enable LAM mode, mask high bits of auser
Quoted 3 lines
pointer.
ARCH_GET_UNTAG_MASK: get current untagged mask.ARCH_GET_MAX_TAG_BITS: the maximum tag bits user can request. zeroif
Quoted 19 lines
LAM is not supported.
The LAM mode is for pre-process, a process has only one chance to set LAM mode.
But there is no API to disable LAM mode. So all of test cases are run under child process.Functions of this test:
MALLOC
-
LAM_U57masks bits 57:62 of a user pointer. Process on user space can dereference such pointers.- Disable LAM, dereference a pointer with metadata above 48 bit or 57 bit lead to trigger SIGSEGV.
TAG_BITSQuoted 7 lines
- Max tag bits of
LAM_U57is 6.Signed-off-by Weihong Zhang <[email protected]>Signed-off-by Kirill A. Shutemov <[email protected]>--- tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/lam.c | 317+++++++++++++++++++++++++++
Quoted 102 lines
2 files changed, 318 insertions(+), 1 deletion(-) create mode 100644
tools/testing/selftests/x86/lam.ctools/testing/selftests/x86/Makefile +1
-1@@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \ - corrupt_xstate_header amx + corrupt_xstate_header amx lam # Some selftests require 32bit support enabled also on 64bit systems
TARGETS_C_32BIT_NEEDED:=ldt_gdtptrace_syscalltools/testing/selftests/x86/lam.c new file +77
-0@@ -0,0 +1,317 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <sys/syscall.h> +#include <time.h> +#include <signal.h> +#include <setjmp.h> +#include <sys/mman.h> +#include <sys/wait.h> +#include <inttypes.h> + +#include "../kselftest.h" + +#ifndef __x86_64__ +# error This test is 64-bit only +#endif + +/* LAM modes, these definitions were copied from kernel code */ +#define LAM_NONE 0 +#define LAM_U57_BITS 6 +/* arch prctl for LAM */ +#define ARCH_GET_UNTAG_MASK 0x4001 +#define ARCH_ENABLE_TAGGED_ADDR 0x4002 +#define ARCH_GET_MAX_TAG_BITS 0x4003 + +/* Specified test function bits */ +#define FUNC_MALLOC 0x1 +#define FUNC_BITS 0x2 + +#define TEST_MASK 0x3 + +#define MALLOC_LEN 32 + +struct testcases { +unsigned int later; +int expected;/* 2: SIGSEGV Error; 1: other errors */ +unsigned long lam; +uint64_t addr; +int (*test_func)(struct testcases *test); +const char *msg; +}; + +int tests_cnt; +jmp_buf segv_env; + +static void segv_handler (int sig) +{ +ksft_print_msg ("Get segmentation fault(%d)." , sig); +siglongjmp (segv_env,1 ); +} + +static inline int cpu_has_lam (void ) +{ +unsigned int cpuinfo[4 ]; + +__cpuid_count (0x7 ,1 , cpuinfo[0 ], cpuinfo[1 ], cpuinfo[2 ], +cpuinfo[3 ]); + +return (cpuinfo[0 ] & (1 <<26 )); +} + +/* +* Set tagged address and read back untag mask. +* check if the untagged mask is expected. +*/ +static int set_lam (unsigned long lam) { +int ret =0 ; +uint64_t ptr =0 ; + +if (lam !=LAM_U57_BITS && lam !=LAM_NONE ) +return -1 ; + +/* Skip check return */ +syscall (SYS_arch_prctl,ARCH_ENABLE_TAGGED_ADDR , lam); + +/* Get untagged mask */
Quoted 9 lines
tools/testing/selftests/x86/lam.c line 82
Quoted 5 lines
tools/testing/selftests/x86/lam.c new file +5
-0+syscall (SYS_arch_prctl,ARCH_GET_UNTAG_MASK , &ptr); + +/* Check mask returned is expected */ +if (lam ==LAM_U57_BITS ) + ret = (ptr != ~(0x3fULL <<57 ));[Hu, Robert]
Any special reason not "bool ret"?
The code didn't involve the type: bool.
error: unknown type name 'bool'
Quoted 10 lines
Quoted 10 lines
tools/testing/selftests/x86/lam.c new file +10
-0+else if (lam ==LAM_NONE ) + ret = (ptr !=-1ULL ); + +return ret; +} + +static unsigned long get_default_tag_bits (void ) { +pid_t pid; +int lam =LAM_NONE ; +int ret =0 ;
Quoted 9 lines
tools/testing/selftests/x86/lam.c line 98
Quoted 5 lines
tools/testing/selftests/x86/lam.c new file +5
-0+ + pid =fork (); +if (pid <0 ) { +perror ("Fork failed." ); + ret =1 ;[Hu, Robert]
In this fault case, at last you "return lam", which inited as 0, not your intended "ret".
Yes, "ret" in here is redundant, will be optimized.
Quoted 16 lines
tools/testing/selftests/x86/lam.c line 100
Quoted 2 lines
tools/testing/selftests/x86/lam.c new file +2
-0+ }else if (pid ==0 ) { +/* Set LAM mode in parent process */ [Hu, Robert]
pid == 0 is child process?Quoted 11 lines
tools/testing/selftests/x86/lam.c new file +11
-0+if (set_lam (LAM_U57_BITS ) ==0 ) + lam =LAM_U57_BITS ; +else + lam =LAM_NONE ; +exit (lam); + }else { +wait (&ret); + lam =WEXITSTATUS (ret); + } + +return lam;
Quoted 9 lines
tools/testing/selftests/x86/lam.c
Quoted 4 lines
tools/testing/selftests/x86/lam.c new file +4
-0+} + +/* According to LAM mode, set metadata in high bits */ static +uint64_t get_metadata (uint64_t src,unsigned long lam) {[Hu, Robert]
This function looks like not "get metadata", but "embed metadata" to canonical address.
Naming looks confusing. Perhapsembed_metadata()?
The name does not express rightly the purpose of this function. Will rename the function.
Quoted 4 lines
Quoted 4 lines
tools/testing/selftests/x86/lam.c new file +4
-0+uint64_t metadata; + +srand (time (NULL )); +/* Get a random value as metadata */
Quoted 9 lines
tools/testing/selftests/x86/lam.c
Quoted 5 lines
tools/testing/selftests/x86/lam.c new file +5
-0+ metadata =rand (); + +switch (lam) { +case LAM_U57_BITS :/* Set metadata in bits 62:57 */ + metadata = (src & ~(0x3fULL << 57)) | ((metadata & 0x3f) <<[Hu, Robert]
Looks like "0x3fULL<< 57" is frequently used across LAM selftests, why not name a macro for "0x3fULL<< 57"?
Add a macro for (0x3fULL << 57) and (0x7fffULL << 48)?.
Quoted 22 lines
tools/testing/selftests/x86/lam.c
Quoted 12 lines
57);
tools/testing/selftests/x86/lam.c new file +11
-0+break ; +default : + metadata = src; +break ; + } + +return metadata; +} + +/* +* Set metadata in user pointer, compare new pointer with original pointer.
Quoted 9 lines
+ * both pointers should point to the same address.
+ */tools/testing/selftests/x86/lam.c new file +7
-0+static int handle_lam_test (void *src,unsigned int lam) { +char *ptr; + +strcpy ((char *)src,"USER POINTER" ); + + ptr = (char *)get_metadata ((uint64_t )src, lam); +if (src == ptr)
Quoted 9 lines
tools/testing/selftests/x86/lam.c
Quoted 6 lines
tools/testing/selftests/x86/lam.c new file +6
-0+return 0 ; + +/* Copy a string into the pointer with metadata */ +strcpy ((char *)ptr,"METADATA POINTER" ); + +return (!!strcmp ((char *)src, (char *)ptr)); }[Hu, Robert]
Why "!!" here? now that function return type is int andstrcmp()returns int.
The " handle_lam_test" only returns 0 and 1. "!!" used to convert return of strcmp to 0 or 1.
Quoted 166 lines
Quoted 94 lines
tools/testing/selftests/x86/lam.c new file +93
-0+ + +int handle_max_bits (struct testcases *test) { +unsigned long exp_bits =get_default_tag_bits (); +unsigned long bits =0 ; + +if (exp_bits !=LAM_NONE ) + exp_bits =LAM_U57_BITS ; + +/* Get LAM max tag bits */ +if (syscall (SYS_arch_prctl,ARCH_GET_MAX_TAG_BITS , &bits) ==-1 ) +return 1 ; + +return (exp_bits != bits); +} + +/* +* Test lam feature through dereference pointer get from malloc. +* @return 0: Pass test. 1: Get failure during test 2: Get SIGSEGV +*/ static int handle_malloc (struct testcases *test) { +char *ptr =NULL ; +int ret =0 ; + +if (test->later ==0 && test->lam !=0 ) +if (set_lam (test->lam) ==-1 ) +return 1 ; + + ptr = (char *)malloc (MALLOC_LEN ); +if (ptr ==NULL ) { +perror ("malloc() failure\n" ); +return 1 ; + } + +/* Set signal handler */ +if (sigsetjmp (segv_env,1 ) ==0 ) { +signal (SIGSEGV , segv_handler); + ret =handle_lam_test (ptr, test->lam); + }else { + ret =2 ; + } + +if (test->later !=0 && test->lam !=0 ) +if (set_lam (test->lam) ==-1 && ret ==0 ) + ret =1 ; + +free (ptr); + +return ret; +} + +static int fork_test (struct testcases *test) { +int ret, child_ret; +pid_t pid; + + pid =fork (); +if (pid <0 ) { +perror ("Fork failed." ); + ret =1 ; + }else if (pid ==0 ) { + ret = test->test_func (test); +exit (ret); + }else { +wait (&child_ret); + ret =WEXITSTATUS (child_ret); + } + +return ret; +} + +static void run_test (struct testcases *test,int count) { +int i, ret =0 ; + +for (i =0 ; i < count; i++) { +struct testcases *t = test + i; + +/* fork a process to run test case */ + ret =fork_test (t); +if (ret !=0 ) + ret = (t->expected == ret); +else + ret = !(t->expected); + + tests_cnt++; +ksft_test_result (ret, t->msg); + } +} + +static struct testcases malloc_cases[] = { + { + .later =0 , + .lam =LAM_U57_BITS , + .test_func = handle_malloc, + .msg = "MALLOC: LAM_U57. Dereferencing pointer withmetadata\n",
tools/testing/selftests/x86/lam.c
Quoted 7 lines
tools/testing/selftests/x86/lam.c new file +7
-0+ }, + { + .later = 1, + .expected = 2, + .lam = LAM_U57_BITS, + .test_func = handle_malloc, + .msg = "MALLOC:[Negtive] Disable LAM. Dereferencingpointer
Quoted 64 lines
with metadata.\n",
tools/testing/selftests/x86/lam.c new file +16
-0+ }, +}; + + +static struct testcases bits_cases[] = { + { + .test_func = handle_max_bits, + .msg ="BITS: Check default tag bits\n" , + }, +}; + +static void cmd_help (void ) +{ +printf ("usage: lam [-h] [-t test list]\n" ); + printf("\t-t test list: run tests specified in the test list, +default:0x%x\n",
TEST_MASK);tools/testing/selftests/x86/lam.c new file +44
-1+printf ("\t\t0x1:malloc; 0x2:max_bits;\n" ); +printf ("\t-h: help\n" ); +} + +int main (int argc,char **argv) +{ +int c =0 ; +unsigned int tests =TEST_MASK ; + + tests_cnt =0 ; + +if (!cpu_has_lam ()) { +ksft_print_msg ("Unsupported LAM feature!\n" ); +return -1 ; + } + +while ((c =getopt (argc, argv,"ht:" )) !=-1 ) { +switch (c) { +case 't' : + tests =strtoul (optarg,NULL ,16 ); +if (!(tests &TEST_MASK )) { +ksft_print_msg ("Invalid argument!\n" ); +return -1 ; + } +break ; +case 'h' : +cmd_help (); +return 0 ; +default : +ksft_print_msg ("Invalid argument\n" ); +return -1 ; + } + } + +if (tests &FUNC_MALLOC ) +run_test (malloc_cases,ARRAY_SIZE (malloc_cases)); + +if (tests &FUNC_BITS ) +run_test (bits_cases,ARRAY_SIZE (bits_cases)); + +ksft_set_plan (tests_cnt); + +return ksft_exit_pass (); +} --2.35.1
PATCH v6 9/11 selftests/x86/lam: Add mmap and SYSCALL test cases for linear-address masking
In reply to Kirill A. ShutemovFrom: Weihong Zhang <[email protected]>
Add mmap and SYSCALL test cases.
SYSCALL test cases:
- LAM supports set metadata in high bits 62:57 (LAM_U57) of a user pointer, pass the pointer to SYSCALL, SYSCALL can dereference the pointer and return correct result.
- Disable LAM, pass a pointer with metadata in high bits to SYSCALL, SYSCALL returns -1 (EFAULT).
MMAP test cases:
- Enable LAM_U57, MMAP with low address (below bits 47), set metadata in high bits of the address, dereference the address should be allowed.
- Enable LAM_U57, MMAP with high address (above bits 47), set metadata in high bits of the address, dereference the address should be allowed.
--- tools/testing/selftests/x86/lam.c | 135 +++++++++++++++++++++++++++++- 1 file changed, 132 insertions(+), 3 deletions(-)
tools/testing/selftests/x86/lam.c +132 -3
@@ -7,6 +7,7 @@#include <signal.h> #include <setjmp.h> #include <sys/mman.h> +#include <sys/utsname.h> #include <sys/wait.h> #include <inttypes.h>
@@ -27,11 +28,18 @@/* Specified test function bits */ #define FUNC_MALLOC 0x1#define FUNC_BITS 0x2 +#define FUNC_MMAP 0x4 +#define FUNC_SYSCALL 0x8 -#define TEST_MASK 0x3 +#define TEST_MASK 0xf + +#define LOW_ADDR (0x1UL << 30) +#define HIGH_ADDR (0x3UL << 48)#define MALLOC_LEN 32 +#define PAGE_SIZE (4 << 10) +struct testcases {unsigned int later;int expected;/* 2: SIGSEGV Error; 1: other errors */
@@ -47,6 +55,7 @@ jmp_buf segv_env;static void segv_handler (int sig) {ksft_print_msg ("Get segmentation fault(%d)." , sig); +siglongjmp (segv_env,1 ); }
@@ -59,6 +68,16 @@ static inline int cpu_has_lam(void)return (cpuinfo[0 ] & (1 <<26 )); } +/* Check 5-level page table feature in CPUID.(EAX=07H, ECX=00H):ECX.[bit 16] */ +static inline int cpu_has_la57 (void ) +{ +unsigned int cpuinfo[4 ]; + +__cpuid_count (0x7 ,0 , cpuinfo[0 ], cpuinfo[1 ], cpuinfo[2 ], cpuinfo[3 ]); + +return (cpuinfo[2 ] & (1 <<16 )); +} +/* * Set tagged address and read back untag mask. * check if the untagged mask is expected.
@@ -204,6 +223,68 @@ static int handle_malloc(struct testcases *test)return ret; } +static int handle_mmap (struct testcases *test) +{ +void *ptr; +unsigned int flags =MAP_PRIVATE |MAP_ANONYMOUS |MAP_FIXED ; +int ret =0 ; + +if (test->later ==0 && test->lam !=0 ) +if (set_lam (test->lam) !=0 ) +return 1 ; + + ptr =mmap ((void *)test->addr,PAGE_SIZE ,PROT_READ |PROT_WRITE , + flags,-1 ,0 ); +if (ptr ==MAP_FAILED ) { +if (test->addr ==HIGH_ADDR ) +if (!cpu_has_la57 ()) +return 3 ;/* unsupport LA57 */ +return 1 ; + } + +if (test->later !=0 && test->lam !=0 ) +if (set_lam (test->lam) !=0 ) + ret =1 ; + +if (ret ==0 ) { +if (sigsetjmp (segv_env,1 ) ==0 ) { +signal (SIGSEGV , segv_handler); + ret =handle_lam_test (ptr, test->lam); + }else { + ret =2 ; + } + } + +munmap (ptr,PAGE_SIZE ); +return ret; +} + +static int handle_syscall (struct testcases *test) +{ +struct utsname unme, *pu; +int ret =0 ; + +if (test->later ==0 && test->lam !=0 ) +if (set_lam (test->lam) !=0 ) +return 1 ; + +if (sigsetjmp (segv_env,1 ) ==0 ) { +signal (SIGSEGV , segv_handler); + pu = (struct utsname *)get_metadata ((uint64_t )&unme, test->lam); + ret =uname (pu); +if (ret <0 ) + ret =1 ; + }else { + ret =2 ; + } + +if (test->later !=0 && test->lam !=0 ) +if (set_lam (test->lam) !=-1 && ret ==0 ) + ret =1 ; + +return ret; +} +static int fork_test (struct testcases *test) {int ret, child_ret;
@@ -259,7 +340,6 @@ static struct testcases malloc_cases[] = { }, }; -static struct testcases bits_cases[] = { { .test_func = handle_max_bits,
@@ -267,11 +347,54 @@ static struct testcases bits_cases[] = { }, }; +static struct testcases syscall_cases[] = { + { + .later =0 , + .lam =LAM_U57_BITS , + .test_func = handle_syscall, + .msg ="SYSCALL: LAM_U57. syscall with metadata\n" , + }, + { + .later =1 , + .expected =1 , + .lam =LAM_U57_BITS , + .test_func = handle_syscall, + .msg ="SYSCALL:[Negtive] Disable LAM. Dereferencing pointer with metadata.\n" , + }, +}; + +static struct testcases mmap_cases[] = { + { + .later =1 , + .expected =0 , + .lam =LAM_U57_BITS , + .addr =HIGH_ADDR , + .test_func = handle_mmap, + .msg ="MMAP: First mmap high address, then set LAM_U57.\n" , + }, + { + .later =0 , + .expected =0 , + .lam =LAM_U57_BITS , + .addr =HIGH_ADDR , + .test_func = handle_mmap, + .msg ="MMAP: First LAM_U57, then High address.\n" , + }, + { + .later =0 , + .expected =0 , + .lam =LAM_U57_BITS , + .addr =LOW_ADDR , + .test_func = handle_mmap, + .msg ="MMAP: First LAM_U57, then Low address.\n" , + }, +}; +static void cmd_help (void ) {printf ("usage: lam [-h] [-t test list]\n" );printf ("\t-t test list: run tests specified in the test list, default:0x%x\n" ,TEST_MASK ); -printf ("\t\t0x1:malloc; 0x2:max_bits;\n" ); +printf ("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall.\n" );printf ("\t-h: help\n" ); }
@@ -311,6 +434,12 @@ int main(int argc, char **argv)if (tests &FUNC_BITS )run_test (bits_cases,ARRAY_SIZE (bits_cases)); +if (tests &FUNC_MMAP ) +run_test (mmap_cases,ARRAY_SIZE (mmap_cases)); + +if (tests &FUNC_SYSCALL ) +run_test (syscall_cases,ARRAY_SIZE (syscall_cases)); +ksft_set_plan (tests_cnt);return ksft_exit_pass ();
Signature · 1 line
2.35.1
RE: [PATCHv6 09/11] selftests/x86/lam: Add mmap and SYSCALL test cases for linear-address masking
Quoted 184 lines
-----Original Message-----
From: Kirill A. Shutemov <[email protected]>
Sent: Monday, August 15, 2022 12:18
To: Dave Hansen <[email protected]>; Lutomirski, Andy <[email protected]>; Peter Zijlstra <[email protected]>
Cc: [email protected]; Kostya Serebryany <[email protected]>; Andrey Ryabinin <[email protected]>; Andrey Konovalov <[email protected]>; Alexander Potapenko <[email protected]>; Taras Madan <[email protected]>; Dmitry Vyukov <[email protected]>; H . J . Lu <[email protected]>; Andi Kleen <[email protected]>; Edgecombe, Rick P <[email protected]>; [email protected]; linux- [email protected]; Zhang, Weihong <[email protected]>; Kirill A . Shutemov <[email protected]>
Subject: [PATCHv6 09/11] selftests/x86/lam: Add mmap and SYSCALL test cases for linear-address maskingFrom: Weihong Zhang <[email protected]>
Add mmap and SYSCALL test cases.
SYSCALL test cases:
- LAM supports set metadata in high bits 62:57 (
LAM_U57) of a user pointer,
pass
the pointer to SYSCALL, SYSCALL can dereference the pointer and return
correct
result.- Disable LAM, pass a pointer with metadata in high bits to SYSCALL, SYSCALL returns -1 (EFAULT).
MMAP test cases:
- Enable
LAM_U57, MMAP with low address (below bits 47), set metadata in high bits of the address, dereference the address should be allowed.- Enable
LAM_U57, MMAP with high address (above bits 47), set metadata in high bits of the address, dereference the address should be allowed.Signed-off-by Weihong Zhang <[email protected]>Signed-off-by Kirill A. Shutemov <[email protected]>--- tools/testing/selftests/x86/lam.c | 135 +++++++++++++++++++++++++++++- 1 file changed, 132 insertions(+), 3 deletions(-)tools/testing/selftests/x86/lam.c +9
-1@@ -7,6 +7,7 @@#include <signal.h> #include <setjmp.h> #include <sys/mman.h> +#include <sys/utsname.h> #include <sys/wait.h> #include <inttypes.h> @@ -27,11 +28,18 @@/* Specified test function bits */ #define FUNC_MALLOC 0x1#define FUNC_BITS 0x2 +#define FUNC_MMAP 0x4 +#define FUNC_SYSCALL 0x8 -#define TEST_MASK 0x3 +#define TEST_MASK 0xf + +#define LOW_ADDR (0x1UL << 30) +#define HIGH_ADDR (0x3UL << 48)#define MALLOC_LEN 32 +#define PAGE_SIZE (4 << 10) +struct testcases {unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ @@ -47,6 +55,7@@
jmp_bufsegv_env; static voidsegv_handler(int sig){tools/testing/selftests/x86/lam.c +73
-1ksft_print_msg ("Get segmentation fault(%d)." , sig); +siglongjmp (segv_env,1 ); }@@ -59,6 +68,16 @@ static inline int cpu_has_lam(void)return (cpuinfo[0 ] & (1 <<26 )); } +/* Check 5-level page table feature in CPUID.(EAX=07H, +ECX=00H):ECX.[bit 16] */ static inline int cpu_has_la57 (void ) { +unsigned int cpuinfo[4 ]; + +__cpuid_count (0x7 ,0 , cpuinfo[0 ], cpuinfo[1 ], cpuinfo[2 ], cpuinfo[3 ]); + +return (cpuinfo[2 ] & (1 <<16 )); +} +/* * Set tagged address and read back untag mask. * check if the untagged mask is expected. @@ -204,6 +223,68 @@ static int handle_malloc(struct testcases *test)return ret; } +static int handle_mmap (struct testcases *test) { +void *ptr; +unsigned int flags =MAP_PRIVATE |MAP_ANONYMOUS |MAP_FIXED ; +int ret =0 ; + +if (test->later ==0 && test->lam !=0 ) +if (set_lam (test->lam) !=0 ) +return 1 ; + + ptr =mmap ((void *)test->addr,PAGE_SIZE ,PROT_READ |PROT_WRITE , + flags,-1 ,0 ); +if (ptr ==MAP_FAILED ) { +if (test->addr ==HIGH_ADDR ) +if (!cpu_has_la57 ()) +return 3 ;/* unsupport LA57 */ +return 1 ; + } + +if (test->later !=0 && test->lam !=0 ) +if (set_lam (test->lam) !=0 ) + ret =1 ; + +if (ret ==0 ) { +if (sigsetjmp (segv_env,1 ) ==0 ) { +signal (SIGSEGV , segv_handler); + ret =handle_lam_test (ptr, test->lam); + }else { + ret =2 ; + } + } + +munmap (ptr,PAGE_SIZE ); +return ret; +} + +static int handle_syscall (struct testcases *test) { +struct utsname unme, *pu; +int ret =0 ; + +if (test->later ==0 && test->lam !=0 ) +if (set_lam (test->lam) !=0 ) +return 1 ; + +if (sigsetjmp (segv_env,1 ) ==0 ) { +signal (SIGSEGV , segv_handler); + pu = (struct utsname *)get_metadata ((uint64_t )&unme, test- >lam); + ret =uname (pu); +if (ret <0 ) + ret =1 ; + }else { + ret =2 ; + } + +if (test->later !=0 && test->lam !=0 ) +if (set_lam (test->lam) !=-1 && ret ==0 ) + ret =1 ; + +return ret; +} +static int fork_test (struct testcases *test) {int ret, child_ret;@@ -259,7 +340,6 @@ static struct testcases malloc_cases[] = { }, }; -static struct testcases bits_cases[] = { { .test_func = handle_max_bits,@@ -267,11 +347,54 @@ static struct testcases bits_cases[] = { }, }; +static struct testcases syscall_cases[] = { + { + .later =0 ,
tools/testing/selftests/x86/lam.c static struct testcases bits_cases[] = { · line 362
Quoted 10 lines
tools/testing/selftests/x86/lam.c +10
-0+ .lam = LAM_U57_BITS, + .test_func = handle_syscall, + .msg = "SYSCALL: LAM_U57. syscall with metadata\n", + }, + { + .later = 1, + .expected = 1, + .lam = LAM_U57_BITS, + .test_func = handle_syscall, + .msg = "SYSCALL:[Negtive] Disable LAM. Dereferencing pointer
[Hu, Robert]
Trivial, Negtive --> Negative.
tools/testing/selftests/x86/lam.c static struct testcases bits_cases[] = { · line 366
Quoted 5 lines
tools/testing/selftests/x86/lam.c +4
-0with metadata.\n", + }, +}; + +static struct testcases mmap_cases[] = {
[Hu, Robert]
Can mmap also have negative case?
Quoted 52 lines
tools/testing/selftests/x86/lam.c +27
-1+ { + .later = 1, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = HIGH_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First mmap high address, then set LAM_U57.\n", + }, + { + .later = 0, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = HIGH_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First LAM_U57, then High address.\n", + }, + { + .later = 0, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = LOW_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First LAM_U57, then Low address.\n", + }, +}; +static void cmd_help (void ) {printf ("usage: lam [-h] [-t test list]\n" );printf ("\t-t test list: run tests specified in the test list, default:0x%x\n" ,TEST_MASK ); -printf ("\t\t0x1:malloc; 0x2:max_bits;\n" ); +printf ("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall.\n" );tools/testing/selftests/x86/lam.c +6
-1printf ("\t-h: help\n" ); }@@ -311,6 +434,12 @@ int main(int argc, char **argv)if (tests &FUNC_BITS )run_test (bits_cases,ARRAY_SIZE (bits_cases)); +if (tests &FUNC_MMAP ) +run_test (mmap_cases,ARRAY_SIZE (mmap_cases)); + +if (tests &FUNC_SYSCALL ) +run_test (syscall_cases,ARRAY_SIZE (syscall_cases)); +ksft_set_plan (tests_cnt);return ksft_exit_pass (); --2.35.1
RE: [PATCHv6 09/11] selftests/x86/lam: Add mmap and SYSCALL test cases for linear-address masking
Quoted 207 lines
-----Original Message-----
From: Hu, Robert <[email protected]>
Sent: Friday, August 19, 2022 4:15 PM
To: Kirill A. Shutemov <[email protected]>; Dave Hansen <[email protected]>; Lutomirski, Andy <[email protected]>; Peter Zijlstra <[email protected]>
Cc: [email protected]; Kostya Serebryany <[email protected]>; Andrey Ryabinin <[email protected]>; Andrey Konovalov <[email protected]>; Alexander Potapenko <[email protected]>; Taras Madan <[email protected]>; Dmitry Vyukov <[email protected]>; H . J . Lu <[email protected]>; Andi Kleen <[email protected]>; Edgecombe, Rick P <[email protected]>; [email protected]; linux- [email protected]; Zhang, Weihong <[email protected]>
Subject: RE: [PATCHv6 09/11] selftests/x86/lam: Add mmap and SYSCALL test cases for linear-address maskingQuoted 8 lines
-----Original Message-----
From: Kirill A. Shutemov <[email protected]>
Sent: Monday, August 15, 2022 12:18
To: Dave Hansen <[email protected]>; Lutomirski, Andy <[email protected]>; Peter Zijlstra <[email protected]>
Cc: [email protected]; Kostya Serebryany <[email protected]>; Andrey Ryabinin <[email protected]>; Andrey Konovalov <[email protected]>; Alexander Potapenko <[email protected]>;Taras
Quoted 1 line
Madan <[email protected]>; Dmitry Vyukov
<[email protected]>; H .
Quoted 3 lines
J . Lu <[email protected]>; Andi Kleen <[email protected]>; Edgecombe, Rick P <[email protected]>; [email protected]; linux- [email protected]; Zhang, Weihong
<[email protected]>; Kirill A .
Quoted 93 lines
Shutemov <[email protected]>
Subject: [PATCHv6 09/11] selftests/x86/lam: Add mmap and SYSCALL test cases for linear-address maskingFrom: Weihong Zhang <[email protected]>
Add mmap and SYSCALL test cases.
SYSCALL test cases:
- LAM supports set metadata in high bits 62:57 (
LAM_U57) of a user
pointer, pass
the pointer to SYSCALL, SYSCALL can dereference the pointer and
return correct
result.- Disable LAM, pass a pointer with metadata in high bits to SYSCALL, SYSCALL returns -1 (EFAULT).
MMAP test cases:
- Enable
LAM_U57, MMAP with low address (below bits 47), set metadata in high bits of the address, dereference the address should be allowed.- Enable
LAM_U57, MMAP with high address (above bits 47), set metadata in high bits of the address, dereference the address should be allowed.Signed-off-by Weihong Zhang <[email protected]>Signed-off-by Kirill A. Shutemov <[email protected]>--- tools/testing/selftests/x86/lam.c | 135 +++++++++++++++++++++++++++++- 1 file changed, 132 insertions(+), 3 deletions(-)tools/testing/selftests/x86/lam.c +9
-1@@ -7,6 +7,7 @@#include <signal.h> #include <setjmp.h> #include <sys/mman.h> +#include <sys/utsname.h> #include <sys/wait.h> #include <inttypes.h> @@ -27,11 +28,18 @@/* Specified test function bits */ #define FUNC_MALLOC 0x1#define FUNC_BITS 0x2 +#define FUNC_MMAP 0x4 +#define FUNC_SYSCALL 0x8 -#define TEST_MASK 0x3 +#define TEST_MASK 0xf + +#define LOW_ADDR (0x1UL << 30) +#define HIGH_ADDR (0x3UL << 48)#define MALLOC_LEN 32 +#define PAGE_SIZE (4 << 10) +struct testcases {unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ @@ -47,6 +55,7@@
jmp_bufsegv_env; static voidsegv_handler(int sig){tools/testing/selftests/x86/lam.c +11
-0ksft_print_msg ("Get segmentation fault(%d)." , sig); +siglongjmp (segv_env,1 ); }@@ -59,6 +68,16 @@ static inline int cpu_has_lam(void)return (cpuinfo[0 ] & (1 <<26 )); } +/* Check 5-level page table feature in CPUID.(EAX=07H, +ECX=00H):ECX.[bit 16] */ static inline int cpu_has_la57 (void ) { +unsigned int cpuinfo[4 ]; + +__cpuid_count (0x7 ,0 , cpuinfo[0 ], cpuinfo[1 ], cpuinfo[2 ], +cpuinfo[3 ]); + +return (cpuinfo[2 ] & (1 <<16 )); +} +/* * Set tagged address and read back untag mask. * check if the untagged mask is expected. tools/testing/selftests/x86/lam.c static int handle_malloc(struct testcases *test) · line 228
Quoted 6 lines
tools/testing/selftests/x86/lam.c +3
-0@@ -204,6 +223,68 @@ static int handle_malloc(struct testcases *test)return ret; } +static int handle_mmap (struct testcases *test) { +void *ptr; + unsigned int flags = MAP_PRIVATE | MAP_ANONYMOUS |
MAP_FIXED;Quoted 34 lines
tools/testing/selftests/x86/lam.c +33
-0+int ret =0 ; + +if (test->later ==0 && test->lam !=0 ) +if (set_lam (test->lam) !=0 ) +return 1 ; + + ptr =mmap ((void *)test->addr,PAGE_SIZE ,PROT_READ |PROT_WRITE , + flags,-1 ,0 ); +if (ptr ==MAP_FAILED ) { +if (test->addr ==HIGH_ADDR ) +if (!cpu_has_la57 ()) +return 3 ;/* unsupport LA57 */ +return 1 ; + } + +if (test->later !=0 && test->lam !=0 ) +if (set_lam (test->lam) !=0 ) + ret =1 ; + +if (ret ==0 ) { +if (sigsetjmp (segv_env,1 ) ==0 ) { +signal (SIGSEGV , segv_handler); + ret =handle_lam_test (ptr, test->lam); + }else { + ret =2 ; + } + } + +munmap (ptr,PAGE_SIZE ); +return ret; +} + +static int handle_syscall (struct testcases *test) {tools/testing/selftests/x86/lam.c static int handle_malloc(struct testcases *test)
Quoted 10 lines
tools/testing/selftests/x86/lam.c +10
-0+struct utsname unme, *pu; +int ret =0 ; + +if (test->later ==0 && test->lam !=0 ) +if (set_lam (test->lam) !=0 ) +return 1 ; + +if (sigsetjmp (segv_env,1 ) ==0 ) { +signal (SIGSEGV , segv_handler); + pu = (struct utsname *)get_metadata((uint64_t)&unme,test-
Quoted 29 lines
tools/testing/selftests/x86/lam.c +14
-1>lam); + ret =uname (pu); +if (ret <0 ) + ret =1 ; + }else { + ret =2 ; + } + +if (test->later !=0 && test->lam !=0 ) +if (set_lam (test->lam) !=-1 && ret ==0 ) + ret =1 ; + +return ret; +} +static int fork_test (struct testcases *test) {int ret, child_ret;@@ -259,7 +340,6 @@ static struct testcases malloc_cases[] = { }, }; -static struct testcases bits_cases[] = { { .test_func = handle_max_bits,@@ -267,11 +347,54 @@ static struct testcases bits_cases[] = { }, };
Quoted 17 lines
tools/testing/selftests/x86/lam.c static struct testcases bits_cases[] = {
Quoted 13 lines
tools/testing/selftests/x86/lam.c +13
-0+static struct testcases syscall_cases[] = { + { + .later =0 , + .lam =LAM_U57_BITS , + .test_func = handle_syscall, + .msg ="SYSCALL: LAM_U57. syscall with metadata\n" , + }, + { + .later =1 , + .expected =1 , + .lam =LAM_U57_BITS , + .test_func = handle_syscall, + .msg = "SYSCALL:[Negtive] Disable LAM. Dereferencingpointer
[Hu, Robert]
Trivial, Negtive --> Negative.
Thanks, will fix the typo.
Quoted 8 lines
tools/testing/selftests/x86/lam.c static struct testcases bits_cases[] = { · line 366
Quoted 5 lines
tools/testing/selftests/x86/lam.c +4
-0with metadata.\n", + }, +}; + +static struct testcases mmap_cases[] = {[Hu, Robert]
Can mmap also have negative case?
Mmap just get a pointer. under U57 mode, negative case for mmap can refer to malloc cases.
Quoted 52 lines
Quoted 52 lines
tools/testing/selftests/x86/lam.c +27
-1+ { + .later = 1, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = HIGH_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First mmap high address, then set LAM_U57.\n", + }, + { + .later = 0, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = HIGH_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First LAM_U57, then High address.\n", + }, + { + .later = 0, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = LOW_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First LAM_U57, then Low address.\n", + }, +}; +static void cmd_help (void ) {printf ("usage: lam [-h] [-t test list]\n" );printf ("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); -printf ("\t\t0x1:malloc; 0x2:max_bits;\n" ); +printf ("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall.\n" );tools/testing/selftests/x86/lam.c +6
-1printf ("\t-h: help\n" ); }@@ -311,6 +434,12 @@ int main(int argc, char **argv)if (tests &FUNC_BITS )run_test (bits_cases,ARRAY_SIZE (bits_cases)); +if (tests &FUNC_MMAP ) +run_test (mmap_cases,ARRAY_SIZE (mmap_cases)); + +if (tests &FUNC_SYSCALL ) +run_test (syscall_cases,ARRAY_SIZE (syscall_cases)); +ksft_set_plan (tests_cnt);return ksft_exit_pass (); --2.35.1
PATCH v6 10/11 selftests/x86/lam: Add io_uring test cases for linear-address masking
In reply to Kirill A. ShutemovFrom: Weihong Zhang <[email protected]>
LAM should be supported in kernel thread, using io_uring to verify LAM feature. The test cases implement read a file through io_uring, the test cases choose an iovec array as receiving buffer, which used to receive data, according to LAM mode, set metadata in high bits of these buffer.
io_uring can deal with these buffers that pointed to pointers with the metadata in high bits.
--- tools/testing/selftests/x86/lam.c | 341 +++++++++++++++++++++++++++++- 1 file changed, 339 insertions(+), 2 deletions(-)
tools/testing/selftests/x86/lam.c +339 -2
@@ -9,8 +9,12 @@#include <sys/mman.h> #include <sys/utsname.h> #include <sys/wait.h> +#include <sys/stat.h> +#include <fcntl.h> #include <inttypes.h> +#include <sys/uio.h> +#include <linux/io_uring.h> #include "../kselftest.h" #ifndef __x86_64__
@@ -30,8 +34,9 @@#define FUNC_BITS 0x2#define FUNC_MMAP 0x4#define FUNC_SYSCALL 0x8 +#define FUNC_URING 0x10 -#define TEST_MASK 0xf +#define TEST_MASK 0x1f#define LOW_ADDR (0x1UL << 30)#define HIGH_ADDR (0x3UL << 48)
@@ -40,6 +45,13 @@#define PAGE_SIZE (4 << 10) +#define barrier () ({ \ + __asm__ __volatile__("" : : : "memory"); \ +}) + +#define URING_QUEUE_SZ 1 +#define URING_BLOCK_SZ 2048 +struct testcases {unsigned int later;int expected;/* 2: SIGSEGV Error; 1: other errors */
@@ -49,6 +61,33 @@ struct testcases {const char *msg; }; +/* Used by CQ of uring, source file handler and file's size */ +struct file_io { +int file_fd; +off_t file_sz; +struct iovec iovecs[]; +}; + +struct io_uring_queue { +unsigned int *head; +unsigned int *tail; +unsigned int *ring_mask; +unsigned int *ring_entries; +unsigned int *flags; +unsigned int *array; +union { +struct io_uring_cqe *cqes; +struct io_uring_sqe *sqes; + } queue; +size_t ring_sz; +}; + +struct io_ring { +int ring_fd; +struct io_uring_queue sq_ring; +struct io_uring_queue cq_ring; +}; +int tests_cnt;jmp_buf segv_env;
@@ -285,6 +324,285 @@ static int handle_syscall(struct testcases *test)return ret; } +int sys_uring_setup (unsigned int entries,struct io_uring_params *p) +{ +return (int )syscall (__NR_io_uring_setup, entries, p); +} + +int sys_uring_enter (int fd,unsigned int to,unsigned int min,unsigned int flags) +{ +return (int )syscall (__NR_io_uring_enter, fd, to, min, flags,NULL ,0 ); +} + +/* Init submission queue and completion queue */ +int mmap_io_uring (struct io_uring_params p,struct io_ring *s) +{ +struct io_uring_queue *sring = &s->sq_ring; +struct io_uring_queue *cring = &s->cq_ring; + + sring->ring_sz = p.sq_off.array + p.sq_entries *sizeof (unsigned int ); + cring->ring_sz = p.cq_off.cqes + p.cq_entries *sizeof (struct io_uring_cqe ); + +if (p.features &IORING_FEAT_SINGLE_MMAP ) { +if (cring->ring_sz > sring->ring_sz) + sring->ring_sz = cring->ring_sz; + + cring->ring_sz = sring->ring_sz; + } + +void *sq_ptr =mmap (0 , sring->ring_sz,PROT_READ |PROT_WRITE , +MAP_SHARED |MAP_POPULATE , s->ring_fd, +IORING_OFF_SQ_RING ); + +if (sq_ptr ==MAP_FAILED ) { +perror ("sub-queue!" ); +return 1 ; + } + +void *cq_ptr = sq_ptr; + +if (!(p.features &IORING_FEAT_SINGLE_MMAP )) { + cq_ptr =mmap (0 , cring->ring_sz,PROT_READ |PROT_WRITE , +MAP_SHARED |MAP_POPULATE , s->ring_fd, +IORING_OFF_CQ_RING ); +if (cq_ptr ==MAP_FAILED ) { +perror ("cpl-queue!" ); +munmap (sq_ptr, sring->ring_sz); +return 1 ; + } + } + + sring->head = sq_ptr + p.sq_off.head; + sring->tail = sq_ptr + p.sq_off.tail; + sring->ring_mask = sq_ptr + p.sq_off.ring_mask; + sring->ring_entries = sq_ptr + p.sq_off.ring_entries; + sring->flags = sq_ptr + p.sq_off.flags; + sring->array = sq_ptr + p.sq_off.array; + +/* Map a queue as mem map */ + s->sq_ring.queue.sqes =mmap (0 , p.sq_entries *sizeof (struct io_uring_sqe ), +PROT_READ |PROT_WRITE ,MAP_SHARED |MAP_POPULATE , + s->ring_fd,IORING_OFF_SQES ); +if (s->sq_ring.queue.sqes ==MAP_FAILED ) { +munmap (sq_ptr, sring->ring_sz); +if (sq_ptr != cq_ptr) { +ksft_print_msg ("failed to mmap uring queue!" ); +munmap (cq_ptr, cring->ring_sz); +return 1 ; + } + } + + cring->head = cq_ptr + p.cq_off.head; + cring->tail = cq_ptr + p.cq_off.tail; + cring->ring_mask = cq_ptr + p.cq_off.ring_mask; + cring->ring_entries = cq_ptr + p.cq_off.ring_entries; + cring->queue.cqes = cq_ptr + p.cq_off.cqes; + +return 0 ; +} + +/* Init io_uring queues */ +int setup_io_uring (struct io_ring *s) +{ +struct io_uring_params para; + +memset (¶,0 ,sizeof (para)); + s->ring_fd =sys_uring_setup (URING_QUEUE_SZ , ¶); +if (s->ring_fd <0 ) +return 1 ; + +return mmap_io_uring (para, s); +} + +/* +* Get data from completion queue. the data buffer saved the file data +* return 0: success; others: error; +*/ +int handle_uring_cq (struct io_ring *s) +{ +struct file_io *fi =NULL ; +struct io_uring_queue *cring = &s->cq_ring; +struct io_uring_cqe *cqe; +unsigned int head; +off_t len =0 ; + + head = *cring->head; + +do { +barrier (); +if (head == *cring->tail) +break ; +/* Get the entry */ + cqe = &cring->queue.cqes[head & *s->cq_ring.ring_mask]; + fi = (struct file_io *)cqe->user_data; +if (cqe->res <0 ) +break ; + +int blocks = (int )(fi->file_sz +URING_BLOCK_SZ -1 ) /URING_BLOCK_SZ ; + +for (int i =0 ; i < blocks; i++) + len += fi->iovecs[i].iov_len; + + head++; + }while (1 ); + + *cring->head = head; +barrier (); + +return (len != fi->file_sz); +} + +/* +* Submit squeue. specify via IORING_OP_READV. +* the buffer need to be set metadata according to LAM mode +*/ +int handle_uring_sq (struct io_ring *ring,struct file_io *fi,unsigned long lam) +{ +int file_fd = fi->file_fd; +struct io_uring_queue *sring = &ring->sq_ring; +unsigned int index =0 , cur_block =0 , tail =0 , next_tail =0 ; +struct io_uring_sqe *sqe; + +off_t remain = fi->file_sz; +int blocks = (int )(remain +URING_BLOCK_SZ -1 ) /URING_BLOCK_SZ ; + +while (remain) { +off_t bytes = remain; +void *buf; + +if (bytes >URING_BLOCK_SZ ) + bytes =URING_BLOCK_SZ ; + + fi->iovecs[cur_block].iov_len = bytes; + +if (posix_memalign (&buf,URING_BLOCK_SZ ,URING_BLOCK_SZ )) +return 1 ; + + fi->iovecs[cur_block].iov_base = (void *)get_metadata ((uint64_t )buf, lam); + remain -= bytes; + cur_block++; + } + + next_tail = *sring->tail; + tail = next_tail; + next_tail++; + +barrier (); + + index = tail & *ring->sq_ring.ring_mask; + + sqe = &ring->sq_ring.queue.sqes[index]; + sqe->fd = file_fd; + sqe->flags =0 ; + sqe->opcode =IORING_OP_READV ; + sqe->addr = (unsigned long )fi->iovecs; + sqe->len = blocks; + sqe->off =0 ; + sqe->user_data = (uint64_t )fi; + + sring->array[index] = index; + tail = next_tail; + +if (*sring->tail != tail) { + *sring->tail = tail; +barrier (); + } + +if (sys_uring_enter (ring->ring_fd,1 ,1 ,IORING_ENTER_GETEVENTS ) <0 ) +return 1 ; + +return 0 ; +} + +/* +* Test LAM in async I/O and io_uring, read current binery through io_uring +* Set metadata in pointers to iovecs buffer. +*/ +int do_uring (unsigned long lam) +{ +struct io_ring *ring; +struct file_io *fi; +struct stat st; +int ret =1 ; +char path[PATH_MAX ]; + +/* get current process path */ +if (readlink ("/proc/self/exe" , path,PATH_MAX ) <=0 ) +return 1 ; + +int file_fd =open (path,O_RDONLY ); + +if (file_fd <0 ) +return 1 ; + +if (fstat (file_fd, &st) <0 ) +return 1 ; + +off_t file_sz = st.st_size; + +int blocks = (int )(file_sz +URING_BLOCK_SZ -1 ) /URING_BLOCK_SZ ; + + fi =malloc (sizeof (*fi) +sizeof (struct iovec ) * blocks); +if (!fi) +return 1 ; + + fi->file_sz = file_sz; + fi->file_fd = file_fd; + + ring =malloc (sizeof (*ring)); +if (!ring) +return 1 ; + +memset (ring,0 ,sizeof (struct io_ring )); + +if (setup_io_uring (ring)) + goto out; + +if (handle_uring_sq (ring, fi, lam)) + goto out; + + ret =handle_uring_cq (ring); + +out: +free (ring); + +for (int i =0 ; i < blocks; i++) { +if (fi->iovecs[i].iov_base) { +uint64_t addr = ((uint64_t )fi->iovecs[i].iov_base); + +switch (lam) { +case LAM_U57_BITS :/* Clear bits 62:57 */ + addr = (addr & ~(0x3fULL <<57 )); +break ; + } +free ((void *)addr); + fi->iovecs[i].iov_base =NULL ; + } + } + +free (fi); + +return ret; +} + +int handle_uring (struct testcases *test) +{ +int ret =0 ; + +if (test->later ==0 && test->lam !=0 ) +if (set_lam (test->lam) !=0 ) +return 1 ; + +if (sigsetjmp (segv_env,1 ) ==0 ) { +signal (SIGSEGV , segv_handler); + ret =do_uring (test->lam); + }else { + ret =2 ; + } + +return ret; +} +static int fork_test (struct testcases *test) {int ret, child_ret;
@@ -324,6 +642,22 @@ static void run_test(struct testcases *test, int count) } } +static struct testcases uring_cases[] = { + { + .later =0 , + .lam =LAM_U57_BITS , + .test_func = handle_uring, + .msg ="URING: LAM_U57. Dereferencing pointer with metadata\n" , + }, + { + .later =1 , + .expected =1 , + .lam =LAM_U57_BITS , + .test_func = handle_uring, + .msg ="URING:[Negtive] Disable LAM. Dereferencing pointer with metadata.\n" , + }, +}; +static struct testcases malloc_cases[] = { { .later =0 ,
@@ -394,7 +728,7 @@ static void cmd_help(void) {printf ("usage: lam [-h] [-t test list]\n" );printf ("\t-t test list: run tests specified in the test list, default:0x%x\n" ,TEST_MASK ); -printf ("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall.\n" ); +printf ("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall; 0x10:io_uring.\n" );printf ("\t-h: help\n" ); }
@@ -440,6 +774,9 @@ int main(int argc, char **argv)if (tests &FUNC_SYSCALL )run_test (syscall_cases,ARRAY_SIZE (syscall_cases)); +if (tests &FUNC_URING ) +run_test (uring_cases,ARRAY_SIZE (uring_cases)); +ksft_set_plan (tests_cnt);return ksft_exit_pass ();
Signature · 1 line
2.35.1
PATCH v6 11/11 selftests/x86/lam: Add inherit test cases for linear-address masking
In reply to Kirill A. ShutemovFrom: Weihong Zhang <[email protected]>
LAM is enabled per-thread and gets inherited on fork(2)/clone(2). exec() reverts LAM status to the default disabled state.
There are two test scenarios:
- Fork test cases:
These cases were used to test the inheritance of LAM for per-thread, Child process generated by fork() should inherit LAM feature from parent process, Child process can get the LAM mode same as parent process.
- Execve test cases:
Processes generated by execve() are different from processes generated by fork(), these processes revert LAM status to disabled status.
--- tools/testing/selftests/x86/lam.c | 123 +++++++++++++++++++++++++++++- 1 file changed, 120 insertions(+), 3 deletions(-)
tools/testing/selftests/x86/lam.c +120 -3
@@ -35,8 +35,9 @@#define FUNC_MMAP 0x4#define FUNC_SYSCALL 0x8#define FUNC_URING 0x10 +#define FUNC_INHERITE 0x20 -#define TEST_MASK 0x1f +#define TEST_MASK 0x3f#define LOW_ADDR (0x1UL << 30)#define HIGH_ADDR (0x3UL << 48)
@@ -169,6 +170,28 @@ static unsigned long get_default_tag_bits(void)return lam; } +/* +* Set tagged address and read back untag mask. +* check if the untag mask is expected. +*/ +static int get_lam (void ) +{ +uint64_t ptr =0 ; +int ret =-1 ; +/* Get untagged mask */ +if (syscall (SYS_arch_prctl,ARCH_GET_UNTAG_MASK , &ptr) ==-1 ) +return -1 ; + +/* Check mask returned is expected */ +if (ptr == ~(0x3fULL <<57 )) + ret =LAM_U57_BITS ; +else if (ptr ==-1ULL ) + ret =LAM_NONE ; + + +return ret; +} +/* According to LAM mode, set metadata in high bits */ static uint64_t get_metadata (uint64_t src,unsigned long lam) {
@@ -623,6 +646,72 @@ static int fork_test(struct testcases *test)return ret; } +static int handle_execve (struct testcases *test) +{ +int ret, child_ret; +int lam = test->lam; +pid_t pid; + + pid =fork (); +if (pid <0 ) { +perror ("Fork failed." ); + ret =1 ; + }else if (pid ==0 ) { +char path[PATH_MAX ]; + +/* Set LAM mode in parent process */ +if (set_lam (lam) !=0 ) +return 1 ; + +/* Get current binary's path and the binary was run by execve */ +if (readlink ("/proc/self/exe" , path,PATH_MAX ) <=0 ) +exit (-1 ); + +/* run binary to get LAM mode and return to parent process */ +if (execlp (path, path,"-t 0x0" ,NULL ) <0 ) { +perror ("error on exec" ); +exit (-1 ); + } + }else { +wait (&child_ret); + ret =WEXITSTATUS (child_ret); +if (ret !=LAM_NONE ) +return 1 ; + } + +return 0 ; +} + +static int handle_inheritance (struct testcases *test) +{ +int ret, child_ret; +int lam = test->lam; +pid_t pid; + +/* Set LAM mode in parent process */ +if (set_lam (lam) !=0 ) +return 1 ; + + pid =fork (); +if (pid <0 ) { +perror ("Fork failed." ); +return 1 ; + }else if (pid ==0 ) { +/* Set LAM mode in parent process */ +int child_lam =get_lam (); + +exit (child_lam); + }else { +wait (&child_ret); + ret =WEXITSTATUS (child_ret); + +if (lam != ret) +return 1 ; + } + +return 0 ; +} +static void run_test (struct testcases *test,int count) {int i, ret =0 ;
@@ -724,11 +813,26 @@ static struct testcases mmap_cases[] = { }, }; +static struct testcases inheritance_cases[] = { + { + .expected =0 , + .lam =LAM_U57_BITS , + .test_func = handle_inheritance, + .msg ="FORK: LAM_U57, child process should get LAM mode same as parent\n" , + }, + { + .expected =0 , + .lam =LAM_U57_BITS , + .test_func = handle_execve, + .msg ="EXECVE: LAM_U57, child process should get disabled LAM mode\n" , + }, +}; +static void cmd_help (void ) {printf ("usage: lam [-h] [-t test list]\n" );printf ("\t-t test list: run tests specified in the test list, default:0x%x\n" ,TEST_MASK ); -printf ("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall; 0x10:io_uring.\n" ); +printf ("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall; 0x10:io_uring; 0x20:inherit;\n" );printf ("\t-h: help\n" ); }
@@ -748,7 +852,7 @@ int main(int argc, char **argv)switch (c) {case 't' : tests =strtoul (optarg,NULL ,16 ); -if (!(tests &TEST_MASK )) { +if (tests && !(tests &TEST_MASK )) {ksft_print_msg ("Invalid argument!\n" );return -1 ; }
@@ -762,6 +866,16 @@ int main(int argc, char **argv) } } +/* +* When tests is 0, it is not a real test case; +* the option used by test case(execve) to check the lam mode in +* process generated by execve, the process read back lam mode and +* check with lam mode in parent process. +*/ +if (!tests) +return (get_lam ()); + +/* Run test cases */ if (tests &FUNC_MALLOC )run_test (malloc_cases,ARRAY_SIZE (malloc_cases));
@@ -777,6 +891,9 @@ int main(int argc, char **argv)if (tests &FUNC_URING )run_test (uring_cases,ARRAY_SIZE (uring_cases)); +if (tests &FUNC_INHERITE ) +run_test (inheritance_cases,ARRAY_SIZE (inheritance_cases)); +ksft_set_plan (tests_cnt);return ksft_exit_pass ();
Signature · 1 line
2.35.1
Quoted 9 lines · Kirill A. Shutemov
Kirill A. Shutemov (7):
x86/mm: FixCR3_ADDR_MASK
x86: CPUID and CR3/CR4 flags for Linear Address Masking
mm: Pass downmm_structtountagged_addr()
x86/mm: Handle LAM on context switch
x86/uaccess: Provideuntagged_addr()and remove tags before address
check
x86/mm: Providearch_prctl()interface for LAM
x86: Expose untagging mask in/proc/$PID/arch_status
Over-all these are not terrible.. I've replied with a few nits; with those fixed:
Quoted 13 lines · Peter Zijlstra
Quoted 9 lines · Kirill A. Shutemov
Kirill A. Shutemov (7):
x86/mm: FixCR3_ADDR_MASK
x86: CPUID and CR3/CR4 flags for Linear Address Masking
mm: Pass downmm_structtountagged_addr()
x86/mm: Handle LAM on context switch
x86/uaccess: Provideuntagged_addr()and remove tags before address
check
x86/mm: Providearch_prctl()interface for LAM
x86: Expose untagging mask in/proc/$PID/arch_status
Quoted 4 lines
Over-all these are not terrible.. I've replied with a few nits; with those fixed:
The patchset is green on the LLVM buildbot:
https://lab.llvm.org/buildbot/#/builders/169, so my Tested-by: tags persist.
I'm going to make minor changes to HWASan runtime to accommodate for the new arch_prctl().
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Liana Sebastian Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg