arm64: Memory Tagging Extension user-space support
Subsystems (15)
Catalin Marinas
maintainerv5 cover arm64: Memory Tagging Extension user-space support
This is the 5th version (4th version here [1]) of the series adding user-space support for the ARMv8.5 Memory Tagging Extension ([2], [3]). The patches are also available on this branch:
git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux devel/mte-v5
There are no user ABI changes from v4 and I do not anticipate further updates affecting the ABI. Subsequent ABI improvements, if needed, will be done in a backwards-compatible manner. The glibc counterpart is also under discussion [4].
My plan is to push these patches into linux-next for wider coverage, with an aim for merging into 5.9 unless major reworking is needed. I would be grateful if mm folks review/ack/nak those patches touching mm/ (and, of course, any other patch in this series, feedback always welcomed). Thank you.
Changes in this version:
- Removed the Device Tree memory node description requirement after agreement with the hardware architects that the CPUID should reflect the features supported by the general purpose memory.
- Dropped the command line argument to disable MTE at boot in the absence of a strong argument in its favour.
- Fixed handling of compound pages (inadvertently clearing valid tags in previously mapped small pages).
- Some reworking of the copy_{user,}highpage() functions.
- Rebased to 5.8-rc2
[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/
[2] https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/enhancing-memory-safety
[3] https://developer.arm.com/-/media/Arm%20Developer%20Community/PDF/Arm_Memory_Tagging_Extension_Whitepaper.pdf
[4] https://sourceware.org/pipermail/libc-alpha/2020-June/115039.html
Catalin Marinas (13):
arm64: mte: Use Normal Tagged attributes for the linear map
arm64: mte: Clear the tags when a page is mapped in user-space with
PROT_MTE
arm64: Avoid unnecessary clear_user_page() indirection
arm64: mte: Tags-aware aware memcmp_pages() implementation
arm64: mte: Add PROT_MTE support to mmap() and mprotect()
mm: Introduce arch_validate_flags()
arm64: mte: Validate the PROT_MTE request via arch_validate_flags()
mm: Allow arm64 mmap(PROT_MTE) on RAM-based files
arm64: mte: Allow user control of the tag check mode via prctl()
arm64: mte: Allow user control of the generated random tags via
prctl()
arm64: mte: Restore the GCR_EL1 register after a suspend
arm64: mte: Add PTRACE_{PEEK,POKE}MTETAGS support
fs: Handle intra-page faults in copy_mount_options()
Kevin Brodsky (1):
mm: Introduce arch_calc_vm_flag_bits()
Steven Price (4):
mm: Add PG_ARCH_2 page flag
mm: Add arch hooks for saving/restoring tags
arm64: mte: Enable swap of tagged pages
arm64: mte: Save tags when hibernating
Vincenzo Frascino (7):
arm64: mte: system register definitions
arm64: mte: CPU feature detection and initial sysreg configuration
arm64: mte: Add specific SIGSEGV codes
arm64: mte: Handle synchronous and asynchronous tag check faults
arm64: mte: Tags-aware copy_{user_,}highpage() implementations
arm64: mte: Kconfig entry
arm64: mte: Add Memory Tagging Extension documentation
Documentation/arm64/cpu-feature-registers.rst | 2 +
Documentation/arm64/elf_hwcaps.rst | 4 +
Documentation/arm64/index.rst | 1 +
.../arm64/memory-tagging-extension.rst | 297 ++++++++++++++++
arch/arm64/Kconfig | 29 ++
arch/arm64/include/asm/cpucaps.h | 3 +-
arch/arm64/include/asm/cpufeature.h | 6 +
arch/arm64/include/asm/hwcap.h | 1 +
arch/arm64/include/asm/kvm_arm.h | 3 +-
arch/arm64/include/asm/memory.h | 17 +-
arch/arm64/include/asm/mman.h | 56 ++-
arch/arm64/include/asm/mte.h | 86 +++++
arch/arm64/include/asm/page.h | 19 +-
arch/arm64/include/asm/pgtable-prot.h | 2 +
arch/arm64/include/asm/pgtable.h | 46 ++-
arch/arm64/include/asm/processor.h | 4 +
arch/arm64/include/asm/sysreg.h | 61 ++++
arch/arm64/include/asm/thread_info.h | 4 +-
arch/arm64/include/uapi/asm/hwcap.h | 1 +
arch/arm64/include/uapi/asm/mman.h | 1 +
arch/arm64/include/uapi/asm/ptrace.h | 4 +
arch/arm64/kernel/Makefile | 1 +
arch/arm64/kernel/cpufeature.c | 61 ++++
arch/arm64/kernel/cpuinfo.c | 1 +
arch/arm64/kernel/entry.S | 37 ++
arch/arm64/kernel/hibernate.c | 118 +++++++
arch/arm64/kernel/mte.c | 331 ++++++++++++++++++
arch/arm64/kernel/process.c | 31 +-
arch/arm64/kernel/ptrace.c | 9 +-
arch/arm64/kernel/signal.c | 8 +
arch/arm64/kernel/suspend.c | 4 +
arch/arm64/kernel/syscall.c | 10 +
arch/arm64/lib/Makefile | 2 +
arch/arm64/lib/mte.S | 151 ++++++++
arch/arm64/mm/Makefile | 1 +
arch/arm64/mm/copypage.c | 25 +-
arch/arm64/mm/dump.c | 4 +
arch/arm64/mm/fault.c | 9 +-
arch/arm64/mm/mmu.c | 22 +-
arch/arm64/mm/mteswap.c | 82 +++++
arch/arm64/mm/proc.S | 8 +-
arch/x86/kernel/signal_compat.c | 2 +-
fs/namespace.c | 24 +-
fs/proc/page.c | 3 +
fs/proc/task_mmu.c | 4 +
include/{linux => asm-generic}/pgtable.h | 222 ++----------
include/linux/kernel-page-flags.h | 1 +
include/linux/mm.h | 8 +
include/linux/mman.h | 22 +-
include/linux/page-flags.h | 3 +
include/linux/pgtable.h | 23 ++
include/trace/events/mmflags.h | 9 +-
include/uapi/asm-generic/siginfo.h | 4 +-
include/uapi/linux/prctl.h | 9 +
mm/Kconfig | 3 +
mm/mmap.c | 9 +
mm/mprotect.c | 6 +
mm/page_io.c | 10 +
mm/shmem.c | 9 +
mm/swapfile.c | 2 +
mm/util.c | 2 +-
tools/vm/page-types.c | 2 +
62 files changed, 1704 insertions(+), 235 deletions(-)
create mode 100644 Documentation/arm64/memory-tagging-extension.rst
create mode 100644 arch/arm64/include/asm/mte.h
create mode 100644 arch/arm64/kernel/mte.c
create mode 100644 arch/arm64/lib/mte.S
create mode 100644 arch/arm64/mm/mteswap.c
copy include/{linux => asm-generic}/pgtable.h (85%)Catalin Marinas
maintainerPATCH v5 1/25 arm64: mte: system register definitions
From: Vincenzo Frascino <[email protected]>
Add Memory Tagging Extension system register definitions together with the relevant bitfields.
---
Notes:
v2:
- Added SET_PSTATE_TCO() macro.
arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/include/asm/sysreg.h | 53 ++++++++++++++++++++++++++++ arch/arm64/include/uapi/asm/ptrace.h | 1 + arch/arm64/kernel/ptrace.c | 2 +- 4 files changed, 56 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_arm.h +1 -0
@@ -12,6 +12,7 @@#include <asm/types.h> /* Hyp Configuration Register (HCR) bits */ +#define HCR_ATA (UL(1) << 56)#define HCR_FWB (UL(1) << 46)#define HCR_API (UL(1) << 41)#define HCR_APK (UL(1) << 40)
arch/arm64/include/asm/sysreg.h +53 -0
123 lines not on this page · raw message
arch/arm64/include/uapi/asm/ptrace.h +1 -0
@@ -51,6 +51,7 @@#define PSR_PAN_BIT 0x00400000#define PSR_UAO_BIT 0x00800000#define PSR_DIT_BIT 0x01000000 +#define PSR_TCO_BIT 0x02000000#define PSR_V_BIT 0x10000000#define PSR_C_BIT 0x20000000#define PSR_Z_BIT 0x40000000
arch/arm64/kernel/ptrace.c +1 -1
@@ -1873,7 +1873,7 @@ void syscall_trace_exit(struct pt_regs *regs)* We also reserve IL for the kernel; SS is handled dynamically. */ #define SPSR_EL1_AARCH64_RES0_BITS \ - (GENMASK_ULL(63, 32) | GENMASK_ULL(27, 25) | GENMASK_ULL(23, 22) | \ + (GENMASK_ULL(63, 32) | GENMASK_ULL(27, 26) | GENMASK_ULL(23, 22) | \ GENMASK_ULL(20, 13) | GENMASK_ULL(5, 5))#define SPSR_EL1_AARCH32_RES0_BITS \ (GENMASK_ULL(63, 32) | GENMASK_ULL(22, 22) | GENMASK_ULL(20, 20))
Catalin Marinas
maintainerPATCH v5 2/25 arm64: mte: CPU feature detection and initial sysreg configuration
In reply to Catalin MarinasFrom: Vincenzo Frascino <[email protected]>
Add the cpufeature and hwcap entries to detect the presence of MTE on the boot CPUs (primary and secondary). Any late secondary CPU not supporting the feature, if detected during boot, will be parked.
In addition, add the minimum SCTLR_EL1 and HCR_EL2 bits for enabling MTE. Without subsequent setting of MAIR, these bits do not have an effect on tag checking.
--- arch/arm64/include/asm/cpucaps.h | 3 ++- arch/arm64/include/asm/cpufeature.h | 6 ++++++ arch/arm64/include/asm/hwcap.h | 1 + arch/arm64/include/asm/kvm_arm.h | 2 +- arch/arm64/include/asm/sysreg.h | 1 + arch/arm64/include/uapi/asm/hwcap.h | 1 + arch/arm64/kernel/cpufeature.c | 30 +++++++++++++++++++++++++++++ arch/arm64/kernel/cpuinfo.c | 1 + 8 files changed, 43 insertions(+), 2 deletions(-)
arch/arm64/include/asm/cpucaps.h +2 -1
@@ -62,7 +62,8 @@#define ARM64_HAS_GENERIC_AUTH 52#define ARM64_HAS_32BIT_EL1 53#define ARM64_BTI 54 +#define ARM64_MTE 55 -#define ARM64_NCAPS 55 +#define ARM64_NCAPS 56#endif /* __ASM_CPUCAPS_H */
arch/arm64/include/asm/cpufeature.h +6 -0
@@ -681,6 +681,12 @@ static inline bool system_uses_irq_prio_masking(void)cpus_have_const_cap (ARM64_HAS_IRQ_PRIO_MASKING ); } +static inline bool system_supports_mte (void ) +{ +return IS_ENABLED (CONFIG_ARM64_MTE ) && +cpus_have_const_cap (ARM64_MTE ); +} +static inline bool system_has_prio_mask_debugging (void ) { return IS_ENABLED(CONFIG_ARM64_DEBUG_PRIORITY_MASKING) &&
arch/arm64/include/asm/hwcap.h +1 -0
@@ -95,6 +95,7 @@#define KERNEL_HWCAP_DGH __khwcap2_feature(DGH)#define KERNEL_HWCAP_RNG __khwcap2_feature(RNG)#define KERNEL_HWCAP_BTI __khwcap2_feature(BTI) +#define KERNEL_HWCAP_MTE __khwcap2_feature(MTE)/* * This yields a mask that user programs can use to figure out what
arch/arm64/include/asm/kvm_arm.h +1 -1
@@ -78,7 +78,7 @@ HCR_AMO | HCR_SWIO | HCR_TIDCP | HCR_RW | HCR_TLOR | \ HCR_FMO | HCR_IMO)#define HCR_VIRT_EXCP_MASK (HCR_VSE | HCR_VI | HCR_VF) -#define HCR_HOST_NVHE_FLAGS (HCR_RW | HCR_API | HCR_APK) +#define HCR_HOST_NVHE_FLAGS (HCR_RW | HCR_API | HCR_APK | HCR_ATA)#define HCR_HOST_VHE_FLAGS (HCR_RW | HCR_TGE | HCR_E2H)/* TCR_EL2 Registers bits */
arch/arm64/include/asm/sysreg.h +1 -0
@@ -613,6 +613,7 @@SCTLR_EL1_SA0 |SCTLR_EL1_SED | SCTLR_ELx_I |\SCTLR_EL1_DZE |SCTLR_EL1_UCT |\SCTLR_EL1_NTWE | SCTLR_ELx_IESB |SCTLR_EL1_SPAN |\ + SCTLR_ELx_ITFSB| SCTLR_ELx_ATA |SCTLR_EL1_ATA0 |\ ENDIAN_SET_EL1 | SCTLR_EL1_UCI | SCTLR_EL1_RES1)/* MAIR_ELx memory attributes (used by Linux) */
arch/arm64/include/uapi/asm/hwcap.h +1 -0
@@ -74,5 +74,6 @@#define HWCAP2_DGH (1 << 15)#define HWCAP2_RNG (1 << 16)#define HWCAP2_BTI (1 << 17) +#define HWCAP2_MTE (1 << 18)#endif /* _UAPI__ASM_HWCAP_H */
arch/arm64/kernel/cpufeature.c +30 -0
@@ -243,6 +243,8 @@ static const struct arm64_ftr_bits ftr_id_aa64pfr1[] = {ARM64_FTR_BITS (FTR_VISIBLE ,FTR_STRICT ,FTR_LOWER_SAFE ,ID_AA64PFR1_SSBS_SHIFT ,4 ,ID_AA64PFR1_SSBS_PSTATE_NI ),ARM64_FTR_BITS (FTR_VISIBLE_IF_IS_ENABLED (CONFIG_ARM64_BTI ),FTR_STRICT ,FTR_LOWER_SAFE ,ID_AA64PFR1_BT_SHIFT ,4 ,0 ), +ARM64_FTR_BITS (FTR_VISIBLE_IF_IS_ENABLED (CONFIG_ARM64_MTE ), +FTR_STRICT ,FTR_LOWER_SAFE ,ID_AA64PFR1_MTE_SHIFT ,4 ,ID_AA64PFR1_MTE_NI ), ARM64_FTR_END, };
@@ -1655,6 +1657,18 @@ static void bti_enable(const struct arm64_cpu_capabilities *__unused) }#endif /* CONFIG_ARM64_BTI */ +#ifdef CONFIG_ARM64_MTE +static void cpu_enable_mte (struct arm64_cpu_capabilities const *cap) +{ +/* all non-zero tags excluded by default */ +write_sysreg_s (SYS_GCR_EL1_RRND |SYS_GCR_EL1_EXCL_MASK ,SYS_GCR_EL1 ); +write_sysreg_s (0 ,SYS_TFSR_EL1 ); +write_sysreg_s (0 ,SYS_TFSRE0_EL1 ); + +isb (); +} +#endif /* CONFIG_ARM64_MTE */ +/* Internal helper functions to match cpu capability type */ static bool cpucap_late_cpu_optional (const struct arm64_cpu_capabilities *cap)
@@ -2054,6 +2068,19 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .sign = FTR_UNSIGNED, },#endif +#ifdef CONFIG_ARM64_MTE + { + .desc = "Memory Tagging Extension", + .capability = ARM64_MTE, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_cpuid_feature, + .sys_reg = SYS_ID_AA64PFR1_EL1, + .field_pos = ID_AA64PFR1_MTE_SHIFT, + .min_field_value = ID_AA64PFR1_MTE, + .sign = FTR_UNSIGNED, + .cpu_enable = cpu_enable_mte, + }, +#endif /* CONFIG_ARM64_MTE */ {}, };
@@ -2170,6 +2197,9 @@ static const struct arm64_cpu_capabilities arm64_elf_hwcaps[] = { HWCAP_MULTI_CAP(ptr_auth_hwcap_addr_matches, CAP_HWCAP, KERNEL_HWCAP_PACA), HWCAP_MULTI_CAP(ptr_auth_hwcap_gen_matches, CAP_HWCAP, KERNEL_HWCAP_PACG),#endif +#ifdef CONFIG_ARM64_MTE + HWCAP_CAP(SYS_ID_AA64PFR1_EL1, ID_AA64PFR1_MTE_SHIFT, FTR_UNSIGNED, ID_AA64PFR1_MTE, CAP_HWCAP, KERNEL_HWCAP_MTE), +#endif /* CONFIG_ARM64_MTE */ {}, };
arch/arm64/kernel/cpuinfo.c +1 -0
@@ -93,6 +93,7 @@ static const char *const hwcap_str[] = {"dgh" ,"rng" ,"bti" , +"mte" , NULL };
Catalin Marinas
maintainerPATCH v5 3/25 arm64: mte: Use Normal Tagged attributes for the linear map
In reply to Catalin MarinasOnce user space is given access to tagged memory, the kernel must be able to clear/save/restore tags visible to the user. This is done via the linear mapping, therefore map it as such. The new MT_NORMAL_TAGGED index for MAIR_EL1 is initially mapped as Normal memory and later changed to Normal Tagged via the cpufeature infrastructure. From a mismatched attribute aliases perspective, the Tagged memory is considered a permission and it won't lead to undefined behaviour.
---
Notes:
v5:
- Move the clearing of the zero page since clear_page() to a later patch.
v3:
- Restrict the safe attribute change in pgattr_change_is_safe() only to Normal to/from Normal-Tagged (old version allow any other type as long as old or new was Normal(-Tagged)).
arch/arm64/include/asm/memory.h | 1 + arch/arm64/include/asm/pgtable-prot.h | 2 ++ arch/arm64/kernel/cpufeature.c | 24 ++++++++++++++++++++++++ arch/arm64/mm/dump.c | 4 ++++ arch/arm64/mm/mmu.c | 22 ++++++++++++++++++++-- arch/arm64/mm/proc.S | 8 ++++++-- 6 files changed, 57 insertions(+), 4 deletions(-)
arch/arm64/include/asm/memory.h +1 -0
@@ -136,6 +136,7 @@#define MT_NORMAL_NC 3#define MT_NORMAL 4#define MT_NORMAL_WT 5 +#define MT_NORMAL_TAGGED 6/* * Memory types for Stage-2 translation
arch/arm64/include/asm/pgtable-prot.h +2 -0
@@ -50,6 +50,7 @@ extern bool arm64_use_ng_mappings;#define PROT_NORMAL_NC (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL_NC))#define PROT_NORMAL_WT (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL_WT))#define PROT_NORMAL (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL)) +#define PROT_NORMAL_TAGGED (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL_TAGGED))#define PROT_SECT_DEVICE_nGnRE (PROT_SECT_DEFAULT | PMD_SECT_PXN | PMD_SECT_UXN | PMD_ATTRINDX(MT_DEVICE_nGnRE))#define PROT_SECT_NORMAL (PROT_SECT_DEFAULT | PMD_SECT_PXN | PMD_SECT_UXN | PMD_ATTRINDX(MT_NORMAL))
@@ -59,6 +60,7 @@ extern bool arm64_use_ng_mappings;#define _HYP_PAGE_DEFAULT _PAGE_DEFAULT#define PAGE_KERNEL __pgprot(PROT_NORMAL) +#define PAGE_KERNEL_TAGGED __pgprot(PROT_NORMAL_TAGGED)#define PAGE_KERNEL_RO __pgprot((PROT_NORMAL & ~PTE_WRITE) | PTE_RDONLY)#define PAGE_KERNEL_ROX __pgprot((PROT_NORMAL & ~(PTE_WRITE | PTE_PXN)) | PTE_RDONLY)#define PAGE_KERNEL_EXEC __pgprot(PROT_NORMAL & ~PTE_PXN)
arch/arm64/kernel/cpufeature.c +24 -0
@@ -1660,13 +1660,37 @@ static void bti_enable(const struct arm64_cpu_capabilities *__unused)#ifdef CONFIG_ARM64_MTE static void cpu_enable_mte (struct arm64_cpu_capabilities const *cap) { +u64 mair; +/* all non-zero tags excluded by default */ write_sysreg_s (SYS_GCR_EL1_RRND |SYS_GCR_EL1_EXCL_MASK ,SYS_GCR_EL1 );write_sysreg_s (0 ,SYS_TFSR_EL1 );write_sysreg_s (0 ,SYS_TFSRE0_EL1 ); +/* +* Update the MT_NORMAL_TAGGED index in MAIR_EL1. Tag checking is +* disabled for the kernel, so there won't be any observable effect +* other than allowing the kernel to read and write tags. +*/ + mair =read_sysreg_s (SYS_MAIR_EL1 ); + mair &= ~MAIR_ATTRIDX (MAIR_ATTR_MASK ,MT_NORMAL_TAGGED ); + mair |=MAIR_ATTRIDX (MAIR_ATTR_NORMAL_TAGGED ,MT_NORMAL_TAGGED ); +write_sysreg_s (mair,SYS_MAIR_EL1 ); +isb (); } + +static int __init system_enable_mte (void ) +{ +if (!system_supports_mte ()) +return 0 ; + +/* Ensure the TLB does not have stale MAIR attributes */ +flush_tlb_all (); + +return 0 ; +} +core_initcall (system_enable_mte);#endif /* CONFIG_ARM64_MTE */ /* Internal helper functions to match cpu capability type */
arch/arm64/mm/dump.c +4 -0
@@ -169,6 +169,10 @@ static const struct prot_bits pte_bits[] = { .mask = PTE_ATTRINDX_MASK, .val = PTE_ATTRINDX(MT_NORMAL), .set = "MEM/NORMAL", + }, { + .mask = PTE_ATTRINDX_MASK, + .val = PTE_ATTRINDX(MT_NORMAL_TAGGED), + .set = "MEM/NORMAL-TAGGED", } };
arch/arm64/mm/mmu.c +20 -2
@@ -121,7 +121,7 @@ static bool pgattr_change_is_safe(u64 old, u64 new)* The following mapping attributes may be updated in live * kernel mappings without the need for break-before-make. */ -static const pteval_t mask =PTE_PXN |PTE_RDONLY |PTE_WRITE |PTE_NG ; +pteval_t mask =PTE_PXN |PTE_RDONLY |PTE_WRITE |PTE_NG ;/* creating or taking down mappings is always safe */ if (old ==0 || new ==0 )
@@ -135,6 +135,19 @@ static bool pgattr_change_is_safe(u64 old, u64 new)if (old & ~new &PTE_NG )return false; +if (system_supports_mte ()) { +/* +* Changing the memory type between Normal and Normal-Tagged +* is safe since Tagged is considered a permission attribute +* from the mismatched attribute aliases perspective. +*/ +if (((old &PTE_ATTRINDX_MASK ) ==PTE_ATTRINDX (MT_NORMAL ) || + (old &PTE_ATTRINDX_MASK ) ==PTE_ATTRINDX (MT_NORMAL_TAGGED )) && + ((new &PTE_ATTRINDX_MASK ) ==PTE_ATTRINDX (MT_NORMAL ) || + (new &PTE_ATTRINDX_MASK ) ==PTE_ATTRINDX (MT_NORMAL_TAGGED ))) + mask |=PTE_ATTRINDX_MASK ; + } +return ((old ^ new) & ~mask) ==0 ; }
@@ -490,7 +503,12 @@ static void __init map_mem(pgd_t *pgdp)if (memblock_is_nomap (reg))continue ; -__map_memblock (pgdp, start, end,PAGE_KERNEL , flags); +/* +* The linear map must allow allocation tags reading/writing +* if MTE is present. Otherwise, it has the same attributes as +* PAGE_KERNEL. +*/ +__map_memblock (pgdp, start, end,PAGE_KERNEL_TAGGED , flags); }/*
arch/arm64/mm/proc.S +6 -2
@@ -44,14 +44,18 @@ #define TCR_KASAN_FLAGS 0 #endif -/* Default MAIR_EL1 */ +/* + * Default MAIR_EL1. MT_NORMAL_TAGGED is initially mapped as Normal memory and + * changed later to Normal Tagged if the system supports MTE. + */ #define MAIR_EL1_SET \ (MAIR_ATTRIDX(MAIR_ATTR_DEVICE_nGnRnE, MT_DEVICE_nGnRnE) | \ MAIR_ATTRIDX(MAIR_ATTR_DEVICE_nGnRE, MT_DEVICE_nGnRE) | \ MAIR_ATTRIDX(MAIR_ATTR_DEVICE_GRE, MT_DEVICE_GRE) | \ MAIR_ATTRIDX(MAIR_ATTR_NORMAL_NC, MT_NORMAL_NC) | \ MAIR_ATTRIDX(MAIR_ATTR_NORMAL, MT_NORMAL) | \ - MAIR_ATTRIDX(MAIR_ATTR_NORMAL_WT, MT_NORMAL_WT)) + MAIR_ATTRIDX(MAIR_ATTR_NORMAL_WT, MT_NORMAL_WT) | \ + MAIR_ATTRIDX(MAIR_ATTR_NORMAL, MT_NORMAL_TAGGED)) #ifdef CONFIG_CPU_PM /**
Catalin Marinas
maintainerPATCH v5 4/25 arm64: mte: Add specific SIGSEGV codes
In reply to Catalin MarinasFrom: Vincenzo Frascino <[email protected]>
Add MTE-specific SIGSEGV codes to siginfo.h and update the x86 BUILD_BUG_ON(NSIGSEGV != 7) compile check.
---
Notes:
v3:
- Fixed the BUILD_BUG_ON(NSIGSEGV != 7) on x86
- Updated the commit log
v2:
- Dropped the #ifdef __aarch64__.
- Renumbered the SEGV_MTE* values to avoid clash with ADI.
arch/x86/kernel/signal_compat.c | 2 +- include/uapi/asm-generic/siginfo.h | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-)
arch/x86/kernel/signal_compat.c +1 -1
@@ -27,7 +27,7 @@ static inline void signal_compat_build_tests(void)*/ BUILD_BUG_ON (NSIGILL !=11 );BUILD_BUG_ON (NSIGFPE !=15 ); -BUILD_BUG_ON (NSIGSEGV !=7 ); +BUILD_BUG_ON (NSIGSEGV !=9 );BUILD_BUG_ON (NSIGBUS !=5 );BUILD_BUG_ON (NSIGTRAP !=5 );BUILD_BUG_ON (NSIGCHLD !=6 );
include/uapi/asm-generic/siginfo.h +3 -1
@@ -229,7 +229,9 @@ typedef struct siginfo {#define SEGV_ACCADI 5/* ADI not enabled for mapped object */ #define SEGV_ADIDERR 6/* Disrupting MCD error */ #define SEGV_ADIPERR 7/* Precise MCD exception */ -#define NSIGSEGV 7 +#define SEGV_MTEAERR 8/* Asynchronous ARM MTE error */ +#define SEGV_MTESERR 9/* Synchronous ARM MTE exception */ +#define NSIGSEGV 9/* * SIGBUS si_codes
Catalin Marinas
maintainerPATCH v5 5/25 arm64: mte: Handle synchronous and asynchronous tag check faults
In reply to Catalin MarinasFrom: Vincenzo Frascino <[email protected]>
The Memory Tagging Extension has two modes of notifying a tag check fault at EL0, configurable through the SCTLR_EL1.TCF0 field:
1. Synchronous raising of a Data Abort exception with DFSC 17.
2. Asynchronous setting of a cumulative bit in TFSRE0_EL1.
Add the exception handler for the synchronous exception and handling of the asynchronous TFSRE0_EL1.TF0 bit setting via a new TIF flag in do_notify_resume().
On a tag check failure in user-space, whether synchronous or asynchronous, a SIGSEGV will be raised on the faulting thread.
---
Notes:
v4:
- Use send_signal_fault() instead of fault_signal_inject() for asynchronous tag check faults as execution can continue even if this signal is masked.
- Add DSB ISH prior to writing TFSRE0_EL1 in the clear_mte_async_tcf macro.
- Move clear_mte_async_tcf just after returning to user since do_notify_resume() may still cause async tag faults via do_signal().
v3:
- Asynchronous tag check faults during the uaccess routines in the kernel are ignored.
- Fix check_mte_async_tcf calling site as it expects the first argument to be the thread flags.
- Move the mte_thread_switch() definition and call to a later patch as this became empty with the removal of async uaccess checking.
- Add dsb() and clearing of TFSRE0_EL1 in flush_mte_state(), in case execve() triggered a asynchronous tag check fault.
- Clear TIF_MTE_ASYNC_FAULT in arch_dup_task_struct() so that the child does not inherit any pending tag fault in the parent.
v2:
- Clear PSTATE.TCO on exception entry (automatically set by the hardware).
- On syscall entry, for asynchronous tag check faults from user space, generate the signal early via syscall restarting.
- Before context switch, save any potential async tag check fault generated by the kernel to the TIF flag (this follows an architecture update where the uaccess routines use the TCF0 mode).
- Moved the flush_mte_state() and mte_thread_switch() function to a new mte.c file.
arch/arm64/include/asm/mte.h | 23 +++++++++++++++++ arch/arm64/include/asm/thread_info.h | 4 ++- arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/entry.S | 37 ++++++++++++++++++++++++++++ arch/arm64/kernel/mte.c | 21 ++++++++++++++++ arch/arm64/kernel/process.c | 5 ++++ arch/arm64/kernel/signal.c | 8 ++++++ arch/arm64/kernel/syscall.c | 10 ++++++++ arch/arm64/mm/fault.c | 9 ++++++- 9 files changed, 116 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/include/asm/mte.h create mode 100644 arch/arm64/kernel/mte.c
arch/arm64/include/asm/mte.h new file +23 -0
@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (C) 2020 ARM Ltd. +*/ +#ifndef __ASM_MTE_H +#define __ASM_MTE_H + +#ifndef __ASSEMBLY__ + +#ifdef CONFIG_ARM64_MTE + +void flush_mte_state (void ); + +#else + +static inline void flush_mte_state (void ) +{ +} + +#endif + +#endif /* __ASSEMBLY__ */ +#endif /* __ASM_MTE_H */
arch/arm64/include/asm/thread_info.h +3 -1
@@ -67,6 +67,7 @@ void arch_release_task_struct(struct task_struct *tsk);#define TIF_FOREIGN_FPSTATE 3/* CPU's FP state is not current's */ #define TIF_UPROBE 4/* uprobe breakpoint or singlestep */ #define TIF_FSCHECK 5/* Check FS is USER_DS on return */ +#define TIF_MTE_ASYNC_FAULT 6/* MTE Asynchronous Tag Check Fault */ #define TIF_SYSCALL_TRACE 8/* syscall trace active */ #define TIF_SYSCALL_AUDIT 9/* syscall auditing */ #define TIF_SYSCALL_TRACEPOINT 10/* syscall tracepoint for ftrace */
@@ -95,10 +96,11 @@ void arch_release_task_struct(struct task_struct *tsk);#define _TIF_FSCHECK (1 << TIF_FSCHECK)#define _TIF_32BIT (1 << TIF_32BIT)#define _TIF_SVE (1 << TIF_SVE) +#define _TIF_MTE_ASYNC_FAULT (1 << TIF_MTE_ASYNC_FAULT)#define _TIF_WORK_MASK (_TIF_NEED_RESCHED | _TIF_SIGPENDING | \ _TIF_NOTIFY_RESUME | _TIF_FOREIGN_FPSTATE | \ - _TIF_UPROBE | _TIF_FSCHECK) + _TIF_UPROBE | _TIF_FSCHECK | _TIF_MTE_ASYNC_FAULT)#define _TIF_SYSCALL_WORK (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | \ _TIF_SYSCALL_TRACEPOINT | _TIF_SECCOMP | \
arch/arm64/kernel/Makefile +1 -0
@@ -64,6 +64,7 @@ obj-$(CONFIG_ARM_SDE_INTERFACE) += sdei.o obj-$(CONFIG_ARM64_SSBD) += ssbd.o obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o +obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso/ probes/ obj-$(CONFIG_COMPAT_VDSO) += vdso32/
arch/arm64/kernel/entry.S +37 -0
@@ -146,6 +146,32 @@ alternative_cb_end #endif .endm + /* Check for MTE asynchronous tag check faults */ + .macro check_mte_async_tcf, flgs, tmp +#ifdef CONFIG_ARM64_MTE +alternative_if_not ARM64_MTE + b 1f +alternative_else_nop_endif + mrs_s \tmp, SYS_TFSRE0_EL1 + tbz \tmp, #SYS_TFSR_EL1_TF0_SHIFT, 1f + /* Asynchronous TCF occurred for TTBR0 access, set the TI flag */ + orr \flgs, \flgs, #_TIF_MTE_ASYNC_FAULT + str \flgs, [tsk, #TSK_TI_FLAGS] + msr_s SYS_TFSRE0_EL1, xzr +1: +#endif + .endm + + /* Clear the MTE asynchronous tag check faults */ + .macro clear_mte_async_tcf +#ifdef CONFIG_ARM64_MTE +alternative_if ARM64_MTE + dsb ish + msr_s SYS_TFSRE0_EL1, xzr +alternative_else_nop_endif +#endif + .endm + .macro kernel_entry, el, regsize = 64 .if \regsize == 32 mov w0, w0 // zero upper 32 bits of x0
@@ -177,6 +203,8 @@ alternative_cb_end ldr x19, [tsk, #TSK_TI_FLAGS] disable_step_tsk x19, x20 + /* Check for asynchronous tag check faults in user space */ + check_mte_async_tcf x19, x22 apply_ssbd 1, x22, x23 ptrauth_keys_install_kernel tsk, x20, x22, x23
@@ -247,6 +275,13 @@ alternative_if ARM64_HAS_IRQ_PRIO_MASKING str x20, [sp, #S_PMR_SAVE] alternative_else_nop_endif + /* Re-enable tag checking (TCO set on exception entry) */ +#ifdef CONFIG_ARM64_MTE +alternative_if ARM64_MTE + SET_PSTATE_TCO(0) +alternative_else_nop_endif +#endif + /* * Registers that may be useful after this macro is invoked: *
@@ -755,6 +790,8 @@ SYM_CODE_START_LOCAL(ret_to_user) and x2, x1, #_TIF_WORK_MASK cbnz x2, work_pending finish_ret_to_user: + /* Ignore asynchronous tag check faults in the uaccess routines */ + clear_mte_async_tcf enable_step_tsk x1, x2 #ifdef CONFIG_GCC_PLUGIN_STACKLEAK bl stackleak_erase
arch/arm64/kernel/mte.c new file +21 -0
@@ -0,0 +1,21 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2020 ARM Ltd. +*/ + +#include <linux/thread_info.h> + +#include <asm/cpufeature.h> +#include <asm/mte.h> +#include <asm/sysreg.h> + +void flush_mte_state (void ) +{ +if (!system_supports_mte ()) +return ; + +/* clear any pending asynchronous tag fault */ +dsb (ish); +write_sysreg_s (0 ,SYS_TFSRE0_EL1 ); +clear_thread_flag (TIF_MTE_ASYNC_FAULT ); +}
arch/arm64/kernel/process.c +5 -0
@@ -52,6 +52,7 @@#include <asm/exec.h> #include <asm/fpsimd.h> #include <asm/mmu_context.h> +#include <asm/mte.h> #include <asm/processor.h> #include <asm/pointer_auth.h> #include <asm/stacktrace.h>
@@ -338,6 +339,7 @@ void flush_thread(void)tls_thread_flush ();flush_ptrace_hw_breakpoint (current);flush_tagged_addr_state (); +flush_mte_state (); }void release_thread (struct task_struct *dead_task)
@@ -370,6 +372,9 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) dst->thread.sve_state =NULL ;clear_tsk_thread_flag (dst,TIF_SVE ); +/* clear any pending asynchronous tag fault raised by the parent */ +clear_tsk_thread_flag (dst,TIF_MTE_ASYNC_FAULT ); +return 0 ; }
arch/arm64/kernel/signal.c +8 -0
@@ -748,6 +748,9 @@ static void setup_return(struct pt_regs *regs, struct k_sigaction *ka, regs->pstate |=PSR_BTYPE_C ; } +/* TCO (Tag Check Override) always cleared for signal handlers */ + regs->pstate &= ~PSR_TCO_BIT ; +if (ka->sa.sa_flags &SA_RESTORER ) sigtramp = ka->sa.sa_restorer;else
@@ -939,6 +942,11 @@ asmlinkage void do_notify_resume(struct pt_regs *regs,if (thread_flags & _TIF_UPROBE)uprobe_notify_resume (regs); +if (thread_flags & _TIF_MTE_ASYNC_FAULT) { +clear_thread_flag (TIF_MTE_ASYNC_FAULT ); +send_sig_fault (SIGSEGV ,SEGV_MTEAERR ,0 , current); + } +if (thread_flags & _TIF_SIGPENDING)do_signal (regs);
arch/arm64/kernel/syscall.c +10 -0
@@ -120,6 +120,16 @@ static void el0_svc_common(struct pt_regs *regs, int scno, int sc_nr,local_daif_restore (DAIF_PROCCTX );user_exit (); +if (system_supports_mte () && (flags & _TIF_MTE_ASYNC_FAULT)) { +/* +* Process the asynchronous tag check fault before the actual +* syscall. do_notify_resume() will send a signal to userspace +* before the syscall is restarted. +*/ + regs->regs[0 ] = -ERESTARTNOINTR ; +return ; + } +if (has_syscall_work (flags)) {/* set default errno for user-issued syscall(-1) */ if (scno ==NO_SYSCALL )
arch/arm64/mm/fault.c +8 -1
@@ -658,6 +658,13 @@ static int do_sea(unsigned long addr, unsigned int esr, struct pt_regs *regs)return 0 ; } +static int do_tag_check_fault (unsigned long addr,unsigned int esr, +struct pt_regs *regs) +{ +do_bad_area (addr, esr, regs); +return 0 ; +} +static const struct fault_info fault_info[] = { { do_bad,SIGKILL ,SI_KERNEL ,"ttbr address size fault" }, { do_bad,SIGKILL ,SI_KERNEL ,"level 1 address size fault" },
@@ -676,7 +683,7 @@ static const struct fault_info fault_info[] = { { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 2 permission fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 3 permission fault" }, { do_sea, SIGBUS, BUS_OBJERR, "synchronous external abort" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 17" }, + { do_tag_check_fault, SIGSEGV, SEGV_MTESERR, "synchronous tag check fault" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 18" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 19" }, { do_sea, SIGKILL, SI_KERNEL, "level 0 (translation table walk)" },
Catalin Marinas
maintainerPATCH v5 6/25 mm: Add PG_ARCH_2 page flag
In reply to Catalin MarinasFrom: Steven Price <[email protected]>
For arm64 MTE support it is necessary to be able to mark pages that contain user space visible tags that will need to be saved/restored e.g. when swapped out.
To support this add a new arch specific flag (PG_ARCH_2) that arch code can opt into using ARCH_USES_PG_ARCH_2.
---
Notes:
New in v4.
fs/proc/page.c | 3 +++ include/linux/kernel-page-flags.h | 1 + include/linux/page-flags.h | 3 +++ include/trace/events/mmflags.h | 9 ++++++++- mm/Kconfig | 3 +++ tools/vm/page-types.c | 2 ++ 6 files changed, 20 insertions(+), 1 deletion(-)
fs/proc/page.c +3 -0
@@ -217,6 +217,9 @@ u64 stable_page_flags(struct page *page) u |=kpf_copy_bit (k,KPF_PRIVATE_2 , PG_private_2); u |=kpf_copy_bit (k,KPF_OWNER_PRIVATE , PG_owner_priv_1); u |=kpf_copy_bit (k,KPF_ARCH , PG_arch_1); +#ifdef CONFIG_ARCH_USES_PG_ARCH_2 + u |=kpf_copy_bit (k,KPF_ARCH_2 , PG_arch_2); +#endif return u; };
include/linux/kernel-page-flags.h +1 -0
@@ -17,5 +17,6 @@#define KPF_ARCH 38#define KPF_UNCACHED 39#define KPF_SOFTDIRTY 40 +#define KPF_ARCH_2 41#endif /* LINUX_KERNEL_PAGE_FLAGS_H */
include/linux/page-flags.h +3 -0
@@ -135,6 +135,9 @@ enum pageflags {#if defined(CONFIG_IDLE_PAGE_TRACKING ) && defined(CONFIG_64BIT ) PG_young, PG_idle, +#endif +#ifdef CONFIG_ARCH_USES_PG_ARCH_2 + PG_arch_2, #endif __NR_PAGEFLAGS,
include/trace/events/mmflags.h +8 -1
@@ -79,6 +79,12 @@#define IF_HAVE_PG_IDLE (flag,string)#endif +#ifdef CONFIG_ARCH_USES_PG_ARCH_2 +#define IF_HAVE_PG_ARCH_2 (flag,string) ,{1UL << flag, string} +#else +#define IF_HAVE_PG_ARCH_2 (flag,string) +#endif +#define __def_pageflag_names \ {1UL << PG_locked, "locked" }, \ {1UL << PG_waiters, "waiters" }, \
@@ -105,7 +111,8 @@ IF_HAVE_PG_MLOCK(PG_mlocked, "mlocked" ) \ IF_HAVE_PG_UNCACHED(PG_uncached, "uncached" ) \ IF_HAVE_PG_HWPOISON(PG_hwpoison, "hwpoison" ) \ IF_HAVE_PG_IDLE(PG_young, "young" ) \ -IF_HAVE_PG_IDLE(PG_idle, "idle" ) +IF_HAVE_PG_IDLE(PG_idle, "idle" ) \ +IF_HAVE_PG_ARCH_2(PG_arch_2, "arch_2" )#define show_page_flags (flags) \ (flags) ? __print_flags(flags, "|", \
mm/Kconfig +3 -0
@@ -872,4 +872,7 @@ config ARCH_HAS_HUGEPD config MAPPING_DIRTY_HELPERS bool +config ARCH_USES_PG_ARCH_2 + bool + endmenu
tools/vm/page-types.c +2 -0
@@ -78,6 +78,7 @@#define KPF_ARCH 38#define KPF_UNCACHED 39#define KPF_SOFTDIRTY 40 +#define KPF_ARCH_2 41/* [48-] take some arbitrary free slots for expanding overloaded flags * not part of kernel API
@@ -135,6 +136,7 @@ static const char * const page_flag_names[] = { [KPF_ARCH] = "h:arch", [KPF_UNCACHED] = "c:uncached", [KPF_SOFTDIRTY] = "f:softdirty", + [KPF_ARCH_2] = "H:arch_2", [KPF_READAHEAD] = "I:readahead", [KPF_SLOB_FREE] = "P:slob_free",
Andrew Morton
maintainerRe: [PATCH v5 06/25] mm: Add PG_ARCH_2 page flag
Quoted 14 lines · On Wed, 24 Jun 2020 18:52:25 +0100 Catalin Marinas <[email protected]>…
From: Steven Price <[email protected]>
For arm64 MTE support it is necessary to be able to mark pages that contain user space visible tags that will need to be saved/restored e.g. when swapped out.
To support this add a new arch specific flag (
PG_ARCH_2) that arch code can opt into usingARCH_USES_PG_ARCH_2....
fs/proc/page.c u64 stable_page_flags(struct page *page) · line 222
Quoted 6 lines
fs/proc/page.c +3
-0@@ -217,6 +217,9 @@ u64 stable_page_flags(struct page *page) u |=kpf_copy_bit (k,KPF_PRIVATE_2 , PG_private_2); u |=kpf_copy_bit (k,KPF_OWNER_PRIVATE , PG_owner_priv_1); u |=kpf_copy_bit (k,KPF_ARCH , PG_arch_1); +#ifdef CONFIG_ARCH_USES_PG_ARCH_2 + u |=kpf_copy_bit (k,KPF_ARCH_2 , PG_arch_2); +#endif
Do we need CONFIG_ARCH_USES_PG_ARCH_2? What would be the downside to giving every architecture a PG_arch_2, but only arm64 uses it (at present)?
Re: [PATCH v5 06/25] mm: Add PG_ARCH_2 page flag
Quoted 13 lines · Andrew Morton
Quoted 11 lines · On Wed, 24 Jun 2020 18:52:25 +0100 Catalin Marinas <[email protected]>…
From: Steven Price <[email protected]>
For arm64 MTE support it is necessary to be able to mark pages that contain user space visible tags that will need to be saved/restored e.g. when swapped out.
To support this add a new arch specific flag (
PG_ARCH_2) that arch code can opt into usingARCH_USES_PG_ARCH_2....
Quoted 13 lines
fs/proc/page.c u64 stable_page_flags(struct page *page) · line 222
Quoted 9 lines
fs/proc/page.c +3
-0@@ -217,6 +217,9 @@ u64 stable_page_flags(struct page *page) u |=kpf_copy_bit (k,KPF_PRIVATE_2 , PG_private_2); u |=kpf_copy_bit (k,KPF_OWNER_PRIVATE , PG_owner_priv_1); u |=kpf_copy_bit (k,KPF_ARCH , PG_arch_1); +#ifdef CONFIG_ARCH_USES_PG_ARCH_2 + u |=kpf_copy_bit (k,KPF_ARCH_2 , PG_arch_2); +#endif Do we need
CONFIG_ARCH_USES_PG_ARCH_2? What would be the downside to giving every architecture aPG_arch_2, but only arm64 uses it (at present)?
32-bit architectures don't have space for it. We could condition it on CONFIG_64BIT instead.
Catalin Marinas
maintainerRe: [PATCH v5 06/25] mm: Add PG_ARCH_2 page flag
Quoted 27 lines · Matthew Wilcox
Quoted 12 lines · Andrew Morton
Quoted 11 lines · On Wed, 24 Jun 2020 18:52:25 +0100 Catalin Marinas <[email protected]>…
From: Steven Price <[email protected]>
For arm64 MTE support it is necessary to be able to mark pages that contain user space visible tags that will need to be saved/restored e.g. when swapped out.
To support this add a new arch specific flag (
PG_ARCH_2) that arch code can opt into usingARCH_USES_PG_ARCH_2....
Quoted 13 lines
fs/proc/page.c u64 stable_page_flags(struct page *page) · line 222
Quoted 9 lines
fs/proc/page.c +3
-0@@ -217,6 +217,9 @@ u64 stable_page_flags(struct page *page) u |=kpf_copy_bit (k,KPF_PRIVATE_2 , PG_private_2); u |=kpf_copy_bit (k,KPF_OWNER_PRIVATE , PG_owner_priv_1); u |=kpf_copy_bit (k,KPF_ARCH , PG_arch_1); +#ifdef CONFIG_ARCH_USES_PG_ARCH_2 + u |=kpf_copy_bit (k,KPF_ARCH_2 , PG_arch_2); +#endif Do we need
CONFIG_ARCH_USES_PG_ARCH_2? What would be the downside to giving every architecture aPG_arch_2, but only arm64 uses it (at present)?
Quoted 2 lines
32-bit architectures don't have space for it. We could condition it on
CONFIG_64BITinstead.
I'll this, though we'd still need some #ifdefs (OTOH, we get rid of the Kconfig entry).
Signature · 1 line
Catalin
Catalin Marinas
maintainerRe: [PATCH v5 06/25] mm: Add PG_ARCH_2 page flag
In reply to Andrew MortonQuoted 11 lines · Andrew Morton
Quoted 10 lines · On Wed, 24 Jun 2020 18:52:25 +0100 Catalin Marinas <[email protected]>…
From: Steven Price <[email protected]>
For arm64 MTE support it is necessary to be able to mark pages that contain user space visible tags that will need to be saved/restored e.g. when swapped out.To support this add a new arch specific flag (
PG_ARCH_2) that arch code can opt into usingARCH_USES_PG_ARCH_2....
Quoted 13 lines
fs/proc/page.c u64 stable_page_flags(struct page *page) · line 222
Quoted 9 lines
fs/proc/page.c +3
-0@@ -217,6 +217,9 @@ u64 stable_page_flags(struct page *page) u |=kpf_copy_bit (k,KPF_PRIVATE_2 , PG_private_2); u |=kpf_copy_bit (k,KPF_OWNER_PRIVATE , PG_owner_priv_1); u |=kpf_copy_bit (k,KPF_ARCH , PG_arch_1); +#ifdef CONFIG_ARCH_USES_PG_ARCH_2 + u |=kpf_copy_bit (k,KPF_ARCH_2 , PG_arch_2); +#endif Do we need
CONFIG_ARCH_USES_PG_ARCH_2? What would be the downside to giving every architecture aPG_arch_2, but only arm64 uses it (at present)?
It turns out we have another issue with this flag. PG_arch_2 in the arm64 MTE patches is used to mark a page as having valid tags. During set_pte_at(), if the mapping type is tagged, we set PG_arch_2 (also setting it in other cases like copy_page). In combination with THP and swap (and some stress-testing to force swap-out), the kernel ends up clearing PG_arch_2 in __split_huge_page_tail(), causing a subsequent set_pte_at() to zero valid tags stored by user.
The quick fix is to add an arch_huge_page_flags_split_preserve macro (need to think of a shorter name) which adds 1L << PG_arch_2 to the preserve list in the above mentioned function. However, I wonder whether it's safe to add both PG_arch_1 and PG_arch_2 to this list. At least on arm and arm64, PG_arch_1 is used to mark a page as D-cache clean (and don't need to do this again after splitting a pmd):
mm/huge_memory.c +4 -0
@@ -2364,6 +2364,10 @@ static void __split_huge_page_tail(struct page *head, int tail, (1L << PG_workingset) | (1L << PG_locked) | (1L << PG_unevictable) | + (1L << PG_arch_1) | +#ifdef CONFIG_64BIT + (1L << PG_arch_2) | +#endif (1L << PG_dirty)));/* ->mapping in first tail page is compound_mapcount */
Thanks.
Signature · 1 line
Catalin
Catalin Marinas
maintainerPATCH v5 7/25 arm64: mte: Clear the tags when a page is mapped in user-space with PROT_MTE
In reply to Catalin MarinasPages allocated by the kernel are not guaranteed to have the tags zeroed, especially as the kernel does not (yet) use MTE itself. To ensure the user can still access such pages when mapped into its address space, clear the tags via set_pte_at(). A new page flag - PG_mte_tagged (PG_arch_2) - is used to track pages with valid allocation tags.
Since the zero page is mapped as pte_special(), it won't be covered by the above set_pte_at() mechanism. Clear its tags during early MTE initialisation.
---
Notes:
v5:
- Fix the handling of compound pages. Previously, set_pte_at() could have erased already valid tags if the first page in a compound one did not have the PG_mte_tagged flag set.
- Move the multi_tag_transfer_size macro from assembler.h to mte.S.
- Ignore pte_special() mappings and clear the tags in the zero page separately (since it's mapped as a special pte).
- Clearing the tags of the zero page was moved to this patch from an earlier one since mte_clear_page_tags() was not available.
New in v4. Replacing a previous page zeroing the tags in clear_page().
arch/arm64/include/asm/mte.h | 16 +++++++++++++++ arch/arm64/include/asm/pgtable.h | 7 +++++++ arch/arm64/kernel/cpufeature.c | 7 +++++++ arch/arm64/kernel/mte.c | 14 +++++++++++++ arch/arm64/lib/Makefile | 2 ++ arch/arm64/lib/mte.S | 34 ++++++++++++++++++++++++++++++++ 6 files changed, 80 insertions(+) create mode 100644 arch/arm64/lib/mte.S
arch/arm64/include/asm/mte.h +16 -0
@@ -7,12 +7,28 @@#ifndef __ASSEMBLY__ +#include <linux/page-flags.h> + +#include <asm/pgtable-types.h> + +void mte_clear_page_tags (void *addr); +#ifdef CONFIG_ARM64_MTE +/* track which pages have valid allocation tags */ +#define PG_mte_tagged PG_arch_2 + +void mte_sync_tags (pte_t *ptep,pte_t pte);void flush_mte_state (void );#else +/* unused if !CONFIG_ARM64_MTE, silence the compiler */ +#define PG_mte_tagged 0 + +static inline void mte_sync_tags (pte_t *ptep,pte_t pte) +{ +}static inline void flush_mte_state (void ) { }
arch/arm64/include/asm/pgtable.h +7 -0
@@ -9,6 +9,7 @@#include <asm/proc-fns.h> #include <asm/memory.h> +#include <asm/mte.h> #include <asm/pgtable-hwdef.h> #include <asm/pgtable-prot.h> #include <asm/tlbflush.h>
@@ -80,6 +81,8 @@ extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)];#define pte_user_exec (pte) (!(pte_val(pte) & PTE_UXN))#define pte_cont (pte) (!!(pte_val(pte) & PTE_CONT))#define pte_devmap (pte) (!!(pte_val(pte) & PTE_DEVMAP)) +#define pte_tagged (pte) ((pte_val(pte) & PTE_ATTRINDX_MASK) == \ + PTE_ATTRINDX(MT_NORMAL_TAGGED))#define pte_cont_addr_end (addr, end) \ ({ unsigned long __boundary = ((addr) + CONT_PTE_SIZE) & CONT_PTE_MASK; \
@@ -274,6 +277,10 @@ static inline void set_pte_at(struct mm_struct *mm, unsigned long addr,if (pte_present (pte) &&pte_user_exec (pte) && !pte_special (pte))__sync_icache_dcache (pte); +if (system_supports_mte () && +pte_present (pte) &&pte_tagged (pte) && !pte_special (pte)) +mte_sync_tags (ptep, pte); +__check_racy_pte_update (mm, ptep, pte);set_pte (ptep, pte);
arch/arm64/kernel/cpufeature.c +7 -0
@@ -75,6 +75,7 @@#include <asm/cpu_ops.h> #include <asm/fpsimd.h> #include <asm/mmu_context.h> +#include <asm/mte.h> #include <asm/processor.h> #include <asm/sysreg.h> #include <asm/traps.h>
@@ -1688,6 +1689,12 @@ static int __init system_enable_mte(void)/* Ensure the TLB does not have stale MAIR attributes */ flush_tlb_all (); +/* +* Clear the tags in the zero page. This needs to be done via the +* linear map which has the Tagged attribute. +*/ +mte_clear_page_tags (lm_alias (empty_zero_page)); +return 0 ; }core_initcall (system_enable_mte);
arch/arm64/kernel/mte.c +14 -0
@@ -3,12 +3,26 @@* Copyright (C) 2020 ARM Ltd. */ +#include <linux/bitops.h> +#include <linux/mm.h> #include <linux/thread_info.h> #include <asm/cpufeature.h> #include <asm/mte.h> #include <asm/sysreg.h> +void mte_sync_tags (pte_t *ptep,pte_t pte) +{ +struct page *page =pte_page (pte); +long i, nr_pages =compound_nr (page); + +/* if PG_mte_tagged is set, tags have already been initialised */ +for (i =0 ; i < nr_pages; i++, page++) { +if (!test_and_set_bit (PG_mte_tagged, &page->flags)) +mte_clear_page_tags (page_address (page)); + } +} +void flush_mte_state (void ) {if (!system_supports_mte ())
arch/arm64/lib/Makefile +2 -0
@@ -16,3 +16,5 @@ lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o obj-$(CONFIG_CRC32) += crc32.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o + +obj-$(CONFIG_ARM64_MTE) += mte.o
arch/arm64/lib/mte.S new file +34 -0
@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2020 ARM Ltd. + */ +#include <linux/linkage.h> + +#include <asm/assembler.h> +#include <asm/sysreg.h> + + .arch armv8.5-a+memtag + +/* + * multitag_transfer_size - set \reg to the block size that is accessed by the + * LDGM/STGM instructions. + */ + .macro multitag_transfer_size, reg, tmp + mrs_s \reg, SYS_GMID_EL1 + ubfx \reg, \reg, #SYS_GMID_EL1_BS_SHIFT, #SYS_GMID_EL1_BS_SIZE + mov \tmp, #4 + lsl \reg, \tmp, \reg + .endm + +/* + * Clear the tags in a page + * x0 - address of the page to be cleared + */ +SYM_FUNC_START(mte_clear_page_tags) + multitag_transfer_size x1, x2 +1: stgm xzr, [x0] + add x0, x0, x1 + tst x0, #(PAGE_SIZE - 1) + b.ne 1b + ret +SYM_FUNC_END(mte_clear_page_tags)
Catalin Marinas
maintainerPATCH v5 8/25 arm64: mte: Tags-aware copy_{user_,}highpage() implementations
In reply to Catalin MarinasFrom: Vincenzo Frascino <[email protected]>
When the Memory Tagging Extension is enabled, the tags need to be preserved across page copy (e.g. for copy-on-write, page migration).
Introduce MTE-aware copy_{user_,}highpage() functions to copy tags to the destination if the source page has the PG_mte_tagged flag set. copy_user_page() does not need to handle tag copying since, with this patch, it is only called by the DAX code where there is no source page structure (and no source tags).
---
Notes:
v5:
- Handle tags in copy_highpage() (previously only copy_user_highpage()).
- Ignore tags in copy_user_page() since it is only called directly by the DAX code where there is no source page structure.
- Fix missing ret in mte_copy_page_tags().
v4:
- Moved the tag copying to a separate function in mte.S and only called if the source page has the PG_mte_tagged flag set.
arch/arm64/include/asm/mte.h | 4 ++++ arch/arm64/include/asm/page.h | 14 +++++++++++--- arch/arm64/lib/mte.S | 19 +++++++++++++++++++ arch/arm64/mm/copypage.c | 25 +++++++++++++++++++++---- 4 files changed, 55 insertions(+), 7 deletions(-)
arch/arm64/include/asm/mte.h +4 -0
@@ -19,6 +19,7 @@ void mte_clear_page_tags(void *addr);#define PG_mte_tagged PG_arch_2void mte_sync_tags (pte_t *ptep,pte_t pte); +void mte_copy_page_tags (void *kto,const void *kfrom);void flush_mte_state (void );#else
@@ -29,6 +30,9 @@ void flush_mte_state(void);static inline void mte_sync_tags (pte_t *ptep,pte_t pte) { } +static inline void mte_copy_page_tags (void *kto,const void *kfrom) +{ +}static inline void flush_mte_state (void ) { }
arch/arm64/include/asm/page.h +11 -3
@@ -15,18 +15,26 @@#include <linux/personality.h> /* for READ_IMPLIES_EXEC */ #include <asm/pgtable-types.h> +struct page ; +struct vm_area_struct ; +extern void __cpu_clear_user_page (void *p,unsigned long user); -extern void __cpu_copy_user_page (void *to,const void *from, -unsigned long user);extern void copy_page (void *to,const void *from);extern void clear_page (void *to); +void copy_user_highpage (struct page *to,struct page *from, +unsigned long vaddr,struct vm_area_struct *vma); +#define __HAVE_ARCH_COPY_USER_HIGHPAGE + +void copy_highpage (struct page *to,struct page *from); +#define __HAVE_ARCH_COPY_HIGHPAGE +#define __alloc_zeroed_user_highpage (movableflags, vma, vaddr) \ alloc_page_vma(GFP_HIGHUSER | __GFP_ZERO | movableflags, vma, vaddr)#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE#define clear_user_page (addr,vaddr,pg) __cpu_clear_user_page(addr, vaddr) -#define copy_user_page (to,from,vaddr,pg) __cpu_copy_user_page(to, from, vaddr) +#define copy_user_page (to, from, vaddr, pg) copy_page(to, from)typedef struct page *pgtable_t ;
arch/arm64/lib/mte.S +19 -0
@@ -5,6 +5,7 @@ #include <linux/linkage.h> #include <asm/assembler.h> +#include <asm/page.h> #include <asm/sysreg.h> .arch armv8.5-a+memtag
@@ -32,3 +33,21 @@ SYM_FUNC_START(mte_clear_page_tags) b.ne 1b ret SYM_FUNC_END(mte_clear_page_tags) + +/* + * Copy the tags from the source page to the destination one + * x0 - address of the destination page + * x1 - address of the source page + */ +SYM_FUNC_START(mte_copy_page_tags) + mov x2, x0 + mov x3, x1 + multitag_transfer_size x5, x6 +1: ldgm x4, [x3] + stgm x4, [x2] + add x2, x2, x5 + add x3, x3, x5 + tst x2, #(PAGE_SIZE - 1) + b.ne 1b + ret +SYM_FUNC_END(mte_copy_page_tags)
arch/arm64/mm/copypage.c +21 -4
@@ -6,18 +6,35 @@* Copyright (C) 2012 ARM Ltd. */ +#include <linux/bitops.h> #include <linux/mm.h> #include <asm/page.h> #include <asm/cacheflush.h> +#include <asm/cpufeature.h> +#include <asm/mte.h> -void __cpu_copy_user_page (void *kto,const void *kfrom,unsigned long vaddr) +void copy_highpage (struct page *to,struct page *from) { -struct page *page =virt_to_page (kto); +struct page *kto =page_address (to); +struct page *kfrom =page_address (from); +copy_page (kto, kfrom); -flush_dcache_page (page); + +if (system_supports_mte () &&test_bit (PG_mte_tagged, &from->flags)) { +set_bit (PG_mte_tagged, &to->flags); +mte_copy_page_tags (kto, kfrom); + } +} +EXPORT_SYMBOL (copy_highpage); + +void copy_user_highpage (struct page *to,struct page *from, +unsigned long vaddr,struct vm_area_struct *vma) +{ +copy_highpage (to, from); +flush_dcache_page (to); } -EXPORT_SYMBOL_GPL (__cpu_copy_user_page); +EXPORT_SYMBOL_GPL (copy_user_highpage);void __cpu_clear_user_page (void *kaddr,unsigned long vaddr) {
Catalin Marinas
maintainerPATCH v5 9/25 arm64: Avoid unnecessary clear_user_page() indirection
In reply to Catalin MarinasSince clear_user_page() calls clear_page() directly, avoid the unnecessary indirection.
---
Notes:
New in v5.
arch/arm64/include/asm/page.h | 3 +-- arch/arm64/mm/copypage.c | 6 ------ 2 files changed, 1 insertion(+), 8 deletions(-)
arch/arm64/include/asm/page.h +1 -2
@@ -18,7 +18,6 @@struct page ;struct vm_area_struct ; -extern void __cpu_clear_user_page (void *p,unsigned long user);extern void copy_page (void *to,const void *from);extern void clear_page (void *to);
@@ -33,7 +32,7 @@ void copy_highpage(struct page *to, struct page *from); alloc_page_vma(GFP_HIGHUSER | __GFP_ZERO | movableflags, vma, vaddr)#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE -#define clear_user_page (addr,vaddr,pg) __cpu_clear_user_page(addr, vaddr) +#define clear_user_page (page, vaddr, pg) clear_page(page)#define copy_user_page (to, from, vaddr, pg) copy_page(to, from)typedef struct page *pgtable_t ;
arch/arm64/mm/copypage.c +0 -6
@@ -35,9 +35,3 @@ void copy_user_highpage(struct page *to, struct page *from,flush_dcache_page (to); }EXPORT_SYMBOL_GPL (copy_user_highpage); - -void __cpu_clear_user_page (void *kaddr,unsigned long vaddr) -{ -clear_page (kaddr); -} -EXPORT_SYMBOL_GPL (__cpu_clear_user_page);
Catalin Marinas
maintainerPATCH v5 10/25 arm64: mte: Tags-aware aware memcmp_pages() implementation
In reply to Catalin MarinasWhen the Memory Tagging Extension is enabled, two pages are identical only if both their data and tags are identical.
Make the generic memcmp_pages() a __weak function and add an arm64-specific implementation which returns non-zero if any of the two pages contain valid MTE tags (PG_mte_tagged set). There isn't much benefit in comparing the tags of two pages since these are normally used for heap allocations and likely to differ anyway.
---
Notes:
v4:
- Remove page tag comparison. This is not very useful to detect identical pages as long as set_pte_at() can zero the tags on a page without copy-on-write if mapped with PROT_MTE. This can be improved if a real case appears but it's unlikely for heap pages to be identical across multiple processes.
- Move the memcmp_pages() function to mte.c.
arch/arm64/kernel/mte.c | 26 ++++++++++++++++++++++++++ mm/util.c | 2 +- 2 files changed, 27 insertions(+), 1 deletion(-)
arch/arm64/kernel/mte.c +26 -0
@@ -5,6 +5,7 @@#include <linux/bitops.h> #include <linux/mm.h> +#include <linux/string.h> #include <linux/thread_info.h> #include <asm/cpufeature.h>
@@ -23,6 +24,31 @@ void mte_sync_tags(pte_t *ptep, pte_t pte) } } +int memcmp_pages (struct page *page1,struct page *page2) +{ +char *addr1, *addr2; +int ret; + + addr1 =page_address (page1); + addr2 =page_address (page2); + ret =memcmp (addr1, addr2,PAGE_SIZE ); + +if (!system_supports_mte () || ret) +return ret; + +/* +* If the page content is identical but at least one of the pages is +* tagged, return non-zero to avoid KSM merging. If only one of the +* pages is tagged, set_pte_at() may zero or change the tags of the +* other page via mte_sync_tags(). +*/ +if (test_bit (PG_mte_tagged, &page1->flags) || +test_bit (PG_mte_tagged, &page2->flags)) +return addr1 != addr2; + +return ret; +} +void flush_mte_state (void ) {if (!system_supports_mte ())
mm/util.c +1 -1
@@ -911,7 +911,7 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen)return res; } -int memcmp_pages (struct page *page1,struct page *page2) +int __weak memcmp_pages (struct page *page1,struct page *page2) {char *addr1, *addr2;int ret;
Catalin Marinas
maintainerPATCH v5 11/25 mm: Introduce arch_calc_vm_flag_bits()
In reply to Catalin MarinasFrom: Kevin Brodsky <[email protected]>
Similarly to arch_calc_vm_prot_bits(), introduce a dummy arch_calc_vm_flag_bits() invoked from calc_vm_flag_bits(). This macro can be overridden by architectures to insert specific VM_* flags derived from the mmap() MAP_* flags.
---
Notes:
v2:
- Updated the comment above arch_calc_vm_prot_bits().
- Changed author since this patch had already been posted (internally).
include/linux/mman.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
include/linux/mman.h +7 -2
@@ -74,13 +74,17 @@ static inline void vm_unacct_memory(long pages) }/* -* Allow architectures to handle additional protection bits +* Allow architectures to handle additional protection and flag bits */ #ifndef arch_calc_vm_prot_bits#define arch_calc_vm_prot_bits (prot, pkey) 0#endif +#ifndef arch_calc_vm_flag_bits +#define arch_calc_vm_flag_bits (flags) 0 +#endif +#ifndef arch_vm_get_page_prot#define arch_vm_get_page_prot (vm_flags) __pgprot(0)#endif
@@ -131,7 +135,8 @@ calc_vm_flag_bits(unsigned long flags)return _calc_vm_trans (flags,MAP_GROWSDOWN ,VM_GROWSDOWN ) |_calc_vm_trans (flags,MAP_DENYWRITE ,VM_DENYWRITE ) |_calc_vm_trans (flags,MAP_LOCKED ,VM_LOCKED ) | -_calc_vm_trans (flags,MAP_SYNC ,VM_SYNC ); +_calc_vm_trans (flags,MAP_SYNC ,VM_SYNC ) | +arch_calc_vm_flag_bits (flags); }unsigned long vm_commit_limit (void );
Andrew Morton
maintainerRe: [PATCH v5 11/25] mm: Introduce arch_calc_vm_flag_bits()
Quoted 16 lines · On Wed, 24 Jun 2020 18:52:30 +0100 Catalin Marinas <[email protected]>…
From: Kevin Brodsky <[email protected]>
Similarly to
arch_calc_vm_prot_bits(), introduce a dummyarch_calc_vm_flag_bits()invoked fromcalc_vm_flag_bits(). This macro can be overridden by architectures to insert specific VM_* flags derived from themmap()MAP_* flags....
include/linux/mman.h +0
-1@@ -74,13 +74,17 @@ static inline void vm_unacct_memory(long pages) }/* -* Allow architectures to handle additional protection bits
include/linux/mman.h static inline void vm_unacct_memory(long pages) · line 86
Quoted 10 lines
include/linux/mman.h +4
-0+* Allow architectures to handle additional protection and flag bits */ #ifndef arch_calc_vm_prot_bits#define arch_calc_vm_prot_bits (prot, pkey) 0#endif +#ifndef arch_calc_vm_flag_bits +#define arch_calc_vm_flag_bits (flags) 0 +#endif
It would be helpful to add a comment specifying which arch header file is responsible for defining arch_calc_vm_flag_bits. Because in the past we've messed this sort of thing up and had different architectures define things in different header files, resulting in build issues as code evolves.
Quoted 13 lines
include/linux/mman.h +2
-1#ifndef arch_vm_get_page_prot#define arch_vm_get_page_prot (vm_flags) __pgprot(0)#endif @@ -131,7 +135,8 @@ calc_vm_flag_bits(unsigned long flags)return _calc_vm_trans (flags,MAP_GROWSDOWN ,VM_GROWSDOWN ) |_calc_vm_trans (flags,MAP_DENYWRITE ,VM_DENYWRITE ) |_calc_vm_trans (flags,MAP_LOCKED ,VM_LOCKED ) | -_calc_vm_trans (flags,MAP_SYNC ,VM_SYNC ); +_calc_vm_trans (flags,MAP_SYNC ,VM_SYNC ) | +arch_calc_vm_flag_bits (flags); }unsigned long vm_commit_limit (void );
Catalin Marinas
maintainerRe: [PATCH v5 11/25] mm: Introduce arch_calc_vm_flag_bits()
Quoted 23 lines · Andrew Morton
Quoted 22 lines · On Wed, 24 Jun 2020 18:52:30 +0100 Catalin Marinas <[email protected]>…
From: Kevin Brodsky <[email protected]>
Similarly toarch_calc_vm_prot_bits(), introduce a dummyarch_calc_vm_flag_bits()invoked fromcalc_vm_flag_bits(). This macro can be overridden by architectures to insert specific VM_* flags derived from themmap()MAP_* flags....
include/linux/mman.h +1
-1@@ -74,13 +74,17 @@ static inline void vm_unacct_memory(long pages) }/* -* Allow architectures to handle additional protection bits +* Allow architectures to handle additional protection and flag bits */ #ifndef arch_calc_vm_prot_bits#define arch_calc_vm_prot_bits (prot, pkey) 0#endif
Quoted 9 lines
include/linux/mman.h static inline void vm_unacct_memory(long pages) · line 86
Quoted 3 lines
include/linux/mman.h +3
-0+#ifndef arch_calc_vm_flag_bits +#define arch_calc_vm_flag_bits (flags) 0 +#endif It would be helpful to add a comment specifying which arch header file is responsible for defining
arch_calc_vm_flag_bits. Because in the past we've messed this sort of thing up and had different architectures define things in different header files, resulting in build issues as code evolves.
I'll add a comment that the overriding definitions need to go in the arch asm/mman.h file.
Thanks.
Signature · 1 line
Catalin
Catalin Marinas
maintainerPATCH v5 12/25 arm64: mte: Add PROT_MTE support to mmap() and mprotect()
In reply to Catalin MarinasTo enable tagging on a memory range, the user must explicitly opt in via a new PROT_MTE flag passed to mmap() or mprotect(). Since this is a new memory type in the AttrIndx field of a pte, simplify the or'ing of these bits over the protection_map[] attributes by making MT_NORMAL index 0.
There are two conditions for arch_vm_get_page_prot() to return the MT_NORMAL_TAGGED memory type: (1) the user requested it via PROT_MTE, registered as VM_MTE in the vm_flags, and (2) the vma supports MTE, decided during the mmap() call (only) and registered as VM_MTE_ALLOWED.
arch_calc_vm_prot_bits() is responsible for registering the user request as VM_MTE. The newly introduced arch_calc_vm_flag_bits() sets VM_MTE_ALLOWED if the mapping is MAP_ANONYMOUS. An MTE-capable filesystem (RAM-based) may be able to set VM_MTE_ALLOWED during its mmap() file ops call.
In addition, update VM_DATA_DEFAULT_FLAGS to allow mprotect(PROT_MTE) on stack or brk area.
The Linux mmap() syscall currently ignores unknown PROT_* flags. In the presence of MTE, an mmap(PROT_MTE) on a file which does not support MTE will not report an error and the memory will not be mapped as Normal Tagged. For consistency, mprotect(PROT_MTE) will not report an error either if the memory range does not support MTE. Two subsequent patches in the series will propose tightening of this behaviour.
---
Notes:
v2:
- Add VM_MTE_ALLOWED to show_smap_vma_flags().
arch/arm64/include/asm/memory.h | 18 +++++++----- arch/arm64/include/asm/mman.h | 44 ++++++++++++++++++++++++++++-- arch/arm64/include/asm/page.h | 2 +- arch/arm64/include/asm/pgtable.h | 7 ++++- arch/arm64/include/uapi/asm/mman.h | 1 + fs/proc/task_mmu.c | 4 +++ include/linux/mm.h | 8 ++++++ 7 files changed, 72 insertions(+), 12 deletions(-)
arch/arm64/include/asm/memory.h +11 -7
@@ -129,14 +129,18 @@/* * Memory types available. +* +* IMPORTANT: MT_NORMAL must be index 0 since vm_get_page_prot() may 'or' in +* the MT_NORMAL_TAGGED memory type for PROT_MTE mappings. Note +* that protection_map[] only contains MT_NORMAL attributes. */ -#define MT_DEVICE_nGnRnE 0 -#define MT_DEVICE_nGnRE 1 -#define MT_DEVICE_GRE 2 -#define MT_NORMAL_NC 3 -#define MT_NORMAL 4 -#define MT_NORMAL_WT 5 -#define MT_NORMAL_TAGGED 6 +#define MT_NORMAL 0 +#define MT_NORMAL_TAGGED 1 +#define MT_NORMAL_NC 2 +#define MT_NORMAL_WT 3 +#define MT_DEVICE_nGnRnE 4 +#define MT_DEVICE_nGnRE 5 +#define MT_DEVICE_GRE 6/* * Memory types for Stage-2 translation
arch/arm64/include/asm/mman.h +41 -3
63 lines not on this page · raw message
arch/arm64/include/asm/page.h +1 -1
@@ -43,7 +43,7 @@ extern int pfn_valid(unsigned long);#endif /* !__ASSEMBLY__ */ -#define VM_DATA_DEFAULT_FLAGS VM_DATA_FLAGS_TSK_EXEC +#define VM_DATA_DEFAULT_FLAGS (VM_DATA_FLAGS_TSK_EXEC | VM_MTE_ALLOWED)#include <asm-generic/getorder.h>
arch/arm64/include/asm/pgtable.h +6 -1
@@ -671,8 +671,13 @@ static inline unsigned long p4d_page_vaddr(p4d_t p4d)static inline pte_t pte_modify (pte_t pte,pgprot_t newprot) { +/* +* Normal and Normal-Tagged are two different memory types and indices +* in MAIR_EL1. The mask below has to include PTE_ATTRINDX_MASK. +*/ const pteval_t mask =PTE_USER |PTE_PXN |PTE_UXN |PTE_RDONLY | -PTE_PROT_NONE |PTE_VALID |PTE_WRITE |PTE_GP ; +PTE_PROT_NONE |PTE_VALID |PTE_WRITE |PTE_GP | +PTE_ATTRINDX_MASK ;/* preserve the hardware dirty information */ if (pte_hw_dirty (pte)) pte =pte_mkdirty (pte);
arch/arm64/include/uapi/asm/mman.h +1 -0
@@ -5,5 +5,6 @@#include <asm-generic/mman.h> #define PROT_BTI 0x10/* BTI guarded page */ +#define PROT_MTE 0x20/* Normal Tagged mapping */ #endif /* ! _UAPI__ASM_MMAN_H */
fs/proc/task_mmu.c +4 -0
@@ -653,6 +653,10 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) [ilog2(VM_MERGEABLE)] = "mg", [ilog2(VM_UFFD_MISSING)]= "um", [ilog2 (VM_UFFD_WP )] ="uw" , +#ifdef CONFIG_ARM64_MTE + [ilog2(VM_MTE)] = "mt", + [ilog2 (VM_MTE_ALLOWED )] ="" , +#endif #ifdef CONFIG_ARCH_HAS_PKEYS/* These come out via ProtectionKey: */ [ilog2(VM_PKEY_BIT0)] = "",
include/linux/mm.h +8 -0
@@ -333,6 +333,14 @@ extern unsigned int kobjsize(const void *objp);# define VM_MAPPED_COPY VM_ARCH_1/* T if mapped copy of data (nommu mmap) */ #endif +#if defined(CONFIG_ARM64_MTE ) +# define VM_MTE VM_HIGH_ARCH_0/* Use Tagged memory for access control */ +# define VM_MTE_ALLOWED VM_HIGH_ARCH_1/* Tagged memory permitted */ +#else +# define VM_MTE VM_NONE +# define VM_MTE_ALLOWED VM_NONE +#endif +#ifndef VM_GROWSUP # define VM_GROWSUP VM_NONE#endif
Catalin Marinas
maintainerPATCH v5 13/25 mm: Introduce arch_validate_flags()
In reply to Catalin MarinasSimilarly to arch_validate_prot() called from do_mprotect_pkey(), an architecture may need to sanity-check the new vm_flags.
Define a dummy function always returning true. In addition to do_mprotect_pkey(), also invoke it from mmap_region() prior to updating vma->vm_page_prot to allow the architecture code to veto potentially inconsistent vm_flags.
---
Notes:
v2:
- Some comments updated.
include/linux/mman.h | 13 +++++++++++++ mm/mmap.c | 9 +++++++++ mm/mprotect.c | 6 ++++++ 3 files changed, 28 insertions(+)
include/linux/mman.h +13 -0
@@ -103,6 +103,19 @@ static inline bool arch_validate_prot(unsigned long prot, unsigned long addr)#define arch_validate_prot arch_validate_prot#endif +#ifndef arch_validate_flags +/* +* This is called from mmap() and mprotect() with the updated vma->vm_flags. +* +* Returns true if the VM_* flags are valid. +*/ +static inline bool arch_validate_flags (unsigned long flags) +{ +return true; +} +#define arch_validate_flags arch_validate_flags +#endif +/* * Optimisation macro. It is equivalent to: * (x & bit1) ? bit2 : 0
mm/mmap.c +9 -0
@@ -1792,6 +1792,15 @@ unsigned long mmap_region(struct file *file, unsigned long addr,vma_set_anonymous (vma); } +/* Allow architectures to sanity-check the vm_flags */ +if (!arch_validate_flags (vma->vm_flags)) { + error = -EINVAL ; +if (file) + goto unmap_and_free_vma; +else + goto free_vma; + } +vma_link (mm, vma, prev, rb_link, rb_parent);/* Once vma denies write, undo our temporary denial count */ if (file) {
mm/mprotect.c +6 -0
@@ -603,6 +603,12 @@ static int do_mprotect_pkey(unsigned long start, size_t len, goto out; } +/* Allow architectures to sanity-check the new flags */ +if (!arch_validate_flags (newflags)) { + error = -EINVAL ; + goto out; + } + error =security_file_mprotect (vma, reqprot, prot);if (error) goto out;
Andrew Morton
maintainerRe: [PATCH v5 13/25] mm: Introduce arch_validate_flags()
Quoted 18 lines · On Wed, 24 Jun 2020 18:52:32 +0100 Catalin Marinas <[email protected]>…
Similarly to
arch_validate_prot()called fromdo_mprotect_pkey(), an architecture may need to sanity-check the newvm_flags.Define a dummy function always returning true. In addition to
do_mprotect_pkey(), also invoke it frommmap_region()prior to updatingvma->vm_page_protto allow the architecture code to veto potentially inconsistentvm_flags....
include/linux/mman.h +2
-0@@ -103,6 +103,19 @@ static inline bool arch_validate_prot(unsigned long prot, unsigned long addr)#define arch_validate_prot arch_validate_prot#endif +#ifndef arch_validate_flags +/*
include/linux/mman.h static inline bool arch_validate_prot(unsigned long prot, unsigned long addr) · line 117
Quoted 10 lines
include/linux/mman.h +10
-0+* This is called from mmap() and mprotect() with the updated vma->vm_flags. +* +* Returns true if the VM_* flags are valid. +*/ +static inline bool arch_validate_flags (unsigned long flags) +{ +return true; +} +#define arch_validate_flags arch_validate_flags +#endif
Again, please let's nail down where the arch should define this.
Otherwise, Acked-by: Andrew Morton <[email protected]>
Catalin Marinas
maintainerPATCH v5 14/25 arm64: mte: Validate the PROT_MTE request via arch_validate_flags()
In reply to Catalin MarinasMake use of the newly introduced arch_validate_flags() hook to sanity-check the PROT_MTE request passed to mmap() and mprotect(). If the mapping does not support MTE, these syscalls will return -EINVAL.
--- arch/arm64/include/asm/mman.h | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
arch/arm64/include/asm/mman.h +13 -1
@@ -49,8 +49,10 @@ static inline pgprot_t arch_vm_get_page_prot(unsigned long vm_flags)* register (1) as VM_MTE in the vma->vm_flags and (2) as * VM_MTE_ALLOWED. Note that the latter can only be set during the * mmap() call since mprotect() does not accept MAP_* flags. +* Checking for VM_MTE only is sufficient since arch_validate_flags() +* does not permit (VM_MTE & !VM_MTE_ALLOWED). */ -if ((vm_flags &VM_MTE ) && (vm_flags &VM_MTE_ALLOWED )) +if (vm_flags &VM_MTE ) prot |=PTE_ATTRINDX (MT_NORMAL_TAGGED );return __pgprot (prot);
@@ -72,4 +74,14 @@ static inline bool arch_validate_prot(unsigned long prot, }#define arch_validate_prot (prot, addr) arch_validate_prot(prot, addr) +static inline bool arch_validate_flags (unsigned long vm_flags) +{ +if (!system_supports_mte ()) +return true; + +/* only allow VM_MTE if VM_MTE_ALLOWED has been set previously */ +return !(vm_flags &VM_MTE ) || (vm_flags &VM_MTE_ALLOWED ); +} +#define arch_validate_flags (vm_flags) arch_validate_flags(vm_flags) +#endif /* ! __ASM_MMAN_H__ */
Catalin Marinas
maintainerPATCH v5 15/25 mm: Allow arm64 mmap(PROT_MTE) on RAM-based files
In reply to Catalin MarinasSince arm64 memory (allocation) tags can only be stored in RAM, mapping files with PROT_MTE is not allowed by default. RAM-based files like those in a tmpfs mount or memfd_create() can support memory tagging, so update the vm_flags accordingly in shmem_mmap().
--- mm/shmem.c | 3 +++ 1 file changed, 3 insertions(+)
mm/shmem.c +3 -0
@@ -2206,6 +2206,9 @@ static int shmem_mmap(struct file *file, struct vm_area_struct *vma) vma->vm_flags &= ~(VM_MAYWRITE ); } +/* arm64 - allow memory tagging on RAM-based files */ + vma->vm_flags |=VM_MTE_ALLOWED ; +file_accessed (file); vma->vm_ops = &shmem_vm_ops; if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
Re: [PATCH v5 15/25] mm: Allow arm64 mmap(PROT_MTE) on RAM-based files
Quoted 10 lines · On Wed, 24 Jun 2020 18:52:34 +0100 Catalin Marinas <[email protected]>…
Since arm64 memory (allocation) tags can only be stored in RAM, mapping files with
PROT_MTEis not allowed by default. RAM-based files like those in a tmpfs mount ormemfd_create()can support memory tagging, so update thevm_flagsaccordingly inshmem_mmap()....
mm/shmem.c static int shmem_mmap(struct file *file, struct vm_area_struct *vma) · line 2214
Quoted 9 lines
mm/shmem.c +3
-0@@ -2206,6 +2206,9 @@ static int shmem_mmap(struct file *file, struct vm_area_struct *vma) vma->vm_flags &= ~(VM_MAYWRITE ); } +/* arm64 - allow memory tagging on RAM-based files */ + vma->vm_flags |=VM_MTE_ALLOWED ; +file_accessed (file); vma->vm_ops = &shmem_vm_ops; if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
Catalin Marinas
maintainerPATCH v5 16/25 arm64: mte: Allow user control of the tag check mode via prctl()
In reply to Catalin MarinasBy default, even if PROT_MTE is set on a memory range, there is no tag check fault reporting (SIGSEGV). Introduce a set of option to the exiting prctl(PR_SET_TAGGED_ADDR_CTRL) to allow user control of the tag check fault mode:
PR_MTE_TCF_NONE - no reporting (default) PR_MTE_TCF_SYNC - synchronous tag check fault reporting PR_MTE_TCF_ASYNC - asynchronous tag check fault reporting
These options translate into the corresponding SCTLR_EL1.TCF0 bitfield, context-switched by the kernel. Note that uaccess done by the kernel is not checked and cannot be configured by the user.
---
Notes:
v3:
- Use SCTLR_EL1_TCF0_NONE instead of 0 for consistency.
- Move mte_thread_switch() in this patch from an earlier one. In addition, it is called after the dsb() in __switch_to() so that any asynchronous tag check faults have been registered in the TFSR_EL1 registers (to be added with the in-kernel MTE support.
v2:
- Handle SCTLR_EL1_TCF0_NONE explicitly for consistency with PR_MTE_TCF_NONE.
- Fix SCTLR_EL1 register setting in flush_mte_state() (thanks to Peter Collingbourne).
- Added ISB to update_sctlr_el1_tcf0() since, with the latest architecture update/fix, the TCF0 field is used by the uaccess routines.
arch/arm64/include/asm/mte.h | 14 ++++++ arch/arm64/include/asm/processor.h | 3 ++ arch/arm64/kernel/mte.c | 77 ++++++++++++++++++++++++++++++ arch/arm64/kernel/process.c | 26 ++++++++-- include/uapi/linux/prctl.h | 6 +++ 5 files changed, 123 insertions(+), 3 deletions(-)
arch/arm64/include/asm/mte.h +14 -0
@@ -21,6 +21,9 @@ void mte_clear_page_tags(void *addr);void mte_sync_tags (pte_t *ptep,pte_t pte);void mte_copy_page_tags (void *kto,const void *kfrom);void flush_mte_state (void ); +void mte_thread_switch (struct task_struct *next); +long set_mte_ctrl (unsigned long arg); +long get_mte_ctrl (void );#else
@@ -36,6 +39,17 @@ static inline void mte_copy_page_tags(void *kto, const void *kfrom)static inline void flush_mte_state (void ) { } +static inline void mte_thread_switch (struct task_struct *next) +{ +} +static inline long set_mte_ctrl (unsigned long arg) +{ +return 0 ; +} +static inline long get_mte_ctrl (void ) +{ +return 0 ; +}#endif
arch/arm64/include/asm/processor.h +3 -0
@@ -151,6 +151,9 @@ struct thread_struct {struct ptrauth_keys_user keys_user;struct ptrauth_keys_kernel keys_kernel;#endif +#ifdef CONFIG_ARM64_MTE +u64 sctlr_tcf0; +#endif }; static inline void arch_thread_struct_whitelist(unsigned long *offset,
arch/arm64/kernel/mte.c +77 -0
93 lines not on this page · raw message
arch/arm64/kernel/process.c +23 -3
@@ -544,6 +544,13 @@ __notrace_funcgraph struct task_struct *__switch_to(struct task_struct *prev,*/ dsb (ish); +/* +* MTE thread switching must happen after the DSB above to ensure that +* any asynchronous tag check faults have been logged in the TFSR*_EL1 +* registers. +*/ +mte_thread_switch (next); +/* the actual thread switch */ last =cpu_switch_to (prev, next);
@@ -603,9 +610,15 @@ static unsigned int tagged_addr_disabled;long set_tagged_addr_ctrl (unsigned long arg) { +unsigned long valid_mask =PR_TAGGED_ADDR_ENABLE ; +if (is_compat_task ())return -EINVAL ; -if (arg & ~PR_TAGGED_ADDR_ENABLE ) + +if (system_supports_mte ()) + valid_mask |=PR_MTE_TCF_MASK ; + +if (arg & ~valid_mask)return -EINVAL ;/*
@@ -615,6 +628,9 @@ long set_tagged_addr_ctrl(unsigned long arg)if (arg &PR_TAGGED_ADDR_ENABLE && tagged_addr_disabled)return -EINVAL ; +if (set_mte_ctrl (arg) !=0 ) +return -EINVAL ; +update_thread_flag (TIF_TAGGED_ADDR , arg &PR_TAGGED_ADDR_ENABLE );return 0 ;
@@ -622,13 +638,17 @@ long set_tagged_addr_ctrl(unsigned long arg)long get_tagged_addr_ctrl (void ) { +long ret =0 ; +if (is_compat_task ())return -EINVAL ;if (test_thread_flag (TIF_TAGGED_ADDR )) -return PR_TAGGED_ADDR_ENABLE ; + ret =PR_TAGGED_ADDR_ENABLE ; -return 0 ; + ret |=get_mte_ctrl (); + +return ret; }/*
include/uapi/linux/prctl.h +6 -0
@@ -233,6 +233,12 @@ struct prctl_mm_map {#define PR_SET_TAGGED_ADDR_CTRL 55#define PR_GET_TAGGED_ADDR_CTRL 56# define PR_TAGGED_ADDR_ENABLE (1UL << 0) +/* MTE tag check fault modes */ +# define PR_MTE_TCF_SHIFT 1 +# define PR_MTE_TCF_NONE (0UL << PR_MTE_TCF_SHIFT) +# define PR_MTE_TCF_SYNC (1UL << PR_MTE_TCF_SHIFT) +# define PR_MTE_TCF_ASYNC (2UL << PR_MTE_TCF_SHIFT) +# define PR_MTE_TCF_MASK (3UL << PR_MTE_TCF_SHIFT)/* Control reclaim behavior when allocating memory */ #define PR_SET_IO_FLUSHER 57
Catalin Marinas
maintainerPATCH v5 17/25 arm64: mte: Allow user control of the generated random tags via prctl()
In reply to Catalin MarinasThe IRG, ADDG and SUBG instructions insert a random tag in the resulting address. Certain tags can be excluded via the GCR_EL1.Exclude bitmap when, for example, the user wants a certain colour for freed buffers. Since the GCR_EL1 register is not accessible at EL0, extend the prctl(PR_SET_TAGGED_ADDR_CTRL) interface to include a 16-bit field in the first argument for controlling which tags can be generated by the above instruction (an include rather than exclude mask). Note that by default all non-zero tags are excluded. This setting is per-thread.
---
Notes:
v5:
- Rename gcr_incl to gcr_user_incl (there will be a subsequent gcr_kernel when support for in-kernel MTE is added).
v2:
- Switch from an exclude mask to an include one for the prctl() interface.
- Reset the allowed tags mask during flush_thread().
arch/arm64/include/asm/processor.h | 1 + arch/arm64/include/asm/sysreg.h | 7 ++++++ arch/arm64/kernel/mte.c | 35 +++++++++++++++++++++++++++--- arch/arm64/kernel/process.c | 2 +- include/uapi/linux/prctl.h | 3 +++ 5 files changed, 44 insertions(+), 4 deletions(-)
arch/arm64/include/asm/processor.h +1 -0
@@ -153,6 +153,7 @@ struct thread_struct {#endif #ifdef CONFIG_ARM64_MTE u64 sctlr_tcf0; +u64 gcr_user_incl;#endif };
arch/arm64/include/asm/sysreg.h +7 -0
@@ -1033,6 +1033,13 @@write_sysreg (__scs_new, sysreg); \ } while (0) +#define sysreg_clear_set_s (sysreg, clear, set) do { \ + u64 __scs_val = read_sysreg_s(sysreg); \ + u64 __scs_new = (__scs_val & ~(u64)(clear)) | (set); \ + if (__scs_new != __scs_val) \ + write_sysreg_s(__scs_new, sysreg); \ +} while (0) +#endif #endif /* __ASM_SYSREG_H */
arch/arm64/kernel/mte.c +32 -3
@@ -71,6 +71,25 @@ static void set_sctlr_el1_tcf0(u64 tcf0)preempt_enable (); } +static void update_gcr_el1_excl (u64 incl) +{ +u64 excl = ~incl &SYS_GCR_EL1_EXCL_MASK ; + +/* +* Note that 'incl' is an include mask (controlled by the user via +* prctl()) while GCR_EL1 accepts an exclude mask. +* No need for ISB since this only affects EL0 currently, implicit +* with ERET. +*/ +sysreg_clear_set_s (SYS_GCR_EL1 ,SYS_GCR_EL1_EXCL_MASK , excl); +} + +static void set_gcr_el1_excl (u64 incl) +{ + current->thread.gcr_user_incl = incl; +update_gcr_el1_excl (incl); +} +void flush_mte_state (void ) {if (!system_supports_mte ())
@@ -82,6 +101,8 @@ void flush_mte_state(void)clear_thread_flag (TIF_MTE_ASYNC_FAULT );/* disable tag checking */ set_sctlr_el1_tcf0 (SCTLR_EL1_TCF0_NONE ); +/* reset tag generation mask */ +set_gcr_el1_excl (0 ); }void mte_thread_switch (struct task_struct *next)
@@ -92,6 +113,7 @@ void mte_thread_switch(struct task_struct *next)/* avoid expensive SCTLR_EL1 accesses if no change */ if (current->thread.sctlr_tcf0 != next->thread.sctlr_tcf0)update_sctlr_el1_tcf0 (next->thread.sctlr_tcf0); +update_gcr_el1_excl (next->thread.gcr_user_incl); }long set_mte_ctrl (unsigned long arg)
@@ -116,23 +138,30 @@ long set_mte_ctrl(unsigned long arg) }set_sctlr_el1_tcf0 (tcf0); +set_gcr_el1_excl ((arg &PR_MTE_TAG_MASK ) >>PR_MTE_TAG_SHIFT );return 0 ; }long get_mte_ctrl (void ) { +unsigned long ret; +if (!system_supports_mte ())return 0 ; + ret = current->thread.gcr_user_incl <<PR_MTE_TAG_SHIFT ; +switch (current->thread.sctlr_tcf0) {case SCTLR_EL1_TCF0_NONE :return PR_MTE_TCF_NONE ;case SCTLR_EL1_TCF0_SYNC : -return PR_MTE_TCF_SYNC ; + ret |=PR_MTE_TCF_SYNC ; +break ;case SCTLR_EL1_TCF0_ASYNC : -return PR_MTE_TCF_ASYNC ; + ret |=PR_MTE_TCF_ASYNC ; +break ; } -return 0 ; +return ret; }
arch/arm64/kernel/process.c +1 -1
@@ -616,7 +616,7 @@ long set_tagged_addr_ctrl(unsigned long arg)return -EINVAL ;if (system_supports_mte ()) - valid_mask |=PR_MTE_TCF_MASK ; + valid_mask |=PR_MTE_TCF_MASK |PR_MTE_TAG_MASK ;if (arg & ~valid_mask)return -EINVAL ;
include/uapi/linux/prctl.h +3 -0
@@ -239,6 +239,9 @@ struct prctl_mm_map {# define PR_MTE_TCF_SYNC (1UL << PR_MTE_TCF_SHIFT)# define PR_MTE_TCF_ASYNC (2UL << PR_MTE_TCF_SHIFT)# define PR_MTE_TCF_MASK (3UL << PR_MTE_TCF_SHIFT) +/* MTE tag inclusion mask */ +# define PR_MTE_TAG_SHIFT 3 +# define PR_MTE_TAG_MASK (0xffffUL << PR_MTE_TAG_SHIFT)/* Control reclaim behavior when allocating memory */ #define PR_SET_IO_FLUSHER 57
Catalin Marinas
maintainerPATCH v5 18/25 arm64: mte: Restore the GCR_EL1 register after a suspend
In reply to Catalin MarinasThe CPU resume/suspend routines only take care of the common system registers. Restore GCR_EL1 in addition via the __cpu_suspend_exit() function.
---
Notes:
New in v3.
arch/arm64/include/asm/mte.h | 4 ++++ arch/arm64/kernel/mte.c | 8 ++++++++ arch/arm64/kernel/suspend.c | 4 ++++ 3 files changed, 16 insertions(+)
arch/arm64/include/asm/mte.h +4 -0
@@ -22,6 +22,7 @@ void mte_sync_tags(pte_t *ptep, pte_t pte);void mte_copy_page_tags (void *kto,const void *kfrom);void flush_mte_state (void );void mte_thread_switch (struct task_struct *next); +void mte_suspend_exit (void );long set_mte_ctrl (unsigned long arg);long get_mte_ctrl (void );
@@ -42,6 +43,9 @@ static inline void flush_mte_state(void)static inline void mte_thread_switch (struct task_struct *next) { } +static inline void mte_suspend_exit (void ) +{ +}static inline long set_mte_ctrl (unsigned long arg) {return 0 ;
arch/arm64/kernel/mte.c +8 -0
@@ -116,6 +116,14 @@ void mte_thread_switch(struct task_struct *next)update_gcr_el1_excl (next->thread.gcr_user_incl); } +void mte_suspend_exit (void ) +{ +if (!system_supports_mte ()) +return ; + +update_gcr_el1_excl (current->thread.gcr_user_incl); +} +long set_mte_ctrl (unsigned long arg) {u64 tcf0;
arch/arm64/kernel/suspend.c +4 -0
@@ -10,6 +10,7 @@#include <asm/daifflags.h> #include <asm/debug-monitors.h> #include <asm/exec.h> +#include <asm/mte.h> #include <asm/memory.h> #include <asm/mmu_context.h> #include <asm/smp_plat.h>
@@ -74,6 +75,9 @@ void notrace __cpu_suspend_exit(void)*/ if (arm64_get_ssbd_state () ==ARM64_SSBD_FORCE_DISABLE )arm64_set_ssbd_mitigation (false); + +/* Restore additional MTE-specific configuration */ +mte_suspend_exit (); }/*
Catalin Marinas
maintainerPATCH v5 19/25 arm64: mte: Add PTRACE_{PEEK,POKE}MTETAGS support
In reply to Catalin MarinasAdd support for bulk setting/getting of the MTE tags in a tracee's address space at 'addr' in the ptrace() syscall prototype. 'data' points to a struct iovec in the tracer's address space with iov_base representing the address of a tracer's buffer of length iov_len. The tags to be copied to/from the tracer's buffer are stored as one tag per byte.
On successfully copying at least one tag, ptrace() returns 0 and updates the tracer's iov_len with the number of tags copied. In case of error, either -EIO or -EFAULT is returned, trying to follow the ptrace() man page.
Note that the tag copying functions are not performance critical, therefore they lack optimisations found in typical memory copy routines.
---
Notes:
v4:
- Following the change to only clear the tags in a page if it is mapped to user with PROT_MTE, ptrace() now will refuse to access tags in pages not previously mapped with PROT_MTE (PG_mte_tagged set). This is primarily to avoid leaking uninitialised tags to user via ptrace().
- Fix SYM_FUNC_END argument typo.
- Rename MTE_ALLOC_* to MTE_GRANULE_*.
- Use uao_user_alternative for the user access in case we ever want to call mte_copy_tags_* with a kernel buffer. It also matches the other uaccess routines in the kernel.
- Simplify arch_ptrace() slightly.
- Reorder down_write_killable() with access_ok() in __access_remote_tags().
- Handle copy length 0 in mte_copy_tags_{to,from}_user().
- Use put_user() instead of __put_user().
New in v3.
arch/arm64/include/asm/mte.h | 17 ++++ arch/arm64/include/uapi/asm/ptrace.h | 3 + arch/arm64/kernel/mte.c | 139 +++++++++++++++++++++++++++ arch/arm64/kernel/ptrace.c | 7 ++ arch/arm64/lib/mte.S | 53 ++++++++++ 5 files changed, 219 insertions(+)
arch/arm64/include/asm/mte.h +17 -0
@@ -5,6 +5,11 @@#ifndef __ASM_MTE_H#define __ASM_MTE_H +#define MTE_GRANULE_SIZE UL(16) +#define MTE_GRANULE_MASK (~(MTE_GRANULE_SIZE - 1)) +#define MTE_TAG_SHIFT 56 +#define MTE_TAG_SIZE 4 +#ifndef __ASSEMBLY__#include <linux/page-flags.h>
@@ -12,6 +17,10 @@#include <asm/pgtable-types.h> void mte_clear_page_tags (void *addr); +unsigned long mte_copy_tags_from_user (void *to,const void __user *from, +unsigned long n); +unsigned long mte_copy_tags_to_user (void __user *to,void *from, +unsigned long n);#ifdef CONFIG_ARM64_MTE
@@ -25,6 +34,8 @@ void mte_thread_switch(struct task_struct *next);void mte_suspend_exit (void );long set_mte_ctrl (unsigned long arg);long get_mte_ctrl (void ); +int mte_ptrace_copy_tags (struct task_struct *child,long request, +unsigned long addr,unsigned long data);#else
@@ -54,6 +65,12 @@ static inline long get_mte_ctrl(void) {return 0 ; } +static inline int mte_ptrace_copy_tags (struct task_struct *child, +long request,unsigned long addr, +unsigned long data) +{ +return -EIO ; +}#endif
arch/arm64/include/uapi/asm/ptrace.h +3 -0
@@ -76,6 +76,9 @@/* syscall emulation path in ptrace */ #define PTRACE_SYSEMU 31#define PTRACE_SYSEMU_SINGLESTEP 32 +/* MTE allocation tag access */ +#define PTRACE_PEEKMTETAGS 33 +#define PTRACE_POKEMTETAGS 34#ifndef __ASSEMBLY__
arch/arm64/kernel/mte.c +139 -0
156 lines not on this page · raw message
arch/arm64/kernel/ptrace.c +7 -0
@@ -34,6 +34,7 @@#include <asm/cpufeature.h> #include <asm/debug-monitors.h> #include <asm/fpsimd.h> +#include <asm/mte.h> #include <asm/pointer_auth.h> #include <asm/stacktrace.h> #include <asm/syscall.h>
@@ -1796,6 +1797,12 @@ const struct user_regset_view *task_user_regset_view(struct task_struct *task)long arch_ptrace (struct task_struct *child,long request,unsigned long addr,unsigned long data) { +switch (request) { +case PTRACE_PEEKMTETAGS : +case PTRACE_POKEMTETAGS : +return mte_ptrace_copy_tags (child, request, addr, data); + } +return ptrace_request (child, request, addr, data); }
arch/arm64/lib/mte.S +53 -0
63 lines not on this page · raw message
Re: [PATCH v5 19/25] arm64: mte: Add PTRACE_{PEEK,POKE}MTETAGS support
Hi Catalin,
I have one point below I wanted to clarify regarding PEEKMTETAGS/POKEMTETAGS.
But before that, I've pushed v2 of the MTE series for GDB here:
https://sourceware.org/git/?p=binutils-gdb.git;a=shortlog;h=refs/heads/users/luisgpm/aarch64-mte-v2
That series adds sctlr and gcr registers to the NT_ARM_MTE (still using a dummy value of 0x407) register set. It would be nice if the Linux Kernel and the debuggers were in sync in terms of supporting this new register set. GDB assumes the register set exists if HWCAP2_MTE is there.
So, if we want to adjust the register set, we should probably consider doing that now. That prevents the situation where debuggers would need to do another check to confirm NT_ARM_MTE is exported. I'd rather avoid that.
What do you think?
Quoted 208 lines · Catalin Marinas
Add support for bulk setting/getting of the MTE tags in a tracee's address space at 'addr' in the
ptrace()syscall prototype. 'data' points to a struct iovec in the tracer's address space withiov_baserepresenting the address of a tracer's buffer of lengthiov_len. The tags to be copied to/from the tracer's buffer are stored as one tag per byte.On successfully copying at least one tag,
ptrace()returns 0 and updates the tracer'siov_lenwith the number of tags copied. In case of error, either-EIOor-EFAULTis returned, trying to follow theptrace()man page.Note that the tag copying functions are not performance critical, therefore they lack optimisations found in typical memory copy routines.
Signed-off-by Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>Cc Alan Hayward <[email protected]>Cc Luis Machado <[email protected]>Cc Omair Javaid <[email protected]>---
Notes:
v4:
- Following the change to only clear the tags in a page if it is mapped to user withPROT_MTE,ptrace()now will refuse to access tags in pages not previously mapped withPROT_MTE(PG_mte_taggedset). This is primarily to avoid leaking uninitialised tags to user viaptrace().
- FixSYM_FUNC_ENDargument typo.
- Rename MTE_ALLOC_* to MTE_GRANULE_*.
- Useuao_user_alternativefor the user access in case we ever want to call mte_copy_tags_* with a kernel buffer. It also matches the other uaccess routines in the kernel.
- Simplifyarch_ptrace()slightly.
- Reorderdown_write_killable()withaccess_ok()in__access_remote_tags().
- Handle copy length 0 in mte_copy_tags_{to,from}_user().
- Useput_user()instead of__put_user().New in v3.
arch/arm64/include/asm/mte.h | 17 ++++ arch/arm64/include/uapi/asm/ptrace.h | 3 + arch/arm64/kernel/mte.c | 139 +++++++++++++++++++++++++++ arch/arm64/kernel/ptrace.c | 7 ++ arch/arm64/lib/mte.S | 53 ++++++++++ 5 files changed, 219 insertions(+)arch/arm64/include/asm/mte.h +17
-0@@ -5,6 +5,11 @@#ifndef __ASM_MTE_H#define __ASM_MTE_H +#define MTE_GRANULE_SIZE UL(16) +#define MTE_GRANULE_MASK (~(MTE_GRANULE_SIZE - 1)) +#define MTE_TAG_SHIFT 56 +#define MTE_TAG_SIZE 4 +#ifndef __ASSEMBLY__#include <linux/page-flags.h> @@ -12,6 +17,10 @@#include <asm/pgtable-types.h> void mte_clear_page_tags (void *addr); +unsigned long mte_copy_tags_from_user (void *to,const void __user *from, +unsigned long n); +unsigned long mte_copy_tags_to_user (void __user *to,void *from, +unsigned long n);#ifdef CONFIG_ARM64_MTE @@ -25,6 +34,8 @@ void mte_thread_switch(struct task_struct *next);void mte_suspend_exit (void );long set_mte_ctrl (unsigned long arg);long get_mte_ctrl (void ); +int mte_ptrace_copy_tags (struct task_struct *child,long request, +unsigned long addr,unsigned long data);#else @@ -54,6 +65,12 @@ static inline long get_mte_ctrl(void) {return 0 ; } +static inline int mte_ptrace_copy_tags (struct task_struct *child, +long request,unsigned long addr, +unsigned long data) +{ +return -EIO ; +}#endif arch/arm64/include/uapi/asm/ptrace.h +3
-0@@ -76,6 +76,9 @@/* syscall emulation path in ptrace */ #define PTRACE_SYSEMU 31#define PTRACE_SYSEMU_SINGLESTEP 32 +/* MTE allocation tag access */ +#define PTRACE_PEEKMTETAGS 33 +#define PTRACE_POKEMTETAGS 34#ifndef __ASSEMBLY__arch/arm64/kernel/mte.c +74
-0@@ -4,14 +4,18 @@*/ #include <linux/bitops.h> +#include <linux/kernel.h> #include <linux/mm.h> #include <linux/prctl.h> #include <linux/sched.h> +#include <linux/sched/mm.h> #include <linux/string.h> #include <linux/thread_info.h> +#include <linux/uio.h> #include <asm/cpufeature.h> #include <asm/mte.h> +#include <asm/ptrace.h> #include <asm/sysreg.h> void mte_sync_tags (pte_t *ptep,pte_t pte)@@ -173,3 +177,138 @@ long get_mte_ctrl(void)return ret; } + +/* +* Access MTE tags in another process' address space as given in mm. Update +* the number of tags copied. Return 0 if any tags copied, error otherwise. +* Inspired by __access_remote_vm(). +*/ +static int __access_remote_tags (struct task_struct *tsk,struct mm_struct *mm, +unsigned long addr,struct iovec *kiov, +unsigned int gup_flags) +{ +struct vm_area_struct *vma; +void __user *buf = kiov->iov_base; +size_t len = kiov->iov_len; +int ret; +int write = gup_flags &FOLL_WRITE ; + +if (!access_ok (buf, len)) +return -EFAULT ; + +if (mmap_read_lock_killable (mm)) +return -EIO ; + +while (len) { +unsigned long tags, offset; +void *maddr; +struct page *page =NULL ; + + ret =get_user_pages_remote (tsk, mm, addr,1 , gup_flags, + &page, &vma,NULL ); +if (ret <=0 ) +break ; + +/* +* Only copy tags if the page has been mapped as PROT_MTE +* (PG_mte_tagged set). Otherwise the tags are not valid and +* not accessible to user. Moreover, an mprotect(PROT_MTE) +* would cause the existing tags to be cleared if the page +* was never mapped with PROT_MTE. +*/ +if (!test_bit (PG_mte_tagged, &page->flags)) { + ret = -EOPNOTSUPP ; +put_page (page); +break ; + } + +/* limit access to the end of the page */ + offset =offset_in_page (addr); + tags =min (len, (PAGE_SIZE - offset) /MTE_GRANULE_SIZE ); + + maddr =page_address (page); +if (write) { + tags =mte_copy_tags_from_user (maddr + offset, buf, tags); +set_page_dirty_lock (page); + }else { + tags =mte_copy_tags_to_user (buf, maddr + offset, tags); + } +put_page (page); + +/* error accessing the tracer's buffer */ +if (!tags) +break ; + + len -= tags; + buf += tags; + addr += tags *MTE_GRANULE_SIZE ; + } +mmap_read_unlock (mm); + +/* return an error if no tags copied */ + kiov->iov_len = buf - kiov->iov_base;
arch/arm64/kernel/mte.c long get_mte_ctrl(void) · line 259
Quoted 10 lines
arch/arm64/kernel/mte.c +10
-0+if (!kiov->iov_len) { +/* check for error accessing the tracee's address space */ +if (ret <=0 ) +return -EIO ; +else +return -EFAULT ; + } + +return 0 ; +}
My understanding is that both the PEEKMTETAGS and POKEMTETAGS can potentially read/write less tags than requested, right? The iov_len field will be updated accordingly.
So the ptrace caller would need to loop and make sure all the tags were read/written, right?
I'm considering the situation where the kernel reads/writes 0 tags (when requested to read/write 1 or more tags) an error we can't recover from. So this may indicate a page without PROT_MTE or an invalid address.
Does that make sense?
Catalin Marinas
maintainerRe: [PATCH v5 19/25] arm64: mte: Add PTRACE_{PEEK,POKE}MTETAGS support
Hi Luis,
Quoted 7 lines · Luis Machado
I have one point below I wanted to clarify regarding PEEKMTETAGS/POKEMTETAGS.
But before that, I've pushed v2 of the MTE series for GDB here:
https://sourceware.org/git/?p=binutils-gdb.git;a=shortlog;h=refs/heads/users/luisgpm/aarch64-mte-v2
Quoted 8 lines
That series adds sctlr and gcr registers to the
NT_ARM_MTE(still using a dummy value of0x407) register set. It would be nice if the Linux Kernel and the debuggers were in sync in terms of supporting this new register set. GDB assumes the register set exists ifHWCAP2_MTEis there.So, if we want to adjust the register set, we should probably consider doing that now. That prevents the situation where debuggers would need to do another check to confirm
NT_ARM_MTEis exported. I'd rather avoid that.
I'm happy to do this before merging, though we need to agree on the semantics.
Do you need both read and write access? Also wondering whether the prctl() value would be a better option than the raw register bits (well, not entirely raw, masking out the irrelevant part).
Signature · 1 line
Catalin
Re: [PATCH v5 19/25] arm64: mte: Add PTRACE_{PEEK,POKE}MTETAGS support
Hi,
Quoted 15 lines · Catalin Marinas
Hi Luis,
Quoted 12 lines · Luis Machado
I have one point below I wanted to clarify regarding PEEKMTETAGS/POKEMTETAGS.
But before that, I've pushed v2 of the MTE series for GDB here:
https://sourceware.org/git/?p=binutils-gdb.git;a=shortlog;h=refs/heads/users/luisgpm/aarch64-mte-v2
That series adds sctlr and gcr registers to the
NT_ARM_MTE(still using a dummy value of0x407) register set. It would be nice if the Linux Kernel and the debuggers were in sync in terms of supporting this new register set. GDB assumes the register set exists ifHWCAP2_MTEis there.
Quoted 8 lines
Quoted 3 lines
So, if we want to adjust the register set, we should probably consider doing that now. That prevents the situation where debuggers would need to do another check to confirm
NT_ARM_MTEis exported. I'd rather avoid that.I'm happy to do this before merging, though we need to agree on the semantics.
Do you need both read and write access? Also wondering whether the
If I recall the previous discussion correctly, Kevin thought access to both of these would be interesting to the user. It sounded like having read-only access was enough. If so,...
Quoted 2 lines
prctl()value would be a better option than the raw register bits (well, not entirely raw, masking out the irrelevant part).
... then exposing the most useful bits to the user would be better, and up to you to define.
I can tweak the GDB patches to turn the sctlr and gcr values into flag fields. Then GDB can just show those in a more meaningful way. I just need to know what the bits would look like.
I'd rather not make these values writable if we don't think there is a good use case for it. Better avoid giving developers more knobs than they need?
Catalin Marinas
maintainerRe: [PATCH v5 19/25] arm64: mte: Add PTRACE_{PEEK,POKE}MTETAGS support
Quoted 30 lines · Luis Machado
Quoted 13 lines · Catalin Marinas
Quoted 12 lines · Luis Machado
I have one point below I wanted to clarify regarding PEEKMTETAGS/POKEMTETAGS.
But before that, I've pushed v2 of the MTE series for GDB here:
https://sourceware.org/git/?p=binutils-gdb.git;a=shortlog;h=refs/heads/users/luisgpm/aarch64-mte-v2
That series adds sctlr and gcr registers to the
NT_ARM_MTE(still using a dummy value of0x407) register set. It would be nice if the Linux Kernel and the debuggers were in sync in terms of supporting this new register set. GDB assumes the register set exists ifHWCAP2_MTEis there.Quoted 8 lines
Quoted 3 lines
So, if we want to adjust the register set, we should probably consider doing that now. That prevents the situation where debuggers would need to do another check to confirm
NT_ARM_MTEis exported. I'd rather avoid that.I'm happy to do this before merging, though we need to agree on the semantics.
Do you need both read and write access? Also wondering whether the
If I recall the previous discussion correctly, Kevin thought access to both of these would be interesting to the user. It sounded like having read-only access was enough. If so,...
Quoted 2 lines
prctl()value would be a better option than the raw register bits (well, not entirely raw, masking out the irrelevant part).
Quoted 6 lines
... then exposing the most useful bits to the user would be better, and up to you to define.
I can tweak the GDB patches to turn the sctlr and gcr values into flag fields. Then GDB can just show those in a more meaningful way. I just need to know what the bits would look like.
We may have some software only behaviour added to these bits at some point (e.g. deliver signal on return from syscall for faults on the uaccess routines). They would not be represented in the SCTLR/GCR registers.
Quoted 2 lines
I'd rather not make these values writable if we don't think there is a good use case for it. Better avoid giving developers more knobs than they need?
There's the CRIU use-case for restoring this but I don't think we do it for other prctl() controls.
Signature · 1 line
Catalin
Catalin Marinas
maintainerRe: [PATCH v5 19/25] arm64: mte: Add PTRACE_{PEEK,POKE}MTETAGS support
In reply to Luis MachadoHi Luis,
Quoted 44 lines · Luis Machado
Quoted 43 lines · Catalin Marinas
arch/arm64/kernel/mte.c +43
-0+/* +* Access MTE tags in another process' address space as given in mm. Update +* the number of tags copied. Return 0 if any tags copied, error otherwise. +* Inspired by __access_remote_vm(). +*/ +static int __access_remote_tags (struct task_struct *tsk,struct mm_struct *mm, +unsigned long addr,struct iovec *kiov, +unsigned int gup_flags) +{ +struct vm_area_struct *vma; +void __user *buf = kiov->iov_base; +size_t len = kiov->iov_len; +int ret; +int write = gup_flags &FOLL_WRITE ; + +if (!access_ok (buf, len)) +return -EFAULT ; + +if (mmap_read_lock_killable (mm)) +return -EIO ; + +while (len) { +unsigned long tags, offset; +void *maddr; +struct page *page =NULL ; + + ret =get_user_pages_remote (tsk, mm, addr,1 , gup_flags, + &page, &vma,NULL ); +if (ret <=0 ) +break ; + +/* +* Only copy tags if the page has been mapped as PROT_MTE +* (PG_mte_tagged set). Otherwise the tags are not valid and +* not accessible to user. Moreover, an mprotect(PROT_MTE) +* would cause the existing tags to be cleared if the page +* was never mapped with PROT_MTE. +*/ +if (!test_bit (PG_mte_tagged, &page->flags)) { + ret = -EOPNOTSUPP ; +put_page (page); +break ; + }
[...]
Quoted 3 lines
My understanding is that both the PEEKMTETAGS and POKEMTETAGS can potentially read/write less tags than requested, right? The
iov_lenfield will be updated accordingly.
Yes.
(I missed this part due to the mix of top/bottom-posting)
Quoted 2 lines
So the ptrace caller would need to loop and make sure all the tags were read/written, right?
Yes. As per the documentation patch, if the ptrace call returns 0, iov_len is updated to the number of tags copied. The caller can retry until it gets a negative return (error) or everything was copied.
Quoted 3 lines
I'm considering the situation where the kernel reads/writes 0 tags (when requested to read/write 1 or more tags) an error we can't recover from. So this may indicate a page without
PROT_MTEor an invalid address.
For this case, it should return -EOPNOTSUPP (see the documentation patch; and, of course, also test the syscall in case I got anything wrong).
Signature · 1 line
Catalin
Catalin Marinas
maintainerPATCH v5 20/25 fs: Handle intra-page faults in copy_mount_options()
In reply to Catalin MarinasThe copy_mount_options() function takes a user pointer argument but no size. It tries to read up to a PAGE_SIZE. However, copy_from_user() is not guaranteed to return all the accessible bytes if, for example, the access crosses a page boundary and gets a fault on the second page. To work around this, the current copy_mount_options() implementation performs two copy_from_user() passes, first to the end of the current page and the second to what's left in the subsequent page.
On arm64 with MTE enabled, access to a user page may trigger a fault after part of the buffer has been copied (when the user pointer tag, bits 56-59, no longer matches the allocation tag stored in memory). Allow copy_mount_options() to handle such intra-page faults by returning -EFAULT only if the first copy_from_user() has not copied any bytes.
---
Notes:
v4:
- Rewrite to avoid arch_has_exact_copy_from_user()
New in v3.
fs/namespace.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
fs/namespace.c +21 -3
@@ -3074,7 +3074,7 @@ static void shrink_submounts(struct mount *mnt)void *copy_mount_options (const void __user * data) {char *copy; -unsigned size; +unsigned size, left;if (!data)return NULL ;
@@ -3085,12 +3085,30 @@ void *copy_mount_options(const void __user * data) size =PAGE_SIZE -offset_in_page (data); -if (copy_from_user (copy, data, size)) { +/* +* Attempt to copy to the end of the first user page. On success, +* left == 0, copy the rest from the second user page (if it is +* accessible). copy_from_user() will zero the part of the kernel +* buffer not copied into. +* +* On architectures with intra-page faults (arm64 with MTE), the read +* from the first page may fail after copying part of the user data +* (left > 0 && left < size). Do not attempt the second copy in this +* case as the end of the valid user buffer has already been reached. +* Ensure, however, that the second part of the kernel buffer is +* zeroed. +*/ + left =copy_from_user (copy, data, size); +if (left == size) {kfree (copy);return ERR_PTR (-EFAULT ); }if (size !=PAGE_SIZE ) { -if (copy_from_user (copy + size, data + size,PAGE_SIZE - size)) +if (left ==0 ) +/* return not relevant, just silence the compiler */ + left =copy_from_user (copy + size, data + size, +PAGE_SIZE - size); +else memset (copy + size,0 ,PAGE_SIZE - size); }return copy;
Catalin Marinas
maintainerPATCH v5 21/25 mm: Add arch hooks for saving/restoring tags
In reply to Catalin MarinasFrom: Steven Price <[email protected]>
Arm's Memory Tagging Extension (MTE) adds some metadata (tags) to every physical page, when swapping pages out to disk it is necessary to save these tags, and later restore them when reading the pages back.
Add some hooks along with dummy implementations to enable the arch code to handle this.
Three new hooks are added to the swap code:
* arch_prepare_to_swap() and
* arch_swap_invalidate_page() / arch_swap_invalidate_area().
One new hook is added to shmem:
* arch_swap_restore_tags()
---
Notes:
New in v4.
include/linux/pgtable.h | 23 +++++++++++++++++++++++ mm/page_io.c | 10 ++++++++++ mm/shmem.c | 6 ++++++ mm/swapfile.c | 2 ++ 4 files changed, 41 insertions(+)
include/linux/pgtable.h +23 -0
@@ -631,6 +631,29 @@ static inline int arch_unmap_one(struct mm_struct *mm, }#endif +#ifndef __HAVE_ARCH_PREPARE_TO_SWAP +static inline int arch_prepare_to_swap (struct page *page) +{ +return 0 ; +} +#endif + +#ifndef __HAVE_ARCH_SWAP_INVALIDATE +static inline void arch_swap_invalidate_page (int type,pgoff_t offset) +{ +} + +static inline void arch_swap_invalidate_area (int type) +{ +} +#endif + +#ifndef __HAVE_ARCH_SWAP_RESTORE_TAGS +static inline void arch_swap_restore_tags (swp_entry_t entry,struct page *page) +{ +} +#endif +#ifndef __HAVE_ARCH_PGD_OFFSET_GATE#define pgd_offset_gate (mm, addr) pgd_offset(mm, addr)#endif
mm/page_io.c +10 -0
@@ -252,6 +252,16 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)unlock_page (page); goto out; } +/* +* Arch code may have to preserve more data than just the page +* contents, e.g. memory tags. +*/ + ret =arch_prepare_to_swap (page); +if (ret) { +set_page_dirty (page); +unlock_page (page); + goto out; + }if (frontswap_store (page) ==0 ) {set_page_writeback (page);unlock_page (page);
mm/shmem.c +6 -0
@@ -1673,6 +1673,12 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index, }wait_on_page_writeback (page); +/* +* Some architectures may have to restore extra metadata to the +* physical page after reading from swap. +*/ +arch_swap_restore_tags (swap, page); +if (shmem_should_replace_page (page, gfp)) { error =shmem_replace_page (&page, gfp, info, index);if (error)
mm/swapfile.c +2 -0
@@ -716,6 +716,7 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,else swap_slot_free_notify =NULL ;while (offset <= end) { +arch_swap_invalidate_page (si->type, offset);frontswap_invalidate_page (si->type, offset);if (swap_slot_free_notify)swap_slot_free_notify (si->bdev, offset);
@@ -2675,6 +2676,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) frontswap_map =frontswap_map_get (p);spin_unlock (&p->lock);spin_unlock (&swap_lock); +arch_swap_invalidate_area (p->type);frontswap_invalidate_area (p->type);frontswap_map_set (p,NULL );mutex_unlock (&swapon_mutex);
Re: [PATCH v5 21/25] mm: Add arch hooks for saving/restoring tags
Quoted 35 lines · On Wed, 24 Jun 2020 18:52:40 +0100 Catalin Marinas <[email protected]>…
From: Steven Price <[email protected]>
Arm's Memory Tagging Extension (MTE) adds some metadata (tags) to every physical page, when swapping pages out to disk it is necessary to save these tags, and later restore them when reading the pages back.
Add some hooks along with dummy implementations to enable the arch code to handle this.
Three new hooks are added to the swap code:
*arch_prepare_to_swap()and
*arch_swap_invalidate_page()/arch_swap_invalidate_area().
One new hook is added to shmem:
*arch_swap_restore_tags()...
include/linux/pgtable.h +12
-0@@ -631,6 +631,29 @@ static inline int arch_unmap_one(struct mm_struct *mm, }#endif +#ifndef __HAVE_ARCH_PREPARE_TO_SWAP +static inline int arch_prepare_to_swap (struct page *page) +{ +return 0 ; +} +#endif + +#ifndef __HAVE_ARCH_SWAP_INVALIDATE +static inline void arch_swap_invalidate_page (int type,pgoff_t offset) +{ +} +
include/linux/pgtable.h static inline int arch_unmap_one(struct mm_struct *mm, · line 655
Quoted 10 lines
include/linux/pgtable.h +10
-0+static inline void arch_swap_invalidate_area (int type) +{ +} +#endif + +#ifndef __HAVE_ARCH_SWAP_RESTORE_TAGS +static inline void arch_swap_restore_tags (swp_entry_t entry,struct page *page) +{ +} +#endif
Presumably these three __HAVE_ARCH_ macros are to be defined in asm/pgtable.h?
Re: [PATCH v5 21/25] mm: Add arch hooks for saving/restoring tags
Quoted 41 lines · Andrew Morton
Quoted 39 lines · On Wed, 24 Jun 2020 18:52:40 +0100 Catalin Marinas <[email protected]>…
From: Steven Price <[email protected]>
Arm's Memory Tagging Extension (MTE) adds some metadata (tags) to every physical page, when swapping pages out to disk it is necessary to save these tags, and later restore them when reading the pages back.
Add some hooks along with dummy implementations to enable the arch code to handle this.
Three new hooks are added to the swap code:
*arch_prepare_to_swap()and
*arch_swap_invalidate_page()/arch_swap_invalidate_area().
One new hook is added to shmem:
*arch_swap_restore_tags()...
include/linux/pgtable.h +16
-0@@ -631,6 +631,29 @@ static inline int arch_unmap_one(struct mm_struct *mm, }#endif +#ifndef __HAVE_ARCH_PREPARE_TO_SWAP +static inline int arch_prepare_to_swap (struct page *page) +{ +return 0 ; +} +#endif + +#ifndef __HAVE_ARCH_SWAP_INVALIDATE +static inline void arch_swap_invalidate_page (int type,pgoff_t offset) +{ +} + +static inline void arch_swap_invalidate_area (int type) +{ +} +#endif
Quoted 8 lines
include/linux/pgtable.h static inline int arch_unmap_one(struct mm_struct *mm, · line 655
Quoted 6 lines
include/linux/pgtable.h +6
-0+ +#ifndef __HAVE_ARCH_SWAP_RESTORE_TAGS +static inline void arch_swap_restore_tags (swp_entry_t entry,struct page *page) +{ +} +#endif Presumably these three
__HAVE_ARCH_macros are to be defined inasm/pgtable.h?
That would be the idea (see patch 22). However:
Catalin - you've renamed __HAVE_ARCH_SWAP_RESTORE_TAGS in patch 22, but not here!
Steve
Catalin Marinas
maintainerRe: [PATCH v5 21/25] mm: Add arch hooks for saving/restoring tags
Quoted 50 lines · Steven Price
Quoted 40 lines · Andrew Morton
Quoted 38 lines · On Wed, 24 Jun 2020 18:52:40 +0100 Catalin Marinas <[email protected]>…
From: Steven Price <[email protected]>
Arm's Memory Tagging Extension (MTE) adds some metadata (tags) to every physical page, when swapping pages out to disk it is necessary to save these tags, and later restore them when reading the pages back.
Add some hooks along with dummy implementations to enable the arch code to handle this.
Three new hooks are added to the swap code:
*arch_prepare_to_swap()and
*arch_swap_invalidate_page()/arch_swap_invalidate_area().
One new hook is added to shmem:
*arch_swap_restore_tags()...
include/linux/pgtable.h +16
-0@@ -631,6 +631,29 @@ static inline int arch_unmap_one(struct mm_struct *mm, }#endif +#ifndef __HAVE_ARCH_PREPARE_TO_SWAP +static inline int arch_prepare_to_swap (struct page *page) +{ +return 0 ; +} +#endif + +#ifndef __HAVE_ARCH_SWAP_INVALIDATE +static inline void arch_swap_invalidate_page (int type,pgoff_t offset) +{ +} + +static inline void arch_swap_invalidate_area (int type) +{ +} +#endif Quoted 8 lines
include/linux/pgtable.h static inline int arch_unmap_one(struct mm_struct *mm, · line 655
Quoted 6 lines
include/linux/pgtable.h +6
-0+ +#ifndef __HAVE_ARCH_SWAP_RESTORE_TAGS +static inline void arch_swap_restore_tags (swp_entry_t entry,struct page *page) +{ +} +#endif Presumably these three
__HAVE_ARCH_macros are to be defined inasm/pgtable.h?
Quoted 4 lines
That would be the idea (see patch 22). However:
Catalin - you've renamed
__HAVE_ARCH_SWAP_RESTORE_TAGSin patch 22, but not here!
This was meant to be arch_swap_restore() and __HAVE_ARCH_SWAP_RESTORE (no tags suffix) and it was originally in include/asm-generic/pgtable.h. With Mike's recent reworking getting rid of this file, I messed up the conflict resolution during rebase and re-introduced this file in patch
22. I'll fix it up, it needs to be only in include/linux/pgtable.h.
Signature · 1 line
Catalin
Catalin Marinas
maintainerPATCH v5 22/25 arm64: mte: Enable swap of tagged pages
In reply to Catalin MarinasFrom: Steven Price <[email protected]>
When swapping pages out to disk it is necessary to save any tags that have been set, and restore when swapping back in. Make use of the new page flag (PG_ARCH_2, locally named PG_mte_tagged) to identify pages with tags. When swapping out these pages the tags are stored in memory and later restored when the pages are brought back in. Because shmem can swap pages back in without restoring the userspace PTE it is also necessary to add a hook for shmem.
---
Notes:
New in v4.
arch/arm64/include/asm/mte.h | 8 + arch/arm64/include/asm/pgtable.h | 32 + arch/arm64/kernel/mte.c | 19 +- arch/arm64/lib/mte.S | 45 ++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/mteswap.c | 82 ++ include/asm-generic/pgtable.h | 1285 ++++++++++++++++++++++++++++++ mm/shmem.c | 2 +- 8 files changed, 1472 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/mm/mteswap.c create mode 100644 include/asm-generic/pgtable.h
arch/arm64/include/asm/mte.h +8 -0
@@ -21,6 +21,14 @@ unsigned long mte_copy_tags_from_user(void *to, const void __user *from, unsigned long n);unsigned long mte_copy_tags_to_user (void __user *to,void *from,unsigned long n); +int mte_save_tags (struct page *page); +void mte_save_page_tags (const void *page_addr,void *tag_storage); +bool mte_restore_tags (swp_entry_t entry,struct page *page); +void mte_restore_page_tags (void *page_addr,const void *tag_storage); +void mte_invalidate_tags (int type,pgoff_t offset); +void mte_invalidate_tags_area (int type); +void *mte_allocate_tag_storage (void ); +void mte_free_tag_storage (char *storage);#ifdef CONFIG_ARM64_MTE
arch/arm64/include/asm/pgtable.h +32 -0
@@ -857,6 +857,38 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,extern int kern_addr_valid (unsigned long addr); +#ifdef CONFIG_ARM64_MTE + +#define __HAVE_ARCH_PREPARE_TO_SWAP +static inline int arch_prepare_to_swap (struct page *page) +{ +if (system_supports_mte ()) +return mte_save_tags (page); +return 0 ; +} + +#define __HAVE_ARCH_SWAP_INVALIDATE +static inline void arch_swap_invalidate_page (int type,pgoff_t offset) +{ +if (system_supports_mte ()) +mte_invalidate_tags (type, offset); +} + +static inline void arch_swap_invalidate_area (int type) +{ +if (system_supports_mte ()) +mte_invalidate_tags_area (type); +} + +#define __HAVE_ARCH_SWAP_RESTORE +static inline void arch_swap_restore (swp_entry_t entry,struct page *page) +{ +if (system_supports_mte () &&mte_restore_tags (entry, page)) +set_bit (PG_mte_tagged, &page->flags); +} + +#endif /* CONFIG_ARM64_MTE */ +/* * On AArch64, the cache coherency is handled via the set_pte_at() function. */
arch/arm64/kernel/mte.c +18 -1
@@ -10,6 +10,8 @@#include <linux/sched.h> #include <linux/sched/mm.h> #include <linux/string.h> +#include <linux/swap.h> +#include <linux/swapops.h> #include <linux/thread_info.h> #include <linux/uio.h>
@@ -18,15 +20,30 @@#include <asm/ptrace.h> #include <asm/sysreg.h> +static void mte_sync_page_tags (struct page *page,pte_t *ptep,bool check_swap) +{ +pte_t old_pte =READ_ONCE (*ptep); + +if (check_swap &&is_swap_pte (old_pte)) { +swp_entry_t entry =pte_to_swp_entry (old_pte); + +if (!non_swap_entry (entry) &&mte_restore_tags (entry, page)) +return ; + } + +mte_clear_page_tags (page_address (page)); +} +void mte_sync_tags (pte_t *ptep,pte_t pte) {struct page *page =pte_page (pte);long i, nr_pages =compound_nr (page); +bool check_swap = nr_pages ==0 ;/* if PG_mte_tagged is set, tags have already been initialised */ for (i =0 ; i < nr_pages; i++, page++) {if (!test_and_set_bit (PG_mte_tagged, &page->flags)) -mte_clear_page_tags (page_address (page)); +mte_sync_page_tags (page, ptep, check_swap); } }
arch/arm64/lib/mte.S +45 -0
48 lines not on this page · raw message
arch/arm64/mm/Makefile +1 -0
@@ -8,6 +8,7 @@ obj-$(CONFIG_PTDUMP_CORE) += dump.o obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o obj-$(CONFIG_NUMA) += numa.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o +obj-$(CONFIG_ARM64_MTE) += mteswap.o KASAN_SANITIZE_physaddr.o += n obj-$(CONFIG_KASAN) += kasan_init.o
arch/arm64/mm/mteswap.c new file +82 -0
82 lines not on this page · raw message
include/asm-generic/pgtable.h new file +1285 -0
1,285 lines not on this page · raw message
mm/shmem.c +1 -1
@@ -1677,7 +1677,7 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index,* Some architectures may have to restore extra metadata to the * physical page after reading from swap. */ -arch_swap_restore_tags (swap, page); +arch_swap_restore (swap, page);if (shmem_should_replace_page (page, gfp)) { error =shmem_replace_page (&page, gfp, info, index);
Re: [PATCH v5 22/25] arm64: mte: Enable swap of tagged pages
Quoted 21 lines · Catalin Marinas
From: Steven Price <[email protected]>
When swapping pages out to disk it is necessary to save any tags that have been set, and restore when swapping back in. Make use of the new page flag (
PG_ARCH_2, locally namedPG_mte_tagged) to identify pages with tags. When swapping out these pages the tags are stored in memory and later restored when the pages are brought back in. Because shmem can swap pages back in without restoring the userspace PTE it is also necessary to add a hook for shmem.Signed-off-by Steven Price <[email protected]>[[email protected]: move function prototypes to mte.h][[email protected]: drop '_tags' from arch_swap_restore_tags()]Signed-off-by Catalin Marinas <[email protected]>Cc Andrew Morton <[email protected]>Cc Will Deacon <[email protected]>---
Notes:
New in v4.
[...]
Quoted 29 lines
arch/arm64/kernel/mte.c +14
-0@@ -10,6 +10,8 @@#include <linux/sched.h> #include <linux/sched/mm.h> #include <linux/string.h> +#include <linux/swap.h> +#include <linux/swapops.h> #include <linux/thread_info.h> #include <linux/uio.h> @@ -18,15 +20,30 @@#include <asm/ptrace.h> #include <asm/sysreg.h> +static void mte_sync_page_tags (struct page *page,pte_t *ptep,bool check_swap) +{ +pte_t old_pte =READ_ONCE (*ptep); + +if (check_swap &&is_swap_pte (old_pte)) { +swp_entry_t entry =pte_to_swp_entry (old_pte); + +if (!non_swap_entry (entry) &&mte_restore_tags (entry, page)) +return ; + } + +mte_clear_page_tags (page_address (page));
arch/arm64/kernel/mte.c line 44
Quoted 10 lines
arch/arm64/kernel/mte.c +3
-0+} +void mte_sync_tags (pte_t *ptep,pte_t pte) {struct page *page =pte_page (pte);long i, nr_pages =compound_nr (page); +bool check_swap = nr_pages ==0 ;/* if PG_mte_tagged is set, tags have already been initialised */ for (i =0 ; i < nr_pages; i++, page++) {
This is broken - for check_swap to be true, nr_pages==0, which means we never enter the loop and nothing happens...
Except I don't believe compound_nr() will return 0 - it's defined as:
static inline unsigned long compound_nr(struct page *page)
{
return 1UL << compound_order(page);
}Changing it to nr_pages==1 works for me.
Steve
Catalin Marinas
maintainerRe: [PATCH v5 22/25] arm64: mte: Enable swap of tagged pages
Quoted 41 lines · Steven Price
Quoted 26 lines · Catalin Marinas
arch/arm64/kernel/mte.c +13
-0@@ -10,6 +10,8 @@#include <linux/sched.h> #include <linux/sched/mm.h> #include <linux/string.h> +#include <linux/swap.h> +#include <linux/swapops.h> #include <linux/thread_info.h> #include <linux/uio.h> @@ -18,15 +20,30 @@#include <asm/ptrace.h> #include <asm/sysreg.h> +static void mte_sync_page_tags (struct page *page,pte_t *ptep,bool check_swap) +{ +pte_t old_pte =READ_ONCE (*ptep); + +if (check_swap &&is_swap_pte (old_pte)) { +swp_entry_t entry =pte_to_swp_entry (old_pte); + +if (!non_swap_entry (entry) &&mte_restore_tags (entry, page)) +return ; + } +arch/arm64/kernel/mte.c line 42
Quoted 10 lines
arch/arm64/kernel/mte.c +4
-0+mte_clear_page_tags (page_address (page)); +} +void mte_sync_tags (pte_t *ptep,pte_t pte) {struct page *page =pte_page (pte);long i, nr_pages =compound_nr (page); +bool check_swap = nr_pages ==0 ;/* if PG_mte_tagged is set, tags have already been initialised */ for (i =0 ; i < nr_pages; i++, page++) {This is broken - for
check_swapto be true, nr_pages==0, which means we never enter the loop and nothing happens...
Quoted 8 lines
Except I don't believe
compound_nr()will return 0 - it's defined as:static inline unsigned long compound_nr(struct page *page) { return 1UL << compound_order(page); }Changing it to nr_pages==1 works for me.
Ah, I had it as compound_order() and changes to compound_nr() but left the 0 check. Fixed locally.
Thanks.
Signature · 1 line
Catalin
Catalin Marinas
maintainerPATCH v5 23/25 arm64: mte: Save tags when hibernating
In reply to Catalin MarinasFrom: Steven Price <[email protected]>
When hibernating the contents of all pages in the system are written to disk, however the MTE tags are not visible to the generic hibernation code. So just before the hibernation image is created copy the tags out of the physical tag storage into standard memory so they will be included in the hibernation image. After hibernation apply the tags back into the physical tag storage.
---
Notes:
New in v4.
arch/arm64/kernel/hibernate.c | 118 ++++++++++++++++++++++++++++++++++ 1 file changed, 118 insertions(+)
arch/arm64/kernel/hibernate.c +118 -0
@@ -31,6 +31,7 @@#include <asm/kexec.h> #include <asm/memory.h> #include <asm/mmu_context.h> +#include <asm/mte.h> #include <asm/pgalloc.h> #include <asm/pgtable-hwdef.h> #include <asm/sections.h>
@@ -285,6 +286,117 @@ static int create_safe_exec_page(void *src_start, size_t length,#define dcache_clean_range (start, end) __flush_dcache_area(start, (end - start)) +#ifdef CONFIG_ARM64_MTE + +static DEFINE_XARRAY (mte_pages); + +static int save_tags (struct page *page,unsigned long pfn) +{ +void *tag_storage, *ret; + + tag_storage =mte_allocate_tag_storage (); +if (!tag_storage) +return -ENOMEM ; + +mte_save_page_tags (page_address (page), tag_storage); + + ret =xa_store (&mte_pages, pfn, tag_storage,GFP_KERNEL ); +if (WARN (xa_is_err (ret),"Failed to store MTE tags" )) { +mte_free_tag_storage (tag_storage); +return xa_err (ret); + }else if (WARN (ret,"swsusp: %s: Duplicate entry" , __func__)) { +mte_free_tag_storage (ret); + } + +return 0 ; +} + +static void swsusp_mte_free_storage (void ) +{ +XA_STATE (xa_state, &mte_pages,0 ); +void *tags; + +xa_lock (&mte_pages); +xas_for_each (&xa_state, tags,ULONG_MAX ) { +mte_free_tag_storage (tags); + } +xa_unlock (&mte_pages); + +xa_destroy (&mte_pages); +} + +static int swsusp_mte_save_tags (void ) +{ +struct zone *zone; +unsigned long pfn, max_zone_pfn; +int ret =0 ; +int n =0 ; + +if (!system_supports_mte ()) +return 0 ; + +for_each_populated_zone (zone) { + max_zone_pfn =zone_end_pfn (zone); +for (pfn = zone->zone_start_pfn; pfn < max_zone_pfn; pfn++) { +struct page *page =pfn_to_online_page (pfn); + +if (!page) +continue ; + +if (!test_bit (PG_mte_tagged, &page->flags)) +continue ; + + ret =save_tags (page, pfn); +if (ret) { +swsusp_mte_free_storage (); + goto out; + } + + n++; + } + } +pr_info ("Saved %d MTE pages\n" , n); + +out: +return ret; +} + +static void swsusp_mte_restore_tags (void ) +{ +XA_STATE (xa_state, &mte_pages,0 ); +int n =0 ; +void *tags; + +xa_lock (&mte_pages); +xas_for_each (&xa_state, tags,ULONG_MAX ) { +unsigned long pfn = xa_state.xa_index; +struct page *page =pfn_to_online_page (pfn); + +mte_restore_page_tags (page_address (page), tags); + +mte_free_tag_storage (tags); + n++; + } +xa_unlock (&mte_pages); + +pr_info ("Restored %d MTE pages\n" , n); + +xa_destroy (&mte_pages); +} + +#else /* CONFIG_ARM64_MTE */ + +static int swsusp_mte_save_tags (void ) +{ +return 0 ; +} + +static void swsusp_mte_restore_tags (void ) +{ +} + +#endif /* CONFIG_ARM64_MTE */ +int swsusp_arch_suspend (void ) {int ret =0 ;
@@ -302,6 +414,10 @@ int swsusp_arch_suspend(void)/* make the crash dump kernel image visible/saveable */ crash_prepare_suspend (); + ret =swsusp_mte_save_tags (); +if (ret) +return ret; + sleep_cpu =smp_processor_id (); ret =swsusp_save (); }else {
@@ -315,6 +431,8 @@ int swsusp_arch_suspend(void)dcache_clean_range (__hyp_text_start, __hyp_text_end); } +swsusp_mte_restore_tags (); +/* make the crash dump kernel image protected again */ crash_post_resume ();
Catalin Marinas
maintainerPATCH v5 24/25 arm64: mte: Kconfig entry
In reply to Catalin MarinasFrom: Vincenzo Frascino <[email protected]>
Add Memory Tagging Extension support to the arm64 kbuild.
---
Notes:
v5:
- Remove duplicate ARMv8.5 menu entry.
v4:
- select ARCH_USES_PG_ARCH_2.
- remove ARCH_NO_SWAP.
- default y.
arch/arm64/Kconfig | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
arch/arm64/Kconfig +29 -0
@@ -1664,6 +1664,35 @@ config ARCH_RANDOM provides a high bandwidth, cryptographically secure hardware random number generator. +config ARM64_AS_HAS_MTE + def_bool $(as-instr,.arch armv8.5-a+memtag) + +config ARM64_MTE + bool "Memory Tagging Extension support" + default y + depends on ARM64_AS_HAS_MTE && ARM64_TAGGED_ADDR_ABI + select ARCH_USES_HIGH_VMA_FLAGS + select ARCH_USES_PG_ARCH_2 + help + Memory Tagging (part of the ARMv8.5 Extensions) provides + architectural support for run-time, always-on detection of + various classes of memory error to aid with software debugging + to eliminate vulnerabilities arising from memory-unsafe + languages. + + This option enables the support for the Memory Tagging + Extension at EL0 (i.e. for userspace). + + Selecting this option allows the feature to be detected at + runtime. Any secondary CPU not implementing this feature will + not be allowed a late bring-up. + + Userspace binaries that want to use this feature must + explicitly opt in. The mechanism for the userspace is + described in: + + Documentation/arm64/memory-tagging-extension.rst. + endmenu config ARM64_SVE
Catalin Marinas
maintainerPATCH v5 25/25 arm64: mte: Add Memory Tagging Extension documentation
In reply to Catalin MarinasFrom: Vincenzo Frascino <[email protected]>
Memory Tagging Extension (part of the ARMv8.5 Extensions) provides a mechanism to detect the sources of memory related errors which may be vulnerable to exploitation, including bounds violations, use-after-free, use-after-return, use-out-of-scope and use before initialization errors.
Add Memory Tagging Extension documentation for the arm64 linux kernel support.
---
Notes:
v4:
- Document behaviour of madvise(MADV_DONTNEED/MADV_FREE).
- Document the initial process state on fork/execve.
- Clarify when the kernel uaccess checks the tags.
- Minor updates to the example code.
- A few other minor clean-ups following review.
v3:
- Modify the uaccess checking conditions: only when the sync mode is selected by the user. In async mode, the kernel uaccesses are not checked.
- Clarify that an include mask of 0 (exclude mask 0xffff) results in always generating tag 0.
- Document the ptrace() interface.
v2:
- Documented the uaccess kernel tag checking mode.
- Removed the BTI definitions from cpu-feature-registers.rst.
- Removed the paragraph stating that MTE depends on the tagged address ABI (while the Kconfig entry does, there is no requirement for the user to enable both).
- Changed the GCR_EL1.Exclude handling description following the change in the prctl() interface (include vs exclude mask).
- Updated the example code.
Documentation/arm64/cpu-feature-registers.rst | 2 + Documentation/arm64/elf_hwcaps.rst | 4 + Documentation/arm64/index.rst | 1 + .../arm64/memory-tagging-extension.rst | 297 ++++++++++++++++++ 4 files changed, 304 insertions(+) create mode 100644 Documentation/arm64/memory-tagging-extension.rst
Documentation/arm64/cpu-feature-registers.rst +2 -0
@@ -174,6 +174,8 @@ infrastructure: +------------------------------+---------+---------+ | Name | bits | visible | +------------------------------+---------+---------+ + | MTE | [11-8] | y | + +------------------------------+---------+---------+ | SSBS | [7-4] | y | +------------------------------+---------+---------+ | BT | [3-0] | y |
Documentation/arm64/elf_hwcaps.rst +4 -0
@@ -240,6 +240,10 @@ HWCAP2_BTI Functionality implied by ID_AA64PFR0_EL1.BT == 0b0001. +HWCAP2_MTE + + Functionality implied by ID_AA64PFR1_EL1.MTE == 0b0010, as described + by Documentation/arm64/memory-tagging-extension.rst. 4. Unused AT_HWCAP bits -----------------------
Documentation/arm64/index.rst +1 -0
@@ -14,6 +14,7 @@ ARM64 Architecture hugetlbpage legacy_instructions memory + memory-tagging-extension pointer-authentication silicon-errata sve
Documentation/arm64/memory-tagging-extension.rst new file +297 -0
297 lines not on this page · raw message
Re: [PATCH v5 25/25] arm64: mte: Add Memory Tagging Extension documentation
Quoted 8 lines · Catalin Marinas
From: Vincenzo Frascino <[email protected]>
Memory Tagging Extension (part of the ARMv8.5 Extensions) provides a mechanism to detect the sources of memory related errors which may be vulnerable to exploitation, including bounds violations, use-after-free, use-after-return, use-out-of-scope and use before initialization errors.
Quoted 7 lines
Add Memory Tagging Extension documentation for the arm64 linux kernel support.
Signed-off-by Vincenzo Frascino <[email protected]>Co-developed-by Catalin Marinas <[email protected]>Signed-off-by Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>
there are are still libc side discussions, but the linux abi looks ok to me from user space pov. i'm adding libc-alpha on cc, the patch set is e.g. at
http://lists.infradead.org/pipermail/linux-arm-kernel/2020-June/579787.html
with a few comments inline.
Quoted 53 lines
---
Notes:
v4:
- Document behaviour ofmadvise(MADV_DONTNEED/MADV_FREE).
- Document the initial process state on fork/execve.
- Clarify when the kernel uaccess checks the tags.
- Minor updates to the example code.
- A few other minor clean-ups following review.v3:
- Modify the uaccess checking conditions: only when the sync mode is selected by the user. In async mode, the kernel uaccesses are not checked.
- Clarify that an include mask of 0 (exclude mask0xffff) results in always generating tag 0.
- Document theptrace()interface.v2:
- Documented the uaccess kernel tag checking mode.
- Removed the BTI definitions fromcpu-feature-registers.rst.
- Removed the paragraph stating that MTE depends on the tagged address ABI (while theKconfigentry does, there is no requirement for the user to enable both).
- Changed the GCR_EL1.Exclude handling description following the change in theprctl()interface (include vs exclude mask).
- Updated the example code.Documentation/arm64/cpu-feature-registers.rst | 2 + Documentation/arm64/elf_hwcaps.rst | 4 + Documentation/arm64/index.rst | 1 + .../arm64/memory-tagging-extension.rst | 297 ++++++++++++++++++ 4 files changed, 304 insertions(+) create mode 100644 Documentation/arm64/memory-tagging-extension.rstDocumentation/arm64/cpu-feature-registers.rst +2
-0@@ -174,6 +174,8 @@ infrastructure: +------------------------------+---------+---------+ | Name | bits | visible | +------------------------------+---------+---------+ + | MTE | [11-8] | y | + +------------------------------+---------+---------+ | SSBS | [7-4] | y | +------------------------------+---------+---------+ | BT | [3-0] | y |Documentation/arm64/elf_hwcaps.rst +0
-0
Documentation/arm64/elf_hwcaps.rst HWCAP2_BTI · line 246
Quoted 7 lines
Documentation/arm64/elf_hwcaps.rst +4
-0@@ -240,6 +240,10 @@ HWCAP2_BTI Functionality implied by ID_AA64PFR0_EL1.BT == 0b0001. +HWCAP2_MTE + + Functionality implied by ID_AA64PFR1_EL1.MTE == 0b0010, as described + by Documentation/arm64/memory-tagging-extension.rst.
OK.
Quoted 42 lines
Documentation/arm64/elf_hwcaps.rst +0
-04. Unused AT_HWCAP bits -----------------------Documentation/arm64/index.rst +1
-0@@ -14,6 +14,7 @@ ARM64 Architecture hugetlbpage legacy_instructions memory + memory-tagging-extension pointer-authentication silicon-errata sveDocumentation/arm64/memory-tagging-extension.rst new file +21
-0@@ -0,0 +1,297 @@ +=============================================== +Memory Tagging Extension (MTE) in AArch64 Linux +=============================================== + +Authors: Vincenzo Frascino <[email protected]> + Catalin Marinas <[email protected]> + +Date: 2020-02-25 + +This document describes the provision of the Memory Tagging Extension +functionality in AArch64 Linux. + +Introduction +============ + +ARMv8.5 based processors introduce the Memory Tagging Extension (MTE) +feature. MTE is built on top of the ARMv8.0 virtual address tagging TBI +(Top Byte Ignore) feature and allows software to access a 4-bit +allocation tag for each 16-byte granule in the physical address space. +Such memory range must be mapped with the Normal-Tagged memory +attribute. A logical tag is derived from bits 59-56 of the virtual
Documentation/arm64/memory-tagging-extension.rst line 31
Quoted 10 lines
Documentation/arm64/memory-tagging-extension.rst new file +10
-0+address used for the memory access. A CPU with MTE enabled will compare +the logical tag against the allocation tag and potentially raise an +exception on mismatch, subject to system registers configuration. + +Userspace Support +================= + +When ``CONFIG_ARM64_MTE`` is selected and Memory Tagging Extension is +supported by the hardware, the kernel advertises the feature to +userspace via ``HWCAP2_MTE``.
OK.
Quoted 15 lines
Documentation/arm64/memory-tagging-extension.rst new file +15
-0+ +PROT_MTE +-------- + +To access the allocation tags, a user process must enable the Tagged +memory attribute on an address range using a new ``prot`` flag for +``mmap()`` and ``mprotect()``: + +``PROT_MTE`` - Pages allow access to the MTE allocation tags. + +The allocation tag is set to 0 when such pages are first mapped in the +user address space and preserved on copy-on-write. ``MAP_SHARED`` is +supported and the allocation tags can be shared between processes. + +**Note**: ``PROT_MTE`` is only supported on ``MAP_ANONYMOUS`` and
Documentation/arm64/memory-tagging-extension.rst line 56
Quoted 10 lines
Documentation/arm64/memory-tagging-extension.rst new file +10
-0+RAM-based file mappings (``tmpfs``, ``memfd``). Passing it to other +types of mapping will result in ``-EINVAL`` returned by these system +calls. + +**Note**: The ``PROT_MTE`` flag (and corresponding memory type) cannot +be cleared by ``mprotect()``. + +**Note**: ``madvise()`` memory ranges with ``MADV_DONTNEED`` and +``MADV_FREE`` may have the allocation tags cleared (set to 0) at any +point after the system call.
OK.
I expect in the future to have a way to query the PROT_MTE status of mappings (e.g. via /proc/self).
The MAP_SHARED behaviour is not entirely clear here but i guess it's possible to have PROT_MTE in one process and no PROT_MTE in others on the same mapping. then allocation tags only affect the process where PROT_MTE was used, later on another process may set PROT_MTE and then the shared allocation tags affect that process too.
The madvise behaviour looks a bit risky from user space pov since now it's not just the memory content that can disappear after a MADV_DONTNEED, but pointer to that memory can become invalid too. but i think this is OK: in libc we will have to say that madvise on memory returned by malloc is not valid.
As noted before, this design is not ideal for stack tagging (mprotecting the initial stack with PROT_MTE may be problematic if we don't know the bounds), but the expectation is to introduce some ELF marking and then linux can just start the process with PROT_MTE stack if the dynamic linker has the marking. Same for the brk area (default PROT_MTE based on ELF marking).
Quoted 39 lines
Documentation/arm64/memory-tagging-extension.rst new file +39
-0+ +Tag Check Faults +---------------- + +When ``PROT_MTE`` is enabled on an address range and a mismatch between +the logical and allocation tags occurs on access, there are three +configurable behaviours: + +- *Ignore* - This is the default mode. The CPU (and kernel) ignores the + tag check fault. + +- *Synchronous* - The kernel raises a ``SIGSEGV`` synchronously, with + ``.si_code = SEGV_MTESERR`` and ``.si_addr = <fault-address>``. The + memory access is not performed. If ``SIGSEGV`` is ignored or blocked + by the offending thread, the containing process is terminated with a + ``coredump``. + +- *Asynchronous* - The kernel raises a ``SIGSEGV``, in the offending + thread, asynchronously following one or multiple tag check faults, + with ``.si_code = SEGV_MTEAERR`` and ``.si_addr = 0`` (the faulting + address is unknown). + +The user can select the above modes, per thread, using the +``prctl(PR_SET_TAGGED_ADDR_CTRL, flags, 0, 0, 0)`` system call where +``flags`` contain one of the following values in the ``PR_MTE_TCF_MASK`` +bit-field: + +- ``PR_MTE_TCF_NONE`` - *Ignore* tag check faults +- ``PR_MTE_TCF_SYNC`` - *Synchronous* tag check fault mode +- ``PR_MTE_TCF_ASYNC`` - *Asynchronous* tag check fault mode + +The current tag check fault mode can be read using the +``prctl(PR_GET_TAGGED_ADDR_CTRL, 0, 0, 0, 0)`` system call. + +Tag checking can also be disabled for a user thread by setting the +``PSTATE.TCO`` bit with ``MSR TCO, #1``. + +**Note**: Signal handlers are always invoked with ``PSTATE.TCO = 0``, +irrespective of the interrupted context. ``PSTATE.TCO`` is restored on
Documentation/arm64/memory-tagging-extension.rst line 105
Quoted 10 lines
Documentation/arm64/memory-tagging-extension.rst new file +10
-0+``sigreturn()``. + +**Note**: There are no *match-all* logical tags available for user +applications. + +**Note**: Kernel accesses to the user address space (e.g. ``read()`` +system call) are not checked if the user thread tag checking mode is +``PR_MTE_TCF_NONE`` or ``PR_MTE_TCF_ASYNC``. If the tag checking mode is +``PR_MTE_TCF_SYNC``, the kernel makes a best effort to check its user +address accesses, however it cannot always guarantee it.
OK.
i know the kernel likes to operate on os-threads, but in userspace this causes the slight wart that if somebody wants to use heap tagging with LD_PRELOADed malloc and the first malloc is called after a thread is already created then the malloc implementation cannot set up the prctl right for all threads in the process. (for userspace i think it is only useful to allow threads with different MTE settings if there are some threads in a process that are not managed by the c runtime and don't call into libc, so as far as normal c code is concerned a per process setting would be nicer). for interposers the workaround is to interpose thread creating libc apis, which is not perfect (libc internally may create threads in not interposable ways e.g. for implementing aio and then use heap memory in such threads), but i think early threads before an LD_PRELOAD initializer may run is not a common scenario and this type of MTE usage is for debugging, i.e. does not have to be perfect.
as noted before (i think by Kevin) it would be nice to query the tag check status of other threads e.g. via a /proc/ thing (but i don't see an immediate need for this other than debugging MTE faults).
Quoted 4 lines
Documentation/arm64/memory-tagging-extension.rst new file +4
-0+ +Excluding Tags in the ``IRG``, ``ADDG`` and ``SUBG`` instructions +----------------------------------------------------------------- +
Documentation/arm64/memory-tagging-extension.rst line 119
Quoted 10 lines
Documentation/arm64/memory-tagging-extension.rst new file +10
-0+The architecture allows excluding certain tags to be randomly generated +via the ``GCR_EL1.Exclude`` register bit-field. By default, Linux +excludes all tags other than 0. A user thread can enable specific tags +in the randomly generated set using the ``prctl(PR_SET_TAGGED_ADDR_CTRL, +flags, 0, 0, 0)`` system call where ``flags`` contains the tags bitmap +in the ``PR_MTE_TAG_MASK`` bit-field. + +**Note**: The hardware uses an exclude mask but the ``prctl()`` +interface provides an include mask. An include mask of ``0`` (exclusion +mask ``0xffff``) results in the CPU always generating tag ``0``.
OK.
Quoted 6 lines
Documentation/arm64/memory-tagging-extension.rst new file +6
-0+ +Initial process state +--------------------- + +On ``execve()``, the new process has the following configuration: +
Documentation/arm64/memory-tagging-extension.rst line 135
Quoted 10 lines
Documentation/arm64/memory-tagging-extension.rst new file +10
-0+- ``PR_TAGGED_ADDR_ENABLE`` set to 0 (disabled) +- Tag checking mode set to ``PR_MTE_TCF_NONE`` +- ``PR_MTE_TAG_MASK`` set to 0 (all tags excluded) +- ``PSTATE.TCO`` set to 0 +- ``PROT_MTE`` not set on any of the initial memory maps + +On ``fork()``, the new process inherits the parent's configuration and +memory map attributes with the exception of the ``madvise()`` ranges +with ``MADV_WIPEONFORK`` which will have the data and tags cleared (set +to 0).
OK.
Quoted 28 lines
Documentation/arm64/memory-tagging-extension.rst new file +28
-0+ +The ``ptrace()`` interface +-------------------------- + +``PTRACE_PEEKMTETAGS`` and ``PTRACE_POKEMTETAGS`` allow a tracer to read +the tags from or set the tags to a tracee's address space. The +``ptrace()`` system call is invoked as ``ptrace(request, pid, addr, +data)`` where: + +- ``request`` - one of ``PTRACE_PEEKMTETAGS`` or ``PTRACE_PEEKMTETAGS``. +- ``pid`` - the tracee's PID. +- ``addr`` - address in the tracee's address space. +- ``data`` - pointer to a ``struct iovec`` where ``iov_base`` points to + a buffer of ``iov_len`` length in the tracer's address space. + +The tags in the tracer's ``iov_base`` buffer are represented as one +4-bit tag per byte and correspond to a 16-byte MTE tag granule in the +tracee's address space. + +**Note**: If ``addr`` is not aligned to a 16-byte granule, the kernel +will use the corresponding aligned address. + +``ptrace()`` return value: + +- 0 - tags were copied, the tracer's ``iov_len`` was updated to the + number of tags transferred. This may be smaller than the requested + ``iov_len`` if the requested address range in the tracee's or the + tracer's space cannot be accessed or does not have valid tags.
Documentation/arm64/memory-tagging-extension.rst line 173
Quoted 10 lines
Documentation/arm64/memory-tagging-extension.rst new file +10
-0+- ``-EPERM`` - the specified process cannot be traced. +- ``-EIO`` - the tracee's address range cannot be accessed (e.g. invalid + address) and no tags copied. ``iov_len`` not updated. +- ``-EFAULT`` - fault on accessing the tracer's memory (``struct iovec`` + or ``iov_base`` buffer) and no tags copied. ``iov_len`` not updated. +- ``-EOPNOTSUPP`` - the tracee's address does not have valid tags (never + mapped with the ``PROT_MTE`` flag). ``iov_len`` not updated. + +**Note**: There are no transient errors for the requests above, so user +programs should not retry in case of a non-zero system call return.
looks OK.
Quoted 38 lines
Documentation/arm64/memory-tagging-extension.rst new file +38
-0+ +Example of correct usage +======================== + +*MTE Example code* + +.. code-block:: c + + /* + * To be compiled with -march=armv8.5-a+memtag + */ + #include <errno.h> + #include <stdio.h> + #include <stdlib.h> + #include <unistd.h> + #include <sys/auxv.h> + #include <sys/mman.h> + #include <sys/prctl.h> + + /* + * From arch/arm64/include/uapi/asm/hwcap.h + */ + #define HWCAP2_MTE (1 << 18) + + /* + * From arch/arm64/include/uapi/asm/mman.h + */ + #define PROT_MTE 0x20 + + /* + * From include/uapi/linux/prctl.h + */ + #define PR_SET_TAGGED_ADDR_CTRL 55 + #define PR_GET_TAGGED_ADDR_CTRL 56 + # define PR_TAGGED_ADDR_ENABLE (1UL << 0) + # define PR_MTE_TCF_SHIFT 1 + # define PR_MTE_TCF_NONE (0UL << PR_MTE_TCF_SHIFT) + # define PR_MTE_TCF_SYNC (1UL << PR_MTE_TCF_SHIFT)
Documentation/arm64/memory-tagging-extension.rst line 221
Quoted 10 lines
Documentation/arm64/memory-tagging-extension.rst new file +10
-0+ # define PR_MTE_TCF_ASYNC (2UL << PR_MTE_TCF_SHIFT) + # define PR_MTE_TCF_MASK (3UL << PR_MTE_TCF_SHIFT) + # define PR_MTE_TAG_SHIFT 3 + # define PR_MTE_TAG_MASK (0xffffUL << PR_MTE_TAG_SHIFT) + + /* + * Insert a random logical tag into the given pointer. + */ + #define insert_random_tag(ptr) ({ \ + __u64 __val; \
i'd use uint64_t from stdint.h or unsigned long in the example (i.e. not a kernel type)
Quoted 66 lines
Documentation/arm64/memory-tagging-extension.rst new file +66
-0+ asm("irg %0, %1" : "=r" (__val) : "r" (ptr)); \ + __val; \ + }) + + /* + * Set the allocation tag on the destination address. + */ + #define set_tag(tagged_addr) do { \ + asm volatile("stg %0, [%0]" : : "r" (tagged_addr) : "memory"); \ + } while (0) + + int main() + { + unsigned char *a; + unsigned long page_sz = sysconf(_SC_PAGESIZE); + unsigned long hwcap2 = getauxval(AT_HWCAP2); + + /* check if MTE is present */ + if (!(hwcap2 & HWCAP2_MTE)) + return EXIT_FAILURE; + + /* + * Enable the tagged address ABI, synchronous MTE tag check faults and + * allow all non-zero tags in the randomly generated set. + */ + if (prctl(PR_SET_TAGGED_ADDR_CTRL, + PR_TAGGED_ADDR_ENABLE | PR_MTE_TCF_SYNC | (0xfffe << PR_MTE_TAG_SHIFT), + 0, 0, 0)) { + perror("prctl() failed"); + return EXIT_FAILURE; + } + + a = mmap(0, page_sz, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (a == MAP_FAILED) { + perror("mmap() failed"); + return EXIT_FAILURE; + } + + /* + * Enable MTE on the above anonymous mmap. The flag could be passed + * directly to mmap() and skip this step. + */ + if (mprotect(a, page_sz, PROT_READ | PROT_WRITE | PROT_MTE)) { + perror("mprotect() failed"); + return EXIT_FAILURE; + } + + /* access with the default tag (0) */ + a[0] = 1; + a[1] = 2; + + printf("a[0] = %hhu a[1] = %hhu\n", a[0], a[1]); + + /* set the logical and allocation tags */ + a = (unsigned char *)insert_random_tag(a); + set_tag(a); + + printf("%p\n", a); + + /* non-zero tag access */ + a[0] = 3; + printf("a[0] = %hhu a[1] = %hhu\n", a[0], a[1]); + + /* + * If MTE is enabled correctly the next instruction will generate an
Documentation/arm64/memory-tagging-extension.rst line 297
Quoted 10 lines
Documentation/arm64/memory-tagging-extension.rst new file +10
-0+ * exception. + */ + printf("Expecting SIGSEGV...\n"); + a[16] = 0xdd; + + /* this should not be printed in the PR_MTE_TCF_SYNC mode */ + printf("...haven't got one\n"); + + return EXIT_FAILURE; + }
OK.
Catalin Marinas
maintainerRe: [PATCH v5 25/25] arm64: mte: Add Memory Tagging Extension documentation
Hi Szabolcs,
Quoted 17 lines · Szabolcs Nagy
Quoted 8 lines · Catalin Marinas
From: Vincenzo Frascino <[email protected]>
Memory Tagging Extension (part of the ARMv8.5 Extensions) provides a mechanism to detect the sources of memory related errors which may be vulnerable to exploitation, including bounds violations, use-after-free, use-after-return, use-out-of-scope and use before initialization errors.
Quoted 7 lines
Add Memory Tagging Extension documentation for the arm64 linux kernel support.
Signed-off-by Vincenzo Frascino <[email protected]>Co-developed-by Catalin Marinas <[email protected]>Signed-off-by Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>
Quoted 6 lines
there are are still libc side discussions, but the linux abi looks ok to me from user space pov. i'm adding libc-alpha on cc, the patch set is e.g. at
http://lists.infradead.org/pipermail/linux-arm-kernel/2020-June/579787.htmlAcked-by Szabolcs Nagy <[email protected]>
Thanks for the review. If there are any ABI changes required as a result of the libc-alpha discussions, please let me know.
Quoted 27 lines
Quoted 14 lines
Documentation/arm64/memory-tagging-extension.rst new file +14
-0+PROT_MTE +-------- + +To access the allocation tags, a user process must enable the Tagged +memory attribute on an address range using a new ``prot`` flag for +``mmap()`` and ``mprotect()``: + +``PROT_MTE`` - Pages allow access to the MTE allocation tags. + +The allocation tag is set to 0 when such pages are first mapped in the +user address space and preserved on copy-on-write. ``MAP_SHARED`` is +supported and the allocation tags can be shared between processes. + +**Note**: ``PROT_MTE`` is only supported on ``MAP_ANONYMOUS`` andDocumentation/arm64/memory-tagging-extension.rst line 56
Quoted 10 lines
Documentation/arm64/memory-tagging-extension.rst new file +10
-0+RAM-based file mappings (``tmpfs``, ``memfd``). Passing it to other +types of mapping will result in ``-EINVAL`` returned by these system +calls. + +**Note**: The ``PROT_MTE`` flag (and corresponding memory type) cannot +be cleared by ``mprotect()``. + +**Note**: ``madvise()`` memory ranges with ``MADV_DONTNEED`` and +``MADV_FREE`` may have the allocation tags cleared (set to 0) at any +point after the system call.OK.
Quoted 2 lines
I expect in the future to have a way to query the
PROT_MTEstatus of mappings (e.g. via/proc/self).
Currently you can do this via /proc/<pid>/smaps.
Quoted 3 lines
The
MAP_SHAREDbehaviour is not entirely clear here but i guess it's possible to havePROT_MTEin one process and noPROT_MTEin others on the same mapping.
Yes, it is.
Quoted 4 lines
then allocation tags only affect the process where
PROT_MTEwas used, later on another process may setPROT_MTEand then the shared allocation tags affect that process too.
Yes. Since PROT_MTE allows access to the allocation tags, each process can control it independently.
Quoted 6 lines
The madvise behaviour looks a bit risky from user space pov since now it's not just the memory content that can disappear after a
MADV_DONTNEED, but pointer to that memory can become invalid too. but i think this is OK: in libc we will have to say that madvise on memory returned by malloc is not valid.
From a kernel perspective, it never returned a tagged pointer on mmap(), so reverting the allocation tag to 0 is fine. I don't really have a better solution here other than not calling madvise() on malloc'ed memory.
A more invasive option may be to return tagged pointers on mmap() and guarantee that the libc will not change them. A subsequent access on DONTNEED memory would restore the original colour.
Quoted 3 lines
As noted before, this design is not ideal for stack tagging (mprotecting the initial stack with
PROT_MTEmay be problematic if we don't know the bounds),
I don't think you need the bounds (with PROT_GROWSDOWN). Maybe the upper one but functions up the call chain should not use stack tagging anyway.
Quoted 5 lines
but
the expectation is to introduce some ELF marking and then linux can just start the process withPROT_MTEstack if the dynamic linker has the marking. Same for the brk area (defaultPROT_MTEbased on ELF marking).
This should work. Since stack tagging cannot use instructions in the NOP space anyway and the program needs recompiling, having an ELF marking would help (for heap tagging, you only need to change the libc and dynamic loader). I think we do similar checks for BTI.
Quoted 51 lines
Quoted 38 lines
Documentation/arm64/memory-tagging-extension.rst new file +38
-0+Tag Check Faults +---------------- + +When ``PROT_MTE`` is enabled on an address range and a mismatch between +the logical and allocation tags occurs on access, there are three +configurable behaviours: + +- *Ignore* - This is the default mode. The CPU (and kernel) ignores the + tag check fault. + +- *Synchronous* - The kernel raises a ``SIGSEGV`` synchronously, with + ``.si_code = SEGV_MTESERR`` and ``.si_addr = <fault-address>``. The + memory access is not performed. If ``SIGSEGV`` is ignored or blocked + by the offending thread, the containing process is terminated with a + ``coredump``. + +- *Asynchronous* - The kernel raises a ``SIGSEGV``, in the offending + thread, asynchronously following one or multiple tag check faults, + with ``.si_code = SEGV_MTEAERR`` and ``.si_addr = 0`` (the faulting + address is unknown). + +The user can select the above modes, per thread, using the +``prctl(PR_SET_TAGGED_ADDR_CTRL, flags, 0, 0, 0)`` system call where +``flags`` contain one of the following values in the ``PR_MTE_TCF_MASK`` +bit-field: + +- ``PR_MTE_TCF_NONE`` - *Ignore* tag check faults +- ``PR_MTE_TCF_SYNC`` - *Synchronous* tag check fault mode +- ``PR_MTE_TCF_ASYNC`` - *Asynchronous* tag check fault mode + +The current tag check fault mode can be read using the +``prctl(PR_GET_TAGGED_ADDR_CTRL, 0, 0, 0, 0)`` system call. + +Tag checking can also be disabled for a user thread by setting the +``PSTATE.TCO`` bit with ``MSR TCO, #1``. + +**Note**: Signal handlers are always invoked with ``PSTATE.TCO = 0``, +irrespective of the interrupted context. ``PSTATE.TCO`` is restored onDocumentation/arm64/memory-tagging-extension.rst line 105
Quoted 10 lines
Documentation/arm64/memory-tagging-extension.rst new file +10
-0+``sigreturn()``. + +**Note**: There are no *match-all* logical tags available for user +applications. + +**Note**: Kernel accesses to the user address space (e.g. ``read()`` +system call) are not checked if the user thread tag checking mode is +``PR_MTE_TCF_NONE`` or ``PR_MTE_TCF_ASYNC``. If the tag checking mode is +``PR_MTE_TCF_SYNC``, the kernel makes a best effort to check its user +address accesses, however it cannot always guarantee it.OK.
Quoted 7 lines
i know the kernel likes to operate on os-threads, but in userspace this causes the slight wart that if somebody wants to use heap tagging with
LD_PRELOADedmalloc and the first malloc is called after a thread is already created then the malloc implementation cannot set up the prctl right for all threads in the process.
Ah, so you can't have a constructor called with LD_PRELOAD.
Quoted 6 lines
(for userspace i think it is only useful to allow threads with different MTE settings if there are some threads in a process that are not managed by the c runtime and don't call into libc, so as far as normal c code is concerned a per process setting would be nicer).
My assumption was that the c runtime would set this up and all threads inherit the initial configuration. How important is the LD_PRELOAD use-case?
The slight trouble with having this setting global is synchronising all the threads. Maybe if we only allow single global configuration (rather than having the option of per-thread and global), user-space could force the synchronisation with something like membarrier().
Quoted 8 lines
for interposers the workaround is
to interpose thread creating libc apis, which is not perfect (libc internally may create threads in not interposable ways e.g. for implementing aio and then use heap memory in such threads), but i think early threads before anLD_PRELOADinitializer may run is not a common scenario and this type of MTE usage is for debugging, i.e. does not have to be perfect.
Feedback welcome, both from the glibc and the bionic/Android camps.
Quoted 4 lines
as noted before (i think by Kevin) it would be nice to query the tag check status of other threads e.g. via a
/proc/thing (but i don't see an immediate need for this other than debugging MTE faults).
I have a plan to add some information in /proc/<pid>/status at some point.
Signature · 1 line
Catalin