arm64: Memory Tagging Extension user-space support
Subsystems (16)
Catalin Marinas
maintainerv4 cover arm64: Memory Tagging Extension user-space support
This is the fourth version (third version here [1]) of the series adding user-space support for the ARMv8.5 Memory Tagging Extension ([2], [3]). The patches are also available on this branch:
git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux devel/mte-v4
Changes in this version:
- Swap and suspend to disk support for saving/restoring tags.
- Deferred the tag zeroing from clear_page() to set_pte_at() to cope with RAM filesystems that do not start from a zeroed page. The copy_page() function was also optimised to only copy tags if the page has been previously mapped as tagged (PROT_MTE). This mechanism requires a new PG_arch_2 flag.
- memcmp_pages() rewritten to always return a no-match if at least one of the pages is tagged (tag comparison avoided).
- ptrace() updated to prevent accessing tags in a page that has not been mapped with PROT_MTE (PG_arch_2 flag not set).
- copy_mount_options() fix re-implemented to avoid arch_has_exact_copy_from_user().
- The CPUID handling has been reworked to ensure that, when the feature is not backed by the DT, the HWCAP is also hidden from user. Note that the DT description is still under internal discussion on whether we need it or not.
- A new early param, arm64.mte_disable, was introduced to facilitate testing with and without MTE on platforms that support it.
- Asynchronous TCF SIGSEGV is no longer forced, so it can be ignored but the user thread.
- CONFIG_ARM64_MTE is now default y since swap is supported.
To do or discuss:
- prctl() accepting an include vs exclude mask for the GCR_EL1.Excl field. There is an ongoing discussion on v3 which accepts an include mask with 0 being a special case equivalent to 1. If this is not desirable, we can change this to an exclude mask.
- mmap(tagged_addr, PROT_MTE) pre-tagging the memory with the tag given in the tagged_addr hint.
- ptrace() to expose the prctl() configuration for the user thread (or the TCF and GCR_EL1.Excl fields).
- coredump (user) to also dump the tags.
- Kselftest patches will be made available.
[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/
[2] https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/enhancing-memory-safety
[3] https://developer.arm.com/-/media/Arm%20Developer%20Community/PDF/Arm_Memory_Tagging_Extension_Whitepaper.pdf
Catalin Marinas (14):
arm64: mte: Use Normal Tagged attributes for the linear map
arm64: mte: Clear the tags when a page is mapped in user-space with
PROT_MTE
arm64: mte: Tags-aware aware memcmp_pages() implementation
arm64: mte: Add PROT_MTE support to mmap() and mprotect()
mm: Introduce arch_validate_flags()
arm64: mte: Validate the PROT_MTE request via arch_validate_flags()
mm: Allow arm64 mmap(PROT_MTE) on RAM-based files
arm64: mte: Allow user control of the tag check mode via prctl()
arm64: mte: Allow user control of the generated random tags via
prctl()
arm64: mte: Restore the GCR_EL1 register after a suspend
arm64: mte: Add PTRACE_{PEEK,POKE}MTETAGS support
fs: Handle intra-page faults in copy_mount_options()
arm64: mte: Check the DT memory nodes for MTE support
arm64: mte: Introduce early param to disable MTE support
Kevin Brodsky (1):
mm: Introduce arch_calc_vm_flag_bits()
Steven Price (4):
mm: Add PG_ARCH_2 page flag
mm: Add arch hooks for saving/restoring tags
arm64: mte: Enable swap of tagged pages
arm64: mte: Save tags when hibernating
Vincenzo Frascino (7):
arm64: mte: system register definitions
arm64: mte: CPU feature detection and initial sysreg configuration
arm64: mte: Add specific SIGSEGV codes
arm64: mte: Handle synchronous and asynchronous tag check faults
arm64: mte: Tags-aware copy_page() implementation
arm64: mte: Kconfig entry
arm64: mte: Add Memory Tagging Extension documentation
.../admin-guide/kernel-parameters.txt | 4 + Documentation/arm64/cpu-feature-registers.rst | 2 + Documentation/arm64/elf_hwcaps.rst | 5 + Documentation/arm64/index.rst | 1 + .../arm64/memory-tagging-extension.rst | 297 ++++++++++++++++ arch/arm64/Kconfig | 33 ++ arch/arm64/boot/dts/arm/fvp-base-revc.dts | 1 + arch/arm64/include/asm/assembler.h | 12 + arch/arm64/include/asm/cpucaps.h | 4 +- arch/arm64/include/asm/cpufeature.h | 12 +- arch/arm64/include/asm/hwcap.h | 1 + arch/arm64/include/asm/kvm_arm.h | 3 +- arch/arm64/include/asm/memory.h | 17 +- arch/arm64/include/asm/mman.h | 78 +++++ arch/arm64/include/asm/mte.h | 86 +++++ arch/arm64/include/asm/page.h | 2 +- arch/arm64/include/asm/pgtable-prot.h | 2 + arch/arm64/include/asm/pgtable.h | 45 ++- arch/arm64/include/asm/processor.h | 4 + arch/arm64/include/asm/sysreg.h | 62 ++++ arch/arm64/include/asm/thread_info.h | 4 +- arch/arm64/include/uapi/asm/hwcap.h | 2 + arch/arm64/include/uapi/asm/mman.h | 14 + arch/arm64/include/uapi/asm/ptrace.h | 4 + arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/cpufeature.c | 126 ++++++- arch/arm64/kernel/cpuinfo.c | 2 + arch/arm64/kernel/entry.S | 37 ++ arch/arm64/kernel/hibernate.c | 118 +++++++ arch/arm64/kernel/mte.c | 324 ++++++++++++++++++ arch/arm64/kernel/process.c | 31 +- arch/arm64/kernel/ptrace.c | 9 +- arch/arm64/kernel/signal.c | 8 + arch/arm64/kernel/suspend.c | 4 + arch/arm64/kernel/syscall.c | 10 + arch/arm64/lib/Makefile | 2 + arch/arm64/lib/mte.S | 140 ++++++++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/copypage.c | 14 +- arch/arm64/mm/dump.c | 4 + arch/arm64/mm/fault.c | 9 +- arch/arm64/mm/mmu.c | 22 +- arch/arm64/mm/mteswap.c | 82 +++++ arch/arm64/mm/proc.S | 8 +- arch/x86/kernel/signal_compat.c | 2 +- fs/namespace.c | 24 +- fs/proc/page.c | 3 + fs/proc/task_mmu.c | 4 + include/asm-generic/pgtable.h | 23 ++ include/linux/kernel-page-flags.h | 1 + include/linux/mm.h | 8 + include/linux/mman.h | 22 +- include/linux/page-flags.h | 3 + include/trace/events/mmflags.h | 9 +- include/uapi/asm-generic/siginfo.h | 4 +- include/uapi/linux/prctl.h | 9 + mm/Kconfig | 3 + mm/mmap.c | 9 + mm/mprotect.c | 6 + mm/page_io.c | 10 + mm/shmem.c | 9 + mm/swapfile.c | 2 + mm/util.c | 2 +- tools/vm/page-types.c | 2 + 64 files changed, 1765 insertions(+), 37 deletions(-) create mode 100644 Documentation/arm64/memory-tagging-extension.rst create mode 100644 arch/arm64/include/asm/mman.h create mode 100644 arch/arm64/include/asm/mte.h create mode 100644 arch/arm64/include/uapi/asm/mman.h create mode 100644 arch/arm64/kernel/mte.c create mode 100644 arch/arm64/lib/mte.S create mode 100644 arch/arm64/mm/mteswap.c
Catalin Marinas
maintainerPATCH v4 1/26 arm64: mte: system register definitions
From: Vincenzo Frascino <[email protected]>
Add Memory Tagging Extension system register definitions together with the relevant bitfields.
---
Notes:
v2:
- Added SET_PSTATE_TCO() macro.
arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/include/asm/sysreg.h | 54 ++++++++++++++++++++++++++++ arch/arm64/include/uapi/asm/ptrace.h | 1 + arch/arm64/kernel/ptrace.c | 2 +- 4 files changed, 57 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_arm.h +1 -0
@@ -12,6 +12,7 @@#include <asm/types.h> /* Hyp Configuration Register (HCR) bits */ +#define HCR_ATA (UL(1) << 56)#define HCR_FWB (UL(1) << 46)#define HCR_API (UL(1) << 41)#define HCR_APK (UL(1) << 40)
arch/arm64/include/asm/sysreg.h +54 -0
129 lines not on this page · raw message
arch/arm64/include/uapi/asm/ptrace.h +1 -0
@@ -50,6 +50,7 @@#define PSR_PAN_BIT 0x00400000#define PSR_UAO_BIT 0x00800000#define PSR_DIT_BIT 0x01000000 +#define PSR_TCO_BIT 0x02000000#define PSR_V_BIT 0x10000000#define PSR_C_BIT 0x20000000#define PSR_Z_BIT 0x40000000
arch/arm64/kernel/ptrace.c +1 -1
@@ -1873,7 +1873,7 @@ void syscall_trace_exit(struct pt_regs *regs)* We also reserve IL for the kernel; SS is handled dynamically. */ #define SPSR_EL1_AARCH64_RES0_BITS \ - (GENMASK_ULL(63, 32) | GENMASK_ULL(27, 25) | GENMASK_ULL(23, 22) | \ + (GENMASK_ULL(63, 32) | GENMASK_ULL(27, 26) | GENMASK_ULL(23, 22) | \ GENMASK_ULL(20, 13) | GENMASK_ULL(11, 10) | GENMASK_ULL(5, 5))#define SPSR_EL1_AARCH32_RES0_BITS \ (GENMASK_ULL(63, 32) | GENMASK_ULL(22, 22) | GENMASK_ULL(20, 20))
Catalin Marinas
maintainerPATCH v4 2/26 arm64: mte: CPU feature detection and initial sysreg configuration
In reply to Catalin MarinasFrom: Vincenzo Frascino <[email protected]>
Add the cpufeature and hwcap entries to detect the presence of MTE on the boot CPUs (primary and secondary). Any late secondary CPU not supporting the feature, if detected during boot, will be parked.
In addition, add the minimum SCTLR_EL1 and HCR_EL2 bits for enabling MTE. Without subsequent setting of MAIR, these bits do not have an effect on tag checking.
--- arch/arm64/include/asm/cpucaps.h | 4 +++- arch/arm64/include/asm/cpufeature.h | 6 ++++++ arch/arm64/include/asm/hwcap.h | 1 + arch/arm64/include/asm/kvm_arm.h | 2 +- arch/arm64/include/asm/sysreg.h | 1 + arch/arm64/include/uapi/asm/hwcap.h | 2 ++ arch/arm64/kernel/cpufeature.c | 30 +++++++++++++++++++++++++++++ arch/arm64/kernel/cpuinfo.c | 2 ++ 8 files changed, 46 insertions(+), 2 deletions(-)
arch/arm64/include/asm/cpucaps.h +3 -1
@@ -61,7 +61,9 @@#define ARM64_HAS_AMU_EXTN 51#define ARM64_HAS_ADDRESS_AUTH 52#define ARM64_HAS_GENERIC_AUTH 53 +/* 54 reserved for ARM64_BTI */ +#define ARM64_MTE 55 -#define ARM64_NCAPS 54 +#define ARM64_NCAPS 56#endif /* __ASM_CPUCAPS_H */
arch/arm64/include/asm/cpufeature.h +6 -0
@@ -674,6 +674,12 @@ static inline bool system_uses_irq_prio_masking(void)cpus_have_const_cap (ARM64_HAS_IRQ_PRIO_MASKING ); } +static inline bool system_supports_mte (void ) +{ +return IS_ENABLED (CONFIG_ARM64_MTE ) && +cpus_have_const_cap (ARM64_MTE ); +} +static inline bool system_has_prio_mask_debugging (void ) { return IS_ENABLED(CONFIG_ARM64_DEBUG_PRIORITY_MASKING) &&
arch/arm64/include/asm/hwcap.h +1 -0
@@ -94,6 +94,7 @@#define KERNEL_HWCAP_BF16 __khwcap2_feature(BF16)#define KERNEL_HWCAP_DGH __khwcap2_feature(DGH)#define KERNEL_HWCAP_RNG __khwcap2_feature(RNG) +#define KERNEL_HWCAP_MTE __khwcap2_feature(MTE)/* * This yields a mask that user programs can use to figure out what
arch/arm64/include/asm/kvm_arm.h +1 -1
@@ -78,7 +78,7 @@ HCR_AMO | HCR_SWIO | HCR_TIDCP | HCR_RW | HCR_TLOR | \ HCR_FMO | HCR_IMO)#define HCR_VIRT_EXCP_MASK (HCR_VSE | HCR_VI | HCR_VF) -#define HCR_HOST_NVHE_FLAGS (HCR_RW | HCR_API | HCR_APK) +#define HCR_HOST_NVHE_FLAGS (HCR_RW | HCR_API | HCR_APK | HCR_ATA)#define HCR_HOST_VHE_FLAGS (HCR_RW | HCR_TGE | HCR_E2H)/* TCR_EL2 Registers bits */
arch/arm64/include/asm/sysreg.h +1 -0
@@ -604,6 +604,7 @@SCTLR_EL1_SA0 |SCTLR_EL1_SED | SCTLR_ELx_I |\SCTLR_EL1_DZE |SCTLR_EL1_UCT |\SCTLR_EL1_NTWE | SCTLR_ELx_IESB |SCTLR_EL1_SPAN |\ + SCTLR_ELx_ITFSB| SCTLR_ELx_ATA |SCTLR_EL1_ATA0 |\ ENDIAN_SET_EL1 | SCTLR_EL1_UCI | SCTLR_EL1_RES1)/* MAIR_ELx memory attributes (used by Linux) */
arch/arm64/include/uapi/asm/hwcap.h +2 -0
@@ -73,5 +73,7 @@#define HWCAP2_BF16 (1 << 14)#define HWCAP2_DGH (1 << 15)#define HWCAP2_RNG (1 << 16) +/* bit 17 reserved for HWCAP2_BTI */ +#define HWCAP2_MTE (1 << 18)#endif /* _UAPI__ASM_HWCAP_H */
arch/arm64/kernel/cpufeature.c +30 -0
@@ -182,6 +182,8 @@ static const struct arm64_ftr_bits ftr_id_aa64pfr0[] = {static const struct arm64_ftr_bits ftr_id_aa64pfr1[] = {ARM64_FTR_BITS (FTR_VISIBLE ,FTR_STRICT ,FTR_LOWER_SAFE ,ID_AA64PFR1_SSBS_SHIFT ,4 ,ID_AA64PFR1_SSBS_PSTATE_NI ), +ARM64_FTR_BITS (FTR_VISIBLE_IF_IS_ENABLED (CONFIG_ARM64_MTE ), +FTR_STRICT ,FTR_LOWER_SAFE ,ID_AA64PFR1_MTE_SHIFT ,4 ,ID_AA64PFR1_MTE_NI ),ARM64_FTR_END , };
@@ -1409,6 +1411,18 @@ static bool can_use_gic_priorities(const struct arm64_cpu_capabilities *entry, }#endif +#ifdef CONFIG_ARM64_MTE +static void cpu_enable_mte (struct arm64_cpu_capabilities const *cap) +{ +/* all non-zero tags excluded by default */ +write_sysreg_s (SYS_GCR_EL1_RRND |SYS_GCR_EL1_EXCL_MASK ,SYS_GCR_EL1 ); +write_sysreg_s (0 ,SYS_TFSR_EL1 ); +write_sysreg_s (0 ,SYS_TFSRE0_EL1 ); + +isb (); +} +#endif /* CONFIG_ARM64_MTE */ +/* Internal helper functions to match cpu capability type */ static bool cpucap_late_cpu_optional (const struct arm64_cpu_capabilities *cap)
@@ -1779,6 +1793,19 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .min_field_value = 1, },#endif +#ifdef CONFIG_ARM64_MTE + { + .desc = "Memory Tagging Extension", + .capability = ARM64_MTE, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_cpuid_feature, + .sys_reg = SYS_ID_AA64PFR1_EL1, + .field_pos = ID_AA64PFR1_MTE_SHIFT, + .min_field_value = ID_AA64PFR1_MTE, + .sign = FTR_UNSIGNED, + .cpu_enable = cpu_enable_mte, + }, +#endif /* CONFIG_ARM64_MTE */ {}, };
@@ -1892,6 +1919,9 @@ static const struct arm64_cpu_capabilities arm64_elf_hwcaps[] = { HWCAP_MULTI_CAP(ptr_auth_hwcap_addr_matches, CAP_HWCAP, KERNEL_HWCAP_PACA), HWCAP_MULTI_CAP(ptr_auth_hwcap_gen_matches, CAP_HWCAP, KERNEL_HWCAP_PACG),#endif +#ifdef CONFIG_ARM64_MTE + HWCAP_CAP(SYS_ID_AA64PFR1_EL1, ID_AA64PFR1_MTE_SHIFT, FTR_UNSIGNED, ID_AA64PFR1_MTE, CAP_HWCAP, KERNEL_HWCAP_MTE), +#endif /* CONFIG_ARM64_MTE */ {}, };
arch/arm64/kernel/cpuinfo.c +2 -0
@@ -92,6 +92,8 @@ static const char *const hwcap_str[] = {"bf16" ,"dgh" ,"rng" , +"" ,/* reserved for BTI */ +"mte" , NULL };
Catalin Marinas
maintainerPATCH v4 3/26 arm64: mte: Use Normal Tagged attributes for the linear map
In reply to Catalin MarinasOnce user space is given access to tagged memory, the kernel must be able to clear/save/restore tags visible to the user. This is done via the linear mapping, therefore map it as such. The new MT_NORMAL_TAGGED index for MAIR_EL1 is initially mapped as Normal memory and later changed to Normal Tagged via the cpufeature infrastructure. From a mismatched attribute aliases perspective, the Tagged memory is considered a permission and it won't lead to undefined behaviour.
The empty_zero_page is cleared to ensure that the tags it contains are already zeroed. The actual tags-aware clear_page() implementation is part of a subsequent patch.
---
Notes:
v3:
- Restrict the safe attribute change in pgattr_change_is_safe() only to Normal to/from Normal-Tagged (old version allow any other type as long as old or new was Normal(-Tagged)).
arch/arm64/include/asm/memory.h | 1 + arch/arm64/include/asm/pgtable-prot.h | 2 ++ arch/arm64/kernel/cpufeature.c | 30 +++++++++++++++++++++++++++ arch/arm64/mm/dump.c | 4 ++++ arch/arm64/mm/mmu.c | 22 ++++++++++++++++++-- arch/arm64/mm/proc.S | 8 +++++-- 6 files changed, 63 insertions(+), 4 deletions(-)
arch/arm64/include/asm/memory.h +1 -0
@@ -136,6 +136,7 @@#define MT_NORMAL_NC 3#define MT_NORMAL 4#define MT_NORMAL_WT 5 +#define MT_NORMAL_TAGGED 6/* * Memory types for Stage-2 translation
arch/arm64/include/asm/pgtable-prot.h +2 -0
@@ -39,6 +39,7 @@ extern bool arm64_use_ng_mappings;#define PROT_NORMAL_NC (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL_NC))#define PROT_NORMAL_WT (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL_WT))#define PROT_NORMAL (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL)) +#define PROT_NORMAL_TAGGED (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL_TAGGED))#define PROT_SECT_DEVICE_nGnRE (PROT_SECT_DEFAULT | PMD_SECT_PXN | PMD_SECT_UXN | PMD_ATTRINDX(MT_DEVICE_nGnRE))#define PROT_SECT_NORMAL (PROT_SECT_DEFAULT | PMD_SECT_PXN | PMD_SECT_UXN | PMD_ATTRINDX(MT_NORMAL))
@@ -48,6 +49,7 @@ extern bool arm64_use_ng_mappings;#define _HYP_PAGE_DEFAULT _PAGE_DEFAULT#define PAGE_KERNEL __pgprot(PROT_NORMAL) +#define PAGE_KERNEL_TAGGED __pgprot(PROT_NORMAL_TAGGED)#define PAGE_KERNEL_RO __pgprot((PROT_NORMAL & ~PTE_WRITE) | PTE_RDONLY)#define PAGE_KERNEL_ROX __pgprot((PROT_NORMAL & ~(PTE_WRITE | PTE_PXN)) | PTE_RDONLY)#define PAGE_KERNEL_EXEC __pgprot(PROT_NORMAL & ~PTE_PXN)
arch/arm64/kernel/cpufeature.c +30 -0
@@ -1414,13 +1414,43 @@ static bool can_use_gic_priorities(const struct arm64_cpu_capabilities *entry,#ifdef CONFIG_ARM64_MTE static void cpu_enable_mte (struct arm64_cpu_capabilities const *cap) { +u64 mair; +/* all non-zero tags excluded by default */ write_sysreg_s (SYS_GCR_EL1_RRND |SYS_GCR_EL1_EXCL_MASK ,SYS_GCR_EL1 );write_sysreg_s (0 ,SYS_TFSR_EL1 );write_sysreg_s (0 ,SYS_TFSRE0_EL1 ); +/* +* Update the MT_NORMAL_TAGGED index in MAIR_EL1. Tag checking is +* disabled for the kernel, so there won't be any observable effect +* other than allowing the kernel to read and write tags. +*/ + mair =read_sysreg_s (SYS_MAIR_EL1 ); + mair &= ~MAIR_ATTRIDX (MAIR_ATTR_MASK ,MT_NORMAL_TAGGED ); + mair |=MAIR_ATTRIDX (MAIR_ATTR_NORMAL_TAGGED ,MT_NORMAL_TAGGED ); +write_sysreg_s (mair,SYS_MAIR_EL1 ); +isb (); } + +static int __init system_enable_mte (void ) +{ +if (!system_supports_mte ()) +return 0 ; + +/* Ensure the TLB does not have stale MAIR attributes */ +flush_tlb_all (); + +/* +* Clear the zero page (again) so that tags are reset. This needs to +* be done via the linear map which has the Tagged attribute. +*/ +clear_page (lm_alias (empty_zero_page)); + +return 0 ; +} +core_initcall (system_enable_mte);#endif /* CONFIG_ARM64_MTE */ /* Internal helper functions to match cpu capability type */
arch/arm64/mm/dump.c +4 -0
@@ -165,6 +165,10 @@ static const struct prot_bits pte_bits[] = { .mask = PTE_ATTRINDX_MASK, .val = PTE_ATTRINDX(MT_NORMAL), .set = "MEM/NORMAL", + }, { + .mask = PTE_ATTRINDX_MASK, + .val = PTE_ATTRINDX(MT_NORMAL_TAGGED), + .set = "MEM/NORMAL-TAGGED", } };
arch/arm64/mm/mmu.c +20 -2
@@ -121,7 +121,7 @@ static bool pgattr_change_is_safe(u64 old, u64 new)* The following mapping attributes may be updated in live * kernel mappings without the need for break-before-make. */ -static const pteval_t mask =PTE_PXN |PTE_RDONLY |PTE_WRITE |PTE_NG ; +pteval_t mask =PTE_PXN |PTE_RDONLY |PTE_WRITE |PTE_NG ;/* creating or taking down mappings is always safe */ if (old ==0 || new ==0 )
@@ -135,6 +135,19 @@ static bool pgattr_change_is_safe(u64 old, u64 new)if (old & ~new &PTE_NG )return false; +if (system_supports_mte ()) { +/* +* Changing the memory type between Normal and Normal-Tagged +* is safe since Tagged is considered a permission attribute +* from the mismatched attribute aliases perspective. +*/ +if (((old &PTE_ATTRINDX_MASK ) ==PTE_ATTRINDX (MT_NORMAL ) || + (old &PTE_ATTRINDX_MASK ) ==PTE_ATTRINDX (MT_NORMAL_TAGGED )) && + ((new &PTE_ATTRINDX_MASK ) ==PTE_ATTRINDX (MT_NORMAL ) || + (new &PTE_ATTRINDX_MASK ) ==PTE_ATTRINDX (MT_NORMAL_TAGGED ))) + mask |=PTE_ATTRINDX_MASK ; + } +return ((old ^ new) & ~mask) ==0 ; }
@@ -489,7 +502,12 @@ static void __init map_mem(pgd_t *pgdp)if (memblock_is_nomap (reg))continue ; -__map_memblock (pgdp, start, end,PAGE_KERNEL , flags); +/* +* The linear map must allow allocation tags reading/writing +* if MTE is present. Otherwise, it has the same attributes as +* PAGE_KERNEL. +*/ +__map_memblock (pgdp, start, end,PAGE_KERNEL_TAGGED , flags); }/*
arch/arm64/mm/proc.S +6 -2
@@ -44,14 +44,18 @@ #define TCR_KASAN_FLAGS 0 #endif -/* Default MAIR_EL1 */ +/* + * Default MAIR_EL1. MT_NORMAL_TAGGED is initially mapped as Normal memory and + * changed later to Normal Tagged if the system supports MTE. + */ #define MAIR_EL1_SET \ (MAIR_ATTRIDX(MAIR_ATTR_DEVICE_nGnRnE, MT_DEVICE_nGnRnE) | \ MAIR_ATTRIDX(MAIR_ATTR_DEVICE_nGnRE, MT_DEVICE_nGnRE) | \ MAIR_ATTRIDX(MAIR_ATTR_DEVICE_GRE, MT_DEVICE_GRE) | \ MAIR_ATTRIDX(MAIR_ATTR_NORMAL_NC, MT_NORMAL_NC) | \ MAIR_ATTRIDX(MAIR_ATTR_NORMAL, MT_NORMAL) | \ - MAIR_ATTRIDX(MAIR_ATTR_NORMAL_WT, MT_NORMAL_WT)) + MAIR_ATTRIDX(MAIR_ATTR_NORMAL_WT, MT_NORMAL_WT) | \ + MAIR_ATTRIDX(MAIR_ATTR_NORMAL, MT_NORMAL_TAGGED)) #ifdef CONFIG_CPU_PM /**
Catalin Marinas
maintainerPATCH v4 4/26 arm64: mte: Add specific SIGSEGV codes
In reply to Catalin MarinasFrom: Vincenzo Frascino <[email protected]>
Add MTE-specific SIGSEGV codes to siginfo.h and update the x86 BUILD_BUG_ON(NSIGSEGV != 7) compile check.
---
Notes:
v3:
- Fixed the BUILD_BUG_ON(NSIGSEGV != 7) on x86
- Updated the commit log
v2:
- Dropped the #ifdef __aarch64__.
- Renumbered the SEGV_MTE* values to avoid clash with ADI.
arch/x86/kernel/signal_compat.c | 2 +- include/uapi/asm-generic/siginfo.h | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-)
arch/x86/kernel/signal_compat.c +1 -1
@@ -27,7 +27,7 @@ static inline void signal_compat_build_tests(void)*/ BUILD_BUG_ON (NSIGILL !=11 );BUILD_BUG_ON (NSIGFPE !=15 ); -BUILD_BUG_ON (NSIGSEGV !=7 ); +BUILD_BUG_ON (NSIGSEGV !=9 );BUILD_BUG_ON (NSIGBUS !=5 );BUILD_BUG_ON (NSIGTRAP !=5 );BUILD_BUG_ON (NSIGCHLD !=6 );
include/uapi/asm-generic/siginfo.h +3 -1
@@ -229,7 +229,9 @@ typedef struct siginfo {#define SEGV_ACCADI 5/* ADI not enabled for mapped object */ #define SEGV_ADIDERR 6/* Disrupting MCD error */ #define SEGV_ADIPERR 7/* Precise MCD exception */ -#define NSIGSEGV 7 +#define SEGV_MTEAERR 8/* Asynchronous ARM MTE error */ +#define SEGV_MTESERR 9/* Synchronous ARM MTE exception */ +#define NSIGSEGV 9/* * SIGBUS si_codes
Catalin Marinas
maintainerPATCH v4 5/26 arm64: mte: Handle synchronous and asynchronous tag check faults
In reply to Catalin MarinasFrom: Vincenzo Frascino <[email protected]>
The Memory Tagging Extension has two modes of notifying a tag check fault at EL0, configurable through the SCTLR_EL1.TCF0 field:
1. Synchronous raising of a Data Abort exception with DFSC 17.
2. Asynchronous setting of a cumulative bit in TFSRE0_EL1.
Add the exception handler for the synchronous exception and handling of the asynchronous TFSRE0_EL1.TF0 bit setting via a new TIF flag in do_notify_resume().
On a tag check failure in user-space, whether synchronous or asynchronous, a SIGSEGV will be raised on the faulting thread.
---
Notes:
v4:
- Use send_signal_fault() instead of fault_signal_inject() for asynchronous tag check faults as execution can continue even if this signal is masked.
- Add DSB ISH prior to writing TFSRE0_EL1 in the clear_mte_async_tcf macro.
- Move clear_mte_async_tcf just after returning to user since do_notify_resume() may still cause async tag faults via do_signal().
v3:
- Asynchronous tag check faults during the uaccess routines in the kernel are ignored.
- Fix check_mte_async_tcf calling site as it expects the first argument to be the thread flags.
- Move the mte_thread_switch() definition and call to a later patch as this became empty with the removal of async uaccess checking.
- Add dsb() and clearing of TFSRE0_EL1 in flush_mte_state(), in case execve() triggered a asynchronous tag check fault.
- Clear TIF_MTE_ASYNC_FAULT in arch_dup_task_struct() so that the child does not inherit any pending tag fault in the parent.
v2:
- Clear PSTATE.TCO on exception entry (automatically set by the hardware).
- On syscall entry, for asynchronous tag check faults from user space, generate the signal early via syscall restarting.
- Before context switch, save any potential async tag check fault generated by the kernel to the TIF flag (this follows an architecture update where the uaccess routines use the TCF0 mode).
- Moved the flush_mte_state() and mte_thread_switch() function to a new mte.c file.
arch/arm64/include/asm/mte.h | 23 +++++++++++++++++ arch/arm64/include/asm/thread_info.h | 4 ++- arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/entry.S | 37 ++++++++++++++++++++++++++++ arch/arm64/kernel/mte.c | 21 ++++++++++++++++ arch/arm64/kernel/process.c | 5 ++++ arch/arm64/kernel/signal.c | 8 ++++++ arch/arm64/kernel/syscall.c | 10 ++++++++ arch/arm64/mm/fault.c | 9 ++++++- 9 files changed, 116 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/include/asm/mte.h create mode 100644 arch/arm64/kernel/mte.c
arch/arm64/include/asm/mte.h new file +23 -0
@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (C) 2020 ARM Ltd. +*/ +#ifndef __ASM_MTE_H +#define __ASM_MTE_H + +#ifndef __ASSEMBLY__ + +#ifdef CONFIG_ARM64_MTE + +void flush_mte_state (void ); + +#else + +static inline void flush_mte_state (void ) +{ +} + +#endif + +#endif /* __ASSEMBLY__ */ +#endif /* __ASM_MTE_H */
arch/arm64/include/asm/thread_info.h +3 -1
@@ -63,6 +63,7 @@ void arch_release_task_struct(struct task_struct *tsk);#define TIF_FOREIGN_FPSTATE 3/* CPU's FP state is not current's */ #define TIF_UPROBE 4/* uprobe breakpoint or singlestep */ #define TIF_FSCHECK 5/* Check FS is USER_DS on return */ +#define TIF_MTE_ASYNC_FAULT 6/* MTE Asynchronous Tag Check Fault */ #define TIF_SYSCALL_TRACE 8/* syscall trace active */ #define TIF_SYSCALL_AUDIT 9/* syscall auditing */ #define TIF_SYSCALL_TRACEPOINT 10/* syscall tracepoint for ftrace */
@@ -91,10 +92,11 @@ void arch_release_task_struct(struct task_struct *tsk);#define _TIF_FSCHECK (1 << TIF_FSCHECK)#define _TIF_32BIT (1 << TIF_32BIT)#define _TIF_SVE (1 << TIF_SVE) +#define _TIF_MTE_ASYNC_FAULT (1 << TIF_MTE_ASYNC_FAULT)#define _TIF_WORK_MASK (_TIF_NEED_RESCHED | _TIF_SIGPENDING | \ _TIF_NOTIFY_RESUME | _TIF_FOREIGN_FPSTATE | \ - _TIF_UPROBE | _TIF_FSCHECK) + _TIF_UPROBE | _TIF_FSCHECK | _TIF_MTE_ASYNC_FAULT)#define _TIF_SYSCALL_WORK (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | \ _TIF_SYSCALL_TRACEPOINT | _TIF_SECCOMP | \
arch/arm64/kernel/Makefile +1 -0
@@ -63,6 +63,7 @@ obj-$(CONFIG_CRASH_CORE) += crash_core.o obj-$(CONFIG_ARM_SDE_INTERFACE) += sdei.o obj-$(CONFIG_ARM64_SSBD) += ssbd.o obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o +obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso/ probes/ obj-$(CONFIG_COMPAT_VDSO) += vdso32/
arch/arm64/kernel/entry.S +37 -0
@@ -145,6 +145,32 @@ alternative_cb_end #endif .endm + /* Check for MTE asynchronous tag check faults */ + .macro check_mte_async_tcf, flgs, tmp +#ifdef CONFIG_ARM64_MTE +alternative_if_not ARM64_MTE + b 1f +alternative_else_nop_endif + mrs_s \tmp, SYS_TFSRE0_EL1 + tbz \tmp, #SYS_TFSR_EL1_TF0_SHIFT, 1f + /* Asynchronous TCF occurred for TTBR0 access, set the TI flag */ + orr \flgs, \flgs, #_TIF_MTE_ASYNC_FAULT + str \flgs, [tsk, #TSK_TI_FLAGS] + msr_s SYS_TFSRE0_EL1, xzr +1: +#endif + .endm + + /* Clear the MTE asynchronous tag check faults */ + .macro clear_mte_async_tcf +#ifdef CONFIG_ARM64_MTE +alternative_if ARM64_MTE + dsb ish + msr_s SYS_TFSRE0_EL1, xzr +alternative_else_nop_endif +#endif + .endm + .macro kernel_entry, el, regsize = 64 .if \regsize == 32 mov w0, w0 // zero upper 32 bits of x0
@@ -176,6 +202,8 @@ alternative_cb_end ldr x19, [tsk, #TSK_TI_FLAGS] disable_step_tsk x19, x20 + /* Check for asynchronous tag check faults in user space */ + check_mte_async_tcf x19, x22 apply_ssbd 1, x22, x23 ptrauth_keys_install_kernel tsk, 1, x20, x22, x23
@@ -244,6 +272,13 @@ alternative_if ARM64_HAS_IRQ_PRIO_MASKING str x20, [sp, #S_PMR_SAVE] alternative_else_nop_endif + /* Re-enable tag checking (TCO set on exception entry) */ +#ifdef CONFIG_ARM64_MTE +alternative_if ARM64_MTE + SET_PSTATE_TCO(0) +alternative_else_nop_endif +#endif + /* * Registers that may be useful after this macro is invoked: *
@@ -748,6 +783,8 @@ ret_to_user: and x2, x1, #_TIF_WORK_MASK cbnz x2, work_pending finish_ret_to_user: + /* Ignore asynchronous tag check faults in the uaccess routines */ + clear_mte_async_tcf enable_step_tsk x1, x2 #ifdef CONFIG_GCC_PLUGIN_STACKLEAK bl stackleak_erase
arch/arm64/kernel/mte.c new file +21 -0
@@ -0,0 +1,21 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2020 ARM Ltd. +*/ + +#include <linux/thread_info.h> + +#include <asm/cpufeature.h> +#include <asm/mte.h> +#include <asm/sysreg.h> + +void flush_mte_state (void ) +{ +if (!system_supports_mte ()) +return ; + +/* clear any pending asynchronous tag fault */ +dsb (ish); +write_sysreg_s (0 ,SYS_TFSRE0_EL1 ); +clear_thread_flag (TIF_MTE_ASYNC_FAULT ); +}
arch/arm64/kernel/process.c +5 -0
@@ -50,6 +50,7 @@#include <asm/exec.h> #include <asm/fpsimd.h> #include <asm/mmu_context.h> +#include <asm/mte.h> #include <asm/processor.h> #include <asm/pointer_auth.h> #include <asm/stacktrace.h>
@@ -323,6 +324,7 @@ void flush_thread(void)tls_thread_flush ();flush_ptrace_hw_breakpoint (current);flush_tagged_addr_state (); +flush_mte_state (); }void release_thread (struct task_struct *dead_task)
@@ -355,6 +357,9 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) dst->thread.sve_state =NULL ;clear_tsk_thread_flag (dst,TIF_SVE ); +/* clear any pending asynchronous tag fault raised by the parent */ +clear_tsk_thread_flag (dst,TIF_MTE_ASYNC_FAULT ); +return 0 ; }
arch/arm64/kernel/signal.c +8 -0
@@ -732,6 +732,9 @@ static void setup_return(struct pt_regs *regs, struct k_sigaction *ka, regs->regs[29 ] = (unsigned long )&user->next_frame->fp; regs->pc = (unsigned long )ka->sa.sa_handler; +/* TCO (Tag Check Override) always cleared for signal handlers */ + regs->pstate &= ~PSR_TCO_BIT ; +if (ka->sa.sa_flags &SA_RESTORER ) sigtramp = ka->sa.sa_restorer;else
@@ -923,6 +926,11 @@ asmlinkage void do_notify_resume(struct pt_regs *regs,if (thread_flags & _TIF_UPROBE)uprobe_notify_resume (regs); +if (thread_flags & _TIF_MTE_ASYNC_FAULT) { +clear_thread_flag (TIF_MTE_ASYNC_FAULT ); +send_sig_fault (SIGSEGV ,SEGV_MTEAERR ,0 , current); + } +if (thread_flags & _TIF_SIGPENDING)do_signal (regs);
arch/arm64/kernel/syscall.c +10 -0
@@ -102,6 +102,16 @@ static void el0_svc_common(struct pt_regs *regs, int scno, int sc_nr,local_daif_restore (DAIF_PROCCTX );user_exit (); +if (system_supports_mte () && (flags & _TIF_MTE_ASYNC_FAULT)) { +/* +* Process the asynchronous tag check fault before the actual +* syscall. do_notify_resume() will send a signal to userspace +* before the syscall is restarted. +*/ + regs->regs[0 ] = -ERESTARTNOINTR ; +return ; + } +if (has_syscall_work (flags)) {/* set default errno for user-issued syscall(-1) */ if (scno ==NO_SYSCALL )
arch/arm64/mm/fault.c +8 -1
@@ -650,6 +650,13 @@ static int do_sea(unsigned long addr, unsigned int esr, struct pt_regs *regs)return 0 ; } +static int do_tag_check_fault (unsigned long addr,unsigned int esr, +struct pt_regs *regs) +{ +do_bad_area (addr, esr, regs); +return 0 ; +} +static const struct fault_info fault_info[] = { { do_bad,SIGKILL ,SI_KERNEL ,"ttbr address size fault" }, { do_bad,SIGKILL ,SI_KERNEL ,"level 1 address size fault" },
@@ -668,7 +675,7 @@ static const struct fault_info fault_info[] = { { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 2 permission fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 3 permission fault" }, { do_sea, SIGBUS, BUS_OBJERR, "synchronous external abort" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 17" }, + { do_tag_check_fault, SIGSEGV, SEGV_MTESERR, "synchronous tag check fault" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 18" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 19" }, { do_sea, SIGKILL, SI_KERNEL, "level 0 (translation table walk)" },
Catalin Marinas
maintainerPATCH v4 6/26 mm: Add PG_ARCH_2 page flag
In reply to Catalin MarinasFrom: Steven Price <[email protected]>
For arm64 MTE support it is necessary to be able to mark pages that contain user space visible tags that will need to be saved/restored e.g. when swapped out.
To support this add a new arch specific flag (PG_ARCH_2) that arch code can opt into using ARCH_USES_PG_ARCH_2.
---
Notes:
New in v4.
fs/proc/page.c | 3 +++ include/linux/kernel-page-flags.h | 1 + include/linux/page-flags.h | 3 +++ include/trace/events/mmflags.h | 9 ++++++++- mm/Kconfig | 3 +++ tools/vm/page-types.c | 2 ++ 6 files changed, 20 insertions(+), 1 deletion(-)
fs/proc/page.c +3 -0
@@ -217,6 +217,9 @@ u64 stable_page_flags(struct page *page) u |=kpf_copy_bit (k,KPF_PRIVATE_2 , PG_private_2); u |=kpf_copy_bit (k,KPF_OWNER_PRIVATE , PG_owner_priv_1); u |=kpf_copy_bit (k,KPF_ARCH , PG_arch_1); +#ifdef CONFIG_ARCH_USES_PG_ARCH_2 + u |=kpf_copy_bit (k,KPF_ARCH_2 , PG_arch_2); +#endif return u; };
include/linux/kernel-page-flags.h +1 -0
@@ -17,5 +17,6 @@#define KPF_ARCH 38#define KPF_UNCACHED 39#define KPF_SOFTDIRTY 40 +#define KPF_ARCH_2 41#endif /* LINUX_KERNEL_PAGE_FLAGS_H */
include/linux/page-flags.h +3 -0
@@ -135,6 +135,9 @@ enum pageflags {#if defined(CONFIG_IDLE_PAGE_TRACKING ) && defined(CONFIG_64BIT ) PG_young, PG_idle, +#endif +#ifdef CONFIG_ARCH_USES_PG_ARCH_2 + PG_arch_2, #endif __NR_PAGEFLAGS,
include/trace/events/mmflags.h +8 -1
@@ -79,6 +79,12 @@#define IF_HAVE_PG_IDLE (flag,string)#endif +#ifdef CONFIG_ARCH_USES_PG_ARCH_2 +#define IF_HAVE_PG_ARCH_2 (flag,string) ,{1UL << flag, string} +#else +#define IF_HAVE_PG_ARCH_2 (flag,string) +#endif +#define __def_pageflag_names \ {1UL << PG_locked, "locked" }, \ {1UL << PG_waiters, "waiters" }, \
@@ -105,7 +111,8 @@ IF_HAVE_PG_MLOCK(PG_mlocked, "mlocked" ) \ IF_HAVE_PG_UNCACHED(PG_uncached, "uncached" ) \ IF_HAVE_PG_HWPOISON(PG_hwpoison, "hwpoison" ) \ IF_HAVE_PG_IDLE(PG_young, "young" ) \ -IF_HAVE_PG_IDLE(PG_idle, "idle" ) +IF_HAVE_PG_IDLE(PG_idle, "idle" ) \ +IF_HAVE_PG_ARCH_2(PG_arch_2, "arch_2" )#define show_page_flags (flags) \ (flags) ? __print_flags(flags, "|", \
mm/Kconfig +3 -0
@@ -867,4 +867,7 @@ config ARCH_HAS_HUGEPD config MAPPING_DIRTY_HELPERS bool +config ARCH_USES_PG_ARCH_2 + bool + endmenu
tools/vm/page-types.c +2 -0
@@ -78,6 +78,7 @@#define KPF_ARCH 38#define KPF_UNCACHED 39#define KPF_SOFTDIRTY 40 +#define KPF_ARCH_2 41/* [48-] take some arbitrary free slots for expanding overloaded flags * not part of kernel API
@@ -135,6 +136,7 @@ static const char * const page_flag_names[] = { [KPF_ARCH] = "h:arch", [KPF_UNCACHED] = "c:uncached", [KPF_SOFTDIRTY] = "f:softdirty", + [KPF_ARCH_2] = "H:arch_2", [KPF_READAHEAD] = "I:readahead", [KPF_SLOB_FREE] = "P:slob_free",
Catalin Marinas
maintainerPATCH v4 7/26 arm64: mte: Clear the tags when a page is mapped in user-space with PROT_MTE
In reply to Catalin MarinasPages allocated by the kernel are not guaranteed to have the tags zeroed, especially as the kernel does not yet use MTE itself. To ensure the user can still access such pages when mapped into its address space, clear the tags via set_pte_at(). A new page flag - PG_mte_tagged (PG_arch_2) - is used to track pages with valid allocation tags.
---
Notes:
New in v4. Replacing a previous page zeroing the tags in clear_page().
arch/arm64/include/asm/assembler.h | 12 ++++++++++++ arch/arm64/include/asm/mte.h | 16 ++++++++++++++++ arch/arm64/include/asm/pgtable.h | 6 ++++++ arch/arm64/kernel/mte.c | 14 ++++++++++++++ arch/arm64/lib/Makefile | 2 ++ arch/arm64/lib/mte.S | 23 +++++++++++++++++++++++ 6 files changed, 73 insertions(+) create mode 100644 arch/arm64/lib/mte.S
arch/arm64/include/asm/assembler.h +12 -0
@@ -21,6 +21,7 @@#include <asm/page.h> #include <asm/pgtable-hwdef.h> #include <asm/ptrace.h> +#include <asm/sysreg.h> #include <asm/thread_info.h> .macro save_and_disable_daif, flags
@@ -736,4 +737,15 @@ USER(\label, ic ivau, \tmp2) // invalidate I line PoU .Lyield_out_\@ : .endm +/* + * multitag_transfer_size - set \reg to the block size that is accessed by the + * LDGM/STGM instructions. + */ + .macro multitag_transfer_size, reg, tmp + mrs_s \reg, SYS_GMID_EL1 + ubfx \reg, \reg, #SYS_GMID_EL1_BS_SHIFT, #SYS_GMID_EL1_BS_SIZE + mov \tmp, #4 + lsl \reg, \tmp, \reg + .endm + #endif /* __ASM_ASSEMBLER_H */
arch/arm64/include/asm/mte.h +16 -0
@@ -7,12 +7,28 @@#ifndef __ASSEMBLY__ +#include <linux/page-flags.h> + +#include <asm/pgtable-types.h> + +void mte_clear_page_tags (void *addr,size_t size); +#ifdef CONFIG_ARM64_MTE +/* track which pages have valid allocation tags */ +#define PG_mte_tagged PG_arch_2 + +void mte_sync_tags (pte_t *ptep,pte_t pte);void flush_mte_state (void );#else +/* unused if !CONFIG_ARM64_MTE, silence the compiler */ +#define PG_mte_tagged 0 + +static inline void mte_sync_tags (pte_t *ptep,pte_t pte) +{ +}static inline void flush_mte_state (void ) { }
arch/arm64/include/asm/pgtable.h +6 -0
@@ -9,6 +9,7 @@#include <asm/proc-fns.h> #include <asm/memory.h> +#include <asm/mte.h> #include <asm/pgtable-hwdef.h> #include <asm/pgtable-prot.h> #include <asm/tlbflush.h>
@@ -80,6 +81,8 @@ extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)];#define pte_user_exec (pte) (!(pte_val(pte) & PTE_UXN))#define pte_cont (pte) (!!(pte_val(pte) & PTE_CONT))#define pte_devmap (pte) (!!(pte_val(pte) & PTE_DEVMAP)) +#define pte_tagged (pte) ((pte_val(pte) & PTE_ATTRINDX_MASK) == \ + PTE_ATTRINDX(MT_NORMAL_TAGGED))#define pte_cont_addr_end (addr, end) \ ({ unsigned long __boundary = ((addr) + CONT_PTE_SIZE) & CONT_PTE_MASK; \
@@ -274,6 +277,9 @@ static inline void set_pte_at(struct mm_struct *mm, unsigned long addr,if (pte_present (pte) &&pte_user_exec (pte) && !pte_special (pte))__sync_icache_dcache (pte); +if (system_supports_mte () &&pte_present (pte) &&pte_tagged (pte)) +mte_sync_tags (ptep, pte); +__check_racy_pte_update (mm, ptep, pte);set_pte (ptep, pte);
arch/arm64/kernel/mte.c +14 -0
@@ -3,12 +3,26 @@* Copyright (C) 2020 ARM Ltd. */ +#include <linux/bitops.h> +#include <linux/mm.h> #include <linux/thread_info.h> #include <asm/cpufeature.h> #include <asm/mte.h> #include <asm/sysreg.h> +void mte_sync_tags (pte_t *ptep,pte_t pte) +{ +struct page *page =pte_page (pte); + +/* if PG_mte_tagged is set, tags have already been initialised */ +if (test_and_set_bit (PG_mte_tagged, &page->flags)) +return ; + +/* tags of the zero page will be cleared on first mapping */ +mte_clear_page_tags (page_address (page),page_size (page)); +} +void flush_mte_state (void ) {if (!system_supports_mte ())
arch/arm64/lib/Makefile +2 -0
@@ -16,3 +16,5 @@ lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o obj-$(CONFIG_CRC32) += crc32.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o + +obj-$(CONFIG_ARM64_MTE) += mte.o
arch/arm64/lib/mte.S new file +23 -0
@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2020 ARM Ltd. + */ +#include <linux/linkage.h> + +#include <asm/assembler.h> + + .arch armv8.5-a+memtag + +/* + * Clear the tags in a (potentially huge) page + * x0 - page address + * x1 - page size + */ +SYM_FUNC_START(mte_clear_page_tags) + multitag_transfer_size x2, x3 +1: stgm xzr, [x0] + add x0, x0, x2 + sub x1, x1, x2 + cbnz x1, 1b + ret +SYM_FUNC_END(mte_clear_page_tags)
Catalin Marinas
maintainerPATCH v4 8/26 arm64: mte: Tags-aware copy_page() implementation
In reply to Catalin MarinasFrom: Vincenzo Frascino <[email protected]>
When the Memory Tagging Extension is enabled, the tags need to be preserved across page copy (e.g. for copy-on-write).
Introduce MTE-aware copy_page() which preserves the tags across page copy.
---
Notes:
v4:
- Moved the tag copying to a separate function in mte.S and only called if the source page has the PG_mte_tagged flag set.
arch/arm64/include/asm/mte.h | 4 ++++ arch/arm64/lib/mte.S | 19 +++++++++++++++++++ arch/arm64/mm/copypage.c | 14 ++++++++++++-- 3 files changed, 35 insertions(+), 2 deletions(-)
arch/arm64/include/asm/mte.h +4 -0
@@ -19,6 +19,7 @@ void mte_clear_page_tags(void *addr, size_t size);#define PG_mte_tagged PG_arch_2void mte_sync_tags (pte_t *ptep,pte_t pte); +void mte_copy_page_tags (void *kto,const void *kfrom);void flush_mte_state (void );#else
@@ -29,6 +30,9 @@ void flush_mte_state(void);static inline void mte_sync_tags (pte_t *ptep,pte_t pte) { } +static inline void mte_copy_page_tags (void *kto,const void *kfrom) +{ +}static inline void flush_mte_state (void ) { }
arch/arm64/lib/mte.S +19 -0
@@ -5,6 +5,7 @@ #include <linux/linkage.h> #include <asm/assembler.h> +#include <asm/page.h> .arch armv8.5-a+memtag
@@ -21,3 +22,21 @@ SYM_FUNC_START(mte_clear_page_tags) cbnz x1, 1b ret SYM_FUNC_END(mte_clear_page_tags) + +/* + * Copy the tags from the source page to the destination one + * x0 - address of the destination page + * x1 - address of the source page + */ +SYM_FUNC_START(mte_copy_page_tags) + mov x2, x0 + mov x3, x1 + multitag_transfer_size x5, x6 +1: ldgm x4, [x3] + stgm x4, [x2] + add x2, x2, x5 + add x3, x3, x5 + tst x2, #(PAGE_SIZE - 1) + b.ne 1b +2: +SYM_FUNC_END(mte_copy_page_tags)
arch/arm64/mm/copypage.c +12 -2
@@ -6,16 +6,26 @@* Copyright (C) 2012 ARM Ltd. */ +#include <linux/bitops.h> #include <linux/mm.h> #include <asm/page.h> #include <asm/cacheflush.h> +#include <asm/cpufeature.h> +#include <asm/mte.h> void __cpu_copy_user_page (void *kto,const void *kfrom,unsigned long vaddr) { -struct page *page =virt_to_page (kto); +struct page *to_page =virt_to_page (kto); +struct page *from_page =virt_to_page (kfrom); +copy_page (kto, kfrom); -flush_dcache_page (page); +if (system_supports_mte () && +test_bit (PG_mte_tagged, &from_page->flags)) { +mte_copy_page_tags (kto, kfrom); +set_bit (PG_mte_tagged, &to_page->flags); + } +flush_dcache_page (to_page); }EXPORT_SYMBOL_GPL (__cpu_copy_user_page);
Catalin Marinas
maintainerPATCH v4 9/26 arm64: mte: Tags-aware aware memcmp_pages() implementation
In reply to Catalin MarinasWhen the Memory Tagging Extension is enabled, two pages are identical only if both their data and tags are identical.
Make the generic memcmp_pages() a __weak function and add an arm64-specific implementation which returns non-zero if any of the two pages contain valid MTE tags (PG_mte_tagged set). There isn't much benefit in comparing the tags of two pages since these are normally used for heap allocations and likely to differ anyway.
---
Notes:
v4:
- Remove page tag comparison. This is not very useful to detect identical pages as long as set_pte_at() can zero the tags on a page without copy-on-write if mapped with PROT_MTE. This can be improved if a real case appears but it's unlikely for heap pages to be identical across multiple processes.
- Move the memcmp_pages() function to mte.c.
arch/arm64/kernel/mte.c | 26 ++++++++++++++++++++++++++ mm/util.c | 2 +- 2 files changed, 27 insertions(+), 1 deletion(-)
arch/arm64/kernel/mte.c +26 -0
@@ -5,6 +5,7 @@#include <linux/bitops.h> #include <linux/mm.h> +#include <linux/string.h> #include <linux/thread_info.h> #include <asm/cpufeature.h>
@@ -23,6 +24,31 @@ void mte_sync_tags(pte_t *ptep, pte_t pte)mte_clear_page_tags (page_address (page),page_size (page)); } +int memcmp_pages (struct page *page1,struct page *page2) +{ +char *addr1, *addr2; +int ret; + + addr1 =page_address (page1); + addr2 =page_address (page2); + ret =memcmp (addr1, addr2,PAGE_SIZE ); + +if (!system_supports_mte () || ret) +return ret; + +/* +* If the page content is identical but at least one of the pages is +* tagged, return non-zero to avoid KSM merging. If only one of the +* pages is tagged, set_pte_at() may zero or change the tags of the +* other page via mte_sync_tags(). +*/ +if (test_bit (PG_mte_tagged, &page1->flags) || +test_bit (PG_mte_tagged, &page2->flags)) +return addr1 != addr2; + +return ret; +} +void flush_mte_state (void ) {if (!system_supports_mte ())
mm/util.c +1 -1
@@ -899,7 +899,7 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen)return res; } -int memcmp_pages (struct page *page1,struct page *page2) +int __weak memcmp_pages (struct page *page1,struct page *page2) {char *addr1, *addr2;int ret;
Catalin Marinas
maintainerPATCH v4 10/26 mm: Introduce arch_calc_vm_flag_bits()
In reply to Catalin MarinasFrom: Kevin Brodsky <[email protected]>
Similarly to arch_calc_vm_prot_bits(), introduce a dummy arch_calc_vm_flag_bits() invoked from calc_vm_flag_bits(). This macro can be overridden by architectures to insert specific VM_* flags derived from the mmap() MAP_* flags.
---
Notes:
v2:
- Updated the comment above arch_calc_vm_prot_bits().
- Changed author since this patch had already been posted (internally).
include/linux/mman.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
include/linux/mman.h +7 -2
@@ -74,13 +74,17 @@ static inline void vm_unacct_memory(long pages) }/* -* Allow architectures to handle additional protection bits +* Allow architectures to handle additional protection and flag bits */ #ifndef arch_calc_vm_prot_bits#define arch_calc_vm_prot_bits (prot, pkey) 0#endif +#ifndef arch_calc_vm_flag_bits +#define arch_calc_vm_flag_bits (flags) 0 +#endif +#ifndef arch_vm_get_page_prot#define arch_vm_get_page_prot (vm_flags) __pgprot(0)#endif
@@ -131,7 +135,8 @@ calc_vm_flag_bits(unsigned long flags)return _calc_vm_trans (flags,MAP_GROWSDOWN ,VM_GROWSDOWN ) |_calc_vm_trans (flags,MAP_DENYWRITE ,VM_DENYWRITE ) |_calc_vm_trans (flags,MAP_LOCKED ,VM_LOCKED ) | -_calc_vm_trans (flags,MAP_SYNC ,VM_SYNC ); +_calc_vm_trans (flags,MAP_SYNC ,VM_SYNC ) | +arch_calc_vm_flag_bits (flags); }unsigned long vm_commit_limit (void );
Catalin Marinas
maintainerPATCH v4 11/26 arm64: mte: Add PROT_MTE support to mmap() and mprotect()
In reply to Catalin MarinasTo enable tagging on a memory range, the user must explicitly opt in via a new PROT_MTE flag passed to mmap() or mprotect(). Since this is a new memory type in the AttrIndx field of a pte, simplify the or'ing of these bits over the protection_map[] attributes by making MT_NORMAL index 0.
There are two conditions for arch_vm_get_page_prot() to return the MT_NORMAL_TAGGED memory type: (1) the user requested it via PROT_MTE, registered as VM_MTE in the vm_flags, and (2) the vma supports MTE, decided during the mmap() call (only) and registered as VM_MTE_ALLOWED.
arch_calc_vm_prot_bits() is responsible for registering the user request as VM_MTE. The newly introduced arch_calc_vm_flag_bits() sets VM_MTE_ALLOWED if the mapping is MAP_ANONYMOUS. An MTE-capable filesystem (RAM-based) may be able to set VM_MTE_ALLOWED during its mmap() file ops call.
In addition, update VM_DATA_DEFAULT_FLAGS to allow mprotect(PROT_MTE) on stack or brk area.
The Linux mmap() syscall currently ignores unknown PROT_* flags. In the presence of MTE, an mmap(PROT_MTE) on a file which does not support MTE will not report an error and the memory will not be mapped as Normal Tagged. For consistency, mprotect(PROT_MTE) will not report an error either if the memory range does not support MTE. Two subsequent patches in the series will propose tightening of this behaviour.
---
Notes:
v2:
- Add VM_MTE_ALLOWED to show_smap_vma_flags().
arch/arm64/include/asm/memory.h | 18 +++++---- arch/arm64/include/asm/mman.h | 64 ++++++++++++++++++++++++++++++ arch/arm64/include/asm/page.h | 2 +- arch/arm64/include/asm/pgtable.h | 7 +++- arch/arm64/include/uapi/asm/mman.h | 14 +++++++ fs/proc/task_mmu.c | 4 ++ include/linux/mm.h | 8 ++++ 7 files changed, 108 insertions(+), 9 deletions(-) create mode 100644 arch/arm64/include/asm/mman.h create mode 100644 arch/arm64/include/uapi/asm/mman.h
arch/arm64/include/asm/memory.h +11 -7
@@ -129,14 +129,18 @@/* * Memory types available. +* +* IMPORTANT: MT_NORMAL must be index 0 since vm_get_page_prot() may 'or' in +* the MT_NORMAL_TAGGED memory type for PROT_MTE mappings. Note +* that protection_map[] only contains MT_NORMAL attributes. */ -#define MT_DEVICE_nGnRnE 0 -#define MT_DEVICE_nGnRE 1 -#define MT_DEVICE_GRE 2 -#define MT_NORMAL_NC 3 -#define MT_NORMAL 4 -#define MT_NORMAL_WT 5 -#define MT_NORMAL_TAGGED 6 +#define MT_NORMAL 0 +#define MT_NORMAL_TAGGED 1 +#define MT_NORMAL_NC 2 +#define MT_NORMAL_WT 3 +#define MT_DEVICE_nGnRnE 4 +#define MT_DEVICE_nGnRE 5 +#define MT_DEVICE_GRE 6/* * Memory types for Stage-2 translation
arch/arm64/include/asm/mman.h new file +64 -0
64 lines not on this page · raw message
arch/arm64/include/asm/page.h +1 -1
@@ -36,7 +36,7 @@ extern int pfn_valid(unsigned long);#endif /* !__ASSEMBLY__ */ -#define VM_DATA_DEFAULT_FLAGS VM_DATA_FLAGS_TSK_EXEC +#define VM_DATA_DEFAULT_FLAGS (VM_DATA_FLAGS_TSK_EXEC | VM_MTE_ALLOWED)#include <asm-generic/getorder.h>
arch/arm64/include/asm/pgtable.h +6 -1
@@ -665,8 +665,13 @@ static inline phys_addr_t pgd_page_paddr(pgd_t pgd)static inline pte_t pte_modify (pte_t pte,pgprot_t newprot) { +/* +* Normal and Normal-Tagged are two different memory types and indices +* in MAIR_EL1. The mask below has to include PTE_ATTRINDX_MASK. +*/ const pteval_t mask =PTE_USER |PTE_PXN |PTE_UXN |PTE_RDONLY | -PTE_PROT_NONE |PTE_VALID |PTE_WRITE ; +PTE_PROT_NONE |PTE_VALID |PTE_WRITE | +PTE_ATTRINDX_MASK ;/* preserve the hardware dirty information */ if (pte_hw_dirty (pte)) pte =pte_mkdirty (pte);
arch/arm64/include/uapi/asm/mman.h new file +14 -0
@@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +#ifndef _UAPI__ASM_MMAN_H +#define _UAPI__ASM_MMAN_H + +#include <asm-generic/mman.h> + +/* +* The generic mman.h file reserves 0x10 and 0x20 for arch-specific PROT_* +* flags. +*/ +/* 0x10 reserved for PROT_BTI */ +#define PROT_MTE 0x20/* Normal Tagged mapping */ + +#endif /* !_UAPI__ASM_MMAN_H */
fs/proc/task_mmu.c +4 -0
@@ -647,6 +647,10 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) [ilog2(VM_MERGEABLE)] = "mg", [ilog2(VM_UFFD_MISSING)]= "um", [ilog2 (VM_UFFD_WP )] ="uw" , +#ifdef CONFIG_ARM64_MTE + [ilog2(VM_MTE)] = "mt", + [ilog2 (VM_MTE_ALLOWED )] ="" , +#endif #ifdef CONFIG_ARCH_HAS_PKEYS/* These come out via ProtectionKey: */ [ilog2(VM_PKEY_BIT0)] = "",
include/linux/mm.h +8 -0
@@ -336,6 +336,14 @@ extern unsigned int kobjsize(const void *objp);# define VM_MPX VM_NONE#endif +#if defined(CONFIG_ARM64_MTE ) +# define VM_MTE VM_HIGH_ARCH_0/* Use Tagged memory for access control */ +# define VM_MTE_ALLOWED VM_HIGH_ARCH_1/* Tagged memory permitted */ +#else +# define VM_MTE VM_NONE +# define VM_MTE_ALLOWED VM_NONE +#endif +#ifndef VM_GROWSUP # define VM_GROWSUP VM_NONE#endif
Re: [PATCH v4 11/26] arm64: mte: Add PROT_MTE support to mmap() and mprotect()
Quoted 5 lines · Catalin Marinas
To enable tagging on a memory range, the user must explicitly opt in via a new
PROT_MTEflag passed tommap()ormprotect(). Since this is a new memory type in the AttrIndx field of a pte, simplify the or'ing of these bits over the protection_map[] attributes by makingMT_NORMALindex 0.
Should the userspace stack always be mapped as if with PROT_MTE if the hardware supports it? Such a change would be invisible to non-MTE aware userspace since it would already need to opt in to tag checking via prctl. This would let userspace avoid a complex stack initialization sequence when running with stack tagging enabled on the main thread.
Peter
Catalin Marinas
maintainerRe: [PATCH v4 11/26] arm64: mte: Add PROT_MTE support to mmap() and mprotect()
Quoted 7 lines · Peter Collingbourne
Quoted 4 lines · Catalin Marinas
To enable tagging on a memory range, the user must explicitly opt in via a new
PROT_MTEflag passed tommap()ormprotect(). Since this is a new memory type in the AttrIndx field of a pte, simplify the or'ing of these bits over the protection_map[] attributes by makingMT_NORMALindex 0.
Quoted 6 lines
Should the userspace stack always be mapped as if with
PROT_MTEif the hardware supports it? Such a change would be invisible to non-MTE aware userspace since it would already need to opt in to tag checking via prctl. This would let userspace avoid a complex stack initialization sequence when running with stack tagging enabled on the main thread.
I don't think the stack initialisation is that difficult. On program startup (can be the dynamic loader). Something like (untested):
register unsigned long stack asm ("sp");
unsigned long page_sz = sysconf(_SC_PAGESIZE);
mprotect((void *)(stack & ~(page_sz - 1)), page_sz,
PROT_READ | PROT_WRITE | PROT_MTE | PROT_GROWSDOWN);(the essential part it PROT_GROWSDOWN so that you don't have to specify a stack lower limit)
I don't like enabling this by default since it will have a small cost even if the application doesn't enable tag checking. The kernel would still have to zero the tags when mapping the stack and preserve them when swapping out.
Another case where this could go wrong is if we want enable some quiet monitoring of user programs: the libc enables PROT_MTE on heap allocations but keeps tag checking disabled as it doesn't want any SIGSEGV; the kernel could enable async TCF and log any faults (rate-limited). Default PROT_MTE stack would get in the way. Anyway, this use-case is something for the future, so far these patches rely on the user solely driving the tag checking mode.
I'm fine, however, with enabling PROT_MTE on the main stack based on some ELF note.
Signature · 1 line
Catalin
Re: [PATCH v4 11/26] arm64: mte: Add PROT_MTE support to mmap() and mprotect()
Quoted 18 lines · Catalin Marinas
Quoted 7 lines · Peter Collingbourne
Quoted 4 lines · Catalin Marinas
To enable tagging on a memory range, the user must explicitly opt in via a new
PROT_MTEflag passed tommap()ormprotect(). Since this is a new memory type in the AttrIndx field of a pte, simplify the or'ing of these bits over the protection_map[] attributes by makingMT_NORMALindex 0.Quoted 6 lines
Should the userspace stack always be mapped as if with
PROT_MTEif the hardware supports it? Such a change would be invisible to non-MTE aware userspace since it would already need to opt in to tag checking via prctl. This would let userspace avoid a complex stack initialization sequence when running with stack tagging enabled on the main thread.I don't think the stack initialisation is that difficult. On program startup (can be the dynamic loader). Something like (untested):
Quoted 8 lines
register unsigned long stack asm ("sp"); unsigned long page_sz = sysconf(_SC_PAGESIZE); mprotect((void *)(stack & ~(page_sz - 1)), page_sz, PROT_READ | PROT_WRITE | PROT_MTE | PROT_GROWSDOWN);(the essential part it
PROT_GROWSDOWNso that you don't have to specify a stack lower limit)
does this work even if the currently mapped stack is more than page_sz? determining the mapped main stack area is i think non-trivial to do in userspace (requires parsing /proc/self/maps or similar).
...
Quoted 2 lines
I'm fine, however, with enabling
PROT_MTEon the main stack based on some ELF note.
note that would likely mean an elf note on the dynamic linker (because a dynamic linked executable may not be loaded by the kernel and ctors in loaded libs run before the executable entry code anyway, so the executable alone cannot be in charge of this decision) i.e. one global switch for all dynamic linked binaries.
i think a dynamic linker can map a new stack and switch to it if it needs to control the properties of the stack at runtime (it's wasteful though).
and i think there should be a runtime mechanism for the brk area:
it should be possible to request that future brk expansions are mapped as PROT_MTE so an mte aware malloc implementation can use brk. i think this is not important in the initial design, but if a prctl flag can do it that may be useful to add (may be at a later time).
(and eventually there should be a way to use PROT_MTE on writable global data and appropriate code generation that takes colors into account when globals are accessed, but that requires significant ELF, ld.so and compiler changes, that need not be part of the initial mte design).
Catalin Marinas
maintainerRe: [PATCH v4 11/26] arm64: mte: Add PROT_MTE support to mmap() and mprotect()
Quoted 25 lines · Szabolcs Nagy
Quoted 24 lines · Catalin Marinas
Quoted 7 lines · Peter Collingbourne
Quoted 4 lines · Catalin Marinas
To enable tagging on a memory range, the user must explicitly opt in via a new
PROT_MTEflag passed tommap()ormprotect(). Since this is a new memory type in the AttrIndx field of a pte, simplify the or'ing of these bits over the protection_map[] attributes by makingMT_NORMALindex 0.Quoted 6 lines
Should the userspace stack always be mapped as if with
PROT_MTEif the hardware supports it? Such a change would be invisible to non-MTE aware userspace since it would already need to opt in to tag checking via prctl. This would let userspace avoid a complex stack initialization sequence when running with stack tagging enabled on the main thread.I don't think the stack initialisation is that difficult. On program startup (can be the dynamic loader). Something like (untested):
register unsigned long stack asm ("sp"); unsigned long page_sz = sysconf(_SC_PAGESIZE); mprotect((void *)(stack & ~(page_sz - 1)), page_sz, PROT_READ | PROT_WRITE | PROT_MTE | PROT_GROWSDOWN);
Quoted 6 lines
Quoted 2 lines
(the essential part it
PROT_GROWSDOWNso that you don't have to specify a stack lower limit)does this work even if the currently mapped stack is more than
page_sz? determining the mapped main stack area is i think non-trivial to do in userspace (requires parsing/proc/self/mapsor similar).
Because of PROT_GROWSDOWN, the kernel adjusts the start of the range down automatically. It is potentially problematic if the top of the stack is more than a page away and you want the whole stack coloured. I haven't run a test but my reading of the kernel code is that the stack vma would be split in this scenario, so the range beyond sp+page_sz won't have PROT_MTE set.
My assumption is that if you do this during program start, the stack is smaller than a page. Alternatively, could we use argv or envp to determine the top of the user stack (the bottom is taken care of by the kernel)?
Quoted 8 lines
Quoted 2 lines
I'm fine, however, with enabling
PROT_MTEon the main stack based on some ELF note.note that would likely mean an elf note on the dynamic linker (because a dynamic linked executable may not be loaded by the kernel and ctors in loaded libs run before the executable entry code anyway, so the executable alone cannot be in charge of this decision) i.e. one global switch for all dynamic linked binaries.
I guess parsing such note in the kernel is only useful for static binaries.
Quoted 3 lines
i think a dynamic linker can map a new stack and switch to it if it needs to control the properties of the stack at runtime (it's wasteful though).
There is already user code to check for HWCAP2_MTE and the prctl(), so adding an mprotect() doesn't look like a significant overhead.
Quoted 6 lines
and i think there should be a runtime mechanism for the brk area:
it should be possible to request that future brk expansions are mapped asPROT_MTEso an mte aware malloc implementation can use brk. i think this is not important in the initial design, but if a prctl flag can do it that may be useful to add (may be at a later time).
Looking at the kernel code briefly, I think this would work. We do end up with two vmas for the brk, only the expansion having PROT_MTE, and I'd to find a way to store the extra flag.
From a coding perspective, it's easier to just set PROT_MTE by default on both brk and initial stack ;) (VM_DATA_DEFAULT_FLAGS).
Quoted 5 lines
(and eventually there should be a way to use
PROT_MTEon writable global data and appropriate code generation that takes colors into account when globals are accessed, but that requires significant ELF, ld.so and compiler changes, that need not be part of the initial mte design).
The .data section needs to be driven by the ELF information. It's also a file mapping and we don't support PROT_MTE on them even if MAP_PRIVATE. There are complications like DAX where the file you mmap for CoW may be hosted on memory that does not support MTE (copied to RAM on write).
Is there a use-case for global data to be tagged?
Signature · 1 line
Catalin
Re: [PATCH v4 11/26] arm64: mte: Add PROT_MTE support to mmap() and mprotect()
Quoted 46 lines · Catalin Marinas
Quoted 5 lines · Szabolcs Nagy
Quoted 4 lines · Catalin Marinas
Quoted 3 lines · Peter Collingbourne
Quoted 1 line · Catalin Marinas
To enable tagging on a memory range, the user must explicitly opt
in via
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
a new
PROT_MTEflag passed tommap()ormprotect(). Since this isa new
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
memory type in the AttrIndx field of a pte, simplify the or'ing of
these
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
bits over the protection_map[] attributes by making
MT_NORMALindex 0.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Should the userspace stack always be mapped as if with
PROT_MTEifthe
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
hardware supports it? Such a change would be invisible to non-MTE aware userspace since it would already need to opt in to tag checking via prctl. This would let userspace avoid a complex stack initialization sequence when running with stack tagging enabled on
the
Quoted 11 lines
Quoted 11 lines
Quoted 1 line
main thread.
I don't think the stack initialisation is that difficult. On program startup (can be the dynamic loader). Something like (untested):
register unsigned long stack asm ("sp"); unsigned long page_sz = sysconf(_SC_PAGESIZE); mprotect((void *)(stack & ~(page_sz - 1)), page_sz, PROT_READ | PROT_WRITE | PROT_MTE | PROT_GROWSDOWN);Quoted 6 lines
Quoted 2 lines
(the essential part it
PROT_GROWSDOWNso that you don't have to specify a stack lower limit)does this work even if the currently mapped stack is more than
page_sz? determining the mapped main stack area is i think non-trivial to do in userspace (requires parsing/proc/self/mapsor similar).Because of
PROT_GROWSDOWN, the kernel adjusts the start of the range down automatically. It is potentially problematic if the top of the stack is more than a page away and you want the whole stack coloured. I haven't run a test but my reading of the kernel code is that the stack vma would be split in this scenario, so the range beyond sp+page_sz won't havePROT_MTEset.
Quoted 5 lines
My assumption is that if you do this during program start, the stack is smaller than a page. Alternatively, could we use argv or envp to determine the top of the user stack (the bottom is taken care of by the kernel)?
PROT_GROWSDOWN seems to work fine in our case, and the extra tag maintenance overhead sounds like a valid argument against setting PROT_MTE unconditionally.
On the other hand, we may end up doing this in the userspace in every process. The reason is, PROT_MTE can not be set on a page that contains a live frame with stack tagging because of mismatching tags (IRG is not affected by PROT_MTE but STG is). So ideally, this should be done at (or near) the program entry point, while the stack is mostly empty.
Quoted 40 lines
Quoted 9 lines
Quoted 3 lines
I'm fine, however, with enabling
PROT_MTEon the main stack based on some ELF note.note that would likely mean an elf note on the dynamic linker (because a dynamic linked executable may not be loaded by the kernel and ctors in loaded libs run before the executable entry code anyway, so the executable alone cannot be in charge of this decision) i.e. one global switch for all dynamic linked binaries.
I guess parsing such note in the kernel is only useful for static binaries.
Quoted 3 lines
i think a dynamic linker can map a new stack and switch to it if it needs to control the properties of the stack at runtime (it's wasteful though).
There is already user code to check for
HWCAP2_MTEand theprctl(), so adding anmprotect()doesn't look like a significant overhead.Quoted 6 lines
and i think there should be a runtime mechanism for the brk area:
it should be possible to request that future brk expansions are mapped asPROT_MTEso an mte aware malloc implementation can use brk. i think this is not important in the initial design, but if a prctl flag can do it that may be useful to add (may be at a later time).Looking at the kernel code briefly, I think this would work. We do end up with two vmas for the brk, only the expansion having
PROT_MTE, and I'd to find a way to store the extra flag.From a coding perspective, it's easier to just set
PROT_MTEby default on both brk and initial stack ;) (VM_DATA_DEFAULT_FLAGS).Quoted 5 lines
(and eventually there should be a way to use
PROT_MTEon writable global data and appropriate code generation that takes colors into account when globals are accessed, but that requires significant ELF, ld.so and compiler changes, that need not be part of the initial mte design).
Quoted 7 lines
The .data section needs to be driven by the ELF information. It's also a file mapping and we don't support
PROT_MTEon them even ifMAP_PRIVATE. There are complications like DAX where the file you mmap for CoW may be hosted on memory that does not support MTE (copied to RAM on write).Is there a use-case for global data to be tagged?
Yes, catching global buffer overflow bugs. They are not nearly as common as heap-based issues though.
Quoted 4 lines
--
Catalin
Catalin Marinas
maintainerRe: [PATCH v4 11/26] arm64: mte: Add PROT_MTE support to mmap() and mprotect()
Quoted 35 lines · Evgenii Stepanov
Quoted 25 lines · Catalin Marinas
Quoted 10 lines · Szabolcs Nagy
Quoted 9 lines · Catalin Marinas
I don't think the stack initialisation is that difficult. On program startup (can be the dynamic loader). Something like (untested):
register unsigned long stack asm ("sp"); unsigned long page_sz = sysconf(_SC_PAGESIZE); mprotect((void *)(stack & ~(page_sz - 1)), page_sz, PROT_READ | PROT_WRITE | PROT_MTE | PROT_GROWSDOWN);Quoted 6 lines
Quoted 2 lines
(the essential part it
PROT_GROWSDOWNso that you don't have to specify a stack lower limit)does this work even if the currently mapped stack is more than
page_sz? determining the mapped main stack area is i think non-trivial to do in userspace (requires parsing/proc/self/mapsor similar).Because of
PROT_GROWSDOWN, the kernel adjusts the start of the range down automatically. It is potentially problematic if the top of the stack is more than a page away and you want the whole stack coloured. I haven't run a test but my reading of the kernel code is that the stack vma would be split in this scenario, so the range beyond sp+page_sz won't havePROT_MTEset.Quoted 4 lines
My assumption is that if you do this during program start, the stack is smaller than a page. Alternatively, could we use argv or envp to determine the top of the user stack (the bottom is taken care of by the kernel)?
PROT_GROWSDOWNseems to work fine in our case, and the extra tag maintenance overhead sounds like a valid argument against settingPROT_MTEunconditionally.
Quoted 5 lines
On the other hand, we may end up doing this in the userspace in every process. The reason is,
PROT_MTEcan not be set on a page that contains a live frame with stack tagging because of mismatching tags (IRG is not affected byPROT_MTEbut STG is). So ideally, this should be done at (or near) the program entry point, while the stack is mostly empty.
Since stack tagging cannot use instructions in the NOP space anyway, I think we need an ELF note to check for the presence of STG etc. and, in addition, we can turn PROT_MTE by default on the initial stack. Maybe on such binaries we could just set PROT_MTE on all anonymous and ramfs mappings (i.e. VM_MTE_ALLOWED implies VM_MTE).
For dynamically linked binaries, we base this decision on the main ELF, not the interpreter, and it would be up to the dynamic loader to reject libraries that have such note when HWCAP2_MTE is not present.
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
(and eventually there should be a way to use
PROT_MTEon writable global data and appropriate code generation that takes colors into account when globals are accessed, but that requires significant ELF, ld.so and compiler changes, that need not be part of the initial mte design).The .data section needs to be driven by the ELF information. It's also a file mapping and we don't support
PROT_MTEon them even ifMAP_PRIVATE. There are complications like DAX where the file you mmap for CoW may be hosted on memory that does not support MTE (copied to RAM on write).
Quoted 4 lines
Quoted 1 line
Is there a use-case for global data to be tagged?
Yes, catching global buffer overflow bugs. They are not nearly as common as heap-based issues though.
OK, so these would be tagged red-zones around global data. IIUC, having different colours for global variables was not considered because of the relocations and relative accesses.
If such red-zone colouring is done during load (the dynamic linker?), we could set PROT_MTE only when MAP_PRIVATE and copied on write to make sure it is in RAM. As above, I think this should be driven by some ELF information.
There's also the option of scrapping PROT_MTE altogether and enabling MTE (default tag 0) on all anonymous and private+copied pages (i.e. those stored in RAM). At this point, I can't really tell whether there will be a performance impact.
Signature · 1 line
Catalin
Re: [PATCH v4 11/26] arm64: mte: Add PROT_MTE support to mmap() and mprotect()
In reply to Catalin MarinasQuoted 3 lines · Catalin Marinas
Quoted 2 lines · Szabolcs Nagy
Quoted 1 line · Catalin Marinas
On Wed, May 27, 2020 at 11:57:39AM -0700, Peter Collingbourne wrote:
[...]
Just jumping in on this point:
Quoted 30 lines
Quoted 16 lines
Quoted 16 lines
Quoted 6 lines
Should the userspace stack always be mapped as if with
PROT_MTEif the hardware supports it? Such a change would be invisible to non-MTE aware userspace since it would already need to opt in to tag checking via prctl. This would let userspace avoid a complex stack initialization sequence when running with stack tagging enabled on the main thread.I don't think the stack initialisation is that difficult. On program startup (can be the dynamic loader). Something like (untested):
register unsigned long stack asm ("sp"); unsigned long page_sz = sysconf(_SC_PAGESIZE); mprotect((void *)(stack & ~(page_sz - 1)), page_sz, PROT_READ | PROT_WRITE | PROT_MTE | PROT_GROWSDOWN);Quoted 6 lines
Quoted 2 lines
(the essential part it
PROT_GROWSDOWNso that you don't have to specify a stack lower limit)does this work even if the currently mapped stack is more than
page_sz? determining the mapped main stack area is i think non-trivial to do in userspace (requires parsing/proc/self/mapsor similar).Because of
PROT_GROWSDOWN, the kernel adjusts the start of the range down automatically. It is potentially problematic if the top of the stack is more than a page away and you want the whole stack coloured. I haven't run a test but my reading of the kernel code is that the stack vma would be split in this scenario, so the range beyond sp+page_sz won't havePROT_MTEset.
Quoted 4 lines
My assumption is that if you do this during program start, the stack is smaller than a page. Alternatively, could we use argv or envp to determine the top of the user stack (the bottom is taken care of by the kernel)?
I don't think you can easily know when the stack ends, but perhaps it doesn't matter.
From memory, the initial stack looks like:
argv/env strings AT_NULL auxv NULL env NULL argv argc <--- sp
If we don't care about tagging the strings correctly, we could step to the end of auxv and tag down from there.
If we do care about tagging the strings, there's probably no good way to find the end of the string area, other than looking up sp in /proc/self/maps. I'm not sure we should trust all past and future kernels to spit out the strings in a predictable order.
Assuming that the last env string has the highest address does not sounds like a good idea to me. It would be easy for someone to break that assumption later without realising.
If we're concerned about this, and reading /proc/self/auxv is deemed unacceptable (likely: some binaries need to work before /proc is mounted) then we could perhaps add a new auxv entry to report the stack base address to the user startup code.
I don't think it matters if all this is "hard" for userspace: only the C library / runtime should be doing this. After libc startup, it's generally too late to do this kind of thing safely.
[...]
Cheers
---Dave
Catalin Marinas
maintainerRe: [PATCH v4 11/26] arm64: mte: Add PROT_MTE support to mmap() and mprotect()
Quoted 53 lines · Dave P Martin
Quoted 33 lines · Catalin Marinas
Quoted 18 lines · Szabolcs Nagy
Quoted 17 lines · Catalin Marinas
Quoted 6 lines · Peter Collingbourne
Should the userspace stack always be mapped as if with
PROT_MTEif the hardware supports it? Such a change would be invisible to non-MTE aware userspace since it would already need to opt in to tag checking via prctl. This would let userspace avoid a complex stack initialization sequence when running with stack tagging enabled on the main thread.I don't think the stack initialisation is that difficult. On program startup (can be the dynamic loader). Something like (untested):
register unsigned long stack asm ("sp"); unsigned long page_sz = sysconf(_SC_PAGESIZE); mprotect((void *)(stack & ~(page_sz - 1)), page_sz, PROT_READ | PROT_WRITE | PROT_MTE | PROT_GROWSDOWN);Quoted 6 lines
Quoted 2 lines
(the essential part it
PROT_GROWSDOWNso that you don't have to specify a stack lower limit)does this work even if the currently mapped stack is more than
page_sz? determining the mapped main stack area is i think non-trivial to do in userspace (requires parsing/proc/self/mapsor similar).Because of
PROT_GROWSDOWN, the kernel adjusts the start of the range down automatically. It is potentially problematic if the top of the stack is more than a page away and you want the whole stack coloured. I haven't run a test but my reading of the kernel code is that the stack vma would be split in this scenario, so the range beyond sp+page_sz won't havePROT_MTEset.Quoted 4 lines
My assumption is that if you do this during program start, the stack is smaller than a page. Alternatively, could we use argv or envp to determine the top of the user stack (the bottom is taken care of by the kernel)?
I don't think you can easily know when the stack ends, but perhaps it doesn't matter.
From memory, the initial stack looks like:
argv/env strings AT_NULL auxv NULL env NULL argv argc <--- sp
Quoted 7 lines
If we don't care about tagging the strings correctly, we could step to the end of auxv and tag down from there.
If we do care about tagging the strings, there's probably no good way to find the end of the string area, other than looking up sp in
/proc/self/maps. I'm not sure we should trust all past and future kernels to spit out the strings in a predictable order.
I don't think we care about tagging whatever the kernel places on the stack since the argv/envp pointers are untagged. An mprotect(PROT_MTE) may or may not cover the environment but it shouldn't matter as the kernel clears the tags on the corresponding pages anyway.
AFAIK stack tagging works by colouring a stack frame on function entry and clearing the tags on return. We would only hit a problem if the function issuing mprotect(sp, PROT_MTE) on and its callers already assumed a PROT_MTE stack. Without PROT_MTE, an STG would be write-ignore, so subsequently turning it on would lead to a mismatch between the pointer and the allocation tags.
So PROT_MTE turning on should happen very early in the user process startup code before any code with stack tagging enabled. Whether you reach the top of the stack with such mprotect() doesn't really matter since up to that point there should not be any use of stack tagging. If that's not possible, for example the glibc code setting up the stack was compiled to stack tagging itself, the kernel would have to enable it when the user process starts. However, I'd only do this based on some ELF note.
Signature · 1 line
Catalin
Re: [PATCH v4 11/26] arm64: mte: Add PROT_MTE support to mmap() and mprotect()
Quoted 62 lines · Catalin Marinas
Quoted 53 lines · Dave P Martin
Quoted 33 lines · Catalin Marinas
Quoted 18 lines · Szabolcs Nagy
Quoted 17 lines · Catalin Marinas
Quoted 6 lines · Peter Collingbourne
Should the userspace stack always be mapped as if with
PROT_MTEif the hardware supports it? Such a change would be invisible to non-MTE aware userspace since it would already need to opt in to tag checking via prctl. This would let userspace avoid a complex stack initialization sequence when running with stack tagging enabled on the main thread.I don't think the stack initialisation is that difficult. On program startup (can be the dynamic loader). Something like (untested):
register unsigned long stack asm ("sp"); unsigned long page_sz = sysconf(_SC_PAGESIZE); mprotect((void *)(stack & ~(page_sz - 1)), page_sz, PROT_READ | PROT_WRITE | PROT_MTE | PROT_GROWSDOWN);Quoted 6 lines
Quoted 2 lines
(the essential part it
PROT_GROWSDOWNso that you don't have to specify a stack lower limit)does this work even if the currently mapped stack is more than
page_sz? determining the mapped main stack area is i think non-trivial to do in userspace (requires parsing/proc/self/mapsor similar).Because of
PROT_GROWSDOWN, the kernel adjusts the start of the range down automatically. It is potentially problematic if the top of the stack is more than a page away and you want the whole stack coloured. I haven't run a test but my reading of the kernel code is that the stack vma would be split in this scenario, so the range beyond sp+page_sz won't havePROT_MTEset.Quoted 4 lines
My assumption is that if you do this during program start, the stack is smaller than a page. Alternatively, could we use argv or envp to determine the top of the user stack (the bottom is taken care of by the kernel)?
I don't think you can easily know when the stack ends, but perhaps it doesn't matter.
From memory, the initial stack looks like:
argv/env strings AT_NULL auxv NULL env NULL argv argc <--- spQuoted 7 lines
If we don't care about tagging the strings correctly, we could step to the end of auxv and tag down from there.
If we do care about tagging the strings, there's probably no good way to find the end of the string area, other than looking up sp in
/proc/self/maps. I'm not sure we should trust all past and future kernels to spit out the strings in a predictable order.
Quoted 4 lines
I don't think we care about tagging whatever the kernel places on the stack since the argv/envp pointers are untagged. An
mprotect(PROT_MTE)may or may not cover the environment but it shouldn't matter as the kernel clears the tags on the corresponding pages anyway.
We have no match-all tag, right? So we do rely on the tags being cleared for the initial stack contents so that using untagged pointers to access it works.
Quoted 7 lines
AFAIK stack tagging works by colouring a stack frame on function entry and clearing the tags on return. We would only hit a problem if the function issuing
mprotect(sp, PROT_MTE)on and its callers already assumed aPROT_MTEstack. WithoutPROT_MTE, an STG would be write-ignore, so subsequently turning it on would lead to a mismatch between the pointer and the allocation tags.
Quoted 8 lines
So
PROT_MTEturning on should happen very early in the user process startup code before any code with stack tagging enabled. Whether you reach the top of the stack with suchmprotect()doesn't really matter since up to that point there should not be any use of stack tagging. If that's not possible, for example the glibc code setting up the stack was compiled to stack tagging itself, the kernel would have to enable it when the user process starts. However, I'd only do this based on some ELF note.
Sounds fair.
This early on, the process shouldn't be exposed to arbitrary, untrusted data. So it's probably not a problem that tagging isn't turned on right from the start.
Cheers
---Dave
Catalin Marinas
maintainerPATCH v4 12/26 mm: Introduce arch_validate_flags()
In reply to Catalin MarinasSimilarly to arch_validate_prot() called from do_mprotect_pkey(), an architecture may need to sanity-check the new vm_flags.
Define a dummy function always returning true. In addition to do_mprotect_pkey(), also invoke it from mmap_region() prior to updating vma->vm_page_prot to allow the architecture code to veto potentially inconsistent vm_flags.
---
Notes:
v2:
- Some comments updated.
include/linux/mman.h | 13 +++++++++++++ mm/mmap.c | 9 +++++++++ mm/mprotect.c | 6 ++++++ 3 files changed, 28 insertions(+)
include/linux/mman.h +13 -0
@@ -103,6 +103,19 @@ static inline bool arch_validate_prot(unsigned long prot, unsigned long addr)#define arch_validate_prot arch_validate_prot#endif +#ifndef arch_validate_flags +/* +* This is called from mmap() and mprotect() with the updated vma->vm_flags. +* +* Returns true if the VM_* flags are valid. +*/ +static inline bool arch_validate_flags (unsigned long flags) +{ +return true; +} +#define arch_validate_flags arch_validate_flags +#endif +/* * Optimisation macro. It is equivalent to: * (x & bit1) ? bit2 : 0
mm/mmap.c +9 -0
@@ -1792,6 +1792,15 @@ unsigned long mmap_region(struct file *file, unsigned long addr,vma_set_anonymous (vma); } +/* Allow architectures to sanity-check the vm_flags */ +if (!arch_validate_flags (vma->vm_flags)) { + error = -EINVAL ; +if (file) + goto unmap_and_free_vma; +else + goto free_vma; + } +vma_link (mm, vma, prev, rb_link, rb_parent);/* Once vma denies write, undo our temporary denial count */ if (file) {
mm/mprotect.c +6 -0
@@ -603,6 +603,12 @@ static int do_mprotect_pkey(unsigned long start, size_t len, goto out; } +/* Allow architectures to sanity-check the new flags */ +if (!arch_validate_flags (newflags)) { + error = -EINVAL ; + goto out; + } + error =security_file_mprotect (vma, reqprot, prot);if (error) goto out;
Catalin Marinas
maintainerPATCH v4 13/26 arm64: mte: Validate the PROT_MTE request via arch_validate_flags()
In reply to Catalin MarinasMake use of the newly introduced arch_validate_flags() hook to sanity-check the PROT_MTE request passed to mmap() and mprotect(). If the mapping does not support MTE, these syscalls will return -EINVAL.
--- arch/arm64/include/asm/mman.h | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
arch/arm64/include/asm/mman.h +15 -1
@@ -44,7 +44,11 @@ static inline unsigned long arch_calc_vm_flag_bits(unsigned long flags)static inline pgprot_t arch_vm_get_page_prot (unsigned long vm_flags) { -return (vm_flags &VM_MTE ) && (vm_flags &VM_MTE_ALLOWED ) ? +/* +* Checking for VM_MTE only is sufficient since arch_validate_flags() +* does not permit (VM_MTE & !VM_MTE_ALLOWED). +*/ +return (vm_flags &VM_MTE ) ?__pgprot (PTE_ATTRINDX (MT_NORMAL_TAGGED )) :__pgprot (0 ); }
@@ -61,4 +65,14 @@ static inline bool arch_validate_prot(unsigned long prot, unsigned long addr) }#define arch_validate_prot arch_validate_prot +static inline bool arch_validate_flags (unsigned long flags) +{ +if (!system_supports_mte ()) +return true; + +/* only allow VM_MTE if VM_MTE_ALLOWED has been set previously */ +return !(flags &VM_MTE ) || (flags &VM_MTE_ALLOWED ); +} +#define arch_validate_flags arch_validate_flags +#endif /* !__ASM_MMAN_H__ */
Catalin Marinas
maintainerPATCH v4 14/26 mm: Allow arm64 mmap(PROT_MTE) on RAM-based files
In reply to Catalin MarinasSince arm64 memory (allocation) tags can only be stored in RAM, mapping files with PROT_MTE is not allowed by default. RAM-based files like those in a tmpfs mount or memfd_create() can support memory tagging, so update the vm_flags accordingly in shmem_mmap().
--- mm/shmem.c | 3 +++ 1 file changed, 3 insertions(+)
mm/shmem.c +3 -0
@@ -2221,6 +2221,9 @@ static int shmem_mmap(struct file *file, struct vm_area_struct *vma) vma->vm_flags &= ~(VM_MAYWRITE ); } +/* arm64 - allow memory tagging on RAM-based files */ + vma->vm_flags |=VM_MTE_ALLOWED ; +file_accessed (file); vma->vm_ops = &shmem_vm_ops; if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
Catalin Marinas
maintainerPATCH v4 15/26 arm64: mte: Allow user control of the tag check mode via prctl()
In reply to Catalin MarinasBy default, even if PROT_MTE is set on a memory range, there is no tag check fault reporting (SIGSEGV). Introduce a set of option to the exiting prctl(PR_SET_TAGGED_ADDR_CTRL) to allow user control of the tag check fault mode:
PR_MTE_TCF_NONE - no reporting (default) PR_MTE_TCF_SYNC - synchronous tag check fault reporting PR_MTE_TCF_ASYNC - asynchronous tag check fault reporting
These options translate into the corresponding SCTLR_EL1.TCF0 bitfield, context-switched by the kernel. Note that uaccess done by the kernel is not checked and cannot be configured by the user.
---
Notes:
v3:
- Use SCTLR_EL1_TCF0_NONE instead of 0 for consistency.
- Move mte_thread_switch() in this patch from an earlier one. In addition, it is called after the dsb() in __switch_to() so that any asynchronous tag check faults have been registered in the TFSR_EL1 registers (to be added with the in-kernel MTE support.
v2:
- Handle SCTLR_EL1_TCF0_NONE explicitly for consistency with PR_MTE_TCF_NONE.
- Fix SCTLR_EL1 register setting in flush_mte_state() (thanks to Peter Collingbourne).
- Added ISB to update_sctlr_el1_tcf0() since, with the latest architecture update/fix, the TCF0 field is used by the uaccess routines.
arch/arm64/include/asm/mte.h | 14 ++++++ arch/arm64/include/asm/processor.h | 3 ++ arch/arm64/kernel/mte.c | 77 ++++++++++++++++++++++++++++++ arch/arm64/kernel/process.c | 26 ++++++++-- include/uapi/linux/prctl.h | 6 +++ 5 files changed, 123 insertions(+), 3 deletions(-)
arch/arm64/include/asm/mte.h +14 -0
@@ -21,6 +21,9 @@ void mte_clear_page_tags(void *addr, size_t size);void mte_sync_tags (pte_t *ptep,pte_t pte);void mte_copy_page_tags (void *kto,const void *kfrom);void flush_mte_state (void ); +void mte_thread_switch (struct task_struct *next); +long set_mte_ctrl (unsigned long arg); +long get_mte_ctrl (void );#else
@@ -36,6 +39,17 @@ static inline void mte_copy_page_tags(void *kto, const void *kfrom)static inline void flush_mte_state (void ) { } +static inline void mte_thread_switch (struct task_struct *next) +{ +} +static inline long set_mte_ctrl (unsigned long arg) +{ +return 0 ; +} +static inline long get_mte_ctrl (void ) +{ +return 0 ; +}#endif
arch/arm64/include/asm/processor.h +3 -0
@@ -151,6 +151,9 @@ struct thread_struct {struct ptrauth_keys_user keys_user;struct ptrauth_keys_kernel keys_kernel;#endif +#ifdef CONFIG_ARM64_MTE +u64 sctlr_tcf0; +#endif }; static inline void arch_thread_struct_whitelist(unsigned long *offset,
arch/arm64/kernel/mte.c +77 -0
93 lines not on this page · raw message
arch/arm64/kernel/process.c +23 -3
@@ -529,6 +529,13 @@ __notrace_funcgraph struct task_struct *__switch_to(struct task_struct *prev,*/ dsb (ish); +/* +* MTE thread switching must happen after the DSB above to ensure that +* any asynchronous tag check faults have been logged in the TFSR*_EL1 +* registers. +*/ +mte_thread_switch (next); +/* the actual thread switch */ last =cpu_switch_to (prev, next);
@@ -588,9 +595,15 @@ static unsigned int tagged_addr_disabled;long set_tagged_addr_ctrl (unsigned long arg) { +unsigned long valid_mask =PR_TAGGED_ADDR_ENABLE ; +if (is_compat_task ())return -EINVAL ; -if (arg & ~PR_TAGGED_ADDR_ENABLE ) + +if (system_supports_mte ()) + valid_mask |=PR_MTE_TCF_MASK ; + +if (arg & ~valid_mask)return -EINVAL ;/*
@@ -600,6 +613,9 @@ long set_tagged_addr_ctrl(unsigned long arg)if (arg &PR_TAGGED_ADDR_ENABLE && tagged_addr_disabled)return -EINVAL ; +if (set_mte_ctrl (arg) !=0 ) +return -EINVAL ; +update_thread_flag (TIF_TAGGED_ADDR , arg &PR_TAGGED_ADDR_ENABLE );return 0 ;
@@ -607,13 +623,17 @@ long set_tagged_addr_ctrl(unsigned long arg)long get_tagged_addr_ctrl (void ) { +long ret =0 ; +if (is_compat_task ())return -EINVAL ;if (test_thread_flag (TIF_TAGGED_ADDR )) -return PR_TAGGED_ADDR_ENABLE ; + ret =PR_TAGGED_ADDR_ENABLE ; -return 0 ; + ret |=get_mte_ctrl (); + +return ret; }/*
include/uapi/linux/prctl.h +6 -0
@@ -233,6 +233,12 @@ struct prctl_mm_map {#define PR_SET_TAGGED_ADDR_CTRL 55#define PR_GET_TAGGED_ADDR_CTRL 56# define PR_TAGGED_ADDR_ENABLE (1UL << 0) +/* MTE tag check fault modes */ +# define PR_MTE_TCF_SHIFT 1 +# define PR_MTE_TCF_NONE (0UL << PR_MTE_TCF_SHIFT) +# define PR_MTE_TCF_SYNC (1UL << PR_MTE_TCF_SHIFT) +# define PR_MTE_TCF_ASYNC (2UL << PR_MTE_TCF_SHIFT) +# define PR_MTE_TCF_MASK (3UL << PR_MTE_TCF_SHIFT)/* Control reclaim behavior when allocating memory */ #define PR_SET_IO_FLUSHER 57
Will Deacon
maintainerRe: [PATCH v4 15/26] arm64: mte: Allow user control of the tag check mode via prctl()
Quoted 33 lines · Catalin Marinas
By default, even if
PROT_MTEis set on a memory range, there is no tag check fault reporting (SIGSEGV). Introduce a set of option to the exitingprctl(PR_SET_TAGGED_ADDR_CTRL)to allow user control of the tag check fault mode:
PR_MTE_TCF_NONE- no reporting (default)PR_MTE_TCF_SYNC- synchronous tag check fault reportingPR_MTE_TCF_ASYNC- asynchronous tag check fault reportingThese options translate into the corresponding SCTLR_EL1.TCF0 bitfield, context-switched by the kernel. Note that uaccess done by the kernel is not checked and cannot be configured by the user.
Signed-off-by Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>---
Notes:
v3:
- UseSCTLR_EL1_TCF0_NONEinstead of 0 for consistency.
- Movemte_thread_switch()in this patch from an earlier one. In addition, it is called after thedsb()in__switch_to()so that any asynchronous tag check faults have been registered in theTFSR_EL1registers (to be added with the in-kernel MTE support.v2:
- HandleSCTLR_EL1_TCF0_NONEexplicitly for consistency withPR_MTE_TCF_NONE.
- FixSCTLR_EL1register setting inflush_mte_state()(thanks to Peter Collingbourne).
- Added ISB toupdate_sctlr_el1_tcf0()since, with the latest architecture update/fix, the TCF0 field is used by the uaccess routines.
Quoted 6 lines
arch/arm64/include/asm/mte.h | 14 ++++++ arch/arm64/include/asm/processor.h | 3 ++ arch/arm64/kernel/mte.c | 77 ++++++++++++++++++++++++++++++ arch/arm64/kernel/process.c | 26 ++++++++-- include/uapi/linux/prctl.h | 6 +++ 5 files changed, 123 insertions(+), 3 deletions(-)
Dave is working on man pages for prctl() (and I think also ptrace). I think it would be /very/ useful for us to have some RFC patches on top of his work adding documentation for the MTE interactions, as we found some other minor issues/inconsistencies as a direct result of writing and reviewing the man page for our existing interfaces.
Cheers,
Will
Re: [PATCH v4 15/26] arm64: mte: Allow user control of the tag check mode via prctl()
Quoted 41 lines · Will Deacon
Quoted 33 lines · Catalin Marinas
By default, even if
PROT_MTEis set on a memory range, there is no tag check fault reporting (SIGSEGV). Introduce a set of option to the exitingprctl(PR_SET_TAGGED_ADDR_CTRL)to allow user control of the tag check fault mode:
PR_MTE_TCF_NONE- no reporting (default)PR_MTE_TCF_SYNC- synchronous tag check fault reportingPR_MTE_TCF_ASYNC- asynchronous tag check fault reportingThese options translate into the corresponding SCTLR_EL1.TCF0 bitfield, context-switched by the kernel. Note that uaccess done by the kernel is not checked and cannot be configured by the user.
Signed-off-by Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>---
Notes:
v3:
- UseSCTLR_EL1_TCF0_NONEinstead of 0 for consistency.
- Movemte_thread_switch()in this patch from an earlier one. In addition, it is called after thedsb()in__switch_to()so that any asynchronous tag check faults have been registered in theTFSR_EL1registers (to be added with the in-kernel MTE support.v2:
- HandleSCTLR_EL1_TCF0_NONEexplicitly for consistency withPR_MTE_TCF_NONE.
- FixSCTLR_EL1register setting inflush_mte_state()(thanks to Peter Collingbourne).
- Added ISB toupdate_sctlr_el1_tcf0()since, with the latest architecture update/fix, the TCF0 field is used by the uaccess routines.Quoted 6 lines
arch/arm64/include/asm/mte.h | 14 ++++++ arch/arm64/include/asm/processor.h | 3 ++ arch/arm64/kernel/mte.c | 77 ++++++++++++++++++++++++++++++ arch/arm64/kernel/process.c | 26 ++++++++-- include/uapi/linux/prctl.h | 6 +++ 5 files changed, 123 insertions(+), 3 deletions(-)
Quoted 5 lines
Dave is working on man pages for
prctl()(and I think also ptrace). I think it would be /very/ useful for us to have some RFC patches on top of his work adding documentation for the MTE interactions, as we found some other minor issues/inconsistencies as a direct result of writing and reviewing the man page for our existing interfaces.
I have a local draft for the address tagging and MTE prctls already btw. I hadn't posted them yet so as to focus on nailing the "easy" stuff down ;)
If I have time I'll try and get them posted today so that people can take a look before next week.
Cheers
---Dave
Will Deacon
maintainerRe: [PATCH v4 15/26] arm64: mte: Allow user control of the tag check mode via prctl()
Quoted 48 lines · Dave Martin
Quoted 41 lines · Will Deacon
Quoted 33 lines · Catalin Marinas
By default, even if
PROT_MTEis set on a memory range, there is no tag check fault reporting (SIGSEGV). Introduce a set of option to the exitingprctl(PR_SET_TAGGED_ADDR_CTRL)to allow user control of the tag check fault mode:
PR_MTE_TCF_NONE- no reporting (default)PR_MTE_TCF_SYNC- synchronous tag check fault reportingPR_MTE_TCF_ASYNC- asynchronous tag check fault reportingThese options translate into the corresponding SCTLR_EL1.TCF0 bitfield, context-switched by the kernel. Note that uaccess done by the kernel is not checked and cannot be configured by the user.
Signed-off-by Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>---
Notes:
v3:
- UseSCTLR_EL1_TCF0_NONEinstead of 0 for consistency.
- Movemte_thread_switch()in this patch from an earlier one. In addition, it is called after thedsb()in__switch_to()so that any asynchronous tag check faults have been registered in theTFSR_EL1registers (to be added with the in-kernel MTE support.v2:
- HandleSCTLR_EL1_TCF0_NONEexplicitly for consistency withPR_MTE_TCF_NONE.
- FixSCTLR_EL1register setting inflush_mte_state()(thanks to Peter Collingbourne).
- Added ISB toupdate_sctlr_el1_tcf0()since, with the latest architecture update/fix, the TCF0 field is used by the uaccess routines.Quoted 6 lines
arch/arm64/include/asm/mte.h | 14 ++++++ arch/arm64/include/asm/processor.h | 3 ++ arch/arm64/kernel/mte.c | 77 ++++++++++++++++++++++++++++++ arch/arm64/kernel/process.c | 26 ++++++++-- include/uapi/linux/prctl.h | 6 +++ 5 files changed, 123 insertions(+), 3 deletions(-)Quoted 5 lines
Dave is working on man pages for
prctl()(and I think also ptrace). I think it would be /very/ useful for us to have some RFC patches on top of his work adding documentation for the MTE interactions, as we found some other minor issues/inconsistencies as a direct result of writing and reviewing the man page for our existing interfaces.
Quoted 6 lines
I have a local draft for the address tagging and MTE prctls already btw. I hadn't posted them yet so as to focus on nailing the "easy" stuff down ;)
If I have time I'll try and get them posted today so that people can take a look before next week.
Oh, great! I wasn't meaning that you should be the one doing it, but if you're already drafted them that's really good. Might make sense for them to appear as RFC patches at the end of this series, to be honest, so the next posting (v5) can all be reviewed together.
But I'll leave that up to you and Catalin to figure out.
Will
Catalin Marinas
maintainerRe: [PATCH v4 15/26] arm64: mte: Allow user control of the tag check mode via prctl()
In reply to Dave MartinQuoted 48 lines · Dave P Martin
Quoted 41 lines · Will Deacon
Quoted 33 lines · Catalin Marinas
By default, even if
PROT_MTEis set on a memory range, there is no tag check fault reporting (SIGSEGV). Introduce a set of option to the exitingprctl(PR_SET_TAGGED_ADDR_CTRL)to allow user control of the tag check fault mode:
PR_MTE_TCF_NONE- no reporting (default)PR_MTE_TCF_SYNC- synchronous tag check fault reportingPR_MTE_TCF_ASYNC- asynchronous tag check fault reportingThese options translate into the corresponding SCTLR_EL1.TCF0 bitfield, context-switched by the kernel. Note that uaccess done by the kernel is not checked and cannot be configured by the user.
Signed-off-by Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>---
Notes:
v3:
- UseSCTLR_EL1_TCF0_NONEinstead of 0 for consistency.
- Movemte_thread_switch()in this patch from an earlier one. In addition, it is called after thedsb()in__switch_to()so that any asynchronous tag check faults have been registered in theTFSR_EL1registers (to be added with the in-kernel MTE support.v2:
- HandleSCTLR_EL1_TCF0_NONEexplicitly for consistency withPR_MTE_TCF_NONE.
- FixSCTLR_EL1register setting inflush_mte_state()(thanks to Peter Collingbourne).
- Added ISB toupdate_sctlr_el1_tcf0()since, with the latest architecture update/fix, the TCF0 field is used by the uaccess routines.Quoted 6 lines
arch/arm64/include/asm/mte.h | 14 ++++++ arch/arm64/include/asm/processor.h | 3 ++ arch/arm64/kernel/mte.c | 77 ++++++++++++++++++++++++++++++ arch/arm64/kernel/process.c | 26 ++++++++-- include/uapi/linux/prctl.h | 6 +++ 5 files changed, 123 insertions(+), 3 deletions(-)Quoted 5 lines
Dave is working on man pages for
prctl()(and I think also ptrace). I think it would be /very/ useful for us to have some RFC patches on top of his work adding documentation for the MTE interactions, as we found some other minor issues/inconsistencies as a direct result of writing and reviewing the man page for our existing interfaces.
Quoted 3 lines
I have a local draft for the address tagging and MTE prctls already btw. I hadn't posted them yet so as to focus on nailing the "easy" stuff down ;)
That's great Dave. Thanks!
Quoted 2 lines
If I have time I'll try and get them posted today so that people can take a look before next week.
Feel free to post them whenever you can. I'll include them in v5 (likely to be posted after the merging window).
Signature · 1 line
Catalin
Catalin Marinas
maintainerPATCH v4 16/26 arm64: mte: Allow user control of the generated random tags via prctl()
In reply to Catalin MarinasThe IRG, ADDG and SUBG instructions insert a random tag in the resulting address. Certain tags can be excluded via the GCR_EL1.Exclude bitmap when, for example, the user wants a certain colour for freed buffers. Since the GCR_EL1 register is not accessible at EL0, extend the prctl(PR_SET_TAGGED_ADDR_CTRL) interface to include a 16-bit field in the first argument for controlling which tags can be generated by the above instruction (an include rather than exclude mask). Note that by default all non-zero tags are excluded. This setting is per-thread.
---
Notes:
v2:
- Switch from an exclude mask to an include one for the prctl() interface.
- Reset the allowed tags mask during flush_thread().
arch/arm64/include/asm/processor.h | 1 + arch/arm64/include/asm/sysreg.h | 7 ++++++ arch/arm64/kernel/mte.c | 35 +++++++++++++++++++++++++++--- arch/arm64/kernel/process.c | 2 +- include/uapi/linux/prctl.h | 3 +++ 5 files changed, 44 insertions(+), 4 deletions(-)
arch/arm64/include/asm/processor.h +1 -0
@@ -153,6 +153,7 @@ struct thread_struct {#endif #ifdef CONFIG_ARM64_MTE u64 sctlr_tcf0; +u64 gcr_incl;#endif };
arch/arm64/include/asm/sysreg.h +7 -0
@@ -981,6 +981,13 @@write_sysreg (__scs_new, sysreg); \ } while (0) +#define sysreg_clear_set_s (sysreg, clear, set) do { \ + u64 __scs_val = read_sysreg_s(sysreg); \ + u64 __scs_new = (__scs_val & ~(u64)(clear)) | (set); \ + if (__scs_new != __scs_val) \ + write_sysreg_s(__scs_new, sysreg); \ +} while (0) +#endif #endif /* __ASM_SYSREG_H */
arch/arm64/kernel/mte.c +32 -3
@@ -71,6 +71,25 @@ static void set_sctlr_el1_tcf0(u64 tcf0)preempt_enable (); } +static void update_gcr_el1_excl (u64 incl) +{ +u64 excl = ~incl &SYS_GCR_EL1_EXCL_MASK ; + +/* +* Note that 'incl' is an include mask (controlled by the user via +* prctl()) while GCR_EL1 accepts an exclude mask. +* No need for ISB since this only affects EL0 currently, implicit +* with ERET. +*/ +sysreg_clear_set_s (SYS_GCR_EL1 ,SYS_GCR_EL1_EXCL_MASK , excl); +} + +static void set_gcr_el1_excl (u64 incl) +{ + current->thread.gcr_incl = incl; +update_gcr_el1_excl (incl); +} +void flush_mte_state (void ) {if (!system_supports_mte ())
@@ -82,6 +101,8 @@ void flush_mte_state(void)clear_thread_flag (TIF_MTE_ASYNC_FAULT );/* disable tag checking */ set_sctlr_el1_tcf0 (SCTLR_EL1_TCF0_NONE ); +/* reset tag generation mask */ +set_gcr_el1_excl (0 ); }void mte_thread_switch (struct task_struct *next)
@@ -92,6 +113,7 @@ void mte_thread_switch(struct task_struct *next)/* avoid expensive SCTLR_EL1 accesses if no change */ if (current->thread.sctlr_tcf0 != next->thread.sctlr_tcf0)update_sctlr_el1_tcf0 (next->thread.sctlr_tcf0); +update_gcr_el1_excl (next->thread.gcr_incl); }long set_mte_ctrl (unsigned long arg)
@@ -116,23 +138,30 @@ long set_mte_ctrl(unsigned long arg) }set_sctlr_el1_tcf0 (tcf0); +set_gcr_el1_excl ((arg &PR_MTE_TAG_MASK ) >>PR_MTE_TAG_SHIFT );return 0 ; }long get_mte_ctrl (void ) { +unsigned long ret; +if (!system_supports_mte ())return 0 ; + ret = current->thread.gcr_incl <<PR_MTE_TAG_SHIFT ; +switch (current->thread.sctlr_tcf0) {case SCTLR_EL1_TCF0_NONE :return PR_MTE_TCF_NONE ;case SCTLR_EL1_TCF0_SYNC : -return PR_MTE_TCF_SYNC ; + ret |=PR_MTE_TCF_SYNC ; +break ;case SCTLR_EL1_TCF0_ASYNC : -return PR_MTE_TCF_ASYNC ; + ret |=PR_MTE_TCF_ASYNC ; +break ; } -return 0 ; +return ret; }
arch/arm64/kernel/process.c +1 -1
@@ -601,7 +601,7 @@ long set_tagged_addr_ctrl(unsigned long arg)return -EINVAL ;if (system_supports_mte ()) - valid_mask |=PR_MTE_TCF_MASK ; + valid_mask |=PR_MTE_TCF_MASK |PR_MTE_TAG_MASK ;if (arg & ~valid_mask)return -EINVAL ;
include/uapi/linux/prctl.h +3 -0
@@ -239,6 +239,9 @@ struct prctl_mm_map {# define PR_MTE_TCF_SYNC (1UL << PR_MTE_TCF_SHIFT)# define PR_MTE_TCF_ASYNC (2UL << PR_MTE_TCF_SHIFT)# define PR_MTE_TCF_MASK (3UL << PR_MTE_TCF_SHIFT) +/* MTE tag inclusion mask */ +# define PR_MTE_TAG_SHIFT 3 +# define PR_MTE_TAG_MASK (0xffffUL << PR_MTE_TAG_SHIFT)/* Control reclaim behavior when allocating memory */ #define PR_SET_IO_FLUSHER 57
Catalin Marinas
maintainerPATCH v4 17/26 arm64: mte: Restore the GCR_EL1 register after a suspend
In reply to Catalin MarinasThe CPU resume/suspend routines only take care of the common system registers. Restore GCR_EL1 in addition via the __cpu_suspend_exit() function.
---
Notes:
New in v3.
arch/arm64/include/asm/mte.h | 4 ++++ arch/arm64/kernel/mte.c | 8 ++++++++ arch/arm64/kernel/suspend.c | 4 ++++ 3 files changed, 16 insertions(+)
arch/arm64/include/asm/mte.h +4 -0
@@ -22,6 +22,7 @@ void mte_sync_tags(pte_t *ptep, pte_t pte);void mte_copy_page_tags (void *kto,const void *kfrom);void flush_mte_state (void );void mte_thread_switch (struct task_struct *next); +void mte_suspend_exit (void );long set_mte_ctrl (unsigned long arg);long get_mte_ctrl (void );
@@ -42,6 +43,9 @@ static inline void flush_mte_state(void)static inline void mte_thread_switch (struct task_struct *next) { } +static inline void mte_suspend_exit (void ) +{ +}static inline long set_mte_ctrl (unsigned long arg) {return 0 ;
arch/arm64/kernel/mte.c +8 -0
@@ -116,6 +116,14 @@ void mte_thread_switch(struct task_struct *next)update_gcr_el1_excl (next->thread.gcr_incl); } +void mte_suspend_exit (void ) +{ +if (!system_supports_mte ()) +return ; + +update_gcr_el1_excl (current->thread.gcr_incl); +} +long set_mte_ctrl (unsigned long arg) {u64 tcf0;
arch/arm64/kernel/suspend.c +4 -0
@@ -9,6 +9,7 @@#include <asm/daifflags.h> #include <asm/debug-monitors.h> #include <asm/exec.h> +#include <asm/mte.h> #include <asm/pgtable.h> #include <asm/memory.h> #include <asm/mmu_context.h>
@@ -74,6 +75,9 @@ void notrace __cpu_suspend_exit(void)*/ if (arm64_get_ssbd_state () ==ARM64_SSBD_FORCE_DISABLE )arm64_set_ssbd_mitigation (false); + +/* Restore additional MTE-specific configuration */ +mte_suspend_exit (); }/*
Catalin Marinas
maintainerPATCH v4 18/26 arm64: mte: Add PTRACE_{PEEK,POKE}MTETAGS support
In reply to Catalin MarinasAdd support for bulk setting/getting of the MTE tags in a tracee's address space at 'addr' in the ptrace() syscall prototype. 'data' points to a struct iovec in the tracer's address space with iov_base representing the address of a tracer's buffer of length iov_len. The tags to be copied to/from the tracer's buffer are stored as one tag per byte.
On successfully copying at least one tag, ptrace() returns 0 and updates the tracer's iov_len with the number of tags copied. In case of error, either -EIO or -EFAULT is returned, trying to follow the ptrace() man page.
Note that the tag copying functions are not performance critical, therefore they lack optimisations found in typical memory copy routines.
---
Notes:
v4:
- Following the change to only clear the tags in a page if it is mapped to user with PROT_MTE, ptrace() now will refuse to access tags in pages not previously mapped with PROT_MTE (PG_mte_tagged set). This is primarily to avoid leaking uninitialised tags to user via ptrace().
- Fix SYM_FUNC_END argument typo.
- Rename MTE_ALLOC_* to MTE_GRANULE_*.
- Use uao_user_alternative for the user access in case we ever want to call mte_copy_tags_* with a kernel buffer. It also matches the other uaccess routines in the kernel.
- Simplify arch_ptrace() slightly.
- Reorder down_write_killable() with access_ok() in __access_remote_tags().
- Handle copy length 0 in mte_copy_tags_{to,from}_user().
- Use put_user() instead of __put_user().
New in v3.
arch/arm64/include/asm/mte.h | 17 ++++ arch/arm64/include/uapi/asm/ptrace.h | 3 + arch/arm64/kernel/mte.c | 139 +++++++++++++++++++++++++++ arch/arm64/kernel/ptrace.c | 7 ++ arch/arm64/lib/mte.S | 53 ++++++++++ 5 files changed, 219 insertions(+)
arch/arm64/include/asm/mte.h +17 -0
@@ -5,6 +5,11 @@#ifndef __ASM_MTE_H#define __ASM_MTE_H +#define MTE_GRANULE_SIZE UL(16) +#define MTE_GRANULE_MASK (~(MTE_GRANULE_SIZE - 1)) +#define MTE_TAG_SHIFT 56 +#define MTE_TAG_SIZE 4 +#ifndef __ASSEMBLY__#include <linux/page-flags.h>
@@ -12,6 +17,10 @@#include <asm/pgtable-types.h> void mte_clear_page_tags (void *addr,size_t size); +unsigned long mte_copy_tags_from_user (void *to,const void __user *from, +unsigned long n); +unsigned long mte_copy_tags_to_user (void __user *to,void *from, +unsigned long n);#ifdef CONFIG_ARM64_MTE
@@ -25,6 +34,8 @@ void mte_thread_switch(struct task_struct *next);void mte_suspend_exit (void );long set_mte_ctrl (unsigned long arg);long get_mte_ctrl (void ); +int mte_ptrace_copy_tags (struct task_struct *child,long request, +unsigned long addr,unsigned long data);#else
@@ -54,6 +65,12 @@ static inline long get_mte_ctrl(void) {return 0 ; } +static inline int mte_ptrace_copy_tags (struct task_struct *child, +long request,unsigned long addr, +unsigned long data) +{ +return -EIO ; +}#endif
arch/arm64/include/uapi/asm/ptrace.h +3 -0
@@ -67,6 +67,9 @@/* syscall emulation path in ptrace */ #define PTRACE_SYSEMU 31#define PTRACE_SYSEMU_SINGLESTEP 32 +/* MTE allocation tag access */ +#define PTRACE_PEEKMTETAGS 33 +#define PTRACE_POKEMTETAGS 34#ifndef __ASSEMBLY__
arch/arm64/kernel/mte.c +139 -0
156 lines not on this page · raw message
arch/arm64/kernel/ptrace.c +7 -0
@@ -34,6 +34,7 @@#include <asm/cpufeature.h> #include <asm/debug-monitors.h> #include <asm/fpsimd.h> +#include <asm/mte.h> #include <asm/pgtable.h> #include <asm/pointer_auth.h> #include <asm/stacktrace.h>
@@ -1797,6 +1798,12 @@ const struct user_regset_view *task_user_regset_view(struct task_struct *task)long arch_ptrace (struct task_struct *child,long request,unsigned long addr,unsigned long data) { +switch (request) { +case PTRACE_PEEKMTETAGS : +case PTRACE_POKEMTETAGS : +return mte_ptrace_copy_tags (child, request, addr, data); + } +return ptrace_request (child, request, addr, data); }
arch/arm64/lib/mte.S +53 -0
63 lines not on this page · raw message
Re: [PATCH v4 18/26] arm64: mte: Add PTRACE_{PEEK,POKE}MTETAGS support
Hi Catalin,
I have a question about siginfo MTE information. I suppose SEGV_MTESERR will be the most useful setting for debugging, right? Does si_addr contain the tagged pointer with the logical tag, a zero-tagged memory address or a tagged pointer with the allocation tag?
From the debugger user's perspective, one would want to see both the logical tag and the allocation tag. And it would be handy to have both available in siginfo. Does that make sense?
Also, when would we see SEGV_MTEAERR, for example? That would provide no additional information about a particular memory address, which is not that useful for the debugger.
Thanks,
Luis
Quoted 368 lines · Catalin Marinas
Add support for bulk setting/getting of the MTE tags in a tracee's address space at 'addr' in the
ptrace()syscall prototype. 'data' points to a struct iovec in the tracer's address space withiov_baserepresenting the address of a tracer's buffer of lengthiov_len. The tags to be copied to/from the tracer's buffer are stored as one tag per byte.On successfully copying at least one tag,
ptrace()returns 0 and updates the tracer'siov_lenwith the number of tags copied. In case of error, either-EIOor-EFAULTis returned, trying to follow theptrace()man page.Note that the tag copying functions are not performance critical, therefore they lack optimisations found in typical memory copy routines.
Signed-off-by Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>Cc Alan Hayward <[email protected]>Cc Luis Machado <[email protected]>Cc Omair Javaid <[email protected]>---
Notes:
v4:
- Following the change to only clear the tags in a page if it is mapped to user withPROT_MTE,ptrace()now will refuse to access tags in pages not previously mapped withPROT_MTE(PG_mte_taggedset). This is primarily to avoid leaking uninitialised tags to user viaptrace().
- FixSYM_FUNC_ENDargument typo.
- Rename MTE_ALLOC_* to MTE_GRANULE_*.
- Useuao_user_alternativefor the user access in case we ever want to call mte_copy_tags_* with a kernel buffer. It also matches the other uaccess routines in the kernel.
- Simplifyarch_ptrace()slightly.
- Reorderdown_write_killable()withaccess_ok()in__access_remote_tags().
- Handle copy length 0 in mte_copy_tags_{to,from}_user().
- Useput_user()instead of__put_user().New in v3.
arch/arm64/include/asm/mte.h | 17 ++++ arch/arm64/include/uapi/asm/ptrace.h | 3 + arch/arm64/kernel/mte.c | 139 +++++++++++++++++++++++++++ arch/arm64/kernel/ptrace.c | 7 ++ arch/arm64/lib/mte.S | 53 ++++++++++ 5 files changed, 219 insertions(+)arch/arm64/include/asm/mte.h +17
-0@@ -5,6 +5,11 @@#ifndef __ASM_MTE_H#define __ASM_MTE_H +#define MTE_GRANULE_SIZE UL(16) +#define MTE_GRANULE_MASK (~(MTE_GRANULE_SIZE - 1)) +#define MTE_TAG_SHIFT 56 +#define MTE_TAG_SIZE 4 +#ifndef __ASSEMBLY__#include <linux/page-flags.h> @@ -12,6 +17,10 @@#include <asm/pgtable-types.h> void mte_clear_page_tags (void *addr,size_t size); +unsigned long mte_copy_tags_from_user (void *to,const void __user *from, +unsigned long n); +unsigned long mte_copy_tags_to_user (void __user *to,void *from, +unsigned long n);#ifdef CONFIG_ARM64_MTE @@ -25,6 +34,8 @@ void mte_thread_switch(struct task_struct *next);void mte_suspend_exit (void );long set_mte_ctrl (unsigned long arg);long get_mte_ctrl (void ); +int mte_ptrace_copy_tags (struct task_struct *child,long request, +unsigned long addr,unsigned long data);#else @@ -54,6 +65,12 @@ static inline long get_mte_ctrl(void) {return 0 ; } +static inline int mte_ptrace_copy_tags (struct task_struct *child, +long request,unsigned long addr, +unsigned long data) +{ +return -EIO ; +}#endif arch/arm64/include/uapi/asm/ptrace.h +3
-0@@ -67,6 +67,9 @@/* syscall emulation path in ptrace */ #define PTRACE_SYSEMU 31#define PTRACE_SYSEMU_SINGLESTEP 32 +/* MTE allocation tag access */ +#define PTRACE_PEEKMTETAGS 33 +#define PTRACE_POKEMTETAGS 34#ifndef __ASSEMBLY__arch/arm64/kernel/mte.c +139
-0@@ -4,14 +4,18 @@*/ #include <linux/bitops.h> +#include <linux/kernel.h> #include <linux/mm.h> #include <linux/prctl.h> #include <linux/sched.h> +#include <linux/sched/mm.h> #include <linux/string.h> #include <linux/thread_info.h> +#include <linux/uio.h> #include <asm/cpufeature.h> #include <asm/mte.h> +#include <asm/ptrace.h> #include <asm/sysreg.h> void mte_sync_tags (pte_t *ptep,pte_t pte)@@ -173,3 +177,138 @@ long get_mte_ctrl(void)return ret; } + +/* +* Access MTE tags in another process' address space as given in mm. Update +* the number of tags copied. Return 0 if any tags copied, error otherwise. +* Inspired by __access_remote_vm(). +*/ +static int __access_remote_tags (struct task_struct *tsk,struct mm_struct *mm, +unsigned long addr,struct iovec *kiov, +unsigned int gup_flags) +{ +struct vm_area_struct *vma; +void __user *buf = kiov->iov_base; +size_t len = kiov->iov_len; +int ret; +int write = gup_flags &FOLL_WRITE ; + +if (!access_ok (buf, len)) +return -EFAULT ; + +if (down_read_killable (&mm->mmap_sem)) +return -EIO ; + +while (len) { +unsigned long tags, offset; +void *maddr; +struct page *page =NULL ; + + ret =get_user_pages_remote (tsk, mm, addr,1 , gup_flags, + &page, &vma,NULL ); +if (ret <=0 ) +break ; + +/* +* Only copy tags if the page has been mapped as PROT_MTE +* (PG_mte_tagged set). Otherwise the tags are not valid and +* not accessible to user. Moreover, an mprotect(PROT_MTE) +* would cause the existing tags to be cleared if the page +* was never mapped with PROT_MTE. +*/ +if (!test_bit (PG_mte_tagged, &page->flags)) { + ret = -EOPNOTSUPP ; +put_page (page); +break ; + } + +/* limit access to the end of the page */ + offset =offset_in_page (addr); + tags =min (len, (PAGE_SIZE - offset) /MTE_GRANULE_SIZE ); + + maddr =page_address (page); +if (write) { + tags =mte_copy_tags_from_user (maddr + offset, buf, tags); +set_page_dirty_lock (page); + }else { + tags =mte_copy_tags_to_user (buf, maddr + offset, tags); + } +put_page (page); + +/* error accessing the tracer's buffer */ +if (!tags) +break ; + + len -= tags; + buf += tags; + addr += tags *MTE_GRANULE_SIZE ; + } +up_read (&mm->mmap_sem); + +/* return an error if no tags copied */ + kiov->iov_len = buf - kiov->iov_base; +if (!kiov->iov_len) { +/* check for error accessing the tracee's address space */ +if (ret <=0 ) +return -EIO ; +else +return -EFAULT ; + } + +return 0 ; +} + +/* +* Copy MTE tags in another process' address space at 'addr' to/from tracer's +* iovec buffer. Return 0 on success. Inspired by ptrace_access_vm(). +*/ +static int access_remote_tags (struct task_struct *tsk,unsigned long addr, +struct iovec *kiov,unsigned int gup_flags) +{ +struct mm_struct *mm; +int ret; + + mm =get_task_mm (tsk); +if (!mm) +return -EPERM ; + +if (!tsk->ptrace || (current != tsk->parent) || + ((get_dumpable (mm) !=SUID_DUMP_USER ) && + !ptracer_capable (tsk, mm->user_ns))) { +mmput (mm); +return -EPERM ; + } + + ret =__access_remote_tags (tsk, mm, addr, kiov, gup_flags); +mmput (mm); + +return ret; +} + +int mte_ptrace_copy_tags (struct task_struct *child,long request, +unsigned long addr,unsigned long data) +{ +int ret; +struct iovec kiov; +struct iovec __user *uiov = (void __user *)data; +unsigned int gup_flags =FOLL_FORCE ; + +if (!system_supports_mte ()) +return -EIO ; + +if (get_user (kiov.iov_base, &uiov->iov_base) || +get_user (kiov.iov_len, &uiov->iov_len)) +return -EFAULT ; + +if (request ==PTRACE_POKEMTETAGS ) + gup_flags |=FOLL_WRITE ; + +/* align addr to the MTE tag granule */ + addr &=MTE_GRANULE_MASK ; + + ret =access_remote_tags (child, addr, &kiov, gup_flags); +if (!ret) + ret =put_user (kiov.iov_len, &uiov->iov_len); + +return ret; +}arch/arm64/kernel/ptrace.c +7
-0@@ -34,6 +34,7 @@#include <asm/cpufeature.h> #include <asm/debug-monitors.h> #include <asm/fpsimd.h> +#include <asm/mte.h> #include <asm/pgtable.h> #include <asm/pointer_auth.h> #include <asm/stacktrace.h> @@ -1797,6 +1798,12 @@ const struct user_regset_view *task_user_regset_view(struct task_struct *task)long arch_ptrace (struct task_struct *child,long request,unsigned long addr,unsigned long data) { +switch (request) { +case PTRACE_PEEKMTETAGS : +case PTRACE_POKEMTETAGS : +return mte_ptrace_copy_tags (child, request, addr, data); + } +return ptrace_request (child, request, addr, data); }arch/arm64/lib/mte.S +53
-0@@ -4,7 +4,9 @@ */ #include <linux/linkage.h> +#include <asm/alternative.h> #include <asm/assembler.h> +#include <asm/mte.h> #include <asm/page.h> .arch armv8.5-a+memtag@@ -40,3 +42,54 @@ SYM_FUNC_START(mte_copy_page_tags) b.ne 1b 2: SYM_FUNC_END(mte_copy_page_tags) + +/* + * Read tags from a user buffer (one tag per byte) and set the corresponding + * tags at the given kernel address. Used by PTRACE_POKEMTETAGS. + * x0 - kernel address (to) + * x1 - user buffer (from) + * x2 - number of tags/bytes (n) + * Returns: + * x0 - number of tags read/set + */ +SYM_FUNC_START(mte_copy_tags_from_user) + mov x3, x1 + cbz x2, 2f +1: + uao_user_alternative 2f, ldrb, ldtrb, w4, x1, 0 + lsl x4, x4, #MTE_TAG_SHIFT + stg x4, [x0], #MTE_GRANULE_SIZE + add x1, x1, #1 + subs x2, x2, #1 + b.ne 1b + + // exception handling and function return +2: sub x0, x1, x3 // update the number of tags set + ret +SYM_FUNC_END(mte_copy_tags_from_user) + +/* + * Get the tags from a kernel address range and write the tag values to the + * given user buffer (one tag per byte). Used by PTRACE_PEEKMTETAGS. + * x0 - user buffer (to) + * x1 - kernel address (from) + * x2 - number of tags/bytes (n) + * Returns: + * x0 - number of tags read/set + */ +SYM_FUNC_START(mte_copy_tags_to_user) + mov x3, x0 + cbz x2, 2f +1: + ldg x4, [x1] + ubfx x4, x4, #MTE_TAG_SHIFT, #MTE_TAG_SIZE + uao_user_alternative 2f, strb, sttrb, w4, x0, 0 + add x0, x0, #1 + add x1, x1, #MTE_GRANULE_SIZE + subs x2, x2, #1 + b.ne 1b + + // exception handling and function return +2: sub x0, x0, x3 // update the number of tags copied + ret +SYM_FUNC_END(mte_copy_tags_to_user)
Catalin Marinas
maintainerRe: [PATCH v4 18/26] arm64: mte: Add PTRACE_{PEEK,POKE}MTETAGS support
Quoted 4 lines · Luis Machado
I have a question about siginfo MTE information. I suppose
SEGV_MTESERRwill be the most useful setting for debugging, right? Doessi_addrcontain the tagged pointer with the logical tag, a zero-tagged memory address or a tagged pointer with the allocation tag?
The si_addr is zero-tagged currently. We were planning to expose the tag in FAR_EL1 as a separate siginfo field. See these discussions:
https://lore.kernel.org/linux-arm-kernel/[email protected]/
https://lore.kernel.org/linux-arm-kernel/[email protected]/
In theory, we could add the tag to si_addr for SEGV_MTESERR, it shouldn't break the existing ABI (well, it depends on how you look at it).
Quoted 3 lines
From the debugger user's perspective, one would want to see both the logical tag and the allocation tag. And it would be handy to have both available in siginfo. Does that make sense?
The debugger can access the allocation tag via PTRACE_PEEKMTETAGS. I don't think the kernel should provide this in siginfo. Also, the signal handler can do an LDG and read the allocation tag directly, no need for it to be in siginfo.
Quoted 3 lines
Also, when would we see
SEGV_MTEAERR, for example? That would provide no additional information about a particular memory address, which is not that useful for the debugger.
Yeah, we can't really do much here since the hardware doesn't provide us such information. The async mode is only useful as a general test to see if your program has MTE faults but for actual debugging you'd have to switch to synchronous. For glibc at least, I think the mode can be driven by an environment variable.
Signature · 1 line
Catalin
Re: [PATCH v4 18/26] arm64: mte: Add PTRACE_{PEEK,POKE}MTETAGS support
Quoted 9 lines · Catalin Marinas
Quoted 4 lines · Luis Machado
I have a question about siginfo MTE information. I suppose
SEGV_MTESERRwill be the most useful setting for debugging, right? Doessi_addrcontain the tagged pointer with the logical tag, a zero-tagged memory address or a tagged pointer with the allocation tag?The
si_addris zero-tagged currently. We were planning to expose the tag inFAR_EL1as a separate siginfo field. See these discussions:Quoted 1 line
https://lore.kernel.org/linux-arm-kernel/[email protected]/
Quoted 6 lines
https://lore.kernel.org/linux-arm-kernel/[email protected]/
In theory, we could add the tag to
si_addrforSEGV_MTESERR, it shouldn't break the existing ABI (well, it depends on how you look at it).
Having additional fields in siginfo that hold useful information is probably best for debuggers. See my comment below about Intel MPX.
Quoted 9 lines
Quoted 3 lines
From the debugger user's perspective, one would want to see both the logical
tag and the allocation tag. And it would be handy to have both available in siginfo. Does that make sense?The debugger can access the allocation tag via
PTRACE_PEEKMTETAGS. I don't think the kernel should provide this in siginfo. Also, the signal handler can do an LDG and read the allocation tag directly, no need for it to be in siginfo.
While the debugger can request this information from the kernel, the debugger has already received a SIGSEGV signal and will have to fetch siginfo for si_code. Having to do another PTRACE_PEEKMTETAGS call just to fetch the allocation tag doesn't sound great. Remember this can travel through TCP to gdbserver so it can call ptrace from the remote's end. It would be best to avoid the round trip.
Also, there seems to be past precedent to include more information in siginfo. For example, Intel MPX includes upper/lower bounds violation data in there.
Regarding using LDG, are you suggesting force-running this particular instruction in the traced process? If so, that isn't the way GDB (in particular, not sure about LLDB) does things.
Quoted 10 lines
Quoted 3 lines
Also, when would we see
SEGV_MTEAERR, for example? That would provide no additional information about a particular memory address, which is not that useful for the debugger.Yeah, we can't really do much here since the hardware doesn't provide us such information. The async mode is only useful as a general test to see if your program has MTE faults but for actual debugging you'd have to switch to synchronous. For glibc at least, I think the mode can be driven by an environment variable.
I suspect SEGV_MTESERR would be a reasonable default then, for whoever is responsible for setting the default settings.
I'm assuming it is not the debugger, as it doesn't know how to toggle prctl settings.
Catalin Marinas
maintainerRe: [PATCH v4 18/26] arm64: mte: Add PTRACE_{PEEK,POKE}MTETAGS support
Quoted 29 lines · Luis Machado
Quoted 9 lines · Catalin Marinas
Quoted 4 lines · Luis Machado
I have a question about siginfo MTE information. I suppose
SEGV_MTESERRwill be the most useful setting for debugging, right? Doessi_addrcontain the tagged pointer with the logical tag, a zero-tagged memory address or a tagged pointer with the allocation tag?The
si_addris zero-tagged currently. We were planning to expose the tag inFAR_EL1as a separate siginfo field. See these discussions:Quoted 6 lines
https://lore.kernel.org/linux-arm-kernel/[email protected]/
https://lore.kernel.org/linux-arm-kernel/[email protected]/In theory, we could add the tag to
si_addrforSEGV_MTESERR, it shouldn't break the existing ABI (well, it depends on how you look at it).Having additional fields in siginfo that hold useful information is probably best for debuggers. See my comment below about Intel MPX.
Quoted 8 lines
Quoted 3 lines
From the debugger user's perspective, one would want to see both the logical
tag and the allocation tag. And it would be handy to have both available in siginfo. Does that make sense?The debugger can access the allocation tag via
PTRACE_PEEKMTETAGS. I don't think the kernel should provide this in siginfo. Also, the signal handler can do an LDG and read the allocation tag directly, no need for it to be in siginfo.
Quoted 6 lines
While the debugger can request this information from the kernel, the debugger has already received a SIGSEGV signal and will have to fetch siginfo for
si_code. Having to do anotherPTRACE_PEEKMTETAGScall just to fetch the allocation tag doesn't sound great. Remember this can travel through TCP to gdbserver so it can call ptrace from the remote's end. It would be best to avoid the round trip.
But given that this is supposed to be a rare event, does another round trip to read some memory matter much?
Quoted 3 lines
Also, there seems to be past precedent to include more information in siginfo. For example, Intel MPX includes upper/lower bounds violation data in there.
There is a possible race here with getting the allocation tag. For example, another thread of the same process unmaps the memory and reading the tag from the user memory is no longer possible. If we are to add such information, we can't guarantee it's always possible (it may differ from the MPX case).
In general, I don't like providing information that's already accessible by other means (unless there's a performance issue but I don't think that's the case here).
Quoted 3 lines
Regarding using LDG, are you suggesting force-running this particular instruction in the traced process? If so, that isn't the way GDB (in particular, not sure about LLDB) does things.
No. What I meant is that if the signal handler itself needs the information, it can execute an LDG. For gdb, the equivalent currently is PTRACE_PEEKMTETAGS.
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines
Also, when would we see
SEGV_MTEAERR, for example? That would provide no additional information about a particular memory address, which is not that useful for the debugger.Yeah, we can't really do much here since the hardware doesn't provide us such information. The async mode is only useful as a general test to see if your program has MTE faults but for actual debugging you'd have to switch to synchronous. For glibc at least, I think the mode can be driven by an environment variable.
Quoted 5 lines
I suspect
SEGV_MTESERRwould be a reasonable default then, for whoever is responsible for setting the default settings.I'm assuming it is not the debugger, as it doesn't know how to toggle prctl settings.
The debugger could set the environment before starting the debugged process. But yes, that would be the C library.
Signature · 6 lines
Catalin _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Catalin Marinas
maintainerPATCH v4 19/26 fs: Handle intra-page faults in copy_mount_options()
In reply to Catalin MarinasThe copy_mount_options() function takes a user pointer argument but no size. It tries to read up to a PAGE_SIZE. However, copy_from_user() is not guaranteed to return all the accessible bytes if, for example, the access crosses a page boundary and gets a fault on the second page. To work around this, the current copy_mount_options() implementation performs two copy_from_user() passes, first to the end of the current page and the second to what's left in the subsequent page.
On arm64 with MTE enabled, access to a user page may trigger a fault after part of the buffer has been copied (when the user pointer tag, bits 56-59, no longer matches the allocation tag stored in memory). Allow copy_mount_options() to handle such intra-page faults by returning -EFAULT only if the first copy_from_user() has not copied any bytes.
---
Notes:
v4:
- Rewrite to avoid arch_has_exact_copy_from_user()
New in v3.
fs/namespace.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
fs/namespace.c +21 -3
@@ -3016,7 +3016,7 @@ static void shrink_submounts(struct mount *mnt)void *copy_mount_options (const void __user * data) {char *copy; -unsigned size; +unsigned size, left;if (!data)return NULL ;
@@ -3027,12 +3027,30 @@ void *copy_mount_options(const void __user * data) size =PAGE_SIZE -offset_in_page (data); -if (copy_from_user (copy, data, size)) { +/* +* Attempt to copy to the end of the first user page. On success, +* left == 0, copy the rest from the second user page (if it is +* accessible). copy_from_user() will zero the part of the kernel +* buffer not copied into. +* +* On architectures with intra-page faults (arm64 with MTE), the read +* from the first page may fail after copying part of the user data +* (left > 0 && left < size). Do not attempt the second copy in this +* case as the end of the valid user buffer has already been reached. +* Ensure, however, that the second part of the kernel buffer is +* zeroed. +*/ + left =copy_from_user (copy, data, size); +if (left == size) {kfree (copy);return ERR_PTR (-EFAULT ); }if (size !=PAGE_SIZE ) { -if (copy_from_user (copy + size, data + size,PAGE_SIZE - size)) +if (left ==0 ) +/* return not relevant, just silence the compiler */ + left =copy_from_user (copy + size, data + size, +PAGE_SIZE - size); +else memset (copy + size,0 ,PAGE_SIZE - size); }return copy;
Catalin Marinas
maintainerPATCH v4 20/26 mm: Add arch hooks for saving/restoring tags
In reply to Catalin MarinasFrom: Steven Price <[email protected]>
Arm's Memory Tagging Extension (MTE) adds some metadata (tags) to every physical page, when swapping pages out to disk it is necessary to save these tags, and later restore them when reading the pages back.
Add some hooks along with dummy implementations to enable the arch code to handle this.
Three new hooks are added to the swap code:
* arch_prepare_to_swap() and
* arch_swap_invalidate_page() / arch_swap_invalidate_area().
One new hook is added to shmem:
* arch_swap_restore_tags()
--- include/asm-generic/pgtable.h | 23 +++++++++++++++++++++++ mm/page_io.c | 10 ++++++++++ mm/shmem.c | 6 ++++++ mm/swapfile.c | 2 ++ 4 files changed, 41 insertions(+)
include/asm-generic/pgtable.h +23 -0
@@ -475,6 +475,29 @@ static inline int arch_unmap_one(struct mm_struct *mm, }#endif +#ifndef __HAVE_ARCH_PREPARE_TO_SWAP +static inline int arch_prepare_to_swap (struct page *page) +{ +return 0 ; +} +#endif + +#ifndef __HAVE_ARCH_SWAP_INVALIDATE +static inline void arch_swap_invalidate_page (int type,pgoff_t offset) +{ +} + +static inline void arch_swap_invalidate_area (int type) +{ +} +#endif + +#ifndef __HAVE_ARCH_SWAP_RESTORE_TAGS +static inline void arch_swap_restore_tags (swp_entry_t entry,struct page *page) +{ +} +#endif +#ifndef __HAVE_ARCH_PGD_OFFSET_GATE#define pgd_offset_gate (mm, addr) pgd_offset(mm, addr)#endif
mm/page_io.c +10 -0
@@ -253,6 +253,16 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)unlock_page (page); goto out; } +/* +* Arch code may have to preserve more data than just the page +* contents, e.g. memory tags. +*/ + ret =arch_prepare_to_swap (page); +if (ret) { +set_page_dirty (page); +unlock_page (page); + goto out; + }if (frontswap_store (page) ==0 ) {set_page_writeback (page);unlock_page (page);
mm/shmem.c +6 -0
@@ -1658,6 +1658,12 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index, }wait_on_page_writeback (page); +/* +* Some architectures may have to restore extra metadata to the +* physical page after reading from swap. +*/ +arch_swap_restore_tags (swap, page); +if (shmem_should_replace_page (page, gfp)) { error =shmem_replace_page (&page, gfp, info, index);if (error)
mm/swapfile.c +2 -0
@@ -722,6 +722,7 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,else swap_slot_free_notify =NULL ;while (offset <= end) { +arch_swap_invalidate_page (si->type, offset);frontswap_invalidate_page (si->type, offset);if (swap_slot_free_notify)swap_slot_free_notify (si->bdev, offset);
@@ -2645,6 +2646,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) frontswap_map =frontswap_map_get (p);spin_unlock (&p->lock);spin_unlock (&swap_lock); +arch_swap_invalidate_area (p->type);frontswap_invalidate_area (p->type);frontswap_map_set (p,NULL );mutex_unlock (&swapon_mutex);
Catalin Marinas
maintainerPATCH v4 21/26 arm64: mte: Enable swap of tagged pages
In reply to Catalin MarinasFrom: Steven Price <[email protected]>
When swapping pages out to disk it is necessary to save any tags that have been set, and restore when swapping back in. Make use of the new page flag (PG_ARCH_2, locally named PG_mte_tagged) to identify pages with tags. When swapping out these pages the tags are stored in memory and later restored when the pages are brought back in. Because shmem can swap pages back in without restoring the userspace PTE it is also necessary to add a hook for shmem.
--- arch/arm64/include/asm/mte.h | 8 ++++ arch/arm64/include/asm/pgtable.h | 32 +++++++++++++ arch/arm64/kernel/mte.c | 10 ++++ arch/arm64/lib/mte.S | 45 ++++++++++++++++++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/mteswap.c | 82 ++++++++++++++++++++++++++++++++ include/asm-generic/pgtable.h | 4 +- mm/shmem.c | 2 +- 8 files changed, 181 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/mm/mteswap.c
arch/arm64/include/asm/mte.h +8 -0
@@ -21,6 +21,14 @@ unsigned long mte_copy_tags_from_user(void *to, const void __user *from, unsigned long n);unsigned long mte_copy_tags_to_user (void __user *to,void *from,unsigned long n); +int mte_save_tags (struct page *page); +void mte_save_page_tags (const void *page_addr,void *tag_storage); +bool mte_restore_tags (swp_entry_t entry,struct page *page); +void mte_restore_page_tags (void *page_addr,const void *tag_storage); +void mte_invalidate_tags (int type,pgoff_t offset); +void mte_invalidate_tags_area (int type); +void *mte_allocate_tag_storage (void ); +void mte_free_tag_storage (char *storage);#ifdef CONFIG_ARM64_MTE
arch/arm64/include/asm/pgtable.h +32 -0
@@ -851,6 +851,38 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,extern int kern_addr_valid (unsigned long addr); +#ifdef CONFIG_ARM64_MTE + +#define __HAVE_ARCH_PREPARE_TO_SWAP +static inline int arch_prepare_to_swap (struct page *page) +{ +if (system_supports_mte ()) +return mte_save_tags (page); +return 0 ; +} + +#define __HAVE_ARCH_SWAP_INVALIDATE +static inline void arch_swap_invalidate_page (int type,pgoff_t offset) +{ +if (system_supports_mte ()) +mte_invalidate_tags (type, offset); +} + +static inline void arch_swap_invalidate_area (int type) +{ +if (system_supports_mte ()) +mte_invalidate_tags_area (type); +} + +#define __HAVE_ARCH_SWAP_RESTORE +static inline void arch_swap_restore (swp_entry_t entry,struct page *page) +{ +if (system_supports_mte () &&mte_restore_tags (entry, page)) +set_bit (PG_mte_tagged, &page->flags); +} + +#endif /* CONFIG_ARM64_MTE */ +#include <asm-generic/pgtable.h> /*
arch/arm64/kernel/mte.c +10 -0
@@ -10,6 +10,8 @@#include <linux/sched.h> #include <linux/sched/mm.h> #include <linux/string.h> +#include <linux/swap.h> +#include <linux/swapops.h> #include <linux/thread_info.h> #include <linux/uio.h>
@@ -21,11 +23,19 @@void mte_sync_tags (pte_t *ptep,pte_t pte) {struct page *page =pte_page (pte); +pte_t old_pte =READ_ONCE (*ptep);/* if PG_mte_tagged is set, tags have already been initialised */ if (test_and_set_bit (PG_mte_tagged, &page->flags))return ; +if (is_swap_pte (old_pte)) { +swp_entry_t entry =pte_to_swp_entry (old_pte); + +if (!non_swap_entry (entry) &&mte_restore_tags (entry, page)) +return ; + } +/* tags of the zero page will be cleared on first mapping */ mte_clear_page_tags (page_address (page),page_size (page)); }
arch/arm64/lib/mte.S +45 -0
48 lines not on this page · raw message
arch/arm64/mm/Makefile +1 -0
@@ -8,6 +8,7 @@ obj-$(CONFIG_PTDUMP_CORE) += dump.o obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o obj-$(CONFIG_NUMA) += numa.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o +obj-$(CONFIG_ARM64_MTE) += mteswap.o KASAN_SANITIZE_physaddr.o += n obj-$(CONFIG_KASAN) += kasan_init.o
arch/arm64/mm/mteswap.c new file +82 -0
82 lines not on this page · raw message
include/asm-generic/pgtable.h +2 -2
@@ -492,8 +492,8 @@ static inline void arch_swap_invalidate_area(int type) }#endif -#ifndef __HAVE_ARCH_SWAP_RESTORE_TAGS -static inline void arch_swap_restore_tags (swp_entry_t entry,struct page *page) +#ifndef __HAVE_ARCH_SWAP_RESTORE +static inline void arch_swap_restore (swp_entry_t entry,struct page *page) { }#endif
mm/shmem.c +1 -1
@@ -1662,7 +1662,7 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index,* Some architectures may have to restore extra metadata to the * physical page after reading from swap. */ -arch_swap_restore_tags (swap, page); +arch_swap_restore (swap, page);if (shmem_should_replace_page (page, gfp)) { error =shmem_replace_page (&page, gfp, info, index);
Catalin Marinas
maintainerPATCH v4 22/26 arm64: mte: Save tags when hibernating
In reply to Catalin MarinasFrom: Steven Price <[email protected]>
When hibernating the contents of all pages in the system are written to disk, however the MTE tags are not visible to the generic hibernation code. So just before the hibernation image is created copy the tags out of the physical tag storage into standard memory so they will be included in the hibernation image. After hibernation apply the tags back into the physical tag storage.
--- arch/arm64/kernel/hibernate.c | 118 ++++++++++++++++++++++++++++++++++ 1 file changed, 118 insertions(+)
arch/arm64/kernel/hibernate.c +118 -0
@@ -31,6 +31,7 @@#include <asm/kexec.h> #include <asm/memory.h> #include <asm/mmu_context.h> +#include <asm/mte.h> #include <asm/pgalloc.h> #include <asm/pgtable.h> #include <asm/pgtable-hwdef.h>
@@ -277,6 +278,117 @@ static int create_safe_exec_page(void *src_start, size_t length,#define dcache_clean_range (start, end) __flush_dcache_area(start, (end - start)) +#ifdef CONFIG_ARM64_MTE + +static DEFINE_XARRAY (mte_pages); + +static int save_tags (struct page *page,unsigned long pfn) +{ +void *tag_storage, *ret; + + tag_storage =mte_allocate_tag_storage (); +if (!tag_storage) +return -ENOMEM ; + +mte_save_page_tags (page_address (page), tag_storage); + + ret =xa_store (&mte_pages, pfn, tag_storage,GFP_KERNEL ); +if (WARN (xa_is_err (ret),"Failed to store MTE tags" )) { +mte_free_tag_storage (tag_storage); +return xa_err (ret); + }else if (WARN (ret,"swsusp: %s: Duplicate entry" , __func__)) { +mte_free_tag_storage (ret); + } + +return 0 ; +} + +static void swsusp_mte_free_storage (void ) +{ +XA_STATE (xa_state, &mte_pages,0 ); +void *tags; + +xa_lock (&mte_pages); +xas_for_each (&xa_state, tags,ULONG_MAX ) { +mte_free_tag_storage (tags); + } +xa_unlock (&mte_pages); + +xa_destroy (&mte_pages); +} + +static int swsusp_mte_save_tags (void ) +{ +struct zone *zone; +unsigned long pfn, max_zone_pfn; +int ret =0 ; +int n =0 ; + +if (!system_supports_mte ()) +return 0 ; + +for_each_populated_zone (zone) { + max_zone_pfn =zone_end_pfn (zone); +for (pfn = zone->zone_start_pfn; pfn < max_zone_pfn; pfn++) { +struct page *page =pfn_to_online_page (pfn); + +if (!page) +continue ; + +if (!test_bit (PG_mte_tagged, &page->flags)) +continue ; + + ret =save_tags (page, pfn); +if (ret) { +swsusp_mte_free_storage (); + goto out; + } + + n++; + } + } +pr_info ("Saved %d MTE pages\n" , n); + +out: +return ret; +} + +static void swsusp_mte_restore_tags (void ) +{ +XA_STATE (xa_state, &mte_pages,0 ); +int n =0 ; +void *tags; + +xa_lock (&mte_pages); +xas_for_each (&xa_state, tags,ULONG_MAX ) { +unsigned long pfn = xa_state.xa_index; +struct page *page =pfn_to_online_page (pfn); + +mte_restore_page_tags (page_address (page), tags); + +mte_free_tag_storage (tags); + n++; + } +xa_unlock (&mte_pages); + +pr_info ("Restored %d MTE pages\n" , n); + +xa_destroy (&mte_pages); +} + +#else /* CONFIG_ARM64_MTE */ + +static int swsusp_mte_save_tags (void ) +{ +return 0 ; +} + +static void swsusp_mte_restore_tags (void ) +{ +} + +#endif /* CONFIG_ARM64_MTE */ +int swsusp_arch_suspend (void ) {int ret =0 ;
@@ -294,6 +406,10 @@ int swsusp_arch_suspend(void)/* make the crash dump kernel image visible/saveable */ crash_prepare_suspend (); + ret =swsusp_mte_save_tags (); +if (ret) +return ret; + sleep_cpu =smp_processor_id (); ret =swsusp_save (); }else {
@@ -307,6 +423,8 @@ int swsusp_arch_suspend(void)dcache_clean_range (__hyp_text_start, __hyp_text_end); } +swsusp_mte_restore_tags (); +/* make the crash dump kernel image protected again */ crash_post_resume ();
Catalin Marinas
maintainerPATCH v4 23/26 arm64: mte: Check the DT memory nodes for MTE support
In reply to Catalin MarinasEven if the ID_AA64PFR1_EL1 register advertises the presence of MTE, it is not guaranteed that the memory system on the SoC supports the feature. In the absence of system-wide MTE support, the behaviour is undefined and the kernel should not enable the MTE memory type in MAIR_EL1.
For FDT, add an 'arm,armv8.5-memtag' property to the /memory nodes and check for its presence during MTE probing. For example:
memory@80000000 {
device_type = "memory";
arm,armv8.5-memtag;
reg = <0x00000000 0x80000000 0 0x80000000>,
<0x00000008 0x80000000 0 0x80000000>;
};If the /memory nodes are not present in DT or if at least one node does not support MTE, the feature will be disabled. On EFI systems, it is assumed that the memory description matches the EFI memory map (if not, it is considered a firmware bug).
MTE is not currently supported on ACPI systems.
---
Notes:
Ongoing (internal) discussions on whether this is the right approach. The issue may need to be solved similarly for ACPI systems.
v4:
- Reworked the CPUID field handling to ensure that it is not exposed to user (via MRS emulation) when the feature is not described in the DT. Previously, only the HWCAP was hidden.
New in v3.
arch/arm64/boot/dts/arm/fvp-base-revc.dts | 1 + arch/arm64/include/asm/cpufeature.h | 6 ++- arch/arm64/kernel/cpufeature.c | 59 ++++++++++++++++++++--- 3 files changed, 58 insertions(+), 8 deletions(-)
arch/arm64/boot/dts/arm/fvp-base-revc.dts +1 -0
@@ -94,6 +94,7 @@ memory@80000000 { device_type = "memory"; + arm,armv8.5-memtag; reg = <0x00000000 0x80000000 0 0x80000000>, <0x00000008 0x80000000 0 0x80000000>; };
arch/arm64/include/asm/cpufeature.h +5 -1
@@ -61,6 +61,7 @@ struct arm64_ftr_bits {u8 shift;u8 width;s64 safe_val;/* safe value for FTR_EXACT features */ +s64 (*filter)(const struct arm64_ftr_bits *,s64 ); };/*
@@ -542,7 +543,10 @@ cpuid_feature_extract_field(u64 features, int field, bool sign)static inline s64 arm64_ftr_value (const struct arm64_ftr_bits *ftrp,u64 val) { -return (s64 )cpuid_feature_extract_field_width (val, ftrp->shift, ftrp->width, ftrp->sign); +s64 fval = (s64 )cpuid_feature_extract_field_width (val, ftrp->shift, ftrp->width, ftrp->sign); +if (ftrp->filter) + fval = ftrp->filter (ftrp, fval); +return fval; }static inline bool id_aa64mmfr0_mixed_endian_el0 (u64 mmfr0)
arch/arm64/kernel/cpufeature.c +52 -7
101 lines not on this page · raw message
Catalin Marinas
maintainerPATCH v4 24/26 arm64: mte: Introduce early param to disable MTE support
In reply to Catalin MarinasFor performance analysis it may be desirable to disable MTE altogether via an early param. Introduce arm64.mte_disable and, if true, filter out the sanitised ID_AA64PFR1_EL1.MTE field to avoid exposing the HWCAP to user.
---
Notes:
New in v4.
Documentation/admin-guide/kernel-parameters.txt | 4 ++++ arch/arm64/kernel/cpufeature.c | 11 +++++++++++ 2 files changed, 15 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +4 -0
@@ -373,6 +373,10 @@ arcrimi= [HW,NET] ARCnet - "RIM I" (entirely mem-mapped) cards Format: <io>,<irq>,<nodeID> + arm64.mte_disable= + [ARM64] Disable Linux support for the Memory + Tagging Extension (both user and in-kernel). + ataflop= [HW,M68k] atarimouse= [HW,MOUSE] Atari Mouse
arch/arm64/kernel/cpufeature.c +11 -0
@@ -126,12 +126,23 @@ static void cpu_enable_cnp(struct arm64_cpu_capabilities const *cap);static bool __system_matches_cap (unsigned int n);#ifdef CONFIG_ARM64_MTE +static bool mte_disable; + +static int __init arm64_mte_disable (char *buf) +{ +return strtobool (buf, &mte_disable); +} +early_param ("arm64.mte_disable" , arm64_mte_disable); +s64 mte_ftr_filter (const struct arm64_ftr_bits *ftrp,s64 val) {struct device_node *np;static bool memory_checked = false;static bool mte_capable = true; +if (mte_disable) +return ID_AA64PFR1_MTE_NI ; +/* EL0-only MTE is not supported by Linux, don't expose it */ if (val <ID_AA64PFR1_MTE )return ID_AA64PFR1_MTE_NI ;
Re: [PATCH v4 24/26] arm64: mte: Introduce early param to disable MTE support
Quoted 21 lines · Catalin Marinas
For performance analysis it may be desirable to disable MTE altogether via an early param. Introduce arm64.mte_disable and, if true, filter out the sanitised ID_AA64PFR1_EL1.MTE field to avoid exposing the HWCAP to user.
Signed-off-by Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>---
Notes:
New in v4.Documentation/admin-guide/kernel-parameters.txt | 4 ++++ arch/arm64/kernel/cpufeature.c | 11 +++++++++++ 2 files changed, 15 insertions(+)Documentation/admin-guide/kernel-parameters.txt +0
-0
Documentation/admin-guide/kernel-parameters.txt line 379
Quoted 7 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0@@ -373,6 +373,10 @@ arcrimi= [HW,NET] ARCnet - "RIM I" (entirely mem-mapped) cards Format: <io>,<irq>,<nodeID> + arm64.mte_disable= + [ARM64] Disable Linux support for the Memory + Tagging Extension (both user and in-kernel). +
Should it really to take parameter (on/off/true/false)? It may lead to expectation that arm64.mte_disable=false should enable MT and, yes, double negatives make it look ugly, so if we do need parameter, can it be arm64.mte=on/off/true/false?
Cheers
Vladimir
Will Deacon
maintainerRe: [PATCH v4 24/26] arm64: mte: Introduce early param to disable MTE support
Quoted 25 lines · Vladimir Murzin
Quoted 24 lines · Catalin Marinas
For performance analysis it may be desirable to disable MTE altogether via an early param. Introduce arm64.mte_disable and, if true, filter out the sanitised ID_AA64PFR1_EL1.MTE field to avoid exposing the HWCAP to user.
Signed-off-by Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>---
Notes:
New in v4.Documentation/admin-guide/kernel-parameters.txt | 4 ++++ arch/arm64/kernel/cpufeature.c | 11 +++++++++++ 2 files changed, 15 insertions(+)Documentation/admin-guide/kernel-parameters.txt +0
-0@@ -373,6 +373,10 @@ arcrimi= [HW,NET] ARCnet - "RIM I" (entirely mem-mapped) cards Format: <io>,<irq>,<nodeID>
Quoted 8 lines
Documentation/admin-guide/kernel-parameters.txt line 379
Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0+ arm64.mte_disable= + [ARM64] Disable Linux support for the Memory + Tagging Extension (both user and in-kernel). +Should it really to take parameter (on/off/true/false)? It may lead to expectation that arm64.mte_disable=false should enable MT and, yes, double negatives make it look ugly, so if we do need parameter, can it be arm64.mte=on/off/true/false?
I don't think "performance analysis" is a good justification for this parameter tbh. We don't tend to add these options for other architectural features, and I don't see why MTE is any different in this regard.
Will
Catalin Marinas
maintainerRe: [PATCH v4 24/26] arm64: mte: Introduce early param to disable MTE support
Quoted 35 lines · Will Deacon
Quoted 25 lines · Vladimir Murzin
Quoted 24 lines · Catalin Marinas
For performance analysis it may be desirable to disable MTE altogether via an early param. Introduce arm64.mte_disable and, if true, filter out the sanitised ID_AA64PFR1_EL1.MTE field to avoid exposing the HWCAP to user.
Signed-off-by Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>---
Notes:
New in v4.Documentation/admin-guide/kernel-parameters.txt | 4 ++++ arch/arm64/kernel/cpufeature.c | 11 +++++++++++ 2 files changed, 15 insertions(+)Documentation/admin-guide/kernel-parameters.txt +0
-0@@ -373,6 +373,10 @@ arcrimi= [HW,NET] ARCnet - "RIM I" (entirely mem-mapped) cards Format: <io>,<irq>,<nodeID>Quoted 8 lines
Documentation/admin-guide/kernel-parameters.txt line 379
Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0+ arm64.mte_disable= + [ARM64] Disable Linux support for the Memory + Tagging Extension (both user and in-kernel). +Should it really to take parameter (on/off/true/false)? It may lead to expectation that arm64.mte_disable=false should enable MT and, yes, double negatives make it look ugly, so if we do need parameter, can it be arm64.mte=on/off/true/false?
Quoted 3 lines
I don't think "performance analysis" is a good justification for this parameter tbh. We don't tend to add these options for other architectural features, and I don't see why MTE is any different in this regard.
There is an expectation of performance impact with MTE enabled, especially if it's running in synchronous mode. For the in-kernel MTE, we could add a parameter which sets sync vs async at boot time rather than a big disable knob. It won't affect user space however.
The other 'justification' is if your hardware has weird unexpected behaviour but I'd like this handled via errata workarounds.
I'll let the people who asked for this to chip in ;). I agree with you that we rarely add these (and I rejected a similar option a few weeks ago on the AMU patchset).
Signature · 1 line
Catalin
Re: [PATCH v4 24/26] arm64: mte: Introduce early param to disable MTE support
Quoted 45 lines · Catalin Marinas
Quoted 35 lines · Will Deacon
Quoted 25 lines · Vladimir Murzin
Quoted 24 lines · Catalin Marinas
For performance analysis it may be desirable to disable MTE altogether via an early param. Introduce arm64.mte_disable and, if true, filter out the sanitised ID_AA64PFR1_EL1.MTE field to avoid exposing the HWCAP to user.
Signed-off-by Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>---
Notes:
New in v4.Documentation/admin-guide/kernel-parameters.txt | 4 ++++ arch/arm64/kernel/cpufeature.c | 11 +++++++++++ 2 files changed, 15 insertions(+)Documentation/admin-guide/kernel-parameters.txt +0
-0@@ -373,6 +373,10 @@ arcrimi= [HW,NET] ARCnet - "RIM I" (entirely mem-mapped) cards Format: <io>,<irq>,<nodeID>Quoted 8 lines
Documentation/admin-guide/kernel-parameters.txt line 379
Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0+ arm64.mte_disable= + [ARM64] Disable Linux support for the Memory + Tagging Extension (both user and in-kernel). +Should it really to take parameter (on/off/true/false)? It may lead to expectation that arm64.mte_disable=false should enable MT and, yes, double negatives make it look ugly, so if we do need parameter, can it be arm64.mte=on/off/true/false?
Quoted 3 lines
I don't think "performance analysis" is a good justification for this parameter tbh. We don't tend to add these options for other architectural features, and I don't see why MTE is any different in this regard.
There is an expectation of performance impact with MTE enabled, especially if it's running in synchronous mode. For the in-kernel MTE, we could add a parameter which sets sync vs async at boot time rather than a big disable knob. It won't affect user space however.
Quoted 6 lines
The other 'justification' is if your hardware has weird unexpected behaviour but I'd like this handled via errata workarounds.
I'll let the people who asked for this to chip in ;). I agree with you that we rarely add these (and I rejected a similar option a few weeks ago on the AMU patchset).
We've been looking into other ways this on/off behavior could be achieved. The "arm,armv8.5-memtag" DT flag already provides what we want - meaning that this flag could be removed if the system did not support MTE.
I did see your remark on "arm64: mte: Check the DT memory nodes for MTE support" questioning whether it was the right approach - is this still the case? --Patrick
Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
Catalin Marinas
maintainerRe: [PATCH v4 24/26] arm64: mte: Introduce early param to disable MTE support
Hi Patrick,
Quoted 30 lines · Patrick Daly
Quoted 29 lines · Catalin Marinas
Quoted 19 lines · Will Deacon
Quoted 9 lines · Vladimir Murzin
Quoted 8 lines · Catalin Marinas
Documentation/admin-guide/kernel-parameters.txt +0
-0@@ -373,6 +373,10 @@ arcrimi= [HW,NET] ARCnet - "RIM I" (entirely mem-mapped) cards Format: <io>,<irq>,<nodeID>Quoted 8 lines
Documentation/admin-guide/kernel-parameters.txt line 379
Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0+ arm64.mte_disable= + [ARM64] Disable Linux support for the Memory + Tagging Extension (both user and in-kernel). +Should it really to take parameter (on/off/true/false)? It may lead to expectation that arm64.mte_disable=false should enable MT and, yes, double negatives make it look ugly, so if we do need parameter, can it be arm64.mte=on/off/true/false?
Quoted 3 lines
I don't think "performance analysis" is a good justification for this parameter tbh. We don't tend to add these options for other architectural features, and I don't see why MTE is any different in this regard.
There is an expectation of performance impact with MTE enabled, especially if it's running in synchronous mode. For the in-kernel MTE, we could add a parameter which sets sync vs async at boot time rather than a big disable knob. It won't affect user space however.
Quoted 8 lines
Quoted 6 lines
The other 'justification' is if your hardware has weird unexpected behaviour but I'd like this handled via errata workarounds.
I'll let the people who asked for this to chip in ;). I agree with you that we rarely add these (and I rejected a similar option a few weeks ago on the AMU patchset).
We've been looking into other ways this on/off behavior could be achieved.
The actual question here is what the on/off behaviour is needed for. We can figure out the best mechanism for this once we know what we want to achieve. My wild guess above was performance analysis but that can be toggled by either kernel boot parameter or run-time sysctl (or just the Kconfig option).
If it is about forcing user space not to use MTE, we may look into some other sysctl controls (we already have one for the tagged address ABI).
If it is for working around hardware not supporting MTE (i.e. no allocation tag storage), this should be handled differently, not by kernel parameter.
Quoted 5 lines
The "arm,armv8.5-memtag" DT flag already provides what we want - meaning that this flag could be removed if the system did not support MTE.
I did see your remark on "arm64: mte: Check the DT memory nodes for MTE support" questioning whether it was the right approach - is this still the case?
My plan is to remove the DT patch altogether _if_ I get confirmation from the CPU designers. The idea is that if ID_AA64PFR1_EL1.MTE > 1, Linux can assume system-wide MTE support. If an MTE-capable CPU is deployed in an SoC without tag storage, a tie-off should change the ID field to 1 (or 0). If we do find hardware with an ID field > 1 and no tag storage, it will be handled as an SoC erratum in the kernel, probably tied to the new SoC Id advertised by firmware (Sudeep had some patches recently).
Thanks.
Signature · 1 line
Catalin
Re: [PATCH v4 24/26] arm64: mte: Introduce early param to disable MTE support
Quoted 42 lines · Catalin Marinas
Hi Patrick,
Quoted 30 lines · Patrick Daly
Quoted 29 lines · Catalin Marinas
Quoted 19 lines · Will Deacon
Quoted 9 lines · Vladimir Murzin
Quoted 8 lines · Catalin Marinas
Documentation/admin-guide/kernel-parameters.txt +0
-0@@ -373,6 +373,10 @@ arcrimi= [HW,NET] ARCnet - "RIM I" (entirely mem-mapped) cards Format: <io>,<irq>,<nodeID>Quoted 8 lines
Documentation/admin-guide/kernel-parameters.txt line 379
Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0+ arm64.mte_disable= + [ARM64] Disable Linux support for the Memory + Tagging Extension (both user and in-kernel). +Should it really to take parameter (on/off/true/false)? It may lead to expectation that arm64.mte_disable=false should enable MT and, yes, double negatives make it look ugly, so if we do need parameter, can it be arm64.mte=on/off/true/false?
Quoted 3 lines
I don't think "performance analysis" is a good justification for this parameter tbh. We don't tend to add these options for other architectural features, and I don't see why MTE is any different in this regard.
There is an expectation of performance impact with MTE enabled, especially if it's running in synchronous mode. For the in-kernel MTE, we could add a parameter which sets sync vs async at boot time rather than a big disable knob. It won't affect user space however.
Quoted 8 lines
Quoted 6 lines
The other 'justification' is if your hardware has weird unexpected behaviour but I'd like this handled via errata workarounds.
I'll let the people who asked for this to chip in ;). I agree with you that we rarely add these (and I rejected a similar option a few weeks ago on the AMU patchset).
We've been looking into other ways this on/off behavior could be achieved.
Quoted 8 lines
The actual question here is what the on/off behaviour is needed for. We can figure out the best mechanism for this once we know what we want to achieve. My wild guess above was performance analysis but that can be toggled by either kernel boot parameter or run-time sysctl (or just the
Kconfigoption).If it is about forcing user space not to use MTE, we may look into some other sysctl controls (we already have one for the tagged address ABI).
We want to allow the end user to be able to easily "opt out" of MTE in favour of better power, perf and battery life.
In terms of deciding policy, a sysctl is much more accessible than reompiling with CONFIG_MTE=n, or replacing userspace libraries with equivalents which don't use PROT_MTE.
--Patrick
Quoted 19 lines
If it is for working around hardware not supporting MTE (i.e. no allocation tag storage), this should be handled differently, not by kernel parameter.
Quoted 5 lines
The "arm,armv8.5-memtag" DT flag already provides what we want - meaning that this flag could be removed if the system did not support MTE.
I did see your remark on "arm64: mte: Check the DT memory nodes for MTE support" questioning whether it was the right approach - is this still the case?
My plan is to remove the DT patch altogether _if_ I get confirmation from the CPU designers. The idea is that if ID_AA64PFR1_EL1.MTE > 1, Linux can assume system-wide MTE support. If an MTE-capable CPU is deployed in an SoC without tag storage, a tie-off should change the ID field to 1 (or 0). If we do find hardware with an ID field > 1 and no tag storage, it will be handled as an SoC erratum in the kernel, probably tied to the new SoC Id advertised by firmware (Sudeep had some patches recently).
Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
Will Deacon
maintainerRe: [PATCH v4 24/26] arm64: mte: Introduce early param to disable MTE support
Quoted 47 lines · Patrick Daly
Quoted 46 lines · Catalin Marinas
Quoted 30 lines · Patrick Daly
Quoted 29 lines · Catalin Marinas
Quoted 19 lines · Will Deacon
Quoted 9 lines · Vladimir Murzin
Quoted 8 lines · Catalin Marinas
Documentation/admin-guide/kernel-parameters.txt +0
-0@@ -373,6 +373,10 @@ arcrimi= [HW,NET] ARCnet - "RIM I" (entirely mem-mapped) cards Format: <io>,<irq>,<nodeID>Quoted 8 lines
Documentation/admin-guide/kernel-parameters.txt line 379
Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0+ arm64.mte_disable= + [ARM64] Disable Linux support for the Memory + Tagging Extension (both user and in-kernel). +Should it really to take parameter (on/off/true/false)? It may lead to expectation that arm64.mte_disable=false should enable MT and, yes, double negatives make it look ugly, so if we do need parameter, can it be arm64.mte=on/off/true/false?
Quoted 3 lines
I don't think "performance analysis" is a good justification for this parameter tbh. We don't tend to add these options for other architectural features, and I don't see why MTE is any different in this regard.
There is an expectation of performance impact with MTE enabled, especially if it's running in synchronous mode. For the in-kernel MTE, we could add a parameter which sets sync vs async at boot time rather than a big disable knob. It won't affect user space however.
Quoted 8 lines
Quoted 6 lines
The other 'justification' is if your hardware has weird unexpected behaviour but I'd like this handled via errata workarounds.
I'll let the people who asked for this to chip in ;). I agree with you that we rarely add these (and I rejected a similar option a few weeks ago on the AMU patchset).
We've been looking into other ways this on/off behavior could be achieved.
The actual question here is what the on/off behaviour is needed for. We can figure out the best mechanism for this once we know what we want to achieve. My wild guess above was performance analysis but that can be toggled by either kernel boot parameter or run-time sysctl (or just the
Kconfigoption).
Quoted 5 lines
Quoted 2 lines
If it is about forcing user space not to use MTE, we may look into some other sysctl controls (we already have one for the tagged address ABI).
We want to allow the end user to be able to easily "opt out" of MTE in favour of better power, perf and battery life.
Who is "the end user" in this case?
If MTE is bad enough for power, performance and battery life that we need a kill switch, then perhaps we shouldn't enable it by default and the few people that want to use it can build a kernel with it enabled. However, then I don't really see what MTE buys you over the existing KASAN implementations.
I thought the general idea was that you could run in the (cheap) "async" mode, and then re-run in the more expensive "sync" mode to further diagnose any failures. That model seems to work well with these patches, since reporting is disabled by default. Are you saying that there is a significant penalty incurred even when reporting is not enabled?
Anyway, we don't offer global runtime/cmdline switches for the vast majority of other architectural features -- instead, we choose a sensible default, and I think we should do the same here.
Will
Re: [PATCH v4 24/26] arm64: mte: Introduce early param to disable MTE support
Quoted 27 lines · Will Deacon
Quoted 7 lines · Patrick Daly
Quoted 6 lines · Catalin Marinas
The actual question here is what the on/off behaviour is needed for. We can figure out the best mechanism for this once we know what we want to achieve. My wild guess above was performance analysis but that can be toggled by either kernel boot parameter or run-time sysctl (or just the
Kconfigoption).Quoted 5 lines
Quoted 2 lines
If it is about forcing user space not to use MTE, we may look into some other sysctl controls (we already have one for the tagged address ABI).
We want to allow the end user to be able to easily "opt out" of MTE in favour of better power, perf and battery life.
Who is "the end user" in this case?
If MTE is bad enough for power, performance and battery life that we need a kill switch, then perhaps we shouldn't enable it by default and the few people that want to use it can build a kernel with it enabled. However, then I don't really see what MTE buys you over the existing KASAN implementations.
I thought the general idea was that you could run in the (cheap) "async" mode, and then re-run in the more expensive "sync" mode to further diagnose any failures. That model seems to work well with these patches, since reporting is disabled by default. Are you saying that there is a significant penalty incurred even when reporting is not enabled?
Quoted 3 lines
Anyway, we don't offer global runtime/cmdline switches for the vast majority of other architectural features -- instead, we choose a sensible default, and I think we should do the same here.
i would not expect mte overhead if userspace processes don't map anything with PROT_MTE and don't enable tag checking with prctl. (i.e. userspace runtimes can "opt out" of mte for better power, perf and battery)
is that not the case?
Catalin Marinas
maintainerRe: [PATCH v4 24/26] arm64: mte: Introduce early param to disable MTE support
In reply to Will DeaconQuoted 48 lines · Will Deacon
Quoted 47 lines · Patrick Daly
Quoted 46 lines · Catalin Marinas
Quoted 30 lines · Patrick Daly
Quoted 29 lines · Catalin Marinas
Quoted 19 lines · Will Deacon
Quoted 9 lines · Vladimir Murzin
Quoted 8 lines · Catalin Marinas
Documentation/admin-guide/kernel-parameters.txt +0
-0@@ -373,6 +373,10 @@ arcrimi= [HW,NET] ARCnet - "RIM I" (entirely mem-mapped) cards Format: <io>,<irq>,<nodeID>Quoted 8 lines
Documentation/admin-guide/kernel-parameters.txt line 379
Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0+ arm64.mte_disable= + [ARM64] Disable Linux support for the Memory + Tagging Extension (both user and in-kernel). +Should it really to take parameter (on/off/true/false)? It may lead to expectation that arm64.mte_disable=false should enable MT and, yes, double negatives make it look ugly, so if we do need parameter, can it be arm64.mte=on/off/true/false?
Quoted 3 lines
I don't think "performance analysis" is a good justification for this parameter tbh. We don't tend to add these options for other architectural features, and I don't see why MTE is any different in this regard.
There is an expectation of performance impact with MTE enabled, especially if it's running in synchronous mode. For the in-kernel MTE, we could add a parameter which sets sync vs async at boot time rather than a big disable knob. It won't affect user space however.
Quoted 8 lines
Quoted 6 lines
The other 'justification' is if your hardware has weird unexpected behaviour but I'd like this handled via errata workarounds.
I'll let the people who asked for this to chip in ;). I agree with you that we rarely add these (and I rejected a similar option a few weeks ago on the AMU patchset).
We've been looking into other ways this on/off behavior could be achieved.
The actual question here is what the on/off behaviour is needed for. We can figure out the best mechanism for this once we know what we want to achieve. My wild guess above was performance analysis but that can be toggled by either kernel boot parameter or run-time sysctl (or just the
Kconfigoption).
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
If it is about forcing user space not to use MTE, we may look into some other sysctl controls (we already have one for the tagged address ABI).
We want to allow the end user to be able to easily "opt out" of MTE in favour of better power, perf and battery life.
Who is "the end user" in this case?
Good question. I have a suspicion it's still the (kernel) developer ;).
Quoted 4 lines
If MTE is bad enough for power, performance and battery life that we need a kill switch, then perhaps we shouldn't enable it by default and the few people that want to use it can build a kernel with it enabled. However, then I don't really see what MTE buys you over the existing KASAN implementations.
MTE is faster than KASan (with async mode the fastest), however I'd expect it to still be noticeable compared to no-MTE. It's a trade-off if you want to find security bugs in your code on a large scale.
Quoted 5 lines
I thought the general idea was that you could run in the (cheap) "async" mode, and then re-run in the more expensive "sync" mode to further diagnose any failures. That model seems to work well with these patches, since reporting is disabled by default. Are you saying that there is a significant penalty incurred even when reporting is not enabled?
The tag checking mode is controlled by the user on a per-process basis. The modes and hardware perf/power expectations:
1. no tag checking - no expected performance penalty from the hardware perspective (tags not fetched from memory).
2. async tag checking - tags fetched from memory but checked asynchronously, so it allows the hardware to perform as well as it can (I don't have numbers yet). Probably a small degradation vs (1).
3. sync tag checking - there is an expectation of further perf/power degradation vs (2).
In addition to the hardware aspects above, you have the software cost for colouring memory both on allocation and on free. By default, a malloc()/free() wouldn't touch the memory (maybe some red zones) but with MTE the libc will have to set the colour. That's faster than a memset since it need to store 4 bits for every 16 bytes of address but slower than not doing it at all. For a calloc(), The memset + tag setting can be combined in a single DC instruction.
So, it really depends on what the user is doing. If we want a knob where the user doesn't even attempt to colour pages (not even (1) above), maybe a user space env variable parsed by the libc is a better option.
While MTE and the tagged addr ABI are complementary (one can still set PROT_MTE without enabling the tagged addr ABI), most likely a libc implementation would try to enable the latter before using MTE. We already have a sysctl to force the tagged addr ABI off. The side-effect is that MTE will be disabled in the C library, so assuming no run-time cost (the libc people to confirm).
The tagged addr sysctl doesn't cover the in-kernel MTE but we can leave the discussion for when we have the patches.
Quoted 3 lines
Anyway, we don't offer global runtime/cmdline switches for the vast majority of other architectural features -- instead, we choose a sensible default, and I think we should do the same here.
The sensible defaults are currently "off" with a user opt-in. I think the question is whether we need a "safety" knob at the kernel level like we did with the sysctl abi.tagged_addr_disabled or we leave it to the user as it sees fit (e.g. env variables) since it doesn't affect the kernel (unlike the tagged addr ABI).
Signature · 1 line
Catalin
Catalin Marinas
maintainerRe: [PATCH v4 24/26] arm64: mte: Introduce early param to disable MTE support
In reply to Vladimir MurzinQuoted 25 lines · Vladimir Murzin
Quoted 24 lines · Catalin Marinas
For performance analysis it may be desirable to disable MTE altogether via an early param. Introduce arm64.mte_disable and, if true, filter out the sanitised ID_AA64PFR1_EL1.MTE field to avoid exposing the HWCAP to user.
Signed-off-by Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>---
Notes:
New in v4.Documentation/admin-guide/kernel-parameters.txt | 4 ++++ arch/arm64/kernel/cpufeature.c | 11 +++++++++++ 2 files changed, 15 insertions(+)Documentation/admin-guide/kernel-parameters.txt +0
-0@@ -373,6 +373,10 @@ arcrimi= [HW,NET] ARCnet - "RIM I" (entirely mem-mapped) cards Format: <io>,<irq>,<nodeID>
Quoted 8 lines
Documentation/admin-guide/kernel-parameters.txt line 379
Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0+ arm64.mte_disable= + [ARM64] Disable Linux support for the Memory + Tagging Extension (both user and in-kernel). +Should it really to take parameter (on/off/true/false)? It may lead to expectation that arm64.mte_disable=false should enable MT and, yes, double negatives make it look ugly, so if we do need parameter, can it be arm64.mte=on/off/true/false?
My reasoning about arm64.mte= was that 'on' may lead people to think it does something even when MTE isn't available on the SoC. So I ended up with an explicit 'disable' in the name. Happy to change it if we don't drop this parameter altogether (in the absence of valid use-cases).
Signature · 1 line
Catalin
Re: [PATCH v4 24/26] arm64: mte: Introduce early param to disable MTE support
In reply to Catalin MarinasQuoted 50 lines · Catalin Marinas
For performance analysis it may be desirable to disable MTE altogether via an early param. Introduce arm64.mte_disable and, if true, filter out the sanitised ID_AA64PFR1_EL1.MTE field to avoid exposing the HWCAP to user.
Signed-off-by Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>---
Notes:
New in v4.Documentation/admin-guide/kernel-parameters.txt | 4 ++++ arch/arm64/kernel/cpufeature.c | 11 +++++++++++ 2 files changed, 15 insertions(+)Documentation/admin-guide/kernel-parameters.txt +4
-0@@ -373,6 +373,10 @@ arcrimi= [HW,NET] ARCnet - "RIM I" (entirely mem-mapped) cards Format: <io>,<irq>,<nodeID> + arm64.mte_disable= + [ARM64] Disable Linux support for the Memory + Tagging Extension (both user and in-kernel). + ataflop= [HW,M68k] atarimouse= [HW,MOUSE] Atari Mousearch/arm64/kernel/cpufeature.c +8
-0@@ -126,12 +126,23 @@ static void cpu_enable_cnp(struct arm64_cpu_capabilities const *cap);static bool __system_matches_cap (unsigned int n);#ifdef CONFIG_ARM64_MTE +static bool mte_disable; + +static int __init arm64_mte_disable (char *buf) +{ +return strtobool (buf, &mte_disable); +} +early_param ("arm64.mte_disable" , arm64_mte_disable); +s64 mte_ftr_filter (const struct arm64_ftr_bits *ftrp,s64 val) {
arch/arm64/kernel/cpufeature.c static void cpu_enable_cnp(struct arm64_cpu_capabilities const *cap); · line 148
Quoted 10 lines
arch/arm64/kernel/cpufeature.c +3
-0struct device_node *np;static bool memory_checked = false;static bool mte_capable = true; +if (mte_disable) +return ID_AA64PFR1_MTE_NI ; +/* EL0-only MTE is not supported by Linux, don't expose it */ if (val <ID_AA64PFR1_MTE )return ID_AA64PFR1_MTE_NI ;
Hi Calatin,
While this patch didn't land upstream, we need an MTE kill-switch for Android GKI. Is this patch OK to take as is? Is it still valid?
Thanks!
Re: [PATCH v4 24/26] arm64: mte: Introduce early param to disable MTE support
Quoted 65 lines · Andrey Konovalov
Quoted 50 lines · Catalin Marinas
For performance analysis it may be desirable to disable MTE altogether via an early param. Introduce arm64.mte_disable and, if true, filter out the sanitised ID_AA64PFR1_EL1.MTE field to avoid exposing the HWCAP to user.
Signed-off-by Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>---
Notes:
New in v4.Documentation/admin-guide/kernel-parameters.txt | 4 ++++ arch/arm64/kernel/cpufeature.c | 11 +++++++++++ 2 files changed, 15 insertions(+)Documentation/admin-guide/kernel-parameters.txt +4
-0@@ -373,6 +373,10 @@ arcrimi= [HW,NET] ARCnet - "RIM I" (entirely mem-mapped) cards Format: <io>,<irq>,<nodeID> + arm64.mte_disable= + [ARM64] Disable Linux support for the Memory + Tagging Extension (both user and in-kernel). + ataflop= [HW,M68k] atarimouse= [HW,MOUSE] Atari Mousearch/arm64/kernel/cpufeature.c +8
-0@@ -126,12 +126,23 @@ static void cpu_enable_cnp(struct arm64_cpu_capabilities const *cap);static bool __system_matches_cap (unsigned int n);#ifdef CONFIG_ARM64_MTE +static bool mte_disable; + +static int __init arm64_mte_disable (char *buf) +{ +return strtobool (buf, &mte_disable); +} +early_param ("arm64.mte_disable" , arm64_mte_disable); +s64 mte_ftr_filter (const struct arm64_ftr_bits *ftrp,s64 val) {arch/arm64/kernel/cpufeature.c static void cpu_enable_cnp(struct arm64_cpu_capabilities const *cap); · line 148
Quoted 10 lines
arch/arm64/kernel/cpufeature.c +3
-0struct device_node *np;static bool memory_checked = false;static bool mte_capable = true; +if (mte_disable) +return ID_AA64PFR1_MTE_NI ; +/* EL0-only MTE is not supported by Linux, don't expose it */ if (val <ID_AA64PFR1_MTE )return ID_AA64PFR1_MTE_NI ;Hi Catalin,
Quoted 2 lines
While this patch didn't land upstream, we need an MTE kill-switch for Android GKI. Is this patch OK to take as is? Is it still valid?
Looking at this more closely: looks like this code no longer exists. What would be the approach to add this kind of switch now?
Thanks!
Catalin Marinas
maintainerRe: [PATCH v4 24/26] arm64: mte: Introduce early param to disable MTE support
In reply to Andrey KonovalovQuoted 54 lines · Andrey Konovalov
Quoted 53 lines · Catalin Marinas
For performance analysis it may be desirable to disable MTE altogether via an early param. Introduce arm64.mte_disable and, if true, filter out the sanitised ID_AA64PFR1_EL1.MTE field to avoid exposing the HWCAP to user.
Signed-off-by Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>---
Notes:
New in v4.Documentation/admin-guide/kernel-parameters.txt | 4 ++++ arch/arm64/kernel/cpufeature.c | 11 +++++++++++ 2 files changed, 15 insertions(+)Documentation/admin-guide/kernel-parameters.txt +4
-0@@ -373,6 +373,10 @@ arcrimi= [HW,NET] ARCnet - "RIM I" (entirely mem-mapped) cards Format: <io>,<irq>,<nodeID> + arm64.mte_disable= + [ARM64] Disable Linux support for the Memory + Tagging Extension (both user and in-kernel). + ataflop= [HW,M68k] atarimouse= [HW,MOUSE] Atari Mousearch/arm64/kernel/cpufeature.c +8
-0@@ -126,12 +126,23 @@ static void cpu_enable_cnp(struct arm64_cpu_capabilities const *cap);static bool __system_matches_cap (unsigned int n);#ifdef CONFIG_ARM64_MTE +static bool mte_disable; + +static int __init arm64_mte_disable (char *buf) +{ +return strtobool (buf, &mte_disable); +} +early_param ("arm64.mte_disable" , arm64_mte_disable); +s64 mte_ftr_filter (const struct arm64_ftr_bits *ftrp,s64 val) {struct device_node *np;static bool memory_checked = false;static bool mte_capable = true;
Quoted 9 lines
arch/arm64/kernel/cpufeature.c static void cpu_enable_cnp(struct arm64_cpu_capabilities const *cap); · line 148
Quoted 6 lines
arch/arm64/kernel/cpufeature.c +3
-0+if (mte_disable) +return ID_AA64PFR1_MTE_NI ; +/* EL0-only MTE is not supported by Linux, don't expose it */ if (val <ID_AA64PFR1_MTE )return ID_AA64PFR1_MTE_NI ;While this patch didn't land upstream, we need an MTE kill-switch for Android GKI. Is this patch OK to take as is? Is it still valid?
As you noticed, this code no longer exists. The CPUID is checked early during boot in proc.S, before the MMU is enabled, as you need to set up the MAIR register.
Now, what do you mean by kill switch? There are multiple levels at which one can disable MTE or some of its effects: memory type (MAIR) level, tag allocation (TCR_EL1.ATA), tag checking (SCTLR_EL1.TCF). Apart from the latter, all the other bits are cached in the TLB which make them more problematic to toggle at run-time.
For the kernel, we can currently disable tag checking via the kasan command line options. For user-space, we don't have a kill switch specific to MTE, however one can disable the tagged addr ABI and presumably the C library will avoid generating tagged heap pointers.
Signature · 1 line
Catalin
Re: [PATCH v4 24/26] arm64: mte: Introduce early param to disable MTE support
Quoted 14 lines · Catalin Marinas
Quoted 3 lines
While this patch didn't land upstream, we need an MTE kill-switch for Android GKI. Is this patch OK to take as is? Is it still valid?
As you noticed, this code no longer exists. The CPUID is checked early during boot in
proc.S, before the MMU is enabled, as you need to set up the MAIR register.Now, what do you mean by kill switch? There are multiple levels at which one can disable MTE or some of its effects: memory type (MAIR) level, tag allocation (TCR_EL1.ATA), tag checking (SCTLR_EL1.TCF). Apart from the latter, all the other bits are cached in the TLB which make them more problematic to toggle at run-time.
Quoted 4 lines
For the kernel, we can currently disable tag checking via the kasan command line options. For user-space, we don't have a kill switch specific to MTE, however one can disable the tagged addr ABI and presumably the C library will avoid generating tagged heap pointers.
Just FTR: As discussed off-the-list, there won't be any need for a kill-switch for userspace MTE.
Thanks!
Catalin Marinas
maintainerPATCH v4 25/26 arm64: mte: Kconfig entry
In reply to Catalin MarinasFrom: Vincenzo Frascino <[email protected]>
Add Memory Tagging Extension support to the arm64 kbuild.
---
Notes:
v4:
- select ARCH_USES_PG_ARCH_2
- remove ARCH_NO_SWAP
- default y
arch/arm64/Kconfig | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
arch/arm64/Kconfig +33 -0
@@ -1606,6 +1606,39 @@ config ARCH_RANDOM endmenu +menu "ARMv8.5 architectural features" + +config ARM64_AS_HAS_MTE + def_bool $(as-instr,.arch armv8.5-a+memtag) + +config ARM64_MTE + bool "Memory Tagging Extension support" + default y + depends on ARM64_AS_HAS_MTE && ARM64_TAGGED_ADDR_ABI + select ARCH_USES_HIGH_VMA_FLAGS + select ARCH_USES_PG_ARCH_2 + help + Memory Tagging (part of the ARMv8.5 Extensions) provides + architectural support for run-time, always-on detection of + various classes of memory error to aid with software debugging + to eliminate vulnerabilities arising from memory-unsafe + languages. + + This option enables the support for the Memory Tagging + Extension at EL0 (i.e. for userspace). + + Selecting this option allows the feature to be detected at + runtime. Any secondary CPU not implementing this feature will + not be allowed a late bring-up. + + Userspace binaries that want to use this feature must + explicitly opt in. The mechanism for the userspace is + described in: + + Documentation/arm64/memory-tagging-extension.rst. + +endmenu + config ARM64_SVE bool "ARM Scalable Vector Extension support" default y
Catalin Marinas
maintainerPATCH v4 26/26 arm64: mte: Add Memory Tagging Extension documentation
In reply to Catalin MarinasFrom: Vincenzo Frascino <[email protected]>
Memory Tagging Extension (part of the ARMv8.5 Extensions) provides a mechanism to detect the sources of memory related errors which may be vulnerable to exploitation, including bounds violations, use-after-free, use-after-return, use-out-of-scope and use before initialization errors.
Add Memory Tagging Extension documentation for the arm64 linux kernel support.
---
Notes:
v4:
- Document behaviour of madvise(MADV_DONTNEED/MADV_FREE).
- Document the initial process state on fork/execve.
- Clarify when the kernel uaccess checks the tags.
- Minor updates to the example code.
- A few other minor clean-ups following review.
v3:
- Modify the uaccess checking conditions: only when the sync mode is selected by the user. In async mode, the kernel uaccesses are not checked.
- Clarify that an include mask of 0 (exclude mask 0xffff) results in always generating tag 0.
- Document the ptrace() interface.
v2:
- Documented the uaccess kernel tag checking mode.
- Removed the BTI definitions from cpu-feature-registers.rst.
- Removed the paragraph stating that MTE depends on the tagged address ABI (while the Kconfig entry does, there is no requirement for the user to enable both).
- Changed the GCR_EL1.Exclude handling description following the change in the prctl() interface (include vs exclude mask).
- Updated the example code.
Documentation/arm64/cpu-feature-registers.rst | 2 + Documentation/arm64/elf_hwcaps.rst | 5 + Documentation/arm64/index.rst | 1 + .../arm64/memory-tagging-extension.rst | 297 ++++++++++++++++++ 4 files changed, 305 insertions(+) create mode 100644 Documentation/arm64/memory-tagging-extension.rst
Documentation/arm64/cpu-feature-registers.rst +2 -0
@@ -174,6 +174,8 @@ infrastructure: +------------------------------+---------+---------+ | Name | bits | visible | +------------------------------+---------+---------+ + | MTE | [11-8] | y | + +------------------------------+---------+---------+ | SSBS | [7-4] | y | +------------------------------+---------+---------+
Documentation/arm64/elf_hwcaps.rst +5 -0
@@ -236,6 +236,11 @@ HWCAP2_RNG Functionality implied by ID_AA64ISAR0_EL1.RNDR == 0b0001. +HWCAP2_MTE + + Functionality implied by ID_AA64PFR1_EL1.MTE == 0b0010, as described + by Documentation/arm64/memory-tagging-extension.rst. + 4. Unused AT_HWCAP bits -----------------------
Documentation/arm64/index.rst +1 -0
@@ -14,6 +14,7 @@ ARM64 Architecture hugetlbpage legacy_instructions memory + memory-tagging-extension pointer-authentication silicon-errata sve
Documentation/arm64/memory-tagging-extension.rst new file +297 -0
297 lines not on this page · raw message