kasan: hw_tags: Disable tagging for stack and page-tables
Subsystems (6)
v4 cover kasan: hw_tags: Disable tagging for stack and page-tables
Stacks and page tables are always accessed with the match-all tag, so assigning a new random tag every time at allocation and setting invalid tag at deallocation time, just adds overhead without improving the detection.
With __GFP_SKIP_KASAN the page keeps its poison tag and KASAN_TAG_KERNEL (match-all tag) is stored in the page flags while keeping the poison tag in the hardware. The benefit of it is that 256 tag setting instruction per 4 kB page aren't needed at allocation and deallocation time.
Thus match-all pointers still work, while non-match tags (other than poison tag) still fault.
__GFP_SKIP_KASAN only skips for KASAN_HW_TAGS mode, so coverage is unchanged.
Benchmark:
The benchmark has two modes. In thread mode, the child process forks and creates N threads. In pgtable mode, the parent maps and faults a specified memory size and then forks repeatedly with children exiting immediately.
Thread benchmark:
2000 iterations, 2000 threads: 2.575 s → 2.229 s (~13.4% faster)
The pgtable samples:
- 2048 MB, 2000 iters 19.08 s → 17.62 s (~7.6% faster)
---
Applies on 7-0-rc1.
Changes since v3->v4:
- Sashiko noticed: https://sashiko.dev/#/patchset/20260424130157.3163009-1-dev.jain%40arm.com
Fix this by honouring vmalloc skip via GFP_SKIP_KASAN only in HW tags case, to avoid unintended skipping in SW/generic KASAN.
- Instead of removing and adding GFP_SKIP_KASAN into gfp_flags, simply
call __get_vm_area_node() without it
- Update GFP_SKIP_KASAN documentation
- Put missing SOB by me
v2->v3:
- Directly skip kasan_unpoison_vmalloc() for GFP_SKIP_KASAN in patch 1
v1->v2:
- Update description/title
- Patch 1: Simplify skip conditions based on the fact that __GFP_SKIP_KASAN
- Patch 2: Specify _GFP_SKIP_KASAN in THREADINFO_GFP and GFP_VMAP_STACK
Muhammad Usama Anjum (3):
vmalloc: add __GFP_SKIP_KASAN support
kasan: skip HW tagging for all kernel thread stacks
mm: skip KASAN tagging for page-allocated page tables
include/asm-generic/pgalloc.h | 2 +- include/linux/gfp_types.h | 6 +++--- include/linux/thread_info.h | 2 +- kernel/fork.c | 5 +++-- mm/vmalloc.c | 13 +++++++++---- 5 files changed, 17 insertions(+), 11 deletions(-)
Signature · 1 line
2.34.1
PATCH v4 1/3 vmalloc: add __GFP_SKIP_KASAN support
From: Muhammad Usama Anjum <[email protected]>
For allocations that will be accessed only with match-all pointers (e.g., kernel stacks), setting tags is wasted work. If the caller already set __GFP_SKIP_KASAN, skip tag setting of vmalloc pages.
Before this patch, __GFP_SKIP_KASAN wasn't being used with vmalloc APIs. So it wasn't being checked. Now its being checked and acted upon. Other KASAN modes are unchanged because __GFP_SKIP_KASAN is ignored for them in the page allocator, and in vmalloc too we ignore this flag for them.
This is a preparatory patch for optimizing kernel stack allocations.
--- include/linux/gfp_types.h | 6 +++--- mm/vmalloc.c | 13 +++++++++---- 2 files changed, 12 insertions(+), 7 deletions(-)
include/linux/gfp_types.h +3 -3
@@ -281,9 +281,9 @@ enum {* * %__GFP_SKIP_KASAN makes KASAN skip unpoisoning on page allocation. * Used for userspace and vmalloc pages; the latter are unpoisoned by -* kasan_unpoison_vmalloc instead. For userspace pages, results in -* poisoning being skipped as well, see should_skip_kasan_poison for -* details. Only effective in HW_TAGS mode. +* kasan_unpoison_vmalloc instead. If passed to vmalloc, kasan_unpoison_vmalloc +* is skipped too. For userspace pages, results in poisoning being skipped as +* well, see should_skip_kasan_poison for details. Only effective in HW_TAGS mode. */ #define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN)#define __GFP_COMP ((__force gfp_t)___GFP_COMP)
mm/vmalloc.c +9 -4
@@ -3939,7 +3939,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, __GFP_NOFAIL | __GFP_ZERO |\ __GFP_NORETRY | __GFP_RETRY_MAYFAIL |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ - GFP_USER | __GFP_NOLOCKDEP) + GFP_USER | __GFP_NOLOCKDEP | __GFP_SKIP_KASAN)static gfp_t vmalloc_fix_flags (gfp_t flags) {
@@ -3980,6 +3980,9 @@ static gfp_t vmalloc_fix_flags(gfp_t flags)* * %__GFP_NOWARN can be used to suppress failure messages. * +* %__GFP_SKIP_KASAN can be used to skip unpoisoning of mapped pages +* (when prot=%PAGE_KERNEL). +* * Can not be called from interrupt nor NMI contexts. * Return: the address of the area or %NULL on failure */
@@ -3993,6 +3996,7 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align,kasan_vmalloc_flags_t kasan_flags =KASAN_VMALLOC_NONE ;unsigned long original_align = align;unsigned int shift =PAGE_SHIFT ; +bool skip_vmalloc_kasan =kasan_hw_tags_enabled () && (gfp_mask & __GFP_SKIP_KASAN);if (WARN_ON_ONCE (!size))return NULL ;
@@ -4023,7 +4027,7 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, again: area =__get_vm_area_node (size, align, shift,VM_ALLOC |VM_UNINITIALIZED | vm_flags, start, end, node, - gfp_mask, caller); + gfp_mask & ~__GFP_SKIP_KASAN, caller);if (!area) {bool nofail = gfp_mask & __GFP_NOFAIL; warn_alloc(gfp_mask, NULL,
@@ -4041,7 +4045,7 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align,* kasan_unpoison_vmalloc(). */ if (pgprot_val (prot) ==pgprot_val (PAGE_KERNEL )) { -if (kasan_hw_tags_enabled ()) { +if (kasan_hw_tags_enabled () && !skip_vmalloc_kasan) {/* * Modify protection bits to allow tagging. * This must be done before mapping.
@@ -4078,7 +4082,8 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, (gfp_mask & __GFP_SKIP_ZERO)) kasan_flags |=KASAN_VMALLOC_INIT ;/* KASAN_VMALLOC_PROT_NORMAL already set if required. */ - area->addr =kasan_unpoison_vmalloc (area->addr, size, kasan_flags); +if (!skip_vmalloc_kasan) + area->addr =kasan_unpoison_vmalloc (area->addr, size, kasan_flags);/* * In this function, newly allocated vm_struct has VM_UNINITIALIZED
Signature · 1 line
2.34.1
Re: [PATCH v4 1/3] vmalloc: add __GFP_SKIP_KASAN support
Quoted 29 lines · Dev Jain
From: Muhammad Usama Anjum <[email protected]>
For allocations that will be accessed only with match-all pointers (e.g., kernel stacks), setting tags is wasted work. If the caller already set
__GFP_SKIP_KASAN, skip tag setting of vmalloc pages.Before this patch,
__GFP_SKIP_KASANwasn't being used with vmalloc APIs. So it wasn't being checked. Now its being checked and acted upon. Other KASAN modes are unchanged because__GFP_SKIP_KASANis ignored for them in the page allocator, and in vmalloc too we ignore this flag for them.This is a preparatory patch for optimizing kernel stack allocations.
Signed-off-by Muhammad Usama Anjum <[email protected]>Co-developed-by Ryan Roberts <[email protected]>Signed-off-by Ryan Roberts <[email protected]>Co-developed-by Dev Jain <[email protected]>Signed-off-by Dev Jain <[email protected]>--- include/linux/gfp_types.h | 6 +++--- mm/vmalloc.c | 13 +++++++++---- 2 files changed, 12 insertions(+), 7 deletions(-)include/linux/gfp_types.h +0
-0
include/linux/gfp_types.h enum { · line 286
Quoted 9 lines
include/linux/gfp_types.h +3
-3@@ -281,9 +281,9 @@ enum {* * %__GFP_SKIP_KASAN makes KASAN skip unpoisoning on page allocation. * Used for userspace and vmalloc pages; the latter are unpoisoned by -* kasan_unpoison_vmalloc instead. For userspace pages, results in -* poisoning being skipped as well, see should_skip_kasan_poison for -* details. Only effective in HW_TAGS mode. +* kasan_unpoison_vmalloc instead. If passed to vmalloc, kasan_unpoison_vmalloc +* is skipped too. For userspace pages, results in poisoning being skipped as +* well, see should_skip_kasan_poison for details. Only effective in HW_TAGS mode.
Nit: keep the comment line length to less than 80, ideally close to the comments above (for esthetic reasons).
PATCH v4 2/3 kasan: skip HW tagging for all kernel thread stacks
In reply to Dev JainFrom: Muhammad Usama Anjum <[email protected]>
HW-tag KASAN never checks kernel stacks because stack pointers carry the match-all tag, so setting/poisoning tags is pure overhead.
- Add __GFP_SKIP_KASAN to THREADINFO_GFP so every stack allocator that uses it skips tagging (fork path plus arch users)
- Add __GFP_SKIP_KASAN to GFP_VMAP_STACK for the fork-specific vmap stacks.
- When reusing cached vmap stacks, skip kasan_unpoison_range() if HW tags are enabled.
Software KASAN is unchanged; this only affects tag-based KASAN.
--- include/linux/thread_info.h | 2 +- kernel/fork.c | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-)
include/linux/thread_info.h +1 -1
@@ -92,7 +92,7 @@ static inline long set_restart_fn(struct restart_block *restart,#define THREAD_ALIGN THREAD_SIZE#endif -#define THREADINFO_GFP (GFP_KERNEL_ACCOUNT | __GFP_ZERO) +#define THREADINFO_GFP (GFP_KERNEL_ACCOUNT | __GFP_ZERO | __GFP_SKIP_KASAN)/* * flag set/clear/test wrappers
kernel/fork.c +3 -2
@@ -204,7 +204,7 @@ static DEFINE_PER_CPU(struct vm_struct *, cached_stacks[NR_CACHED_STACKS]);* accounting is performed by the code assigning/releasing stacks to tasks. * We need a zeroed memory without __GFP_ACCOUNT. */ -#define GFP_VMAP_STACK (GFP_KERNEL | __GFP_ZERO) +#define GFP_VMAP_STACK (GFP_KERNEL | __GFP_ZERO | __GFP_SKIP_KASAN)struct vm_stack {struct rcu_head rcu;
@@ -342,7 +342,8 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) }/* Reset stack metadata. */ -kasan_unpoison_range (vm_area->addr,THREAD_SIZE ); +if (!kasan_hw_tags_enabled ()) +kasan_unpoison_range (vm_area->addr,THREAD_SIZE ); stack =kasan_reset_tag (vm_area->addr);
Signature · 1 line
2.34.1
Re: [PATCH v4 2/3] kasan: skip HW tagging for all kernel thread stacks
Quoted 12 lines · Dev Jain
From: Muhammad Usama Anjum <[email protected]>
HW-tag KASAN never checks kernel stacks because stack pointers carry the match-all tag, so setting/poisoning tags is pure overhead.
- Add
__GFP_SKIP_KASANtoTHREADINFO_GFPso every stack allocator that uses it skips tagging (fork path plus arch users)
- Add__GFP_SKIP_KASANtoGFP_VMAP_STACKfor the fork-specific vmap stacks.
- When reusing cached vmap stacks, skipkasan_unpoison_range()if HW tags are enabled.
Quoted 4 lines
Software KASAN is unchanged; this only affects tag-based KASAN.
Signed-off-by Muhammad Usama Anjum <[email protected]>Signed-off-by Dev Jain <[email protected]>
PATCH v4 3/3 mm: skip KASAN tagging for page-allocated page tables
In reply to Dev JainFrom: Muhammad Usama Anjum <[email protected]>
Page tables are always accessed via the linear mapping with a match-all tag, so HW-tag KASAN never checks them. For page-allocated tables (PTEs and PGDs etc), avoid the tag setup and poisoning overhead by using __GFP_SKIP_KASAN. SLUB-backed page tables are unchanged for now. (They aren't widely used and require more SLUB related skip logic. Leave it later.)
--- include/asm-generic/pgalloc.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/asm-generic/pgalloc.h +1 -1
@@ -4,7 +4,7 @@#ifdef CONFIG_MMU -#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO) +#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO | __GFP_SKIP_KASAN)#define GFP_PGTABLE_USER (GFP_PGTABLE_KERNEL | __GFP_ACCOUNT)/**
Signature · 1 line
2.34.1