kasan: hw_tags: Disable tagging for stack and page-tables
Subsystems (6)
v3 cover kasan: hw_tags: Disable tagging for stack and page-tables
Stacks and page tables are always accessed with the match-all tag, so assigning a new random tag every time at allocation and setting invalid tag at deallocation time, just adds overhead without improving the detection.
With __GFP_SKIP_KASAN the page keeps its poison tag and KASAN_TAG_KERNEL (match-all tag) is stored in the page flags while keeping the poison tag in the hardware. The benefit of it is that 256 tag setting instruction per 4 kB page aren't needed at allocation and deallocation time.
Thus match-all pointers still work, while non-match tags (other than poison tag) still fault.
__GFP_SKIP_KASAN only skips for KASAN_HW_TAGS mode, so coverage is unchanged.
Benchmark:
The benchmark has two modes. In thread mode, the child process forks and creates N threads. In pgtable mode, the parent maps and faults a specified memory size and then forks repeatedly with children exiting immediately.
Thread benchmark:
2000 iterations, 2000 threads: 2.575 s → 2.229 s (~13.4% faster)
The pgtable samples:
- 2048 MB, 2000 iters 19.08 s → 17.62 s (~7.6% faster)
---
Changes since v2:
- Directly skip kasan_unpoison_vmalloc() for GFP_SKIP_KASAN in patch 1
Changes since v1:
- Update description/title
- Patch 1: Simplify skip conditions based on the fact that __GFP_SKIP_KASAN
- Patch 2: Specify _GFP_SKIP_KASAN in THREADINFO_GFP and GFP_VMAP_STACK
Muhammad Usama Anjum (3):
vmalloc: add __GFP_SKIP_KASAN support
kasan: skip HW tagging for all kernel thread stacks
mm: skip KASAN tagging for page-allocated page tables
include/asm-generic/pgalloc.h | 2 +- include/linux/thread_info.h | 2 +- kernel/fork.c | 5 +++-- mm/vmalloc.c | 20 +++++++++++++++++--- 4 files changed, 22 insertions(+), 7 deletions(-)
Signature · 1 line
2.34.1
PATCH v3 1/3 vmalloc: add __GFP_SKIP_KASAN support
From: Muhammad Usama Anjum <[email protected]>
For allocations that will be accessed only with match-all pointers (e.g., kernel stacks), setting tags is wasted work. If the caller already set __GFP_SKIP_KASAN, skip tag setting of vmalloc pages.
Before this patch, __GFP_SKIP_KASAN wasn't being used with vmalloc APIs. So it wasn't being checked. Now its being checked and acted upon. Other KASAN modes are unchanged because __GFP_SKIP_KASAN isn't defined there.
This is a preparatory patch for optimizing kernel stack allocations.
--- mm/vmalloc.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)
mm/vmalloc.c +17 -3
@@ -3939,7 +3939,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, __GFP_NOFAIL | __GFP_ZERO |\ __GFP_NORETRY | __GFP_RETRY_MAYFAIL |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ - GFP_USER | __GFP_NOLOCKDEP) + GFP_USER | __GFP_NOLOCKDEP | __GFP_SKIP_KASAN)static gfp_t vmalloc_fix_flags (gfp_t flags) {
@@ -3980,6 +3980,9 @@ static gfp_t vmalloc_fix_flags(gfp_t flags)* * %__GFP_NOWARN can be used to suppress failure messages. * +* %__GFP_SKIP_KASAN can be used to skip unpoisoning of mapped pages +* (when prot=%PAGE_KERNEL). +* * Can not be called from interrupt nor NMI contexts. * Return: the address of the area or %NULL on failure */
@@ -3993,6 +3996,10 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align,kasan_vmalloc_flags_t kasan_flags =KASAN_VMALLOC_NONE ;unsigned long original_align = align;unsigned int shift =PAGE_SHIFT ; +bool skip_vmalloc_kasan = gfp_mask & __GFP_SKIP_KASAN; + +/* Don't skip metadata kasan unpoisoning */ + gfp_mask &= ~__GFP_SKIP_KASAN;if (WARN_ON_ONCE (!size))return NULL ;
@@ -4041,7 +4048,7 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align,* kasan_unpoison_vmalloc(). */ if (pgprot_val (prot) ==pgprot_val (PAGE_KERNEL )) { -if (kasan_hw_tags_enabled ()) { +if (kasan_hw_tags_enabled () && !skip_vmalloc_kasan) {/* * Modify protection bits to allow tagging. * This must be done before mapping.
@@ -4054,6 +4061,12 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align,* poisoned and zeroed by kasan_unpoison_vmalloc(). */ gfp_mask |= __GFP_SKIP_KASAN | __GFP_SKIP_ZERO; + }else if (skip_vmalloc_kasan) { +/* +* Skip page_alloc unpoisoning physical pages backing +* VM_ALLOC mapping, as requested by caller. +*/ + gfp_mask |= __GFP_SKIP_KASAN; }/* Take note that the mapping is PAGE_KERNEL. */
@@ -4078,7 +4091,8 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, (gfp_mask & __GFP_SKIP_ZERO)) kasan_flags |=KASAN_VMALLOC_INIT ;/* KASAN_VMALLOC_PROT_NORMAL already set if required. */ - area->addr =kasan_unpoison_vmalloc (area->addr, size, kasan_flags); +if (!skip_vmalloc_kasan) + area->addr =kasan_unpoison_vmalloc (area->addr, size, kasan_flags);/* * In this function, newly allocated vm_struct has VM_UNINITIALIZED
Signature · 1 line
2.34.1
Re: [PATCH v3 1/3] vmalloc: add __GFP_SKIP_KASAN support
Quoted 11 lines · Dev Jain
From: Muhammad Usama Anjum <[email protected]>
For allocations that will be accessed only with match-all pointers (e.g., kernel stacks), setting tags is wasted work. If the caller already set
__GFP_SKIP_KASAN, skip tag setting of vmalloc pages.Before this patch,
__GFP_SKIP_KASANwasn't being used with vmalloc APIs. So it wasn't being checked. Now its being checked and acted upon. Other KASAN modes are unchanged because__GFP_SKIP_KASANisn't defined there.
Quoted 5 lines
This is a preparatory patch for optimizing kernel stack allocations.
Co-developed-by Ryan Roberts <[email protected]>Co-developed-by Dev Jain <[email protected]>Signed-off-by Muhammad Usama Anjum <[email protected]>
Co-developers need to sign off as well. See submitting-patches.rst. Same comment about your SoB as on patch 3.
Quoted 48 lines
--- mm/vmalloc.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)mm/vmalloc.c +9
-2@@ -3939,7 +3939,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, __GFP_NOFAIL | __GFP_ZERO |\ __GFP_NORETRY | __GFP_RETRY_MAYFAIL |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ - GFP_USER | __GFP_NOLOCKDEP) + GFP_USER | __GFP_NOLOCKDEP | __GFP_SKIP_KASAN)static gfp_t vmalloc_fix_flags (gfp_t flags) {@@ -3980,6 +3980,9 @@ static gfp_t vmalloc_fix_flags(gfp_t flags)* * %__GFP_NOWARN can be used to suppress failure messages. * +* %__GFP_SKIP_KASAN can be used to skip unpoisoning of mapped pages +* (when prot=%PAGE_KERNEL). +* * Can not be called from interrupt nor NMI contexts. * Return: the address of the area or %NULL on failure */ @@ -3993,6 +3996,10 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align,kasan_vmalloc_flags_t kasan_flags =KASAN_VMALLOC_NONE ;unsigned long original_align = align;unsigned int shift =PAGE_SHIFT ; +bool skip_vmalloc_kasan = gfp_mask & __GFP_SKIP_KASAN; + +/* Don't skip metadata kasan unpoisoning */ + gfp_mask &= ~__GFP_SKIP_KASAN;if (WARN_ON_ONCE (!size))return NULL ;@@ -4041,7 +4048,7 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align,* kasan_unpoison_vmalloc(). */ if (pgprot_val (prot) ==pgprot_val (PAGE_KERNEL )) { -if (kasan_hw_tags_enabled ()) { +if (kasan_hw_tags_enabled () && !skip_vmalloc_kasan) {/* * Modify protection bits to allow tagging. * This must be done before mapping.
mm/vmalloc.c void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, · line 4070
Quoted 10 lines
mm/vmalloc.c +6
-0@@ -4054,6 +4061,12 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align,* poisoned and zeroed by kasan_unpoison_vmalloc(). */ gfp_mask |= __GFP_SKIP_KASAN | __GFP_SKIP_ZERO; + }else if (skip_vmalloc_kasan) { +/* +* Skip page_alloc unpoisoning physical pages backing +* VM_ALLOC mapping, as requested by caller. +*/ + gfp_mask |= __GFP_SKIP_KASAN; }
This playing around with some of the GFP flags meant for metadata and the actual page allocation gets confusing. You remove __GFP_SKIP_KASAN early from gfp_mask, add it back here. You might as well just remove it when calling __get_vm_area_node() and we won't have to figure out why it's added back above.
The __GFP_SKIP_ZERO flag is meant for the page allocator and used in this function later to actually tell kasan to initialise the memory (not skip this). __GFP_SKIP_KASAN, OTOH, is used to actually tell both vmalloc() and the underlying page allocator to avoid tagging. I wonder whether it would be better to have a VM_SKIP_KASAN flag instead and leave the GFP flags alone.
Signature · 1 line
Catalin
Re: [PATCH v3 1/3] vmalloc: add __GFP_SKIP_KASAN support
Quoted 12 lines · Catalin Marinas
Quoted 11 lines · Dev Jain
From: Muhammad Usama Anjum <[email protected]>
For allocations that will be accessed only with match-all pointers (e.g., kernel stacks), setting tags is wasted work. If the caller already set
__GFP_SKIP_KASAN, skip tag setting of vmalloc pages.Before this patch,
__GFP_SKIP_KASANwasn't being used with vmalloc APIs. So it wasn't being checked. Now its being checked and acted upon. Other KASAN modes are unchanged because__GFP_SKIP_KASANisn't defined there.
Quoted 8 lines
Quoted 5 lines
This is a preparatory patch for optimizing kernel stack allocations.
Co-developed-by Ryan Roberts <[email protected]>Co-developed-by Dev Jain <[email protected]>Signed-off-by Muhammad Usama Anjum <[email protected]>Co-developers need to sign off as well. See
submitting-patches.rst. Same comment about your SoB as on patch 3.
oops I forgot, I'll do that.
Quoted 57 lines
Quoted 57 lines
--- mm/vmalloc.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)mm/vmalloc.c +14
-2@@ -3939,7 +3939,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, __GFP_NOFAIL | __GFP_ZERO |\ __GFP_NORETRY | __GFP_RETRY_MAYFAIL |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ - GFP_USER | __GFP_NOLOCKDEP) + GFP_USER | __GFP_NOLOCKDEP | __GFP_SKIP_KASAN)static gfp_t vmalloc_fix_flags (gfp_t flags) {@@ -3980,6 +3980,9 @@ static gfp_t vmalloc_fix_flags(gfp_t flags)* * %__GFP_NOWARN can be used to suppress failure messages. * +* %__GFP_SKIP_KASAN can be used to skip unpoisoning of mapped pages +* (when prot=%PAGE_KERNEL). +* * Can not be called from interrupt nor NMI contexts. * Return: the address of the area or %NULL on failure */ @@ -3993,6 +3996,10 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align,kasan_vmalloc_flags_t kasan_flags =KASAN_VMALLOC_NONE ;unsigned long original_align = align;unsigned int shift =PAGE_SHIFT ; +bool skip_vmalloc_kasan = gfp_mask & __GFP_SKIP_KASAN; + +/* Don't skip metadata kasan unpoisoning */ + gfp_mask &= ~__GFP_SKIP_KASAN;if (WARN_ON_ONCE (!size))return NULL ;@@ -4041,7 +4048,7 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align,* kasan_unpoison_vmalloc(). */ if (pgprot_val (prot) ==pgprot_val (PAGE_KERNEL )) { -if (kasan_hw_tags_enabled ()) { +if (kasan_hw_tags_enabled () && !skip_vmalloc_kasan) {/* * Modify protection bits to allow tagging. * This must be done before mapping. @@ -4054,6 +4061,12 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align,* poisoned and zeroed by kasan_unpoison_vmalloc(). */ gfp_mask |= __GFP_SKIP_KASAN | __GFP_SKIP_ZERO; + }else if (skip_vmalloc_kasan) { +/* +* Skip page_alloc unpoisoning physical pages backing +* VM_ALLOC mapping, as requested by caller. +*/
Quoted 8 lines
mm/vmalloc.c void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, · line 4070
Quoted 2 lines
mm/vmalloc.c +1
-0+ gfp_mask |= __GFP_SKIP_KASAN; }This playing around with some of the GFP flags meant for metadata and the actual page allocation gets confusing. You remove
__GFP_SKIP_KASANearly fromgfp_mask, add it back here. You might as well just remove it when calling__get_vm_area_node()and we won't have to figure out why it's added back above.
Okay I'll do this.
Quoted 7 lines
The
__GFP_SKIP_ZEROflag is meant for the page allocator and used in this function later to actually tell kasan to initialise the memory (not skip this).__GFP_SKIP_KASAN, OTOH, is used to actually tell bothvmalloc()and the underlying page allocator to avoid tagging. I wonder whether it would be better to have aVM_SKIP_KASANflag instead and leave the GFP flags alone.
Lemme reply on the other reply
Quoted 1 line
Re: [PATCH v3 1/3] vmalloc: add __GFP_SKIP_KASAN support
In reply to Dev JainQuoted 14 lines · Dev Jain
mm/vmalloc.c +1
-1@@ -3939,7 +3939,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, __GFP_NOFAIL | __GFP_ZERO |\ __GFP_NORETRY | __GFP_RETRY_MAYFAIL |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ - GFP_USER | __GFP_NOLOCKDEP) + GFP_USER | __GFP_NOLOCKDEP | __GFP_SKIP_KASAN)static gfp_t vmalloc_fix_flags (gfp_t flags) {
mm/vmalloc.c static gfp_t vmalloc_fix_flags(gfp_t flags) · line 3984
Quoted 5 lines
mm/vmalloc.c +2
-0@@ -3980,6 +3980,9 @@ static gfp_t vmalloc_fix_flags(gfp_t flags)* * %__GFP_NOWARN can be used to suppress failure messages. * +* %__GFP_SKIP_KASAN can be used to skip unpoisoning of mapped pages +* (when prot=%PAGE_KERNEL).
I just realised, if we go with this flag for vmalloc(), there's also a comment in gfp_types.h implying that pages are unpoisoned by kasan_unpoison_vmalloc() instead. This is no longer the case with this patch.
A VM_SKIP_KASAN flag may have been nicer but we already have THREADINFO_GFP and GFP_VMAP_STACK, so all those call sites would have to be moved to call the lower-level __vmalloc_node_range().
Signature · 1 line
Catalin
Re: [PATCH v3 1/3] vmalloc: add __GFP_SKIP_KASAN support
Quoted 26 lines · Catalin Marinas
Quoted 14 lines · Dev Jain
mm/vmalloc.c +1
-1@@ -3939,7 +3939,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, __GFP_NOFAIL | __GFP_ZERO |\ __GFP_NORETRY | __GFP_RETRY_MAYFAIL |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ - GFP_USER | __GFP_NOLOCKDEP) + GFP_USER | __GFP_NOLOCKDEP | __GFP_SKIP_KASAN)static gfp_t vmalloc_fix_flags (gfp_t flags) {mm/vmalloc.c static gfp_t vmalloc_fix_flags(gfp_t flags) · line 3984
Quoted 5 lines
mm/vmalloc.c +2
-0@@ -3980,6 +3980,9 @@ static gfp_t vmalloc_fix_flags(gfp_t flags)* * %__GFP_NOWARN can be used to suppress failure messages. * +* %__GFP_SKIP_KASAN can be used to skip unpoisoning of mapped pages +* (when prot=%PAGE_KERNEL). I just realised, if we go with this flag for
vmalloc(), there's also a comment ingfp_types.himplying that pages are unpoisoned bykasan_unpoison_vmalloc()instead. This is no longer the case with this patch.
Quoted 3 lines
A
VM_SKIP_KASANflag may have been nicer but we already haveTHREADINFO_GFPandGFP_VMAP_STACK, so all those call sites would have to be moved to call the lower-level__vmalloc_node_range().
Yep we were discussing this only on v2, and realized the wider cleanup is not trivial and should be made separately.
But I should change the documentation of GFP_SKIP_KASAN to say that if it passed on to vmalloc then it will also skip unpoisoning.
Quoted 1 line
PATCH v3 2/3 kasan: skip HW tagging for all kernel thread stacks
In reply to Dev JainFrom: Muhammad Usama Anjum <[email protected]>
HW-tag KASAN never checks kernel stacks because stack pointers carry the match-all tag, so setting/poisoning tags is pure overhead.
- Add __GFP_SKIP_KASAN to THREADINFO_GFP so every stack allocator that uses it skips tagging (fork path plus arch users)
- Add __GFP_SKIP_KASAN to GFP_VMAP_STACK for the fork-specific vmap stacks.
- When reusing cached vmap stacks, skip kasan_unpoison_range() if HW tags are enabled.
Software KASAN is unchanged; this only affects tag-based KASAN.
--- include/linux/thread_info.h | 2 +- kernel/fork.c | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-)
include/linux/thread_info.h +1 -1
@@ -92,7 +92,7 @@ static inline long set_restart_fn(struct restart_block *restart,#define THREAD_ALIGN THREAD_SIZE#endif -#define THREADINFO_GFP (GFP_KERNEL_ACCOUNT | __GFP_ZERO) +#define THREADINFO_GFP (GFP_KERNEL_ACCOUNT | __GFP_ZERO | __GFP_SKIP_KASAN)/* * flag set/clear/test wrappers
kernel/fork.c +3 -2
@@ -202,7 +202,7 @@ static DEFINE_PER_CPU(struct vm_struct *, cached_stacks[NR_CACHED_STACKS]);* accounting is performed by the code assigning/releasing stacks to tasks. * We need a zeroed memory without __GFP_ACCOUNT. */ -#define GFP_VMAP_STACK (GFP_KERNEL | __GFP_ZERO) +#define GFP_VMAP_STACK (GFP_KERNEL | __GFP_ZERO | __GFP_SKIP_KASAN)struct vm_stack {struct rcu_head rcu;
@@ -340,7 +340,8 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) }/* Reset stack metadata. */ -kasan_unpoison_range (vm_area->addr,THREAD_SIZE ); +if (!kasan_hw_tags_enabled ()) +kasan_unpoison_range (vm_area->addr,THREAD_SIZE ); stack =kasan_reset_tag (vm_area->addr);
Signature · 1 line
2.34.1
PATCH v3 3/3 mm: skip KASAN tagging for page-allocated page tables
In reply to Dev JainFrom: Muhammad Usama Anjum <[email protected]>
Page tables are always accessed via the linear mapping with a match-all tag, so HW-tag KASAN never checks them. For page-allocated tables (PTEs and PGDs etc), avoid the tag setup and poisoning overhead by using __GFP_SKIP_KASAN. SLUB-backed page tables are unchanged for now. (They aren't widely used and require more SLUB related skip logic. Leave it later.)
--- include/asm-generic/pgalloc.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/asm-generic/pgalloc.h +1 -1
@@ -4,7 +4,7 @@#ifdef CONFIG_MMU -#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO) +#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO | __GFP_SKIP_KASAN)#define GFP_PGTABLE_USER (GFP_PGTABLE_KERNEL | __GFP_ACCOUNT)/**
Signature · 1 line
2.34.1
Re: [PATCH v3 3/3] mm: skip KASAN tagging for page-allocated page tables
Quoted 9 lines · Dev Jain
From: Muhammad Usama Anjum <[email protected]>
Page tables are always accessed via the linear mapping with a match-all tag, so HW-tag KASAN never checks them. For page-allocated tables (PTEs and PGDs etc), avoid the tag setup and poisoning overhead by using
__GFP_SKIP_KASAN. SLUB-backed page tables are unchanged for now. (They aren't widely used and require more SLUB related skip logic. Leave it later.)
Quoted 4 lines
Reviewed-by Ryan Roberts <[email protected]>Signed-off-by Muhammad Usama Anjum <[email protected]>Reviewed-by Catalin Marinas <[email protected]>
Your signed-off-by is missing. You must add it if you are reposting someone else's patches.
Also I only got cc'ed on patch 3. You should normally cc all people on all patches. Maybe you could skip this if the patches are some independent cleanups but even so, I'd still cc all the others at least on the cover letter.
Signature · 1 line
Catalin
Re: [PATCH v3 3/3] mm: skip KASAN tagging for page-allocated page tables
Quoted 10 lines · Catalin Marinas
Quoted 9 lines · Dev Jain
From: Muhammad Usama Anjum <[email protected]>
Page tables are always accessed via the linear mapping with a match-all tag, so HW-tag KASAN never checks them. For page-allocated tables (PTEs and PGDs etc), avoid the tag setup and poisoning overhead by using
__GFP_SKIP_KASAN. SLUB-backed page tables are unchanged for now. (They aren't widely used and require more SLUB related skip logic. Leave it later.)
Quoted 7 lines
Quoted 4 lines
Reviewed-by Ryan Roberts <[email protected]>Signed-off-by Muhammad Usama Anjum <[email protected]>Reviewed-by Catalin Marinas <[email protected]>Your signed-off-by is missing. You must add it if you are reposting someone else's patches.
Okay I read submitting-patches properly and didn't know we should add SOB even if we are sending it as it is ...
"Any further SoBs (Signed-off-by:'s) following the author's SoB are from people handling and transporting the patch, but were not involved in its development. SoB chains should reflect the **real** route a patch took as it was propagated to the maintainers and ultimately to Linus, with the first SoB entry signalling primary authorship of a single author."
So I'll re-order the tags as
Quoted 5 lines
Also I only got cc'ed on patch 3. You should normally cc all people on all patches. Maybe you could skip this if the patches are some independent cleanups but even so, I'd still cc all the others at least on the cover letter.
Oops didn't realize my script was not taking a union of the result of get_maintainer on each patch. I'll fix that.