kasan: hw_tags: some micro-optimizations
Subsystems (4)
cover kasan: hw_tags: some micro-optimizations
Patch 1 uses GFP_SKIP_KASAN to skip unpoisoning of a slab page in the page allocator, since slab allocator itself poisons the slab page immediately.
Patch 2 and 3 remove wasted work while poisoning the tail end of the vmalloc/slab allocation.
---
Based on 7.1-rc2.
Dev Jain (3):
mm/slub: hw_tags: skip page-allocator unpoisoning on slab allocation
kasan: avoid re-poisoning tag-based kmalloc redzones
vmalloc: hw_tags: optimize vmalloc redzoning
include/linux/kasan.h | 17 +++++++++---- mm/kasan/common.c | 55 +++++++++++++++++++++++++++++++++---------- mm/kasan/hw_tags.c | 13 ++++++---- mm/page_alloc.c | 2 +- mm/slub.c | 22 ++++++++++++----- 5 files changed, 79 insertions(+), 30 deletions(-)
Signature · 1 line
2.43.0
PATCH 1/3 mm/slub: hw_tags: skip page-allocator unpoisoning on slab allocation
When a new slab page is allocated, the buddy will unpoison the page. Then slab immediately poisons the page via kasan_poison_slab(). This is wasted work.
Similar to what is done in vmalloc currently, use GFP_SKIP_KASAN (hw tags flag only) to skip unpoisoning of the slab page.
--- mm/page_alloc.c | 2 +- mm/slub.c | 11 +++++++++-- 2 files changed, 10 insertions(+), 3 deletions(-)
mm/page_alloc.c +1 -1
@@ -7723,7 +7723,7 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsignedstruct alloc_context ac = { };struct page *page; -VM_WARN_ON_ONCE (gfp_flags & ~__GFP_ACCOUNT); +VM_WARN_ON_ONCE (gfp_flags & ~(__GFP_ACCOUNT | __GFP_SKIP_KASAN));/* * In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is * unsafe in NMI. If spin_trylock() is called from hard IRQ the current
mm/slub.c +9 -2
@@ -3269,9 +3269,16 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node,struct slab *slab;unsigned int order =oo_order (oo); +/* +* New slab pages are immediately poisoned by kasan_poison_slab() +* before any object is handed out, so page allocator unpoisoning +* is wasted work for HW_TAGS KASAN. +*/ + flags |= __GFP_SKIP_KASAN; +if (unlikely (!allow_spin)) - page =alloc_frozen_pages_nolock (0 /* __GFP_COMP is implied */ , - node, order); + page =alloc_frozen_pages_nolock (__GFP_SKIP_KASAN, + node, order);else if (node ==NUMA_NO_NODE ) page =alloc_frozen_pages (flags, order);else
Signature · 1 line
2.43.0
Re: [PATCH 1/3] mm/slub: hw_tags: skip page-allocator unpoisoning on slab allocation
Quoted 31 lines · Dev Jain
When a new slab page is allocated, the buddy will unpoison the page. Then slab immediately poisons the page via
kasan_poison_slab(). This is wasted work.Similar to what is done in vmalloc currently, use
GFP_SKIP_KASAN(hw tags flag only) to skip unpoisoning of the slab page.Signed-off-by Dev Jain <[email protected]>--- mm/page_alloc.c | 2 +- mm/slub.c | 11 +++++++++-- 2 files changed, 10 insertions(+), 3 deletions(-)mm/page_alloc.c +1
-1@@ -7723,7 +7723,7 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsignedstruct alloc_context ac = { };struct page *page; -VM_WARN_ON_ONCE (gfp_flags & ~__GFP_ACCOUNT); +VM_WARN_ON_ONCE (gfp_flags & ~(__GFP_ACCOUNT | __GFP_SKIP_KASAN));/* * In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is * unsafe in NMI. If spin_trylock() is called from hard IRQ the current mm/slub.c +0
-0
mm/slub.c static inline struct slab *alloc_slab_page(gfp_t flags, int node, · line 3277
Quoted 9 lines
mm/slub.c +6
-0@@ -3269,9 +3269,16 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node,struct slab *slab;unsigned int order =oo_order (oo); +/* +* New slab pages are immediately poisoned by kasan_poison_slab() +* before any object is handed out, so page allocator unpoisoning +* is wasted work for HW_TAGS KASAN. +*/ + flags |= __GFP_SKIP_KASAN;
You will also want to elide kasan_poison_slab() right? In which case, it might be better to handle __GFP_SKIP_KASAN in allocate_slab() (which calls alloc_slab_page()), because that's the place where kasan_poison_slab() currently is and it's probably better to keep the logic together.
Note that there is a wrinkle though; logically, there are different types of memory poison; KASAN_PAGE_FREE, KASAN_PAGE_REDZONE, KASAN_SLAB_REDZONE, etc. Memory returned by the page allocator with __GFP_SKIP_KASAN set, will have KASAN_PAGE_FREE poison (I think). But kasan_poison_slab() sets KASAN_SLAB_REDZONE poison.
However, this is only distinguished in practice for KASAN_GENERIC. For KASAN_SW_TAGS and KASAN_HW_TAGS these distinct logical types all map to the same KASAN_TAG_INVALID tag. So this optimization can only be safely applied to KASAN_SW_TAGS and KASAN_HW_TAGS.
It would be nice if this could be abstracted away somehow...
mm/slub.c static inline struct slab *alloc_slab_page(gfp_t flags, int node, · line 3273
Quoted 3 lines
mm/slub.c +1
-1+if (unlikely (!allow_spin)) - page = alloc_frozen_pages_nolock(0/* __GFP_COMP is implied */,
nit: you may want to keep this comment around?
Quoted 6 lines
mm/slub.c +2
-1- node, order); + page =alloc_frozen_pages_nolock (__GFP_SKIP_KASAN, + node, order); else if (node == NUMA_NO_NODE) page =alloc_frozen_pages (flags, order); else
Re: [PATCH 1/3] mm/slub: hw_tags: skip page-allocator unpoisoning on slab allocation
Quoted 35 lines · Ryan Roberts
Quoted 34 lines · Dev Jain
When a new slab page is allocated, the buddy will unpoison the page. Then slab immediately poisons the page via
kasan_poison_slab(). This is wasted work.Similar to what is done in vmalloc currently, use
GFP_SKIP_KASAN(hw tags flag only) to skip unpoisoning of the slab page.Signed-off-by Dev Jain <[email protected]>--- mm/page_alloc.c | 2 +- mm/slub.c | 11 +++++++++-- 2 files changed, 10 insertions(+), 3 deletions(-)mm/page_alloc.c +1
-1@@ -7723,7 +7723,7 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsignedstruct alloc_context ac = { };struct page *page; -VM_WARN_ON_ONCE (gfp_flags & ~__GFP_ACCOUNT); +VM_WARN_ON_ONCE (gfp_flags & ~(__GFP_ACCOUNT | __GFP_SKIP_KASAN));/* * In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is * unsafe in NMI. If spin_trylock() is called from hard IRQ the current mm/slub.c +0
-0@@ -3269,9 +3269,16 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node,struct slab *slab;unsigned int order =oo_order (oo);
Quoted 11 lines
mm/slub.c static inline struct slab *alloc_slab_page(gfp_t flags, int node, · line 3277
Quoted 6 lines
mm/slub.c +6
-0+/* +* New slab pages are immediately poisoned by kasan_poison_slab() +* before any object is handed out, so page allocator unpoisoning +* is wasted work for HW_TAGS KASAN. +*/ + flags |= __GFP_SKIP_KASAN;You will also want to elide
kasan_poison_slab()right? In which case, it might be better to handle__GFP_SKIP_KASANinallocate_slab()(which callsalloc_slab_page()), because that's the place wherekasan_poison_slab()currently is and it's probably better to keep the logic together.
Okay.
Quoted 11 lines
Note that there is a wrinkle though; logically, there are different types of memory poison;
KASAN_PAGE_FREE,KASAN_PAGE_REDZONE,KASAN_SLAB_REDZONE, etc. Memory returned by the page allocator with__GFP_SKIP_KASANset, will haveKASAN_PAGE_FREEpoison (I think). Butkasan_poison_slab()setsKASAN_SLAB_REDZONEpoison.However, this is only distinguished in practice for
KASAN_GENERIC. ForKASAN_SW_TAGSandKASAN_HW_TAGSthese distinct logical types all map to the sameKASAN_TAG_INVALIDtag. So this optimization can only be safely applied toKASAN_SW_TAGSandKASAN_HW_TAGS.
GFP_SKIP_KASAN is only for HW_TAGS.
I will mention the KASAN_SLAB_REDZONE = KASAN_PAGE_FREE for hw tags in the description.
Quoted 8 lines
It would be nice if this could be abstracted away somehow...
mm/slub.c static inline struct slab *alloc_slab_page(gfp_t flags, int node, · line 3273
Quoted 3 lines
mm/slub.c +1
-1+if (unlikely (!allow_spin)) - page = alloc_frozen_pages_nolock(0/* __GFP_COMP is implied */,nit: you may want to keep this comment around?
Yes I will keep this.
Quoted 8 lines
Quoted 7 lines
mm/slub.c +2
-1- node, order); + page =alloc_frozen_pages_nolock (__GFP_SKIP_KASAN, + node, order); else if (node == NUMA_NO_NODE) page =alloc_frozen_pages (flags, order); else
PATCH 2/3 kasan: avoid re-poisoning tag-based kmalloc redzones
In reply to Dev JainWhen we allocate object from slab, kasan will unpoison the entire object. In case of allocation from kmalloc caches, the actual allocation size request can be less than the size of the kmalloc cache. kasan poisons the bytes following allocation size up till object size to catch OOB.
We can do this operation in one shot: while unpoisoning the object upon allocation, only unpoison up till allocation size bytes, so that the bytes following that up till object size remain poisoned.
Currently when we free an object into the slab, we use KASAN_SLAB_FREE for poisoning, and use KASAN_SLAB_REDZONE for poisoning the tail end. For tag-based kasan, these two are equal, as opposed to generic kasan. So we make this optimization only for tag-based kasan.
--- include/linux/kasan.h | 17 +++++++++---- mm/kasan/common.c | 55 +++++++++++++++++++++++++++++++++---------- mm/slub.c | 11 +++++---- 3 files changed, 61 insertions(+), 22 deletions(-)
include/linux/kasan.h +12 -5
@@ -102,6 +102,12 @@ static inline bool kasan_has_integrated_init(void)return kasan_hw_tags_enabled (); } +static inline bool kasan_has_tag_based_kmalloc_redzones (void ) +{ +return kasan_enabled () && + (IS_ENABLED (CONFIG_KASAN_SW_TAGS ) ||kasan_hw_tags_enabled ()); +} +#ifdef CONFIG_KASAN void __kasan_unpoison_range (const void *addr,size_t size); static __always_inline void kasan_unpoison_range(const void *addr, size_t size)
@@ -244,13 +250,14 @@ static __always_inline void kasan_kfree_large(void *ptr)__kasan_kfree_large (ptr, _RET_IP_); } -void *__must_check __kasan_slab_alloc (struct kmem_cache *s, -void *object,gfp_t flags,bool init); +void *__must_check __kasan_slab_alloc (struct kmem_cache *s,void *object, +size_t size,gfp_t flags,bool init); static __always_inline void * __must_check kasan_slab_alloc( -struct kmem_cache *s,void *object,gfp_t flags,bool init) +struct kmem_cache *s,void *object,size_t size, +gfp_t flags,bool init) {if (kasan_enabled ()) -return __kasan_slab_alloc (s, object, flags, init); +return __kasan_slab_alloc (s, object, size, flags, init);return object; }
@@ -437,7 +444,7 @@ static inline bool kasan_slab_free(struct kmem_cache *s, void *object, }static inline void kasan_kfree_large (void *ptr) {}static inline void *kasan_slab_alloc (struct kmem_cache *s,void *object, -gfp_t flags,bool init) +size_t size,gfp_t flags,bool init) {return object; }
mm/kasan/common.c +42 -13
101 lines not on this page · raw message
mm/slub.c +7 -4
@@ -4550,8 +4550,9 @@ bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,* replacement of current poisoning under certain debug option, and * won't break other sanity checks. */ -if (kmem_cache_debug_flags (s,SLAB_STORE_USER |SLAB_RED_ZONE ) && - (s->flags &SLAB_KMALLOC )) +if ((s->flags &SLAB_KMALLOC ) && + (kmem_cache_debug_flags (s,SLAB_STORE_USER |SLAB_RED_ZONE ) || +kasan_has_tag_based_kmalloc_redzones ())) zero_size = orig_size;/*
@@ -4573,7 +4574,8 @@ bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,* As p[i] might get tagged, memset and kmemleak hook come after KASAN. */ for (i =0 ; i < size; i++) { - p[i] =kasan_slab_alloc (s, p[i], init_flags, kasan_init); + p[i] =kasan_slab_alloc (s, p[i], orig_size, init_flags, + kasan_init);if (p[i] && init && (!kasan_init || !kasan_has_integrated_init ()))memset (p[i],0 , zero_size);
@@ -7615,7 +7617,8 @@ static void early_kmem_cache_node_alloc(int node)#ifdef CONFIG_SLUB_DEBUG init_object (kmem_cache_node, n,SLUB_RED_ACTIVE );#endif - n =kasan_slab_alloc (kmem_cache_node, n,GFP_KERNEL , false); + n =kasan_slab_alloc (kmem_cache_node, n, kmem_cache_node->object_size, +GFP_KERNEL , false); slab->freelist =get_freepointer (kmem_cache_node, n); slab->inuse =1 ; kmem_cache_node->per_node[node].node = n;
Signature · 1 line
2.43.0
Re: [PATCH 2/3] kasan: avoid re-poisoning tag-based kmalloc redzones
Quoted 9 lines · Dev Jain
When we allocate object from slab, kasan will unpoison the entire object. In case of allocation from kmalloc caches, the actual allocation size request can be less than the size of the kmalloc cache. kasan poisons the bytes following allocation size up till object size to catch OOB.
We can do this operation in one shot: while unpoisoning the object upon allocation, only unpoison up till allocation size bytes, so that the bytes following that up till object size remain poisoned.
Quoted 7 lines
Currently when we free an object into the slab, we use
KASAN_SLAB_FREEfor poisoning, and useKASAN_SLAB_REDZONEfor poisoning the tail end. For tag-based kasan, these two are equal, as opposed to generic kasan. So we make this optimization only for tag-based kasan.Signed-off-by Dev Jain <[email protected]>---
Sashiko: https://sashiko.dev/#/patchset/[email protected]
So I'll explain what Sashiko is saying with examples. If I am mis-interpreting then please correct me!
Problem 1:
p = kzalloc(96, GFP_KERNEL); // kmalloc-128
p = krealloc(p, 120, GFP_KERNEL | __GFP_ZERO);
Before my patch, kzalloc zeroed bytes 0..127, krealloc exposes bytes 96..119 which are already zero from kzalloc.
After my patch, kzalloc zeroes only 0..95, krealloc may think old size is 128, so it does not zero 96..119, 96..119 may contain stale data.
Why is this a problem? Surely caller of krealloc shouldn't be relying on a previous zeroing. krealloc documentation explicitly says that GFP_ZERO is ignored.
Problem 2 for SW tags:
p = kzalloc(97, GFP_KERNEL); // kmalloc-128, granule = 16
Before patch:
KASAN unpoisons 0..111
memset zeroes 0..127
p[100] will not be caught by KASAN, but reads zero
After patch:
KASAN unpoisons 0..111
memset zeroes only 0..96
p[100] will not be caught by KASAN and reads stale data
Again why is this a problem?
Quoted 197 lines
include/linux/kasan.h | 17 +++++++++---- mm/kasan/common.c | 55 +++++++++++++++++++++++++++++++++---------- mm/slub.c | 11 +++++---- 3 files changed, 61 insertions(+), 22 deletions(-)include/linux/kasan.h +12
-5@@ -102,6 +102,12 @@ static inline bool kasan_has_integrated_init(void)return kasan_hw_tags_enabled (); } +static inline bool kasan_has_tag_based_kmalloc_redzones (void ) +{ +return kasan_enabled () && + (IS_ENABLED (CONFIG_KASAN_SW_TAGS ) ||kasan_hw_tags_enabled ()); +} +#ifdef CONFIG_KASAN void __kasan_unpoison_range (const void *addr,size_t size); static __always_inline void kasan_unpoison_range(const void *addr, size_t size)@@ -244,13 +250,14 @@ static __always_inline void kasan_kfree_large(void *ptr)__kasan_kfree_large (ptr, _RET_IP_); } -void *__must_check __kasan_slab_alloc (struct kmem_cache *s, -void *object,gfp_t flags,bool init); +void *__must_check __kasan_slab_alloc (struct kmem_cache *s,void *object, +size_t size,gfp_t flags,bool init); static __always_inline void * __must_check kasan_slab_alloc( -struct kmem_cache *s,void *object,gfp_t flags,bool init) +struct kmem_cache *s,void *object,size_t size, +gfp_t flags,bool init) {if (kasan_enabled ()) -return __kasan_slab_alloc (s, object, flags, init); +return __kasan_slab_alloc (s, object, size, flags, init);return object; }@@ -437,7 +444,7 @@ static inline bool kasan_slab_free(struct kmem_cache *s, void *object, }static inline void kasan_kfree_large (void *ptr) {}static inline void *kasan_slab_alloc (struct kmem_cache *s,void *object, -gfp_t flags,bool init) +size_t size,gfp_t flags,bool init) {return object; }mm/kasan/common.c +42
-13@@ -326,14 +326,25 @@ void __kasan_kfree_large(void *ptr, unsigned long ip)/* The object will be poisoned by kasan_poison_pages(). */ } +static inline size_t slab_unpoison_size (struct kmem_cache *cache,size_t size) +{ +if (kasan_has_tag_based_kmalloc_redzones () &&is_kmalloc_cache (cache)) +return min_t (size_t, size, cache->object_size); + +return cache->object_size; +} +static inline void unpoison_slab_object (struct kmem_cache *cache,void *object, -gfp_t flags,bool init) +size_t size,gfp_t flags,bool init) {/* -* Unpoison the whole object. For kmalloc() allocations, -* poison_kmalloc_redzone() will do precise poisoning. +* For tag-based modes, kmalloc redzones all use the same invalid tag. +* Keep the tail poisoned and only unpoison the requested allocation +* size. Generic KASAN keeps distinct shadow values for free objects and +* redzones, so it still unpoisons the whole object and later poisons +* the precise redzone. */ -kasan_unpoison (object, cache->object_size, init); +kasan_unpoison (object,slab_unpoison_size (cache, size), init);/* Save alloc info (if possible) for non-kmalloc() allocations. */ if (kasan_stack_collection_enabled () && !is_kmalloc_cache (cache))@@ -341,7 +352,8 @@ static inline void unpoison_slab_object(struct kmem_cache *cache, void *object, }void *__must_check __kasan_slab_alloc (struct kmem_cache *cache, -void *object,gfp_t flags,bool init) +void *object,size_t size, +gfp_t flags,bool init) {u8 tag;void *tagged_object;@@ -363,11 +375,18 @@ void * __must_check __kasan_slab_alloc(struct kmem_cache *cache, tagged_object =set_tag (object, tag);/* Unpoison the object and save alloc info for non-kmalloc() allocations. */ -unpoison_slab_object (cache, tagged_object, flags, init); +unpoison_slab_object (cache, tagged_object, size, flags, init);return tagged_object; } +static inline void save_kmalloc_alloc_info (struct kmem_cache *cache, +void *object,gfp_t flags) +{ +if (kasan_stack_collection_enabled () &&is_kmalloc_cache (cache)) +kasan_save_alloc_info (cache, object, flags); +} +static inline void poison_kmalloc_redzone (struct kmem_cache *cache,const void *object,size_t size,gfp_t flags) {@@ -394,8 +413,7 @@ static inline void poison_kmalloc_redzone(struct kmem_cache *cache,* Save alloc info (if possible) for kmalloc() allocations. * This also rewrites the alloc info when called from kasan_krealloc(). */ -if (kasan_stack_collection_enabled () &&is_kmalloc_cache (cache)) -kasan_save_alloc_info (cache, (void *)object, flags); +save_kmalloc_alloc_info (cache, (void *)object, flags); }@@ -411,8 +429,14 @@ void * __must_check __kasan_kmalloc(struct kmem_cache *cache, const void *objectif (is_kfence_address (object))return (void *)object; -/* The object has already been unpoisoned by kasan_slab_alloc(). */ -poison_kmalloc_redzone (cache, object, size, flags); +/* +* For tag-based modes, the object has already been precisely +* unpoisoned by kasan_slab_alloc(). The tail remains poisoned. +*/ +if (kasan_has_tag_based_kmalloc_redzones ()) +save_kmalloc_alloc_info (cache, (void *)object, flags); +else +poison_kmalloc_redzone (cache, object, size, flags);/* Keep the tag that was set by kasan_slab_alloc(). */ return (void *)object;@@ -561,11 +585,16 @@ void __kasan_mempool_unpoison_object(void *ptr, size_t size, unsigned long ip)return ;/* Unpoison the object and save alloc info for non-kmalloc() allocations. */ -unpoison_slab_object (slab->slab_cache, ptr, flags, false); +unpoison_slab_object (slab->slab_cache, ptr, size, flags, false);/* Poison the redzone and save alloc info for kmalloc() allocations. */ -if (is_kmalloc_cache (slab->slab_cache)) -poison_kmalloc_redzone (slab->slab_cache, ptr, size, flags); +if (is_kmalloc_cache (slab->slab_cache)) { +if (kasan_has_tag_based_kmalloc_redzones ()) +save_kmalloc_alloc_info (slab->slab_cache, ptr, flags); +else +poison_kmalloc_redzone (slab->slab_cache, ptr, size, + flags); + } }bool __kasan_check_byte (const void *address,unsigned long ip)mm/slub.c +7
-4@@ -4550,8 +4550,9 @@ bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,* replacement of current poisoning under certain debug option, and * won't break other sanity checks. */ -if (kmem_cache_debug_flags (s,SLAB_STORE_USER |SLAB_RED_ZONE ) && - (s->flags &SLAB_KMALLOC )) +if ((s->flags &SLAB_KMALLOC ) && + (kmem_cache_debug_flags (s,SLAB_STORE_USER |SLAB_RED_ZONE ) || +kasan_has_tag_based_kmalloc_redzones ())) zero_size = orig_size;/* @@ -4573,7 +4574,8 @@ bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,* As p[i] might get tagged, memset and kmemleak hook come after KASAN. */ for (i =0 ; i < size; i++) { - p[i] =kasan_slab_alloc (s, p[i], init_flags, kasan_init); + p[i] =kasan_slab_alloc (s, p[i], orig_size, init_flags, + kasan_init);if (p[i] && init && (!kasan_init || !kasan_has_integrated_init ()))memset (p[i],0 , zero_size);@@ -7615,7 +7617,8 @@ static void early_kmem_cache_node_alloc(int node)#ifdef CONFIG_SLUB_DEBUG init_object (kmem_cache_node, n,SLUB_RED_ACTIVE );#endif - n =kasan_slab_alloc (kmem_cache_node, n,GFP_KERNEL , false); + n =kasan_slab_alloc (kmem_cache_node, n, kmem_cache_node->object_size, +GFP_KERNEL , false); slab->freelist =get_freepointer (kmem_cache_node, n); slab->inuse =1 ; kmem_cache_node->per_node[node].node = n;
Re: [PATCH 2/3] kasan: avoid re-poisoning tag-based kmalloc redzones
Quoted 30 lines · Dev Jain
Quoted 9 lines · Dev Jain
When we allocate object from slab, kasan will unpoison the entire object. In case of allocation from kmalloc caches, the actual allocation size request can be less than the size of the kmalloc cache. kasan poisons the bytes following allocation size up till object size to catch OOB.
We can do this operation in one shot: while unpoisoning the object upon allocation, only unpoison up till allocation size bytes, so that the bytes following that up till object size remain poisoned.
Quoted 7 lines
Currently when we free an object into the slab, we use
KASAN_SLAB_FREEfor poisoning, and useKASAN_SLAB_REDZONEfor poisoning the tail end. For tag-based kasan, these two are equal, as opposed to generic kasan. So we make this optimization only for tag-based kasan.Signed-off-by Dev Jain <[email protected]>---
Sashiko: https://sashiko.dev/#/patchset/[email protected]
So I'll explain what Sashiko is saying with examples. If I am mis-interpreting then please correct me!
Problem 1:
p =
kzalloc(96, GFP_KERNEL); // kmalloc-128
p =krealloc(p, 120, GFP_KERNEL | __GFP_ZERO);
Quoted 7 lines
Before my patch, kzalloc zeroed bytes 0..127, krealloc exposes bytes 96..119 which are already zero from kzalloc.
After my patch, kzalloc zeroes only 0..95, krealloc may think old size is 128, so it does not zero 96..119, 96..119 may contain stale data.
Why is this a problem?
Intuitively, I would say it's a problem if you're leaking when you previously were not.
Quoted 3 lines
Surely caller of krealloc shouldn't be
relying on a previous zeroing. krealloc documentation explicitly says thatGFP_ZEROis ignored.
What would happen if you did:
p = kzalloc(96, GFP_KERNEL); // kmalloc-128
p = krealloc(p, 129, GFP_KERNEL | __GFP_ZERO);
Presumably the krealloc() would have to move the whole allocation to the next biggest bin? Would p[128] be zeroed? If so, then I'd say your change is problematic.
Quoted 12 lines
Problem 2 for SW tags:
p =
kzalloc(97, GFP_KERNEL); // kmalloc-128, granule = 16Before patch:
KASAN unpoisons 0..111
memset zeroes 0..127
p[100] will not be caught by KASAN, but reads zero
Quoted 7 lines
After patch:
KASAN unpoisons 0..111
memset zeroes only 0..96
p[100] will not be caught by KASAN and reads stale dataAgain why is this a problem?
Because you have defeated KASAN and are leaking data? I think you'd want to zero at least to the end of the KASAN granule?
Thanks,
Ryan
Quoted 199 lines
Quoted 198 lines
include/linux/kasan.h | 17 +++++++++---- mm/kasan/common.c | 55 +++++++++++++++++++++++++++++++++---------- mm/slub.c | 11 +++++---- 3 files changed, 61 insertions(+), 22 deletions(-)include/linux/kasan.h +12
-5@@ -102,6 +102,12 @@ static inline bool kasan_has_integrated_init(void)return kasan_hw_tags_enabled (); } +static inline bool kasan_has_tag_based_kmalloc_redzones (void ) +{ +return kasan_enabled () && + (IS_ENABLED (CONFIG_KASAN_SW_TAGS ) ||kasan_hw_tags_enabled ()); +} +#ifdef CONFIG_KASAN void __kasan_unpoison_range (const void *addr,size_t size); static __always_inline void kasan_unpoison_range(const void *addr, size_t size)@@ -244,13 +250,14 @@ static __always_inline void kasan_kfree_large(void *ptr)__kasan_kfree_large (ptr, _RET_IP_); } -void *__must_check __kasan_slab_alloc (struct kmem_cache *s, -void *object,gfp_t flags,bool init); +void *__must_check __kasan_slab_alloc (struct kmem_cache *s,void *object, +size_t size,gfp_t flags,bool init); static __always_inline void * __must_check kasan_slab_alloc( -struct kmem_cache *s,void *object,gfp_t flags,bool init) +struct kmem_cache *s,void *object,size_t size, +gfp_t flags,bool init) {if (kasan_enabled ()) -return __kasan_slab_alloc (s, object, flags, init); +return __kasan_slab_alloc (s, object, size, flags, init);return object; }@@ -437,7 +444,7 @@ static inline bool kasan_slab_free(struct kmem_cache *s, void *object, }static inline void kasan_kfree_large (void *ptr) {}static inline void *kasan_slab_alloc (struct kmem_cache *s,void *object, -gfp_t flags,bool init) +size_t size,gfp_t flags,bool init) {return object; }mm/kasan/common.c +42
-13@@ -326,14 +326,25 @@ void __kasan_kfree_large(void *ptr, unsigned long ip)/* The object will be poisoned by kasan_poison_pages(). */ } +static inline size_t slab_unpoison_size (struct kmem_cache *cache,size_t size) +{ +if (kasan_has_tag_based_kmalloc_redzones () &&is_kmalloc_cache (cache)) +return min_t (size_t, size, cache->object_size); + +return cache->object_size; +} +static inline void unpoison_slab_object (struct kmem_cache *cache,void *object, -gfp_t flags,bool init) +size_t size,gfp_t flags,bool init) {/* -* Unpoison the whole object. For kmalloc() allocations, -* poison_kmalloc_redzone() will do precise poisoning. +* For tag-based modes, kmalloc redzones all use the same invalid tag. +* Keep the tail poisoned and only unpoison the requested allocation +* size. Generic KASAN keeps distinct shadow values for free objects and +* redzones, so it still unpoisons the whole object and later poisons +* the precise redzone. */ -kasan_unpoison (object, cache->object_size, init); +kasan_unpoison (object,slab_unpoison_size (cache, size), init);/* Save alloc info (if possible) for non-kmalloc() allocations. */ if (kasan_stack_collection_enabled () && !is_kmalloc_cache (cache))@@ -341,7 +352,8 @@ static inline void unpoison_slab_object(struct kmem_cache *cache, void *object, }void *__must_check __kasan_slab_alloc (struct kmem_cache *cache, -void *object,gfp_t flags,bool init) +void *object,size_t size, +gfp_t flags,bool init) {u8 tag;void *tagged_object;@@ -363,11 +375,18 @@ void * __must_check __kasan_slab_alloc(struct kmem_cache *cache, tagged_object =set_tag (object, tag);/* Unpoison the object and save alloc info for non-kmalloc() allocations. */ -unpoison_slab_object (cache, tagged_object, flags, init); +unpoison_slab_object (cache, tagged_object, size, flags, init);return tagged_object; } +static inline void save_kmalloc_alloc_info (struct kmem_cache *cache, +void *object,gfp_t flags) +{ +if (kasan_stack_collection_enabled () &&is_kmalloc_cache (cache)) +kasan_save_alloc_info (cache, object, flags); +} +static inline void poison_kmalloc_redzone (struct kmem_cache *cache,const void *object,size_t size,gfp_t flags) {@@ -394,8 +413,7 @@ static inline void poison_kmalloc_redzone(struct kmem_cache *cache,* Save alloc info (if possible) for kmalloc() allocations. * This also rewrites the alloc info when called from kasan_krealloc(). */ -if (kasan_stack_collection_enabled () &&is_kmalloc_cache (cache)) -kasan_save_alloc_info (cache, (void *)object, flags); +save_kmalloc_alloc_info (cache, (void *)object, flags); }@@ -411,8 +429,14 @@ void * __must_check __kasan_kmalloc(struct kmem_cache *cache, const void *objectif (is_kfence_address (object))return (void *)object; -/* The object has already been unpoisoned by kasan_slab_alloc(). */ -poison_kmalloc_redzone (cache, object, size, flags); +/* +* For tag-based modes, the object has already been precisely +* unpoisoned by kasan_slab_alloc(). The tail remains poisoned. +*/ +if (kasan_has_tag_based_kmalloc_redzones ()) +save_kmalloc_alloc_info (cache, (void *)object, flags); +else +poison_kmalloc_redzone (cache, object, size, flags);/* Keep the tag that was set by kasan_slab_alloc(). */ return (void *)object;@@ -561,11 +585,16 @@ void __kasan_mempool_unpoison_object(void *ptr, size_t size, unsigned long ip)return ;/* Unpoison the object and save alloc info for non-kmalloc() allocations. */ -unpoison_slab_object (slab->slab_cache, ptr, flags, false); +unpoison_slab_object (slab->slab_cache, ptr, size, flags, false);/* Poison the redzone and save alloc info for kmalloc() allocations. */ -if (is_kmalloc_cache (slab->slab_cache)) -poison_kmalloc_redzone (slab->slab_cache, ptr, size, flags); +if (is_kmalloc_cache (slab->slab_cache)) { +if (kasan_has_tag_based_kmalloc_redzones ()) +save_kmalloc_alloc_info (slab->slab_cache, ptr, flags); +else +poison_kmalloc_redzone (slab->slab_cache, ptr, size, + flags); + } }bool __kasan_check_byte (const void *address,unsigned long ip)mm/slub.c +7
-4@@ -4550,8 +4550,9 @@ bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,* replacement of current poisoning under certain debug option, and * won't break other sanity checks. */ -if (kmem_cache_debug_flags (s,SLAB_STORE_USER |SLAB_RED_ZONE ) && - (s->flags &SLAB_KMALLOC )) +if ((s->flags &SLAB_KMALLOC ) && + (kmem_cache_debug_flags (s,SLAB_STORE_USER |SLAB_RED_ZONE ) || +kasan_has_tag_based_kmalloc_redzones ())) zero_size = orig_size;/* @@ -4573,7 +4574,8 @@ bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,* As p[i] might get tagged, memset and kmemleak hook come after KASAN. */ for (i =0 ; i < size; i++) { - p[i] =kasan_slab_alloc (s, p[i], init_flags, kasan_init); + p[i] =kasan_slab_alloc (s, p[i], orig_size, init_flags, + kasan_init);if (p[i] && init && (!kasan_init || !kasan_has_integrated_init ()))memset (p[i],0 , zero_size);@@ -7615,7 +7617,8 @@ static void early_kmem_cache_node_alloc(int node)#ifdef CONFIG_SLUB_DEBUG init_object (kmem_cache_node, n,SLUB_RED_ACTIVE );#endif - n =kasan_slab_alloc (kmem_cache_node, n,GFP_KERNEL , false); + n =kasan_slab_alloc (kmem_cache_node, n, kmem_cache_node->object_size, +GFP_KERNEL , false); slab->freelist =get_freepointer (kmem_cache_node, n); slab->inuse =1 ; kmem_cache_node->per_node[node].node = n;
Re: [PATCH 2/3] kasan: avoid re-poisoning tag-based kmalloc redzones
Quoted 68 lines · Ryan Roberts
Quoted 30 lines · Dev Jain
Quoted 9 lines · Dev Jain
When we allocate object from slab, kasan will unpoison the entire object. In case of allocation from kmalloc caches, the actual allocation size request can be less than the size of the kmalloc cache. kasan poisons the bytes following allocation size up till object size to catch OOB.
We can do this operation in one shot: while unpoisoning the object upon allocation, only unpoison up till allocation size bytes, so that the bytes following that up till object size remain poisoned.
Quoted 7 lines
Currently when we free an object into the slab, we use
KASAN_SLAB_FREEfor poisoning, and useKASAN_SLAB_REDZONEfor poisoning the tail end. For tag-based kasan, these two are equal, as opposed to generic kasan. So we make this optimization only for tag-based kasan.Signed-off-by Dev Jain <[email protected]>---
Sashiko: https://sashiko.dev/#/patchset/[email protected]
So I'll explain what Sashiko is saying with examples. If I am mis-interpreting then please correct me!
Problem 1:
p =
kzalloc(96, GFP_KERNEL); // kmalloc-128
p =krealloc(p, 120, GFP_KERNEL | __GFP_ZERO);Quoted 7 lines
Before my patch, kzalloc zeroed bytes 0..127, krealloc exposes bytes 96..119 which are already zero from kzalloc.
After my patch, kzalloc zeroes only 0..95, krealloc may think old size is 128, so it does not zero 96..119, 96..119 may contain stale data.
Why is this a problem?
Intuitively, I would say it's a problem if you're leaking when you previously were not.
Quoted 3 lines
Surely caller of krealloc shouldn't be
relying on a previous zeroing. krealloc documentation explicitly says thatGFP_ZEROis ignored.What would happen if you did:
p =
kzalloc(96, GFP_KERNEL); // kmalloc-128
p =krealloc(p, 129, GFP_KERNEL | __GFP_ZERO);Presumably the
krealloc()would have to move the whole allocation to the next biggest bin? Would p[128] be zeroed? If so, then I'd say your change is problematic.Quoted 14 lines
Problem 2 for SW tags:
p =
kzalloc(97, GFP_KERNEL); // kmalloc-128, granule = 16Before patch:
KASAN unpoisons 0..111
memset zeroes 0..127
p[100] will not be caught by KASAN, but reads zeroAfter patch:
Quoted 8 lines
Quoted 5 lines
KASAN unpoisons 0..111
memset zeroes only 0..96
p[100] will not be caught by KASAN and reads stale dataAgain why is this a problem?
Because you have defeated KASAN and are leaking data? I think you'd want to zero at least to the end of the KASAN granule?
I got the problem now. The problem is that krealloc() does not know what was the old allocation size.
So only talking about HW tags, can't we simply do the following. We can use HW tags integrated init to initialize only the part we want to unpoison.
mm/kasan/common.c +1 -1
@@ -474,7 +474,7 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag* Part of it might already have been unpoisoned, but it's unknown * how big that part is. */ -kasan_unpoison (object, size, false); +kasan_unpoison (object, size,want_init_on_alloc (flags)); slab =virt_to_slab (object);
mm/slub.c +1 -1
@@ -6686,7 +6686,7 @@ __do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags, goto alloc_new;/* Zero out spare memory. */ -if (want_init_on_alloc (flags)) { +if (want_init_on_alloc (flags) && !kasan_has_integrated_init ()) {kasan_disable_current ();if (orig_size && orig_size < new_size)memset (kasan_reset_tag (p) + orig_size,0 , new_size - orig_size);
Quoted 206 lines
Thanks,
RyanQuoted 199 lines
Quoted 198 lines
include/linux/kasan.h | 17 +++++++++---- mm/kasan/common.c | 55 +++++++++++++++++++++++++++++++++---------- mm/slub.c | 11 +++++---- 3 files changed, 61 insertions(+), 22 deletions(-)include/linux/kasan.h +12
-5@@ -102,6 +102,12 @@ static inline bool kasan_has_integrated_init(void)return kasan_hw_tags_enabled (); } +static inline bool kasan_has_tag_based_kmalloc_redzones (void ) +{ +return kasan_enabled () && + (IS_ENABLED (CONFIG_KASAN_SW_TAGS ) ||kasan_hw_tags_enabled ()); +} +#ifdef CONFIG_KASAN void __kasan_unpoison_range (const void *addr,size_t size); static __always_inline void kasan_unpoison_range(const void *addr, size_t size)@@ -244,13 +250,14 @@ static __always_inline void kasan_kfree_large(void *ptr)__kasan_kfree_large (ptr, _RET_IP_); } -void *__must_check __kasan_slab_alloc (struct kmem_cache *s, -void *object,gfp_t flags,bool init); +void *__must_check __kasan_slab_alloc (struct kmem_cache *s,void *object, +size_t size,gfp_t flags,bool init); static __always_inline void * __must_check kasan_slab_alloc( -struct kmem_cache *s,void *object,gfp_t flags,bool init) +struct kmem_cache *s,void *object,size_t size, +gfp_t flags,bool init) {if (kasan_enabled ()) -return __kasan_slab_alloc (s, object, flags, init); +return __kasan_slab_alloc (s, object, size, flags, init);return object; }@@ -437,7 +444,7 @@ static inline bool kasan_slab_free(struct kmem_cache *s, void *object, }static inline void kasan_kfree_large (void *ptr) {}static inline void *kasan_slab_alloc (struct kmem_cache *s,void *object, -gfp_t flags,bool init) +size_t size,gfp_t flags,bool init) {return object; }mm/kasan/common.c +42
-13@@ -326,14 +326,25 @@ void __kasan_kfree_large(void *ptr, unsigned long ip)/* The object will be poisoned by kasan_poison_pages(). */ } +static inline size_t slab_unpoison_size (struct kmem_cache *cache,size_t size) +{ +if (kasan_has_tag_based_kmalloc_redzones () &&is_kmalloc_cache (cache)) +return min_t (size_t, size, cache->object_size); + +return cache->object_size; +} +static inline void unpoison_slab_object (struct kmem_cache *cache,void *object, -gfp_t flags,bool init) +size_t size,gfp_t flags,bool init) {/* -* Unpoison the whole object. For kmalloc() allocations, -* poison_kmalloc_redzone() will do precise poisoning. +* For tag-based modes, kmalloc redzones all use the same invalid tag. +* Keep the tail poisoned and only unpoison the requested allocation +* size. Generic KASAN keeps distinct shadow values for free objects and +* redzones, so it still unpoisons the whole object and later poisons +* the precise redzone. */ -kasan_unpoison (object, cache->object_size, init); +kasan_unpoison (object,slab_unpoison_size (cache, size), init);/* Save alloc info (if possible) for non-kmalloc() allocations. */ if (kasan_stack_collection_enabled () && !is_kmalloc_cache (cache))@@ -341,7 +352,8 @@ static inline void unpoison_slab_object(struct kmem_cache *cache, void *object, }void *__must_check __kasan_slab_alloc (struct kmem_cache *cache, -void *object,gfp_t flags,bool init) +void *object,size_t size, +gfp_t flags,bool init) {u8 tag;void *tagged_object;@@ -363,11 +375,18 @@ void * __must_check __kasan_slab_alloc(struct kmem_cache *cache, tagged_object =set_tag (object, tag);/* Unpoison the object and save alloc info for non-kmalloc() allocations. */ -unpoison_slab_object (cache, tagged_object, flags, init); +unpoison_slab_object (cache, tagged_object, size, flags, init);return tagged_object; } +static inline void save_kmalloc_alloc_info (struct kmem_cache *cache, +void *object,gfp_t flags) +{ +if (kasan_stack_collection_enabled () &&is_kmalloc_cache (cache)) +kasan_save_alloc_info (cache, object, flags); +} +static inline void poison_kmalloc_redzone (struct kmem_cache *cache,const void *object,size_t size,gfp_t flags) {@@ -394,8 +413,7 @@ static inline void poison_kmalloc_redzone(struct kmem_cache *cache,* Save alloc info (if possible) for kmalloc() allocations. * This also rewrites the alloc info when called from kasan_krealloc(). */ -if (kasan_stack_collection_enabled () &&is_kmalloc_cache (cache)) -kasan_save_alloc_info (cache, (void *)object, flags); +save_kmalloc_alloc_info (cache, (void *)object, flags); }@@ -411,8 +429,14 @@ void * __must_check __kasan_kmalloc(struct kmem_cache *cache, const void *objectif (is_kfence_address (object))return (void *)object; -/* The object has already been unpoisoned by kasan_slab_alloc(). */ -poison_kmalloc_redzone (cache, object, size, flags); +/* +* For tag-based modes, the object has already been precisely +* unpoisoned by kasan_slab_alloc(). The tail remains poisoned. +*/ +if (kasan_has_tag_based_kmalloc_redzones ()) +save_kmalloc_alloc_info (cache, (void *)object, flags); +else +poison_kmalloc_redzone (cache, object, size, flags);/* Keep the tag that was set by kasan_slab_alloc(). */ return (void *)object;@@ -561,11 +585,16 @@ void __kasan_mempool_unpoison_object(void *ptr, size_t size, unsigned long ip)return ;/* Unpoison the object and save alloc info for non-kmalloc() allocations. */ -unpoison_slab_object (slab->slab_cache, ptr, flags, false); +unpoison_slab_object (slab->slab_cache, ptr, size, flags, false);/* Poison the redzone and save alloc info for kmalloc() allocations. */ -if (is_kmalloc_cache (slab->slab_cache)) -poison_kmalloc_redzone (slab->slab_cache, ptr, size, flags); +if (is_kmalloc_cache (slab->slab_cache)) { +if (kasan_has_tag_based_kmalloc_redzones ()) +save_kmalloc_alloc_info (slab->slab_cache, ptr, flags); +else +poison_kmalloc_redzone (slab->slab_cache, ptr, size, + flags); + } }bool __kasan_check_byte (const void *address,unsigned long ip)mm/slub.c +7
-4@@ -4550,8 +4550,9 @@ bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,* replacement of current poisoning under certain debug option, and * won't break other sanity checks. */ -if (kmem_cache_debug_flags (s,SLAB_STORE_USER |SLAB_RED_ZONE ) && - (s->flags &SLAB_KMALLOC )) +if ((s->flags &SLAB_KMALLOC ) && + (kmem_cache_debug_flags (s,SLAB_STORE_USER |SLAB_RED_ZONE ) || +kasan_has_tag_based_kmalloc_redzones ())) zero_size = orig_size;/* @@ -4573,7 +4574,8 @@ bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,* As p[i] might get tagged, memset and kmemleak hook come after KASAN. */ for (i =0 ; i < size; i++) { - p[i] =kasan_slab_alloc (s, p[i], init_flags, kasan_init); + p[i] =kasan_slab_alloc (s, p[i], orig_size, init_flags, + kasan_init);if (p[i] && init && (!kasan_init || !kasan_has_integrated_init ()))memset (p[i],0 , zero_size);@@ -7615,7 +7617,8 @@ static void early_kmem_cache_node_alloc(int node)#ifdef CONFIG_SLUB_DEBUG init_object (kmem_cache_node, n,SLUB_RED_ACTIVE );#endif - n =kasan_slab_alloc (kmem_cache_node, n,GFP_KERNEL , false); + n =kasan_slab_alloc (kmem_cache_node, n, kmem_cache_node->object_size, +GFP_KERNEL , false); slab->freelist =get_freepointer (kmem_cache_node, n); slab->inuse =1 ; kmem_cache_node->per_node[node].node = n;
Re: [PATCH 2/3] kasan: avoid re-poisoning tag-based kmalloc redzones
Quoted 27 lines
I got the problem now. The problem is that
krealloc()does not know what was the old allocation size.So only talking about HW tags, can't we simply do the following. We can use HW tags integrated init to initialize only the part we want to unpoison.
mm/kasan/common.c +1
-1@@ -474,7 +474,7 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag* Part of it might already have been unpoisoned, but it's unknown * how big that part is. */ -kasan_unpoison (object, size, false); +kasan_unpoison (object, size,want_init_on_alloc (flags)); slab =virt_to_slab (object);mm/slub.c +0
-0
mm/slub.c __do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags, · line 6692
Quoted 8 lines
mm/slub.c +1
-1@@ -6686,7 +6686,7 @@ __do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags, goto alloc_new;/* Zero out spare memory. */ -if (want_init_on_alloc (flags)) { +if (want_init_on_alloc (flags) && !kasan_has_integrated_init ()) {kasan_disable_current ();if (orig_size && orig_size < new_size)memset (kasan_reset_tag (p) + orig_size,0 , new_size - orig_size);
Okay, this would overwrite the data which we still need.
Quoted 5 lines
Quoted 4 lines
Re: [PATCH 2/3] kasan: avoid re-poisoning tag-based kmalloc redzones
In reply to Dev JainQuoted 10 lines
Problem 1:
p =
kzalloc(96, GFP_KERNEL); // kmalloc-128
p =krealloc(p, 120, GFP_KERNEL | __GFP_ZERO);Before my patch, kzalloc zeroed bytes 0..127, krealloc exposes bytes 96..119 which are already zero from kzalloc.
Quoted 6 lines
After my patch, kzalloc zeroes only 0..95, krealloc may think old size is 128, so it does not zero 96..119, 96..119 may contain stale data.
Why is this a problem? Surely caller of krealloc shouldn't be relying on a previous zeroing. krealloc documentation explicitly says that
GFP_ZEROis ignored.
My bad for looking at a very old tree :( the documentation in the upstream kernel does not say that.
PATCH 3/3 vmalloc: hw_tags: optimize vmalloc redzoning
In reply to Dev JainIf the allocation size is less than a page, vmalloc first unpoisons the entire page, then poisons the tail with KASAN_TAG_INVALID (for hw tags), to catch OOB.
Instead, unpoison the allocation length, and then poison the tail, saving some work.
--- mm/kasan/hw_tags.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
mm/kasan/hw_tags.c +8 -5
@@ -364,9 +364,6 @@ void *__kasan_unpoison_vmalloc(const void *start, unsigned long size, tag = (flags &KASAN_VMALLOC_KEEP_TAG ) ?get_tag (start) :kasan_random_tag (); start =set_tag (start, tag); -/* Unpoison and initialize memory up to size. */ -kasan_unpoison (start, size, flags &KASAN_VMALLOC_INIT ); -/* * Explicitly poison and initialize the in-page vmalloc() redzone. * Unlike software KASAN modes, hardware tag-based KASAN doesn't
@@ -375,8 +372,14 @@ void *__kasan_unpoison_vmalloc(const void *start, unsigned long size, redzone_start =round_up ((unsigned long )start + size,KASAN_GRANULE_SIZE ); redzone_size =round_up (redzone_start,PAGE_SIZE ) - redzone_start; -kasan_poison ((void *)redzone_start, redzone_size,KASAN_TAG_INVALID , - flags &KASAN_VMALLOC_INIT ); + +/* Unpoison and initialize memory before the redzone. */ +kasan_unpoison (start, redzone_start - (unsigned long )start, + flags &KASAN_VMALLOC_INIT ); + +if (redzone_size) +kasan_poison ((void *)redzone_start, redzone_size, +KASAN_TAG_INVALID , flags &KASAN_VMALLOC_INIT );/* * Set per-page tag flags to allow accessing physical memory for the
Signature · 1 line
2.43.0
Re: [PATCH 3/3] vmalloc: hw_tags: optimize vmalloc redzoning
Quoted 6 lines · Dev Jain
If the allocation size is less than a page, vmalloc first unpoisons the entire page, then poisons the tail with
KASAN_TAG_INVALID(for hw tags), to catch OOB.Instead, unpoison the allocation length, and then poison the tail, saving some work.
My bad, I see that the actual allocation size (and not page-aligned size) is passed down to kasan_unpoison_vmalloc, so this patch is a no-op.
I'll drop it.
Quoted 37 lines
Signed-off-by Dev Jain <[email protected]>--- mm/kasan/hw_tags.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)mm/kasan/hw_tags.c +8
-5@@ -364,9 +364,6 @@ void *__kasan_unpoison_vmalloc(const void *start, unsigned long size, tag = (flags &KASAN_VMALLOC_KEEP_TAG ) ?get_tag (start) :kasan_random_tag (); start =set_tag (start, tag); -/* Unpoison and initialize memory up to size. */ -kasan_unpoison (start, size, flags &KASAN_VMALLOC_INIT ); -/* * Explicitly poison and initialize the in-page vmalloc() redzone. * Unlike software KASAN modes, hardware tag-based KASAN doesn't @@ -375,8 +372,14 @@ void *__kasan_unpoison_vmalloc(const void *start, unsigned long size, redzone_start =round_up ((unsigned long )start + size,KASAN_GRANULE_SIZE ); redzone_size =round_up (redzone_start,PAGE_SIZE ) - redzone_start; -kasan_poison ((void *)redzone_start, redzone_size,KASAN_TAG_INVALID , - flags &KASAN_VMALLOC_INIT ); + +/* Unpoison and initialize memory before the redzone. */ +kasan_unpoison (start, redzone_start - (unsigned long )start, + flags &KASAN_VMALLOC_INIT ); + +if (redzone_size) +kasan_poison ((void *)redzone_start, redzone_size, +KASAN_TAG_INVALID , flags &KASAN_VMALLOC_INIT );/* * Set per-page tag flags to allow accessing physical memory for the
Harry Yoo (Oracle)
maintainerI have a few questions...
Does the performance of KASAN_HW_TAGS matter?
Yes, the performance of Hardware Tag-Based KASAN matters because it is meant to be used in production on mobile devices as it utilizes cutting-edge (to me) hardware support instead of compiler instrumentation.
Do you have data or are you planning to measure the performance impact of this work?
Quoted 2 lines · Dev Jain
Patch 1 uses
GFP_SKIP_KASANto skip unpoisoning of a slab page in the page allocator, since slab allocator itself poisons the slab page immediately.
If __GFP_SKIP_KASAN skips unpoisoning, why should the slab allocator poison the whole page (kasan_poison_slab())? It's already poisoned.
Quoted 21 lines
Patch 2 and 3 remove wasted work while poisoning the tail end of the vmalloc/slab allocation.
---
Based on 7.1-rc2.Dev Jain (3):
mm/slub:hw_tags: skip page-allocator unpoisoning on slab allocation
kasan: avoid re-poisoning tag-based kmalloc redzones
vmalloc:hw_tags: optimize vmalloc redzoninginclude/linux/kasan.h | 17 +++++++++---- mm/kasan/common.c | 55 +++++++++++++++++++++++++++++++++---------- mm/kasan/hw_tags.c | 13 ++++++---- mm/page_alloc.c | 2 +- mm/slub.c | 22 ++++++++++++----- 5 files changed, 79 insertions(+), 30 deletions(-)Signature · 1 line
2.43.0
Signature · 2 lines
Cheers, Harry / Hyeonggon
Quoted 11 lines · Harry Yoo (Oracle)
I have a few questions...
Does the performance of
KASAN_HW_TAGSmatter?Yes, the performance of Hardware Tag-Based KASAN matters because it is meant to be used in production on mobile devices as it utilizes cutting-edge (to me) hardware support instead of compiler instrumentation.
Do you have data or are you planning to measure the performance impact of this work?
I will get back on this.
Quoted 7 lines · Dev Jain
Quoted 2 lines
Patch 1 uses
GFP_SKIP_KASANto skip unpoisoning of a slab page in the page allocator, since slab allocator itself poisons the slab page immediately.If
__GFP_SKIP_KASANskips unpoisoning, why should the slab allocator poison the whole page (kasan_poison_slab())? It's already poisoned.
Nice observation! Although it may not always be the case that the slab page is already poisoned. If should_skip_kasan_poison() in the buddy code triggers, then a freed page is not poisoned.
But, the point of kasan_poison_slab() is still served without it - an OOB for a slab object is still caught probabilistically because the allocation has a random tag.
Quoted 23 lines
Quoted 22 lines
Patch 2 and 3 remove wasted work while poisoning the tail end of the vmalloc/slab allocation.
---
Based on 7.1-rc2.Dev Jain (3):
mm/slub:hw_tags: skip page-allocator unpoisoning on slab allocation
kasan: avoid re-poisoning tag-based kmalloc redzones
vmalloc:hw_tags: optimize vmalloc redzoninginclude/linux/kasan.h | 17 +++++++++---- mm/kasan/common.c | 55 +++++++++++++++++++++++++++++++++---------- mm/kasan/hw_tags.c | 13 ++++++---- mm/page_alloc.c | 2 +- mm/slub.c | 22 ++++++++++++----- 5 files changed, 79 insertions(+), 30 deletions(-)Signature · 1 line
2.43.0