kasan: add software tag-based mode for arm64
Subsystems (13)
v8 cover kasan: add software tag-based mode for arm64
This patchset adds a new software tag-based mode to KASAN [1]. (Initially this mode was called KHWASAN, but it got renamed,
see the naming rationale at the end of this section).
The plan is to implement HWASan [2] for the kernel with the incentive, that it's going to have comparable to KASAN performance, but in the same time consume much less memory, trading that off for somewhat imprecise bug detection and being supported only for arm64.
The underlying ideas of the approach used by software tag-based KASAN are:
1. By using the Top Byte Ignore (TBI) arm64 CPU feature, we can store pointer tags in the top byte of each kernel pointer.
2. Using shadow memory, we can store memory tags for each chunk of kernel memory.
3. On each memory allocation, we can generate a random tag, embed it into the returned pointer and set the memory tags that correspond to this chunk of memory to the same value.
4. By using compiler instrumentation, before each memory access we can add a check that the pointer tag matches the tag of the memory that is being accessed.
5. On a tag mismatch we report an error.
With this patchset the existing KASAN mode gets renamed to generic KASAN, with the word "generic" meaning that the implementation can be supported by any architecture as it is purely software.
The new mode this patchset adds is called software tag-based KASAN. The word "tag-based" refers to the fact that this mode uses tags embedded into the top byte of kernel pointers and the TBI arm64 CPU feature that allows to dereference such pointers. The word "software" here means that shadow memory manipulation and tag checking on pointer dereference is done in software. As it is the only tag-based implementation right now, "software tag-based" KASAN is sometimes referred to as simply "tag-based" in this patchset.
A potential expansion of this mode is a hardware tag-based mode, which would use hardware memory tagging support (announced by Arm [3]) instead of compiler instrumentation and manual shadow memory manipulation.
Same as generic KASAN, software tag-based KASAN is strictly a debugging feature.
[1] https://www.kernel.org/doc/html/latest/dev-tools/kasan.html
[2] http://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html
====== Rationale
On mobile devices generic KASAN's memory usage is significant problem. One of the main reasons to have tag-based KASAN is to be able to perform a similar set of checks as the generic one does, but with lower memory requirements.
Comment from Vishwath Mohan <[email protected]>:
I don't have data on-hand, but anecdotally both ASAN and KASAN have proven problematic to enable for environments that don't tolerate the increased memory pressure well. This includes,
(a) Low-memory form factors - Wear, TV, Things, lower-tier phones like Go,
(c) Connected components like Pixel's visual core [1].
These are both places I'd love to have a low(er) memory footprint option at my disposal.
Comment from Evgenii Stepanov <[email protected]>:
Looking at a live Android device under load, slab (according to /proc/meminfo) + kernel stack take 8-10% available RAM (~350MB). KASAN's overhead of 2x - 3x on top of it is not insignificant.
Not having this overhead enables near-production use - ex. running KASAN/KHWASAN kernel on a personal, daily-use device to catch bugs that do not reproduce in test configuration. These are the ones that often cost the most engineering time to track down.
CPU overhead is bad, but generally tolerable. RAM is critical, in our experience. Once it gets low enough, OOM-killer makes your life miserable.
====== Technical details
Software tag-based KASAN mode is implemented in a very similar way to the generic one. This patchset essentially does the following:
1. TCR_TBI1 is set to enable Top Byte Ignore.
2. Shadow memory is used (with a different scale, 1:16, so each shadow byte corresponds to 16 bytes of kernel memory) to store memory tags.
3. All slab objects are aligned to shadow scale, which is 16 bytes.
4. All pointers returned from the slab allocator are tagged with a random tag and the corresponding shadow memory is poisoned with the same value.
5. Compiler instrumentation is used to insert tag checks. Either by calling callbacks or by inlining them (CONFIG_KASAN_OUTLINE and CONFIG_KASAN_INLINE flags are reused).
6. When a tag mismatch is detected in callback instrumentation mode KASAN simply prints a bug report. In case of inline instrumentation, clang inserts a brk instruction, and KASAN has it's own brk handler, which reports the bug.
7. The memory in between slab objects is marked with a reserved tag, and acts as a redzone.
8. When a slab object is freed it's marked with a reserved tag.
Bug detection is imprecise for two reasons:
1. We won't catch some small out-of-bounds accesses, that fall into the same shadow cell, as the last byte of a slab object.
2. We only have 1 byte to store tags, which means we have a 1/256 probability of a tag match for an incorrect access (actually even slightly less due to reserved tag values).
Despite that there's a particular type of bugs that tag-based KASAN can detect compared to generic KASAN: use-after-free after the object has been allocated by someone else.
====== Testing
Some kernel developers voiced a concern that changing the top byte of kernel pointers may lead to subtle bugs that are difficult to discover. To address this concern deliberate testing has been performed.
It doesn't seem feasible to do some kind of static checking to find potential issues with pointer tagging, so a dynamic approach was taken. All pointer comparisons/subtractions have been instrumented in an LLVM compiler pass and a kernel module that would print a bug report whenever two pointers with different tags are being compared/subtracted (ignoring comparisons with NULL pointers and with pointers obtained by casting an error code to a pointer type) has been used. Then the kernel has been booted in QEMU and on an Odroid C2 board and syzkaller has been run.
This yielded the following results.
The two places that look interesting are:
is_vmalloc_addr in include/linux/mm.his_kernel_rodata in mm/util.c
Here we compare a pointer with some fixed untagged values to make sure that the pointer lies in a particular part of the kernel address space. Since tag-based KASAN doesn't add tags to pointers that belong to rodata or vmalloc regions, this should work as is. To make sure debug checks to those two functions that check that the result doesn't change whether we operate on pointers with or without untagging has been added.
A few other cases that don't look that interesting:
Comparing pointers to achieve unique sorting order of pointee objects (e.g. sorting locks addresses before performing a double lock):
tty_ldisc_lock_pair_timeout in drivers/tty/tty_ldisc.c pipe_double_lock in fs/pipe.c unix_state_double_lock in net/unix/af_unix.c lock_two_nondirectories in fs/inode.c mutex_lock_double in kernel/events/core.c
ep_cmp_ffd in fs/eventpoll.cfsnotify_compare_groups fs/notify/mark.c
Nothing needs to be done here, since the tags embedded into pointers don't change, so the sorting order would still be unique.
Checks that a pointer belongs to some particular allocation:
is_sibling_entry in lib/radix-tree.c object_is_on_stack in include/linux/sched/task_stack.h
Nothing needs to be done here either, since two pointers can only belong to the same allocation if they have the same tag.
Overall, since the kernel boots and works, there are no critical bugs. As for the rest, the traditional kernel testing way (use until fails) is the only one that looks feasible.
Another point here is that tag-based KASAN is available under a separate config option that needs to be deliberately enabled. Even though it might be used in a "near-production" environment to find bugs that are not found during fuzzing or running tests, it is still a debug tool.
====== Benchmarks
The following numbers were collected on Odroid C2 board. Both generic and tag-based KASAN were used in inline instrumentation mode.
Boot time [1]:
* ~1.7 sec for clean kernel
* ~5.0 sec for generic KASAN
* ~5.0 sec for tag-based KASAN
Network performance [2]:
* 8.33 Gbits/sec for clean kernel
* 3.17 Gbits/sec for generic KASAN
* 2.85 Gbits/sec for tag-based KASAN
Slab memory usage after boot [3]:
* ~40 kb for clean kernel
* ~105 kb (~260% overhead) for generic KASAN
* ~47 kb (~20% overhead) for tag-based KASAN
KASAN memory overhead consists of three main parts:
1. Increased slab memory usage due to redzones.
2. Shadow memory (the whole reserved once during boot).
3. Quaratine (grows gradually until some preset limit; the more the limit, the more the chance to detect a use-after-free).
Comparing tag-based vs generic KASAN for each of these points:
1. 20% vs 260% overhead.
2. 1/16th vs 1/8th of physical memory.
3. Tag-based KASAN doesn't require quarantine.
[1] Time before the ext4 driver is initialized.
[2] Measured as iperf -s & iperf -c 127.0.0.1 -t 30.
[3] Measured as cat /proc/meminfo | grep Slab.
====== Some notes
A few notes:
1. The patchset can be found here:
https://github.com/xairy/kasan-prototype/tree/khwasan
2. Building requires a recent Clang version (7.0.0 or later).
3. Stack instrumentation is not supported yet and will be added later.
====== Changes
Changes in v7:
- Rebased onto 7876320f (4.19-rc4).
- Renamed KHWASAN to software tag-based KASAN (see the top of the cover letter for details).
- Explicitly called tag-based KASAN a debug tool.
- Reused kasan_init_slab_obj() callback to preassign tags to caches without constructors, remove khwasan_preset_sl(u/a)b_tag().
- Moved move obj_to_index to include/linux/slab_def.h from mm/slab.c.
- Moved cache->s_mem untagging to alloc_slabmgmt() for SLAB.
- Fixed check_memory_region() to correctly handle user memory accesses and size == 0 case.
- Merged __no_sanitize_hwaddress into __no_sanitize_address.
- Defined KASAN_SET_TAG and KASAN_RESET_TAG macros for non KASAN builds to avoid duplication of __kimg_to_phys, _virt_addr_is_linear and page_to_virt macros.
- Fixed and simplified find_first_bad_addr for generic KASAN.
- Use non symbolized example KASAN report in documentation.
- Mention clang version requirements for both KASAN modes in the Kconfig options and in the documentation.
- Various small fixes.
Changes in v6:
- Rebased onto 050cdc6c (4.19-rc1+).
- Added notes regarding patchset testing into the cover letter.
Changes in v5:
- Rebased onto 1ffaddd029 (4.18-rc8).
- Preassign tags for objects from caches with constructors and SLAB_TYPESAFE_BY_RCU caches.
- Fix SLAB allocator support by untagging page->s_mem in kasan_poison_slab().
- Performed dynamic testing to find potential places where pointer tagging might result in bugs [1].
- Clarified and fixed memory usage benchmarks in the cover letter.
- Added a rationale for having KHWASAN to the cover letter.
Changes in v4:
- Fixed SPDX comment style in mm/kasan/kasan.h.
- Fixed mm/kasan/kasan.h changes being included in a wrong patch.
- Swapped "khwasan, arm64: fix up fault handling logic" and "khwasan: add tag related helper functions" patches order.
- Rebased onto 6f0d349d (4.18-rc2+).
Changes in v3:
- Minor documentation fixes.
- Fixed CFLAGS variable name in KASAN makefile.
- Added a "SPDX-License-Identifier: GPL-2.0" line to all source files under mm/kasan.
- Rebased onto 81e97f013 (4.18-rc1+).
Changes in v2:
- Changed kmalloc_large_node_hook to return tagged pointer instead of using an output argument.
- Fix checking whether -fsanitize=hwaddress is supported by the compiler.
- Removed duplication of -fno-builtin for KASAN and KHWASAN.
- Removed {} block for one line for_each_possible_cpu loop.
- Made set_track() static inline as it is used only in common.c.
- Moved optimal_redzone() to common.c.
- Fixed using tagged pointer for shadow calculation in kasan_unpoison_shadow().
- Restored setting cache->align in kasan_cache_create(), which was accidentally lost.
- Simplified __kasan_slab_free(), kasan_alloc_pages() and kasan_kmalloc().
- Removed tagging from kasan_kmalloc_large().
- Added page_kasan_tag_reset() to kasan_poison_slab() and removed !PageSlab() check from page_to_virt.
- Reset pointer tag in _virt_addr_is_linear.
- Set page tag for each page when multiple pages are allocated or freed.
- Added a comment as to why we ignore cma allocated pages.
Changes in v1:
- Rebased onto 4.17-rc4.
- Updated benchmarking stats.
- Documented compiler version requirements, memory usage and slowdown.
- Dropped kvm patches, as clang + arm64 + kvm is completely broken [1].
Changes in RFC v3:
- Renamed CONFIG_KASAN_CLASSIC and CONFIG_KASAN_TAGS to CONFIG_KASAN_GENERIC and CONFIG_KASAN_HW respectively.
- Switch to -fsanitize=kernel-hwaddress instead of -fsanitize=hwaddress.
- Removed unnecessary excessive shadow initialization.
- Removed khwasan_enabled flag (it?s not needed since KHWASAN is initialized before any slab caches are used).
- Split out kasan_report.c and khwasan_report.c from report.c.
- Moved more common KASAN and KHWASAN functions to common.c.
- Added tagging to pagealloc.
- Rebased onto 4.17-rc1.
- Temporarily dropped patch that adds kvm support (arm64 + kvm + clang combo is broken right now [2]).
Changes in RFC v2:
- Removed explicit casts to u8 * for kasan_mem_to_shadow() calls.
- Introduced KASAN_TCR_FLAGS for setting the TCR_TBI1 flag.
- Added a comment regarding the non-atomic RMW sequence in khwasan_random_tag().
- Made all tag related functions accept const void *.
- Untagged pointers in __kimg_to_phys, which is used by virt_to_phys.
- Untagged pointers in show_ptr in fault handling logic.
- Untagged pointers passed to KVM.
- Added two reserved tag values: 0xFF and 0xFE.
- Used the reserved tag 0xFF to disable validity checking (to resolve the issue with pointer tag being lost after page_address + kmap usage).
- Used the reserved tag 0xFE to mark redzones and freed objects.
- Added mnemonics for esr manipulation in KHWASAN brk handler.
- Added a comment about the -recover flag.
- Some minor cleanups and fixes.
- Rebased onto 3215b9d5 (4.16-rc6+).
- Tested on real hardware (Odroid C2 board).
- Added better benchmarks.
[1] https://lkml.org/lkml/2018/7/18/765
[2] https://lkml.org/lkml/2018/4/19/775
Andrey Konovalov (20):
kasan, mm: change hooks signatures
kasan: move common generic and tag-based code to common.c
kasan: rename source files to reflect the new naming scheme
kasan: add CONFIG_KASAN_GENERIC and CONFIG_KASAN_SW_TAGS kasan, arm64: adjust shadow size for tag-based mode
kasan: initialize shadow to 0xff for tag-based mode
kasan, arm64: untag address in __kimg_to_phys and _virt_addr_is_linear
kasan: add tag related helper functions
kasan: preassign tags to objects with ctors or SLAB_TYPESAFE_BY_RCU
mm: move obj_to_index to include/linux/slab_def.h
kasan, arm64: fix up fault handling logic
kasan, arm64: enable top byte ignore for the kernel
kasan, mm: perform untagged pointers comparison in krealloc
kasan: split out generic_report.c from report.c
kasan: add bug reporting routines for tag-based mode
kasan: add hooks implementation for tag-based mode
kasan, arm64: add brk handler for inline instrumentation kasan, mm, arm64: tag non slab memory allocated via pagealloc
kasan: update documentation
kasan: add SPDX-License-Identifier mark to source files
Documentation/dev-tools/kasan.rst | 232 +++++----
arch/arm64/Kconfig | 1 +
arch/arm64/Makefile | 2 +-
arch/arm64/include/asm/brk-imm.h | 2 +
arch/arm64/include/asm/memory.h | 36 +-
arch/arm64/include/asm/pgtable-hwdef.h | 1 +
arch/arm64/kernel/traps.c | 68 ++-
arch/arm64/mm/fault.c | 3 +
arch/arm64/mm/kasan_init.c | 18 +-
arch/arm64/mm/proc.S | 8 +-
include/linux/compiler-clang.h | 5 +-
include/linux/kasan.h | 83 +++-
include/linux/mm.h | 29 ++
include/linux/page-flags-layout.h | 10 +
include/linux/slab_def.h | 13 +
lib/Kconfig.kasan | 87 +++-
mm/cma.c | 11 +
mm/kasan/Makefile | 15 +-
mm/kasan/{kasan.c => common.c} | 659 +++++++++----------------
mm/kasan/generic.c | 344 +++++++++++++
mm/kasan/generic_report.c | 153 ++++++
mm/kasan/{kasan_init.c => init.c} | 1 +
mm/kasan/kasan.h | 83 +++-
mm/kasan/quarantine.c | 1 +
mm/kasan/report.c | 272 +++-------
mm/kasan/tags.c | 161 ++++++
mm/kasan/tags_report.c | 58 +++
mm/page_alloc.c | 1 +
mm/slab.c | 29 +-
mm/slab.h | 2 +-
mm/slab_common.c | 6 +-
mm/slub.c | 41 +-
scripts/Makefile.kasan | 27 +-
33 files changed, 1634 insertions(+), 828 deletions(-)
rename mm/kasan/{kasan.c => common.c} (59%)
create mode 100644 mm/kasan/generic.c
create mode 100644 mm/kasan/generic_report.c
rename mm/kasan/{kasan_init.c => init.c} (99%)
create mode 100644 mm/kasan/tags.c
create mode 100644 mm/kasan/tags_report.cSignature · 1 line
2.19.0.397.gdd90340f6a-goog
PATCH v8 1/20 kasan, mm: change hooks signatures
Tag-based KASAN changes the value of the top byte of pointers returned from the kernel allocation functions (such as kmalloc). This patch updates KASAN hooks signatures and their usage in SLAB and SLUB code to reflect that.
--- include/linux/kasan.h | 43 +++++++++++++++++++++++++++++-------------- mm/kasan/kasan.c | 30 ++++++++++++++++++------------ mm/slab.c | 12 ++++++------ mm/slab.h | 2 +- mm/slab_common.c | 4 ++-- mm/slub.c | 15 +++++++-------- 6 files changed, 63 insertions(+), 43 deletions(-)
include/linux/kasan.h +29 -14
@@ -51,16 +51,16 @@ void kasan_cache_shutdown(struct kmem_cache *cache);void kasan_poison_slab (struct page *page);void kasan_unpoison_object_data (struct kmem_cache *cache,void *object);void kasan_poison_object_data (struct kmem_cache *cache,void *object); -void kasan_init_slab_obj (struct kmem_cache *cache,const void *object); +void *kasan_init_slab_obj (struct kmem_cache *cache,const void *object); -void kasan_kmalloc_large (const void *ptr,size_t size,gfp_t flags); +void *kasan_kmalloc_large (const void *ptr,size_t size,gfp_t flags);void kasan_kfree_large (void *ptr,unsigned long ip);void kasan_poison_kfree (void *ptr,unsigned long ip); -void kasan_kmalloc (struct kmem_cache *s,const void *object,size_t size, +void *kasan_kmalloc (struct kmem_cache *s,const void *object,size_t size,gfp_t flags); -void kasan_krealloc (const void *object,size_t new_size,gfp_t flags); +void *kasan_krealloc (const void *object,size_t new_size,gfp_t flags); -void kasan_slab_alloc (struct kmem_cache *s,void *object,gfp_t flags); +void *kasan_slab_alloc (struct kmem_cache *s,void *object,gfp_t flags);bool kasan_slab_free (struct kmem_cache *s,void *object,unsigned long ip);struct kasan_cache {
@@ -105,19 +105,34 @@ static inline void kasan_unpoison_object_data(struct kmem_cache *cache, void *object) {}static inline void kasan_poison_object_data (struct kmem_cache *cache,void *object) {} -static inline void kasan_init_slab_obj (struct kmem_cache *cache, -const void *object) {} +static inline void *kasan_init_slab_obj (struct kmem_cache *cache, +const void *object) +{ +return ptr; +} -static inline void kasan_kmalloc_large (void *ptr,size_t size,gfp_t flags) {} +static inline void *kasan_kmalloc_large (void *ptr,size_t size,gfp_t flags) +{ +return ptr; +}static inline void kasan_kfree_large (void *ptr,unsigned long ip) {}static inline void kasan_poison_kfree (void *ptr,unsigned long ip) {} -static inline void kasan_kmalloc (struct kmem_cache *s,const void *object, -size_t size,gfp_t flags) {} -static inline void kasan_krealloc (const void *object,size_t new_size, -gfp_t flags) {} +static inline void *kasan_kmalloc (struct kmem_cache *s,const void *object, +size_t size,gfp_t flags) +{ +return (void *)object; +} +static inline void *kasan_krealloc (const void *object,size_t new_size, +gfp_t flags) +{ +return (void *)object; +} -static inline void kasan_slab_alloc (struct kmem_cache *s,void *object, -gfp_t flags) {} +static inline void *kasan_slab_alloc (struct kmem_cache *s,void *object, +gfp_t flags) +{ +return object; +}static inline bool kasan_slab_free (struct kmem_cache *s,void *object,unsigned long ip) {
mm/kasan/kasan.c +18 -12
@@ -474,20 +474,22 @@ struct kasan_free_meta *get_free_info(struct kmem_cache *cache,return (void *)object + cache->kasan_info.free_meta_offset; } -void kasan_init_slab_obj (struct kmem_cache *cache,const void *object) +void *kasan_init_slab_obj (struct kmem_cache *cache,const void *object) {struct kasan_alloc_meta *alloc_info;if (!(cache->flags &SLAB_KASAN )) -return ; +return (void *)object; alloc_info =get_alloc_info (cache, object);__memset (alloc_info,0 ,sizeof (*alloc_info)); + +return (void *)object; } -void kasan_slab_alloc (struct kmem_cache *cache,void *object,gfp_t flags) +void *kasan_slab_alloc (struct kmem_cache *cache,void *object,gfp_t flags) { -kasan_kmalloc (cache, object, cache->object_size, flags); +return kasan_kmalloc (cache, object, cache->object_size, flags); } static bool __kasan_slab_free(struct kmem_cache *cache, void *object,
@@ -528,7 +530,7 @@ bool kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip)return __kasan_slab_free (cache, object, ip, true); } -void kasan_kmalloc (struct kmem_cache *cache,const void *object,size_t size, +void *kasan_kmalloc (struct kmem_cache *cache,const void *object,size_t size,gfp_t flags) {unsigned long redzone_start;
@@ -538,7 +540,7 @@ void kasan_kmalloc(struct kmem_cache *cache, const void *object, size_t size,quarantine_reduce ();if (unlikely (object ==NULL )) -return ; +return NULL ; redzone_start =round_up ((unsigned long )(object + size),KASAN_SHADOW_SCALE_SIZE );
@@ -551,10 +553,12 @@ void kasan_kmalloc(struct kmem_cache *cache, const void *object, size_t size,if (cache->flags &SLAB_KASAN )set_track (&get_alloc_info (cache, object)->alloc_track, flags); + +return (void *)object; }EXPORT_SYMBOL (kasan_kmalloc); -void kasan_kmalloc_large (const void *ptr,size_t size,gfp_t flags) +void *kasan_kmalloc_large (const void *ptr,size_t size,gfp_t flags) {struct page *page;unsigned long redzone_start;
@@ -564,7 +568,7 @@ void kasan_kmalloc_large(const void *ptr, size_t size, gfp_t flags)quarantine_reduce ();if (unlikely (ptr ==NULL )) -return ; +return NULL ; page =virt_to_page (ptr); redzone_start = round_up((unsigned long)(ptr + size),
@@ -574,21 +578,23 @@ void kasan_kmalloc_large(const void *ptr, size_t size, gfp_t flags)kasan_unpoison_shadow (ptr, size);kasan_poison_shadow ((void *)redzone_start, redzone_end - redzone_start,KASAN_PAGE_REDZONE ); + +return (void *)ptr; } -void kasan_krealloc (const void *object,size_t size,gfp_t flags) +void *kasan_krealloc (const void *object,size_t size,gfp_t flags) {struct page *page;if (unlikely (object ==ZERO_SIZE_PTR )) -return ; +return ZERO_SIZE_PTR ; page =virt_to_head_page (object);if (unlikely (!PageSlab (page))) -kasan_kmalloc_large (object, size, flags); +return kasan_kmalloc_large (object, size, flags);else -kasan_kmalloc (page->slab_cache, object, size, flags); +return kasan_kmalloc (page->slab_cache, object, size, flags); }void kasan_poison_kfree (void *ptr,unsigned long ip)
mm/slab.c +6 -6
@@ -3551,7 +3551,7 @@ void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags) {void *ret =slab_alloc (cachep, flags, _RET_IP_); -kasan_slab_alloc (cachep, ret, flags); + ret =kasan_slab_alloc (cachep, ret, flags);trace_kmem_cache_alloc (_RET_IP_, ret, cachep->object_size, cachep->size, flags);
@@ -3617,7 +3617,7 @@ kmem_cache_alloc_trace(struct kmem_cache *cachep, gfp_t flags, size_t size) ret =slab_alloc (cachep, flags, _RET_IP_); -kasan_kmalloc (cachep, ret, size, flags); + ret =kasan_kmalloc (cachep, ret, size, flags);trace_kmalloc (_RET_IP_, ret, size, cachep->size, flags);return ret;
@@ -3641,7 +3641,7 @@ void *kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid) {void *ret =slab_alloc_node (cachep, flags, nodeid, _RET_IP_); -kasan_slab_alloc (cachep, ret, flags); + ret =kasan_slab_alloc (cachep, ret, flags);trace_kmem_cache_alloc_node (_RET_IP_, ret, cachep->object_size, cachep->size, flags, nodeid);
@@ -3660,7 +3660,7 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep, ret =slab_alloc_node (cachep, flags, nodeid, _RET_IP_); -kasan_kmalloc (cachep, ret, size, flags); + ret =kasan_kmalloc (cachep, ret, size, flags);trace_kmalloc_node (_RET_IP_, ret, size, cachep->size, flags, nodeid);
@@ -3679,7 +3679,7 @@ __do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller)if (unlikely (ZERO_OR_NULL_PTR (cachep)))return cachep; ret =kmem_cache_alloc_node_trace (cachep, flags, node, size); -kasan_kmalloc (cachep, ret, size, flags); + ret =kasan_kmalloc (cachep, ret, size, flags);return ret; }
@@ -3715,7 +3715,7 @@ static __always_inline void *__do_kmalloc(size_t size, gfp_t flags,return cachep; ret =slab_alloc (cachep, flags, caller); -kasan_kmalloc (cachep, ret, size, flags); + ret =kasan_kmalloc (cachep, ret, size, flags);trace_kmalloc (caller, ret, size, cachep->size, flags);
mm/slab.h +1 -1
@@ -441,7 +441,7 @@ static inline void slab_post_alloc_hook(struct kmem_cache *s, gfp_t flags,kmemleak_alloc_recursive (object, s->object_size,1 , s->flags, flags); -kasan_slab_alloc (s, object, flags); + p[i] =kasan_slab_alloc (s, object, flags); }if (memcg_kmem_enabled ())
mm/slab_common.c +2 -2
@@ -1183,7 +1183,7 @@ void *kmalloc_order(size_t size, gfp_t flags, unsigned int order) page =alloc_pages (flags, order); ret = page ?page_address (page) :NULL ;kmemleak_alloc (ret, size,1 , flags); -kasan_kmalloc_large (ret, size, flags); + ret =kasan_kmalloc_large (ret, size, flags);return ret; }EXPORT_SYMBOL (kmalloc_order);
@@ -1461,7 +1461,7 @@ static __always_inline void *__do_krealloc(const void *p, size_t new_size, ks =ksize (p);if (ks >= new_size) { -kasan_krealloc ((void *)p, new_size, flags); + p =kasan_krealloc ((void *)p, new_size, flags);return (void *)p; }
mm/slub.c +7 -8
@@ -1334,10 +1334,10 @@ static inline void dec_slabs_node(struct kmem_cache *s, int node,* Hooks for other subsystems that check memory allocations. In a typical * production configuration these hooks all should produce no code at all. */ -static inline void kmalloc_large_node_hook (void *ptr,size_t size,gfp_t flags) +static inline void *kmalloc_large_node_hook (void *ptr,size_t size,gfp_t flags) {kmemleak_alloc (ptr, size,1 , flags); -kasan_kmalloc_large (ptr, size, flags); +return kasan_kmalloc_large (ptr, size, flags); } static __always_inline void kfree_hook(void *x)
@@ -2730,7 +2730,7 @@ void *kmem_cache_alloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) {void *ret =slab_alloc (s, gfpflags, _RET_IP_);trace_kmalloc (_RET_IP_, ret, size, s->size, gfpflags); -kasan_kmalloc (s, ret, size, gfpflags); + ret =kasan_kmalloc (s, ret, size, gfpflags);return ret; }EXPORT_SYMBOL (kmem_cache_alloc_trace);
@@ -2758,7 +2758,7 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *s,trace_kmalloc_node (_RET_IP_, ret, size, s->size, gfpflags, node); -kasan_kmalloc (s, ret, size, gfpflags); + ret =kasan_kmalloc (s, ret, size, gfpflags);return ret; }EXPORT_SYMBOL (kmem_cache_alloc_node_trace);
@@ -3748,7 +3748,7 @@ void *__kmalloc(size_t size, gfp_t flags)trace_kmalloc (_RET_IP_, ret, size, s->size, flags); -kasan_kmalloc (s, ret, size, flags); + ret =kasan_kmalloc (s, ret, size, flags);return ret; }
@@ -3765,8 +3765,7 @@ static void *kmalloc_large_node(size_t size, gfp_t flags, int node)if (page) ptr =page_address (page); -kmalloc_large_node_hook (ptr, size, flags); -return ptr; +return kmalloc_large_node_hook (ptr, size, flags); }void *__kmalloc_node (size_t size,gfp_t flags,int node)
@@ -3793,7 +3792,7 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node)trace_kmalloc_node (_RET_IP_, ret, size, s->size, flags, node); -kasan_kmalloc (s, ret, size, flags); + ret =kasan_kmalloc (s, ret, size, flags);return ret; }
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
kbuild test robot
build botRe: [PATCH v8 01/20] kasan, mm: change hooks signatures
Thank you for the patch! Yet something to improve:
Hi Andrey,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on linus/master]
[also build test ERROR on v4.19-rc4 next-20180919]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Andrey-Konovalov/kasan-add-software-tag-based-mode-for-arm64/20180920-172444
config: x86_64-randconfig-x013-201837 (attached as .config)
compiler: gcc-7 (Debian 7.3.0-1) 7.3.0
reproduce:
# save the attached .config to linux build tree
make ARCH=x86_64 All errors (new ones prefixed by >>):
In file included from include/linux/slab.h:129:0,
from include/linux/crypto.h:24,
from arch/x86/kernel/asm-offsets.c:9:
include/linux/kasan.h: In function 'kasan_init_slab_obj':
>> include/linux/kasan.h:111:9: error: 'ptr' undeclared (first use in this function)
return ptr;
^~~ include/linux/kasan.h:111:9: note: each undeclared identifier is reported only once for each function it appears in make[2]: *** [arch/x86/kernel/asm-offsets.s] Error 1
make[2]: Target '__build' not remade because of errors.
make[1]: *** [prepare0] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [sub-make] Error 2
vim +/ptr +111 include/linux/kasan.h
102
103 static inline void kasan_poison_slab(struct page *page) {}
104 static inline void kasan_unpoison_object_data(struct kmem_cache *cache,
105 void *object) {}
106 static inline void kasan_poison_object_data(struct kmem_cache *cache,
107 void *object) {}
108 static inline void *kasan_init_slab_obj(struct kmem_cache *cache,
109 const void *object)
110 {
> 111 return ptr;
112 }
113
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config.gz
Type: application/gzip
Size: 30217 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20180921/ddb08485/attachment-0001.gz>
Re: [PATCH v8 01/20] kasan, mm: change hooks signatures
Quoted 17 lines · kbuild test robot
Hi Andrey,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on linus/master]
[also build test ERROR on v4.19-rc4 next-20180919]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]url: https://github.com/0day-ci/linux/commits/Andrey-Konovalov/kasan-add-software-tag-based-mode-for-arm64/20180920-172444
config: x86_64-randconfig-x013-201837 (attached as .config)
compiler: gcc-7 (Debian 7.3.0-1) 7.3.0
reproduce:# save the attached .config to linux build tree make ARCH=x86_64All errors (new ones prefixed by >>):
Quoted 7 lines
In file included from
include/linux/slab.h:129:0,from include/linux/crypto.h:24, from arch/x86/kernel/asm-offsets.c:9:
include/linux/kasan.h: In function 'kasan_init_slab_obj':Quoted 1 line
Quoted 1 line
include/linux/kasan.h:111:9: error: 'ptr' undeclared (first use in this function)return ptr; ^~~
Right, will fix in v9.
Quoted 32 lines
include/linux/kasan.h:111:9: note: each undeclared identifier is reported only once for each function it appears in make[2]: *** [arch/x86/kernel/asm-offsets.s] Error 1
make[2]: Target '__build' not remade because of errors.
make[1]: *** [prepare0] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [sub-make] Error 2vim +/ptr +111
include/linux/kasan.h102 103 static inline void kasan_poison_slab(struct page *page) {} 104 static inline void kasan_unpoison_object_data(struct kmem_cache *cache, 105 void *object) {} 106 static inline void kasan_poison_object_data(struct kmem_cache *cache, 107 void *object) {} 108 static inline void *kasan_init_slab_obj(struct kmem_cache *cache, 109 const void *object) 110 {Quoted 1 line
111 return ptr;
112 }
113---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation--
You received this message because you are subscribed to the Google Groups "kasan-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+unsubscribe at googlegroups.com. To post to this group, send email to kasan-dev at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/201809211218.10PA6RGy%25fengguang.wu%40intel.com.
For more options, visit https://groups.google.com/d/optout.
kbuild test robot
build botRe: [PATCH v8 01/20] kasan, mm: change hooks signatures
In reply to Andrey KonovalovThank you for the patch! Yet something to improve:
Hi Andrey,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on linus/master]
[also build test ERROR on v4.19-rc4 next-20180919]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Andrey-Konovalov/kasan-add-software-tag-based-mode-for-arm64/20180920-172444
config: microblaze-mmu_defconfig (attached as .config)
compiler: microblaze-linux-gcc (GCC) 8.1.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=8.1.0 make.cross ARCH=microblaze
All errors (new ones prefixed by >>):
In file included from include/linux/slab.h:129,
from include/linux/irq.h:21,
from include/asm-generic/hardirq.h:13,
from ./arch/microblaze/include/generated/asm/hardirq.h:1, from include/linux/hardirq.h:9,
from include/linux/interrupt.h:11,
from include/linux/kernel_stat.h:9,
from arch/microblaze/kernel/asm-offsets.c:14:
include/linux/kasan.h: In function 'kasan_init_slab_obj':
>> include/linux/kasan.h:111:9: error: 'ptr' undeclared (first use in this function); did you mean 'qstr'?
return ptr;
^~~
qstr include/linux/kasan.h:111:9: note: each undeclared identifier is reported only once for each function it appears in make[2]: *** [arch/microblaze/kernel/asm-offsets.s] Error 1
make[2]: Target '__build' not remade because of errors.
make[1]: *** [prepare0] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [sub-make] Error 2
vim +111 include/linux/kasan.h
102
103 static inline void kasan_poison_slab(struct page *page) {}
104 static inline void kasan_unpoison_object_data(struct kmem_cache *cache,
105 void *object) {}
106 static inline void kasan_poison_object_data(struct kmem_cache *cache,
107 void *object) {}
108 static inline void *kasan_init_slab_obj(struct kmem_cache *cache,
109 const void *object)
110 {
> 111 return ptr;
112 }
113
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config.gz
Type: application/gzip
Size: 12813 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20180921/a18e090f/attachment.gz>
PATCH v8 2/20 kasan: move common generic and tag-based code to common.c
In reply to Andrey KonovalovTag-based KASAN reuses a significant part of the generic KASAN code, so move the common parts to common.c without any functional changes.
--- mm/kasan/Makefile | 5 +- mm/kasan/common.c | 603 ++++++++++++++++++++++++++++++++++++++++++++++ mm/kasan/kasan.c | 570 +------------------------------------------ mm/kasan/kasan.h | 5 + 4 files changed, 614 insertions(+), 569 deletions(-) create mode 100644 mm/kasan/common.c
mm/kasan/Makefile +4 -1
@@ -1,11 +1,14 @@ # SPDX-License-Identifier: GPL-2.0 KASAN_SANITIZE := n +UBSAN_SANITIZE_common.o := n UBSAN_SANITIZE_kasan.o := n KCOV_INSTRUMENT := n CFLAGS_REMOVE_kasan.o = -pg # Function splitter causes unnecessary splits in __asan_load1/__asan_store1 # see: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63533 + +CFLAGS_common.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector) CFLAGS_kasan.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector) -obj-y := kasan.o report.o kasan_init.o quarantine.o +obj-y := common.o kasan.o report.o kasan_init.o quarantine.o
mm/kasan/common.c new file +603 -0
603 lines not on this page · raw message
mm/kasan/kasan.c +2 -568
600 lines not on this page · raw message
mm/kasan/kasan.h +5 -0
@@ -105,6 +105,11 @@ static inline const void *kasan_shadow_to_mem(const void *shadow_addr) << KASAN_SHADOW_SCALE_SHIFT); } +void kasan_poison_shadow (const void *address,size_t size,u8 value); + +void check_memory_region (unsigned long addr,size_t size,bool write, +unsigned long ret_ip); +void kasan_report (unsigned long addr,size_t size,bool is_write,unsigned long ip);void kasan_report_invalid_free (void *object,unsigned long ip);
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
PATCH v8 3/20 kasan: rename source files to reflect the new naming scheme
In reply to Andrey KonovalovWe now have two KASAN modes: generic KASAN and tag-based KASAN. Rename kasan.c to generic.c to reflect that. Also rename kasan_init.c to init.c as it contains initialization code for both KASAN modes.
---
mm/kasan/Makefile | 8 ++++----
mm/kasan/{kasan.c => generic.c} | 0
mm/kasan/{kasan_init.c => init.c} | 0
3 files changed, 4 insertions(+), 4 deletions(-)
rename mm/kasan/{kasan.c => generic.c} (100%)
rename mm/kasan/{kasan_init.c => init.c} (100%)mm/kasan/Makefile +4 -4
@@ -1,14 +1,14 @@ # SPDX-License-Identifier: GPL-2.0 KASAN_SANITIZE := n UBSAN_SANITIZE_common.o := n -UBSAN_SANITIZE_kasan.o := n +UBSAN_SANITIZE_generic.o := n KCOV_INSTRUMENT := n -CFLAGS_REMOVE_kasan.o = -pg +CFLAGS_REMOVE_generic.o = -pg # Function splitter causes unnecessary splits in __asan_load1/__asan_store1 # see: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63533 CFLAGS_common.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector) -CFLAGS_kasan.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector) +CFLAGS_generic.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector) -obj-y := common.o kasan.o report.o kasan_init.o quarantine.o +obj-y := common.o generic.o report.o init.o quarantine.o
mm/kasan/generic.c renamed from mm/kasan/kasan.c +0 -0
mm/kasan/init.c renamed from mm/kasan/kasan_init.c +0 -0
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
PATCH v8 4/20 kasan: add CONFIG_KASAN_GENERIC and CONFIG_KASAN_SW_TAGS
In reply to Andrey KonovalovThis commit splits the current CONFIG_KASAN config option into two:
1. CONFIG_KASAN_GENERIC, that enables the generic KASAN mode (the one that exists now);
2. CONFIG_KASAN_SW_TAGS, that enables the software tag-based KASAN mode.
The name CONFIG_KASAN_SW_TAGS is chosen as in the future we will have another hardware tag-based KASAN mode, that will rely on hardware memory tagging support in arm64.
With CONFIG_KASAN_SW_TAGS enabled, compiler options are changed to instrument kernel files with -fsantize=kernel-hwaddress (except the ones for which KASAN_SANITIZE := n is set).
Both CONFIG_KASAN_GENERIC and CONFIG_KASAN_SW_TAGS support both CONFIG_KASAN_INLINE and CONFIG_KASAN_OUTLINE instrumentation modes.
This commit also adds empty placeholder (for now) implementation of tag-based KASAN specific hooks inserted by the compiler and adjusts common hooks implementation to compile correctly with each of the config options.
--- arch/arm64/Kconfig | 1 + include/linux/compiler-clang.h | 5 +- include/linux/kasan.h | 16 +++++-- lib/Kconfig.kasan | 87 +++++++++++++++++++++++++++------- mm/kasan/Makefile | 6 ++- mm/kasan/generic.c | 2 +- mm/kasan/kasan.h | 3 +- mm/kasan/tags.c | 75 +++++++++++++++++++++++++++++ mm/slub.c | 2 +- scripts/Makefile.kasan | 27 ++++++++++- 10 files changed, 194 insertions(+), 30 deletions(-) create mode 100644 mm/kasan/tags.c
arch/arm64/Kconfig +1 -0
@@ -105,6 +105,7 @@ config ARM64 select HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) + select HAVE_ARCH_KASAN_SW_TAGS if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT
include/linux/compiler-clang.h +3 -2
@@ -17,11 +17,12 @@#define KASAN_ABI_VERSION 5/* emulate gcc's __SANITIZE_ADDRESS__ flag */ -#if __has_feature (address_sanitizer) +#if __has_feature (address_sanitizer) ||__has_feature (hwaddress_sanitizer)#define __SANITIZE_ADDRESS__#endif -#define __no_sanitize_address __attribute__((no_sanitize("address"))) +#define __no_sanitize_address \ + __attribute__((no_sanitize("address", "hwaddress")))/* * Not all versions of clang implement the the type-generic versions
include/linux/kasan.h +12 -4
@@ -45,8 +45,6 @@ void kasan_free_pages(struct page *page, unsigned int order);void kasan_cache_create (struct kmem_cache *cache,unsigned int *size,slab_flags_t *flags); -void kasan_cache_shrink (struct kmem_cache *cache); -void kasan_cache_shutdown (struct kmem_cache *cache);void kasan_poison_slab (struct page *page);void kasan_unpoison_object_data (struct kmem_cache *cache,void *object);
@@ -97,8 +95,6 @@ static inline void kasan_free_pages(struct page *page, unsigned int order) {}static inline void kasan_cache_create (struct kmem_cache *cache,unsigned int *size,slab_flags_t *flags) {} -static inline void kasan_cache_shrink (struct kmem_cache *cache) {} -static inline void kasan_cache_shutdown (struct kmem_cache *cache) {}static inline void kasan_poison_slab (struct page *page) {} static inline void kasan_unpoison_object_data(struct kmem_cache *cache,
@@ -155,4 +151,16 @@ static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }#endif /* CONFIG_KASAN */ +#ifdef CONFIG_KASAN_GENERIC + +void kasan_cache_shrink (struct kmem_cache *cache); +void kasan_cache_shutdown (struct kmem_cache *cache); + +#else /* CONFIG_KASAN_GENERIC */ + +static inline void kasan_cache_shrink (struct kmem_cache *cache) {} +static inline void kasan_cache_shutdown (struct kmem_cache *cache) {} + +#endif /* CONFIG_KASAN_GENERIC */ +#endif /* LINUX_KASAN_H */
lib/Kconfig.kasan +69 -18
120 lines not on this page · raw message
mm/kasan/Makefile +5 -1
@@ -2,6 +2,7 @@ KASAN_SANITIZE := n UBSAN_SANITIZE_common.o := n UBSAN_SANITIZE_generic.o := n +UBSAN_SANITIZE_tags.o := n KCOV_INSTRUMENT := n CFLAGS_REMOVE_generic.o = -pg
@@ -10,5 +11,8 @@ CFLAGS_REMOVE_generic.o = -pg CFLAGS_common.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector) CFLAGS_generic.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector) +CFLAGS_tags.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector) -obj-y := common.o generic.o report.o init.o quarantine.o +obj-$(CONFIG_KASAN) := common.o init.o report.o +obj-$(CONFIG_KASAN_GENERIC) += generic.o quarantine.o +obj-$(CONFIG_KASAN_SW_TAGS) += tags.o
mm/kasan/generic.c +1 -1
@@ -1,5 +1,5 @@/* -* This file contains core KASAN code. +* This file contains core generic KASAN code. * * Copyright (c) 2014 Samsung Electronics Co., Ltd. * Author: Andrey Ryabinin <[email protected]>
mm/kasan/kasan.h +2 -1
@@ -114,7 +114,8 @@ void kasan_report(unsigned long addr, size_t size, bool is_write, unsigned long ip);void kasan_report_invalid_free (void *object,unsigned long ip); -#if defined(CONFIG_SLAB ) || defined(CONFIG_SLUB ) +#if defined(CONFIG_KASAN_GENERIC ) && \ + (defined(CONFIG_SLAB ) || defined(CONFIG_SLUB ))void quarantine_put (struct kasan_free_meta *info,struct kmem_cache *cache);void quarantine_reduce (void );void quarantine_remove_cache (struct kmem_cache *cache);
mm/kasan/tags.c new file +75 -0
75 lines not on this page · raw message
mm/slub.c +1 -1
@@ -2954,7 +2954,7 @@ static __always_inline void slab_free(struct kmem_cache *s, struct page *page,do_slab_free (s, page, head, tail, cnt, addr); } -#ifdef CONFIG_KASAN +#ifdef CONFIG_KASAN_GENERIC void ___cache_free (struct kmem_cache *cache,void *x,unsigned long addr) {do_slab_free (cache,virt_to_head_page (x), x,NULL ,1 , addr);
scripts/Makefile.kasan +25 -2
@@ -1,5 +1,5 @@ # SPDX-License-Identifier: GPL-2.0 -ifdef CONFIG_KASAN +ifdef CONFIG_KASAN_GENERIC ifdef CONFIG_KASAN_INLINE call_threshold := 10000 else
@@ -42,6 +42,29 @@ ifdef CONFIG_KASAN_EXTRA CFLAGS_KASAN += $(call cc-option, -fsanitize-address-use-after-scope) endif -CFLAGS_KASAN_NOSANITIZE := -fno-builtin +endif + +ifdef CONFIG_KASAN_SW_TAGS + +ifdef CONFIG_KASAN_INLINE + instrumentation_flags := -mllvm -hwasan-mapping-offset=$(KASAN_SHADOW_OFFSET) +else + instrumentation_flags := -mllvm -hwasan-instrument-with-calls=1 +endif +CFLAGS_KASAN := -fsanitize=kernel-hwaddress \ + -mllvm -hwasan-instrument-stack=0 \ + $(instrumentation_flags) + +ifeq ($(call cc-option, $(CFLAGS_KASAN) -Werror),) + ifneq ($(CONFIG_COMPILE_TEST),y) + $(warning Cannot use CONFIG_KASAN_SW_TAGS: \ + -fsanitize=hwaddress is not supported by compiler) + endif +endif + +endif + +ifdef CONFIG_KASAN +CFLAGS_KASAN_NOSANITIZE := -fno-builtin endif
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
PATCH v8 5/20 kasan, arm64: adjust shadow size for tag-based mode
In reply to Andrey KonovalovTag-based KASAN uses 1 shadow byte for 16 bytes of kernel memory, so it requires 1/16th of the kernel virtual address space for the shadow memory.
This commit sets KASAN_SHADOW_SCALE_SHIFT to 4 when the tag-based KASAN mode is enabled.
--- arch/arm64/Makefile | 2 +- arch/arm64/include/asm/memory.h | 13 +++++++++---- 2 files changed, 10 insertions(+), 5 deletions(-)
arch/arm64/Makefile +1 -1
@@ -94,7 +94,7 @@ endif # KASAN_SHADOW_OFFSET = VA_START + (1 << (VA_BITS - KASAN_SHADOW_SCALE_SHIFT)) # - (1 << (64 - KASAN_SHADOW_SCALE_SHIFT)) # in 32-bit arithmetic -KASAN_SHADOW_SCALE_SHIFT := 3 +KASAN_SHADOW_SCALE_SHIFT := $(if $(CONFIG_KASAN_SW_TAGS), 4, 3) KASAN_SHADOW_OFFSET := $(shell printf "0x%08x00000000\n" $$(( \ (0xffffffff & (-1 << ($(CONFIG_ARM64_VA_BITS) - 32))) \ + (1 << ($(CONFIG_ARM64_VA_BITS) - 32 - $(KASAN_SHADOW_SCALE_SHIFT))) \
arch/arm64/include/asm/memory.h +9 -4
@@ -74,12 +74,17 @@#define KERNEL_END _end/* -* KASAN requires 1/8th of the kernel virtual address space for the shadow -* region. KASAN can bloat the stack significantly, so double the (minimum) -* stack size when KASAN is in use. +* Generic and tag-based KASAN require 1/8th and 1/16th of the kernel virtual +* address space for the shadow region respectively. They can bloat the stack +* significantly, so double the (minimum) stack size when they are in use. */ -#ifdef CONFIG_KASAN +#ifdef CONFIG_KASAN_GENERIC #define KASAN_SHADOW_SCALE_SHIFT 3 +#endif +#ifdef CONFIG_KASAN_SW_TAGS +#define KASAN_SHADOW_SCALE_SHIFT 4 +#endif +#ifdef CONFIG_KASAN #define KASAN_SHADOW_SIZE (UL(1) << (VA_BITS - KASAN_SHADOW_SCALE_SHIFT))#define KASAN_THREAD_SHIFT 1#else
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
PATCH v8 6/20 kasan: initialize shadow to 0xff for tag-based mode
In reply to Andrey KonovalovA tag-based KASAN shadow memory cell contains a memory tag, that corresponds to the tag in the top byte of the pointer, that points to that memory. The native top byte value of kernel pointers is 0xff, so with tag-based KASAN we need to initialize shadow memory to 0xff.
--- arch/arm64/mm/kasan_init.c | 16 ++++++++++++++-- include/linux/kasan.h | 8 ++++++++ mm/kasan/common.c | 3 ++- 3 files changed, 24 insertions(+), 3 deletions(-)
arch/arm64/mm/kasan_init.c +14 -2
@@ -44,6 +44,15 @@ static phys_addr_t __init kasan_alloc_zeroed_page(int node)return __pa (p); } +static phys_addr_t __init kasan_alloc_raw_page (int node) +{ +void *p =memblock_virt_alloc_try_nid_raw (PAGE_SIZE ,PAGE_SIZE , +__pa (MAX_DMA_ADDRESS ), +MEMBLOCK_ALLOC_ACCESSIBLE , + node); +return __pa (p); +} +static pte_t *__init kasan_pte_offset (pmd_t *pmdp,unsigned long addr,int node,bool early) {
@@ -89,7 +98,9 @@ static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr,do {phys_addr_t page_phys = early ?__pa_symbol (kasan_zero_page) - :kasan_alloc_zeroed_page (node); + :kasan_alloc_raw_page (node); +if (!early) +memset (__va (page_phys),KASAN_SHADOW_INIT ,PAGE_SIZE ); next = addr +PAGE_SIZE ;set_pte (ptep,pfn_pte (__phys_to_pfn (page_phys),PAGE_KERNEL )); }while (ptep++, addr = next, addr != end &&pte_none (READ_ONCE (*ptep)));
@@ -139,6 +150,7 @@ asmlinkage void __init kasan_early_init(void) KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT)));BUILD_BUG_ON (!IS_ALIGNED (KASAN_SHADOW_START ,PGDIR_SIZE ));BUILD_BUG_ON (!IS_ALIGNED (KASAN_SHADOW_END ,PGDIR_SIZE )); +kasan_pgd_populate (KASAN_SHADOW_START ,KASAN_SHADOW_END ,NUMA_NO_NODE , true); }
@@ -235,7 +247,7 @@ void __init kasan_init(void)set_pte (&kasan_zero_pte[i],pfn_pte (sym_to_pfn (kasan_zero_page),PAGE_KERNEL_RO )); -memset (kasan_zero_page,0 ,PAGE_SIZE ); +memset (kasan_zero_page,KASAN_SHADOW_INIT ,PAGE_SIZE );cpu_replace_ttbr1 (lm_alias (swapper_pg_dir));/* At this point kasan is fully initialized. Enable error messages */
include/linux/kasan.h +8 -0
@@ -153,6 +153,8 @@ static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }#ifdef CONFIG_KASAN_GENERIC +#define KASAN_SHADOW_INIT 0 +void kasan_cache_shrink (struct kmem_cache *cache);void kasan_cache_shutdown (struct kmem_cache *cache);
@@ -163,4 +165,10 @@ static inline void kasan_cache_shutdown(struct kmem_cache *cache) {}#endif /* CONFIG_KASAN_GENERIC */ +#ifdef CONFIG_KASAN_SW_TAGS + +#define KASAN_SHADOW_INIT 0xFF + +#endif /* CONFIG_KASAN_SW_TAGS */ +#endif /* LINUX_KASAN_H */
mm/kasan/common.c +2 -1
@@ -473,11 +473,12 @@ int kasan_module_alloc(void *addr, size_t size) ret =__vmalloc_node_range (shadow_size,1 , shadow_start, shadow_start + shadow_size, -GFP_KERNEL | __GFP_ZERO, +GFP_KERNEL ,PAGE_KERNEL ,VM_NO_GUARD ,NUMA_NO_NODE ,__builtin_return_address (0 ));if (ret) { +__memset (ret,KASAN_SHADOW_INIT , shadow_size);find_vm_area (addr)->flags |=VM_KASAN ;kmemleak_ignore (ret);return 0 ;
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
PATCH v8 7/20 kasan, arm64: untag address in __kimg_to_phys and _virt_addr_is_linear
In reply to Andrey Konovalov__kimg_to_phys (which is used by virt_to_phys) and _virt_addr_is_linear (which is used by virt_addr_valid) assume that the top byte of the address is 0xff, which isn't always the case with tag-based KASAN.
This patch resets the tag in those macros.
--- arch/arm64/include/asm/memory.h | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
arch/arm64/include/asm/memory.h +12 -2
@@ -92,6 +92,15 @@#define KASAN_THREAD_SHIFT 0#endif +#ifdef CONFIG_KASAN_SW_TAGS +#define KASAN_TAG_SHIFTED (tag) ((unsigned long)(tag) << 56) +#define KASAN_SET_TAG (addr, tag) (((addr) & ~KASAN_TAG_SHIFTED(0xff)) | \ + KASAN_TAG_SHIFTED(tag)) +#define KASAN_RESET_TAG (addr) KASAN_SET_TAG(addr, 0xff) +#else +#define KASAN_RESET_TAG (addr) addr +#endif +#define MIN_THREAD_SHIFT (14 + KASAN_THREAD_SHIFT)/*
@@ -232,7 +241,7 @@ static inline unsigned long kaslr_offset(void)#define __is_lm_address (addr) (!!((addr) & BIT(VA_BITS - 1)))#define __lm_to_phys (addr) (((addr) & ~PAGE_OFFSET) + PHYS_OFFSET) -#define __kimg_to_phys (addr) ((addr) - kimage_voffset) +#define __kimg_to_phys (addr) (KASAN_RESET_TAG(addr) - kimage_voffset)#define __virt_to_phys_nodebug (x) ({ \ phys_addr_t __x = (phys_addr_t)(x); \
@@ -308,7 +317,8 @@ static inline void *phys_to_virt(phys_addr_t x)#endif #endif -#define _virt_addr_is_linear (kaddr) (((u64)(kaddr)) >= PAGE_OFFSET) +#define _virt_addr_is_linear (kaddr) (KASAN_RESET_TAG((u64)(kaddr)) >= \ + PAGE_OFFSET)#define virt_addr_valid (kaddr) (_virt_addr_is_linear(kaddr) && \ _virt_addr_valid(kaddr))
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
PATCH v8 8/20 kasan: add tag related helper functions
In reply to Andrey KonovalovThis commit adds a few helper functions, that are meant to be used to work with tags embedded in the top byte of kernel pointers: to set, to get or to reset (set to 0xff) the top byte.
--- arch/arm64/mm/kasan_init.c | 2 ++ include/linux/kasan.h | 13 +++++++++ mm/kasan/kasan.h | 55 ++++++++++++++++++++++++++++++++++++++ mm/kasan/tags.c | 37 +++++++++++++++++++++++++ 4 files changed, 107 insertions(+)
arch/arm64/mm/kasan_init.c +2 -0
@@ -250,6 +250,8 @@ void __init kasan_init(void)memset (kasan_zero_page,KASAN_SHADOW_INIT ,PAGE_SIZE );cpu_replace_ttbr1 (lm_alias (swapper_pg_dir)); +kasan_init_tags (); +/* At this point kasan is fully initialized. Enable error messages */ init_task.kasan_depth =0 ;pr_info ("KernelAddressSanitizer initialized\n" );
include/linux/kasan.h +13 -0
@@ -169,6 +169,19 @@ static inline void kasan_cache_shutdown(struct kmem_cache *cache) {}#define KASAN_SHADOW_INIT 0xFF +void kasan_init_tags (void ); + +void *kasan_reset_tag (const void *addr); + +#else /* CONFIG_KASAN_SW_TAGS */ + +static inline void kasan_init_tags (void ) { } + +static inline void *kasan_reset_tag (const void *addr) +{ +return (void *)addr; +} +#endif /* CONFIG_KASAN_SW_TAGS */ #endif /* LINUX_KASAN_H */
mm/kasan/kasan.h +55 -0
67 lines not on this page · raw message
mm/kasan/tags.c +37 -0
@@ -38,6 +38,43 @@#include "kasan.h" #include "../slab.h" +static DEFINE_PER_CPU(u32, prng_state); + +void kasan_init_tags (void ) +{ +int cpu; + +for_each_possible_cpu (cpu) +per_cpu (prng_state, cpu) =get_random_u32 (); +} + +/* +* If a preemption happens between this_cpu_read and this_cpu_write, the only +* side effect is that we'll give a few allocated in different contexts objects +* the same tag. Since tag-based KASAN is meant to be used a probabilistic +* bug-detection debug feature, this doesn?t have significant negative impact. +* +* Ideally the tags use strong randomness to prevent any attempts to predict +* them during explicit exploit attempts. But strong randomness is expensive, +* and we did an intentional trade-off to use a PRNG. This non-atomic RMW +* sequence has in fact positive effect, since interrupts that randomly skew +* PRNG at unpredictable points do only good. +*/ +u8 random_tag (void ) +{ +u32 state =this_cpu_read (prng_state); + + state =1664525 * state +1013904223 ; +this_cpu_write (prng_state, state); + +return (u8)(state % (KASAN_TAG_MAX +1 )); +} + +void *kasan_reset_tag (const void *addr) +{ +return reset_tag (addr); +} +void check_memory_region (unsigned long addr,size_t size,bool write,unsigned long ret_ip) {
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
PATCH v8 9/20 kasan: preassign tags to objects with ctors or SLAB_TYPESAFE_BY_RCU
In reply to Andrey KonovalovAn object constructor can initialize pointers within this objects based on the address of the object. Since the object address might be tagged, we need to assign a tag before calling constructor.
The implemented approach is to assign tags to objects with constructors when a slab is allocated and call constructors once as usual. The downside is that such object would always have the same tag when it is reallocated, so we won't catch use-after-frees on it.
Also pressign tags for objects from SLAB_TYPESAFE_BY_RCU caches, since they can be validy accessed after having been freed.
--- mm/slab.c | 2 +- mm/slub.c | 24 ++++++++++++++---------- 2 files changed, 15 insertions(+), 11 deletions(-)
mm/slab.c +1 -1
@@ -2574,7 +2574,7 @@ static void cache_init_objs(struct kmem_cache *cachep,for (i =0 ; i < cachep->num; i++) { objp =index_to_obj (cachep, page, i); -kasan_init_slab_obj (cachep, objp); + objp =kasan_init_slab_obj (cachep, objp);/* constructor could break poison info */ if (DEBUG ==0 && cachep->ctor) {
mm/slub.c +14 -10
@@ -1413,16 +1413,17 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s,#endif } -static void setup_object (struct kmem_cache *s,struct page *page, +static void *setup_object (struct kmem_cache *s,struct page *page,void *object) {setup_object_debug (s, page, object); -kasan_init_slab_obj (s, object); + object =kasan_init_slab_obj (s, object);if (unlikely (s->ctor)) {kasan_unpoison_object_data (s, object); s->ctor (object);kasan_poison_object_data (s, object); } +return object; }/*
@@ -1530,16 +1531,16 @@ static bool shuffle_freelist(struct kmem_cache *s, struct page *page)/* First entry is used as the base of the freelist */ cur =next_freelist_entry (s, page, &pos, start, page_limit, freelist_count); + cur =setup_object (s, page, cur); page->freelist = cur;for (idx =1 ; idx < page->objects; idx++) { -setup_object (s, page, cur); next =next_freelist_entry (s, page, &pos, start, page_limit, freelist_count); + next =setup_object (s, page, next);set_freepointer (s, cur, next); cur = next; } -setup_object (s, page, cur);set_freepointer (s, cur,NULL );return true;
@@ -1561,7 +1562,7 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)struct page *page;struct kmem_cache_order_objects oo = s->oo;gfp_t alloc_gfp; -void *start, *p; +void *start, *p, *next;int idx, order;bool shuffle;
@@ -1613,13 +1614,16 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)if (!shuffle) {for_each_object_idx (p, idx, s, start, page->objects) { -setup_object (s, page, p); -if (likely (idx < page->objects)) -set_freepointer (s, p, p + s->size); -else +if (likely (idx < page->objects)) { + next = p + s->size; + next =setup_object (s, page, next); +set_freepointer (s, p, next); + }else set_freepointer (s, p,NULL ); } - page->freelist =fixup_red_left (s, start); + start =fixup_red_left (s, start); + start =setup_object (s, page, start); + page->freelist = start; } page->inuse = page->objects;
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
Dmitry Vyukov
reviewerRe: [PATCH v8 09/20] kasan: preassign tags to objects with ctors or SLAB_TYPESAFE_BY_RCU
Quoted 93 lines · Andrey Konovalov
An object constructor can initialize pointers within this objects based on the address of the object. Since the object address might be tagged, we need to assign a tag before calling constructor.
The implemented approach is to assign tags to objects with constructors when a slab is allocated and call constructors once as usual. The downside is that such object would always have the same tag when it is reallocated, so we won't catch use-after-frees on it.
Also pressign tags for objects from
SLAB_TYPESAFE_BY_RCUcaches, since they can be validy accessed after having been freed.Signed-off-by Andrey Konovalov <[email protected]>--- mm/slab.c | 2 +- mm/slub.c | 24 ++++++++++++++---------- 2 files changed, 15 insertions(+), 11 deletions(-)mm/slab.c +1
-1@@ -2574,7 +2574,7 @@ static void cache_init_objs(struct kmem_cache *cachep,for (i =0 ; i < cachep->num; i++) { objp =index_to_obj (cachep, page, i); -kasan_init_slab_obj (cachep, objp); + objp =kasan_init_slab_obj (cachep, objp);/* constructor could break poison info */ if (DEBUG ==0 && cachep->ctor) {mm/slub.c +8
-9@@ -1413,16 +1413,17 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s,#endif } -static void setup_object (struct kmem_cache *s,struct page *page, +static void *setup_object (struct kmem_cache *s,struct page *page,void *object) {setup_object_debug (s, page, object); -kasan_init_slab_obj (s, object); + object =kasan_init_slab_obj (s, object);if (unlikely (s->ctor)) {kasan_unpoison_object_data (s, object); s->ctor (object);kasan_poison_object_data (s, object); } +return object; }/* @@ -1530,16 +1531,16 @@ static bool shuffle_freelist(struct kmem_cache *s, struct page *page)/* First entry is used as the base of the freelist */ cur =next_freelist_entry (s, page, &pos, start, page_limit, freelist_count); + cur =setup_object (s, page, cur); page->freelist = cur;for (idx =1 ; idx < page->objects; idx++) { -setup_object (s, page, cur); next =next_freelist_entry (s, page, &pos, start, page_limit, freelist_count); + next =setup_object (s, page, next);set_freepointer (s, cur, next); cur = next; } -setup_object (s, page, cur);set_freepointer (s, cur,NULL );return true;@@ -1561,7 +1562,7 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)struct page *page;struct kmem_cache_order_objects oo = s->oo;gfp_t alloc_gfp; -void *start, *p; +void *start, *p, *next;int idx, order;bool shuffle;@@ -1613,13 +1614,16 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)if (!shuffle) {for_each_object_idx (p, idx, s, start, page->objects) { -setup_object (s, page, p); -if (likely (idx < page->objects)) -set_freepointer (s, p, p + s->size); -else +if (likely (idx < page->objects)) { + next = p + s->size;
mm/slub.c static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) · line 1627
Quoted 10 lines
mm/slub.c +6
-1+ next =setup_object (s, page, next); +set_freepointer (s, p, next); + }else set_freepointer (s, p,NULL ); } - page->freelist =fixup_red_left (s, start); + start =fixup_red_left (s, start); + start =setup_object (s, page, start); + page->freelist = start; }
Just want to double-check that this is correct.
We now do an additional setup_object call after the loop, but we do 1 less in the loop. So total number of calls should be the same, right? However, after the loop we call setup_object for the first object (?), but inside of the loop we skip the call for the last object (?). Am I missing something, or we call ctor twice for the last object and don't call it for the first one?
Quoted 3 lines
mm/slub.c +0
-1page->inuse = page->objects; --2.19.0.397.gdd90340f6a-goog
Re: [PATCH v8 09/20] kasan: preassign tags to objects with ctors or SLAB_TYPESAFE_BY_RCU
Quoted 21 lines · Dmitry Vyukov
Quoted 8 lines · Andrey Konovalov
if (!shuffle) { for_each_object_idx(p, idx, s, start, page->objects) {mm/slub.c +2
-4-setup_object (s, page, p); -if (likely (idx < page->objects)) -set_freepointer (s, p, p + s->size); -else +if (likely (idx < page->objects)) { + next = p + s->size;mm/slub.c static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) · line 1627
Quoted 10 lines
mm/slub.c +6
-1+ next =setup_object (s, page, next); +set_freepointer (s, p, next); + }else set_freepointer (s, p,NULL ); } - page->freelist =fixup_red_left (s, start); + start =fixup_red_left (s, start); + start =setup_object (s, page, start); + page->freelist = start; }
Quoted 7 lines
Just want to double-check that this is correct.
We now do an additionalsetup_objectcall after the loop, but we do 1 less in the loop. So total number of calls should be the same, right? However, after the loop we callsetup_objectfor the first object (?), but inside of the loop we skip the call for the last object (?). Am I missing something, or we call ctor twice for the last object and don't call it for the first one?
Inside the loop we call setup_object for the "next" object. So we start iterating on the first one, but call setup_object for the second. Then the loop moves on to the second one and calls setup_object for the third. And so on. So the loop calls setup_object for every object (including the last one) except for the first one.
The idea is that we want the freelist pointer that is stored in the current object to have a tagged pointer to the next one, so we need to assign a tag to the next object before storing the pointer in the current one.
Dmitry Vyukov
reviewerRe: [PATCH v8 09/20] kasan: preassign tags to objects with ctors or SLAB_TYPESAFE_BY_RCU
Quoted 36 lines · Andrey Konovalov
Quoted 21 lines · Dmitry Vyukov
Quoted 8 lines · Andrey Konovalov
if (!shuffle) { for_each_object_idx(p, idx, s, start, page->objects) {mm/slub.c +2
-4-setup_object (s, page, p); -if (likely (idx < page->objects)) -set_freepointer (s, p, p + s->size); -else +if (likely (idx < page->objects)) { + next = p + s->size;mm/slub.c static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) · line 1627
Quoted 10 lines
mm/slub.c +6
-1+ next =setup_object (s, page, next); +set_freepointer (s, p, next); + }else set_freepointer (s, p,NULL ); } - page->freelist =fixup_red_left (s, start); + start =fixup_red_left (s, start); + start =setup_object (s, page, start); + page->freelist = start; }Quoted 7 lines
Just want to double-check that this is correct.
We now do an additionalsetup_objectcall after the loop, but we do 1 less in the loop. So total number of calls should be the same, right? However, after the loop we callsetup_objectfor the first object (?), but inside of the loop we skip the call for the last object (?). Am I missing something, or we call ctor twice for the last object and don't call it for the first one?Inside the loop we call
setup_objectfor the "next" object. So we start iterating on the first one, but callsetup_objectfor the second. Then the loop moves on to the second one and callssetup_objectfor the third. And so on. So the loop callssetup_objectfor every object (including the last one) except for the first one.
Quoted 4 lines
The idea is that we want the freelist pointer that is stored in the current object to have a tagged pointer to the next one, so we need to assign a tag to the next object before storing the pointer in the current one.
Ah, OK, then false alarm.
PATCH v8 10/20 mm: move obj_to_index to include/linux/slab_def.h
In reply to Andrey KonovalovWhile with SLUB we can actually preassign tags for caches with contructors and store them in pointers in the freelist, SLAB doesn't allow that since the freelist is stored as an array of indexes, so there are no pointers to store the tags.
Instead we compute the tag twice, once when a slab is created before calling the constructor and then again each time when an object is allocated with kmalloc. Tag is computed simply by taking the lowest byte of the index that corresponds to the object. However in kasan_kmalloc we only have access to the objects pointer, so we need a way to find out which index this object corresponds to.
This patch moves obj_to_index from slab.c to include/linux/slab_def.h to be reused by KASAN.
--- include/linux/slab_def.h | 13 +++++++++++++ mm/slab.c | 13 ------------- 2 files changed, 13 insertions(+), 13 deletions(-)
include/linux/slab_def.h +13 -0
@@ -104,4 +104,17 @@ static inline void *nearest_obj(struct kmem_cache *cache, struct page *page, return object; } +/* +* We want to avoid an expensive divide : (offset / cache->size) +* Using the fact that size is a constant for a particular cache, +* we can replace (offset / cache->size) by +* reciprocal_divide(offset, cache->reciprocal_buffer_size) +*/ +static inline unsigned int obj_to_index (const struct kmem_cache *cache, +const struct page *page,void *obj) +{ +u32 offset = (obj - page->s_mem); +return reciprocal_divide (offset, cache->reciprocal_buffer_size); +} +#endif /* _LINUX_SLAB_DEF_H */
mm/slab.c +0 -13
@@ -406,19 +406,6 @@ static inline void *index_to_obj(struct kmem_cache *cache, struct page *page,return page->s_mem + cache->size * idx; } -/* -* We want to avoid an expensive divide : (offset / cache->size) -* Using the fact that size is a constant for a particular cache, -* we can replace (offset / cache->size) by -* reciprocal_divide(offset, cache->reciprocal_buffer_size) -*/ -static inline unsigned int obj_to_index (const struct kmem_cache *cache, -const struct page *page,void *obj) -{ -u32 offset = (obj - page->s_mem); -return reciprocal_divide (offset, cache->reciprocal_buffer_size); -} -#define BOOT_CPUCACHE_ENTRIES 1/* internal cache of cache description objs */ static struct kmem_cache kmem_cache_boot = {
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
PATCH v8 11/20 kasan, arm64: fix up fault handling logic
In reply to Andrey Konovalovshow_pte in arm64 fault handling relies on the fact that the top byte of a kernel pointer is 0xff, which isn't always the case with tag-based KASAN.
This patch resets the top byte in show_pte.
--- arch/arm64/mm/fault.c | 3 +++ 1 file changed, 3 insertions(+)
arch/arm64/mm/fault.c +3 -0
@@ -32,6 +32,7 @@#include <linux/perf_event.h> #include <linux/preempt.h> #include <linux/hugetlb.h> +#include <linux/kasan.h> #include <asm/bug.h> #include <asm/cmpxchg.h>
@@ -134,6 +135,8 @@ void show_pte(unsigned long addr)pgd_t *pgdp;pgd_t pgd; + addr = (unsigned long )kasan_reset_tag ((void *)addr); +if (addr <TASK_SIZE ) {/* TTBR0 */ mm = current->active_mm;
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
PATCH v8 12/20 kasan, arm64: enable top byte ignore for the kernel
In reply to Andrey KonovalovTag-based KASAN uses the Top Byte Ignore feature of arm64 CPUs to store a pointer tag in the top byte of each pointer. This commit enables the TCR_TBI1 bit, which enables Top Byte Ignore for the kernel, when tag-based KASAN is used.
--- arch/arm64/include/asm/pgtable-hwdef.h | 1 + arch/arm64/mm/proc.S | 8 +++++++- 2 files changed, 8 insertions(+), 1 deletion(-)
arch/arm64/include/asm/pgtable-hwdef.h +1 -0
@@ -289,6 +289,7 @@#define TCR_A1 (UL(1) << 22)#define TCR_ASID16 (UL(1) << 36)#define TCR_TBI0 (UL(1) << 37) +#define TCR_TBI1 (UL(1) << 38)#define TCR_HA (UL(1) << 39)#define TCR_HD (UL(1) << 40)#define TCR_NFD1 (UL(1) << 54)
arch/arm64/mm/proc.S +7 -1
@@ -47,6 +47,12 @@ /* PTWs cacheable, inner/outer WBWA */ #define TCR_CACHE_FLAGS TCR_IRGN_WBWA | TCR_ORGN_WBWA +#ifdef CONFIG_KASAN_SW_TAGS +#define TCR_KASAN_FLAGS TCR_TBI1 +#else +#define TCR_KASAN_FLAGS 0 +#endif + #define MAIR(attr, mt) ((attr) << ((mt) * 8)) /*
@@ -440,7 +446,7 @@ ENTRY(__cpu_setup) */ ldr x10, =TCR_TxSZ(VA_BITS) | TCR_CACHE_FLAGS | TCR_SMP_FLAGS | \ TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ - TCR_TBI0 | TCR_A1 + TCR_TBI0 | TCR_A1 | TCR_KASAN_FLAGS tcr_set_idmap_t0sz x10, x9 /*
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
PATCH v8 13/20 kasan, mm: perform untagged pointers comparison in krealloc
In reply to Andrey KonovalovThe krealloc function checks where the same buffer was reused or a new one allocated by comparing kernel pointers. Tag-based KASAN changes memory tag on the krealloc'ed chunk of memory and therefore also changes the pointer tag of the returned pointer. Therefore we need to perform comparison on untagged (with tags reset) pointers to check whether it's the same memory region or not.
--- mm/slab_common.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/slab_common.c +1 -1
@@ -1513,7 +1513,7 @@ void *krealloc(const void *p, size_t new_size, gfp_t flags) } ret =__do_krealloc (p, new_size, flags); -if (ret && p != ret) +if (ret &&kasan_reset_tag (p) !=kasan_reset_tag (ret))kfree (p);return ret;
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
PATCH v8 14/20 kasan: split out generic_report.c from report.c
In reply to Andrey KonovalovThis patch moves generic KASAN specific error reporting routines to generic_report.c without any functional changes, leaving common error reporting code in report.c to be later reused by tag-based KASAN.
--- mm/kasan/Makefile | 4 +- mm/kasan/generic_report.c | 158 +++++++++++++++++++++++++ mm/kasan/kasan.h | 7 ++ mm/kasan/report.c | 234 +++++++++----------------------------- mm/kasan/tags_report.c | 39 +++++++ 5 files changed, 257 insertions(+), 185 deletions(-) create mode 100644 mm/kasan/generic_report.c create mode 100644 mm/kasan/tags_report.c
mm/kasan/Makefile +2 -2
@@ -14,5 +14,5 @@ CFLAGS_generic.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector) CFLAGS_tags.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector) obj-$(CONFIG_KASAN) := common.o init.o report.o -obj-$(CONFIG_KASAN_GENERIC) += generic.o quarantine.o -obj-$(CONFIG_KASAN_SW_TAGS) += tags.o +obj-$(CONFIG_KASAN_GENERIC) += generic.o generic_report.o quarantine.o +obj-$(CONFIG_KASAN_SW_TAGS) += tags.o tags_report.o
mm/kasan/generic_report.c new file +158 -0
158 lines not on this page · raw message
mm/kasan/kasan.h +7 -0
@@ -109,11 +109,18 @@ static inline const void *kasan_shadow_to_mem(const void *shadow_addr) << KASAN_SHADOW_SCALE_SHIFT); } +static inline bool addr_has_shadow (const void *addr) +{ +return (addr >=kasan_shadow_to_mem ((void *)KASAN_SHADOW_START )); +} +void kasan_poison_shadow (const void *address,size_t size,u8 value);void check_memory_region (unsigned long addr,size_t size,bool write,unsigned long ret_ip); +const char *get_bug_type (struct kasan_access_info *info); +void kasan_report (unsigned long addr,size_t size,bool is_write,unsigned long ip);void kasan_report_invalid_free (void *object,unsigned long ip);
mm/kasan/report.c +51 -183
315 lines not on this page · raw message
mm/kasan/tags_report.c new file +39 -0
@@ -0,0 +1,39 @@ +/* +* This file contains tag-based KASAN specific error reporting code. +* +* Copyright (c) 2014 Samsung Electronics Co., Ltd. +* Author: Andrey Ryabinin <[email protected]> +* +* Some code borrowed from https://github.com/xairy/kasan-prototype by +* Andrey Konovalov <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +*/ + +#include <linux/bitops.h> +#include <linux/ftrace.h> +#include <linux/init.h> +#include <linux/kernel.h> +#include <linux/mm.h> +#include <linux/printk.h> +#include <linux/sched.h> +#include <linux/slab.h> +#include <linux/stackdepot.h> +#include <linux/stacktrace.h> +#include <linux/string.h> +#include <linux/types.h> +#include <linux/kasan.h> +#include <linux/module.h> + +#include <asm/sections.h> + +#include "kasan.h" +#include "../slab.h" + +const char *get_bug_type (struct kasan_access_info *info) +{ +return "invalid-access" ; +}
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
PATCH v8 15/20 kasan: add bug reporting routines for tag-based mode
In reply to Andrey KonovalovThis commit adds rountines, that print tag-based KASAN error reports. Those are quite similar to generic KASAN, the difference is:
1. The way tag-based KASAN finds the first bad shadow cell (with a mismatching tag). Tag-based KASAN compares memory tags from the shadow memory to the pointer tag.
2. Tag-based KASAN reports all bugs with the "KASAN: invalid-access" header.
Also simplify generic KASAN find_first_bad_addr.
--- mm/kasan/generic_report.c | 16 ++++------- mm/kasan/kasan.h | 5 ++++ mm/kasan/report.c | 57 +++++++++++++++++++++------------------ mm/kasan/tags_report.c | 18 +++++++++++++ 4 files changed, 59 insertions(+), 37 deletions(-)
mm/kasan/generic_report.c +5 -11
@@ -33,16 +33,13 @@#include "kasan.h" #include "../slab.h" -static const void *find_first_bad_addr (const void *addr,size_t size) +void *find_first_bad_addr (void *addr,size_t size) { -u8 shadow_val = *(u8 *)kasan_mem_to_shadow (addr); -const void *first_bad_addr = addr; +void *p = addr; -while (!shadow_val && first_bad_addr < addr + size) { - first_bad_addr +=KASAN_SHADOW_SCALE_SIZE ; - shadow_val = *(u8 *)kasan_mem_to_shadow (first_bad_addr); - } -return first_bad_addr; +while (p < addr + size && !(*(u8 *)kasan_mem_to_shadow (p))) + p +=KASAN_SHADOW_SCALE_SIZE ; +return p; }static const char *get_shadow_bug_type (struct kasan_access_info *info)
@@ -50,9 +47,6 @@ static const char *get_shadow_bug_type(struct kasan_access_info *info)const char *bug_type ="unknown-crash" ;u8 *shadow_addr; - info->first_bad_addr =find_first_bad_addr (info->access_addr, - info->access_size); - shadow_addr = (u8 *)kasan_mem_to_shadow (info->first_bad_addr);/*
mm/kasan/kasan.h +5 -0
@@ -119,6 +119,7 @@ void kasan_poison_shadow(const void *address, size_t size, u8 value);void check_memory_region (unsigned long addr,size_t size,bool write,unsigned long ret_ip); +void *find_first_bad_addr (void *addr,size_t size);const char *get_bug_type (struct kasan_access_info *info); void kasan_report(unsigned long addr, size_t size,
@@ -139,6 +140,8 @@ static inline void quarantine_remove_cache(struct kmem_cache *cache) { }#ifdef CONFIG_KASAN_SW_TAGS +void print_tags (u8 addr_tag,const void *addr); +#define KASAN_TAG_SHIFT 56#define KASAN_TAG_MASK (0xFFUL << KASAN_TAG_SHIFT)
@@ -166,6 +169,8 @@ static inline void *reset_tag(const void *addr)#else /* CONFIG_KASAN_SW_TAGS */ +static inline void print_tags (u8 addr_tag,const void *addr) { } +static inline u8 random_tag (void ) {return 0 ;
mm/kasan/report.c +31 -26
89 lines not on this page · raw message
mm/kasan/tags_report.c +18 -0
@@ -37,3 +37,21 @@ const char *get_bug_type(struct kasan_access_info *info) {return "invalid-access" ; } + +void *find_first_bad_addr (void *addr,size_t size) +{ +u8 tag =get_tag (addr); +void *p =reset_tag (addr); +void *end = p + size; + +while (p < end && tag == *(u8 *)kasan_mem_to_shadow (p)) + p +=KASAN_SHADOW_SCALE_SIZE ; +return p; +} + +void print_tags (u8 addr_tag,const void *addr) +{ +u8 *shadow = (u8 *)kasan_mem_to_shadow (addr); + +pr_err ("Pointer tag: [%02x], memory tag: [%02x]\n" , addr_tag, *shadow); +}
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
PATCH v8 16/20 kasan: add hooks implementation for tag-based mode
In reply to Andrey KonovalovThis commit adds tag-based KASAN specific hooks implementation and adjusts common generic and tag-based KASAN ones.
1. When a new slab cache is created, tag-based KASAN rounds up the size of the objects in this cache to KASAN_SHADOW_SCALE_SIZE (== 16).
2. On each kmalloc tag-based KASAN generates a random tag, sets the shadow memory, that corresponds to this object to this tag, and embeds this tag value into the top byte of the returned pointer.
3. On each kfree tag-based KASAN poisons the shadow memory with a random tag to allow detection of use-after-free bugs.
The rest of the logic of the hook implementation is very much similar to the one provided by generic KASAN. Tag-based KASAN saves allocation and free stack metadata to the slab object the same way generic KASAN does.
--- mm/kasan/common.c | 118 ++++++++++++++++++++++++++++++++++++++-------- mm/kasan/kasan.h | 8 ++++ mm/kasan/tags.c | 48 +++++++++++++++++++ 3 files changed, 155 insertions(+), 19 deletions(-)
mm/kasan/common.c +99 -19
@@ -140,6 +140,13 @@ void kasan_poison_shadow(const void *address, size_t size, u8 value) {void *shadow_start, *shadow_end; +/* +* Perform shadow offset calculation based on untagged address, as +* some of the callers (e.g. kasan_poison_object_data) pass tagged +* addresses to this function. +*/ + address =reset_tag (address); + shadow_start =kasan_mem_to_shadow (address); shadow_end =kasan_mem_to_shadow (address + size);
@@ -148,11 +155,24 @@ void kasan_poison_shadow(const void *address, size_t size, u8 value)void kasan_unpoison_shadow (const void *address,size_t size) { -kasan_poison_shadow (address, size,0 ); +u8 tag =get_tag (address); + +/* +* Perform shadow offset calculation based on untagged address, as +* some of the callers (e.g. kasan_unpoison_object_data) pass tagged +* addresses to this function. +*/ + address =reset_tag (address); + +kasan_poison_shadow (address, size, tag);if (size &KASAN_SHADOW_MASK ) {u8 *shadow = (u8 *)kasan_mem_to_shadow (address + size); - *shadow = size &KASAN_SHADOW_MASK ; + +if (IS_ENABLED (CONFIG_KASAN_SW_TAGS )) + *shadow = tag; +else + *shadow = size &KASAN_SHADOW_MASK ; } }
@@ -200,8 +220,9 @@ void kasan_unpoison_stack_above_sp_to(const void *watermark)void kasan_alloc_pages (struct page *page,unsigned int order) { -if (likely (!PageHighMem (page))) -kasan_unpoison_shadow (page_address (page),PAGE_SIZE << order); +if (unlikely (PageHighMem (page))) +return ; +kasan_unpoison_shadow (page_address (page),PAGE_SIZE << order); }void kasan_free_pages (struct page *page,unsigned int order)
@@ -218,6 +239,9 @@ void kasan_free_pages(struct page *page, unsigned int order)*/ static inline unsigned int optimal_redzone (unsigned int object_size) { +if (IS_ENABLED (CONFIG_KASAN_SW_TAGS )) +return 0 ; +return object_size <=64 -16 ?16 : object_size <= 128 - 32 ? 32 :
@@ -232,6 +256,7 @@ void kasan_cache_create(struct kmem_cache *cache, unsigned int *size, slab_flags_t *flags) {unsigned int orig_size = *size; +unsigned int redzone_size;int redzone_adjust;/* Add alloc meta. */
@@ -239,20 +264,20 @@ void kasan_cache_create(struct kmem_cache *cache, unsigned int *size, *size +=sizeof (struct kasan_alloc_meta );/* Add free meta. */ -if (cache->flags &SLAB_TYPESAFE_BY_RCU || cache->ctor || - cache->object_size <sizeof (struct kasan_free_meta )) { +if (IS_ENABLED (CONFIG_KASAN_GENERIC ) && + (cache->flags &SLAB_TYPESAFE_BY_RCU || cache->ctor || + cache->object_size <sizeof (struct kasan_free_meta ))) { cache->kasan_info.free_meta_offset = *size; *size +=sizeof (struct kasan_free_meta ); } - redzone_adjust =optimal_redzone (cache->object_size) - - (*size - cache->object_size); + redzone_size =optimal_redzone (cache->object_size); + redzone_adjust = redzone_size - (*size - cache->object_size);if (redzone_adjust >0 ) *size += redzone_adjust; *size = min_t(unsigned int, KMALLOC_MAX_SIZE, -max (*size, cache->object_size + -optimal_redzone (cache->object_size))); +max (*size, cache->object_size + redzone_size));/* * If the metadata doesn't fit, don't enable KASAN at all.
@@ -265,6 +290,8 @@ void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,return ; } + cache->align =round_up (cache->align,KASAN_SHADOW_SCALE_SIZE ); + *flags |=SLAB_KASAN ; }
@@ -319,6 +346,28 @@ void *kasan_init_slab_obj(struct kmem_cache *cache, const void *object) alloc_info =get_alloc_info (cache, object);__memset (alloc_info,0 ,sizeof (*alloc_info)); +/* +* Since it's desirable to only call object contructors ones during +* slab allocation, we preassign tags to all such objects. +* Also preassign tags for SLAB_TYPESAFE_BY_RCU slabs to avoid +* use-after-free reports. +* For SLAB allocator we can't preassign tags randomly since the +* freelist is stored as an array of indexes instead of a linked +* list. Assign tags based on objects indexes, so that objects that +* are next to each other get different tags. +*/ +if (IS_ENABLED (CONFIG_KASAN_SW_TAGS ) && + (cache->ctor || cache->flags &SLAB_TYPESAFE_BY_RCU )) { +#ifdef CONFIG_SLAB +struct page *page =virt_to_page (object); +u8 tag = (u8 )obj_to_index (cache, page, (void *)object); +#else +u8 tag =random_tag (); +#endif + + object =set_tag (object, tag); + } +return (void *)object; }
@@ -327,15 +376,30 @@ void *kasan_slab_alloc(struct kmem_cache *cache, void *object, gfp_t flags)return kasan_kmalloc (cache, object, cache->object_size, flags); } +static inline bool shadow_invalid (u8 tag,s8 shadow_byte) +{ +if (IS_ENABLED (CONFIG_KASAN_GENERIC )) +return shadow_byte <0 || + shadow_byte >=KASAN_SHADOW_SCALE_SIZE ; +else +return tag != (u8 )shadow_byte; +} +static bool __kasan_slab_free (struct kmem_cache *cache,void *object,unsigned long ip,bool quarantine) {s8 shadow_byte; +u8 tag; +void *tagged_object;unsigned long rounded_up_size; + tag =get_tag (object); + tagged_object = object; + object =reset_tag (object); +if (unlikely (nearest_obj (cache,virt_to_head_page (object), object) != object)) { -kasan_report_invalid_free (object, ip); +kasan_report_invalid_free (tagged_object, ip);return true; }
@@ -344,20 +408,22 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object,return false; shadow_byte =READ_ONCE (*(s8 *)kasan_mem_to_shadow (object)); -if (shadow_byte <0 || shadow_byte >=KASAN_SHADOW_SCALE_SIZE ) { -kasan_report_invalid_free (object, ip); +if (shadow_invalid (tag, shadow_byte)) { +kasan_report_invalid_free (tagged_object, ip);return true; } rounded_up_size =round_up (cache->object_size,KASAN_SHADOW_SCALE_SIZE );kasan_poison_shadow (object, rounded_up_size,KASAN_KMALLOC_FREE ); -if (!quarantine ||unlikely (!(cache->flags &SLAB_KASAN ))) +if ((IS_ENABLED (CONFIG_KASAN_GENERIC ) && !quarantine) || +unlikely (!(cache->flags &SLAB_KASAN )))return false;set_track (&get_alloc_info (cache, object)->free_track,GFP_NOWAIT );quarantine_put (get_free_info (cache, object), cache); -return true; + +return IS_ENABLED (CONFIG_KASAN_GENERIC ); }bool kasan_slab_free (struct kmem_cache *cache,void *object,unsigned long ip)
@@ -370,6 +436,7 @@ void *kasan_kmalloc(struct kmem_cache *cache, const void *object, size_t size, {unsigned long redzone_start;unsigned long redzone_end; +u8 tag;if (gfpflags_allow_blocking (flags))quarantine_reduce ();
@@ -382,14 +449,27 @@ void *kasan_kmalloc(struct kmem_cache *cache, const void *object, size_t size, redzone_end =round_up ((unsigned long )object + cache->object_size,KASAN_SHADOW_SCALE_SIZE ); -kasan_unpoison_shadow (object, size); +/* See the comment in kasan_init_slab_obj regarding preassigned tags */ +if (IS_ENABLED (CONFIG_KASAN_SW_TAGS ) && + (cache->ctor || cache->flags &SLAB_TYPESAFE_BY_RCU )) { +#ifdef CONFIG_SLAB +struct page *page =virt_to_page (object); + + tag = (u8 )obj_to_index (cache, page, (void *)object); +#else + tag =get_tag (object); +#endif + }else + tag =random_tag (); + +kasan_unpoison_shadow (set_tag (object, tag), size);kasan_poison_shadow ((void *)redzone_start, redzone_end - redzone_start,KASAN_KMALLOC_REDZONE );if (cache->flags &SLAB_KASAN )set_track (&get_alloc_info (cache, object)->alloc_track, flags); -return (void *)object; +return set_tag (object, tag); }EXPORT_SYMBOL (kasan_kmalloc);
@@ -439,7 +519,7 @@ void kasan_poison_kfree(void *ptr, unsigned long ip) page =virt_to_head_page (ptr);if (unlikely (!PageSlab (page))) { -if (ptr !=page_address (page)) { +if (reset_tag (ptr) !=page_address (page)) {kasan_report_invalid_free (ptr, ip);return ; }
@@ -452,7 +532,7 @@ void kasan_poison_kfree(void *ptr, unsigned long ip)void kasan_kfree_large (void *ptr,unsigned long ip) { -if (ptr !=page_address (virt_to_head_page (ptr))) +if (reset_tag (ptr) !=page_address (virt_to_head_page (ptr)))kasan_report_invalid_free (ptr, ip);/* The object will be poisoned by page_alloc. */ }
mm/kasan/kasan.h +8 -0
@@ -12,10 +12,18 @@#define KASAN_TAG_INVALID 0xFE/* inaccessible memory tag */ #define KASAN_TAG_MAX 0xFD/* maximum value for random tags */ +#ifdef CONFIG_KASAN_GENERIC #define KASAN_FREE_PAGE 0xFF/* page was freed */ #define KASAN_PAGE_REDZONE 0xFE/* redzone for kmalloc_large allocations */ #define KASAN_KMALLOC_REDZONE 0xFC/* redzone inside slub object */ #define KASAN_KMALLOC_FREE 0xFB/* object was freed (kmem_cache_free/kfree) */ +#else +#define KASAN_FREE_PAGE KASAN_TAG_INVALID +#define KASAN_PAGE_REDZONE KASAN_TAG_INVALID +#define KASAN_KMALLOC_REDZONE KASAN_TAG_INVALID +#define KASAN_KMALLOC_FREE KASAN_TAG_INVALID +#endif +#define KASAN_GLOBAL_REDZONE 0xFA/* redzone for global variable */ /*
mm/kasan/tags.c +48 -0
78 lines not on this page · raw message
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
Dmitry Vyukov
reviewerRe: [PATCH v8 16/20] kasan: add hooks implementation for tag-based mode
Quoted 136 lines · Andrey Konovalov
This commit adds tag-based KASAN specific hooks implementation and adjusts common generic and tag-based KASAN ones.
1. When a new slab cache is created, tag-based KASAN rounds up the size of the objects in this cache to
KASAN_SHADOW_SCALE_SIZE(== 16).2. On each kmalloc tag-based KASAN generates a random tag, sets the shadow memory, that corresponds to this object to this tag, and embeds this tag value into the top byte of the returned pointer.
3. On each kfree tag-based KASAN poisons the shadow memory with a random tag to allow detection of use-after-free bugs.
The rest of the logic of the hook implementation is very much similar to the one provided by generic KASAN. Tag-based KASAN saves allocation and free stack metadata to the slab object the same way generic KASAN does.
Signed-off-by Andrey Konovalov <[email protected]>--- mm/kasan/common.c | 118 ++++++++++++++++++++++++++++++++++++++-------- mm/kasan/kasan.h | 8 ++++ mm/kasan/tags.c | 48 +++++++++++++++++++ 3 files changed, 155 insertions(+), 19 deletions(-)mm/kasan/common.c +37
-10@@ -140,6 +140,13 @@ void kasan_poison_shadow(const void *address, size_t size, u8 value) {void *shadow_start, *shadow_end; +/* +* Perform shadow offset calculation based on untagged address, as +* some of the callers (e.g. kasan_poison_object_data) pass tagged +* addresses to this function. +*/ + address =reset_tag (address); + shadow_start =kasan_mem_to_shadow (address); shadow_end =kasan_mem_to_shadow (address + size);@@ -148,11 +155,24 @@ void kasan_poison_shadow(const void *address, size_t size, u8 value)void kasan_unpoison_shadow (const void *address,size_t size) { -kasan_poison_shadow (address, size,0 ); +u8 tag =get_tag (address); + +/* +* Perform shadow offset calculation based on untagged address, as +* some of the callers (e.g. kasan_unpoison_object_data) pass tagged +* addresses to this function. +*/ + address =reset_tag (address); + +kasan_poison_shadow (address, size, tag);if (size &KASAN_SHADOW_MASK ) {u8 *shadow = (u8 *)kasan_mem_to_shadow (address + size); - *shadow = size &KASAN_SHADOW_MASK ; + +if (IS_ENABLED (CONFIG_KASAN_SW_TAGS )) + *shadow = tag; +else + *shadow = size &KASAN_SHADOW_MASK ; } }@@ -200,8 +220,9 @@ void kasan_unpoison_stack_above_sp_to(const void *watermark)void kasan_alloc_pages (struct page *page,unsigned int order) { -if (likely (!PageHighMem (page))) -kasan_unpoison_shadow (page_address (page),PAGE_SIZE << order); +if (unlikely (PageHighMem (page))) +return ; +kasan_unpoison_shadow (page_address (page),PAGE_SIZE << order); }void kasan_free_pages (struct page *page,unsigned int order)@@ -218,6 +239,9 @@ void kasan_free_pages(struct page *page, unsigned int order)*/ static inline unsigned int optimal_redzone (unsigned int object_size) { +if (IS_ENABLED (CONFIG_KASAN_SW_TAGS )) +return 0 ; +return object_size <=64 -16 ?16 : object_size <= 128 - 32 ? 32 :@@ -232,6 +256,7 @@ void kasan_cache_create(struct kmem_cache *cache, unsigned int *size, slab_flags_t *flags) {unsigned int orig_size = *size; +unsigned int redzone_size;int redzone_adjust;/* Add alloc meta. */ @@ -239,20 +264,20 @@ void kasan_cache_create(struct kmem_cache *cache, unsigned int *size, *size +=sizeof (struct kasan_alloc_meta );/* Add free meta. */ -if (cache->flags &SLAB_TYPESAFE_BY_RCU || cache->ctor || - cache->object_size <sizeof (struct kasan_free_meta )) { +if (IS_ENABLED (CONFIG_KASAN_GENERIC ) && + (cache->flags &SLAB_TYPESAFE_BY_RCU || cache->ctor || + cache->object_size <sizeof (struct kasan_free_meta ))) { cache->kasan_info.free_meta_offset = *size; *size +=sizeof (struct kasan_free_meta ); } - redzone_adjust =optimal_redzone (cache->object_size) - - (*size - cache->object_size); + redzone_size =optimal_redzone (cache->object_size); + redzone_adjust = redzone_size - (*size - cache->object_size);if (redzone_adjust >0 ) *size += redzone_adjust; *size = min_t(unsigned int, KMALLOC_MAX_SIZE, -max (*size, cache->object_size + -optimal_redzone (cache->object_size))); +max (*size, cache->object_size + redzone_size));/* * If the metadata doesn't fit, don't enable KASAN at all. @@ -265,6 +290,8 @@ void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,return ; } + cache->align =round_up (cache->align,KASAN_SHADOW_SCALE_SIZE ); + *flags |=SLAB_KASAN ; }
mm/kasan/common.c void *kasan_init_slab_obj(struct kmem_cache *cache, const void *object) · line 350
Quoted 5 lines
mm/kasan/common.c +2
-0@@ -319,6 +346,28 @@ void *kasan_init_slab_obj(struct kmem_cache *cache, const void *object) alloc_info =get_alloc_info (cache, object);__memset (alloc_info,0 ,sizeof (*alloc_info)); +/* +* Since it's desirable to only call object contructors ones during
s/ones/once/
mm/kasan/common.c void *kasan_init_slab_obj(struct kmem_cache *cache, const void *object) · line 351
Quoted 1 line
mm/kasan/common.c +1
-0+* slab allocation, we preassign tags to all such objects.
While we are here, it can make sense to mention that we can't repaint objects with ctors after reallocation (even for non-SLAB_TYPESAFE_BY_RCU) because the ctor code can memorize pointer to the object somewhere (e.g. in the object itself). Then if we repaint it, the old memorized pointer will become invalid.
Quoted 5 lines
mm/kasan/common.c +5
-0+* Also preassign tags for SLAB_TYPESAFE_BY_RCU slabs to avoid +* use-after-free reports. +* For SLAB allocator we can't preassign tags randomly since the +* freelist is stored as an array of indexes instead of a linked +* list. Assign tags based on objects indexes, so that objects that
mm/kasan/common.c void *kasan_init_slab_obj(struct kmem_cache *cache, const void *object) · line 366
Quoted 10 lines
mm/kasan/common.c +10
-0+* are next to each other get different tags. +*/ +if (IS_ENABLED (CONFIG_KASAN_SW_TAGS ) && + (cache->ctor || cache->flags &SLAB_TYPESAFE_BY_RCU )) { +#ifdef CONFIG_SLAB +struct page *page =virt_to_page (object); +u8 tag = (u8 )obj_to_index (cache, page, (void *)object); +#else +u8 tag =random_tag (); +#endif
This looks much better now as compared to the 2 additional callbacks in the previous version.
Quoted 79 lines
mm/kasan/common.c +27
-6+ + object =set_tag (object, tag); + } +return (void *)object; }@@ -327,15 +376,30 @@ void *kasan_slab_alloc(struct kmem_cache *cache, void *object, gfp_t flags)return kasan_kmalloc (cache, object, cache->object_size, flags); } +static inline bool shadow_invalid (u8 tag,s8 shadow_byte) +{ +if (IS_ENABLED (CONFIG_KASAN_GENERIC )) +return shadow_byte <0 || + shadow_byte >=KASAN_SHADOW_SCALE_SIZE ; +else +return tag != (u8 )shadow_byte; +} +static bool __kasan_slab_free (struct kmem_cache *cache,void *object,unsigned long ip,bool quarantine) {s8 shadow_byte; +u8 tag; +void *tagged_object;unsigned long rounded_up_size; + tag =get_tag (object); + tagged_object = object; + object =reset_tag (object); +if (unlikely (nearest_obj (cache,virt_to_head_page (object), object) != object)) { -kasan_report_invalid_free (object, ip); +kasan_report_invalid_free (tagged_object, ip);return true; }@@ -344,20 +408,22 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object,return false; shadow_byte =READ_ONCE (*(s8 *)kasan_mem_to_shadow (object)); -if (shadow_byte <0 || shadow_byte >=KASAN_SHADOW_SCALE_SIZE ) { -kasan_report_invalid_free (object, ip); +if (shadow_invalid (tag, shadow_byte)) { +kasan_report_invalid_free (tagged_object, ip);return true; } rounded_up_size =round_up (cache->object_size,KASAN_SHADOW_SCALE_SIZE );kasan_poison_shadow (object, rounded_up_size,KASAN_KMALLOC_FREE ); -if (!quarantine ||unlikely (!(cache->flags &SLAB_KASAN ))) +if ((IS_ENABLED (CONFIG_KASAN_GENERIC ) && !quarantine) || +unlikely (!(cache->flags &SLAB_KASAN )))return false;set_track (&get_alloc_info (cache, object)->free_track,GFP_NOWAIT );quarantine_put (get_free_info (cache, object), cache); -return true; + +return IS_ENABLED (CONFIG_KASAN_GENERIC ); }bool kasan_slab_free (struct kmem_cache *cache,void *object,unsigned long ip)@@ -370,6 +436,7 @@ void *kasan_kmalloc(struct kmem_cache *cache, const void *object, size_t size, {unsigned long redzone_start;unsigned long redzone_end; +u8 tag;if (gfpflags_allow_blocking (flags))quarantine_reduce ();@@ -382,14 +449,27 @@ void *kasan_kmalloc(struct kmem_cache *cache, const void *object, size_t size, redzone_end =round_up ((unsigned long )object + cache->object_size,KASAN_SHADOW_SCALE_SIZE ); -kasan_unpoison_shadow (object, size);
mm/kasan/common.c void *kasan_kmalloc(struct kmem_cache *cache, const void *object, size_t size, · line 461
Quoted 10 lines
mm/kasan/common.c +10
-0+/* See the comment in kasan_init_slab_obj regarding preassigned tags */ +if (IS_ENABLED (CONFIG_KASAN_SW_TAGS ) && + (cache->ctor || cache->flags &SLAB_TYPESAFE_BY_RCU )) { +#ifdef CONFIG_SLAB +struct page *page =virt_to_page (object); + + tag = (u8 )obj_to_index (cache, page, (void *)object); +#else + tag =get_tag (object); +#endif
This kinda _almost_ matches the chunk of code in kasan_init_slab_obj, but not exactly. Wonder if there is some nice way to unify this code?
Maybe something like:
static u8 tag_for_object(struct kmem_cache *cache, const void *object, new bool)
{
if (!IS_ENABLED(CONFIG_KASAN_SW_TAGS) ||
!cache->ctor && !(cache->flags & SLAB_TYPESAFE_BY_RCU))
return random_tag();
#ifdef CONFIG_SLAB
struct page *page = virt_to_page(object);
return (u8)obj_to_index(cache, page, (void *)object);
#else
return new ? random_tag() : get_tag(object);#endif
}
Then we can call this in both places.
As a side effect this will assign tags to pointers during slab initialization even if we don't have ctors, but it should be fine (?).
Quoted 143 lines
mm/kasan/common.c +7
-3+ }else + tag =random_tag (); + +kasan_unpoison_shadow (set_tag (object, tag), size);kasan_poison_shadow ((void *)redzone_start, redzone_end - redzone_start,KASAN_KMALLOC_REDZONE );if (cache->flags &SLAB_KASAN )set_track (&get_alloc_info (cache, object)->alloc_track, flags); -return (void *)object; +return set_tag (object, tag); }EXPORT_SYMBOL (kasan_kmalloc);@@ -439,7 +519,7 @@ void kasan_poison_kfree(void *ptr, unsigned long ip) page =virt_to_head_page (ptr);if (unlikely (!PageSlab (page))) { -if (ptr !=page_address (page)) { +if (reset_tag (ptr) !=page_address (page)) {kasan_report_invalid_free (ptr, ip);return ; }@@ -452,7 +532,7 @@ void kasan_poison_kfree(void *ptr, unsigned long ip)void kasan_kfree_large (void *ptr,unsigned long ip) { -if (ptr !=page_address (virt_to_head_page (ptr))) +if (reset_tag (ptr) !=page_address (virt_to_head_page (ptr)))kasan_report_invalid_free (ptr, ip);/* The object will be poisoned by page_alloc. */ }mm/kasan/kasan.h +8
-0@@ -12,10 +12,18 @@#define KASAN_TAG_INVALID 0xFE/* inaccessible memory tag */ #define KASAN_TAG_MAX 0xFD/* maximum value for random tags */ +#ifdef CONFIG_KASAN_GENERIC #define KASAN_FREE_PAGE 0xFF/* page was freed */ #define KASAN_PAGE_REDZONE 0xFE/* redzone for kmalloc_large allocations */ #define KASAN_KMALLOC_REDZONE 0xFC/* redzone inside slub object */ #define KASAN_KMALLOC_FREE 0xFB/* object was freed (kmem_cache_free/kfree) */ +#else +#define KASAN_FREE_PAGE KASAN_TAG_INVALID +#define KASAN_PAGE_REDZONE KASAN_TAG_INVALID +#define KASAN_KMALLOC_REDZONE KASAN_TAG_INVALID +#define KASAN_KMALLOC_FREE KASAN_TAG_INVALID +#endif +#define KASAN_GLOBAL_REDZONE 0xFA/* redzone for global variable */ /* mm/kasan/tags.c +48
-1@@ -78,15 +78,60 @@ void *kasan_reset_tag(const void *addr)void check_memory_region (unsigned long addr,size_t size,bool write,unsigned long ret_ip) { +u8 tag; +u8 *shadow_first, *shadow_last, *shadow; +void *untagged_addr; + +if (unlikely (size ==0 )) +return ; + + tag =get_tag ((const void *)addr); + +/* +* Ignore accesses for pointers tagged with 0xff (native kernel +* pointer tag) to suppress false positives caused by kmap. +* +* Some kernel code was written to account for archs that don't keep +* high memory mapped all the time, but rather map and unmap particular +* pages when needed. Instead of storing a pointer to the kernel memory, +* this code saves the address of the page structure and offset within +* that page for later use. Those pages are then mapped and unmapped +* with kmap/kunmap when necessary and virt_to_page is used to get the +* virtual address of the page. For arm64 (that keeps the high memory +* mapped all the time), kmap is turned into a page_address call. + +* The issue is that with use of the page_address + virt_to_page +* sequence the top byte value of the original pointer gets lost (gets +* set to KASAN_TAG_KERNEL (0xFF)). +*/ +if (tag ==KASAN_TAG_KERNEL ) +return ; + + untagged_addr =reset_tag ((const void *)addr); +if (unlikely (untagged_addr < +kasan_shadow_to_mem ((void *)KASAN_SHADOW_START ))) { +kasan_report (addr, size, write, ret_ip); +return ; + } + shadow_first =kasan_mem_to_shadow (untagged_addr); + shadow_last =kasan_mem_to_shadow (untagged_addr + size -1 ); +for (shadow = shadow_first; shadow <= shadow_last; shadow++) { +if (*shadow != tag) { +kasan_report (addr, size, write, ret_ip); +return ; + } + } }#define DEFINE_HWASAN_LOAD_STORE (size) \ void __hwasan_load##size##_noabort(unsigned long addr) \ { \ + check_memory_region(addr, size, false, _RET_IP_); \ } \ EXPORT_SYMBOL(__hwasan_load##size##_noabort); \ void __hwasan_store##size##_noabort(unsigned long addr) \ { \ + check_memory_region(addr, size, true, _RET_IP_); \ } \ EXPORT_SYMBOL(__hwasan_store##size##_noabort)@@ -98,15 +143,18 @@ DEFINE_HWASAN_LOAD_STORE(16);void __hwasan_loadN_noabort (unsigned long addr,unsigned long size) { +check_memory_region (addr, size, false, _RET_IP_); }EXPORT_SYMBOL (__hwasan_loadN_noabort);void __hwasan_storeN_noabort (unsigned long addr,unsigned long size) { +check_memory_region (addr, size, true, _RET_IP_); }EXPORT_SYMBOL (__hwasan_storeN_noabort);void __hwasan_tag_memory (unsigned long addr,u8 tag,unsigned long size) { +kasan_poison_shadow ((void *)addr, size, tag); }EXPORT_SYMBOL (__hwasan_tag_memory); --2.19.0.397.gdd90340f6a-goog
Re: [PATCH v8 16/20] kasan: add hooks implementation for tag-based mode
Quoted 6 lines · Dmitry Vyukov
mm/kasan/common.c void *kasan_init_slab_obj(struct kmem_cache *cache, const void *object) · line 350
Quoted 2 lines · Andrey Konovalov
mm/kasan/common.c +2
-0+/* +* Since it's desirable to only call object contructors ones during s/ones/once/
Will fix.
Quoted 8 lines
Quoted 2 lines
+ * slab allocation, we preassign tags to all such objects.
While we are here, it can make sense to mention that we can't repaint objects with ctors after reallocation (even for non-SLAB_TYPESAFE_BY_RCU) because the ctor code can memorize pointer to the object somewhere (e.g. in the object itself). Then if we repaint it, the old memorized pointer will become invalid.
Will mention.
Quoted 27 lines
mm/kasan/common.c void *kasan_kmalloc(struct kmem_cache *cache, const void *object, size_t size, · line 461
Quoted 11 lines
mm/kasan/common.c +10
-1-kasan_unpoison_shadow (object, size); +/* See the comment in kasan_init_slab_obj regarding preassigned tags */ +if (IS_ENABLED (CONFIG_KASAN_SW_TAGS ) && + (cache->ctor || cache->flags &SLAB_TYPESAFE_BY_RCU )) { +#ifdef CONFIG_SLAB +struct page *page =virt_to_page (object); + + tag = (u8 )obj_to_index (cache, page, (void *)object); +#else + tag =get_tag (object); +#endif This kinda _almost_ matches the chunk of code in
kasan_init_slab_obj, but not exactly. Wonder if there is some nice way to unify this code?Maybe something like:
static u8
tag_for_object(struct kmem_cache *cache, const void *object, new bool){ if (!IS_ENABLED(CONFIG_KASAN_SW_TAGS) || !cache->ctor && !(cache->flags & SLAB_TYPESAFE_BY_RCU)) return random_tag(); #ifdef CONFIG_SLAB struct page *page = virt_to_page(object); return (u8)obj_to_index(cache, page, (void *)object); #else return new ? random_tag() : get_tag(object);
Quoted 4 lines
#endif
}Then we can call this in both places.
Will do, however I think it's better to do the CONFIG_KASAN_SW_TAGS check outside this helper function.
Quoted 2 lines
As a side effect this will assign tags to pointers during slab initialization even if we don't have ctors, but it should be fine (?).
We don't have to assign tag in this case, can just leave 0xff.
PATCH v8 17/20 kasan, arm64: add brk handler for inline instrumentation
In reply to Andrey KonovalovTag-based KASAN inline instrumentation mode (which embeds checks of shadow memory into the generated code, instead of inserting a callback) generates a brk instruction when a tag mismatch is detected.
This commit adds a tag-based KASAN specific brk handler, that decodes the immediate value passed to the brk instructions (to extract information about the memory access that triggered the mismatch), reads the register values (x0 contains the guilty address) and reports the bug.
--- arch/arm64/include/asm/brk-imm.h | 2 + arch/arm64/kernel/traps.c | 68 +++++++++++++++++++++++++++++++- include/linux/kasan.h | 3 ++ 3 files changed, 71 insertions(+), 2 deletions(-)
arch/arm64/include/asm/brk-imm.h +2 -0
@@ -16,10 +16,12 @@* 0x400: for dynamic BRK instruction * 0x401: for compile time BRK instruction * 0x800: kernel-mode BUG() and WARN() traps +* 0x9xx: tag-based KASAN trap (allowed values 0x900 - 0x9ff) */ #define FAULT_BRK_IMM 0x100#define KGDB_DYN_DBG_BRK_IMM 0x400#define KGDB_COMPILED_DBG_BRK_IMM 0x401#define BUG_BRK_IMM 0x800 +#define KASAN_BRK_IMM 0x900#endif
arch/arm64/kernel/traps.c +66 -2
105 lines not on this page · raw message
include/linux/kasan.h +3 -0
@@ -173,6 +173,9 @@ void kasan_init_tags(void);void *kasan_reset_tag (const void *addr); +void kasan_report (unsigned long addr,size_t size, +bool is_write,unsigned long ip); +#else /* CONFIG_KASAN_SW_TAGS */ static inline void kasan_init_tags (void ) { }
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
PATCH v8 18/20 kasan, mm, arm64: tag non slab memory allocated via pagealloc
In reply to Andrey KonovalovTag-based KASAN doesn't check memory accesses through pointers tagged with 0xff. When page_address is used to get pointer to memory that corresponds to some page, the tag of the resulting pointer gets set to 0xff, even though the allocated memory might have been tagged differently.
For slab pages it's impossible to recover the correct tag to return from page_address, since the page might contain multiple slab objects tagged with different values, and we can't know in advance which one of them is going to get accessed. For non slab pages however, we can recover the tag in page_address, since the whole page was marked with the same tag.
This patch adds tagging to non slab memory allocated with pagealloc. To set the tag of the pointer returned from page_address, the tag gets stored to page->flags when the memory gets allocated.
--- arch/arm64/include/asm/memory.h | 9 ++++++++- include/linux/mm.h | 29 +++++++++++++++++++++++++++++ include/linux/page-flags-layout.h | 10 ++++++++++ mm/cma.c | 11 +++++++++++ mm/kasan/common.c | 15 +++++++++++++-- mm/page_alloc.c | 1 + mm/slab.c | 2 +- 7 files changed, 73 insertions(+), 4 deletions(-)
arch/arm64/include/asm/memory.h +8 -1
@@ -98,6 +98,7 @@ KASAN_TAG_SHIFTED(tag))#define KASAN_RESET_TAG (addr) KASAN_SET_TAG(addr, 0xff)#else +#define KASAN_SET_TAG (addr, tag) addr#define KASAN_RESET_TAG (addr) addr#endif
@@ -309,7 +310,13 @@ static inline void *phys_to_virt(phys_addr_t x)#define __virt_to_pgoff (kaddr) (((u64)(kaddr) & ~PAGE_OFFSET) / PAGE_SIZE * sizeof(struct page))#define __page_to_voff (kaddr) (((u64)(kaddr) & ~VMEMMAP_START) * PAGE_SIZE / sizeof(struct page)) -#define page_to_virt (page) ((void *)((__page_to_voff(page)) | PAGE_OFFSET)) +#define page_to_virt (page) ({ \ + unsigned long __addr = \ + ((__page_to_voff(page)) | PAGE_OFFSET); \ + __addr = KASAN_SET_TAG(__addr, page_kasan_tag(page)); \ + ((void *)__addr); \ +}) +#define virt_to_page (vaddr) ((struct page *)((__virt_to_pgoff(vaddr)) | VMEMMAP_START))#define _virt_addr_valid (kaddr) pfn_valid((((u64)(kaddr) & ~PAGE_OFFSET) \
include/linux/mm.h +29 -0
@@ -804,6 +804,7 @@ vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf);#define NODES_PGOFF (SECTIONS_PGOFF - NODES_WIDTH)#define ZONES_PGOFF (NODES_PGOFF - ZONES_WIDTH)#define LAST_CPUPID_PGOFF (ZONES_PGOFF - LAST_CPUPID_WIDTH) +#define KASAN_TAG_PGOFF (LAST_CPUPID_PGOFF - KASAN_TAG_WIDTH)/* * Define the bit shifts to access each section. For non-existent
@@ -814,6 +815,7 @@ vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf);#define NODES_PGSHIFT (NODES_PGOFF * (NODES_WIDTH != 0))#define ZONES_PGSHIFT (ZONES_PGOFF * (ZONES_WIDTH != 0))#define LAST_CPUPID_PGSHIFT (LAST_CPUPID_PGOFF * (LAST_CPUPID_WIDTH != 0)) +#define KASAN_TAG_PGSHIFT (KASAN_TAG_PGOFF * (KASAN_TAG_WIDTH != 0))/* NODE:ZONE or SECTION:ZONE is used to ID a zone for the buddy allocator */ #ifdef NODE_NOT_IN_PAGE_FLAGS
@@ -836,6 +838,7 @@ vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf);#define NODES_MASK ((1UL << NODES_WIDTH) - 1)#define SECTIONS_MASK ((1UL << SECTIONS_WIDTH) - 1)#define LAST_CPUPID_MASK ((1UL << LAST_CPUPID_SHIFT) - 1) +#define KASAN_TAG_MASK ((1UL << KASAN_TAG_WIDTH) - 1)#define ZONEID_MASK ((1UL << ZONEID_SHIFT) - 1)static inline enum zone_type page_zonenum (const struct page *page)
@@ -1081,6 +1084,32 @@ static inline bool cpupid_match_pid(struct task_struct *task, int cpupid) }#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_KASAN_SW_TAGS +static inline u8 page_kasan_tag (const struct page *page) +{ +return (page->flags >>KASAN_TAG_PGSHIFT ) &KASAN_TAG_MASK ; +} + +static inline void page_kasan_tag_set (struct page *page,u8 tag) +{ + page->flags &= ~(KASAN_TAG_MASK <<KASAN_TAG_PGSHIFT ); + page->flags |= (tag &KASAN_TAG_MASK ) <<KASAN_TAG_PGSHIFT ; +} + +static inline void page_kasan_tag_reset (struct page *page) +{ +page_kasan_tag_set (page,0xff ); +} +#else +static inline u8 page_kasan_tag (const struct page *page) +{ +return 0xff ; +} + +static inline void page_kasan_tag_set (struct page *page,u8 tag) { } +static inline void page_kasan_tag_reset (struct page *page) { } +#endif +static inline struct zone *page_zone (const struct page *page) {return &NODE_DATA (page_to_nid (page))->node_zones[page_zonenum (page)];
include/linux/page-flags-layout.h +10 -0
@@ -82,6 +82,16 @@#define LAST_CPUPID_WIDTH 0#endif +#ifdef CONFIG_KASAN_SW_TAGS +#define KASAN_TAG_WIDTH 8 +#if SECTIONS_WIDTH +NODES_WIDTH +ZONES_WIDTH +LAST_CPUPID_WIDTH +KASAN_TAG_WIDTH \ + >BITS_PER_LONG -NR_PAGEFLAGS +#error "KASAN: not enough bits in page flags for tag" +#endif +#else +#define KASAN_TAG_WIDTH 0 +#endif +/* * We are going to use the flags for the page to node mapping if its in * there. This includes the case where there is no node, so it is implicit.
mm/cma.c +11 -0
@@ -407,6 +407,7 @@ struct page *cma_alloc(struct cma *cma, size_t count, unsigned int align,unsigned long pfn =-1 ;unsigned long start =0 ;unsigned long bitmap_maxno, bitmap_no, bitmap_count; +size_t i;struct page *page =NULL ;int ret = -ENOMEM ;
@@ -466,6 +467,16 @@ struct page *cma_alloc(struct cma *cma, size_t count, unsigned int align,trace_cma_alloc (pfn, page, count, align); +/* +* CMA can allocate multiple page blocks, which results in different +* blocks being marked with different tags. Reset the tags to ignore +* those page blocks. +*/ +if (page) { +for (i =0 ; i < count; i++) +page_kasan_tag_reset (page + i); + } +if (ret && !no_warn) {pr_err ("%s: alloc failed, req-size: %zu pages, ret: %d\n" , __func__, count, ret);
mm/kasan/common.c +13 -2
@@ -220,8 +220,15 @@ void kasan_unpoison_stack_above_sp_to(const void *watermark)void kasan_alloc_pages (struct page *page,unsigned int order) { +u8 tag; +unsigned long i; +if (unlikely (PageHighMem (page)))return ; + + tag =random_tag (); +for (i =0 ; i < (1 << order); i++) +page_kasan_tag_set (page + i, tag);kasan_unpoison_shadow (page_address (page),PAGE_SIZE << order); }
@@ -319,6 +326,10 @@ struct kasan_free_meta *get_free_info(struct kmem_cache *cache,void kasan_poison_slab (struct page *page) { +unsigned long i; + +for (i =0 ; i < (1 <<compound_order (page)); i++) +page_kasan_tag_reset (page + i);kasan_poison_shadow (page_address (page),PAGE_SIZE <<compound_order (page),KASAN_KMALLOC_REDZONE );
@@ -519,7 +530,7 @@ void kasan_poison_kfree(void *ptr, unsigned long ip) page =virt_to_head_page (ptr);if (unlikely (!PageSlab (page))) { -if (reset_tag (ptr) !=page_address (page)) { +if (ptr !=page_address (page)) {kasan_report_invalid_free (ptr, ip);return ; }
@@ -532,7 +543,7 @@ void kasan_poison_kfree(void *ptr, unsigned long ip)void kasan_kfree_large (void *ptr,unsigned long ip) { -if (reset_tag (ptr) !=page_address (virt_to_head_page (ptr))) +if (ptr !=page_address (virt_to_head_page (ptr)))kasan_report_invalid_free (ptr, ip);/* The object will be poisoned by page_alloc. */ }
mm/page_alloc.c +1 -0
@@ -1176,6 +1176,7 @@ static void __meminit __init_single_page(struct page *page, unsigned long pfn,init_page_count (page);page_mapcount_reset (page);page_cpupid_reset_last (page); +page_kasan_tag_reset (page);INIT_LIST_HEAD (&page->lru);#ifdef WANT_PAGE_VIRTUAL
mm/slab.c +1 -1
@@ -2357,7 +2357,7 @@ static void *alloc_slabmgmt(struct kmem_cache *cachep,void *freelist;void *addr =page_address (page); - page->s_mem = addr + colour_off; + page->s_mem =kasan_reset_tag (addr) + colour_off; page->active =0 ;if (OBJFREELIST_SLAB (cachep))
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
PATCH v8 19/20 kasan: update documentation
In reply to Andrey KonovalovThis patch updates KASAN documentation to reflect the addition of the new tag-based mode.
--- Documentation/dev-tools/kasan.rst | 232 ++++++++++++++++++------------ 1 file changed, 138 insertions(+), 94 deletions(-)
Documentation/dev-tools/kasan.rst +138 -94
@@ -4,15 +4,25 @@ The Kernel Address Sanitizer (KASAN) Overview -------- -KernelAddressSANitizer (KASAN) is a dynamic memory error detector. It provides -a fast and comprehensive solution for finding use-after-free and out-of-bounds -bugs. +KernelAddressSANitizer (KASAN) is a dynamic memory error detector designed to +find out-of-bound and use-after-free bugs. KASAN has two modes: generic KASAN +(similar to userspace ASan) and software tag-based KASAN (similar to userspace +HWASan). -KASAN uses compile-time instrumentation for checking every memory access, -therefore you will need a GCC version 4.9.2 or later. GCC 5.0 or later is -required for detection of out-of-bounds accesses to stack or global variables. +KASAN uses compile-time instrumentation to insert validity checks before every +memory access, and therefore requires a compiler version that supports that. -Currently KASAN is supported only for the x86_64 and arm64 architectures. +Generic KASAN is supported in both GCC and Clang. With GCC it requires version +4.9.2 or later for basic support and version 5.0 or later for detection of +out-of-bounds accesses for stack and global variables and for inline +instrumentation mode (see the Usage section). With Clang it requires version +3.7.0 or later and it doesn't support detection of out-of-bounds accesses for +global variables yet. + +Tag-based KASAN is only supported in Clang and requires version 7.0.0 or later. + +Currently generic KASAN is supported for the x86_64, arm64 and xtensa +architectures, and tag-based KASAN is supported only for arm64. Usage -----
@@ -21,12 +31,14 @@ To enable KASAN configure kernel with:: CONFIG_KASAN = y -and choose between CONFIG_KASAN_OUTLINE and CONFIG_KASAN_INLINE. Outline and -inline are compiler instrumentation types. The former produces smaller binary -the latter is 1.1 - 2 times faster. Inline instrumentation requires a GCC -version 5.0 or later. +and choose between CONFIG_KASAN_GENERIC (to enable generic KASAN) and +CONFIG_KASAN_SW_TAGS (to enable software tag-based KASAN). -KASAN works with both SLUB and SLAB memory allocators. +You also need to choose between CONFIG_KASAN_OUTLINE and CONFIG_KASAN_INLINE. +Outline and inline are compiler instrumentation types. The former produces +smaller binary while the latter is 1.1 - 2 times faster. + +Both KASAN modes work with both SLUB and SLAB memory allocators. For better bug detection and nicer reporting, enable CONFIG_STACKTRACE. To disable instrumentation for specific files or directories, add a line
@@ -43,85 +55,85 @@ similar to the following to the respective kernel Makefile: Error reports ~~~~~~~~~~~~~ -A typical out of bounds access report looks like this:: +A typical out-of-bounds access generic KASAN report looks like this:: ================================================================== - BUG: AddressSanitizer: out of bounds access in kmalloc_oob_right+0x65/0x75 [test_kasan] at addr ffff8800693bc5d3 - Write of size 1 by task modprobe/1689 - ============================================================================= - BUG kmalloc-128 (Not tainted): kasan error - ----------------------------------------------------------------------------- - - Disabling lock debugging due to kernel taint - INFO: Allocated in kmalloc_oob_right+0x3d/0x75 [test_kasan] age=0 cpu=0 pid=1689 - __slab_alloc+0x4b4/0x4f0 - kmem_cache_alloc_trace+0x10b/0x190 - kmalloc_oob_right+0x3d/0x75 [test_kasan] - init_module+0x9/0x47 [test_kasan] - do_one_initcall+0x99/0x200 - load_module+0x2cb3/0x3b20 - SyS_finit_module+0x76/0x80 - system_call_fastpath+0x12/0x17 - INFO: Slab 0xffffea0001a4ef00 objects=17 used=7 fp=0xffff8800693bd728 flags=0x100000000004080 - INFO: Object 0xffff8800693bc558 @offset=1368 fp=0xffff8800693bc720 - - Bytes b4 ffff8800693bc548: 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ - Object ffff8800693bc558: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk - Object ffff8800693bc568: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk - Object ffff8800693bc578: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk - Object ffff8800693bc588: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk - Object ffff8800693bc598: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk - Object ffff8800693bc5a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk - Object ffff8800693bc5b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk - Object ffff8800693bc5c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk. - Redzone ffff8800693bc5d8: cc cc cc cc cc cc cc cc ........ - Padding ffff8800693bc718: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ - CPU: 0 PID: 1689 Comm: modprobe Tainted: G B 3.18.0-rc1-mm1+ #98 - Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014 - ffff8800693bc000 0000000000000000 ffff8800693bc558 ffff88006923bb78 - ffffffff81cc68ae 00000000000000f3 ffff88006d407600 ffff88006923bba8 - ffffffff811fd848 ffff88006d407600 ffffea0001a4ef00 ffff8800693bc558 + BUG: KASAN: slab-out-of-bounds in kmalloc_oob_right+0xa8/0xbc [test_kasan] + Write of size 1 at addr ffff8801f44ec37b by task insmod/2760 + + CPU: 1 PID: 2760 Comm: insmod Not tainted 4.19.0-rc3+ #698 + Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014 Call Trace: - [<ffffffff81cc68ae>] dump_stack+0x46/0x58 - [<ffffffff811fd848>] print_trailer+0xf8/0x160 - [<ffffffffa00026a7>] ? kmem_cache_oob+0xc3/0xc3 [test_kasan] - [<ffffffff811ff0f5>] object_err+0x35/0x40 - [<ffffffffa0002065>] ? kmalloc_oob_right+0x65/0x75 [test_kasan] - [<ffffffff8120b9fa>] kasan_report_error+0x38a/0x3f0 - [<ffffffff8120a79f>] ? kasan_poison_shadow+0x2f/0x40 - [<ffffffff8120b344>] ? kasan_unpoison_shadow+0x14/0x40 - [<ffffffff8120a79f>] ? kasan_poison_shadow+0x2f/0x40 - [<ffffffffa00026a7>] ? kmem_cache_oob+0xc3/0xc3 [test_kasan] - [<ffffffff8120a995>] __asan_store1+0x75/0xb0 - [<ffffffffa0002601>] ? kmem_cache_oob+0x1d/0xc3 [test_kasan] - [<ffffffffa0002065>] ? kmalloc_oob_right+0x65/0x75 [test_kasan] - [<ffffffffa0002065>] kmalloc_oob_right+0x65/0x75 [test_kasan] - [<ffffffffa00026b0>] init_module+0x9/0x47 [test_kasan] - [<ffffffff810002d9>] do_one_initcall+0x99/0x200 - [<ffffffff811e4e5c>] ? __vunmap+0xec/0x160 - [<ffffffff81114f63>] load_module+0x2cb3/0x3b20 - [<ffffffff8110fd70>] ? m_show+0x240/0x240 - [<ffffffff81115f06>] SyS_finit_module+0x76/0x80 - [<ffffffff81cd3129>] system_call_fastpath+0x12/0x17 + dump_stack+0x94/0xd8 + print_address_description+0x73/0x280 + kasan_report+0x144/0x187 + __asan_report_store1_noabort+0x17/0x20 + kmalloc_oob_right+0xa8/0xbc [test_kasan] + kmalloc_tests_init+0x16/0x700 [test_kasan] + do_one_initcall+0xa5/0x3ae + do_init_module+0x1b6/0x547 + load_module+0x75df/0x8070 + __do_sys_init_module+0x1c6/0x200 + __x64_sys_init_module+0x6e/0xb0 + do_syscall_64+0x9f/0x2c0 + entry_SYSCALL_64_after_hwframe+0x44/0xa9 + RIP: 0033:0x7f96443109da + RSP: 002b:00007ffcf0b51b08 EFLAGS: 00000202 ORIG_RAX: 00000000000000af + RAX: ffffffffffffffda RBX: 000055dc3ee521a0 RCX: 00007f96443109da + RDX: 00007f96445cff88 RSI: 0000000000057a50 RDI: 00007f9644992000 + RBP: 000055dc3ee510b0 R08: 0000000000000003 R09: 0000000000000000 + R10: 00007f964430cd0a R11: 0000000000000202 R12: 00007f96445cff88 + R13: 000055dc3ee51090 R14: 0000000000000000 R15: 0000000000000000 + + Allocated by task 2760: + save_stack+0x43/0xd0 + kasan_kmalloc+0xa7/0xd0 + kmem_cache_alloc_trace+0xe1/0x1b0 + kmalloc_oob_right+0x56/0xbc [test_kasan] + kmalloc_tests_init+0x16/0x700 [test_kasan] + do_one_initcall+0xa5/0x3ae + do_init_module+0x1b6/0x547 + load_module+0x75df/0x8070 + __do_sys_init_module+0x1c6/0x200 + __x64_sys_init_module+0x6e/0xb0 + do_syscall_64+0x9f/0x2c0 + entry_SYSCALL_64_after_hwframe+0x44/0xa9 + + Freed by task 815: + save_stack+0x43/0xd0 + __kasan_slab_free+0x135/0x190 + kasan_slab_free+0xe/0x10 + kfree+0x93/0x1a0 + umh_complete+0x6a/0xa0 + call_usermodehelper_exec_async+0x4c3/0x640 + ret_from_fork+0x35/0x40 + + The buggy address belongs to the object at ffff8801f44ec300 + which belongs to the cache kmalloc-128 of size 128 + The buggy address is located 123 bytes inside of + 128-byte region [ffff8801f44ec300, ffff8801f44ec380) + The buggy address belongs to the page: + page:ffffea0007d13b00 count:1 mapcount:0 mapping:ffff8801f7001640 index:0x0 + flags: 0x200000000000100(slab) + raw: 0200000000000100 ffffea0007d11dc0 0000001a0000001a ffff8801f7001640 + raw: 0000000000000000 0000000080150015 00000001ffffffff 0000000000000000 + page dumped because: kasan: bad access detected + Memory state around the buggy address: - ffff8800693bc300: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc - ffff8800693bc380: fc fc 00 00 00 00 00 00 00 00 00 00 00 00 00 fc - ffff8800693bc400: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc - ffff8800693bc480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc - ffff8800693bc500: fc fc fc fc fc fc fc fc fc fc fc 00 00 00 00 00 - >ffff8800693bc580: 00 00 00 00 00 00 00 00 00 00 03 fc fc fc fc fc - ^ - ffff8800693bc600: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc - ffff8800693bc680: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc - ffff8800693bc700: fc fc fc fc fb fb fb fb fb fb fb fb fb fb fb fb - ffff8800693bc780: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb - ffff8800693bc800: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb + ffff8801f44ec200: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb + ffff8801f44ec280: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc + >ffff8801f44ec300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 + ^ + ffff8801f44ec380: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb + ffff8801f44ec400: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc ================================================================== -The header of the report discribe what kind of bug happened and what kind of -access caused it. It's followed by the description of the accessed slub object -(see 'SLUB Debug output' section in Documentation/vm/slub.rst for details) and -the description of the accessed memory page. +The header of the report provides a short summary of what kind of bug happened +and what kind of access caused it. It's followed by a stack trace of the bad +access, a stack trace of where the accessed memory was allocated (in case bad +access happens on a slab object), and a stack trace of where the object was +freed (in case of a use-after-free bug report). Next comes a description of +the accessed slab object and information about the accessed memory page. In the last section the report shows memory state around the accessed address. Reading this part requires some understanding of how KASAN works.
@@ -138,18 +150,24 @@ inaccessible memory like redzones or freed memory (see mm/kasan/kasan.h). In the report above the arrows point to the shadow byte 03, which means that the accessed address is partially accessible. +For tag-based KASAN this last report section shows the memory tags around the +accessed address (see Implementation details section). + Implementation details ---------------------- +Generic KASAN +~~~~~~~~~~~~~ + From a high level, our approach to memory error detection is similar to that of kmemcheck: use shadow memory to record whether each byte of memory is safe -to access, and use compile-time instrumentation to check shadow memory on each -memory access. +to access, and use compile-time instrumentation to insert checks of shadow +memory on each memory access. -AddressSanitizer dedicates 1/8 of kernel memory to its shadow memory -(e.g. 16TB to cover 128TB on x86_64) and uses direct mapping with a scale and -offset to translate a memory address to its corresponding shadow address. +Generic KASAN dedicates 1/8th of kernel memory to its shadow memory (e.g. 16TB +to cover 128TB on x86_64) and uses direct mapping with a scale and offset to +translate a memory address to its corresponding shadow address. Here is the function which translates an address to its corresponding shadow address::
@@ -162,12 +180,38 @@ address:: where ``KASAN_SHADOW_SCALE_SHIFT = 3``. -Compile-time instrumentation used for checking memory accesses. Compiler inserts -function calls (__asan_load*(addr), __asan_store*(addr)) before each memory -access of size 1, 2, 4, 8 or 16. These functions check whether memory access is -valid or not by checking corresponding shadow memory. +Compile-time instrumentation is used to insert memory access checks. Compiler +inserts function calls (__asan_load*(addr), __asan_store*(addr)) before each +memory access of size 1, 2, 4, 8 or 16. These functions check whether memory +access is valid or not by checking corresponding shadow memory. GCC 5.0 has possibility to perform inline instrumentation. Instead of making function calls GCC directly inserts the code to check the shadow memory. This option significantly enlarges kernel but it gives x1.1-x2 performance boost over outline instrumented kernel. + +Software tag-based KASAN +~~~~~~~~~~~~~~~~~~~~~~~~ + +Tag-based KASAN uses the Top Byte Ignore (TBI) feature of modern arm64 CPUs to +store a pointer tag in the top byte of kernel pointers. Like generic KASAN it +uses shadow memory to store memory tags associated with each 16-byte memory +cell (therefore it dedicates 1/16th of the kernel memory for shadow memory). + +On each memory allocation tag-based KASAN generates a random tag, tags the +allocated memory with this tag, and embeds this tag into the returned pointer. +Software tag-based KASAN uses compile-time instrumentation to insert checks +before each memory access. These checks make sure that tag of the memory that +is being accessed is equal to tag of the pointer that is used to access this +memory. In case of a tag mismatch tag-based KASAN prints a bug report. + +Software tag-based KASAN also has two instrumentation modes (outline, that +emits callbacks to check memory accesses; and inline, that performs the shadow +memory checks inline). With outline instrumentation mode, a bug report is +simply printed from the function that performs the access check. With inline +instrumentation a brk instruction is emitted by the compiler, and a dedicated +brk handler is used to print bug reports. + +A potential expansion of this mode is a hardware tag-based mode, which would +use hardware memory tagging support instead of compiler instrumentation and +manual shadow memory manipulation.
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
PATCH v8 20/20 kasan: add SPDX-License-Identifier mark to source files
In reply to Andrey KonovalovThis patch adds a "SPDX-License-Identifier: GPL-2.0" mark to all source files under mm/kasan.
--- mm/kasan/common.c | 1 + mm/kasan/generic.c | 1 + mm/kasan/generic_report.c | 1 + mm/kasan/init.c | 1 + mm/kasan/quarantine.c | 1 + mm/kasan/report.c | 1 + mm/kasan/tags.c | 1 + mm/kasan/tags_report.c | 1 + 8 files changed, 8 insertions(+)
mm/kasan/common.c +1 -0
@@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * This file contains common generic and tag-based KASAN code. *
mm/kasan/generic.c +1 -0
@@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * This file contains core generic KASAN code. *
mm/kasan/generic_report.c +1 -0
@@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * This file contains generic KASAN specific error reporting code. *
mm/kasan/init.c +1 -0
@@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * This file contains some kasan initialization code. *
mm/kasan/quarantine.c +1 -0
@@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * KASAN quarantine. *
mm/kasan/report.c +1 -0
@@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * This file contains common generic and tag-based KASAN error reporting code. *
mm/kasan/tags.c +1 -0
@@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * This file contains core tag-based KASAN code. *
mm/kasan/tags_report.c +1 -0
@@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * This file contains tag-based KASAN specific error reporting code. *
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
Accidentally marked the series as v8 instead of v7, I hope that is not an issue.
Quoted 3 lines · Andrey Konovalov
This patchset adds a new software tag-based mode to KASAN [1]. (Initially this mode was called KHWASAN, but it got renamed,
see the naming rationale at the end of this section).
Andrey, I remember you giving a Reviewed-by to this patchset, but since there are somewhat significant naming related changes in the last version, could you take another look? Thanks!