KFENCE: A low-overhead sampling-based memory safety error detector
Subsystems (13)
Marco Elver
maintainerv2 cover KFENCE: A low-overhead sampling-based memory safety error detector
This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors. This series enables KFENCE for the x86 and arm64 architectures, and adds KFENCE hooks to the SLAB and SLUB allocators.
KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.
KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error.
Guarded allocations are set up based on a sample interval (can be set via kfence.sample_interval). After expiration of the sample interval, the next allocation through the main allocator (SLAB or SLUB) returns a guarded allocation from the KFENCE object pool. At this point, the timer is reset, and the next allocation is set up after the expiration of the interval.
To enable/disable a KFENCE allocation through the main allocator's fast-path without overhead, KFENCE relies on static branches via the static keys infrastructure. The static branch is toggled to redirect the allocation to KFENCE.
The KFENCE memory pool is of fixed size, and if the pool is exhausted no further KFENCE allocations occur. The default config is conservative with only 255 objects, resulting in a pool size of 2 MiB (with 4 KiB pages).
We have verified by running synthetic benchmarks (sysbench I/O, hackbench) that a kernel with KFENCE is performance-neutral compared to a non-KFENCE baseline kernel.
KFENCE is inspired by GWP-ASan [1], a userspace tool with similar properties. The name "KFENCE" is a homage to the Electric Fence Malloc Debugger [2].
For more details, see Documentation/dev-tools/kfence.rst added in the series -- also viewable here:
https://raw.githubusercontent.com/google/kasan/kfence/Documentation/dev-tools/kfence.rst
[1] http://llvm.org/docs/GwpAsan.html
[2] https://linux.die.net/man/3/efence
v2:
* Various comment/documentation changes (see details in patches).
* Various smaller fixes (see details in patches).
* Change all reports to reference the kfence object, "kfence-#nn".
* Skip allocation/free internals stack trace.
* Rework KMEMLEAK compatibility patch.
RFC/v1: https://lkml.kernel.org/r/[email protected]
Alexander Potapenko (6):
mm: add Kernel Electric-Fence infrastructure x86, kfence: enable KFENCE for x86
mm, kfence: insert KFENCE hooks for SLAB
mm, kfence: insert KFENCE hooks for SLUB kfence, kasan: make KFENCE compatible with KASAN kfence, kmemleak: make KFENCE compatible with KMEMLEAK
Marco Elver (4):
arm64, kfence: enable KFENCE for ARM64 kfence, lockdep: make KFENCE compatible with lockdep kfence, Documentation: add KFENCE documentation
kfence: add test suite
Documentation/dev-tools/index.rst | 1 + Documentation/dev-tools/kfence.rst | 291 +++++++++++ MAINTAINERS | 11 + arch/arm64/Kconfig | 1 + arch/arm64/include/asm/kfence.h | 39 ++ arch/arm64/mm/fault.c | 4 + arch/x86/Kconfig | 2 + arch/x86/include/asm/kfence.h | 60 +++ arch/x86/mm/fault.c | 4 + include/linux/kfence.h | 174 +++++++ init/main.c | 2 + kernel/locking/lockdep.c | 8 + lib/Kconfig.debug | 1 + lib/Kconfig.kfence | 78 +++ mm/Makefile | 1 + mm/kasan/common.c | 7 + mm/kfence/Makefile | 6 + mm/kfence/core.c | 733 +++++++++++++++++++++++++++ mm/kfence/kfence.h | 102 ++++ mm/kfence/kfence_test.c | 777 +++++++++++++++++++++++++++++ mm/kfence/report.c | 219 ++++++++ mm/kmemleak.c | 6 + mm/slab.c | 46 +- mm/slab_common.c | 6 +- mm/slub.c | 72 ++- 25 files changed, 2619 insertions(+), 32 deletions(-) create mode 100644 Documentation/dev-tools/kfence.rst create mode 100644 arch/arm64/include/asm/kfence.h create mode 100644 arch/x86/include/asm/kfence.h create mode 100644 include/linux/kfence.h create mode 100644 lib/Kconfig.kfence create mode 100644 mm/kfence/Makefile create mode 100644 mm/kfence/core.c create mode 100644 mm/kfence/kfence.h create mode 100644 mm/kfence/kfence_test.c create mode 100644 mm/kfence/report.c
Signature · 1 line
2.28.0.618.gf4bc123cb7-goog
Marco Elver
maintainerPATCH v2 1/10 mm: add Kernel Electric-Fence infrastructure
From: Alexander Potapenko <[email protected]>
This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors.
KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.
KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error.
Guarded allocations are set up based on a sample interval (can be set via kfence.sample_interval). After expiration of the sample interval, a guarded allocation from the KFENCE object pool is returned to the main allocator (SLAB or SLUB). At this point, the timer is reset, and the next allocation is set up after the expiration of the interval.
To enable/disable a KFENCE allocation through the main allocator's fast-path without overhead, KFENCE relies on static branches via the static keys infrastructure. The static branch is toggled to redirect the allocation to KFENCE. To date, we have verified by running synthetic benchmarks (sysbench I/O workloads) that a kernel compiled with KFENCE is performance-neutral compared to the non-KFENCE baseline.
For more details, see Documentation/dev-tools/kfence.rst (added later in the series).
---
v2:
* Add missing __printf attribute to seq_con_printf, and fix new warning. [reported by kernel test robot <[email protected]>]
* Fix up some comments [reported by Jonathan Cameron].
* Remove 2 cases of redundant stack variable initialization [reported by Jonathan Cameron].
* Fix printf format [reported by kernel test robot <[email protected]>].
* Print (in kfence-#nn) after address, to more clearly establish link between first and second stacktrace [reported by Andrey Konovalov].
* Make choice between KASAN and KFENCE clearer in Kconfig help text [suggested by Dave Hansen].
* Document CONFIG_KFENCE_SAMPLE_INTERVAL=0.
* Shorten memory corruption report line length.
* Make /sys/module/kfence/parameters/sample_interval root-writable for all builds (to enable debugging, automatic dynamic tweaking).
* Reports by Dmitry Vyukov:
* Do not store negative size for right-located objects
* Only cache-align addresses of right-located objects.
* Run toggle_allocation_gate() after KFENCE is enabled.
* Add empty line between allocation and free stacks.
* Add comment about SLAB_TYPESAFE_BY_RCU.
* Also skip internals for allocation/free stacks.
* s/KFENCE_FAULT_INJECTION/KFENCE_STRESS_TEST_FAULTS/ as FAULT_INJECTION is already overloaded in different contexts.
* Parenthesis for macro variable.
* Lower max of KFENCE_NUM_OBJECTS config variable.
--- MAINTAINERS | 11 + include/linux/kfence.h | 174 ++++++++++ init/main.c | 2 + lib/Kconfig.debug | 1 + lib/Kconfig.kfence | 65 ++++ mm/Makefile | 1 + mm/kfence/Makefile | 3 + mm/kfence/core.c | 733 +++++++++++++++++++++++++++++++++++++++++ mm/kfence/kfence.h | 102 ++++++ mm/kfence/report.c | 219 ++++++++++++ 10 files changed, 1311 insertions(+) create mode 100644 include/linux/kfence.h create mode 100644 lib/Kconfig.kfence create mode 100644 mm/kfence/Makefile create mode 100644 mm/kfence/core.c create mode 100644 mm/kfence/kfence.h create mode 100644 mm/kfence/report.c
MAINTAINERS +11 -0
@@ -9673,6 +9673,17 @@ F: include/linux/keyctl.h F: include/uapi/linux/keyctl.h F: security/keys/ +KFENCE +M: Alexander Potapenko <[email protected]> +M: Marco Elver <[email protected]> +R: Dmitry Vyukov <[email protected]> +L: [email protected] +S: Maintained +F: Documentation/dev-tools/kfence.rst +F: include/linux/kfence.h +F: lib/Kconfig.kfence +F: mm/kfence/ + KFIFO M: Stefani Seibold <[email protected]> S: Maintained
include/linux/kfence.h new file +174 -0
174 lines not on this page · raw message
init/main.c +2 -0
@@ -39,6 +39,7 @@#include <linux/security.h> #include <linux/smp.h> #include <linux/profile.h> +#include <linux/kfence.h> #include <linux/rcupdate.h> #include <linux/moduleparam.h> #include <linux/kallsyms.h>
@@ -942,6 +943,7 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)hrtimers_init ();softirq_init ();timekeeping_init (); +kfence_init ();/* * For best initial stack canary entropy, prepare it after:
lib/Kconfig.debug +1 -0
@@ -880,6 +880,7 @@ config DEBUG_STACKOVERFLOW If in doubt, say "N". source "lib/Kconfig.kasan" +source "lib/Kconfig.kfence" endmenu # "Memory Debugging"
lib/Kconfig.kfence new file +65 -0
65 lines not on this page · raw message
mm/Makefile +1 -0
@@ -81,6 +81,7 @@ obj-$(CONFIG_PAGE_POISONING) += page_poison.o obj-$(CONFIG_SLAB) += slab.o obj-$(CONFIG_SLUB) += slub.o obj-$(CONFIG_KASAN) += kasan/ +obj-$(CONFIG_KFENCE) += kfence/ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_MEMORY_HOTPLUG) += memory_hotplug.o obj-$(CONFIG_MEMTEST) += memtest.o
mm/kfence/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-$(CONFIG_KFENCE) := core.o report.o
mm/kfence/core.c new file +733 -0
733 lines not on this page · raw message
mm/kfence/kfence.h new file +102 -0
102 lines not on this page · raw message
mm/kfence/report.c new file +219 -0
219 lines not on this page · raw message
Signature · 1 line
2.28.0.618.gf4bc123cb7-goog
Re: [PATCH v2 01/10] mm: add Kernel Electric-Fence infrastructure
Tue, 15 Sep 2020 15:20:37 +0200
Quoted 14 lines
From: Alexander Potapenko <[email protected]>
This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors.
KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.
Quoted 6 lines
KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error.
To help understand the magic of KFENCE, a simple diagram looks needed to illustrate the relations between obj and guard pages, something like the below asiic chart.
|-----------------|-----------------------------------|------------------| | left guard page | the page containing KFENCE object | right guard page | |-----------------|-----------------------------------|------------------|
Marco Elver
maintainerRe: [PATCH v2 01/10] mm: add Kernel Electric-Fence infrastructure
Quoted 24 lines · Hillf Danton
Tue, 15 Sep 2020 15:20:37 +0200
Quoted 14 lines
From: Alexander Potapenko <[email protected]>
This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors.
KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.
Quoted 6 lines
KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error.
Quoted 8 lines
To help understand the magic of KFENCE, a simple diagram looks needed to illustrate the relations between obj and guard pages, something like the below asiic chart.
|-----------------|-----------------------------------|------------------| | left guard page | the page containing KFENCE object | right guard page | |-----------------|-----------------------------------|------------------|
Would the one we have in Documentation be what you're after?
https://lore.kernel.org/linux-arm-kernel/[email protected]/
(at "The following figure illustrates the page layout::")
Let us know if you'd like that copied into the commit message.
Thanks,
-- Marco
Re: [PATCH v2 01/10] mm: add Kernel Electric-Fence infrastructure
Quoted 27 lines · On Thu, 17 Sep 2020 15:36:38 +0200 Marco Elver wrote:
Quoted 26 lines · Hillf Danton
Tue, 15 Sep 2020 15:20:37 +0200
Quoted 14 lines
From: Alexander Potapenko <[email protected]>
This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors.
KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.
Quoted 6 lines
KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error.
To help understand the magic of KFENCE, a simple diagram looks needed to illustrate the relations between obj and guard pages, something like the below asiic chart.
Quoted 6 lines
Quoted 4 lines
|-----------------|-----------------------------------|------------------| | left guard page | the page containing KFENCE object | right guard page | |-----------------|-----------------------------------|------------------|Would the one we have in Documentation be what you're after?
Yes, that's great.
Quoted 5 lines
https://lore.kernel.org/linux-arm-kernel/[email protected]/
(at "The following figure illustrates the page layout::")Let us know if you'd like that copied into the commit message.
Then the copy can help more readers.
Thanks
Hillf
Marco Elver
maintainerPATCH v2 2/10 x86, kfence: enable KFENCE for x86
In reply to Marco ElverFrom: Alexander Potapenko <[email protected]>
Add architecture specific implementation details for KFENCE and enable KFENCE for the x86 architecture. In particular, this implements the required interface in <asm/kfence.h> for setting up the pool and providing helper functions for protecting and unprotecting pages.
For x86, we need to ensure that the pool uses 4K pages, which is done using the set_memory_4k() helper function.
--- arch/x86/Kconfig | 2 ++ arch/x86/include/asm/kfence.h | 60 +++++++++++++++++++++++++++++++++++ arch/x86/mm/fault.c | 4 +++ 3 files changed, 66 insertions(+) create mode 100644 arch/x86/include/asm/kfence.h
arch/x86/Kconfig +2 -0
@@ -144,6 +144,8 @@ config X86 select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 select HAVE_ARCH_KASAN_VMALLOC if X86_64 + select HAVE_ARCH_KFENCE + select HAVE_ARCH_KFENCE_STATIC_POOL select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS if MMU select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT
arch/x86/include/asm/kfence.h new file +60 -0
60 lines not on this page · raw message
arch/x86/mm/fault.c +4 -0
@@ -9,6 +9,7 @@#include <linux/kdebug.h> /* oops_begin/end, ... */ #include <linux/extable.h> /* search_exception_tables */ #include <linux/memblock.h> /* max_low_pfn */ +#include <linux/kfence.h> /* kfence_handle_page_fault */ #include <linux/kprobes.h> /* NOKPROBE_SYMBOL, ... */ #include <linux/mmiotrace.h> /* kmmio_handler, ... */ #include <linux/perf_event.h> /* perf_sw_event */
@@ -701,6 +702,9 @@ no_context(struct pt_regs *regs, unsigned long error_code, }#endif +if (kfence_handle_page_fault (address)) +return ; +/* * 32-bit: *
Signature · 1 line
2.28.0.618.gf4bc123cb7-goog
Marco Elver
maintainerPATCH v2 3/10 arm64, kfence: enable KFENCE for ARM64
In reply to Marco ElverAdd architecture specific implementation details for KFENCE and enable KFENCE for the arm64 architecture. In particular, this implements the required interface in <asm/kfence.h>. Currently, the arm64 version does not yet use a statically allocated memory pool, at the cost of a pointer load for each is_kfence_address().
---
For ARM64, we would like to solicit feedback on what the best option is to obtain a constant address for __kfence_pool. One option is to declare a memory range in the memory layout to be dedicated to KFENCE (like is done for KASAN), however, it is unclear if this is the best available option. We would like to avoid touching the memory layout.
--- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/kfence.h | 39 +++++++++++++++++++++++++++++++++ arch/arm64/mm/fault.c | 4 ++++ 3 files changed, 44 insertions(+) create mode 100644 arch/arm64/include/asm/kfence.h
arch/arm64/Kconfig +1 -0
@@ -132,6 +132,7 @@ config ARM64 select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN + select HAVE_ARCH_KFENCE if (!ARM64_16K_PAGES && !ARM64_64K_PAGES) select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT
arch/arm64/include/asm/kfence.h new file +39 -0
@@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_KFENCE_H +#define __ASM_KFENCE_H + +#include <linux/kfence.h> +#include <linux/log2.h> +#include <linux/mm.h> + +#include <asm/cacheflush.h> + +#define KFENCE_SKIP_ARCH_FAULT_HANDLER "el1_sync" + +/* +* FIXME: Support HAVE_ARCH_KFENCE_STATIC_POOL: Use the statically allocated +* __kfence_pool, to avoid the extra pointer load for is_kfence_address(). By +* default, however, we do not have struct pages for static allocations. +*/ + +static inline bool arch_kfence_initialize_pool (void ) +{ +const unsigned int num_pages =ilog2 (roundup_pow_of_two (KFENCE_POOL_SIZE /PAGE_SIZE )); +struct page *pages =alloc_pages (GFP_KERNEL , num_pages); + +if (!pages) +return false; + + __kfence_pool =page_address (pages); +return true; +} + +static inline bool kfence_protect_page (unsigned long addr,bool protect) +{ +set_memory_valid (addr,1 , !protect); + +return true; +} + +#endif /* __ASM_KFENCE_H */
arch/arm64/mm/fault.c +4 -0
@@ -10,6 +10,7 @@#include <linux/acpi.h> #include <linux/bitfield.h> #include <linux/extable.h> +#include <linux/kfence.h> #include <linux/signal.h> #include <linux/mm.h> #include <linux/hardirq.h>
@@ -310,6 +311,9 @@ static void __do_kernel_fault(unsigned long addr, unsigned int esr, "Ignoring spurious kernel translation fault at virtual address %016lx\n", addr)) return; +if (kfence_handle_page_fault (addr)) +return ; +if (is_el1_permission_fault (addr, esr, regs)) {if (esr & ESR_ELx_WNR) msg ="write to read-only memory" ;
Signature · 1 line
2.28.0.618.gf4bc123cb7-goog
Marco Elver
maintainerPATCH v2 4/10 mm, kfence: insert KFENCE hooks for SLAB
In reply to Marco ElverFrom: Alexander Potapenko <[email protected]>
Inserts KFENCE hooks into the SLAB allocator.
We note the addition of the 'orig_size' argument to slab_alloc*() functions, to be able to pass the originally requested size to KFENCE. When KFENCE is disabled, there is no additional overhead, since these functions are __always_inline.
--- mm/slab.c | 46 ++++++++++++++++++++++++++++++++++------------ mm/slab_common.c | 6 +++++- 2 files changed, 39 insertions(+), 13 deletions(-)
mm/slab.c +34 -12
@@ -100,6 +100,7 @@#include <linux/seq_file.h> #include <linux/notifier.h> #include <linux/kallsyms.h> +#include <linux/kfence.h> #include <linux/cpu.h> #include <linux/sysctl.h> #include <linux/module.h>
@@ -3206,7 +3207,7 @@ static void *____cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, } static __always_inline void * -slab_alloc_node (struct kmem_cache *cachep,gfp_t flags,int nodeid, +slab_alloc_node (struct kmem_cache *cachep,gfp_t flags,int nodeid,size_t orig_size,unsigned long caller) {unsigned long save_flags;
@@ -3219,6 +3220,10 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,if (unlikely (!cachep))return NULL ; + ptr =kfence_alloc (cachep, orig_size, flags); +if (unlikely (ptr)) + goto out_hooks; +cache_alloc_debugcheck_before (cachep, flags);local_irq_save (save_flags);
@@ -3251,6 +3256,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,if (unlikely (slab_want_init_on_alloc (flags, cachep)) && ptr)memset (ptr,0 , cachep->object_size); +out_hooks:slab_post_alloc_hook (cachep, objcg, flags,1 , &ptr);return ptr; }
@@ -3288,7 +3294,7 @@ __do_cache_alloc(struct kmem_cache *cachep, gfp_t flags)#endif /* CONFIG_NUMA */ static __always_inline void * -slab_alloc (struct kmem_cache *cachep,gfp_t flags,unsigned long caller) +slab_alloc (struct kmem_cache *cachep,gfp_t flags,size_t orig_size,unsigned long caller) {unsigned long save_flags;void *objp;
@@ -3299,6 +3305,10 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller)if (unlikely (!cachep))return NULL ; + objp =kfence_alloc (cachep, orig_size, flags); +if (unlikely (objp)) + goto leave; +cache_alloc_debugcheck_before (cachep, flags);local_irq_save (save_flags); objp =__do_cache_alloc (cachep, flags);
@@ -3309,6 +3319,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller)if (unlikely (slab_want_init_on_alloc (flags, cachep)) && objp)memset (objp,0 , cachep->object_size); +leave:slab_post_alloc_hook (cachep, objcg, flags,1 , &objp);return objp; }
@@ -3414,6 +3425,11 @@ static void cache_flusharray(struct kmem_cache *cachep, struct array_cache *ac) static __always_inline void __cache_free(struct kmem_cache *cachep, void *objp,unsigned long caller) { +if (kfence_free (objp)) { +kmemleak_free_recursive (objp, cachep->flags); +return ; + } +/* Put the object into the quarantine, don't touch it for now. */ if (kasan_slab_free (cachep, objp, _RET_IP_))return ;
@@ -3479,7 +3495,7 @@ void ___cache_free(struct kmem_cache *cachep, void *objp,*/ void *kmem_cache_alloc (struct kmem_cache *cachep,gfp_t flags) { -void *ret =slab_alloc (cachep, flags, _RET_IP_); +void *ret =slab_alloc (cachep, flags, cachep->object_size, _RET_IP_);trace_kmem_cache_alloc (_RET_IP_, ret, cachep->object_size, cachep->size, flags);
@@ -3512,7 +3528,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,local_irq_disable ();for (i =0 ; i < size; i++) { -void *objp =__do_cache_alloc (s, flags); +void *objp =kfence_alloc (s, s->object_size, flags) ?:__do_cache_alloc (s, flags);if (unlikely (!objp)) goto error;
@@ -3545,7 +3561,7 @@ kmem_cache_alloc_trace(struct kmem_cache *cachep, gfp_t flags, size_t size) {void *ret; - ret =slab_alloc (cachep, flags, _RET_IP_); + ret =slab_alloc (cachep, flags, size, _RET_IP_); ret =kasan_kmalloc (cachep, ret, size, flags); trace_kmalloc(_RET_IP_, ret,
@@ -3571,7 +3587,7 @@ EXPORT_SYMBOL(kmem_cache_alloc_trace);*/ void *kmem_cache_alloc_node (struct kmem_cache *cachep,gfp_t flags,int nodeid) { -void *ret =slab_alloc_node (cachep, flags, nodeid, _RET_IP_); +void *ret =slab_alloc_node (cachep, flags, nodeid, cachep->object_size, _RET_IP_); trace_kmem_cache_alloc_node(_RET_IP_, ret, cachep->object_size, cachep->size,
@@ -3589,7 +3605,7 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep, {void *ret; - ret =slab_alloc_node (cachep, flags, nodeid, _RET_IP_); + ret =slab_alloc_node (cachep, flags, nodeid, size, _RET_IP_); ret =kasan_kmalloc (cachep, ret, size, flags); trace_kmalloc_node(_RET_IP_, ret,
@@ -3650,7 +3666,7 @@ static __always_inline void *__do_kmalloc(size_t size, gfp_t flags, cachep =kmalloc_slab (size, flags);if (unlikely (ZERO_OR_NULL_PTR (cachep)))return cachep; - ret =slab_alloc (cachep, flags, caller); + ret =slab_alloc (cachep, flags, size, caller); ret =kasan_kmalloc (cachep, ret, size, flags); trace_kmalloc(caller, ret,
@@ -4138,18 +4154,24 @@ void __check_heap_object(const void *ptr, unsigned long n, struct page *page, bool to_user) {struct kmem_cache *cachep; -unsigned int objnr; +unsigned int objnr =0 ;unsigned long offset; +bool is_kfence =is_kfence_address (ptr); ptr =kasan_reset_tag (ptr);/* Find and validate object. */ cachep = page->slab_cache; - objnr =obj_to_index (cachep, page, (void *)ptr); -BUG_ON (objnr >= cachep->num); +if (!is_kfence) { + objnr =obj_to_index (cachep, page, (void *)ptr); +BUG_ON (objnr >= cachep->num); + }/* Find offset within object. */ - offset = ptr -index_to_obj (cachep, page, objnr) -obj_offset (cachep); +if (is_kfence_address (ptr)) + offset = ptr -kfence_object_start (ptr); +else + offset = ptr -index_to_obj (cachep, page, objnr) -obj_offset (cachep);/* Allow address range falling entirely within usercopy region. */ if (offset >= cachep->useroffset &&
mm/slab_common.c +5 -1
@@ -12,6 +12,7 @@#include <linux/memory.h> #include <linux/cache.h> #include <linux/compiler.h> +#include <linux/kfence.h> #include <linux/module.h> #include <linux/cpu.h> #include <linux/uaccess.h>
@@ -448,6 +449,9 @@ static int shutdown_cache(struct kmem_cache *s)/* free asan quarantined objects */ kasan_cache_shutdown (s); +if (!kfence_shutdown_cache (s)) +return -EBUSY ; +if (__kmem_cache_shutdown (s) !=0 )return -EBUSY ;
@@ -1171,7 +1175,7 @@ size_t ksize(const void *objp)if (unlikely (ZERO_OR_NULL_PTR (objp)) || !__kasan_check_read (objp,1 ))return 0 ; - size =__ksize (objp); + size =kfence_ksize (objp) ?:__ksize (objp);/* * We assume that ksize callers could use whole allocated area, * so we need to unpoison this area.
Signature · 1 line
2.28.0.618.gf4bc123cb7-goog
Re: [PATCH v2 04/10] mm, kfence: insert KFENCE hooks for SLAB
mm/slab.c static void *____cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, · line 3212
Quoted 8 lines · Marco Elver
mm/slab.c +1
-1@@ -3206,7 +3207,7 @@ static void *____cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, } static __always_inline void * -slab_alloc_node (struct kmem_cache *cachep,gfp_t flags,int nodeid, +slab_alloc_node (struct kmem_cache *cachep,gfp_t flags,int nodeid,size_t orig_size,unsigned long caller) {
The size of the object is available via a field in kmem_cache. And a pointer to the current kmem_cache is already passed to the function. Why is there a need to add an additional parameter?
Alexander Potapenko
maintainerRe: [PATCH v2 04/10] mm, kfence: insert KFENCE hooks for SLAB
Quoted 9 lines
mm/slab.c static void *____cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, · line 3212
Quoted 5 lines
static
__always_inlinevoid *mm/slab.c +1
-1-slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, +slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, size_t orig_size, unsigned long caller) {The size of the object is available via a field in
kmem_cache. And a pointer to the currentkmem_cacheis already passed to the function. Why is there a need to add an additional parameter?
That's because we want to do our best detecting bugs on kmalloc-allocated objects.
kmalloc is using size classes, so e.g. when allocating 272 bytes the object will be padded to 512.
As a result, placing that object at the end of the page won't really help to detect out-of-bound accesses that are off by less than 270 bytes.
We probably need to better clarify this in the patch description.
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Marco Elver
maintainerPATCH v2 5/10 mm, kfence: insert KFENCE hooks for SLUB
In reply to Marco ElverFrom: Alexander Potapenko <[email protected]>
Inserts KFENCE hooks into the SLUB allocator.
We note the addition of the 'orig_size' argument to slab_alloc*() functions, to be able to pass the originally requested size to KFENCE. When KFENCE is disabled, there is no additional overhead, since these functions are __always_inline.
--- mm/slub.c | 72 ++++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 53 insertions(+), 19 deletions(-)
mm/slub.c +53 -19
@@ -27,6 +27,7 @@#include <linux/ctype.h> #include <linux/debugobjects.h> #include <linux/kallsyms.h> +#include <linux/kfence.h> #include <linux/memory.h> #include <linux/math64.h> #include <linux/fault-inject.h>
@@ -1557,6 +1558,11 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s,void *old_tail = *tail ? *tail : *head;int rsize; +if (is_kfence_address (next)) { +slab_free_hook (s, next); +return true; + } +/* Head and tail of the reconstructed freelist */ *head =NULL ; *tail =NULL ;
@@ -2660,7 +2666,8 @@ static inline void *get_freelist(struct kmem_cache *s, struct page *page)* already disabled (which is the case for bulk allocation). */ static void *___slab_alloc (struct kmem_cache *s,gfp_t gfpflags,int node, -unsigned long addr,struct kmem_cache_cpu *c) +unsigned long addr,struct kmem_cache_cpu *c, +size_t orig_size) {void *freelist;struct page *page;
@@ -2763,7 +2770,8 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,* cpu changes by refetching the per cpu area pointer. */ static void *__slab_alloc (struct kmem_cache *s,gfp_t gfpflags,int node, -unsigned long addr,struct kmem_cache_cpu *c) +unsigned long addr,struct kmem_cache_cpu *c, +size_t orig_size) {void *p;unsigned long flags;
@@ -2778,7 +2786,7 @@ static void *__slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, c =this_cpu_ptr (s->cpu_slab);#endif - p =___slab_alloc (s, gfpflags, node, addr, c); + p =___slab_alloc (s, gfpflags, node, addr, c, orig_size);local_irq_restore (flags);return p; }
@@ -2805,7 +2813,7 @@ static __always_inline void maybe_wipe_obj_freeptr(struct kmem_cache *s,* Otherwise we can simply pick the next object from the lockless free list. */ static __always_inline void *slab_alloc_node(struct kmem_cache *s, -gfp_t gfpflags,int node,unsigned long addr) +gfp_t gfpflags,int node,unsigned long addr,size_t orig_size) {void *object;struct kmem_cache_cpu *c;
@@ -2816,6 +2824,11 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, s =slab_pre_alloc_hook (s, &objcg,1 , gfpflags);if (!s)return NULL ; + + object =kfence_alloc (s, orig_size, gfpflags); +if (unlikely (object)) + goto out; + redo:/* * Must read kmem_cache cpu data via this cpu ptr. Preemption is
@@ -2853,7 +2866,7 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, object = c->freelist; page = c->page;if (unlikely (!object || !node_match (page, node))) { - object =__slab_alloc (s, gfpflags, node, addr, c); + object =__slab_alloc (s, gfpflags, node, addr, c, orig_size);stat (s,ALLOC_SLOWPATH ); }else {void *next_object =get_freepointer_safe (s, object);
@@ -2889,20 +2902,21 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,if (unlikely (slab_want_init_on_alloc (gfpflags, s)) && object)memset (object,0 , s->object_size); +out:slab_post_alloc_hook (s, objcg, gfpflags,1 , &object);return object; } static __always_inline void *slab_alloc(struct kmem_cache *s, -gfp_t gfpflags,unsigned long addr) +gfp_t gfpflags,unsigned long addr,size_t orig_size) { -return slab_alloc_node (s, gfpflags,NUMA_NO_NODE , addr); +return slab_alloc_node (s, gfpflags,NUMA_NO_NODE , addr, orig_size); }void *kmem_cache_alloc (struct kmem_cache *s,gfp_t gfpflags) { -void *ret =slab_alloc (s, gfpflags, _RET_IP_); +void *ret =slab_alloc (s, gfpflags, _RET_IP_, s->object_size);trace_kmem_cache_alloc (_RET_IP_, ret, s->object_size, s->size, gfpflags);
@@ -2914,7 +2928,7 @@ EXPORT_SYMBOL(kmem_cache_alloc);#ifdef CONFIG_TRACING void *kmem_cache_alloc_trace (struct kmem_cache *s,gfp_t gfpflags,size_t size) { -void *ret =slab_alloc (s, gfpflags, _RET_IP_); +void *ret =slab_alloc (s, gfpflags, _RET_IP_, size);trace_kmalloc (_RET_IP_, ret, size, s->size, gfpflags); ret =kasan_kmalloc (s, ret, size, gfpflags);return ret;
@@ -2925,7 +2939,7 @@ EXPORT_SYMBOL(kmem_cache_alloc_trace);#ifdef CONFIG_NUMA void *kmem_cache_alloc_node (struct kmem_cache *s,gfp_t gfpflags,int node) { -void *ret =slab_alloc_node (s, gfpflags, node, _RET_IP_); +void *ret =slab_alloc_node (s, gfpflags, node, _RET_IP_, s->object_size);trace_kmem_cache_alloc_node (_RET_IP_, ret, s->object_size, s->size, gfpflags, node);
@@ -2939,7 +2953,7 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *s,gfp_t gfpflags, int node, size_t size) { - void *ret = slab_alloc_node(s, gfpflags, node, _RET_IP_); + void *ret = slab_alloc_node(s, gfpflags, node, _RET_IP_, size);trace_kmalloc_node (_RET_IP_, ret, size, s->size, gfpflags, node);
@@ -2973,6 +2987,9 @@ static void __slab_free(struct kmem_cache *s, struct page *page,stat (s,FREE_SLOWPATH ); +if (kfence_free (head)) +return ; +if (kmem_cache_debug (s) && !free_debug_processing (s, page, head, tail, cnt, addr))return ;
@@ -3216,6 +3233,13 @@ int build_detached_freelist(struct kmem_cache *s, size_t size, df->s =cache_from_obj (s, object);/* Support for memcg */ } +if (is_kfence_address (object)) { +slab_free_hook (df->s, object); +WARN_ON (!kfence_free (object)); + p[size] =NULL ;/* mark object processed */ +return size; + } +/* Start new detached freelist */ df->page = page;set_freepointer (df->s, object,NULL );
@@ -3290,8 +3314,14 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, c =this_cpu_ptr (s->cpu_slab);for (i =0 ; i < size; i++) { -void *object = c->freelist; +void *object =kfence_alloc (s, s->object_size, flags); +if (unlikely (object)) { + p[i] = object; +continue ; + } + + object = c->freelist;if (unlikely (!object)) {/* * We may have removed an object from c->freelist using
@@ -3307,7 +3337,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,* of re-populating per CPU c->freelist */ p[i] =___slab_alloc (s, flags,NUMA_NO_NODE , - _RET_IP_, c); + _RET_IP_, c, size);if (unlikely (!p[i])) goto error;
@@ -3962,7 +3992,7 @@ void *__kmalloc(size_t size, gfp_t flags)if (unlikely (ZERO_OR_NULL_PTR (s)))return s; - ret =slab_alloc (s, flags, _RET_IP_); + ret =slab_alloc (s, flags, _RET_IP_, size);trace_kmalloc (_RET_IP_, ret, size, s->size, flags);
@@ -4010,7 +4040,7 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node)if (unlikely (ZERO_OR_NULL_PTR (s)))return s; - ret =slab_alloc_node (s, flags, node, _RET_IP_); + ret =slab_alloc_node (s, flags, node, _RET_IP_, size);trace_kmalloc_node (_RET_IP_, ret, size, s->size, flags, node);
@@ -4036,6 +4066,7 @@ void __check_heap_object(const void *ptr, unsigned long n, struct page *page,struct kmem_cache *s;unsigned int offset;size_t object_size; +bool is_kfence =is_kfence_address (ptr); ptr =kasan_reset_tag (ptr);
@@ -4048,10 +4079,13 @@ void __check_heap_object(const void *ptr, unsigned long n, struct page *page, to_user, 0, n);/* Find offset within object. */ - offset = (ptr -page_address (page)) % s->size; +if (is_kfence) + offset = ptr -kfence_object_start (ptr); +else + offset = (ptr -page_address (page)) % s->size;/* Adjust for redzone and reject if within the redzone. */ -if (kmem_cache_debug_flags (s,SLAB_RED_ZONE )) { +if (!is_kfence &&kmem_cache_debug_flags (s,SLAB_RED_ZONE )) {if (offset < s->red_left_pad)usercopy_abort ("SLUB object in left red zone" , s->name, to_user, offset, n);
@@ -4460,7 +4494,7 @@ void *__kmalloc_track_caller(size_t size, gfp_t gfpflags, unsigned long caller)if (unlikely (ZERO_OR_NULL_PTR (s)))return s; - ret =slab_alloc (s, gfpflags, caller); + ret =slab_alloc (s, gfpflags, caller, size);/* Honor the call site pointer we received. */ trace_kmalloc (caller, ret, size, s->size, gfpflags);
@@ -4491,7 +4525,7 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,if (unlikely (ZERO_OR_NULL_PTR (s)))return s; - ret =slab_alloc_node (s, gfpflags, node, caller); + ret =slab_alloc_node (s, gfpflags, node, caller, size);/* Honor the call site pointer we received. */ trace_kmalloc_node (caller, ret, size, s->size, gfpflags, node);
Signature · 1 line
2.28.0.618.gf4bc123cb7-goog
Re: [PATCH v2 05/10] mm, kfence: insert KFENCE hooks for SLUB
mm/slub.c static __always_inline void *slab_alloc_node(struct kmem_cache *s, · line 2919
Quoted 4 lines · Marco Elver
void *kmem_cache_alloc(struct
kmem_cache*s,gfp_tgfpflags) {mm/slub.c +1
-1-void *ret =slab_alloc (s, gfpflags, _RET_IP_); +void *ret =slab_alloc (s, gfpflags, _RET_IP_, s->object_size);
The additional size parameter is a part of a struct kmem_cache that is already passed to the function. Why does the parameter list need to be expanded?
Alexander Potapenko
maintainerRe: [PATCH v2 05/10] mm, kfence: insert KFENCE hooks for SLUB
Quoted 11 lines · Christopher Lameter
mm/slub.c static __always_inline void *slab_alloc_node(struct kmem_cache *s, · line 2919
Quoted 4 lines · Marco Elver
void *kmem_cache_alloc(struct
kmem_cache*s,gfp_tgfpflags) {mm/slub.c +1
-1-void *ret =slab_alloc (s, gfpflags, _RET_IP_); +void *ret =slab_alloc (s, gfpflags, _RET_IP_, s->object_size);The additional size parameter is a part of a struct
kmem_cachethat is already passed to the function. Why does the parameter list need to be expanded?
See my response to the similar question about the SLAB allocator:
https://lore.kernel.org/linux-arm-kernel/CAG_fn=XMc8NPZPFtUE=rdoR=XJH4F+TxZs-w5n4VuaWKTjcasw@mail.gmail.com/
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Marco Elver
maintainerPATCH v2 6/10 kfence, kasan: make KFENCE compatible with KASAN
In reply to Marco ElverFrom: Alexander Potapenko <[email protected]>
We make KFENCE compatible with KASAN for testing KFENCE itself. In particular, KASAN helps to catch any potential corruptions to KFENCE state, or other corruptions that may be a result of freepointer corruptions in the main allocators.
To indicate that the combination of the two is generally discouraged, CONFIG_EXPERT=y should be set. It also gives us the nice property that KFENCE will be build-tested by allyesconfig builds.
--- lib/Kconfig.kfence | 2 +- mm/kasan/common.c | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-)
lib/Kconfig.kfence +1 -1
@@ -10,7 +10,7 @@ config HAVE_ARCH_KFENCE_STATIC_POOL menuconfig KFENCE bool "KFENCE: low-overhead sampling-based memory safety error detector" - depends on HAVE_ARCH_KFENCE && !KASAN && (SLAB || SLUB) + depends on HAVE_ARCH_KFENCE && (!KASAN || EXPERT) && (SLAB || SLUB) depends on JUMP_LABEL # To ensure performance, require jump labels select STACKTRACE help
mm/kasan/common.c +7 -0
@@ -18,6 +18,7 @@#include <linux/init.h> #include <linux/kasan.h> #include <linux/kernel.h> +#include <linux/kfence.h> #include <linux/kmemleak.h> #include <linux/linkage.h> #include <linux/memblock.h>
@@ -396,6 +397,9 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object, tagged_object = object; object =reset_tag (object); +if (is_kfence_address (object)) +return false; +if (unlikely (nearest_obj (cache,virt_to_head_page (object), object) != object)) {kasan_report_invalid_free (tagged_object, ip);
@@ -444,6 +448,9 @@ static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,if (unlikely (object ==NULL ))return NULL ; +if (is_kfence_address (object)) +return (void *)object; + redzone_start =round_up ((unsigned long )(object + size),KASAN_SHADOW_SCALE_SIZE ); redzone_end = round_up((unsigned long)object + cache->object_size,
Signature · 1 line
2.28.0.618.gf4bc123cb7-goog
Marco Elver
maintainerPATCH v2 7/10 kfence, kmemleak: make KFENCE compatible with KMEMLEAK
In reply to Marco ElverFrom: Alexander Potapenko <[email protected]>
Add compatibility with KMEMLEAK, by making KMEMLEAK aware of the KFENCE memory pool. This allows building debug kernels with both enabled, which also helped in debugging KFENCE.
---
v2:
* Rework using delete_object_part() [suggested by Catalin Marinas].
--- mm/kmemleak.c | 6 ++++++ 1 file changed, 6 insertions(+)
mm/kmemleak.c +6 -0
@@ -97,6 +97,7 @@#include <linux/atomic.h> #include <linux/kasan.h> +#include <linux/kfence.h> #include <linux/kmemleak.h> #include <linux/memory_hotplug.h>
@@ -1948,6 +1949,11 @@ void __init kmemleak_init(void) KMEMLEAK_GREY, GFP_ATOMIC);create_object ((unsigned long )__bss_start, __bss_stop - __bss_start,KMEMLEAK_GREY ,GFP_ATOMIC ); +#if defined(CONFIG_KFENCE ) && defined(CONFIG_HAVE_ARCH_KFENCE_STATIC_POOL ) +/* KFENCE objects are located in .bss, which may confuse kmemleak. Skip them. */ +delete_object_part ((unsigned long )__kfence_pool,KFENCE_POOL_SIZE ); +#endif +/* only register .data..ro_after_init if not within .data */ if (&__start_ro_after_init < &_sdata || &__end_ro_after_init > &_edata) create_object((unsigned long)__start_ro_after_init,
Signature · 1 line
2.28.0.618.gf4bc123cb7-goog
Marco Elver
maintainerPATCH v2 8/10 kfence, lockdep: make KFENCE compatible with lockdep
In reply to Marco ElverLockdep checks that dynamic key registration is only performed on keys that are not static objects. With KFENCE, it is possible that such a dynamically allocated key is a KFENCE object which may, however, be allocated from a static memory pool (if HAVE_ARCH_KFENCE_STATIC_POOL).
Therefore, ignore KFENCE-allocated objects in static_obj().
--- kernel/locking/lockdep.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/locking/lockdep.c +8 -0
@@ -38,6 +38,7 @@#include <linux/seq_file.h> #include <linux/spinlock.h> #include <linux/kallsyms.h> +#include <linux/kfence.h> #include <linux/interrupt.h> #include <linux/stacktrace.h> #include <linux/debug_locks.h>
@@ -755,6 +756,13 @@ static int static_obj(const void *obj)if (arch_is_kernel_initmem_freed (addr))return 0 ; +/* +* KFENCE objects may be allocated from a static memory pool, but are +* not actually static objects. +*/ +if (is_kfence_address (obj)) +return 0 ; +/* * static variable? */
Signature · 1 line
2.28.0.618.gf4bc123cb7-goog
Marco Elver
maintainerPATCH v2 9/10 kfence, Documentation: add KFENCE documentation
In reply to Marco ElverAdd KFENCE documentation in dev-tools/kfence.rst, and add to index.
---
v2:
* Many clarifications based on comments from Andrey Konovalov.
* Document CONFIG_KFENCE_SAMPLE_INTERVAL=0 usage.
* Make use-cases between KASAN and KFENCE clearer.
* Be clearer about the fact the pool is fixed size.
* Update based on reporting changes.
* Explicitly mention max supported allocation size is PAGE_SIZE.
--- Documentation/dev-tools/index.rst | 1 + Documentation/dev-tools/kfence.rst | 291 +++++++++++++++++++++++++++++ 2 files changed, 292 insertions(+) create mode 100644 Documentation/dev-tools/kfence.rst
Documentation/dev-tools/index.rst +1 -0
@@ -22,6 +22,7 @@ whole; patches welcome! ubsan kmemleak kcsan + kfence gdb-kernel-debugging kgdb kselftest
Documentation/dev-tools/kfence.rst new file +291 -0
291 lines not on this page · raw message
Signature · 1 line
2.28.0.618.gf4bc123cb7-goog
Marco Elver
maintainerPATCH v2 10/10 kfence: add test suite
In reply to Marco ElverAdd KFENCE test suite, testing various error detection scenarios. Makes use of KUnit for test organization. Since KFENCE's interface to obtain error reports is via the console, the test verifies that KFENCE outputs expected reports to the console.
---
v2:
* Update for shortened memory corruption report.
--- lib/Kconfig.kfence | 13 + mm/kfence/Makefile | 3 + mm/kfence/kfence_test.c | 777 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 793 insertions(+) create mode 100644 mm/kfence/kfence_test.c
lib/Kconfig.kfence +13 -0
@@ -62,4 +62,17 @@ config KFENCE_STRESS_TEST_FAULTS The option is only to test KFENCE; set to 0 if you are unsure. +config KFENCE_KUNIT_TEST + tristate "KFENCE integration test suite" if !KUNIT_ALL_TESTS + default KUNIT_ALL_TESTS + depends on TRACEPOINTS && KUNIT + help + Test suite for KFENCE, testing various error detection scenarios with + various allocation types, and checking that reports are correctly + output to console. + + Say Y here if you want the test to be built into the kernel and run + during boot; say M if you want the test to build as a module; say N + if you are unsure. + endif # KFENCE
mm/kfence/Makefile +3 -0
@@ -1,3 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_KFENCE) := core.o report.o + +CFLAGS_kfence_test.o := -g -fno-omit-frame-pointer -fno-optimize-sibling-calls +obj-$(CONFIG_KFENCE_KUNIT_TEST) += kfence_test.o
mm/kfence/kfence_test.c new file +777 -0
777 lines not on this page · raw message
Signature · 1 line
2.28.0.618.gf4bc123cb7-goog
kernel test robot
build botRe: [PATCH v2 10/10] kfence: add test suite
I love your patch! Perhaps something to improve:
Hi Marco,
I love your patch! Perhaps something to improve:
[auto build test WARNING on hnaz-linux-mm/master]
[also build test WARNING on arm64/for-next/core linus/master v5.9-rc5] [cannot apply to tip/x86/core next-20200915]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Marco-Elver/KFENCE-A-low-overhead-sampling-based-memory-safety-error-detector/20200915-212305 base: https://github.com/hnaz/linux-mm master config: i386-allyesconfig (attached as .config) compiler: gcc-9 (Debian 9.3.0-15) 9.3.0 reproduce (this is a W=1 build): # save the attached .config to linux build tree make W=1 ARCH=i386
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
mm/kfence/kfence_test.c: In function 'report_matches':
>> mm/kfence/kfence_test.c:150:1: warning: the frame size of 1032 bytes is larger than 1024 bytes [-Wframe-larger-than=]
150 | }
| ^# https://github.com/0day-ci/linux/commit/e52f64e8d3cbfe2423913871780cf012b2abeb21
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Marco-Elver/KFENCE-A-low-overhead-sampling-based-memory-safety-error-detector/20200915-212305 git checkout e52f64e8d3cbfe2423913871780cf012b2abeb21
vim +150 mm/kfence/kfence_test.c
74
75 /* Check observed report matches information in @r. */
76 static bool report_matches(const struct expect_report *r)
77 {
78 bool ret = false;
79 unsigned long flags;
80 typeof(observed.lines) expect;
81 const char *end;
82 char *cur;
83
84 /* Doubled-checked locking. */
85 if (!report_available())
86 return false;
87
88 /* Generate expected report contents. */
89
90 /* Title */
91 cur = expect[0];
92 end = &expect[0][sizeof(expect[0]) - 1];
93 switch (r->type) {
94 case KFENCE_ERROR_OOB:
95 cur += scnprintf(cur, end - cur, "BUG: KFENCE: out-of-bounds");
96 break;
97 case KFENCE_ERROR_UAF:
98 cur += scnprintf(cur, end - cur, "BUG: KFENCE: use-after-free");
99 break;
100 case KFENCE_ERROR_CORRUPTION:
101 cur += scnprintf(cur, end - cur, "BUG: KFENCE: memory corruption");
102 break;
103 case KFENCE_ERROR_INVALID:
104 cur += scnprintf(cur, end - cur, "BUG: KFENCE: invalid access");
105 break;
106 case KFENCE_ERROR_INVALID_FREE:
107 cur += scnprintf(cur, end - cur, "BUG: KFENCE: invalid free");
108 break;
109 }
110
111 scnprintf(cur, end - cur, " in %pS", r->fn);
112 /* The exact offset won't match, remove it; also strip module name. */
113 cur = strchr(expect[0], '+');
114 if (cur)
115 *cur = '\0';
116
117 /* Access information */
118 cur = expect[1];
119 end = &expect[1][sizeof(expect[1]) - 1];
120
121 switch (r->type) {
122 case KFENCE_ERROR_OOB:
123 cur += scnprintf(cur, end - cur, "Out-of-bounds access at");
124 break;
125 case KFENCE_ERROR_UAF:
126 cur += scnprintf(cur, end - cur, "Use-after-free access at");
127 break;
128 case KFENCE_ERROR_CORRUPTION:
129 cur += scnprintf(cur, end - cur, "Corrupted memory at");
130 break;
131 case KFENCE_ERROR_INVALID:
132 cur += scnprintf(cur, end - cur, "Invalid access at");
133 break;
134 case KFENCE_ERROR_INVALID_FREE:
135 cur += scnprintf(cur, end - cur, "Invalid free of");
136 break;
137 }
138
139 cur += scnprintf(cur, end - cur, " 0x" PTR_FMT, (void *)r->addr);
140
141 spin_lock_irqsave(&observed.lock, flags);
142 if (!report_available())
143 goto out; /* A new report is being captured. */
144
145 /* Finally match expected output to what we actually observed. */
146 ret = strstr(observed.lines[0], expect[0]) && strstr(observed.lines[1], expect[1]);
147 out:
148 spin_unlock_irqrestore(&observed.lock, flags);
149 return ret;
> 150 }151
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Quoted 47 lines · Marco Elver
This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors. This series enables KFENCE for the x86 and arm64 architectures, and adds KFENCE hooks to the SLAB and SLUB allocators.
KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.
KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error.
Guarded allocations are set up based on a sample interval (can be set via kfence.sample_interval). After expiration of the sample interval, the next allocation through the main allocator (SLAB or SLUB) returns a guarded allocation from the KFENCE object pool. At this point, the timer is reset, and the next allocation is set up after the expiration of the interval.
To enable/disable a KFENCE allocation through the main allocator's fast-path without overhead, KFENCE relies on static branches via the static keys infrastructure. The static branch is toggled to redirect the allocation to KFENCE.
The KFENCE memory pool is of fixed size, and if the pool is exhausted no further KFENCE allocations occur. The default config is conservative with only 255 objects, resulting in a pool size of 2 MiB (with 4 KiB pages).
We have verified by running synthetic benchmarks (sysbench I/O, hackbench) that a kernel with KFENCE is performance-neutral compared to a non-KFENCE baseline kernel.
KFENCE is inspired by GWP-ASan [1], a userspace tool with similar properties. The name "KFENCE" is a homage to the Electric Fence Malloc Debugger [2].
Quoted 7 lines
For more details, see
Documentation/dev-tools/kfence.rstadded in the series -- also viewable here:https://raw.githubusercontent.com/google/kasan/kfence/Documentation/dev-tools/kfence.rst
[1] http://llvm.org/docs/GwpAsan.html
[2] https://linux.die.net/man/3/efence
I see all of my comments from v1 are resolved. So this is:
for the series.
Quoted 63 lines
v2:
* Various comment/documentation changes (see details in patches).
* Various smaller fixes (see details in patches).
* Change all reports to reference the kfence object, "kfence-#nn".
* Skip allocation/free internals stack trace.
* Rework KMEMLEAK compatibility patch.RFC/v1: https://lkml.kernel.org/r/[email protected]
Alexander Potapenko (6):
mm: add Kernel Electric-Fence infrastructure x86, kfence: enable KFENCE for x86
mm, kfence: insert KFENCE hooks for SLAB
mm, kfence: insert KFENCE hooks for SLUB kfence, kasan: make KFENCE compatible with KASAN kfence, kmemleak: make KFENCE compatible with KMEMLEAKMarco Elver (4):
arm64, kfence: enable KFENCE for ARM64 kfence, lockdep: make KFENCE compatible with lockdep kfence, Documentation: add KFENCE documentation
kfence: add test suiteDocumentation/dev-tools/index.rst | 1 + Documentation/dev-tools/kfence.rst | 291 +++++++++++ MAINTAINERS | 11 + arch/arm64/Kconfig | 1 + arch/arm64/include/asm/kfence.h | 39 ++ arch/arm64/mm/fault.c | 4 + arch/x86/Kconfig | 2 + arch/x86/include/asm/kfence.h | 60 +++ arch/x86/mm/fault.c | 4 + include/linux/kfence.h | 174 +++++++ init/main.c | 2 + kernel/locking/lockdep.c | 8 + lib/Kconfig.debug | 1 + lib/Kconfig.kfence | 78 +++ mm/Makefile | 1 + mm/kasan/common.c | 7 + mm/kfence/Makefile | 6 + mm/kfence/core.c | 733 +++++++++++++++++++++++++++ mm/kfence/kfence.h | 102 ++++ mm/kfence/kfence_test.c | 777 +++++++++++++++++++++++++++++ mm/kfence/report.c | 219 ++++++++ mm/kmemleak.c | 6 + mm/slab.c | 46 +- mm/slab_common.c | 6 +- mm/slub.c | 72 ++- 25 files changed, 2619 insertions(+), 32 deletions(-) create mode 100644 Documentation/dev-tools/kfence.rst create mode 100644 arch/arm64/include/asm/kfence.h create mode 100644 arch/x86/include/asm/kfence.h create mode 100644 include/linux/kfence.h create mode 100644 lib/Kconfig.kfence create mode 100644 mm/kfence/Makefile create mode 100644 mm/kfence/core.c create mode 100644 mm/kfence/kfence.h create mode 100644 mm/kfence/kfence_test.c create mode 100644 mm/kfence/report.c--
2.28.0.618.gf4bc123cb7-goog
Quoted 42 lines · Marco Elver
This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors. This series enables KFENCE for the x86 and arm64 architectures, and adds KFENCE hooks to the SLAB and SLUB allocators.
KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.
KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error.
Guarded allocations are set up based on a sample interval (can be set via kfence.sample_interval). After expiration of the sample interval, the next allocation through the main allocator (SLAB or SLUB) returns a guarded allocation from the KFENCE object pool. At this point, the timer is reset, and the next allocation is set up after the expiration of the interval.
To enable/disable a KFENCE allocation through the main allocator's fast-path without overhead, KFENCE relies on static branches via the static keys infrastructure. The static branch is toggled to redirect the allocation to KFENCE.
The KFENCE memory pool is of fixed size, and if the pool is exhausted no further KFENCE allocations occur. The default config is conservative with only 255 objects, resulting in a pool size of 2 MiB (with 4 KiB pages).
We have verified by running synthetic benchmarks (sysbench I/O, hackbench) that a kernel with KFENCE is performance-neutral compared to a non-KFENCE baseline kernel.
Quoted 6 lines
KFENCE is inspired by GWP-ASan [1], a userspace tool with similar properties. The name "KFENCE" is a homage to the Electric Fence Malloc Debugger [2].
For more details, see
Documentation/dev-tools/kfence.rstadded in the series -- also viewable here:
Does anybody else grow tried of all those different *imperfect* versions of in- kernel memory safety error detectors? KASAN-generic, KFENCE, KASAN-tag-based etc. Then, we have old things like page_poison, SLUB debugging, debug_pagealloc etc which are pretty much inefficient to detect bugs those days compared to KASAN. Can't we work towards having a single implementation and clean up all those mess?
Marco Elver
maintainerQuoted 51 lines · Qian Cai
Quoted 42 lines · Marco Elver
This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors. This series enables KFENCE for the x86 and arm64 architectures, and adds KFENCE hooks to the SLAB and SLUB allocators.
KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.
KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error.
Guarded allocations are set up based on a sample interval (can be set via kfence.sample_interval). After expiration of the sample interval, the next allocation through the main allocator (SLAB or SLUB) returns a guarded allocation from the KFENCE object pool. At this point, the timer is reset, and the next allocation is set up after the expiration of the interval.
To enable/disable a KFENCE allocation through the main allocator's fast-path without overhead, KFENCE relies on static branches via the static keys infrastructure. The static branch is toggled to redirect the allocation to KFENCE.
The KFENCE memory pool is of fixed size, and if the pool is exhausted no further KFENCE allocations occur. The default config is conservative with only 255 objects, resulting in a pool size of 2 MiB (with 4 KiB pages).
We have verified by running synthetic benchmarks (sysbench I/O, hackbench) that a kernel with KFENCE is performance-neutral compared to a non-KFENCE baseline kernel.
Quoted 6 lines
KFENCE is inspired by GWP-ASan [1], a userspace tool with similar properties. The name "KFENCE" is a homage to the Electric Fence Malloc Debugger [2].
For more details, see
Documentation/dev-tools/kfence.rstadded in the series -- also viewable here:
Quoted 6 lines
Does anybody else grow tried of all those different *imperfect* versions of in- kernel memory safety error detectors? KASAN-generic, KFENCE, KASAN-tag-based etc. Then, we have old things like
page_poison, SLUB debugging,debug_pageallocetc which are pretty much inefficient to detect bugs those days compared to KASAN. Can't we work towards having a single implementation and clean up all those mess?
If you have suggestions on how to get a zero-overhead, precise ("perfect") memory safety error detector without new hardware extensions, we're open to suggestions -- many people over many years have researched this problems, and while we're making progress for C (and C++), the fact remains that what you're asking is likely impossible. This might be useful background:
https://arxiv.org/pdf/1802.09517.pdf
The fact remains that requirements and environments vary across applications and usecases. Maybe for one usecase (debugging, test env) normal KASAN is just fine. But that doesn't work for production, where we want to have max performance.
MTE will get us closer (no silicon yet, and ARM64 only for now), but depending on implementation might come with small overheads, although quite acceptable for most environments with increasing processing power modern CPUs deliver.
Yet for other environments, where even a small performance regression is unacceptable, and where it's infeasible to capture in tests what the workloads execute, KFENCE is a very attractive option.
There have also been discussions on using Rust in the kernel [1], but this is just not feasible for core kernel code in the near future (even then, you'll still need dynamic error detection tools for all the unsafe bits, of which there are many in an OS kernel).
[1] https://lwn.net/Articles/829858/
Thanks,
-- Marco