Skip to contentSkip to conversation

KFENCE: A low-overhead sampling-based memory safety error detector

PATCH Superseded Marco Elver 4 participants 33 messages Oct 29, 2020 – Oct 30, 2020

Subsystems (11)

v6 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Marco Elver

author maintainer
#0 lore raw

v6 cover KFENCE: A low-overhead sampling-based memory safety error detector

This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors. This series enables KFENCE for the x86 and arm64 architectures, and adds KFENCE hooks to the SLAB and SLUB allocators.

KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.

KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error.

Guarded allocations are set up based on a sample interval (can be set via kfence.sample_interval). After expiration of the sample interval, the next allocation through the main allocator (SLAB or SLUB) returns a guarded allocation from the KFENCE object pool. At this point, the timer is reset, and the next allocation is set up after the expiration of the interval.

To enable/disable a KFENCE allocation through the main allocator's fast-path without overhead, KFENCE relies on static branches via the static keys infrastructure. The static branch is toggled to redirect the allocation to KFENCE.

The KFENCE memory pool is of fixed size, and if the pool is exhausted no further KFENCE allocations occur. The default config is conservative with only 255 objects, resulting in a pool size of 2 MiB (with 4 KiB pages).

We have verified by running synthetic benchmarks (sysbench I/O, hackbench) that a kernel with KFENCE is performance-neutral compared to a non-KFENCE baseline kernel.

KFENCE is inspired by GWP-ASan [1], a userspace tool with similar properties. The name "KFENCE" is a homage to the Electric Fence Malloc Debugger [2].

For more details, see Documentation/dev-tools/kfence.rst added in the series -- also viewable here:

https://raw.githubusercontent.com/google/kasan/kfence/Documentation/dev-tools/kfence.rst

[1] http://llvm.org/docs/GwpAsan.html
[2] https://linux.die.net/man/3/efence

v6:
* Record allocation and free task pids, and show them in reports. This information helps more easily identify e.g. racy use-after-frees.

v5: https://lkml.kernel.org/r/[email protected]
* Lots of smaller fixes (see details in patches).
* Optimize is_kfence_address() by using better in-range check.
* Removal of HAVE_ARCH_KFENCE_STATIC_POOL and static pool support in favor of memblock_alloc'd pool only, as it avoids all issues with virt_to translations. With the new optimizations to is_kfence_address(), we measure no noticeable performance impact.
* Taint with TAINT_BAD_PAGE, to distinguish memory errors from regular warnings (also used by SL*B/KASAN/etc. for memory errors).
* Rework sample_interval parameter dynamic setting semantics.
* Rework kfence_shutdown_cache().
* Fix obj_to_index+objs_per_slab_page, which among other things is required when using memcg accounted allocations.
* Rebase to 5.10-rc1.

v4: https://lkml.kernel.org/r/[email protected]
* MAINTAINERS: Split out from first patch.
* Make static memory pool's attrs entirely arch-dependent.
* Fix report generation if __slab_free tail-called.
* Clarify RCU test comment [reported by Paul E. McKenney].

v3: https://lkml.kernel.org/r/[email protected]
* Rewrite SLAB/SLUB patch descriptions to clarify need for 'orig_size'.
* Various smaller fixes (see details in patches).

v2: https://lkml.kernel.org/r/[email protected]
* Various comment/documentation changes (see details in patches).
* Various smaller fixes (see details in patches).
* Change all reports to reference the kfence object, "kfence-#nn".
* Skip allocation/free internals stack trace.
* Rework KMEMLEAK compatibility patch.

RFC/v1: https://lkml.kernel.org/r/[email protected]

Alexander Potapenko (5):
mm: add Kernel Electric-Fence infrastructure x86, kfence: enable KFENCE for x86
mm, kfence: insert KFENCE hooks for SLAB mm, kfence: insert KFENCE hooks for SLUB kfence, kasan: make KFENCE compatible with KASAN

Marco Elver (4):
arm64, kfence: enable KFENCE for ARM64
kfence, Documentation: add KFENCE documentation
kfence: add test suite
MAINTAINERS: Add entry for KFENCE

 Documentation/dev-tools/index.rst  |   1 +
 Documentation/dev-tools/kfence.rst | 291 ++++++++++
 MAINTAINERS                        |  11 +
 arch/arm64/Kconfig                 |   1 +
 arch/arm64/include/asm/kfence.h    |  19 +
 arch/arm64/mm/fault.c              |   4 +
 arch/arm64/mm/mmu.c                |   7 +-
 arch/x86/Kconfig                   |   1 +
 arch/x86/include/asm/kfence.h      |  65 +++
 arch/x86/mm/fault.c                |   4 +
 include/linux/kfence.h             | 191 +++++++
 include/linux/slab_def.h           |   3 +
 include/linux/slub_def.h           |   3 +
 init/main.c                        |   3 +
 lib/Kconfig.debug                  |   1 +
 lib/Kconfig.kfence                 |  73 +++
 mm/Makefile                        |   1 +
 mm/kasan/common.c                  |  15 +
 mm/kasan/generic.c                 |   3 +-
 mm/kfence/Makefile                 |   6 +
 mm/kfence/core.c                   | 821 ++++++++++++++++++++++++++++
 mm/kfence/kfence.h                 | 107 ++++
 mm/kfence/kfence_test.c            | 822 +++++++++++++++++++++++++++++
 mm/kfence/report.c                 | 235 +++++++++
 mm/slab.c                          |  37 +-
 mm/slab_common.c                   |   5 +-
 mm/slub.c                          |  72 ++-
 27 files changed, 2771 insertions(+), 31 deletions(-)
 create mode 100644 Documentation/dev-tools/kfence.rst
 create mode 100644 arch/arm64/include/asm/kfence.h
 create mode 100644 arch/x86/include/asm/kfence.h
 create mode 100644 include/linux/kfence.h
 create mode 100644 lib/Kconfig.kfence
 create mode 100644 mm/kfence/Makefile
 create mode 100644 mm/kfence/core.c
 create mode 100644 mm/kfence/kfence.h
 create mode 100644 mm/kfence/kfence_test.c
 create mode 100644 mm/kfence/report.c
Signature · 1 line
2.29.1.341.ge80a0c044ae-goog
author maintainer
#1 lore raw

PATCH v6 1/9 mm: add Kernel Electric-Fence infrastructure

From: Alexander Potapenko <[email protected]>

This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors.

KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.

KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error. To detect out-of-bounds writes to memory within the object's page itself, KFENCE also uses pattern-based redzones. The following figure illustrates the page layout:

---+-----------+-----------+-----------+-----------+-----------+---

     | xxxxxxxxx | O :       | xxxxxxxxx |       : O | xxxxxxxxx |
     | xxxxxxxxx | B :       | xxxxxxxxx |       : B | xxxxxxxxx |
     | x GUARD x | J : RED-  | x GUARD x | RED-  : J | x GUARD x |
     | xxxxxxxxx | E :  ZONE | xxxxxxxxx |  ZONE : E | xxxxxxxxx |
     | xxxxxxxxx | C :       | xxxxxxxxx |       : C | xxxxxxxxx |
     | xxxxxxxxx | T :       | xxxxxxxxx |       : T | xxxxxxxxx |

---+-----------+-----------+-----------+-----------+-----------+---

Guarded allocations are set up based on a sample interval (can be set via kfence.sample_interval). After expiration of the sample interval, a guarded allocation from the KFENCE object pool is returned to the main allocator (SLAB or SLUB). At this point, the timer is reset, and the next allocation is set up after the expiration of the interval.

To enable/disable a KFENCE allocation through the main allocator's fast-path without overhead, KFENCE relies on static branches via the static keys infrastructure. The static branch is toggled to redirect the allocation to KFENCE. To date, we have verified by running synthetic benchmarks (sysbench I/O workloads) that a kernel compiled with KFENCE is performance-neutral compared to the non-KFENCE baseline.

For more details, see Documentation/dev-tools/kfence.rst (added later in the series).

---
v6:
* Record allocation and free task pids, and show them in reports. This information helps more easily identify e.g. racy use-after-frees.

v5:
* MAJOR CHANGE: Removal of HAVE_ARCH_KFENCE_STATIC_POOL and static pool support in favor of memblock_alloc'd pool only, as it avoids all issues with virt_to translations. With the new optimizations to is_kfence_address(), we measure no noticeable performance impact.
* Verify we do not end up with a compound head page.
* Fix reporting of corruptions to never show object contents.
* Reformat kfence_alloc [suggested by Mark Rutland].
* Taint with TAINT_BAD_PAGE, to distinguish memory errors from regular warnings (also used by SL*B/KASAN/etc. for memory errors).
* Show OOB offset bytes in report.
* Rework kfence_shutdown_cache().
* Set page fields to fix obj_to_index+objs_per_slab_page.
* Suggestions/Reports by Jann Horn:
* Move generic page allocation code to core.c.
* Use KERN_ERR for dump_stack_print_info.
* Make __kfence_pool pointer __ro_after_init.
* Fix typos.
* Add likely hint for check_canary_byte.
* Make for_each_canary __always_inline.
* Add comment about IPIs for static key toggling.
* Check for non-null pointer in is_kfence_address(), in case KFENCE is never initialized.
* Rework sample_interval parameter dynamic setting semantics.
* Fix redzone checking.
* Optimize is_kfence_address() by using better in-range check.

v4:
* Make static memory pool's attrs entirely arch-dependent.
* Revert MAINTAINERS, and make separate patch.
* Fix report generation if __slab_free tail-called.

v3:
* Reports by SeongJae Park:
* Remove reference to Documentation/dev-tools/kfence.rst.
* Remove redundant braces.
* Use CONFIG_KFENCE_NUM_OBJECTS instead of ARRAY_SIZE(...).
* Align some comments.
* Add figure from Documentation/dev-tools/kfence.rst added later in series to patch description.

v2:
* Add missing __printf attribute to seq_con_printf, and fix new warning. [reported by kernel test robot <[email protected]>]
* Fix up some comments [reported by Jonathan Cameron].
* Remove 2 cases of redundant stack variable initialization [reported by Jonathan Cameron].
* Fix printf format [reported by kernel test robot <[email protected]>].
* Print (in kfence-#nn) after address, to more clearly establish link between first and second stacktrace [reported by Andrey Konovalov].
* Make choice between KASAN and KFENCE clearer in Kconfig help text [suggested by Dave Hansen].
* Document CONFIG_KFENCE_SAMPLE_INTERVAL=0.
* Shorten memory corruption report line length.
* Make /sys/module/kfence/parameters/sample_interval root-writable for all builds (to enable debugging, automatic dynamic tweaking).
* Reports by Dmitry Vyukov:
* Do not store negative size for right-located objects
* Only cache-align addresses of right-located objects.
* Run toggle_allocation_gate() after KFENCE is enabled.
* Add empty line between allocation and free stacks.
* Add comment about SLAB_TYPESAFE_BY_RCU.
* Also skip internals for allocation/free stacks.
* s/KFENCE_FAULT_INJECTION/KFENCE_STRESS_TEST_FAULTS/ as FAULT_INJECTION is already overloaded in different contexts.
* Parenthesis for macro variable.
* Lower max of KFENCE_NUM_OBJECTS config variable.

---
 include/linux/kfence.h | 191 ++++++++++
 init/main.c            |   3 +
 lib/Kconfig.debug      |   1 +
 lib/Kconfig.kfence     |  58 +++
 mm/Makefile            |   1 +
 mm/kfence/Makefile     |   3 +
 mm/kfence/core.c       | 821 +++++++++++++++++++++++++++++++++++++++++
 mm/kfence/kfence.h     | 107 ++++++
 mm/kfence/report.c     | 235 ++++++++++++
 9 files changed, 1420 insertions(+)
 create mode 100644 include/linux/kfence.h
 create mode 100644 lib/Kconfig.kfence
 create mode 100644 mm/kfence/Makefile
 create mode 100644 mm/kfence/core.c
 create mode 100644 mm/kfence/kfence.h
 create mode 100644 mm/kfence/report.c
include/linux/kfence.h new file +191 -0
new file mode 100644
index 000000000000..a729cf8c1412
@@ -0,0 +1,191 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef _LINUX_KFENCE_H
+#define _LINUX_KFENCE_H
+
+#include <linux/mm.h>
+#include <linux/static_key.h>
+#include <linux/types.h>
+
+#ifdef CONFIG_KFENCE
+
+/*
+ * We allocate an even number of pages, as it simplifies calculations to map
+ * address to metadata indices; effectively, the very first page serves as an
+ * extended guard page, but otherwise has no special purpose.
+ */
+#define KFENCE_POOL_SIZE ((CONFIG_KFENCE_NUM_OBJECTS + 1) * 2 * PAGE_SIZE)
+extern char *__kfence_pool;
+
+DECLARE_STATIC_KEY_FALSE(kfence_allocation_key);
+
+/**
+ * is_kfence_address() - check if an address belongs to KFENCE pool
+ * @addr: address to check
+ *
+ * Return: true or false depending on whether the address is within the KFENCE
+ * object range.
+ *
+ * KFENCE objects live in a separate page range and are not to be intermixed
+ * with regular heap objects (e.g. KFENCE objects must never be added to the
+ * allocator freelists). Failing to do so may and will result in heap
+ * corruptions, therefore is_kfence_address() must be used to check whether
+ * an object requires specific handling.
+ */
+static __always_inline bool is_kfence_address(const void *addr)
+{
+	/*
+	 * The non-NULL check is required in case the __kfence_pool pointer was
+	 * never initialized; keep it in the slow-path after the range-check.
+	 */
+	return unlikely((unsigned long)((char *)addr - __kfence_pool) < KFENCE_POOL_SIZE && addr);
+}
+
+/**
+ * kfence_alloc_pool() - allocate the KFENCE pool via memblock
+ */
+void __init kfence_alloc_pool(void);
+
+/**
+ * kfence_init() - perform KFENCE initialization at boot time
+ *
+ * Requires that kfence_alloc_pool() was called before. This sets up the
+ * allocation gate timer, and requires that workqueues are available.
+ */
+void __init kfence_init(void);
+
+/**
+ * kfence_shutdown_cache() - handle shutdown_cache() for KFENCE objects
+ * @s: cache being shut down
+ *
+ * Before shutting down a cache, one must ensure there are no remaining objects
+ * allocated from it. Because KFENCE objects are not referenced from the cache
+ * directly, we need to check them here.
+ *
+ * Note that shutdown_cache() is internal to SL*B, and kmem_cache_destroy() does
+ * not return if allocated objects still exist: it prints an error message and
+ * simply aborts destruction of a cache, leaking memory.
+ *
+ * If the only such objects are KFENCE objects, we will not leak the entire
+ * cache, but instead try to provide more useful debug info by making allocated
+ * objects "zombie allocations". Objects may then still be used or freed (which
+ * is handled gracefully), but usage will result in showing KFENCE error reports
+ * which include stack traces to the user of the object, the original allocation
+ * site, and caller to shutdown_cache().
+ */
+void kfence_shutdown_cache(struct kmem_cache *s);
+
+/*
+ * Allocate a KFENCE object. Allocators must not call this function directly,
+ * use kfence_alloc() instead.
+ */
+void *__kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags);
+
+/**
+ * kfence_alloc() - allocate a KFENCE object with a low probability
+ * @s:     struct kmem_cache with object requirements
+ * @size:  exact size of the object to allocate (can be less than @s->size
+ *         e.g. for kmalloc caches)
+ * @flags: GFP flags
+ *
+ * Return:
+ * * NULL     - must proceed with allocating as usual,
+ * * non-NULL - pointer to a KFENCE object.
+ *
+ * kfence_alloc() should be inserted into the heap allocation fast path,
+ * allowing it to transparently return KFENCE-allocated objects with a low
+ * probability using a static branch (the probability is controlled by the
+ * kfence.sample_interval boot parameter).
+ */
+static __always_inline void *kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags)
+{
+	if (static_branch_unlikely(&kfence_allocation_key))
+		return __kfence_alloc(s, size, flags);
+	return NULL;
+}
+
+/**
+ * kfence_ksize() - get actual amount of memory allocated for a KFENCE object
+ * @addr: pointer to a heap object
+ *
+ * Return:
+ * * 0     - not a KFENCE object, must call __ksize() instead,
+ * * non-0 - this many bytes can be accessed without causing a memory error.
+ *
+ * kfence_ksize() returns the number of bytes requested for a KFENCE object at
+ * allocation time. This number may be less than the object size of the
+ * corresponding struct kmem_cache.
+ */
+size_t kfence_ksize(const void *addr);
+
+/**
+ * kfence_object_start() - find the beginning of a KFENCE object
+ * @addr - address within a KFENCE-allocated object
+ *
+ * Return: address of the beginning of the object.
+ *
+ * SL[AU]B-allocated objects are laid out within a page one by one, so it is
+ * easy to calculate the beginning of an object given a pointer inside it and
+ * the object size. The same is not true for KFENCE, which places a single
+ * object at either end of the page. This helper function is used to find the
+ * beginning of a KFENCE-allocated object.
+ */
+void *kfence_object_start(const void *addr);
+
+/*
+ * Release a KFENCE-allocated object to KFENCE pool. Allocators must not call
+ * this function directly, use kfence_free() instead.
+ */
+void __kfence_free(void *addr);
+
+/**
+ * kfence_free() - try to release an arbitrary heap object to KFENCE pool
+ * @addr: object to be freed
+ *
+ * Return:
+ * * false - object doesn't belong to KFENCE pool and was ignored,
+ * * true  - object was released to KFENCE pool.
+ *
+ * Release a KFENCE object and mark it as freed. May be called on any object,
+ * even non-KFENCE objects, to simplify integration of the hooks into the
+ * allocator's free codepath. The allocator must check the return value to
+ * determine if it was a KFENCE object or not.
+ */
+static __always_inline __must_check bool kfence_free(void *addr)
+{
+	if (!is_kfence_address(addr))
+		return false;
+	__kfence_free(addr);
+	return true;
+}
+
+/**
+ * kfence_handle_page_fault() - perform page fault handling for KFENCE pages
+ * @addr: faulting address
+ *
+ * Return:
+ * * false - address outside KFENCE pool,
+ * * true  - page fault handled by KFENCE, no additional handling required.
+ *
+ * A page fault inside KFENCE pool indicates a memory error, such as an
+ * out-of-bounds access, a use-after-free or an invalid memory access. In these
+ * cases KFENCE prints an error message and marks the offending page as
+ * present, so that the kernel can proceed.
+ */
+bool __must_check kfence_handle_page_fault(unsigned long addr);
+
+#else /* CONFIG_KFENCE */
+
+static inline bool is_kfence_address(const void *addr) { return false; }
+static inline void kfence_alloc_pool(void) { }
+static inline void kfence_init(void) { }
+static inline void kfence_shutdown_cache(struct kmem_cache *s) { }
+static inline void *kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags) { return NULL; }
+static inline size_t kfence_ksize(const void *addr) { return 0; }
+static inline void *kfence_object_start(const void *addr) { return NULL; }
+static inline bool __must_check kfence_free(void *addr) { return false; }
+static inline bool __must_check kfence_handle_page_fault(unsigned long addr) { return false; }
+
+#endif
+
+#endif /* _LINUX_KFENCE_H */
init/main.c +3 -0
index 130376ec10ba..548746bd6fd6 100644
@@ -40,6 +40,7 @@
 #include <linux/security.h>
 #include <linux/smp.h>
 #include <linux/profile.h>
+#include <linux/kfence.h>
 #include <linux/rcupdate.h>
 #include <linux/moduleparam.h>
 #include <linux/kallsyms.h>
@@ -816,6 +817,7 @@ static void __init mm_init(void)
 	 */
 	page_ext_init_flatmem();
 	init_debug_pagealloc();
+	kfence_alloc_pool();
 	report_meminit();
 	mem_init();
 	kmem_cache_init();
@@ -945,6 +947,7 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)
 	hrtimers_init();
 	softirq_init();
 	timekeeping_init();
+	kfence_init();
 
 	/*
 	 * For best initial stack canary entropy, prepare it after:
lib/Kconfig.debug +1 -0
index d7a7bc3b6098..052fcb2cf0c7 100644
@@ -878,6 +878,7 @@ config DEBUG_STACKOVERFLOW
 	  If in doubt, say "N".
 
 source "lib/Kconfig.kasan"
+source "lib/Kconfig.kfence"
 
 endmenu # "Memory Debugging"
 
lib/Kconfig.kfence new file +58 -0

58 lines not on this page · raw message

mm/Makefile +1 -0
index d73aed0fc99c..eb0993adb49e 100644
@@ -81,6 +81,7 @@ obj-$(CONFIG_PAGE_POISONING) += page_poison.o
 obj-$(CONFIG_SLAB) += slab.o
 obj-$(CONFIG_SLUB) += slub.o
 obj-$(CONFIG_KASAN)	+= kasan/
+obj-$(CONFIG_KFENCE) += kfence/
 obj-$(CONFIG_FAILSLAB) += failslab.o
 obj-$(CONFIG_MEMORY_HOTPLUG) += memory_hotplug.o
 obj-$(CONFIG_MEMTEST)		+= memtest.o
mm/kfence/Makefile new file +3 -0
new file mode 100644
index 000000000000..d991e9a349f0
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_KFENCE) := core.o report.o
mm/kfence/core.c new file +821 -0

821 lines not on this page · raw message

mm/kfence/kfence.h new file +107 -0

107 lines not on this page · raw message

mm/kfence/report.c new file +235 -0

235 lines not on this page · raw message

Signature · 1 line
2.29.1.341.ge80a0c044ae-goog
#2 lore raw

Re: [PATCH v6 1/9] mm: add Kernel Electric-Fence infrastructure

Quoted 3 lines · Marco Elver

This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors.

[...]

Quoted 1 line
include/linux/kfence.h +0 -0

[...]

Quoted 5 lines

+/**
+ * is_kfence_address() - check if an address belongs to KFENCE pool
+ * @addr: address to check
+ *
+ * Return: true or false depending on whether the address is within the KFENCE

Quoted 8 lines

+ * object range.
+ *
+ * KFENCE objects live in a separate page range and are not to be intermixed
+ * with regular heap objects (e.g. KFENCE objects must never be added to the
+ * allocator freelists). Failing to do so may and will result in heap
+ * corruptions, therefore is_kfence_address() must be used to check whether
+ * an object requires specific handling.
+ */

It might be worth noting in the comment that this is one of the few parts of KFENCE that are highly performance-sensitive, since that was an important point during the review.

Quoted 8 lines
include/linux/kfence.h +8 -0
+static __always_inline bool is_kfence_address(const void *addr)
+{
+       /*
+        * The non-NULL check is required in case the __kfence_pool pointer was
+        * never initialized; keep it in the slow-path after the range-check.
+        */
+       return unlikely((unsigned long)((char *)addr - __kfence_pool) < KFENCE_POOL_SIZE && addr);
+}

[...]

Quoted 1 line
lib/Kconfig.kfence +0 -0

[...]

Quoted 11 lines
lib/Kconfig.kfence +11 -0
+config KFENCE_STRESS_TEST_FAULTS
+       int "Stress testing of fault handling and error reporting"
+       default 0
+       depends on EXPERT
+       help
+         The inverse probability with which to randomly protect KFENCE object
+         pages, resulting in spurious use-after-frees. The main purpose of
+         this option is to stress test KFENCE with concurrent error reports
+         and allocations/frees. A value of 0 disables stress testing logic.
+
+         The option is only to test KFENCE; set to 0 if you are unsure.

[...]

Quoted 1 line
mm/kfence/core.c +0 -0

[...]

mm/kfence/core.c line 36

Quoted 3 lines
mm/kfence/core.c +3 -0
+#ifndef CONFIG_KFENCE_STRESS_TEST_FAULTS /* Only defined with CONFIG_EXPERT. */
+#define CONFIG_KFENCE_STRESS_TEST_FAULTS 0
+#endif

I think you can make this prettier by writing the Kconfig appropriately. See e.g. ARCH_MMAP_RND_BITS:

config ARCH_MMAP_RND_BITS
int "Number of bits to use for ASLR of mmap base address" if EXPERT range ARCH_MMAP_RND_BITS_MIN ARCH_MMAP_RND_BITS_MAX default ARCH_MMAP_RND_BITS_DEFAULT if ARCH_MMAP_RND_BITS_DEFAULT default ARCH_MMAP_RND_BITS_MIN
depends on HAVE_ARCH_MMAP_RND_BITS

So instead of 'depends on EXPERT', I think the proper way would be to append ' if EXPERT' to the line
'int "Stress testing of fault handling and error reporting"', so that only whether the option is user-visible depends on EXPERT, and non-EXPERT configs automatically use the default value.

[...]

Quoted 5 lines
mm/kfence/core.c +5 -0
+static inline unsigned long metadata_to_pageaddr(const struct kfence_metadata *meta)
+{
+       unsigned long offset = (meta - kfence_metadata + 1) * PAGE_SIZE * 2;
+       unsigned long pageaddr = (unsigned long)&__kfence_pool[offset];
+

mm/kfence/core.c line 170

Quoted 10 lines
mm/kfence/core.c +10 -0
+       /* The checks do not affect performance; only called from slow-paths. */
+
+       /* Only call with a pointer into kfence_metadata. */
+       if (KFENCE_WARN_ON(meta < kfence_metadata ||
+                          meta >= kfence_metadata + CONFIG_KFENCE_NUM_OBJECTS))
+               return 0;
+
+       /*
+        * This metadata object only ever maps to 1 page; verify the calculation
+        * happens and that the stored address was not corrupted.

nit: This reads a bit weirdly to me. Maybe "; verify that the stored address is in the expected range"? But feel free to leave it as-is if you prefer it that way.

Quoted 6 lines
mm/kfence/core.c +6 -0
+        */
+       if (KFENCE_WARN_ON(ALIGN_DOWN(meta->addr, PAGE_SIZE) != pageaddr))
+               return 0;
+
+       return pageaddr;
+}

[...]

mm/kfence/core.c line 235

Quoted 12 lines
mm/kfence/core.c +12 -0
+/* __always_inline this to ensure we won't do an indirect call to fn. */
+static __always_inline void for_each_canary(const struct kfence_metadata *meta, bool (*fn)(u8 *))
+{
+       const unsigned long pageaddr = ALIGN_DOWN(meta->addr, PAGE_SIZE);
+       unsigned long addr;
+
+       lockdep_assert_held(&meta->lock);
+
+       /* Check left of object. */
+       for (addr = pageaddr; addr < meta->addr; addr++) {
+               if (!fn((u8 *)addr))
+                       break;

It could be argued that "return" instead of "break" would be cleaner here if the API is supposed to be "invoke fn() on each canary byte, but stop when fn() returns false". But I suppose it doesn't really matter, so either way is fine.

Quoted 11 lines
mm/kfence/core.c +11 -0
+       }
+
+       /* Check right of object. */
+       for (addr = meta->addr + meta->size; addr < pageaddr + PAGE_SIZE; addr++) {
+               if (!fn((u8 *)addr))
+                       break;
+       }
+}
+
+static void *kfence_guarded_alloc(struct kmem_cache *cache, size_t size, gfp_t gfp)
+{

[...]

mm/kfence/core.c line 314

Quoted 7 lines
mm/kfence/core.c +7 -0
+       /* Set required struct page fields. */
+       page = virt_to_page(meta->addr);
+       page->slab_cache = cache;
+       if (IS_ENABLED(CONFIG_SLUB))
+               page->objects = 1;
+       if (IS_ENABLED(CONFIG_SLAB))
+               page->s_mem = addr;

Maybe move the last 4 lines over into the "hooks for SLAB" and "hooks for SLUB" patches?

[...]

Quoted 1 line

+}

[...]

Quoted 1 line
mm/kfence/report.c +0 -0

[...]

Quoted 2 lines

+/*
+ * Get the number of stack entries to skip get out of MM internals. @type is

s/to skip get out/to skip to get out/ ?

Quoted 4 lines

+ * optional, and if set to NULL, assumes an allocation or free stack.
+ */

mm/kfence/report.c +2 -0
+static int get_stack_skipnr(const unsigned long stack_entries[], int num_entries,
+                           const enum kfence_error_type *type)

[...]

Quoted 3 lines
mm/kfence/report.c +3 -0
+void kfence_report_error(unsigned long address, const struct kfence_metadata *meta,
+                        enum kfence_error_type type)
+{

[...]

mm/kfence/report.c line 200

Quoted 9 lines
mm/kfence/report.c +9 -0
+       case KFENCE_ERROR_CORRUPTION: {
+               size_t bytes_to_show = 16;
+
+               pr_err("BUG: KFENCE: memory corruption in %pS\n\n", (void *)stack_entries[skipnr]);
+               pr_err("Corrupted memory at 0x" PTR_FMT " ", (void *)address);
+
+               if (address < meta->addr)
+                       bytes_to_show = min(bytes_to_show, meta->addr - address);
+               print_diff_canary((u8 *)address, bytes_to_show);

If the object was located on the right side, but with 1 byte padding to the right due to alignment, and a 1-byte OOB write had clobbered the canary byte on the right side, we would later detect a KFENCE_ERROR_CORRUPTION at offset 0xfff inside the page, right? In that case, I think we'd end up trying to read 15 canary bytes from the following guard page and take a page fault?

You may want to do something like:

unsigned long canary_end = (address < meta->addr) ? meta->addr :
address | (PAGE_SIZE-1);
bytes_to_show = min(bytes_to_show, canary_end);

Quoted 3 lines
mm/kfence/report.c +3 -0
+               pr_cont(" (in kfence-#%zd):\n", object_index);
+               break;
+       }
author maintainer
#3 lore raw

Re: [PATCH v6 1/9] mm: add Kernel Electric-Fence infrastructure

Quoted 21 lines · Jann Horn
Quoted 3 lines · Marco Elver

This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors.

[...]

Quoted 1 line
include/linux/kfence.h +0 -0

[...]

Quoted 5 lines

+/**
+ * is_kfence_address() - check if an address belongs to KFENCE pool
+ * @addr: address to check
+ *
+ * Return: true or false depending on whether the address is within the KFENCE

Quoted 8 lines

+ * object range.
+ *
+ * KFENCE objects live in a separate page range and are not to be intermixed
+ * with regular heap objects (e.g. KFENCE objects must never be added to the
+ * allocator freelists). Failing to do so may and will result in heap
+ * corruptions, therefore is_kfence_address() must be used to check whether
+ * an object requires specific handling.
+ */

Quoted 3 lines

It might be worth noting in the comment that this is one of the few parts of KFENCE that are highly performance-sensitive, since that was an important point during the review.

Done, thanks.

Quoted 39 lines
Quoted 8 lines
include/linux/kfence.h +8 -0
+static __always_inline bool is_kfence_address(const void *addr)
+{
+       /*
+        * The non-NULL check is required in case the __kfence_pool pointer was
+        * never initialized; keep it in the slow-path after the range-check.
+        */
+       return unlikely((unsigned long)((char *)addr - __kfence_pool) < KFENCE_POOL_SIZE && addr);
+}

[...]

Quoted 1 line
lib/Kconfig.kfence +0 -0

[...]

Quoted 11 lines
lib/Kconfig.kfence +11 -0
+config KFENCE_STRESS_TEST_FAULTS
+       int "Stress testing of fault handling and error reporting"
+       default 0
+       depends on EXPERT
+       help
+         The inverse probability with which to randomly protect KFENCE object
+         pages, resulting in spurious use-after-frees. The main purpose of
+         this option is to stress test KFENCE with concurrent error reports
+         and allocations/frees. A value of 0 disables stress testing logic.
+
+         The option is only to test KFENCE; set to 0 if you are unsure.

[...]

Quoted 1 line
mm/kfence/core.c +0 -0

[...]

mm/kfence/core.c line 36

Quoted 3 lines
mm/kfence/core.c +3 -0
+#ifndef CONFIG_KFENCE_STRESS_TEST_FAULTS /* Only defined with CONFIG_EXPERT. */
+#define CONFIG_KFENCE_STRESS_TEST_FAULTS 0
+#endif

I think you can make this prettier by writing the Kconfig appropriately. See e.g. ARCH_MMAP_RND_BITS:

config ARCH_MMAP_RND_BITS
int "Number of bits to use for ASLR of mmap base address" if EXPERT range ARCH_MMAP_RND_BITS_MIN ARCH_MMAP_RND_BITS_MAX default ARCH_MMAP_RND_BITS_DEFAULT if ARCH_MMAP_RND_BITS_DEFAULT default ARCH_MMAP_RND_BITS_MIN
depends on HAVE_ARCH_MMAP_RND_BITS

Quoted 5 lines

So instead of 'depends on EXPERT', I think the proper way would be to append ' if EXPERT' to the line
'int "Stress testing of fault handling and error reporting"', so that only whether the option is user-visible depends on EXPERT, and non-EXPERT configs automatically use the default value.

I guess the idea was to not pollute the config in non-EXPERT configs, but it probably doesn't matter much. Changed it to the suggested cleaner approach.

Quoted 12 lines

[...]

Quoted 11 lines
mm/kfence/core.c +11 -0
+static inline unsigned long metadata_to_pageaddr(const struct kfence_metadata *meta)
+{
+       unsigned long offset = (meta - kfence_metadata + 1) * PAGE_SIZE * 2;
+       unsigned long pageaddr = (unsigned long)&__kfence_pool[offset];
+
+       /* The checks do not affect performance; only called from slow-paths. */
+
+       /* Only call with a pointer into kfence_metadata. */
+       if (KFENCE_WARN_ON(meta < kfence_metadata ||
+                          meta >= kfence_metadata + CONFIG_KFENCE_NUM_OBJECTS))
+               return 0;
Quoted 8 lines

mm/kfence/core.c line 170

Quoted 4 lines
mm/kfence/core.c +4 -0
+
+       /*
+        * This metadata object only ever maps to 1 page; verify the calculation
+        * happens and that the stored address was not corrupted.

nit: This reads a bit weirdly to me. Maybe "; verify that the stored address is in the expected range"? But feel free to leave it as-is if you prefer it that way.

Hmm, that really sounds weird... I've changed it. :-)

Quoted 16 lines
Quoted 6 lines
mm/kfence/core.c +6 -0
+        */
+       if (KFENCE_WARN_ON(ALIGN_DOWN(meta->addr, PAGE_SIZE) != pageaddr))
+               return 0;
+
+       return pageaddr;
+}

[...]

Quoted 9 lines
mm/kfence/core.c +9 -0
+/* __always_inline this to ensure we won't do an indirect call to fn. */
+static __always_inline void for_each_canary(const struct kfence_metadata *meta, bool (*fn)(u8 *))
+{
+       const unsigned long pageaddr = ALIGN_DOWN(meta->addr, PAGE_SIZE);
+       unsigned long addr;
+
+       lockdep_assert_held(&meta->lock);
+
+       /* Check left of object. */
Quoted 8 lines

mm/kfence/core.c line 235

Quoted 3 lines
mm/kfence/core.c +3 -0
+       for (addr = pageaddr; addr < meta->addr; addr++) {
+               if (!fn((u8 *)addr))
+                       break;

It could be argued that "return" instead of "break" would be cleaner here if the API is supposed to be "invoke fn() on each canary byte, but stop when fn() returns false". But I suppose it doesn't really matter, so either way is fine.

Hmm, perhaps if there are corruptions on either side of an object printing both errors (which includes indications of which bytes were corrupted) might give more insights into what went wrong. Printing errors for every canary byte on one side didn't make much sense though, hence the break.

Until we see this in the wild, let's err on the side of "more information might be better".

Quoted 20 lines
Quoted 11 lines
mm/kfence/core.c +11 -0
+       }
+
+       /* Check right of object. */
+       for (addr = meta->addr + meta->size; addr < pageaddr + PAGE_SIZE; addr++) {
+               if (!fn((u8 *)addr))
+                       break;
+       }
+}
+
+static void *kfence_guarded_alloc(struct kmem_cache *cache, size_t size, gfp_t gfp)
+{

[...]

mm/kfence/core.c line 314

Quoted 7 lines
mm/kfence/core.c +7 -0
+       /* Set required struct page fields. */
+       page = virt_to_page(meta->addr);
+       page->slab_cache = cache;
+       if (IS_ENABLED(CONFIG_SLUB))
+               page->objects = 1;
+       if (IS_ENABLED(CONFIG_SLAB))
+               page->s_mem = addr;
Quoted 2 lines

Maybe move the last 4 lines over into the "hooks for SLAB" and "hooks for SLUB" patches?

Done.

Quoted 9 lines

[...]

Quoted 1 line

+}

[...]

Quoted 1 line
mm/kfence/report.c +0 -0

[...]

Quoted 2 lines

+/*
+ * Get the number of stack entries to skip get out of MM internals. @type is

s/to skip get out/to skip to get out/ ?

Done.

Quoted 26 lines
Quoted 4 lines

+ * optional, and if set to NULL, assumes an allocation or free stack.
+ */

mm/kfence/report.c +2 -0
+static int get_stack_skipnr(const unsigned long stack_entries[], int num_entries,
+                           const enum kfence_error_type *type)

[...]

Quoted 3 lines
mm/kfence/report.c +3 -0
+void kfence_report_error(unsigned long address, const struct kfence_metadata *meta,
+                        enum kfence_error_type type)
+{

[...]

mm/kfence/report.c line 200

Quoted 9 lines
mm/kfence/report.c +9 -0
+       case KFENCE_ERROR_CORRUPTION: {
+               size_t bytes_to_show = 16;
+
+               pr_err("BUG: KFENCE: memory corruption in %pS\n\n", (void *)stack_entries[skipnr]);
+               pr_err("Corrupted memory at 0x" PTR_FMT " ", (void *)address);
+
+               if (address < meta->addr)
+                       bytes_to_show = min(bytes_to_show, meta->addr - address);
+               print_diff_canary((u8 *)address, bytes_to_show);

If the object was located on the right side, but with 1 byte padding to the right due to alignment, and a 1-byte OOB write had clobbered the canary byte on the right side, we would later detect a KFENCE_ERROR_CORRUPTION at offset 0xfff inside the page, right? In that case, I think we'd end up trying to read 15 canary bytes from the following guard page and take a page fault?

Quoted 5 lines

You may want to do something like:

unsigned long canary_end = (address < meta->addr) ? meta->addr :
address | (PAGE_SIZE-1);
bytes_to_show = min(bytes_to_show, canary_end);

print_diff_canary() calculates max_addr using PAGE_ALIGN(), and we won't read from the next page. I think I'll move all this logic into print_diff_canary() to simplify.

Thanks,
-- Marco

author maintainer
#4 lore raw

PATCH v6 2/9 x86, kfence: enable KFENCE for x86

In reply to Marco Elver

From: Alexander Potapenko <[email protected]>

Add architecture specific implementation details for KFENCE and enable KFENCE for the x86 architecture. In particular, this implements the required interface in <asm/kfence.h> for setting up the pool and providing helper functions for protecting and unprotecting pages.

For x86, we need to ensure that the pool uses 4K pages, which is done using the set_memory_4k() helper function.

---
v5:
* MAJOR CHANGE: Switch to the memblock_alloc'd pool. Running benchmarks with the newly optimized is_kfence_address(), no difference between baseline and KFENCE is observed.
* Suggested by Jann Horn:
* Move x86 kfence_handle_page_fault before oops handling.
* WARN_ON in kfence_protect_page if non-4K pages.
* Better comments for x86 kfence_protect_page.

v4:
* Define __kfence_pool_attrs.

---
 arch/x86/Kconfig              |  1 +
 arch/x86/include/asm/kfence.h | 65 +++++++++++++++++++++++++++++++++++
 arch/x86/mm/fault.c           |  4 +++
 3 files changed, 70 insertions(+)
 create mode 100644 arch/x86/include/asm/kfence.h
arch/x86/Kconfig +1 -0
index f6946b81f74a..c9ec6b5ba358 100644
@@ -144,6 +144,7 @@ config X86
 	select HAVE_ARCH_JUMP_LABEL_RELATIVE
 	select HAVE_ARCH_KASAN			if X86_64
 	select HAVE_ARCH_KASAN_VMALLOC		if X86_64
+	select HAVE_ARCH_KFENCE
 	select HAVE_ARCH_KGDB
 	select HAVE_ARCH_MMAP_RND_BITS		if MMU
 	select HAVE_ARCH_MMAP_RND_COMPAT_BITS	if MMU && COMPAT
arch/x86/include/asm/kfence.h new file +65 -0

65 lines not on this page · raw message

arch/x86/mm/fault.c +4 -0
index 82bf37a5c9ec..380638745f42 100644
@@ -9,6 +9,7 @@
 #include <linux/kdebug.h>		/* oops_begin/end, ...		*/
 #include <linux/extable.h>		/* search_exception_tables	*/
 #include <linux/memblock.h>		/* max_low_pfn			*/
+#include <linux/kfence.h>		/* kfence_handle_page_fault	*/
 #include <linux/kprobes.h>		/* NOKPROBE_SYMBOL, ...		*/
 #include <linux/mmiotrace.h>		/* kmmio_handler, ...		*/
 #include <linux/perf_event.h>		/* perf_sw_event		*/
@@ -725,6 +726,9 @@ no_context(struct pt_regs *regs, unsigned long error_code,
 	if (IS_ENABLED(CONFIG_EFI))
 		efi_recover_from_page_fault(address);
 
+	if (kfence_handle_page_fault(address))
+		return;
+
 oops:
 	/*
 	 * Oops. The kernel tried to access some bad page. We'll have to
Signature · 1 line
2.29.1.341.ge80a0c044ae-goog
#5 lore raw

Re: [PATCH v6 2/9] x86, kfence: enable KFENCE for x86

Quoted 12 lines · Marco Elver

Add architecture specific implementation details for KFENCE and enable KFENCE for the x86 architecture. In particular, this implements the required interface in <asm/kfence.h> for setting up the pool and providing helper functions for protecting and unprotecting pages.

For x86, we need to ensure that the pool uses 4K pages, which is done using the set_memory_4k() helper function.

[...]

Quoted 1 line
arch/x86/mm/fault.c +0 -0

[...]

arch/x86/mm/fault.c no_context(struct pt_regs *regs, unsigned long error_code, · line 730

Quoted 6 lines
arch/x86/mm/fault.c +2 -0
@@ -725,6 +726,9 @@ no_context(struct pt_regs *regs, unsigned long error_code,
        if (IS_ENABLED(CONFIG_EFI))
                efi_recover_from_page_fault(address);
 
+       if (kfence_handle_page_fault(address))
+               return;

We can also get to this point due to an attempt to execute a data page. That's very unlikely (given that the same thing would also crash if you tried to do it with normal heap memory, and KFENCE allocations are extremely rare); but we might want to try to avoid handling such faults as KFENCE faults, since KFENCE will assume that it has resolved the fault and retry execution of the faulting instruction. Once kernel protection keys are introduced, those might cause the same kind of trouble.

So we might want to gate this on a check like "if ((error_code & X86_PF_PROT) == 0)" (meaning "only handle the fault if the fault was caused by no page being present", see enum x86_pf_error_code).

Unrelated sidenote: Since we're hooking after exception fixup handling, the debug-only KFENCE_STRESS_TEST_FAULTS can probably still cause some behavioral differences through spurious faults in places like copy_user_enhanced_fast_string (where the exception table entries are used even if the *kernel* pointer, not the user pointer, causes a fault). But since KFENCE_STRESS_TEST_FAULTS is exclusively for KFENCE development, the difference might not matter. And ordering them the other way around definitely isn't possible, because the kernel relies on being able to fixup OOB reads. So there probably isn't really anything we can do better here; it's just something to keep in mind. Maybe you can add a little warning to the help text for that Kconfig entry that warns people about this?

Quoted 7 lines
arch/x86/mm/fault.c +1 -1
+
 oops:
        /*
         * Oops. The kernel tried to access some bad page. We'll have to
--

2.29.1.341.ge80a0c044ae-goog

author maintainer
#6 lore raw

Re: [PATCH v6 2/9] x86, kfence: enable KFENCE for x86

Quoted 32 lines · Jann Horn
Quoted 12 lines · Marco Elver

Add architecture specific implementation details for KFENCE and enable KFENCE for the x86 architecture. In particular, this implements the required interface in <asm/kfence.h> for setting up the pool and providing helper functions for protecting and unprotecting pages.

For x86, we need to ensure that the pool uses 4K pages, which is done using the set_memory_4k() helper function.

[...]

Quoted 1 line
arch/x86/mm/fault.c +0 -0

[...]

arch/x86/mm/fault.c no_context(struct pt_regs *regs, unsigned long error_code, · line 730

Quoted 6 lines
arch/x86/mm/fault.c +2 -0
@@ -725,6 +726,9 @@ no_context(struct pt_regs *regs, unsigned long error_code,
        if (IS_ENABLED(CONFIG_EFI))
                efi_recover_from_page_fault(address);
 
+       if (kfence_handle_page_fault(address))
+               return;

We can also get to this point due to an attempt to execute a data page. That's very unlikely (given that the same thing would also crash if you tried to do it with normal heap memory, and KFENCE allocations are extremely rare); but we might want to try to avoid handling such faults as KFENCE faults, since KFENCE will assume that it has resolved the fault and retry execution of the faulting instruction. Once kernel protection keys are introduced, those might cause the same kind of trouble.

Quoted 3 lines

So we might want to gate this on a check like "if ((error_code & X86_PF_PROT) == 0)" (meaning "only handle the fault if the fault was caused by no page being present", see enum x86_pf_error_code).

Good point. Will fix in v7.

Quoted 4 lines

Unrelated sidenote: Since we're hooking after exception fixup handling, the debug-only KFENCE_STRESS_TEST_FAULTS can probably still cause some behavioral differences through spurious faults in places like copy_user_enhanced_fast_string (where the exception table entries

Quoted 8 lines

are used even if the *kernel* pointer, not the user pointer, causes a fault). But since KFENCE_STRESS_TEST_FAULTS is exclusively for KFENCE development, the difference might not matter. And ordering them the other way around definitely isn't possible, because the kernel relies on being able to fixup OOB reads. So there probably isn't really anything we can do better here; it's just something to keep in mind. Maybe you can add a little warning to the help text for that Kconfig entry that warns people about this?

Thanks for pointing it out, but that option really is *only* to stress kfence with concurrent allocations/frees/page faults. If anybody enables this option for anything other than testing kfence, it's their own fault. ;-)
I'll try to add a generic note to the Kconfig entry, but what you mention here seems quite x86-specific.

Thanks,
-- Marco

#7 lore raw

Re: [PATCH v6 2/9] x86, kfence: enable KFENCE for x86

Quoted 23 lines · Marco Elver
Quoted 22 lines · Jann Horn
Quoted 12 lines · Marco Elver

Add architecture specific implementation details for KFENCE and enable KFENCE for the x86 architecture. In particular, this implements the required interface in <asm/kfence.h> for setting up the pool and providing helper functions for protecting and unprotecting pages.

For x86, we need to ensure that the pool uses 4K pages, which is done using the set_memory_4k() helper function.

[...]

Quoted 1 line
arch/x86/mm/fault.c +0 -0

[...]

Quoted 6 lines
arch/x86/mm/fault.c +2 -0
@@ -725,6 +726,9 @@ no_context(struct pt_regs *regs, unsigned long error_code,
        if (IS_ENABLED(CONFIG_EFI))
                efi_recover_from_page_fault(address);
 
+       if (kfence_handle_page_fault(address))
+               return;

[...]

Quoted 13 lines
Quoted 4 lines

Unrelated sidenote: Since we're hooking after exception fixup handling, the debug-only KFENCE_STRESS_TEST_FAULTS can probably still cause some behavioral differences through spurious faults in places like copy_user_enhanced_fast_string (where the exception table entries

Quoted 8 lines

are used even if the *kernel* pointer, not the user pointer, causes a fault). But since KFENCE_STRESS_TEST_FAULTS is exclusively for KFENCE development, the difference might not matter. And ordering them the other way around definitely isn't possible, because the kernel relies on being able to fixup OOB reads. So there probably isn't really anything we can do better here; it's just something to keep in mind. Maybe you can add a little warning to the help text for that Kconfig entry that warns people about this?

Quoted 4 lines

Thanks for pointing it out, but that option really is *only* to stress kfence with concurrent allocations/frees/page faults. If anybody enables this option for anything other than testing kfence, it's their own fault. ;-)

Sounds fair. :P

Quoted 2 lines

I'll try to add a generic note to the Kconfig entry, but what you mention here seems quite x86-specific.

(FWIW, I think it could currently also happen on arm64 in the rare cases where KERNEL_DS is used. But luckily Christoph Hellwig has already gotten rid of most places that did that.)

author maintainer
#8 lore raw

PATCH v6 3/9 arm64, kfence: enable KFENCE for ARM64

In reply to Marco Elver

Add architecture specific implementation details for KFENCE and enable KFENCE for the arm64 architecture. In particular, this implements the required interface in <asm/kfence.h>.

KFENCE requires that attributes for pages from its memory pool can individually be set. Therefore, force the entire linear map to be mapped at page granularity. Doing so may result in extra memory allocated for page tables in case rodata=full is not set; however, currently CONFIG_RODATA_FULL_DEFAULT_ENABLED=y is the default, and the common case is therefore not affected by this change.

---
v5:
* Move generic page allocation code to core.c [suggested by Jann Horn].
* Remove comment about HAVE_ARCH_KFENCE_STATIC_POOL, since we no longer support static pools.
* Force page granularity for the linear map [suggested by Mark Rutland].

---
 arch/arm64/Kconfig              |  1 +
 arch/arm64/include/asm/kfence.h | 19 +++++++++++++++++++
 arch/arm64/mm/fault.c           |  4 ++++
 arch/arm64/mm/mmu.c             |  7 ++++++-
 4 files changed, 30 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/include/asm/kfence.h
arch/arm64/Kconfig +1 -0
index f858c352f72a..2f8b32dddd8b 100644
@@ -135,6 +135,7 @@ config ARM64
 	select HAVE_ARCH_JUMP_LABEL_RELATIVE
 	select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48)
 	select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN
+	select HAVE_ARCH_KFENCE if (!ARM64_16K_PAGES && !ARM64_64K_PAGES)
 	select HAVE_ARCH_KGDB
 	select HAVE_ARCH_MMAP_RND_BITS
 	select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT
arch/arm64/include/asm/kfence.h new file +19 -0
new file mode 100644
index 000000000000..5ac0f599cc9a
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef __ASM_KFENCE_H
+#define __ASM_KFENCE_H
+
+#include <asm/cacheflush.h>
+
+#define KFENCE_SKIP_ARCH_FAULT_HANDLER "el1_sync"
+
+static inline bool arch_kfence_init_pool(void) { return true; }
+
+static inline bool kfence_protect_page(unsigned long addr, bool protect)
+{
+	set_memory_valid(addr, 1, !protect);
+
+	return true;
+}
+
+#endif /* __ASM_KFENCE_H */
arch/arm64/mm/fault.c +4 -0
index 94c99c1c19e3..ec8ed2943484 100644
@@ -10,6 +10,7 @@
 #include <linux/acpi.h>
 #include <linux/bitfield.h>
 #include <linux/extable.h>
+#include <linux/kfence.h>
 #include <linux/signal.h>
 #include <linux/mm.h>
 #include <linux/hardirq.h>
@@ -312,6 +313,9 @@ static void __do_kernel_fault(unsigned long addr, unsigned int esr,
 	    "Ignoring spurious kernel translation fault at virtual address %016lx\n", addr))
 		return;
 
+	if (kfence_handle_page_fault(addr))
+		return;
+
 	if (is_el1_permission_fault(addr, esr, regs)) {
 		if (esr & ESR_ELx_WNR)
 			msg = "write to read-only memory";
arch/arm64/mm/mmu.c +6 -1
index 1c0f3e02f731..86be6d1a78ab 100644
@@ -1449,7 +1449,12 @@ int arch_add_memory(int nid, u64 start, u64 size,
 {
 	int ret, flags = 0;
 
-	if (rodata_full || debug_pagealloc_enabled())
+	/*
+	 * KFENCE requires linear map to be mapped at page granularity, so that
+	 * it is possible to protect/unprotect single pages in the KFENCE pool.
+	 */
+	if (rodata_full || debug_pagealloc_enabled() ||
+	    IS_ENABLED(CONFIG_KFENCE))
 		flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS;
 
 	__create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start),
Signature · 1 line
2.29.1.341.ge80a0c044ae-goog
#9 lore raw

Re: [PATCH v6 3/9] arm64, kfence: enable KFENCE for ARM64

Quoted 10 lines · Marco Elver

Add architecture specific implementation details for KFENCE and enable KFENCE for the arm64 architecture. In particular, this implements the required interface in <asm/kfence.h>.

KFENCE requires that attributes for pages from its memory pool can individually be set. Therefore, force the entire linear map to be mapped at page granularity. Doing so may result in extra memory allocated for page tables in case rodata=full is not set; however, currently CONFIG_RODATA_FULL_DEFAULT_ENABLED=y is the default, and the common case is therefore not affected by this change.

[...]

Quoted 1 line
arch/arm64/Kconfig +0 -0

[...]

Quoted 1 line

+ select HAVE_ARCH_KFENCE if (!ARM64_16K_PAGES && !ARM64_64K_PAGES)

"if ARM64_4K_PAGES"?

[...]

Quoted 1 line
arch/arm64/mm/fault.c +0 -0

[...]

arch/arm64/mm/fault.c static void __do_kernel_fault(unsigned long addr, unsigned int esr, · line 317

Quoted 6 lines
arch/arm64/mm/fault.c +2 -0
@@ -312,6 +313,9 @@ static void __do_kernel_fault(unsigned long addr, unsigned int esr,
            "Ignoring spurious kernel translation fault at virtual address %016lx\n", addr))
                return;
 
+       if (kfence_handle_page_fault(addr))
+               return;

As in the X86 case, we may want to ensure that this doesn't run for permission faults, only for non-present pages. Maybe move this down into the third branch of the "if" block below (neither permission fault nor NULL deref)?

Quoted 4 lines
arch/arm64/mm/fault.c +1 -0
+
        if (is_el1_permission_fault(addr, esr, regs)) {
                if (esr & ESR_ELx_WNR)
                        msg = "write to read-only memory";
reviewer
#10 lore raw

Re: [PATCH v6 3/9] arm64, kfence: enable KFENCE for ARM64

Quoted 5 lines · Jann Horn
Quoted 4 lines · Marco Elver
arch/arm64/mm/fault.c +0 -0
@@ -312,6 +313,9 @@ static void __do_kernel_fault(unsigned long addr, unsigned int esr,
            "Ignoring spurious kernel translation fault at virtual address %016lx\n", addr))
                return;
 
Quoted 7 lines

arch/arm64/mm/fault.c static void __do_kernel_fault(unsigned long addr, unsigned int esr, · line 317

Quoted 2 lines
arch/arm64/mm/fault.c +2 -0
+       if (kfence_handle_page_fault(addr))
+               return;

As in the X86 case, we may want to ensure that this doesn't run for permission faults, only for non-present pages. Maybe move this down into the third branch of the "if" block below (neither permission fault nor NULL deref)?

I think that'd make sense. Those cases *should* be mutually exclusive, but it'd be more robust to do the KFENCE checks in that last block so that if something goes wrong wrong within KFENCE we can't get stuck in a loop failing to service an instruction abort or similar.

Either that, or factor out an is_el1_translation_fault() and only do the KFENCE check and is_spurious_el1_translation_fault() check under that.

Thanks,
Mark.

reviewer
#11 lore raw

Re: [PATCH v6 3/9] arm64, kfence: enable KFENCE for ARM64

In reply to Marco Elver
Quoted 34 lines · Marco Elver

Add architecture specific implementation details for KFENCE and enable KFENCE for the arm64 architecture. In particular, this implements the required interface in <asm/kfence.h>.

KFENCE requires that attributes for pages from its memory pool can individually be set. Therefore, force the entire linear map to be mapped at page granularity. Doing so may result in extra memory allocated for page tables in case rodata=full is not set; however, currently CONFIG_RODATA_FULL_DEFAULT_ENABLED=y is the default, and the common case is therefore not affected by this change.

---
v5:
* Move generic page allocation code to core.c [suggested by Jann Horn].
* Remove comment about HAVE_ARCH_KFENCE_STATIC_POOL, since we no longer support static pools.
* Force page granularity for the linear map [suggested by Mark Rutland].

---
 arch/arm64/Kconfig              |  1 +
 arch/arm64/include/asm/kfence.h | 19 +++++++++++++++++++
 arch/arm64/mm/fault.c           |  4 ++++
 arch/arm64/mm/mmu.c             |  7 ++++++-
 4 files changed, 30 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/include/asm/kfence.h
arch/arm64/Kconfig +0 -0
index f858c352f72a..2f8b32dddd8b 100644

arch/arm64/Kconfig config ARM64 · line 138

Quoted 4 lines
arch/arm64/Kconfig +1 -0
@@ -135,6 +135,7 @@ config ARM64
 	select HAVE_ARCH_JUMP_LABEL_RELATIVE
 	select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48)
 	select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN
+	select HAVE_ARCH_KFENCE if (!ARM64_16K_PAGES && !ARM64_64K_PAGES)

Why does this depend on the page size?

If this is functional, but has a larger overhead on 16K or 64K, I'd suggest removing the dependency, and just updating the Kconfig help text to explain that.

Otherwise, this patch looks fine to me.

Thanks,
Mark.

author maintainer
#12 lore raw

Re: [PATCH v6 3/9] arm64, kfence: enable KFENCE for ARM64

Quoted 43 lines · Mark Rutland
Quoted 34 lines · Marco Elver

Add architecture specific implementation details for KFENCE and enable KFENCE for the arm64 architecture. In particular, this implements the required interface in <asm/kfence.h>.

KFENCE requires that attributes for pages from its memory pool can individually be set. Therefore, force the entire linear map to be mapped at page granularity. Doing so may result in extra memory allocated for page tables in case rodata=full is not set; however, currently CONFIG_RODATA_FULL_DEFAULT_ENABLED=y is the default, and the common case is therefore not affected by this change.

---
v5:
* Move generic page allocation code to core.c [suggested by Jann Horn].
* Remove comment about HAVE_ARCH_KFENCE_STATIC_POOL, since we no longer support static pools.
* Force page granularity for the linear map [suggested by Mark Rutland].

---
 arch/arm64/Kconfig              |  1 +
 arch/arm64/include/asm/kfence.h | 19 +++++++++++++++++++
 arch/arm64/mm/fault.c           |  4 ++++
 arch/arm64/mm/mmu.c             |  7 ++++++-
 4 files changed, 30 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/include/asm/kfence.h
arch/arm64/Kconfig +0 -0
index f858c352f72a..2f8b32dddd8b 100644

arch/arm64/Kconfig config ARM64 · line 138

Quoted 4 lines
arch/arm64/Kconfig +1 -0
@@ -135,6 +135,7 @@ config ARM64
      select HAVE_ARCH_JUMP_LABEL_RELATIVE
      select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48)
      select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN
+     select HAVE_ARCH_KFENCE if (!ARM64_16K_PAGES && !ARM64_64K_PAGES)

Why does this depend on the page size?

Quoted 3 lines

If this is functional, but has a larger overhead on 16K or 64K, I'd suggest removing the dependency, and just updating the Kconfig help text to explain that.

Good point, I don't think anything is requiring us to force 4K pages. Let's remove it.

Thanks,
-- Marco

Quoted 4 lines

Otherwise, this patch looks fine to me.

Thanks,
Mark.

author maintainer
#13 lore raw

PATCH v6 4/9 mm, kfence: insert KFENCE hooks for SLAB

In reply to Marco Elver

From: Alexander Potapenko <[email protected]>

Inserts KFENCE hooks into the SLAB allocator.

To pass the originally requested size to KFENCE, add an argument 'orig_size' to slab_alloc*(). The additional argument is required to preserve the requested original size for kmalloc() allocations, which uses size classes (e.g. an allocation of 272 bytes will return an object of size 512). Therefore, kmem_cache::size does not represent the kmalloc-caller's requested size, and we must introduce the argument 'orig_size' to propagate the originally requested size to KFENCE.

Without the originally requested size, we would not be able to detect out-of-bounds accesses for objects placed at the end of a KFENCE object page if that object is not equal to the kmalloc-size class it was bucketed into.

When KFENCE is disabled, there is no additional overhead, since slab_alloc*() functions are __always_inline.

---
v5:
* New kfence_shutdown_cache(): we need to defer kfence_shutdown_cache() to before the cache is actually freed. In case of SLAB_TYPESAFE_BY_RCU, the objects may still legally be used until the next RCU grace period.
* Fix objs_per_slab_page for kfence objects.
* Revert and use fixed obj_to_index() in __check_heap_object().

v3:
* Rewrite patch description to clarify need for 'orig_size' [reported by Christopher Lameter].

---
 include/linux/slab_def.h |  3 +++
 mm/slab.c                | 37 ++++++++++++++++++++++++++++---------
 mm/slab_common.c         |  5 ++++-
 3 files changed, 35 insertions(+), 10 deletions(-)
include/linux/slab_def.h +3 -0
index 9eb430c163c2..3aa5e1e73ab6 100644
@@ -2,6 +2,7 @@
 #ifndef _LINUX_SLAB_DEF_H
 #define	_LINUX_SLAB_DEF_H
 
+#include <linux/kfence.h>
 #include <linux/reciprocal_div.h>
 
 /*
@@ -114,6 +115,8 @@ static inline unsigned int obj_to_index(const struct kmem_cache *cache,
 static inline int objs_per_slab_page(const struct kmem_cache *cache,
 				     const struct page *page)
 {
+	if (is_kfence_address(page_address(page)))
+		return 1;
 	return cache->num;
 }
 
mm/slab.c +28 -9
index b1113561b98b..ebff1c333558 100644
@@ -100,6 +100,7 @@
 #include	<linux/seq_file.h>
 #include	<linux/notifier.h>
 #include	<linux/kallsyms.h>
+#include	<linux/kfence.h>
 #include	<linux/cpu.h>
 #include	<linux/sysctl.h>
 #include	<linux/module.h>
@@ -3208,7 +3209,7 @@ static void *____cache_alloc_node(struct kmem_cache *cachep, gfp_t flags,
 }
 
 static __always_inline void *
-slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,
+slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, size_t orig_size,
 		   unsigned long caller)
 {
 	unsigned long save_flags;
@@ -3221,6 +3222,10 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,
 	if (unlikely(!cachep))
 		return NULL;
 
+	ptr = kfence_alloc(cachep, orig_size, flags);
+	if (unlikely(ptr))
+		goto out_hooks;
+
 	cache_alloc_debugcheck_before(cachep, flags);
 	local_irq_save(save_flags);
 
@@ -3253,6 +3258,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,
 	if (unlikely(slab_want_init_on_alloc(flags, cachep)) && ptr)
 		memset(ptr, 0, cachep->object_size);
 
+out_hooks:
 	slab_post_alloc_hook(cachep, objcg, flags, 1, &ptr);
 	return ptr;
 }
@@ -3290,7 +3296,7 @@ __do_cache_alloc(struct kmem_cache *cachep, gfp_t flags)
 #endif /* CONFIG_NUMA */
 
 static __always_inline void *
-slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller)
+slab_alloc(struct kmem_cache *cachep, gfp_t flags, size_t orig_size, unsigned long caller)
 {
 	unsigned long save_flags;
 	void *objp;
@@ -3301,6 +3307,10 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller)
 	if (unlikely(!cachep))
 		return NULL;
 
+	objp = kfence_alloc(cachep, orig_size, flags);
+	if (unlikely(objp))
+		goto out;
+
 	cache_alloc_debugcheck_before(cachep, flags);
 	local_irq_save(save_flags);
 	objp = __do_cache_alloc(cachep, flags);
@@ -3311,6 +3321,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller)
 	if (unlikely(slab_want_init_on_alloc(flags, cachep)) && objp)
 		memset(objp, 0, cachep->object_size);
 
+out:
 	slab_post_alloc_hook(cachep, objcg, flags, 1, &objp);
 	return objp;
 }
@@ -3416,6 +3427,11 @@ static void cache_flusharray(struct kmem_cache *cachep, struct array_cache *ac)
 static __always_inline void __cache_free(struct kmem_cache *cachep, void *objp,
 					 unsigned long caller)
 {
+	if (kfence_free(objp)) {
+		kmemleak_free_recursive(objp, cachep->flags);
+		return;
+	}
+
 	/* Put the object into the quarantine, don't touch it for now. */
 	if (kasan_slab_free(cachep, objp, _RET_IP_))
 		return;
@@ -3481,7 +3497,7 @@ void ___cache_free(struct kmem_cache *cachep, void *objp,
  */
 void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags)
 {
-	void *ret = slab_alloc(cachep, flags, _RET_IP_);
+	void *ret = slab_alloc(cachep, flags, cachep->object_size, _RET_IP_);
 
 	trace_kmem_cache_alloc(_RET_IP_, ret,
 			       cachep->object_size, cachep->size, flags);
@@ -3514,7 +3530,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,
 
 	local_irq_disable();
 	for (i = 0; i < size; i++) {
-		void *objp = __do_cache_alloc(s, flags);
+		void *objp = kfence_alloc(s, s->object_size, flags) ?: __do_cache_alloc(s, flags);
 
 		if (unlikely(!objp))
 			goto error;
@@ -3547,7 +3563,7 @@ kmem_cache_alloc_trace(struct kmem_cache *cachep, gfp_t flags, size_t size)
 {
 	void *ret;
 
-	ret = slab_alloc(cachep, flags, _RET_IP_);
+	ret = slab_alloc(cachep, flags, size, _RET_IP_);
 
 	ret = kasan_kmalloc(cachep, ret, size, flags);
 	trace_kmalloc(_RET_IP_, ret,
@@ -3573,7 +3589,7 @@ EXPORT_SYMBOL(kmem_cache_alloc_trace);
  */
 void *kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid)
 {
-	void *ret = slab_alloc_node(cachep, flags, nodeid, _RET_IP_);
+	void *ret = slab_alloc_node(cachep, flags, nodeid, cachep->object_size, _RET_IP_);
 
 	trace_kmem_cache_alloc_node(_RET_IP_, ret,
 				    cachep->object_size, cachep->size,
@@ -3591,7 +3607,7 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep,
 {
 	void *ret;
 
-	ret = slab_alloc_node(cachep, flags, nodeid, _RET_IP_);
+	ret = slab_alloc_node(cachep, flags, nodeid, size, _RET_IP_);
 
 	ret = kasan_kmalloc(cachep, ret, size, flags);
 	trace_kmalloc_node(_RET_IP_, ret,
@@ -3652,7 +3668,7 @@ static __always_inline void *__do_kmalloc(size_t size, gfp_t flags,
 	cachep = kmalloc_slab(size, flags);
 	if (unlikely(ZERO_OR_NULL_PTR(cachep)))
 		return cachep;
-	ret = slab_alloc(cachep, flags, caller);
+	ret = slab_alloc(cachep, flags, size, caller);
 
 	ret = kasan_kmalloc(cachep, ret, size, flags);
 	trace_kmalloc(caller, ret,
@@ -4151,7 +4167,10 @@ void __check_heap_object(const void *ptr, unsigned long n, struct page *page,
 	BUG_ON(objnr >= cachep->num);
 
 	/* Find offset within object. */
-	offset = ptr - index_to_obj(cachep, page, objnr) - obj_offset(cachep);
+	if (is_kfence_address(ptr))
+		offset = ptr - kfence_object_start(ptr);
+	else
+		offset = ptr - index_to_obj(cachep, page, objnr) - obj_offset(cachep);
 
 	/* Allow address range falling entirely within usercopy region. */
 	if (offset >= cachep->useroffset &&
mm/slab_common.c +4 -1
index f9ccd5dc13f3..13125773dae2 100644
@@ -12,6 +12,7 @@
 #include <linux/memory.h>
 #include <linux/cache.h>
 #include <linux/compiler.h>
+#include <linux/kfence.h>
 #include <linux/module.h>
 #include <linux/cpu.h>
 #include <linux/uaccess.h>
@@ -435,6 +436,7 @@ static void slab_caches_to_rcu_destroy_workfn(struct work_struct *work)
 	rcu_barrier();
 
 	list_for_each_entry_safe(s, s2, &to_destroy, list) {
+		kfence_shutdown_cache(s);
 #ifdef SLAB_SUPPORTS_SYSFS
 		sysfs_slab_release(s);
 #else
@@ -460,6 +462,7 @@ static int shutdown_cache(struct kmem_cache *s)
 		list_add_tail(&s->list, &slab_caches_to_rcu_destroy);
 		schedule_work(&slab_caches_to_rcu_destroy_work);
 	} else {
+		kfence_shutdown_cache(s);
 #ifdef SLAB_SUPPORTS_SYSFS
 		sysfs_slab_unlink(s);
 		sysfs_slab_release(s);
@@ -1171,7 +1174,7 @@ size_t ksize(const void *objp)
 	if (unlikely(ZERO_OR_NULL_PTR(objp)) || !__kasan_check_read(objp, 1))
 		return 0;
 
-	size = __ksize(objp);
+	size = kfence_ksize(objp) ?: __ksize(objp);
 	/*
 	 * We assume that ksize callers could use whole allocated area,
 	 * so we need to unpoison this area.
Signature · 1 line
2.29.1.341.ge80a0c044ae-goog
#14 lore raw

Re: [PATCH v6 4/9] mm, kfence: insert KFENCE hooks for SLAB

Quoted 1 line · Marco Elver

Inserts KFENCE hooks into the SLAB allocator.

[...]

Quoted 1 line
mm/slab.c +0 -0

[...]

mm/slab.c static void cache_flusharray(struct kmem_cache *cachep, struct array_cache *ac) · line 3433

Quoted 8 lines
mm/slab.c +4 -0
@@ -3416,6 +3427,11 @@ static void cache_flusharray(struct kmem_cache *cachep, struct array_cache *ac)
 static __always_inline void __cache_free(struct kmem_cache *cachep, void *objp,
                                         unsigned long caller)
 {
+       if (kfence_free(objp)) {
+               kmemleak_free_recursive(objp, cachep->flags);
+               return;
+       }

This looks dodgy. Normally kmemleak is told that an object is being freed *before* the object is actually released. I think that if this races really badly, we'll make kmemleak stumble over this bit in create_object():

kmemleak_stop("Cannot insert 0x%lx into the object search tree (overlaps existing)\n",

      ptr);
Quoted 4 lines
mm/slab.c +1 -0
+
        /* Put the object into the quarantine, don't touch it for now. */
        if (kasan_slab_free(cachep, objp, _RET_IP_))
                return;
author maintainer
#15 lore raw

Re: [PATCH v6 4/9] mm, kfence: insert KFENCE hooks for SLAB

Quoted 19 lines · Jann Horn
Quoted 1 line · Marco Elver

Inserts KFENCE hooks into the SLAB allocator.

[...]

Quoted 1 line
mm/slab.c +0 -0

[...]

mm/slab.c static void cache_flusharray(struct kmem_cache *cachep, struct array_cache *ac) · line 3433

Quoted 8 lines
mm/slab.c +4 -0
@@ -3416,6 +3427,11 @@ static void cache_flusharray(struct kmem_cache *cachep, struct array_cache *ac)
 static __always_inline void __cache_free(struct kmem_cache *cachep, void *objp,
                                         unsigned long caller)
 {
+       if (kfence_free(objp)) {
+               kmemleak_free_recursive(objp, cachep->flags);
+               return;
+       }

This looks dodgy. Normally kmemleak is told that an object is being freed *before* the object is actually released. I think that if this races really badly, we'll make kmemleak stumble over this bit in create_object():

Quoted 3 lines

kmemleak_stop("Cannot insert 0x%lx into the object search tree (overlaps existing)\n",

      ptr);

Good catch. Although extremely unlikely, let's just avoid it by moving the freeing after.

Quoted 5 lines
Quoted 5 lines
mm/slab.c +1 -0
+
        /* Put the object into the quarantine, don't touch it for now. */
        if (kasan_slab_free(cachep, objp, _RET_IP_))
                return;
author maintainer
#16 lore raw

PATCH v6 5/9 mm, kfence: insert KFENCE hooks for SLUB

In reply to Marco Elver

From: Alexander Potapenko <[email protected]>

Inserts KFENCE hooks into the SLUB allocator.

To pass the originally requested size to KFENCE, add an argument 'orig_size' to slab_alloc*(). The additional argument is required to preserve the requested original size for kmalloc() allocations, which uses size classes (e.g. an allocation of 272 bytes will return an object of size 512). Therefore, kmem_cache::size does not represent the kmalloc-caller's requested size, and we must introduce the argument 'orig_size' to propagate the originally requested size to KFENCE.

Without the originally requested size, we would not be able to detect out-of-bounds accesses for objects placed at the end of a KFENCE object page if that object is not equal to the kmalloc-size class it was bucketed into.

When KFENCE is disabled, there is no additional overhead, since slab_alloc*() functions are __always_inline.

---
v5:
* Fix obj_to_index for kfence objects.

v3:
* Rewrite patch description to clarify need for 'orig_size' [reported by Christopher Lameter].

---
 include/linux/slub_def.h |  3 ++
 mm/slub.c                | 72 +++++++++++++++++++++++++++++-----------
 2 files changed, 56 insertions(+), 19 deletions(-)
include/linux/slub_def.h +3 -0
index 1be0ed5befa1..dcde82a4434c 100644
@@ -7,6 +7,7 @@
  *
  * (C) 2007 SGI, Christoph Lameter
  */
+#include <linux/kfence.h>
 #include <linux/kobject.h>
 #include <linux/reciprocal_div.h>
 
@@ -185,6 +186,8 @@ static inline unsigned int __obj_to_index(const struct kmem_cache *cache,
 static inline unsigned int obj_to_index(const struct kmem_cache *cache,
 					const struct page *page, void *obj)
 {
+	if (is_kfence_address(obj))
+		return 0;
 	return __obj_to_index(cache, page_address(page), obj);
 }
 
mm/slub.c +53 -19

218 lines not on this page · raw message

Signature · 1 line
2.29.1.341.ge80a0c044ae-goog
Reviewed-by on v6 5/9
#17 lore raw

Re: [PATCH v6 5/9] mm, kfence: insert KFENCE hooks for SLUB

Quoted 15 lines · Marco Elver

Inserts KFENCE hooks into the SLUB allocator.

To pass the originally requested size to KFENCE, add an argument 'orig_size' to slab_alloc*(). The additional argument is required to preserve the requested original size for kmalloc() allocations, which uses size classes (e.g. an allocation of 272 bytes will return an object of size 512). Therefore, kmem_cache::size does not represent the kmalloc-caller's requested size, and we must introduce the argument 'orig_size' to propagate the originally requested size to KFENCE.

Without the originally requested size, we would not be able to detect out-of-bounds accesses for objects placed at the end of a KFENCE object page if that object is not equal to the kmalloc-size class it was bucketed into.

Quoted 7 lines

When KFENCE is disabled, there is no additional overhead, since slab_alloc*() functions are __always_inline.

if you fix one nit:

[...]

Quoted 1 line
mm/slub.c +0 -0

[...]

mm/slub.c static inline void *get_freelist(struct kmem_cache *s, struct page *page) · line 2668

Quoted 7 lines
mm/slub.c +2 -1
@@ -2658,7 +2664,8 @@ static inline void *get_freelist(struct kmem_cache *s, struct page *page)
  * already disabled (which is the case for bulk allocation).
  */
 static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
-                         unsigned long addr, struct kmem_cache_cpu *c)
+                         unsigned long addr, struct kmem_cache_cpu *c,
+                         size_t orig_size)

orig_size is added as a new argument, but never used. (And if you remove this argument, __slab_alloc will also not be using its orig_size argument anymore.)

Quoted 22 lines
mm/slub.c +3 -2
 {
        void *freelist;
        struct page *page;
@@ -2763,7 +2770,8 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
  * cpu changes by refetching the per cpu area pointer.
  */
 static void *__slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
-                         unsigned long addr, struct kmem_cache_cpu *c)
+                         unsigned long addr, struct kmem_cache_cpu *c,
+                         size_t orig_size)
 {
        void *p;
        unsigned long flags;
@@ -2778,7 +2786,7 @@ static void *__slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
        c = this_cpu_ptr(s->cpu_slab);
 #endif
 
-       p = ___slab_alloc(s, gfpflags, node, addr, c);
+       p = ___slab_alloc(s, gfpflags, node, addr, c, orig_size);
        local_irq_restore(flags);
        return p;
 }
author maintainer
#18 lore raw

PATCH v6 6/9 kfence, kasan: make KFENCE compatible with KASAN

In reply to Marco Elver

From: Alexander Potapenko <[email protected]>

We make KFENCE compatible with KASAN for testing KFENCE itself. In particular, KASAN helps to catch any potential corruptions to KFENCE state, or other corruptions that may be a result of freepointer corruptions in the main allocators.

To indicate that the combination of the two is generally discouraged, CONFIG_EXPERT=y should be set. It also gives us the nice property that KFENCE will be build-tested by allyesconfig builds.

---
v5:
* Also guard kasan_unpoison_shadow with is_kfence_address(), as it may be called from SL*B internals, currently ksize().
* Make kasan_record_aux_stack() compatible with KFENCE, which may be called from outside KASAN runtime.

---
 lib/Kconfig.kfence |  2 +-
 mm/kasan/common.c  | 15 +++++++++++++++
 mm/kasan/generic.c |  3 ++-
 3 files changed, 18 insertions(+), 2 deletions(-)
lib/Kconfig.kfence +1 -1
index d24baa3bce4a..639b48cc75d4 100644
@@ -5,7 +5,7 @@ config HAVE_ARCH_KFENCE
 
 menuconfig KFENCE
 	bool "KFENCE: low-overhead sampling-based memory safety error detector"
-	depends on HAVE_ARCH_KFENCE && !KASAN && (SLAB || SLUB)
+	depends on HAVE_ARCH_KFENCE && (!KASAN || EXPERT) && (SLAB || SLUB)
 	depends on JUMP_LABEL # To ensure performance, require jump labels
 	select STACKTRACE
 	help
mm/kasan/common.c +15 -0
index 950fd372a07e..ac1d404fb41e 100644
@@ -18,6 +18,7 @@
 #include <linux/init.h>
 #include <linux/kasan.h>
 #include <linux/kernel.h>
+#include <linux/kfence.h>
 #include <linux/kmemleak.h>
 #include <linux/linkage.h>
 #include <linux/memblock.h>
@@ -141,6 +142,14 @@ void kasan_unpoison_shadow(const void *address, size_t size)
 	 */
 	address = reset_tag(address);
 
+	/*
+	 * We may be called from SL*B internals, such as ksize(): with a size
+	 * not a multiple of machine-word size, avoid poisoning the invalid
+	 * portion of the word for KFENCE memory.
+	 */
+	if (is_kfence_address(address))
+		return;
+
 	kasan_poison_shadow(address, size, tag);
 
 	if (size & KASAN_SHADOW_MASK) {
@@ -396,6 +405,9 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object,
 	tagged_object = object;
 	object = reset_tag(object);
 
+	if (is_kfence_address(object))
+		return false;
+
 	if (unlikely(nearest_obj(cache, virt_to_head_page(object), object) !=
 	    object)) {
 		kasan_report_invalid_free(tagged_object, ip);
@@ -444,6 +456,9 @@ static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,
 	if (unlikely(object == NULL))
 		return NULL;
 
+	if (is_kfence_address(object))
+		return (void *)object;
+
 	redzone_start = round_up((unsigned long)(object + size),
 				KASAN_SHADOW_SCALE_SIZE);
 	redzone_end = round_up((unsigned long)object + cache->object_size,
mm/kasan/generic.c +2 -1
index 248264b9cb76..1069ecd1cd55 100644
@@ -21,6 +21,7 @@
 #include <linux/init.h>
 #include <linux/kasan.h>
 #include <linux/kernel.h>
+#include <linux/kfence.h>
 #include <linux/kmemleak.h>
 #include <linux/linkage.h>
 #include <linux/memblock.h>
@@ -332,7 +333,7 @@ void kasan_record_aux_stack(void *addr)
 	struct kasan_alloc_meta *alloc_info;
 	void *object;
 
-	if (!(page && PageSlab(page)))
+	if (is_kfence_address(addr) || !(page && PageSlab(page)))
 		return;
 
 	cache = page->slab_cache;
Signature · 1 line
2.29.1.341.ge80a0c044ae-goog
Reviewed-by on v6 6/9
#19 lore raw

Re: [PATCH v6 6/9] kfence, kasan: make KFENCE compatible with KASAN

Quoted 5 lines · Marco Elver

We make KFENCE compatible with KASAN for testing KFENCE itself. In particular, KASAN helps to catch any potential corruptions to KFENCE state, or other corruptions that may be a result of freepointer corruptions in the main allocators.

Quoted 8 lines

To indicate that the combination of the two is generally discouraged, CONFIG_EXPERT=y should be set. It also gives us the nice property that KFENCE will be build-tested by allyesconfig builds.

with one nit:

[...]

Quoted 1 line
mm/kasan/common.c +0 -0

[...]

mm/kasan/common.c void kasan_unpoison_shadow(const void *address, size_t size) · line 151

Quoted 11 lines
mm/kasan/common.c +7 -0
@@ -141,6 +142,14 @@ void kasan_unpoison_shadow(const void *address, size_t size)
         */
        address = reset_tag(address);
 
+       /*
+        * We may be called from SL*B internals, such as ksize(): with a size
+        * not a multiple of machine-word size, avoid poisoning the invalid
+        * portion of the word for KFENCE memory.
+        */
+       if (is_kfence_address(address))
+               return;

It might be helpful if you could add a comment that explains that kasan_poison_object_data() does not need a similar guard because kasan_poison_object_data() is always paired with kasan_unpoison_object_data() - that threw me off a bit at first.

author maintainer
#20 lore raw

Re: [PATCH v6 6/9] kfence, kasan: make KFENCE compatible with KASAN

Quoted 11 lines · Jann Horn
Quoted 9 lines · Marco Elver

We make KFENCE compatible with KASAN for testing KFENCE itself. In particular, KASAN helps to catch any potential corruptions to KFENCE state, or other corruptions that may be a result of freepointer corruptions in the main allocators.

To indicate that the combination of the two is generally discouraged, CONFIG_EXPERT=y should be set. It also gives us the nice property that KFENCE will be build-tested by allyesconfig builds.

Quoted 6 lines
Quoted 4 lines

Thanks!

Quoted 9 lines

with one nit:

[...]

Quoted 1 line
mm/kasan/common.c +0 -0

[...]

Quoted 4 lines
mm/kasan/common.c +0 -0
@@ -141,6 +142,14 @@ void kasan_unpoison_shadow(const void *address, size_t size)
         */
        address = reset_tag(address);
 
Quoted 12 lines

mm/kasan/common.c void kasan_unpoison_shadow(const void *address, size_t size) · line 151

Quoted 7 lines
mm/kasan/common.c +7 -0
+       /*
+        * We may be called from SL*B internals, such as ksize(): with a size
+        * not a multiple of machine-word size, avoid poisoning the invalid
+        * portion of the word for KFENCE memory.
+        */
+       if (is_kfence_address(address))
+               return;

It might be helpful if you could add a comment that explains that kasan_poison_object_data() does not need a similar guard because kasan_poison_object_data() is always paired with kasan_unpoison_object_data() - that threw me off a bit at first.

Well, KFENCE objects should never be poisoned/unpoisoned because the kasan_alloc and free hooks have a kfence guard, and none of the code in sl*b.c that does kasan_{poison,unpoison}_object_data() should be executed for KFENCE objects.

But I just noticed that kernel/scs.c seems to kasan_poison and unpoison objects, and keeps them poisoned for most of the object lifetime. I think we better add a kfence guard to kasan_poison_shadow() as well.

Thanks,
-- Marco

#21 lore raw

Re: [PATCH v6 6/9] kfence, kasan: make KFENCE compatible with KASAN

Quoted 42 lines · Marco Elver
Quoted 10 lines · Jann Horn
Quoted 9 lines · Marco Elver

We make KFENCE compatible with KASAN for testing KFENCE itself. In particular, KASAN helps to catch any potential corruptions to KFENCE state, or other corruptions that may be a result of freepointer corruptions in the main allocators.

To indicate that the combination of the two is generally discouraged, CONFIG_EXPERT=y should be set. It also gives us the nice property that KFENCE will be build-tested by allyesconfig builds.

Quoted 6 lines
Quoted 4 lines

Thanks!

Quoted 9 lines

with one nit:

[...]

Quoted 1 line
mm/kasan/common.c +0 -0

[...]

Quoted 4 lines
mm/kasan/common.c +0 -0
@@ -141,6 +142,14 @@ void kasan_unpoison_shadow(const void *address, size_t size)
         */
        address = reset_tag(address);
 
Quoted 12 lines

mm/kasan/common.c void kasan_unpoison_shadow(const void *address, size_t size) · line 151

Quoted 7 lines
mm/kasan/common.c +7 -0
+       /*
+        * We may be called from SL*B internals, such as ksize(): with a size
+        * not a multiple of machine-word size, avoid poisoning the invalid
+        * portion of the word for KFENCE memory.
+        */
+       if (is_kfence_address(address))
+               return;

It might be helpful if you could add a comment that explains that kasan_poison_object_data() does not need a similar guard because kasan_poison_object_data() is always paired with kasan_unpoison_object_data() - that threw me off a bit at first.

Quoted 8 lines

Well, KFENCE objects should never be poisoned/unpoisoned because the kasan_alloc and free hooks have a kfence guard, and none of the code in sl*b.c that does kasan_{poison,unpoison}_object_data() should be executed for KFENCE objects.

But I just noticed that kernel/scs.c seems to kasan_poison and unpoison objects, and keeps them poisoned for most of the object lifetime.

FWIW, I wouldn't be surprised if other parts of the kernel also ended up wanting to have in-object redzones eventually - e.g. inside skb buffers, which have a struct skb_shared_info at the end. AFAIU at the moment, KASAN can't catch small OOB accesses from these buffers because of the following structure.

Quoted 2 lines

I think we better add a kfence guard to
kasan_poison_shadow() as well.

Sounds good.

author maintainer
#22 lore raw

Re: [PATCH v6 6/9] kfence, kasan: make KFENCE compatible with KASAN

Quoted 53 lines · Jann Horn
Quoted 42 lines · Marco Elver
Quoted 10 lines · Jann Horn
Quoted 9 lines · Marco Elver

We make KFENCE compatible with KASAN for testing KFENCE itself. In particular, KASAN helps to catch any potential corruptions to KFENCE state, or other corruptions that may be a result of freepointer corruptions in the main allocators.

To indicate that the combination of the two is generally discouraged, CONFIG_EXPERT=y should be set. It also gives us the nice property that KFENCE will be build-tested by allyesconfig builds.

Quoted 6 lines
Quoted 4 lines

Thanks!

Quoted 9 lines

with one nit:

[...]

Quoted 1 line
mm/kasan/common.c +0 -0

[...]

Quoted 4 lines
mm/kasan/common.c +0 -0
@@ -141,6 +142,14 @@ void kasan_unpoison_shadow(const void *address, size_t size)
         */
        address = reset_tag(address);
 
Quoted 12 lines

mm/kasan/common.c void kasan_unpoison_shadow(const void *address, size_t size) · line 151

Quoted 7 lines
mm/kasan/common.c +7 -0
+       /*
+        * We may be called from SL*B internals, such as ksize(): with a size
+        * not a multiple of machine-word size, avoid poisoning the invalid
+        * portion of the word for KFENCE memory.
+        */
+       if (is_kfence_address(address))
+               return;

It might be helpful if you could add a comment that explains that kasan_poison_object_data() does not need a similar guard because kasan_poison_object_data() is always paired with kasan_unpoison_object_data() - that threw me off a bit at first.

Quoted 8 lines

Well, KFENCE objects should never be poisoned/unpoisoned because the kasan_alloc and free hooks have a kfence guard, and none of the code in sl*b.c that does kasan_{poison,unpoison}_object_data() should be executed for KFENCE objects.

But I just noticed that kernel/scs.c seems to kasan_poison and unpoison objects, and keeps them poisoned for most of the object lifetime.

Quoted 5 lines

FWIW, I wouldn't be surprised if other parts of the kernel also ended up wanting to have in-object redzones eventually - e.g. inside skb buffers, which have a struct skb_shared_info at the end. AFAIU at the moment, KASAN can't catch small OOB accesses from these buffers because of the following structure.

Sure, and it might also become more interesting with MTE-based KASAN.

But, currently we recommend not to enable generic KASAN+KFENCE, because it'd be redundant if the instrumentation price for generic (or SW-tag) KASAN is already paid. The changes here are also mostly for testing KFENCE itself.

That may change with MTE-based KASAN, however, which may have modes where stack traces aren't collected and having KFENCE to get actionable debug-info across a fleet of machines may still be wanted. But that story is still evolving. The code here is only for the generic and SW-tag based KASAN modes, and MTE will have its own kasan_{un,}poison_shadow (afaik it's being renamed to kasan_{un,}poison_memory) which works just fine with KFENCE AFAIK.

Quoted 4 lines
Quoted 2 lines

I think we better add a kfence guard to
kasan_poison_shadow() as well.

Sounds good.

author maintainer
#23 lore raw

PATCH v6 7/9 kfence, Documentation: add KFENCE documentation

In reply to Marco Elver

Add KFENCE documentation in dev-tools/kfence.rst, and add to index.

---
v3:
* Re-introduce reference to Documentation/dev-tools/kfence.rst.

v2:
* Many clarifications based on comments from Andrey Konovalov.
* Document CONFIG_KFENCE_SAMPLE_INTERVAL=0 usage.
* Make use-cases between KASAN and KFENCE clearer.
* Be clearer about the fact the pool is fixed size.
* Update based on reporting changes.
* Explicitly mention max supported allocation size is PAGE_SIZE.

---
 Documentation/dev-tools/index.rst  |   1 +
 Documentation/dev-tools/kfence.rst | 291 +++++++++++++++++++++++++++++
 lib/Kconfig.kfence                 |   2 +
 3 files changed, 294 insertions(+)
 create mode 100644 Documentation/dev-tools/kfence.rst
Documentation/dev-tools/index.rst +1 -0
index f7809c7b1ba9..1b1cf4f5c9d9 100644
@@ -22,6 +22,7 @@ whole; patches welcome!
    ubsan
    kmemleak
    kcsan
+   kfence
    gdb-kernel-debugging
    kgdb
    kselftest
Documentation/dev-tools/kfence.rst new file +291 -0

291 lines not on this page · raw message

lib/Kconfig.kfence +2 -0
index 639b48cc75d4..3c4066aca4fc 100644
@@ -14,6 +14,8 @@ menuconfig KFENCE
 	  to have negligible cost to permit enabling it in production
 	  environments.
 
+	  See <file:Documentation/dev-tools/kfence.rst> for more details.
+
 	  Note that, KFENCE is not a substitute for explicit testing with tools
 	  such as KASAN. KFENCE can detect a subset of bugs that KASAN can
 	  detect, albeit at very different performance profiles. If you can
Signature · 1 line
2.29.1.341.ge80a0c044ae-goog
#24 lore raw

Re: [PATCH v6 7/9] kfence, Documentation: add KFENCE documentation

Quoted 1 line · Marco Elver

Add KFENCE documentation in dev-tools/kfence.rst, and add to index.

[...]

Documentation/dev-tools/kfence.rst line 52

Quoted 10 lines
Documentation/dev-tools/kfence.rst new file +10 -0
+The KFENCE memory pool is of fixed size, and if the pool is exhausted, no
+further KFENCE allocations occur. With ``CONFIG_KFENCE_NUM_OBJECTS`` (default
+255), the number of available guarded objects can be controlled. Each object
+requires 2 pages, one for the object itself and the other one used as a guard
+page; object pages are interleaved with guard pages, and every object page is
+therefore surrounded by two guard pages.
+
+The total memory dedicated to the KFENCE memory pool can be computed as::
+
+    ( #objects + 1 ) * 2 * PAGE_SIZE

Plus memory overhead from shattered hugepages. With the default object count, on x86, we allocate 2MiB of memory pool, but if we have to shatter a 2MiB hugepage for that, we may cause the allocation of one extra page table, or 4KiB. Of course that's pretty much negligible. But on arm64 it's worse, because there we have to disable hugepages in the linear map completely. So on a device with 4GiB memory, we might end up with something on the order of 4GiB/2MiB * 0x1000 bytes = 8MiB of extra L1 page tables that wouldn't have been needed otherwise - significantly more than the default memory pool size.

If the memory overhead is documented, this detail should probably be documented, too.

Quoted 2 lines
Documentation/dev-tools/kfence.rst new file +2 -0
+Using the default config, and assuming a page size of 4 KiB, results in
+dedicating 2 MiB to the KFENCE memory pool.

[...]

Quoted 1 line

+For such errors, the address where the corruption as well as the invalidly

nit: "the address where the corruption occurred" or "the address of the corruption"

Quoted 6 lines
Documentation/dev-tools/kfence.rst new file +6 -0
+written bytes (offset from the address) are shown; in this representation, '.'
+denote untouched bytes. In the example above ``0xac`` is the value written to
+the invalid address at offset 0, and the remaining '.' denote that no following
+bytes have been touched. Note that, real values are only shown for
+``CONFIG_DEBUG_KERNEL=y`` builds; to avoid information disclosure for non-debug
+builds, '!' is used instead to denote invalidly written bytes.

[...]

Documentation/dev-tools/kfence.rst line 230

Quoted 6 lines
Documentation/dev-tools/kfence.rst new file +6 -0
+KFENCE objects each reside on a dedicated page, at either the left or right
+page boundaries selected at random. The pages to the left and right of the
+object page are "guard pages", whose attributes are changed to a protected
+state, and cause page faults on any attempted access. Such page faults are then
+intercepted by KFENCE, which handles the fault gracefully by reporting an
+out-of-bounds access.

... and marking the page as accessible so that the faulting code can continue (wrongly) executing.

[...]

Documentation/dev-tools/kfence.rst line 260

Quoted 4 lines
Documentation/dev-tools/kfence.rst new file +4 -0
+Interface
+---------
+
+The following describes the functions which are used by allocators as well page

nit: "as well as"?

Quoted 1 line

+handling code to set up and deal with KFENCE allocations.

#25 lore raw

Re: [PATCH v6 7/9] kfence, Documentation: add KFENCE documentation

Quoted 16 lines · Jann Horn
Quoted 1 line · Marco Elver

Add KFENCE documentation in dev-tools/kfence.rst, and add to index.

[...]

Documentation/dev-tools/kfence.rst line 52

Quoted 10 lines
Documentation/dev-tools/kfence.rst new file +10 -0
+The KFENCE memory pool is of fixed size, and if the pool is exhausted, no
+further KFENCE allocations occur. With ``CONFIG_KFENCE_NUM_OBJECTS`` (default
+255), the number of available guarded objects can be controlled. Each object
+requires 2 pages, one for the object itself and the other one used as a guard
+page; object pages are interleaved with guard pages, and every object page is
+therefore surrounded by two guard pages.
+
+The total memory dedicated to the KFENCE memory pool can be computed as::
+
+    ( #objects + 1 ) * 2 * PAGE_SIZE

Plus memory overhead from shattered hugepages. With the default object

Quoted 8 lines

count, on x86, we allocate 2MiB of memory pool, but if we have to shatter a 2MiB hugepage for that, we may cause the allocation of one extra page table, or 4KiB. Of course that's pretty much negligible. But on arm64 it's worse, because there we have to disable hugepages in the linear map completely. So on a device with 4GiB memory, we might end up with something on the order of 4GiB/2MiB * 0x1000 bytes = 8MiB of extra L1 page tables that wouldn't have been needed otherwise - significantly more than the default memory pool size.

Note that with CONFIG_RODATA_FULL_DEFAULT_ENABLED (which is on by default now) these hugepages are already disabled (see patch 3/9)

Quoted 2 lines

If the memory overhead is documented, this detail should probably be documented, too.

But, yes, documenting that also makes sense.

Quoted 37 lines
Quoted 2 lines
Documentation/dev-tools/kfence.rst new file +2 -0
+Using the default config, and assuming a page size of 4 KiB, results in
+dedicating 2 MiB to the KFENCE memory pool.

[...]

Quoted 1 line

+For such errors, the address where the corruption as well as the invalidly

nit: "the address where the corruption occurred" or "the address of the corruption"

Quoted 6 lines
Documentation/dev-tools/kfence.rst new file +6 -0
+written bytes (offset from the address) are shown; in this representation, '.'
+denote untouched bytes. In the example above ``0xac`` is the value written to
+the invalid address at offset 0, and the remaining '.' denote that no following
+bytes have been touched. Note that, real values are only shown for
+``CONFIG_DEBUG_KERNEL=y`` builds; to avoid information disclosure for non-debug
+builds, '!' is used instead to denote invalidly written bytes.

[...]

Documentation/dev-tools/kfence.rst line 230

Quoted 6 lines
Documentation/dev-tools/kfence.rst new file +6 -0
+KFENCE objects each reside on a dedicated page, at either the left or right
+page boundaries selected at random. The pages to the left and right of the
+object page are "guard pages", whose attributes are changed to a protected
+state, and cause page faults on any attempted access. Such page faults are then
+intercepted by KFENCE, which handles the fault gracefully by reporting an
+out-of-bounds access.

... and marking the page as accessible so that the faulting code can continue (wrongly) executing.

[...]

Documentation/dev-tools/kfence.rst line 260

Quoted 4 lines
Documentation/dev-tools/kfence.rst new file +4 -0
+Interface
+---------
+
+The following describes the functions which are used by allocators as well page

nit: "as well as"?

Quoted 1 line

+handling code to set up and deal with KFENCE allocations.

Signature · 10 lines
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
author maintainer
#26 lore raw

PATCH v6 8/9 kfence: add test suite

In reply to Marco Elver

Add KFENCE test suite, testing various error detection scenarios. Makes use of KUnit for test organization. Since KFENCE's interface to obtain error reports is via the console, the test verifies that KFENCE outputs expected reports to the console.

---
v5:
* Add better memory corruption test.
* Test sl*b_def.h primitives.

v4:
* Clarify RCU test comment [reported by Paul E. McKenney].

v3:
* Lower line buffer size to avoid warnings of using more than 1024 bytes stack usage [reported by kernel test robot <[email protected]>].

v2:
* Update for shortened memory corruption report.

---
 lib/Kconfig.kfence      |  13 +
 mm/kfence/Makefile      |   3 +
 mm/kfence/kfence_test.c | 822 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 838 insertions(+)
 create mode 100644 mm/kfence/kfence_test.c
lib/Kconfig.kfence +13 -0
index 3c4066aca4fc..ebafb4538b6e 100644
@@ -57,4 +57,17 @@ config KFENCE_STRESS_TEST_FAULTS
 
 	  The option is only to test KFENCE; set to 0 if you are unsure.
 
+config KFENCE_KUNIT_TEST
+	tristate "KFENCE integration test suite" if !KUNIT_ALL_TESTS
+	default KUNIT_ALL_TESTS
+	depends on TRACEPOINTS && KUNIT
+	help
+	  Test suite for KFENCE, testing various error detection scenarios with
+	  various allocation types, and checking that reports are correctly
+	  output to console.
+
+	  Say Y here if you want the test to be built into the kernel and run
+	  during boot; say M if you want the test to build as a module; say N
+	  if you are unsure.
+
 endif # KFENCE
mm/kfence/Makefile +3 -0
index d991e9a349f0..6872cd5e5390 100644
@@ -1,3 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 
 obj-$(CONFIG_KFENCE) := core.o report.o
+
+CFLAGS_kfence_test.o := -g -fno-omit-frame-pointer -fno-optimize-sibling-calls
+obj-$(CONFIG_KFENCE_KUNIT_TEST) += kfence_test.o
mm/kfence/kfence_test.c new file +822 -0

822 lines not on this page · raw message

Signature · 1 line
2.29.1.341.ge80a0c044ae-goog
#27 lore raw

Re: [PATCH v6 8/9] kfence: add test suite

Quoted 4 lines · Marco Elver

Add KFENCE test suite, testing various error detection scenarios. Makes use of KUnit for test organization. Since KFENCE's interface to obtain error reports is via the console, the test verifies that KFENCE outputs expected reports to the console.

[...]

Quoted 1 line
mm/kfence/kfence_test.c +0 -0

[...]

Quoted 4 lines
mm/kfence/kfence_test.c +4 -0
+static void *test_alloc(struct kunit *test, size_t size, gfp_t gfp, enum allocation_policy policy)
+{
+       void *alloc;
+       unsigned long timeout, resched_after;

[...]

Quoted 11 lines
mm/kfence/kfence_test.c +11 -0
+       /*
+        * 100x the sample interval should be more than enough to ensure we get
+        * a KFENCE allocation eventually.
+        */
+       timeout = jiffies + msecs_to_jiffies(100 * CONFIG_KFENCE_SAMPLE_INTERVAL);
+       /*
+        * Especially for non-preemption kernels, ensure the allocation-gate
+        * timer has time to catch up.
+        */
+       resched_after = jiffies + msecs_to_jiffies(CONFIG_KFENCE_SAMPLE_INTERVAL);
+       do {

[...]

mm/kfence/kfence_test.c line 283

Quoted 2 lines
mm/kfence/kfence_test.c +2 -0
+               if (time_after(jiffies, resched_after))
+                       cond_resched();

You probably meant to recalculate resched_after after the call to cond_resched()?

Quoted 5 lines
mm/kfence/kfence_test.c +5 -0
+       } while (time_before(jiffies, timeout));
+
+       KUNIT_ASSERT_TRUE_MSG(test, false, "failed to allocate from KFENCE");
+       return NULL; /* Unreachable. */
+}

[...]

Quoted 6 lines

+/*
+ * KFENCE is unable to detect an OOB if the allocation's alignment requirements
+ * leave a gap between the object and the guard page. Specifically, an
+ * allocation of e.g. 73 bytes is aligned on 8 and 128 bytes for SLUB or SLAB
+ * respectively. Therefore it is impossible for the allocated object to adhere
+ * to either of the page boundaries.

Should this be "to the left page boundary" instead of "to either of the page boundaries"?

Quoted 1 line

+ * However, we test that an access to memory beyond the gap result in KFENCE

*results

Quoted 3 lines

+ * detecting an OOB access.
+ */
+static void test_kmalloc_aligned_oob_read(struct kunit *test)

author maintainer
#28 lore raw

Re: [PATCH v6 8/9] kfence: add test suite

Quoted 25 lines · Jann Horn
Quoted 4 lines · Marco Elver

Add KFENCE test suite, testing various error detection scenarios. Makes use of KUnit for test organization. Since KFENCE's interface to obtain error reports is via the console, the test verifies that KFENCE outputs expected reports to the console.

[...]

Quoted 1 line
mm/kfence/kfence_test.c +0 -0

[...]

Quoted 4 lines
mm/kfence/kfence_test.c +4 -0
+static void *test_alloc(struct kunit *test, size_t size, gfp_t gfp, enum allocation_policy policy)
+{
+       void *alloc;
+       unsigned long timeout, resched_after;

[...]

mm/kfence/kfence_test.c line 252

Quoted 11 lines
mm/kfence/kfence_test.c +11 -0
+       /*
+        * 100x the sample interval should be more than enough to ensure we get
+        * a KFENCE allocation eventually.
+        */
+       timeout = jiffies + msecs_to_jiffies(100 * CONFIG_KFENCE_SAMPLE_INTERVAL);
+       /*
+        * Especially for non-preemption kernels, ensure the allocation-gate
+        * timer has time to catch up.
+        */
+       resched_after = jiffies + msecs_to_jiffies(CONFIG_KFENCE_SAMPLE_INTERVAL);
+       do {
Quoted 6 lines

[...]

mm/kfence/kfence_test.c line 283

Quoted 2 lines
mm/kfence/kfence_test.c +2 -0
+               if (time_after(jiffies, resched_after))
+                       cond_resched();

You probably meant to recalculate resched_after after the call to cond_resched()?

This is intentional. After @resched_after is reached, every failed allocation attempt will result in a cond_resched(), because we know the sample interval has elapsed and KFENCE should have kicked in. So we just want to ensure the delayed work gets to run as soon as possible, and just keep yielding.

Added a clarifying comment.

Quoted 13 lines
Quoted 5 lines
mm/kfence/kfence_test.c +5 -0
+       } while (time_before(jiffies, timeout));
+
+       KUNIT_ASSERT_TRUE_MSG(test, false, "failed to allocate from KFENCE");
+       return NULL; /* Unreachable. */
+}

[...]

Quoted 6 lines

+/*
+ * KFENCE is unable to detect an OOB if the allocation's alignment requirements
+ * leave a gap between the object and the guard page. Specifically, an
+ * allocation of e.g. 73 bytes is aligned on 8 and 128 bytes for SLUB or SLAB
+ * respectively. Therefore it is impossible for the allocated object to adhere
+ * to either of the page boundaries.

Quoted 2 lines

Should this be "to the left page boundary" instead of "to either of the page boundaries"?

Thanks for spotting. I think it's "Therefore it is impossible for the allocated object to contiguously line up with the right guard page."

Quoted 9 lines
Quoted 1 line

+ * However, we test that an access to memory beyond the gap result in KFENCE

*results

Quoted 3 lines

+ * detecting an OOB access.
+ */
+static void test_kmalloc_aligned_oob_read(struct kunit *test)

Thanks, will address these for v7.

author maintainer
#29 lore raw

PATCH v6 9/9 MAINTAINERS: Add entry for KFENCE

In reply to Marco Elver

Add entry for KFENCE maintainers.

---
v4:
* Split out from first patch.

---
 MAINTAINERS | 11 +++++++++++
 1 file changed, 11 insertions(+)
MAINTAINERS +11 -0
index e73636b75f29..2a257c865795 100644
@@ -9753,6 +9753,17 @@ F:	include/linux/keyctl.h
 F:	include/uapi/linux/keyctl.h
 F:	security/keys/
 
+KFENCE
+M:	Alexander Potapenko <[email protected]>
+M:	Marco Elver <[email protected]>
+R:	Dmitry Vyukov <[email protected]>
+L:	[email protected]
+S:	Maintained
+F:	Documentation/dev-tools/kfence.rst
+F:	include/linux/kfence.h
+F:	lib/Kconfig.kfence
+F:	mm/kfence/
+
 KFIFO
 M:	Stefani Seibold <[email protected]>
 S:	Maintained
Signature · 1 line
2.29.1.341.ge80a0c044ae-goog
#30 lore raw

Re: [PATCH v6 9/9] MAINTAINERS: Add entry for KFENCE

Quoted 7 lines · Marco Elver

Add entry for KFENCE maintainers.

[...]

Quoted 1 line
MAINTAINERS +0 -0

[...]

MAINTAINERS F: include/linux/keyctl.h · line 9765

Quoted 10 lines
MAINTAINERS +10 -0
+KFENCE
+M:     Alexander Potapenko <[email protected]>
+M:     Marco Elver <[email protected]>
+R:     Dmitry Vyukov <[email protected]>
+L:     [email protected]
+S:     Maintained
+F:     Documentation/dev-tools/kfence.rst
+F:     include/linux/kfence.h
+F:     lib/Kconfig.kfence
+F:     mm/kfence/

Plus arch/*/include/asm/kfence.h?

#31 lore raw
In reply to Marco Elver
Quoted 5 lines · Marco Elver

This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors. This series enables KFENCE for the x86 and arm64 architectures, and adds KFENCE hooks to the SLAB and SLUB allocators.

I think this is getting close to a good state, just a couple minor issues left.

Now that the magic "embed the memory pool in the BSS section" stuff is gone, this series looks fairly straightforward.

author maintainer
#32 lore raw
Quoted 8 lines · Jann Horn
Quoted 5 lines · Marco Elver

This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors. This series enables KFENCE for the x86 and arm64 architectures, and adds KFENCE hooks to the SLAB and SLUB allocators.

I think this is getting close to a good state, just a couple minor issues left.

Thanks for your comments. We'll address all of them for v7.

Quoted 2 lines

Now that the magic "embed the memory pool in the BSS section" stuff is gone, this series looks fairly straightforward.

Good to hear. :-)

Thanks,
-- Marco