KFENCE: A low-overhead sampling-based memory safety error detector
Subsystems (13)
Marco Elver
maintainerv3 cover KFENCE: A low-overhead sampling-based memory safety error detector
This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors. This series enables KFENCE for the x86 and arm64 architectures, and adds KFENCE hooks to the SLAB and SLUB allocators.
KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.
KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error.
Guarded allocations are set up based on a sample interval (can be set via kfence.sample_interval). After expiration of the sample interval, the next allocation through the main allocator (SLAB or SLUB) returns a guarded allocation from the KFENCE object pool. At this point, the timer is reset, and the next allocation is set up after the expiration of the interval.
To enable/disable a KFENCE allocation through the main allocator's fast-path without overhead, KFENCE relies on static branches via the static keys infrastructure. The static branch is toggled to redirect the allocation to KFENCE.
The KFENCE memory pool is of fixed size, and if the pool is exhausted no further KFENCE allocations occur. The default config is conservative with only 255 objects, resulting in a pool size of 2 MiB (with 4 KiB pages).
We have verified by running synthetic benchmarks (sysbench I/O, hackbench) that a kernel with KFENCE is performance-neutral compared to a non-KFENCE baseline kernel.
KFENCE is inspired by GWP-ASan [1], a userspace tool with similar properties. The name "KFENCE" is a homage to the Electric Fence Malloc Debugger [2].
For more details, see Documentation/dev-tools/kfence.rst added in the series -- also viewable here:
https://raw.githubusercontent.com/google/kasan/kfence/Documentation/dev-tools/kfence.rst
[1] http://llvm.org/docs/GwpAsan.html
[2] https://linux.die.net/man/3/efence
v3:
* Rewrite SLAB/SLUB patch descriptions to clarify need for 'orig_size'.
* Various smaller fixes (see details in patches).
v2: https://lkml.kernel.org/r/[email protected]
* Various comment/documentation changes (see details in patches).
* Various smaller fixes (see details in patches).
* Change all reports to reference the kfence object, "kfence-#nn".
* Skip allocation/free internals stack trace.
* Rework KMEMLEAK compatibility patch.
RFC/v1: https://lkml.kernel.org/r/[email protected]
Alexander Potapenko (6):
mm: add Kernel Electric-Fence infrastructure x86, kfence: enable KFENCE for x86
mm, kfence: insert KFENCE hooks for SLAB
mm, kfence: insert KFENCE hooks for SLUB kfence, kasan: make KFENCE compatible with KASAN kfence, kmemleak: make KFENCE compatible with KMEMLEAK
Marco Elver (4):
arm64, kfence: enable KFENCE for ARM64 kfence, lockdep: make KFENCE compatible with lockdep kfence, Documentation: add KFENCE documentation
kfence: add test suite
Documentation/dev-tools/index.rst | 1 + Documentation/dev-tools/kfence.rst | 291 +++++++++++ MAINTAINERS | 11 + arch/arm64/Kconfig | 1 + arch/arm64/include/asm/kfence.h | 39 ++ arch/arm64/mm/fault.c | 4 + arch/x86/Kconfig | 2 + arch/x86/include/asm/kfence.h | 60 +++ arch/x86/mm/fault.c | 4 + include/linux/kfence.h | 174 +++++++ init/main.c | 2 + kernel/locking/lockdep.c | 8 + lib/Kconfig.debug | 1 + lib/Kconfig.kfence | 78 +++ mm/Makefile | 1 + mm/kasan/common.c | 7 + mm/kfence/Makefile | 6 + mm/kfence/core.c | 733 +++++++++++++++++++++++++++ mm/kfence/kfence.h | 102 ++++ mm/kfence/kfence_test.c | 777 +++++++++++++++++++++++++++++ mm/kfence/report.c | 219 ++++++++ mm/kmemleak.c | 6 + mm/slab.c | 46 +- mm/slab_common.c | 6 +- mm/slub.c | 72 ++- 25 files changed, 2619 insertions(+), 32 deletions(-) create mode 100644 Documentation/dev-tools/kfence.rst create mode 100644 arch/arm64/include/asm/kfence.h create mode 100644 arch/x86/include/asm/kfence.h create mode 100644 include/linux/kfence.h create mode 100644 lib/Kconfig.kfence create mode 100644 mm/kfence/Makefile create mode 100644 mm/kfence/core.c create mode 100644 mm/kfence/kfence.h create mode 100644 mm/kfence/kfence_test.c create mode 100644 mm/kfence/report.c
Signature · 1 line
2.28.0.681.g6f77f65b4e-goog
Marco Elver
maintainerPATCH v3 1/10 mm: add Kernel Electric-Fence infrastructure
From: Alexander Potapenko <[email protected]>
This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors.
KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.
KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error. To detect out-of-bounds writes to memory within the object's page itself, KFENCE also uses pattern-based redzones. The following figure illustrates the page layout:
---+-----------+-----------+-----------+-----------+-----------+---
| xxxxxxxxx | O : | xxxxxxxxx | : O | xxxxxxxxx |
| xxxxxxxxx | B : | xxxxxxxxx | : B | xxxxxxxxx |
| x GUARD x | J : RED- | x GUARD x | RED- : J | x GUARD x |
| xxxxxxxxx | E : ZONE | xxxxxxxxx | ZONE : E | xxxxxxxxx |
| xxxxxxxxx | C : | xxxxxxxxx | : C | xxxxxxxxx |
| xxxxxxxxx | T : | xxxxxxxxx | : T | xxxxxxxxx |---+-----------+-----------+-----------+-----------+-----------+---
Guarded allocations are set up based on a sample interval (can be set via kfence.sample_interval). After expiration of the sample interval, a guarded allocation from the KFENCE object pool is returned to the main allocator (SLAB or SLUB). At this point, the timer is reset, and the next allocation is set up after the expiration of the interval.
To enable/disable a KFENCE allocation through the main allocator's fast-path without overhead, KFENCE relies on static branches via the static keys infrastructure. The static branch is toggled to redirect the allocation to KFENCE. To date, we have verified by running synthetic benchmarks (sysbench I/O workloads) that a kernel compiled with KFENCE is performance-neutral compared to the non-KFENCE baseline.
For more details, see Documentation/dev-tools/kfence.rst (added later in the series).
---
v3:
* Reports by SeongJae Park:
* Remove reference to Documentation/dev-tools/kfence.rst.
* Remove redundant braces.
* Use CONFIG_KFENCE_NUM_OBJECTS instead of ARRAY_SIZE(...).
* Align some comments.
* Add figure from Documentation/dev-tools/kfence.rst added later in series to patch description.
v2:
* Add missing __printf attribute to seq_con_printf, and fix new warning. [reported by kernel test robot <[email protected]>]
* Fix up some comments [reported by Jonathan Cameron].
* Remove 2 cases of redundant stack variable initialization [reported by Jonathan Cameron].
* Fix printf format [reported by kernel test robot <[email protected]>].
* Print (in kfence-#nn) after address, to more clearly establish link between first and second stacktrace [reported by Andrey Konovalov].
* Make choice between KASAN and KFENCE clearer in Kconfig help text [suggested by Dave Hansen].
* Document CONFIG_KFENCE_SAMPLE_INTERVAL=0.
* Shorten memory corruption report line length.
* Make /sys/module/kfence/parameters/sample_interval root-writable for all builds (to enable debugging, automatic dynamic tweaking).
* Reports by Dmitry Vyukov:
* Do not store negative size for right-located objects
* Only cache-align addresses of right-located objects.
* Run toggle_allocation_gate() after KFENCE is enabled.
* Add empty line between allocation and free stacks.
* Add comment about SLAB_TYPESAFE_BY_RCU.
* Also skip internals for allocation/free stacks.
* s/KFENCE_FAULT_INJECTION/KFENCE_STRESS_TEST_FAULTS/ as FAULT_INJECTION is already overloaded in different contexts.
* Parenthesis for macro variable.
* Lower max of KFENCE_NUM_OBJECTS config variable.
--- MAINTAINERS | 11 + include/linux/kfence.h | 174 ++++++++++ init/main.c | 2 + lib/Kconfig.debug | 1 + lib/Kconfig.kfence | 63 ++++ mm/Makefile | 1 + mm/kfence/Makefile | 3 + mm/kfence/core.c | 733 +++++++++++++++++++++++++++++++++++++++++ mm/kfence/kfence.h | 102 ++++++ mm/kfence/report.c | 219 ++++++++++++ 10 files changed, 1309 insertions(+) create mode 100644 include/linux/kfence.h create mode 100644 lib/Kconfig.kfence create mode 100644 mm/kfence/Makefile create mode 100644 mm/kfence/core.c create mode 100644 mm/kfence/kfence.h create mode 100644 mm/kfence/report.c
MAINTAINERS +11 -0
@@ -9673,6 +9673,17 @@ F: include/linux/keyctl.h F: include/uapi/linux/keyctl.h F: security/keys/ +KFENCE +M: Alexander Potapenko <[email protected]> +M: Marco Elver <[email protected]> +R: Dmitry Vyukov <[email protected]> +L: [email protected] +S: Maintained +F: Documentation/dev-tools/kfence.rst +F: include/linux/kfence.h +F: lib/Kconfig.kfence +F: mm/kfence/ + KFIFO M: Stefani Seibold <[email protected]> S: Maintained
include/linux/kfence.h new file +174 -0
174 lines not on this page · raw message
init/main.c +2 -0
@@ -39,6 +39,7 @@#include <linux/security.h> #include <linux/smp.h> #include <linux/profile.h> +#include <linux/kfence.h> #include <linux/rcupdate.h> #include <linux/moduleparam.h> #include <linux/kallsyms.h>
@@ -942,6 +943,7 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)hrtimers_init ();softirq_init ();timekeeping_init (); +kfence_init ();/* * For best initial stack canary entropy, prepare it after:
lib/Kconfig.debug +1 -0
@@ -880,6 +880,7 @@ config DEBUG_STACKOVERFLOW If in doubt, say "N". source "lib/Kconfig.kasan" +source "lib/Kconfig.kfence" endmenu # "Memory Debugging"
lib/Kconfig.kfence new file +63 -0
63 lines not on this page · raw message
mm/Makefile +1 -0
@@ -81,6 +81,7 @@ obj-$(CONFIG_PAGE_POISONING) += page_poison.o obj-$(CONFIG_SLAB) += slab.o obj-$(CONFIG_SLUB) += slub.o obj-$(CONFIG_KASAN) += kasan/ +obj-$(CONFIG_KFENCE) += kfence/ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_MEMORY_HOTPLUG) += memory_hotplug.o obj-$(CONFIG_MEMTEST) += memtest.o
mm/kfence/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-$(CONFIG_KFENCE) := core.o report.o
mm/kfence/core.c new file +733 -0
733 lines not on this page · raw message
mm/kfence/kfence.h new file +102 -0
102 lines not on this page · raw message
mm/kfence/report.c new file +219 -0
219 lines not on this page · raw message
Signature · 1 line
2.28.0.681.g6f77f65b4e-goog
Re: [PATCH v3 01/10] mm: add Kernel Electric-Fence infrastructure
Quoted 113 lines · On Mon, 21 Sep 2020 15:26:02 +0200 Marco Elver <[email protected]> wrote:
From: Alexander Potapenko <[email protected]>
This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors.
KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.
KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error. To detect out-of-bounds writes to memory within the object's page itself, KFENCE also uses pattern-based redzones. The following figure illustrates the page layout:
---+-----------+-----------+-----------+-----------+-----------+---
| xxxxxxxxx | O : | xxxxxxxxx | : O | xxxxxxxxx | | xxxxxxxxx | B : | xxxxxxxxx | : B | xxxxxxxxx | | x GUARD x | J : RED- | x GUARD x | RED- : J | x GUARD x | | xxxxxxxxx | E : ZONE | xxxxxxxxx | ZONE : E | xxxxxxxxx | | xxxxxxxxx | C : | xxxxxxxxx | : C | xxxxxxxxx | | xxxxxxxxx | T : | xxxxxxxxx | : T | xxxxxxxxx |---+-----------+-----------+-----------+-----------+-----------+---
Guarded allocations are set up based on a sample interval (can be set via kfence.sample_interval). After expiration of the sample interval, a guarded allocation from the KFENCE object pool is returned to the main allocator (SLAB or SLUB). At this point, the timer is reset, and the next allocation is set up after the expiration of the interval.
To enable/disable a KFENCE allocation through the main allocator's fast-path without overhead, KFENCE relies on static branches via the static keys infrastructure. The static branch is toggled to redirect the allocation to KFENCE. To date, we have verified by running synthetic benchmarks (sysbench I/O workloads) that a kernel compiled with KFENCE is performance-neutral compared to the non-KFENCE baseline.
For more details, see
Documentation/dev-tools/kfence.rst(added later in the series).Reviewed-by Dmitry Vyukov <[email protected]>Co-developed-by Marco Elver <[email protected]>Signed-off-by Marco Elver <[email protected]>Signed-off-by Alexander Potapenko <[email protected]>---
v3:
* Reports by SeongJae Park:
* Remove reference toDocumentation/dev-tools/kfence.rst.
* Remove redundant braces.
* UseCONFIG_KFENCE_NUM_OBJECTSinstead ofARRAY_SIZE(...).
* Align some comments.
* Add figure fromDocumentation/dev-tools/kfence.rstadded later in series to patch description.v2:
* Add missing__printfattribute toseq_con_printf, and fix new warning. [reported by kernel test robot <[email protected]>]
* Fix up some comments [reported by Jonathan Cameron].
* Remove 2 cases of redundant stack variable initialization [reported by Jonathan Cameron].
* Fix printf format [reported by kernel test robot <[email protected]>].
* Print (in kfence-#nn) after address, to more clearly establish link between first and second stacktrace [reported by Andrey Konovalov].
* Make choice between KASAN and KFENCE clearer inKconfighelp text [suggested by Dave Hansen].
* DocumentCONFIG_KFENCE_SAMPLE_INTERVAL=0.
* Shorten memory corruption report line length.
* Make/sys/module/kfence/parameters/sample_intervalroot-writable for all builds (to enable debugging, automatic dynamic tweaking).
* Reports by Dmitry Vyukov:
* Do not store negative size for right-located objects
* Only cache-align addresses of right-located objects.
* Runtoggle_allocation_gate()after KFENCE is enabled.
* Add empty line between allocation and free stacks.
* Add comment aboutSLAB_TYPESAFE_BY_RCU.
* Also skip internals for allocation/free stacks.
* s/KFENCE_FAULT_INJECTION/KFENCE_STRESS_TEST_FAULTS/ asFAULT_INJECTIONis already overloaded in different contexts.
* Parenthesis for macro variable.
* Lower max ofKFENCE_NUM_OBJECTSconfig variable.--- MAINTAINERS | 11 + include/linux/kfence.h | 174 ++++++++++ init/main.c | 2 + lib/Kconfig.debug | 1 + lib/Kconfig.kfence | 63 ++++ mm/Makefile | 1 + mm/kfence/Makefile | 3 + mm/kfence/core.c | 733 +++++++++++++++++++++++++++++++++++++++++ mm/kfence/kfence.h | 102 ++++++ mm/kfence/report.c | 219 ++++++++++++ 10 files changed, 1309 insertions(+) create mode 100644 include/linux/kfence.h create mode 100644 lib/Kconfig.kfence create mode 100644 mm/kfence/Makefile create mode 100644 mm/kfence/core.c create mode 100644 mm/kfence/kfence.h create mode 100644 mm/kfence/report.cMAINTAINERS +0
-0
MAINTAINERS F: include/linux/keyctl.h · line 9682
Quoted 10 lines
MAINTAINERS +7
-0@@ -9673,6 +9673,17 @@ F: include/linux/keyctl.h F: include/uapi/linux/keyctl.h F: security/keys/ +KFENCE +M: Alexander Potapenko <[email protected]> +M: Marco Elver <[email protected]> +R: Dmitry Vyukov <[email protected]> +L: [email protected] +S: Maintained +F: Documentation/dev-tools/kfence.rst
This patch doesn't introduce this file yet, right? How about using a separate final patch for MAINTAINERS update?
Other than that,
Thanks,
SeongJae Park
Marco Elver
maintainerRe: [PATCH v3 01/10] mm: add Kernel Electric-Fence infrastructure
Quoted 119 lines · SeongJae Park' via kasan-dev
Quoted 116 lines · On Mon, 21 Sep 2020 15:26:02 +0200 Marco Elver <[email protected]> wrote:
From: Alexander Potapenko <[email protected]>
This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors.
KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.
KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error. To detect out-of-bounds writes to memory within the object's page itself, KFENCE also uses pattern-based redzones. The following figure illustrates the page layout:
---+-----------+-----------+-----------+-----------+-----------+---
| xxxxxxxxx | O : | xxxxxxxxx | : O | xxxxxxxxx | | xxxxxxxxx | B : | xxxxxxxxx | : B | xxxxxxxxx | | x GUARD x | J : RED- | x GUARD x | RED- : J | x GUARD x | | xxxxxxxxx | E : ZONE | xxxxxxxxx | ZONE : E | xxxxxxxxx | | xxxxxxxxx | C : | xxxxxxxxx | : C | xxxxxxxxx | | xxxxxxxxx | T : | xxxxxxxxx | : T | xxxxxxxxx |---+-----------+-----------+-----------+-----------+-----------+---
Guarded allocations are set up based on a sample interval (can be set via kfence.sample_interval). After expiration of the sample interval, a guarded allocation from the KFENCE object pool is returned to the main allocator (SLAB or SLUB). At this point, the timer is reset, and the next allocation is set up after the expiration of the interval.
To enable/disable a KFENCE allocation through the main allocator's fast-path without overhead, KFENCE relies on static branches via the static keys infrastructure. The static branch is toggled to redirect the allocation to KFENCE. To date, we have verified by running synthetic benchmarks (sysbench I/O workloads) that a kernel compiled with KFENCE is performance-neutral compared to the non-KFENCE baseline.
For more details, see
Documentation/dev-tools/kfence.rst(added later in the series).Reviewed-by Dmitry Vyukov <[email protected]>Co-developed-by Marco Elver <[email protected]>Signed-off-by Marco Elver <[email protected]>Signed-off-by Alexander Potapenko <[email protected]>---
v3:
* Reports by SeongJae Park:
* Remove reference toDocumentation/dev-tools/kfence.rst.
* Remove redundant braces.
* UseCONFIG_KFENCE_NUM_OBJECTSinstead ofARRAY_SIZE(...).
* Align some comments.
* Add figure fromDocumentation/dev-tools/kfence.rstadded later in series to patch description.v2:
* Add missing__printfattribute toseq_con_printf, and fix new warning. [reported by kernel test robot <[email protected]>]
* Fix up some comments [reported by Jonathan Cameron].
* Remove 2 cases of redundant stack variable initialization [reported by Jonathan Cameron].
* Fix printf format [reported by kernel test robot <[email protected]>].
* Print (in kfence-#nn) after address, to more clearly establish link between first and second stacktrace [reported by Andrey Konovalov].
* Make choice between KASAN and KFENCE clearer inKconfighelp text [suggested by Dave Hansen].
* DocumentCONFIG_KFENCE_SAMPLE_INTERVAL=0.
* Shorten memory corruption report line length.
* Make/sys/module/kfence/parameters/sample_intervalroot-writable for all builds (to enable debugging, automatic dynamic tweaking).
* Reports by Dmitry Vyukov:
* Do not store negative size for right-located objects
* Only cache-align addresses of right-located objects.
* Runtoggle_allocation_gate()after KFENCE is enabled.
* Add empty line between allocation and free stacks.
* Add comment aboutSLAB_TYPESAFE_BY_RCU.
* Also skip internals for allocation/free stacks.
* s/KFENCE_FAULT_INJECTION/KFENCE_STRESS_TEST_FAULTS/ asFAULT_INJECTIONis already overloaded in different contexts.
* Parenthesis for macro variable.
* Lower max ofKFENCE_NUM_OBJECTSconfig variable.--- MAINTAINERS | 11 + include/linux/kfence.h | 174 ++++++++++ init/main.c | 2 + lib/Kconfig.debug | 1 + lib/Kconfig.kfence | 63 ++++ mm/Makefile | 1 + mm/kfence/Makefile | 3 + mm/kfence/core.c | 733 +++++++++++++++++++++++++++++++++++++++++ mm/kfence/kfence.h | 102 ++++++ mm/kfence/report.c | 219 ++++++++++++ 10 files changed, 1309 insertions(+) create mode 100644 include/linux/kfence.h create mode 100644 lib/Kconfig.kfence create mode 100644 mm/kfence/Makefile create mode 100644 mm/kfence/core.c create mode 100644 mm/kfence/kfence.h create mode 100644 mm/kfence/report.cMAINTAINERS +0
-0@@ -9673,6 +9673,17 @@ F: include/linux/keyctl.h F: include/uapi/linux/keyctl.h F: security/keys/
Quoted 10 lines
MAINTAINERS F: include/linux/keyctl.h · line 9682
Quoted 7 lines
MAINTAINERS +7
-0+KFENCE +M: Alexander Potapenko <[email protected]> +M: Marco Elver <[email protected]> +R: Dmitry Vyukov <[email protected]> +L: [email protected] +S: Maintained +F: Documentation/dev-tools/kfence.rstThis patch doesn't introduce this file yet, right? How about using a separate final patch for
MAINTAINERSupdate?
Sure.
Quoted 3 lines
Other than that,
Reviewed-by SeongJae Park <[email protected]>
Thanks!
Re: [PATCH v3 01/10] mm: add Kernel Electric-Fence infrastructure
In reply to Marco ElverQuoted 555 lines · Marco Elver
From: Alexander Potapenko <[email protected]>
This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors.
KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.
KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error. To detect out-of-bounds writes to memory within the object's page itself, KFENCE also uses pattern-based redzones. The following figure illustrates the page layout:
---+-----------+-----------+-----------+-----------+-----------+---
| xxxxxxxxx | O : | xxxxxxxxx | : O | xxxxxxxxx | | xxxxxxxxx | B : | xxxxxxxxx | : B | xxxxxxxxx | | x GUARD x | J : RED- | x GUARD x | RED- : J | x GUARD x | | xxxxxxxxx | E : ZONE | xxxxxxxxx | ZONE : E | xxxxxxxxx | | xxxxxxxxx | C : | xxxxxxxxx | : C | xxxxxxxxx | | xxxxxxxxx | T : | xxxxxxxxx | : T | xxxxxxxxx |---+-----------+-----------+-----------+-----------+-----------+---
Guarded allocations are set up based on a sample interval (can be set via kfence.sample_interval). After expiration of the sample interval, a guarded allocation from the KFENCE object pool is returned to the main allocator (SLAB or SLUB). At this point, the timer is reset, and the next allocation is set up after the expiration of the interval.
To enable/disable a KFENCE allocation through the main allocator's fast-path without overhead, KFENCE relies on static branches via the static keys infrastructure. The static branch is toggled to redirect the allocation to KFENCE. To date, we have verified by running synthetic benchmarks (sysbench I/O workloads) that a kernel compiled with KFENCE is performance-neutral compared to the non-KFENCE baseline.
For more details, see
Documentation/dev-tools/kfence.rst(added later in the series).Reviewed-by Dmitry Vyukov <[email protected]>Co-developed-by Marco Elver <[email protected]>Signed-off-by Marco Elver <[email protected]>Signed-off-by Alexander Potapenko <[email protected]>---
v3:
* Reports by SeongJae Park:
* Remove reference toDocumentation/dev-tools/kfence.rst.
* Remove redundant braces.
* UseCONFIG_KFENCE_NUM_OBJECTSinstead ofARRAY_SIZE(...).
* Align some comments.
* Add figure fromDocumentation/dev-tools/kfence.rstadded later in series to patch description.v2:
* Add missing__printfattribute toseq_con_printf, and fix new warning. [reported by kernel test robot <[email protected]>]
* Fix up some comments [reported by Jonathan Cameron].
* Remove 2 cases of redundant stack variable initialization [reported by Jonathan Cameron].
* Fix printf format [reported by kernel test robot <[email protected]>].
* Print (in kfence-#nn) after address, to more clearly establish link between first and second stacktrace [reported by Andrey Konovalov].
* Make choice between KASAN and KFENCE clearer inKconfighelp text [suggested by Dave Hansen].
* DocumentCONFIG_KFENCE_SAMPLE_INTERVAL=0.
* Shorten memory corruption report line length.
* Make/sys/module/kfence/parameters/sample_intervalroot-writable for all builds (to enable debugging, automatic dynamic tweaking).
* Reports by Dmitry Vyukov:
* Do not store negative size for right-located objects
* Only cache-align addresses of right-located objects.
* Runtoggle_allocation_gate()after KFENCE is enabled.
* Add empty line between allocation and free stacks.
* Add comment aboutSLAB_TYPESAFE_BY_RCU.
* Also skip internals for allocation/free stacks.
* s/KFENCE_FAULT_INJECTION/KFENCE_STRESS_TEST_FAULTS/ asFAULT_INJECTIONis already overloaded in different contexts.
* Parenthesis for macro variable.
* Lower max ofKFENCE_NUM_OBJECTSconfig variable.--- MAINTAINERS | 11 + include/linux/kfence.h | 174 ++++++++++ init/main.c | 2 + lib/Kconfig.debug | 1 + lib/Kconfig.kfence | 63 ++++ mm/Makefile | 1 + mm/kfence/Makefile | 3 + mm/kfence/core.c | 733 +++++++++++++++++++++++++++++++++++++++++ mm/kfence/kfence.h | 102 ++++++ mm/kfence/report.c | 219 ++++++++++++ 10 files changed, 1309 insertions(+) create mode 100644 include/linux/kfence.h create mode 100644 lib/Kconfig.kfence create mode 100644 mm/kfence/Makefile create mode 100644 mm/kfence/core.c create mode 100644 mm/kfence/kfence.h create mode 100644 mm/kfence/report.cMAINTAINERS +11
-0@@ -9673,6 +9673,17 @@ F: include/linux/keyctl.h F: include/uapi/linux/keyctl.h F: security/keys/ +KFENCE +M: Alexander Potapenko <[email protected]> +M: Marco Elver <[email protected]> +R: Dmitry Vyukov <[email protected]> +L: [email protected] +S: Maintained +F: Documentation/dev-tools/kfence.rst +F: include/linux/kfence.h +F: lib/Kconfig.kfence +F: mm/kfence/ + KFIFO M: Stefani Seibold <[email protected]> S: Maintainedinclude/linux/kfence.h new file +174
-0@@ -0,0 +1,174 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _LINUX_KFENCE_H +#define _LINUX_KFENCE_H + +#include <linux/mm.h> +#include <linux/percpu.h> +#include <linux/static_key.h> +#include <linux/types.h> + +#ifdef CONFIG_KFENCE + +/* +* We allocate an even number of pages, as it simplifies calculations to map +* address to metadata indices; effectively, the very first page serves as an +* extended guard page, but otherwise has no special purpose. +*/ +#define KFENCE_POOL_SIZE ((CONFIG_KFENCE_NUM_OBJECTS + 1) * 2 * PAGE_SIZE) +#ifdef CONFIG_HAVE_ARCH_KFENCE_STATIC_POOL +extern char __kfence_pool[KFENCE_POOL_SIZE ]; +#else +extern char *__kfence_pool; +#endif + +extern struct static_key_false kfence_allocation_key; + +/** +* is_kfence_address() - check if an address belongs to KFENCE pool +* @addr: address to check +* +* Return: true or false depending on whether the address is within the KFENCE +* object range. +* +* KFENCE objects live in a separate page range and are not to be intermixed +* with regular heap objects (e.g. KFENCE objects must never be added to the +* allocator freelists). Failing to do so may and will result in heap +* corruptions, therefore is_kfence_address() must be used to check whether +* an object requires specific handling. +*/ +static __always_inline bool is_kfence_address(const void *addr) +{ +return unlikely ((char *)addr >= __kfence_pool && + (char *)addr < __kfence_pool +KFENCE_POOL_SIZE ); +} + +/** +* kfence_init() - perform KFENCE initialization at boot time +*/ +void kfence_init (void ); + +/** +* kfence_shutdown_cache() - handle shutdown_cache() for KFENCE objects +* @s: cache being shut down +* +* Return: true on success, false if any leftover objects persist. +* +* Before shutting down a cache, one must ensure there are no remaining objects +* allocated from it. KFENCE objects are not referenced from the cache, so +* kfence_shutdown_cache() takes care of them. +*/ +bool __must_check kfence_shutdown_cache (struct kmem_cache *s); + +/* +* Allocate a KFENCE object. Allocators must not call this function directly, +* use kfence_alloc() instead. +*/ +void *__kfence_alloc (struct kmem_cache *s,size_t size,gfp_t flags); + +/** +* kfence_alloc() - allocate a KFENCE object with a low probability +* @s: struct kmem_cache with object requirements +* @size: exact size of the object to allocate (can be less than @s->size +* e.g. for kmalloc caches) +* @flags: GFP flags +* +* Return: +* * NULL - must proceed with allocating as usual, +* * non-NULL - pointer to a KFENCE object. +* +* kfence_alloc() should be inserted into the heap allocation fast path, +* allowing it to transparently return KFENCE-allocated objects with a low +* probability using a static branch (the probability is controlled by the +* kfence.sample_interval boot parameter). +*/ +static __always_inline void *kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags) +{ +return static_branch_unlikely (&kfence_allocation_key) ?__kfence_alloc (s, size, flags) : +NULL ; +} + +/** +* kfence_ksize() - get actual amount of memory allocated for a KFENCE object +* @addr: pointer to a heap object +* +* Return: +* * 0 - not a KFENCE object, must call __ksize() instead, +* * non-0 - this many bytes can be accessed without causing a memory error. +* +* kfence_ksize() returns the number of bytes requested for a KFENCE object at +* allocation time. This number may be less than the object size of the +* corresponding struct kmem_cache. +*/ +size_t kfence_ksize (const void *addr); + +/** +* kfence_object_start() - find the beginning of a KFENCE object +* @addr - address within a KFENCE-allocated object +* +* Return: address of the beginning of the object. +* +* SL[AU]B-allocated objects are laid out within a page one by one, so it is +* easy to calculate the beginning of an object given a pointer inside it and +* the object size. The same is not true for KFENCE, which places a single +* object at either end of the page. This helper function is used to find the +* beginning of a KFENCE-allocated object. +*/ +void *kfence_object_start (const void *addr); + +/* +* Release a KFENCE-allocated object to KFENCE pool. Allocators must not call +* this function directly, use kfence_free() instead. +*/ +void __kfence_free (void *addr); + +/** +* kfence_free() - try to release an arbitrary heap object to KFENCE pool +* @addr: object to be freed +* +* Return: +* * false - object doesn't belong to KFENCE pool and was ignored, +* * true - object was released to KFENCE pool. +* +* Release a KFENCE object and mark it as freed. May be called on any object, +* even non-KFENCE objects, to simplify integration of the hooks into the +* allocator's free codepath. The allocator must check the return value to +* determine if it was a KFENCE object or not. +*/ +static __always_inline __must_check bool kfence_free(void *addr) +{ +if (!is_kfence_address (addr)) +return false; +__kfence_free (addr); +return true; +} + +/** +* kfence_handle_page_fault() - perform page fault handling for KFENCE pages +* @addr: faulting address +* +* Return: +* * false - address outside KFENCE pool, +* * true - page fault handled by KFENCE, no additional handling required. +* +* A page fault inside KFENCE pool indicates a memory error, such as an +* out-of-bounds access, a use-after-free or an invalid memory access. In these +* cases KFENCE prints an error message and marks the offending page as +* present, so that the kernel can proceed. +*/ +bool __must_check kfence_handle_page_fault (unsigned long addr); + +#else /* CONFIG_KFENCE */ + +static inline bool is_kfence_address (const void *addr) {return false; } +static inline void kfence_init (void ) { } +static inline bool __must_check kfence_shutdown_cache (struct kmem_cache *s) {return true; } +static inline void *kfence_alloc (struct kmem_cache *s,size_t size,gfp_t flags) {return NULL ; } +static inline size_t kfence_ksize (const void *addr) {return 0 ; } +static inline void *kfence_object_start (const void *addr) {return NULL ; } +static inline bool __must_check kfence_free (void *addr) {return false; } +static inline bool __must_check kfence_handle_page_fault (unsigned long addr) {return false; } + +#endif + +#endif /* _LINUX_KFENCE_H */ init/main.c +2
-0@@ -39,6 +39,7 @@#include <linux/security.h> #include <linux/smp.h> #include <linux/profile.h> +#include <linux/kfence.h> #include <linux/rcupdate.h> #include <linux/moduleparam.h> #include <linux/kallsyms.h> @@ -942,6 +943,7 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)hrtimers_init ();softirq_init ();timekeeping_init (); +kfence_init ();/* * For best initial stack canary entropy, prepare it after: lib/Kconfig.debug +1
-0@@ -880,6 +880,7 @@ config DEBUG_STACKOVERFLOW If in doubt, say "N". source "lib/Kconfig.kasan" +source "lib/Kconfig.kfence" endmenu # "Memory Debugging"lib/Kconfig.kfence new file +63
-0@@ -0,0 +1,63 @@ +# SPDX-License-Identifier: GPL-2.0-only + +config HAVE_ARCH_KFENCE + bool + +config HAVE_ARCH_KFENCE_STATIC_POOL + bool + help + If the architecture supports using the static pool. + +menuconfig KFENCE + bool "KFENCE: low-overhead sampling-based memory safety error detector" + depends on HAVE_ARCH_KFENCE && !KASAN && (SLAB || SLUB) + depends on JUMP_LABEL # To ensure performance, require jump labels + select STACKTRACE + help + KFENCE is low-overhead sampling-based detector for heap out-of-bounds + access, use-after-free, and invalid-free errors. KFENCE is designed + to have negligible cost to permit enabling it in production + environments. + + Note that, KFENCE is not a substitute for explicit testing with tools + such as KASAN. KFENCE can detect a subset of bugs that KASAN can + detect, albeit at very different performance profiles. If you can + afford to use KASAN, continue using KASAN, for example in test + environments. If your kernel targets production use, and cannot + enable KASAN due to its cost, consider using KFENCE. + +if KFENCE + +config KFENCE_SAMPLE_INTERVAL + int "Default sample interval in milliseconds" + default 100 + help + The KFENCE sample interval determines the frequency with which heap + allocations will be guarded by KFENCE. May be overridden via boot + parameter "kfence.sample_interval". + + Set this to 0 to disable KFENCE by default, in which case only + setting "kfence.sample_interval" to a non-zero value enables KFENCE. + +config KFENCE_NUM_OBJECTS + int "Number of guarded objects available" + default 255 + range 1 16383 + help + The number of guarded objects available. For each KFENCE object, 2 + pages are required; with one containing the object and two adjacent + ones used as guard pages. + +config KFENCE_STRESS_TEST_FAULTS + int "Stress testing of fault handling and error reporting" + default 0 + depends on EXPERT + help + The inverse probability with which to randomly protect KFENCE object + pages, resulting in spurious use-after-frees. The main purpose of + this option is to stress test KFENCE with concurrent error reports + and allocations/frees. A value of 0 disables stress testing logic. + + The option is only to test KFENCE; set to 0 if you are unsure. + +endif # KFENCEmm/Makefile +1
-0@@ -81,6 +81,7 @@ obj-$(CONFIG_PAGE_POISONING) += page_poison.o obj-$(CONFIG_SLAB) += slab.o obj-$(CONFIG_SLUB) += slub.o obj-$(CONFIG_KASAN) += kasan/ +obj-$(CONFIG_KFENCE) += kfence/ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_MEMORY_HOTPLUG) += memory_hotplug.o obj-$(CONFIG_MEMTEST) += memtest.omm/kfence/Makefile new file +3
-0@@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-$(CONFIG_KFENCE) := core.o report.omm/kfence/core.c new file +116
-0@@ -0,0 +1,733 @@ +// SPDX-License-Identifier: GPL-2.0 + +#define pr_fmt (fmt) "kfence: " fmt + +#include <linux/atomic.h> +#include <linux/bug.h> +#include <linux/debugfs.h> +#include <linux/kcsan-checks.h> +#include <linux/kfence.h> +#include <linux/list.h> +#include <linux/lockdep.h> +#include <linux/moduleparam.h> +#include <linux/random.h> +#include <linux/rcupdate.h> +#include <linux/seq_file.h> +#include <linux/slab.h> +#include <linux/spinlock.h> +#include <linux/string.h> + +#include <asm/kfence.h> + +#include "kfence.h" + +/* Disables KFENCE on the first warning assuming an irrecoverable error. */ +#define KFENCE_WARN_ON (cond) \ + ({ \ + const bool __cond = WARN_ON(cond); \ + if (unlikely(__cond)) \ + WRITE_ONCE(kfence_enabled, false); \ + __cond; \ + }) + +#ifndef CONFIG_KFENCE_STRESS_TEST_FAULTS /* Only defined with CONFIG_EXPERT. */ +#define CONFIG_KFENCE_STRESS_TEST_FAULTS 0 +#endif + +/* === Data ================================================================= */ + +static unsigned long kfence_sample_interval__read_mostly =CONFIG_KFENCE_SAMPLE_INTERVAL ; + +#ifdef MODULE_PARAM_PREFIX +#undef MODULE_PARAM_PREFIX +#endif +#define MODULE_PARAM_PREFIX "kfence." +module_param_named (sample_interval, kfence_sample_interval, ulong,0600 ); + +static bool kfence_enabled__read_mostly ; + +/* +* The pool of pages used for guard pages and objects. If supported, allocated +* statically, so that is_kfence_address() avoids a pointer load, and simply +* compares against a constant address. Assume that if KFENCE is compiled into +* the kernel, it is usually enabled, and the space is to be allocated one way +* or another. +*/ +#ifdef CONFIG_HAVE_ARCH_KFENCE_STATIC_POOL +char __kfence_pool[KFENCE_POOL_SIZE ]__aligned(KFENCE_POOL_ALIGNMENT) ; +#else +char *__kfence_pool__read_mostly ; +#endif +EXPORT_SYMBOL (__kfence_pool);/* Export for test modules. */ + +/* +* Per-object metadata, with one-to-one mapping of object metadata to +* backing pages (in __kfence_pool). +*/ +static_assert (CONFIG_KFENCE_NUM_OBJECTS >0 ); +struct kfence_metadata kfence_metadata[CONFIG_KFENCE_NUM_OBJECTS ]; + +/* Freelist with available objects. */ +static struct list_head kfence_freelist =LIST_HEAD_INIT (kfence_freelist); +static DEFINE_RAW_SPINLOCK (kfence_freelist_lock);/* Lock protecting freelist. */ + +/* The static key to set up a KFENCE allocation. */ +DEFINE_STATIC_KEY_FALSE (kfence_allocation_key); + +/* Gates the allocation, ensuring only one succeeds in a given period. */ +static atomic_t allocation_gate =ATOMIC_INIT (1 ); + +/* Wait queue to wake up allocation-gate timer task. */ +static DECLARE_WAIT_QUEUE_HEAD (allocation_wait); + +/* Statistics counters for debugfs. */ +enum kfence_counter_id { +KFENCE_COUNTER_ALLOCATED , +KFENCE_COUNTER_ALLOCS , +KFENCE_COUNTER_FREES , +KFENCE_COUNTER_BUGS , +KFENCE_COUNTER_COUNT , +}; +static atomic_long_t counters[KFENCE_COUNTER_COUNT ]; +static const char *const counter_names[] = { + [KFENCE_COUNTER_ALLOCATED ] ="currently allocated" , + [KFENCE_COUNTER_ALLOCS ] ="total allocations" , + [KFENCE_COUNTER_FREES ] ="total frees" , + [KFENCE_COUNTER_BUGS ] ="total bugs" , +}; +static_assert (ARRAY_SIZE (counter_names) ==KFENCE_COUNTER_COUNT ); + +/* === Internals ============================================================ */ + +static bool kfence_protect (unsigned long addr) +{ +return !KFENCE_WARN_ON (!kfence_protect_page (ALIGN_DOWN (addr,PAGE_SIZE ), true)); +} + +static bool kfence_unprotect (unsigned long addr) +{ +return !KFENCE_WARN_ON (!kfence_protect_page (ALIGN_DOWN (addr,PAGE_SIZE ), false)); +} + +static inline struct kfence_metadata *addr_to_metadata (unsigned long addr) +{ +long index; + +/* The checks do not affect performance; only called from slow-paths. */
mm/kfence/core.c line 126
Quoted 10 lines
mm/kfence/core.c new file +10
-0+ +if (!is_kfence_address ((void *)addr)) +return NULL ; + +/* +* May be an invalid index if called with an address at the edge of +* __kfence_pool, in which case we would report an "invalid access" +* error. +*/ + index = (addr - (unsigned long )__kfence_pool) / (PAGE_SIZE *2 ) -1 ;
Why do we subtract 1 here? We do have the metadata entry reserved for something?
Quoted 686 lines
mm/kfence/core.c new file +607
-0+if (index <0 || index >=CONFIG_KFENCE_NUM_OBJECTS ) +return NULL ; + +return &kfence_metadata[index]; +} + +static inline unsigned long metadata_to_pageaddr (const struct kfence_metadata *meta) +{ +unsigned long offset = (meta - kfence_metadata +1 ) *PAGE_SIZE *2 ; +unsigned long pageaddr = (unsigned long )&__kfence_pool[offset]; + +/* The checks do not affect performance; only called from slow-paths. */ + +/* Only call with a pointer into kfence_metadata. */ +if (KFENCE_WARN_ON (meta < kfence_metadata || + meta >= kfence_metadata +CONFIG_KFENCE_NUM_OBJECTS )) +return 0 ; + +/* +* This metadata object only ever maps to 1 page; verify the calculation +* happens and that the stored address was not corrupted. +*/ +if (KFENCE_WARN_ON (ALIGN_DOWN (meta->addr,PAGE_SIZE ) != pageaddr)) +return 0 ; + +return pageaddr; +} + +/* +* Update the object's metadata state, including updating the alloc/free stacks +* depending on the state transition. +*/ +static noinline void metadata_update_state(struct kfence_metadata *meta, +enum kfence_object_state next) +{ +unsigned long *entries = next ==KFENCE_OBJECT_FREED ? meta->free_stack : meta->alloc_stack; +/* +* Skip over 1 (this) functions; noinline ensures we do not accidentally +* skip over the caller by never inlining. +*/ +const int nentries =stack_trace_save (entries,KFENCE_STACK_DEPTH ,1 ); + +lockdep_assert_held (&meta->lock); + +if (next ==KFENCE_OBJECT_FREED ) + meta->num_free_stack = nentries; +else + meta->num_alloc_stack = nentries; + +/* +* Pairs with READ_ONCE() in +* kfence_shutdown_cache(), +* kfence_handle_page_fault(). +*/ +WRITE_ONCE (meta->state, next); +} + +/* Write canary byte to @addr. */ +static inline bool set_canary_byte (u8 *addr) +{ + *addr =KFENCE_CANARY_PATTERN (addr); +return true; +} + +/* Check canary byte at @addr. */ +static inline bool check_canary_byte (u8 *addr) +{ +if (*addr ==KFENCE_CANARY_PATTERN (addr)) +return true; + +atomic_long_inc (&counters[KFENCE_COUNTER_BUGS ]); +kfence_report_error ((unsigned long )addr,addr_to_metadata ((unsigned long )addr), +KFENCE_ERROR_CORRUPTION ); +return false; +} + +static inline void for_each_canary (const struct kfence_metadata *meta,bool (*fn)(u8 *)) +{ +unsigned long addr; + +lockdep_assert_held (&meta->lock); + +for (addr =ALIGN_DOWN (meta->addr,PAGE_SIZE ); addr < meta->addr; addr++) { +if (!fn ((u8 *)addr)) +break ; + } + +for (addr = meta->addr + meta->size; addr <PAGE_ALIGN (meta->addr); addr++) { +if (!fn ((u8 *)addr)) +break ; + } +} + +static void *kfence_guarded_alloc (struct kmem_cache *cache,size_t size,gfp_t gfp) +{ +struct kfence_metadata *meta =NULL ; +unsigned long flags; +void *addr; + +/* Try to obtain a free object. */ +raw_spin_lock_irqsave (&kfence_freelist_lock, flags); +if (!list_empty (&kfence_freelist)) { + meta = list_entry(kfence_freelist.next, struct kfence_metadata, list); +list_del_init (&meta->list); + } +raw_spin_unlock_irqrestore (&kfence_freelist_lock, flags); +if (!meta) +return NULL ; + +if (unlikely (!raw_spin_trylock_irqsave (&meta->lock, flags))) { +/* +* This is extremely unlikely -- we are reporting on a +* use-after-free, which locked meta->lock, and the reporting +* code via printk calls kmalloc() which ends up in +* kfence_alloc() and tries to grab the same object that we're +* reporting on. While it has never been observed, lockdep does +* report that there is a possibility of deadlock. Fix it by +* using trylock and bailing out gracefully. +*/ +raw_spin_lock_irqsave (&kfence_freelist_lock, flags); +/* Put the object back on the freelist. */ +list_add_tail (&meta->list, &kfence_freelist); +raw_spin_unlock_irqrestore (&kfence_freelist_lock, flags); + +return NULL ; + } + + meta->addr =metadata_to_pageaddr (meta); +/* Unprotect if we're reusing this page. */ +if (meta->state ==KFENCE_OBJECT_FREED ) +kfence_unprotect (meta->addr); + +/* +* Note: for allocations made before RNG initialization, will always +* return zero. We still benefit from enabling KFENCE as early as +* possible, even when the RNG is not yet available, as this will allow +* KFENCE to detect bugs due to earlier allocations. The only downside +* is that the out-of-bounds accesses detected are deterministic for +* such allocations. +*/ +if (prandom_u32_max (2 )) { +/* Allocate on the "right" side, re-calculate address. */ + meta->addr +=PAGE_SIZE - size; + meta->addr =ALIGN_DOWN (meta->addr, cache->align); + } + +/* Update remaining metadata. */ +metadata_update_state (meta,KFENCE_OBJECT_ALLOCATED ); +/* Pairs with READ_ONCE() in kfence_shutdown_cache(). */ +WRITE_ONCE (meta->cache, cache); + meta->size = size; +for_each_canary (meta, set_canary_byte); +virt_to_page (meta->addr)->slab_cache = cache; + +raw_spin_unlock_irqrestore (&meta->lock, flags); + +/* Memory initialization. */ + +/* +* We check slab_want_init_on_alloc() ourselves, rather than letting +* SL*B do the initialization, as otherwise we might overwrite KFENCE's +* redzone. +*/ + addr = (void *)meta->addr; +if (unlikely (slab_want_init_on_alloc (gfp, cache))) +memzero_explicit (addr, size); +if (cache->ctor) + cache->ctor (addr); + +if (CONFIG_KFENCE_STRESS_TEST_FAULTS && !prandom_u32_max (CONFIG_KFENCE_STRESS_TEST_FAULTS )) +kfence_protect (meta->addr);/* Random "faults" by protecting the object. */ + +atomic_long_inc (&counters[KFENCE_COUNTER_ALLOCATED ]); +atomic_long_inc (&counters[KFENCE_COUNTER_ALLOCS ]); + +return addr; +} + +static void kfence_guarded_free (void *addr,struct kfence_metadata *meta) +{ +struct kcsan_scoped_access assert_page_exclusive; +unsigned long flags; + +raw_spin_lock_irqsave (&meta->lock, flags); + +if (meta->state !=KFENCE_OBJECT_ALLOCATED || meta->addr != (unsigned long )addr) { +/* Invalid or double-free, bail out. */ +atomic_long_inc (&counters[KFENCE_COUNTER_BUGS ]); +kfence_report_error ((unsigned long )addr, meta,KFENCE_ERROR_INVALID_FREE ); +raw_spin_unlock_irqrestore (&meta->lock, flags); +return ; + } + +/* Detect racy use-after-free, or incorrect reallocation of this page by KFENCE. */ +kcsan_begin_scoped_access ((void *)ALIGN_DOWN ((unsigned long )addr,PAGE_SIZE ),PAGE_SIZE , +KCSAN_ACCESS_SCOPED |KCSAN_ACCESS_WRITE |KCSAN_ACCESS_ASSERT , + &assert_page_exclusive); + +if (CONFIG_KFENCE_STRESS_TEST_FAULTS ) +kfence_unprotect ((unsigned long )addr);/* To check canary bytes. */ + +/* Restore page protection if there was an OOB access. */ +if (meta->unprotected_page) { +kfence_protect (meta->unprotected_page); + meta->unprotected_page =0 ; + } + +/* Check canary bytes for memory corruption. */ +for_each_canary (meta, check_canary_byte); + +/* +* Clear memory if init-on-free is set. While we protect the page, the +* data is still there, and after a use-after-free is detected, we +* unprotect the page, so the data is still accessible. +*/ +if (unlikely (slab_want_init_on_free (meta->cache))) +memzero_explicit (addr, meta->size); + +/* Mark the object as freed. */ +metadata_update_state (meta,KFENCE_OBJECT_FREED ); + +raw_spin_unlock_irqrestore (&meta->lock, flags); + +/* Protect to detect use-after-frees. */ +kfence_protect ((unsigned long )addr); + +/* Add it to the tail of the freelist for reuse. */ +raw_spin_lock_irqsave (&kfence_freelist_lock, flags); +KFENCE_WARN_ON (!list_empty (&meta->list)); +list_add_tail (&meta->list, &kfence_freelist); +kcsan_end_scoped_access (&assert_page_exclusive); +raw_spin_unlock_irqrestore (&kfence_freelist_lock, flags); + +atomic_long_dec (&counters[KFENCE_COUNTER_ALLOCATED ]); +atomic_long_inc (&counters[KFENCE_COUNTER_FREES ]); +} + +static void rcu_guarded_free (struct rcu_head *h) +{ + struct kfence_metadata *meta = container_of(h, struct kfence_metadata, rcu_head); + +kfence_guarded_free ((void *)meta->addr, meta); +} + +static bool __init kfence_initialize_pool (void ) +{ +unsigned long addr; +struct page *pages; +int i; + +if (!arch_kfence_initialize_pool ()) +return false; + + addr = (unsigned long )__kfence_pool; + pages =virt_to_page (addr); + +/* +* Set up object pages: they must have PG_slab set, to avoid freeing +* these as real pages. +* +* We also want to avoid inserting kfence_free() in the kfree() +* fast-path in SLUB, and therefore need to ensure kfree() correctly +* enters __slab_free() slow-path. +*/ +for (i =0 ; i <KFENCE_POOL_SIZE /PAGE_SIZE ; i++) { +if (!i || (i %2 )) +continue ; + +__SetPageSlab (&pages[i]); + } + +/* +* Protect the first 2 pages. The first page is mostly unnecessary, and +* merely serves as an extended guard page. However, adding one +* additional page in the beginning gives us an even number of pages, +* which simplifies the mapping of address to metadata index. +*/ +for (i =0 ; i <2 ; i++) { +if (unlikely (!kfence_protect (addr))) +return false; + + addr +=PAGE_SIZE ; + } + +for (i =0 ; i <CONFIG_KFENCE_NUM_OBJECTS ; i++) { +struct kfence_metadata *meta = &kfence_metadata[i]; + +/* Initialize metadata. */ +INIT_LIST_HEAD (&meta->list); +raw_spin_lock_init (&meta->lock); + meta->state =KFENCE_OBJECT_UNUSED ; + meta->addr = addr;/* Initialize for validation in metadata_to_pageaddr(). */ +list_add_tail (&meta->list, &kfence_freelist); + +/* Protect the right redzone. */ +if (unlikely (!kfence_protect (addr +PAGE_SIZE ))) +return false; + + addr +=2 *PAGE_SIZE ; + } + +return true; +} + +/* === DebugFS Interface ==================================================== */ + +static int stats_show (struct seq_file *seq,void *v) +{ +int i; + +seq_printf (seq,"enabled: %i\n" ,READ_ONCE (kfence_enabled)); +for (i =0 ; i <KFENCE_COUNTER_COUNT ; i++) +seq_printf (seq,"%s: %ld\n" , counter_names[i],atomic_long_read (&counters[i])); + +return 0 ; +} +DEFINE_SHOW_ATTRIBUTE (stats); + +/* +* debugfs seq_file operations for /sys/kernel/debug/kfence/objects. +* start_object() and next_object() return the object index + 1, because NULL is used +* to stop iteration. +*/ +static void *start_object (struct seq_file *seq,loff_t *pos) +{ +if (*pos <CONFIG_KFENCE_NUM_OBJECTS ) +return (void *)((long )*pos +1 ); +return NULL ; +} + +static void stop_object (struct seq_file *seq,void *v) +{ +} + +static void *next_object (struct seq_file *seq,void *v,loff_t *pos) +{ + ++*pos; +if (*pos <CONFIG_KFENCE_NUM_OBJECTS ) +return (void *)((long )*pos +1 ); +return NULL ; +} + +static int show_object (struct seq_file *seq,void *v) +{ +struct kfence_metadata *meta = &kfence_metadata[(long )v -1 ]; +unsigned long flags; + +raw_spin_lock_irqsave (&meta->lock, flags); +kfence_print_object (seq, meta); +raw_spin_unlock_irqrestore (&meta->lock, flags); +seq_puts (seq,"---------------------------------\n" ); + +return 0 ; +} + +static const struct seq_operations object_seqops = { + .start = start_object, + .next = next_object, + .stop = stop_object, + .show = show_object, +}; + +static int open_objects (struct inode *inode,struct file *file) +{ +return seq_open (file, &object_seqops); +} + +static const struct file_operations objects_fops = { + .open = open_objects, + .read = seq_read, + .llseek = seq_lseek, +}; + +static int __init kfence_debugfs_init (void ) +{ +struct dentry *kfence_dir =debugfs_create_dir ("kfence" ,NULL ); + +debugfs_create_file ("stats" ,0444 , kfence_dir,NULL , &stats_fops); +debugfs_create_file ("objects" ,0400 , kfence_dir,NULL , &objects_fops); +return 0 ; +} + +late_initcall (kfence_debugfs_init); + +/* === Allocation Gate Timer ================================================ */ + +/* +* Set up delayed work, which will enable and disable the static key. We need to +* use a work queue (rather than a simple timer), since enabling and disabling a +* static key cannot be done from an interrupt. +*/ +static struct delayed_work kfence_timer; +static void toggle_allocation_gate (struct work_struct *work) +{ +if (!READ_ONCE (kfence_enabled)) +return ; + +/* Enable static key, and await allocation to happen. */ +atomic_set (&allocation_gate,0 ); +static_branch_enable (&kfence_allocation_key); +wait_event (allocation_wait,atomic_read (&allocation_gate) !=0 ); + +/* Disable static key and reset timer. */ +static_branch_disable (&kfence_allocation_key); +schedule_delayed_work (&kfence_timer,msecs_to_jiffies (kfence_sample_interval)); +} +static DECLARE_DELAYED_WORK(kfence_timer, toggle_allocation_gate); + +/* === Public interface ===================================================== */ + +void __init kfence_init (void ) +{ +/* Setting kfence_sample_interval to 0 on boot disables KFENCE. */ +if (!kfence_sample_interval) +return ; + +if (!kfence_initialize_pool ()) { +pr_err ("%s failed\n" , __func__); +return ; + } + +WRITE_ONCE (kfence_enabled, true); +schedule_delayed_work (&kfence_timer,0 ); +pr_info ("initialized - using %lu bytes for %d objects" ,KFENCE_POOL_SIZE , +CONFIG_KFENCE_NUM_OBJECTS ); +if (IS_ENABLED (CONFIG_DEBUG_KERNEL )) +pr_cont (" at 0x%px-0x%px\n" , (void *)__kfence_pool, + (void *)(__kfence_pool +KFENCE_POOL_SIZE )); +else +pr_cont ("\n" ); +} + +bool kfence_shutdown_cache (struct kmem_cache *s) +{ +unsigned long flags; +struct kfence_metadata *meta; +int i; + +for (i =0 ; i <CONFIG_KFENCE_NUM_OBJECTS ; i++) { +bool in_use; + + meta = &kfence_metadata[i]; + +/* +* If we observe some inconsistent cache and state pair where we +* should have returned false here, cache destruction is racing +* with either kmem_cache_alloc() or kmem_cache_free(). Taking +* the lock will not help, as different critical section +* serialization will have the same outcome. +*/ +if (READ_ONCE (meta->cache) != s || +READ_ONCE (meta->state) !=KFENCE_OBJECT_ALLOCATED ) +continue ; + +raw_spin_lock_irqsave (&meta->lock, flags); + in_use = meta->cache == s && meta->state ==KFENCE_OBJECT_ALLOCATED ; +raw_spin_unlock_irqrestore (&meta->lock, flags); + +if (in_use) +return false; + } + +for (i =0 ; i <CONFIG_KFENCE_NUM_OBJECTS ; i++) { + meta = &kfence_metadata[i]; + +/* See above. */ +if (READ_ONCE (meta->cache) != s ||READ_ONCE (meta->state) !=KFENCE_OBJECT_FREED ) +continue ; + +raw_spin_lock_irqsave (&meta->lock, flags); +if (meta->cache == s && meta->state ==KFENCE_OBJECT_FREED ) + meta->cache =NULL ; +raw_spin_unlock_irqrestore (&meta->lock, flags); + } + +return true; +} + +void *__kfence_alloc (struct kmem_cache *s,size_t size,gfp_t flags) +{ +/* +* allocation_gate only needs to become non-zero, so it doesn't make +* sense to continue writing to it and pay the associated contention +* cost, in case we have a large number of concurrent allocations. +*/ +if (atomic_read (&allocation_gate) ||atomic_inc_return (&allocation_gate) >1 ) +return NULL ; +wake_up (&allocation_wait); + +if (!READ_ONCE (kfence_enabled)) +return NULL ; + +if (size >PAGE_SIZE ) +return NULL ; + +return kfence_guarded_alloc (s, size, flags); +} + +size_t kfence_ksize (const void *addr) +{ +const struct kfence_metadata *meta =addr_to_metadata ((unsigned long )addr); + +/* +* Read locklessly -- if there is a race with __kfence_alloc(), this is +* either a use-after-free or invalid access. +*/ +return meta ? meta->size :0 ; +} + +void *kfence_object_start (const void *addr) +{ +const struct kfence_metadata *meta =addr_to_metadata ((unsigned long )addr); + +/* +* Read locklessly -- if there is a race with __kfence_alloc(), this is +* either a use-after-free or invalid access. +*/ +return meta ? (void *)meta->addr :NULL ; +} + +void __kfence_free (void *addr) +{ +struct kfence_metadata *meta =addr_to_metadata ((unsigned long )addr); + +/* +* If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing +* the object, as the object page may be recycled for other-typed +* objects once it has been freed. +*/ +if (unlikely (meta->cache->flags &SLAB_TYPESAFE_BY_RCU )) +call_rcu (&meta->rcu_head, rcu_guarded_free); +else +kfence_guarded_free (addr, meta); +} + +bool kfence_handle_page_fault (unsigned long addr) +{ +const int page_index = (addr - (unsigned long )__kfence_pool) /PAGE_SIZE ; +struct kfence_metadata *to_report =NULL ; +enum kfence_error_type error_type; +unsigned long flags; + +if (!is_kfence_address ((void *)addr)) +return false; + +if (!READ_ONCE (kfence_enabled))/* If disabled at runtime ... */ +return kfence_unprotect (addr);/* ... unprotect and proceed. */ + +atomic_long_inc (&counters[KFENCE_COUNTER_BUGS ]); + +if (page_index %2 ) { +/* This is a redzone, report a buffer overflow. */ +struct kfence_metadata *meta; +int distance =0 ; + + meta =addr_to_metadata (addr -PAGE_SIZE ); +if (meta &&READ_ONCE (meta->state) ==KFENCE_OBJECT_ALLOCATED ) { + to_report = meta; +/* Data race ok; distance calculation approximate. */ + distance = addr -data_race (meta->addr + meta->size); + } + + meta =addr_to_metadata (addr +PAGE_SIZE ); +if (meta &&READ_ONCE (meta->state) ==KFENCE_OBJECT_ALLOCATED ) { +/* Data race ok; distance calculation approximate. */ +if (!to_report || distance >data_race (meta->addr) - addr) + to_report = meta; + } + +if (!to_report) + goto out; + +raw_spin_lock_irqsave (&to_report->lock, flags); + to_report->unprotected_page = addr; + error_type =KFENCE_ERROR_OOB ; + +/* +* If the object was freed before we took the look we can still +* report this as an OOB -- the report will simply show the +* stacktrace of the free as well. +*/ + }else { + to_report =addr_to_metadata (addr); +if (!to_report) + goto out; + +raw_spin_lock_irqsave (&to_report->lock, flags); + error_type =KFENCE_ERROR_UAF ; +/* +* We may race with __kfence_alloc(), and it is possible that a +* freed object may be reallocated. We simply report this as a +* use-after-free, with the stack trace showing the place where +* the object was re-allocated. +*/ + } + +out: +if (to_report) { +kfence_report_error (addr, to_report, error_type); +raw_spin_unlock_irqrestore (&to_report->lock, flags); + }else { +/* This may be a UAF or OOB access, but we can't be sure. */ +kfence_report_error (addr,NULL ,KFENCE_ERROR_INVALID ); + } + +return kfence_unprotect (addr);/* Unprotect and let access proceed. */ +}mm/kfence/kfence.h new file +73
-0@@ -0,0 +1,102 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef MM_KFENCE_KFENCE_H +#define MM_KFENCE_KFENCE_H + +#include <linux/mm.h> +#include <linux/slab.h> +#include <linux/spinlock.h> +#include <linux/types.h> + +#include "../slab.h" /* for struct kmem_cache */ + +/* For non-debug builds, avoid leaking kernel pointers into dmesg. */ +#ifdef CONFIG_DEBUG_KERNEL +#define PTR_FMT "%px" +#else +#define PTR_FMT "%p" +#endif + +/* + * Get the canary byte pattern for @addr. Use a pattern that varies based on the + * lower 3 bits of the address, to detect memory corruptions with higher + * probability, where similar constants are used. + */ +#define KFENCE_CANARY_PATTERN(addr) ((u8)0xaa ^ (u8)((unsigned long)(addr) & 0x7)) + +/* Maximum stack depth for reports. */ +#define KFENCE_STACK_DEPTH 64 + +/* KFENCE object states. */ +enum kfence_object_state { + KFENCE_OBJECT_UNUSED, /* Object is unused. */ + KFENCE_OBJECT_ALLOCATED, /* Object is currently allocated. */ + KFENCE_OBJECT_FREED, /* Object was allocated, and then freed. */ +}; + +/* KFENCE metadata per guarded allocation. */ +struct kfence_metadata { + struct list_head list; /* Freelist node; access under kfence_freelist_lock. */ + struct rcu_head rcu_head; /* For delayed freeing. */ + + /* + * Lock protecting below data; to ensure consistency of the below data, + * since the following may execute concurrently: __kfence_alloc(), + * __kfence_free(), kfence_handle_page_fault(). However, note that we + * cannot grab the same metadata off the freelist twice, and multiple + * __kfence_alloc() cannot run concurrently on the same metadata. + */ + raw_spinlock_t lock; + + /* The current state of the object; see above. */ + enum kfence_object_state state; + + /* + * Allocated object address; cannot be calculated from size, because of + * alignment requirements. + * + * Invariant: ALIGN_DOWN(addr, PAGE_SIZE) is constant. + */ + unsigned long addr; + + /* + * The size of the original allocation. + */ + size_t size; + + /* + * The kmem_cache cache of the last allocation; NULL if never allocated + * or the cache has already been destroyed. + */ + struct kmem_cache *cache; + + /*
mm/kfence/kfence.h line 83
Quoted 10 lines
mm/kfence/kfence.h new file +10
-0+* In case of an invalid access, the page that was unprotected; we +* optimistically only store address. +*/ +unsigned long unprotected_page; + +/* Allocation and free stack information. */ +int num_alloc_stack; +int num_free_stack; +unsigned long alloc_stack[KFENCE_STACK_DEPTH ]; +unsigned long free_stack[KFENCE_STACK_DEPTH ];
It was a concious decision to not use stackdepot, right? Perhaps it makes sense to document the reason somewhere.
Quoted 247 lines
mm/kfence/kfence.h new file +19
-0+}; + +extern struct kfence_metadata kfence_metadata[CONFIG_KFENCE_NUM_OBJECTS ]; + +/* KFENCE error types for report generation. */ +enum kfence_error_type { +KFENCE_ERROR_OOB ,/* Detected a out-of-bounds access. */ +KFENCE_ERROR_UAF ,/* Detected a use-after-free access. */ +KFENCE_ERROR_CORRUPTION ,/* Detected a memory corruption on free. */ +KFENCE_ERROR_INVALID ,/* Invalid access of unknown type. */ +KFENCE_ERROR_INVALID_FREE ,/* Invalid free. */ +}; + +void kfence_report_error (unsigned long address,const struct kfence_metadata *meta, +enum kfence_error_type type); + +void kfence_print_object (struct seq_file *seq,const struct kfence_metadata *meta); + +#endif /* MM_KFENCE_KFENCE_H */ mm/kfence/report.c new file +219
-1@@ -0,0 +1,219 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <stdarg.h> + +#include <linux/kernel.h> +#include <linux/lockdep.h> +#include <linux/printk.h> +#include <linux/seq_file.h> +#include <linux/stacktrace.h> +#include <linux/string.h> + +#include <asm/kfence.h> + +#include "kfence.h" + +/* Helper function to either print to a seq_file or to console. */ +__printf(2, 3) +static void seq_con_printf (struct seq_file *seq,const char *fmt, ...) +{ +va_list args; + +va_start (args, fmt); +if (seq) +seq_vprintf (seq, fmt, args); +else +vprintk (fmt, args); +va_end (args); +} + +/* +* Get the number of stack entries to skip get out of MM internals. @type is +* optional, and if set to NULL, assumes an allocation or free stack. +*/ +static int get_stack_skipnr (const unsigned long stack_entries[],int num_entries, +const enum kfence_error_type *type) +{ +char buf[64 ]; +int skipnr, fallback =0 ; +bool is_access_fault = false; + +if (type) { +/* Depending on error type, find different stack entries. */ +switch (*type) { +case KFENCE_ERROR_UAF : +case KFENCE_ERROR_OOB : +case KFENCE_ERROR_INVALID : + is_access_fault = true; +break ; +case KFENCE_ERROR_CORRUPTION : +case KFENCE_ERROR_INVALID_FREE : +break ; + } + } + +for (skipnr =0 ; skipnr < num_entries; skipnr++) { +int len =scnprintf (buf,sizeof (buf),"%ps" , (void *)stack_entries[skipnr]); + +if (is_access_fault) { +if (!strncmp (buf,KFENCE_SKIP_ARCH_FAULT_HANDLER , len)) + goto found; + }else { +if (str_has_prefix (buf,"kfence_" ) ||str_has_prefix (buf,"__kfence_" )) + fallback = skipnr +1 ;/* In case of tail calls into kfence. */ + +/* Also the *_bulk() variants by only checking prefixes. */ +if (str_has_prefix (buf,"kfree" ) || +str_has_prefix (buf,"kmem_cache_free" ) || +str_has_prefix (buf,"__kmalloc" ) || +str_has_prefix (buf,"kmem_cache_alloc" )) + goto found; + } + } +if (fallback < num_entries) +return fallback; +found: + skipnr++; +return skipnr < num_entries ? skipnr :0 ; +} + +static void kfence_print_stack (struct seq_file *seq,const struct kfence_metadata *meta, +bool show_alloc) +{ +const unsigned long *entries = show_alloc ? meta->alloc_stack : meta->free_stack; +const int nentries = show_alloc ? meta->num_alloc_stack : meta->num_free_stack; + +if (nentries) { +/* Skip allocation/free internals stack. */ +int i =get_stack_skipnr (entries, nentries,NULL ); + +/* stack_trace_seq_print() does not exist; open code our own. */ +for (; i < nentries; i++) +seq_con_printf (seq," %pS\n" , (void *)entries[i]); + }else { +seq_con_printf (seq," no %s stack\n" , show_alloc ?"allocation" :"deallocation" ); + } +} + +void kfence_print_object (struct seq_file *seq,const struct kfence_metadata *meta) +{ +const int size =abs (meta->size); +const unsigned long start = meta->addr; +const struct kmem_cache *const cache = meta->cache; + +lockdep_assert_held (&meta->lock); + +if (meta->state ==KFENCE_OBJECT_UNUSED ) { +seq_con_printf (seq,"kfence-#%zd unused\n" , meta - kfence_metadata); +return ; + } + +seq_con_printf (seq, +"kfence-#%zd [0x" PTR_FMT "-0x" PTR_FMT +", size=%d, cache=%s] allocated in:\n" , + meta - kfence_metadata, (void *)start, (void *)(start + size -1 ), size, + (cache && cache->name) ? cache->name :"<destroyed>" ); +kfence_print_stack (seq, meta, true); + +if (meta->state ==KFENCE_OBJECT_FREED ) { +seq_con_printf (seq,"\nfreed in:\n" ); +kfence_print_stack (seq, meta, false); + } +} + +/* +* Show bytes at @addr that are different from the expected canary values, up to +* @max_bytes. +*/ +static void print_diff_canary (const u8 *addr,size_t max_bytes) +{ +const u8 *max_addr =min ((const u8 *)PAGE_ALIGN ((unsigned long )addr), addr + max_bytes); + +pr_cont ("[" ); +for (; addr < max_addr; addr++) { +if (*addr ==KFENCE_CANARY_PATTERN (addr)) +pr_cont (" ." ); +else if (IS_ENABLED (CONFIG_DEBUG_KERNEL )) +pr_cont (" 0x%02x" , *addr); +else /* Do not leak kernel memory in non-debug builds. */ +pr_cont (" !" ); + } +pr_cont (" ]" ); +} + +void kfence_report_error (unsigned long address,const struct kfence_metadata *meta, +enum kfence_error_type type) +{ +unsigned long stack_entries[KFENCE_STACK_DEPTH ] = {0 }; +int num_stack_entries =stack_trace_save (stack_entries,KFENCE_STACK_DEPTH ,1 ); +int skipnr =get_stack_skipnr (stack_entries, num_stack_entries, &type); +const ptrdiff_t object_index = meta ? meta - kfence_metadata :-1 ; + +/* Require non-NULL meta, except if KFENCE_ERROR_INVALID. */ +if (WARN_ON (type !=KFENCE_ERROR_INVALID && !meta)) +return ; + +if (meta) +lockdep_assert_held (&meta->lock); +/* +* Because we may generate reports in printk-unfriendly parts of the +* kernel, such as scheduler code, the use of printk() could deadlock. +* Until such time that all printing code here is safe in all parts of +* the kernel, accept the risk, and just get our message out (given the +* system might already behave unpredictably due to the memory error). +* As such, also disable lockdep to hide warnings, and avoid disabling +* lockdep for the rest of the kernel. +*/ +lockdep_off (); + +pr_err ("==================================================================\n" ); +/* Print report header. */ +switch (type) { +case KFENCE_ERROR_OOB : +pr_err ("BUG: KFENCE: out-of-bounds in %pS\n\n" , (void *)stack_entries[skipnr]); +pr_err ("Out-of-bounds access at 0x" PTR_FMT " (%s of kfence-#%zd):\n" , + (void *)address, address < meta->addr ?"left" :"right" , object_index); +break ; +case KFENCE_ERROR_UAF : +pr_err ("BUG: KFENCE: use-after-free in %pS\n\n" , (void *)stack_entries[skipnr]); +pr_err ("Use-after-free access at 0x" PTR_FMT " (in kfence-#%zd):\n" , + (void *)address, object_index); +break ; +case KFENCE_ERROR_CORRUPTION : +pr_err ("BUG: KFENCE: memory corruption in %pS\n\n" , (void *)stack_entries[skipnr]); +pr_err ("Corrupted memory at 0x" PTR_FMT " " , (void *)address); +print_diff_canary ((u8 *)address,16 ); +pr_cont (" (in kfence-#%zd):\n" , object_index); +break ; +case KFENCE_ERROR_INVALID : +pr_err ("BUG: KFENCE: invalid access in %pS\n\n" , (void *)stack_entries[skipnr]); +pr_err ("Invalid access at 0x" PTR_FMT ":\n" , (void *)address); +break ; +case KFENCE_ERROR_INVALID_FREE : +pr_err ("BUG: KFENCE: invalid free in %pS\n\n" , (void *)stack_entries[skipnr]); +pr_err ("Invalid free of 0x" PTR_FMT " (in kfence-#%zd):\n" , (void *)address, + object_index); +break ; + } + +/* Print stack trace and object info. */ +stack_trace_print (stack_entries + skipnr, num_stack_entries - skipnr,0 ); + +if (meta) { +pr_err ("\n" ); +kfence_print_object (NULL , meta); + } + +/* Print report footer. */ +pr_err ("\n" ); +dump_stack_print_info (KERN_DEFAULT ); +pr_err ("==================================================================\n" ); + +lockdep_on (); + +if (panic_on_warn) +panic ("panic_on_warn set ...\n" ); + +/* We encountered a memory unsafety error, taint the kernel! */ +add_taint (TAINT_WARN ,LOCKDEP_STILL_OK ); +} --2.28.0.681.g6f77f65b4e-goog
Marco Elver
maintainerRe: [PATCH v3 01/10] mm: add Kernel Electric-Fence infrastructure
On Tue, Sep 29, 2020 at 02:42PM +0200, Andrey Konovalov wrote:
[...]
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ */ + index = (addr - (unsigned long)__kfence_pool) / (PAGE_SIZE * 2) - 1;Why do we subtract 1 here? We do have the metadata entry reserved for something?
Above the declaration of __kfence_pool it says:
* We allocate an even number of pages, as it simplifies calculations to map
* address to metadata indices; effectively, the very first page serves as an
* extended guard page, but otherwise has no special purpose.
Hopefully that clarifies the - 1 here.
[...]
Quoted 8 lines
mm/kfence/kfence.h line 83
Quoted 5 lines
mm/kfence/kfence.h new file +5
-0+/* Allocation and free stack information. */ +int num_alloc_stack; +int num_free_stack; +unsigned long alloc_stack[KFENCE_STACK_DEPTH ]; +unsigned long free_stack[KFENCE_STACK_DEPTH ];It was a concious decision to not use stackdepot, right? Perhaps it makes sense to document the reason somewhere.
Yes; we want to avoid the dynamic allocations that stackdepot does.
[...]
Thanks,
-- Marco
Re: [PATCH v3 01/10] mm: add Kernel Electric-Fence infrastructure
Quoted 8 lines · Marco Elver
On Tue, Sep 29, 2020 at 02:42PM +0200, Andrey Konovalov wrote:
[...]Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ */ + index = (addr - (unsigned long)__kfence_pool) / (PAGE_SIZE * 2) - 1;Why do we subtract 1 here? We do have the metadata entry reserved for something?
Quoted 7 lines
Above the declaration of
__kfence_poolit says:* We allocate an even number of pages, as it simplifies calculations to map
* address to metadata indices; effectively, the very first page serves as an
* extended guard page, but otherwise has no special purpose.Hopefully that clarifies the
- 1here.
So there are two guard pages at the beginning and only then a page that holds an object?
Marco Elver
maintainerRe: [PATCH v3 01/10] mm: add Kernel Electric-Fence infrastructure
Quoted 11 lines · Andrey Konovalov
Quoted 10 lines · Marco Elver
On Tue, Sep 29, 2020 at 02:42PM +0200, Andrey Konovalov wrote:
[...]Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ */ + index = (addr - (unsigned long)__kfence_pool) / (PAGE_SIZE * 2) - 1;Why do we subtract 1 here? We do have the metadata entry reserved for something?
Above the declaration of
__kfence_poolit says:
Quoted 8 lines
Quoted 5 lines
* We allocate an even number of pages, as it simplifies calculations to map
* address to metadata indices; effectively, the very first page serves as an
* extended guard page, but otherwise has no special purpose.Hopefully that clarifies the
- 1here.So there are two guard pages at the beginning and only then a page that holds an object?
Yes, correct.
Re: [PATCH v3 01/10] mm: add Kernel Electric-Fence infrastructure
Quoted 17 lines · Marco Elver
Quoted 15 lines · Andrey Konovalov
Quoted 14 lines · Marco Elver
On Tue, Sep 29, 2020 at 02:42PM +0200, Andrey Konovalov wrote:
[...]Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ */ + index = (addr - (unsigned long)__kfence_pool) / (PAGE_SIZE * 2) - 1;Why do we subtract 1 here? We do have the metadata entry reserved for something?
Above the declaration of
__kfence_poolit says:* We allocate an even number of pages, as it simplifies calculations to map
* address to metadata indices; effectively, the very first page serves as an
* extended guard page, but otherwise has no special purpose.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Hopefully that clarifies the
- 1here.So there are two guard pages at the beginning and only then a page that holds an object?
Yes, correct.
OK, I see. This isn't directly clear from the comment though, at least for me :)
Mark Rutland
reviewerRe: [PATCH v3 01/10] mm: add Kernel Electric-Fence infrastructure
In reply to Marco ElverQuoted 31 lines · Marco Elver
From: Alexander Potapenko <[email protected]>
This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors.
KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.
KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error. To detect out-of-bounds writes to memory within the object's page itself, KFENCE also uses pattern-based redzones. The following figure illustrates the page layout:
---+-----------+-----------+-----------+-----------+-----------+---
| xxxxxxxxx | O : | xxxxxxxxx | : O | xxxxxxxxx | | xxxxxxxxx | B : | xxxxxxxxx | : B | xxxxxxxxx | | x GUARD x | J : RED- | x GUARD x | RED- : J | x GUARD x | | xxxxxxxxx | E : ZONE | xxxxxxxxx | ZONE : E | xxxxxxxxx | | xxxxxxxxx | C : | xxxxxxxxx | : C | xxxxxxxxx | | xxxxxxxxx | T : | xxxxxxxxx | : T | xxxxxxxxx |
Quoted 7 lines
---+-----------+-----------+-----------+-----------+-----------+---
Guarded allocations are set up based on a sample interval (can be set via kfence.sample_interval). After expiration of the sample interval, a guarded allocation from the KFENCE object pool is returned to the main allocator (SLAB or SLUB). At this point, the timer is reset, and the next allocation is set up after the expiration of the interval.
From other sub-threads it sounds like these addresses are not part of the linear/direct map. Having kmalloc return addresses outside of the linear map is going to break anything that relies on virt<->phys conversions, and is liable to make DMA corrupt memory. There were problems of that sort with VMAP_STACK, and this is why kvmalloc() is separate from kmalloc().
Have you tested with CONFIG_DEBUG_VIRTUAL? I'd expect that to scream.
I strongly suspect this isn't going to be safe unless you always use an in-place carevout from the linear map (which could be the linear alias of a static carevout).
[...]
include/linux/kfence.h line 89
Quoted 5 lines
include/linux/kfence.h new file +5
-0+static __always_inline void *kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags) +{ +return static_branch_unlikely (&kfence_allocation_key) ?__kfence_alloc (s, size, flags) : +NULL ; +}
Minor (unrelated) nit, but this would be easier to read as:
static __always_inline void *kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags)
{
if (static_branch_unlikely(&kfence_allocation_key))
return __kfence_alloc(s, size, flags);
return NULL;
}Thanks,
Mark.
Marco Elver
maintainerRe: [PATCH v3 01/10] mm: add Kernel Electric-Fence infrastructure
On Tue, 29 Sep 2020 at 16:24, Mark Rutland <[email protected]> wrote:
[...]
Quoted 8 lines
From other sub-threads it sounds like these addresses are not part of the linear/direct map. Having kmalloc return addresses outside of the linear map is going to break anything that relies on virt<->phys conversions, and is liable to make DMA corrupt memory. There were problems of that sort with
VMAP_STACK, and this is whykvmalloc()is separate fromkmalloc().
Quoted 5 lines
Have you tested with
CONFIG_DEBUG_VIRTUAL? I'd expect that to scream.I strongly suspect this isn't going to be safe unless you always use an in-place carevout from the linear map (which could be the linear alias of a static carevout).
That's an excellent point, thank you! Indeed, on arm64, a version with naive static-pool screams with CONFIG_DEBUG_VIRTUAL.
We'll try to put together an arm64 version using a carveout as you suggest.
Quoted 10 lines
[...]
include/linux/kfence.h line 89
Quoted 5 lines
include/linux/kfence.h new file +5
-0+static __always_inline void *kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags) +{ +return static_branch_unlikely (&kfence_allocation_key) ?__kfence_alloc (s, size, flags) : +NULL ; +}Minor (unrelated) nit, but this would be easier to read as:
Quoted 6 lines
static
__always_inlinevoid *kfence_alloc(structkmem_cache*s,size_tsize,gfp_tflags){ if (static_branch_unlikely(&kfence_allocation_key)) return __kfence_alloc(s, size, flags); return NULL; }
Will fix for v5.
Thanks,
-- Marco
Mark Rutland
reviewerRe: [PATCH v3 01/10] mm: add Kernel Electric-Fence infrastructure
Quoted 12 lines · Marco Elver
On Tue, 29 Sep 2020 at 16:24, Mark Rutland <[email protected]> wrote:
[...]Quoted 10 lines
From other sub-threads it sounds like these addresses are not part of the linear/direct map. Having kmalloc return addresses outside of the linear map is going to break anything that relies on virt<->phys conversions, and is liable to make DMA corrupt memory. There were problems of that sort with
VMAP_STACK, and this is whykvmalloc()is separate fromkmalloc().Have you tested with
CONFIG_DEBUG_VIRTUAL? I'd expect that to scream.
Quoted 8 lines
Quoted 3 lines
I strongly suspect this isn't going to be safe unless you always use an in-place carevout from the linear map (which could be the linear alias of a static carevout).
That's an excellent point, thank you! Indeed, on arm64, a version with naive static-pool screams with
CONFIG_DEBUG_VIRTUAL.We'll try to put together an arm64 version using a carveout as you suggest.
Great, thanks!
Just to be clear, the concerns for DMA and virt<->phys conversions also apply to x86 (the x86 virt<->phys conversion behaviour is more forgiving in the common case, but still has cases that can go wrong).
Other than the code to initialize the page tables for the careveout, I think the carevout code can be geenric.
Thanks,
Mark.
Alexander Potapenko
maintainerRe: [PATCH v3 01/10] mm: add Kernel Electric-Fence infrastructure
Quoted 21 lines · Mark Rutland
Quoted 19 lines · Marco Elver
On Tue, 29 Sep 2020 at 16:24, Mark Rutland <[email protected]> wrote:
[...]Quoted 8 lines
From other sub-threads it sounds like these addresses are not part of the linear/direct map. Having kmalloc return addresses outside of the linear map is going to break anything that relies on virt<->phys conversions, and is liable to make DMA corrupt memory. There were problems of that sort with
VMAP_STACK, and this is whykvmalloc()is separate fromkmalloc().Quoted 5 lines
Have you tested with
CONFIG_DEBUG_VIRTUAL? I'd expect that to scream.I strongly suspect this isn't going to be safe unless you always use an in-place carevout from the linear map (which could be the linear alias of a static carevout).
That's an excellent point, thank you! Indeed, on arm64, a version with naive static-pool screams with
CONFIG_DEBUG_VIRTUAL.
Quoted 7 lines
Quoted 1 line
We'll try to put together an arm64 version using a carveout as you suggest.
Great, thanks!
Just to be clear, the concerns for DMA and virt<->phys conversions also apply to x86 (the x86 virt<->phys conversion behaviour is more forgiving in the common case, but still has cases that can go wrong).
To clarify, shouldn't kmalloc/kmem_cache allocations used with DMA be allocated with explicit GFP_DMA?
If so, how practical would it be to just skip such allocations in KFENCE allocator?
Re: [PATCH v3 01/10] mm: add Kernel Electric-Fence infrastructure
Quoted 27 lines · Alexander Potapenko
Quoted 25 lines · Mark Rutland
Quoted 12 lines · Marco Elver
On Tue, 29 Sep 2020 at 16:24, Mark Rutland <[email protected]> wrote:
[...]Quoted 10 lines
From other sub-threads it sounds like these addresses are not part of the linear/direct map. Having kmalloc return addresses outside of the linear map is going to break anything that relies on virt<->phys conversions, and is liable to make DMA corrupt memory. There were problems of that sort with
VMAP_STACK, and this is whykvmalloc()is separate fromkmalloc().Have you tested with
CONFIG_DEBUG_VIRTUAL? I'd expect that to scream.Quoted 8 lines
Quoted 3 lines
I strongly suspect this isn't going to be safe unless you always use an in-place carevout from the linear map (which could be the linear alias of a static carevout).
That's an excellent point, thank you! Indeed, on arm64, a version with naive static-pool screams with
CONFIG_DEBUG_VIRTUAL.We'll try to put together an arm64 version using a carveout as you suggest.
Great, thanks!
Quoted 8 lines
Quoted 3 lines
Just to be clear, the concerns for DMA and virt<->phys conversions also apply to x86 (the x86 virt<->phys conversion behaviour is more forgiving in the common case, but still has cases that can go wrong).
To clarify, shouldn't kmalloc/kmem_cache allocations used with DMA be allocated with explicit
GFP_DMA?
If so, how practical would it be to just skip such allocations in KFENCE allocator?
AFAIK GFP_DMA doesn't really mean "I will use this allocation for DMA"; it means "I will use this allocation for DMA using some ancient hardware (e.g. stuff on the ISA bus?) that only supports 16-bit physical addresses (or maybe different limits on other architectures)".
There's also GFP_DMA32, which means the same thing, except with 32-bit physical addresses.
You can see in e.g. __dma_direct_alloc_pages() that the GFP_DMA32 and GFP_DMA flags are only used if the hardware can't address the full physical address space supported by the CPU.
Alexander Potapenko
maintainerRe: [PATCH v3 01/10] mm: add Kernel Electric-Fence infrastructure
In reply to Mark RutlandQuoted 35 lines · Mark Rutland
Quoted 33 lines · Marco Elver
From: Alexander Potapenko <[email protected]>
This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors.
KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.
KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error. To detect out-of-bounds writes to memory within the object's page itself, KFENCE also uses pattern-based redzones. The following figure illustrates the page layout:
---+-----------+-----------+-----------+-----------+-----------+---
| xxxxxxxxx | O : | xxxxxxxxx | : O | xxxxxxxxx | | xxxxxxxxx | B : | xxxxxxxxx | : B | xxxxxxxxx | | x GUARD x | J : RED- | x GUARD x | RED- : J | x GUARD x | | xxxxxxxxx | E : ZONE | xxxxxxxxx | ZONE : E | xxxxxxxxx | | xxxxxxxxx | C : | xxxxxxxxx | : C | xxxxxxxxx | | xxxxxxxxx | T : | xxxxxxxxx | : T | xxxxxxxxx |---+-----------+-----------+-----------+-----------+-----------+---
Quoted 8 lines
Quoted 5 lines
Guarded allocations are set up based on a sample interval (can be set via kfence.sample_interval). After expiration of the sample interval, a guarded allocation from the KFENCE object pool is returned to the main allocator (SLAB or SLUB). At this point, the timer is reset, and the next allocation is set up after the expiration of the interval.
From other sub-threads it sounds like these addresses are not part of the linear/direct map.
For x86 these addresses belong to .bss, i.e. "kernel text mapping" section, isn't that the linear map?
I also don't see lm_alias being used much outside arm64 code.
Quoted 7 lines
Having kmalloc return addresses outside of the
linear map is going to break anything that relies on virt<->phys conversions, and is liable to make DMA corrupt memory. There were problems of that sort withVMAP_STACK, and this is whykvmalloc()is separate fromkmalloc().Have you tested with
CONFIG_DEBUG_VIRTUAL? I'd expect that to scream.
Just checked - it doesn't scream on x86.
Quoted 23 lines
I strongly suspect this isn't going to be safe unless you always use an in-place carevout from the linear map (which could be the linear alias of a static carevout).
[...]
include/linux/kfence.h line 89
Quoted 5 lines
include/linux/kfence.h new file +5
-0+static __always_inline void *kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags) +{ +return static_branch_unlikely (&kfence_allocation_key) ?__kfence_alloc (s, size, flags) : +NULL ; +}Minor (unrelated) nit, but this would be easier to read as:
static
__always_inlinevoid *kfence_alloc(structkmem_cache*s,size_tsize,gfp_tflags){ if (static_branch_unlikely(&kfence_allocation_key)) return __kfence_alloc(s, size, flags); return NULL; }Thanks,
Mark.
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Mark Rutland
reviewerRe: [PATCH v3 01/10] mm: add Kernel Electric-Fence infrastructure
Quoted 42 lines · Alexander Potapenko
Quoted 41 lines · Mark Rutland
Quoted 38 lines · Marco Elver
From: Alexander Potapenko <[email protected]>
This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors.
KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.
KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error. To detect out-of-bounds writes to memory within the object's page itself, KFENCE also uses pattern-based redzones. The following figure illustrates the page layout:
---+-----------+-----------+-----------+-----------+-----------+---
| xxxxxxxxx | O : | xxxxxxxxx | : O | xxxxxxxxx | | xxxxxxxxx | B : | xxxxxxxxx | : B | xxxxxxxxx | | x GUARD x | J : RED- | x GUARD x | RED- : J | x GUARD x | | xxxxxxxxx | E : ZONE | xxxxxxxxx | ZONE : E | xxxxxxxxx | | xxxxxxxxx | C : | xxxxxxxxx | : C | xxxxxxxxx | | xxxxxxxxx | T : | xxxxxxxxx | : T | xxxxxxxxx |---+-----------+-----------+-----------+-----------+-----------+---
Guarded allocations are set up based on a sample interval (can be set via kfence.sample_interval). After expiration of the sample interval, a guarded allocation from the KFENCE object pool is returned to the main allocator (SLAB or SLUB). At this point, the timer is reset, and the next allocation is set up after the expiration of the interval.
Quoted 4 lines
Quoted 2 lines
From other sub-threads it sounds like these addresses are not part of the linear/direct map.
For x86 these addresses belong to .bss, i.e. "kernel text mapping" section, isn't that the linear map?
No; the "linear map" is the "direct mapping" on x86, and the "image" or "kernel text mapping" is a distinct VA region. The image mapping aliases (i.e. uses the same physical pages as) a portion of the linear map, and every page in the linear map has a struct page.
Fon the x86_64 ivirtual memory layout, see:
https://www.kernel.org/doc/html/latest/x86/x86_64/mm.html
Originally, the kernel image lived in the linear map, but it was split out into a distinct VA range (among other things) to permit KASLR. When that split was made, the x86 virt_to_*() helpers were updated to detect when they were passed a kernel image address, and automatically fix that up as-if they'd been handed the linear map alias of that address.
For going one-way from virt->{phys,page} that works ok, but it doesn't survive the round-trip, and introduces redundant work into each virt_to_*() call.
As it was largely arch code that was using image addresses, we didn't bother with the fixup on arm64, as we preferred the stronger warning. At the time I was also under the impression that on x86 they wanted to get rid of the automatic fixup, but that doesn't seem to have happened.
Thanks,
Mark.
Marco Elver
maintainerPATCH v3 2/10 x86, kfence: enable KFENCE for x86
In reply to Marco ElverFrom: Alexander Potapenko <[email protected]>
Add architecture specific implementation details for KFENCE and enable KFENCE for the x86 architecture. In particular, this implements the required interface in <asm/kfence.h> for setting up the pool and providing helper functions for protecting and unprotecting pages.
For x86, we need to ensure that the pool uses 4K pages, which is done using the set_memory_4k() helper function.
--- arch/x86/Kconfig | 2 ++ arch/x86/include/asm/kfence.h | 60 +++++++++++++++++++++++++++++++++++ arch/x86/mm/fault.c | 4 +++ 3 files changed, 66 insertions(+) create mode 100644 arch/x86/include/asm/kfence.h
arch/x86/Kconfig +2 -0
@@ -144,6 +144,8 @@ config X86 select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 select HAVE_ARCH_KASAN_VMALLOC if X86_64 + select HAVE_ARCH_KFENCE + select HAVE_ARCH_KFENCE_STATIC_POOL select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS if MMU select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT
arch/x86/include/asm/kfence.h new file +60 -0
60 lines not on this page · raw message
arch/x86/mm/fault.c +4 -0
@@ -9,6 +9,7 @@#include <linux/kdebug.h> /* oops_begin/end, ... */ #include <linux/extable.h> /* search_exception_tables */ #include <linux/memblock.h> /* max_low_pfn */ +#include <linux/kfence.h> /* kfence_handle_page_fault */ #include <linux/kprobes.h> /* NOKPROBE_SYMBOL, ... */ #include <linux/mmiotrace.h> /* kmmio_handler, ... */ #include <linux/perf_event.h> /* perf_sw_event */
@@ -701,6 +702,9 @@ no_context(struct pt_regs *regs, unsigned long error_code, }#endif +if (kfence_handle_page_fault (address)) +return ; +/* * 32-bit: *
Signature · 1 line
2.28.0.681.g6f77f65b4e-goog
Marco Elver
maintainerPATCH v3 3/10 arm64, kfence: enable KFENCE for ARM64
In reply to Marco ElverAdd architecture specific implementation details for KFENCE and enable KFENCE for the arm64 architecture. In particular, this implements the required interface in <asm/kfence.h>. Currently, the arm64 version does not yet use a statically allocated memory pool, at the cost of a pointer load for each is_kfence_address().
---
For ARM64, we would like to solicit feedback on what the best option is to obtain a constant address for __kfence_pool. One option is to declare a memory range in the memory layout to be dedicated to KFENCE (like is done for KASAN), however, it is unclear if this is the best available option. We would like to avoid touching the memory layout.
--- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/kfence.h | 39 +++++++++++++++++++++++++++++++++ arch/arm64/mm/fault.c | 4 ++++ 3 files changed, 44 insertions(+) create mode 100644 arch/arm64/include/asm/kfence.h
arch/arm64/Kconfig +1 -0
@@ -132,6 +132,7 @@ config ARM64 select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN + select HAVE_ARCH_KFENCE if (!ARM64_16K_PAGES && !ARM64_64K_PAGES) select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT
arch/arm64/include/asm/kfence.h new file +39 -0
@@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_KFENCE_H +#define __ASM_KFENCE_H + +#include <linux/kfence.h> +#include <linux/log2.h> +#include <linux/mm.h> + +#include <asm/cacheflush.h> + +#define KFENCE_SKIP_ARCH_FAULT_HANDLER "el1_sync" + +/* +* FIXME: Support HAVE_ARCH_KFENCE_STATIC_POOL: Use the statically allocated +* __kfence_pool, to avoid the extra pointer load for is_kfence_address(). By +* default, however, we do not have struct pages for static allocations. +*/ + +static inline bool arch_kfence_initialize_pool (void ) +{ +const unsigned int num_pages =ilog2 (roundup_pow_of_two (KFENCE_POOL_SIZE /PAGE_SIZE )); +struct page *pages =alloc_pages (GFP_KERNEL , num_pages); + +if (!pages) +return false; + + __kfence_pool =page_address (pages); +return true; +} + +static inline bool kfence_protect_page (unsigned long addr,bool protect) +{ +set_memory_valid (addr,1 , !protect); + +return true; +} + +#endif /* __ASM_KFENCE_H */
arch/arm64/mm/fault.c +4 -0
@@ -10,6 +10,7 @@#include <linux/acpi.h> #include <linux/bitfield.h> #include <linux/extable.h> +#include <linux/kfence.h> #include <linux/signal.h> #include <linux/mm.h> #include <linux/hardirq.h>
@@ -310,6 +311,9 @@ static void __do_kernel_fault(unsigned long addr, unsigned int esr, "Ignoring spurious kernel translation fault at virtual address %016lx\n", addr)) return; +if (kfence_handle_page_fault (addr)) +return ; +if (is_el1_permission_fault (addr, esr, regs)) {if (esr & ESR_ELx_WNR) msg ="write to read-only memory" ;
Signature · 1 line
2.28.0.681.g6f77f65b4e-goog
Will Deacon
maintainerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
Quoted 10 lines · Marco Elver
Add architecture specific implementation details for KFENCE and enable KFENCE for the arm64 architecture. In particular, this implements the required interface in <
asm/kfence.h>. Currently, the arm64 version does not yet use a statically allocated memory pool, at the cost of a pointer load for eachis_kfence_address().Reviewed-by Dmitry Vyukov <[email protected]>Co-developed-by Alexander Potapenko <[email protected]>Signed-off-by Alexander Potapenko <[email protected]>Signed-off-by Marco Elver <[email protected]>
Quoted 6 lines
---
For ARM64, we would like to solicit feedback on what the best option is to obtain a constant address for__kfence_pool. One option is to declare a memory range in the memory layout to be dedicated to KFENCE (like is done for KASAN), however, it is unclear if this is the best available option. We would like to avoid touching the memory layout.
Sorry for the delay on this.
Given that the pool is relatively small (i.e. when compared with our virtual address space), dedicating an area of virtual space sounds like it makes the most sense here. How early do you need it to be available?
An alternative approach would be to patch in the address at runtime, with something like a static key to swizzle off the direct __kfence_pool load once we're up and running.
Will
Alexander Potapenko
maintainerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
Quoted 19 lines · Will Deacon
Quoted 10 lines · Marco Elver
Add architecture specific implementation details for KFENCE and enable KFENCE for the arm64 architecture. In particular, this implements the required interface in <
asm/kfence.h>. Currently, the arm64 version does not yet use a statically allocated memory pool, at the cost of a pointer load for eachis_kfence_address().Reviewed-by Dmitry Vyukov <[email protected]>Co-developed-by Alexander Potapenko <[email protected]>Signed-off-by Alexander Potapenko <[email protected]>Signed-off-by Marco Elver <[email protected]>Quoted 6 lines
---
For ARM64, we would like to solicit feedback on what the best option is to obtain a constant address for__kfence_pool. One option is to declare a memory range in the memory layout to be dedicated to KFENCE (like is done for KASAN), however, it is unclear if this is the best available option. We would like to avoid touching the memory layout.
Quoted 1 line
Sorry for the delay on this.
NP, thanks for looking!
Quoted 3 lines
Given that the pool is relatively small (i.e. when compared with our virtual address space), dedicating an area of virtual space sounds like it makes the most sense here. How early do you need it to be available?
Yes, having a dedicated address sounds good.
We're inserting kfence_init() into start_kernel() after timekeeping_init(). So way after mm_init(), if that matters.
Quoted 3 lines
An alternative approach would be to patch in the address at runtime, with something like a static key to swizzle off the direct
__kfence_poolload once we're up and running.
IIUC there's no such thing as address patching in the kernel at the moment, at least static keys work differently?
I am not sure how much we need to randomize this address range (we don't on x86 anyway).
Quoted 6 lines
Will
--
You received this message because you are subscribed to the Google Groups "kasan-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/20200921143059.GO2139%40willie-the-truck.
--
Alexander Potapenko
Software Engineer
Google Germany GmbH
Erika-Mann-Straße, 33
80636 München
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Alexander Potapenko
maintainerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
Quoted 25 lines · Alexander Potapenko
Quoted 19 lines · Will Deacon
Quoted 10 lines · Marco Elver
Add architecture specific implementation details for KFENCE and enable KFENCE for the arm64 architecture. In particular, this implements the required interface in <
asm/kfence.h>. Currently, the arm64 version does not yet use a statically allocated memory pool, at the cost of a pointer load for eachis_kfence_address().Reviewed-by Dmitry Vyukov <[email protected]>Co-developed-by Alexander Potapenko <[email protected]>Signed-off-by Alexander Potapenko <[email protected]>Signed-off-by Marco Elver <[email protected]>Quoted 6 lines
---
For ARM64, we would like to solicit feedback on what the best option is to obtain a constant address for__kfence_pool. One option is to declare a memory range in the memory layout to be dedicated to KFENCE (like is done for KASAN), however, it is unclear if this is the best available option. We would like to avoid touching the memory layout.Quoted 1 line
Sorry for the delay on this.
NP, thanks for looking!
Quoted 7 lines
Quoted 3 lines
Given that the pool is relatively small (i.e. when compared with our virtual address space), dedicating an area of virtual space sounds like it makes the most sense here. How early do you need it to be available?
Yes, having a dedicated address sounds good.
We're insertingkfence_init()intostart_kernel()aftertimekeeping_init(). So way aftermm_init(), if that matters.
The question is though, how big should that dedicated area be? Right now KFENCE_NUM_OBJECTS can be up to 16383 (which makes the pool size 64MB), but this number actually comes from the limitation on static objects, so we might want to increase that number on arm64.
Will Deacon
maintainerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
Quoted 30 lines · Alexander Potapenko
Quoted 29 lines · Alexander Potapenko
Quoted 19 lines · Will Deacon
Quoted 10 lines · Marco Elver
Add architecture specific implementation details for KFENCE and enable KFENCE for the arm64 architecture. In particular, this implements the required interface in <
asm/kfence.h>. Currently, the arm64 version does not yet use a statically allocated memory pool, at the cost of a pointer load for eachis_kfence_address().Reviewed-by Dmitry Vyukov <[email protected]>Co-developed-by Alexander Potapenko <[email protected]>Signed-off-by Alexander Potapenko <[email protected]>Signed-off-by Marco Elver <[email protected]>Quoted 6 lines
---
For ARM64, we would like to solicit feedback on what the best option is to obtain a constant address for__kfence_pool. One option is to declare a memory range in the memory layout to be dedicated to KFENCE (like is done for KASAN), however, it is unclear if this is the best available option. We would like to avoid touching the memory layout.Quoted 1 line
Sorry for the delay on this.
NP, thanks for looking!
Quoted 3 lines
Given that the pool is relatively small (i.e. when compared with our virtual address space), dedicating an area of virtual space sounds like it makes the most sense here. How early do you need it to be available?
Quoted 8 lines
Quoted 3 lines
Yes, having a dedicated address sounds good.
We're insertingkfence_init()intostart_kernel()aftertimekeeping_init(). So way aftermm_init(), if that matters.The question is though, how big should that dedicated area be? Right now
KFENCE_NUM_OBJECTScan be up to 16383 (which makes the pool size 64MB), but this number actually comes from the limitation on static objects, so we might want to increase that number on arm64.
What happens on x86 and why would we do something different?
Will
Marco Elver
maintainerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
On Mon, 21 Sep 2020 at 19:44, Will Deacon <[email protected]> wrote:
[...]
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 7 lines
Quoted 5 lines
For ARM64, we would like to solicit feedback on what the best option is to obtain a constant address for
__kfence_pool. One option is to declare a memory range in the memory layout to be dedicated to KFENCE (like is done for KASAN), however, it is unclear if this is the best available option. We would like to avoid touching the memory layout.Sorry for the delay on this.
NP, thanks for looking!
Quoted 3 lines
Given that the pool is relatively small (i.e. when compared with our virtual address space), dedicating an area of virtual space sounds like it makes the most sense here. How early do you need it to be available?
Yes, having a dedicated address sounds good.
We're insertingkfence_init()intostart_kernel()aftertimekeeping_init(). So way aftermm_init(), if that matters.
Quoted 6 lines
Quoted 4 lines
The question is though, how big should that dedicated area be? Right now
KFENCE_NUM_OBJECTScan be up to 16383 (which makes the pool size 64MB), but this number actually comes from the limitation on static objects, so we might want to increase that number on arm64.What happens on x86 and why would we do something different?
On x86 we just do char __kfence_pool[KFENCE_POOL_SIZE] ...; to statically allocate the pool. On arm64 this doesn't seem to work because static memory doesn't have struct pages?
Thanks,
-- Marco
Mark Rutland
reviewerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
Quoted 25 lines · Marco Elver
On Mon, 21 Sep 2020 at 19:44, Will Deacon <[email protected]> wrote:
[...]Quoted 23 lines
Quoted 22 lines
Quoted 10 lines
Quoted 7 lines
Quoted 5 lines
For ARM64, we would like to solicit feedback on what the best option is to obtain a constant address for
__kfence_pool. One option is to declare a memory range in the memory layout to be dedicated to KFENCE (like is done for KASAN), however, it is unclear if this is the best available option. We would like to avoid touching the memory layout.Sorry for the delay on this.
NP, thanks for looking!
Quoted 7 lines
Quoted 3 lines
Given that the pool is relatively small (i.e. when compared with our virtual address space), dedicating an area of virtual space sounds like it makes the most sense here. How early do you need it to be available?
Yes, having a dedicated address sounds good.
We're insertingkfence_init()intostart_kernel()aftertimekeeping_init(). So way aftermm_init(), if that matters.The question is though, how big should that dedicated area be? Right now
KFENCE_NUM_OBJECTScan be up to 16383 (which makes the pool size 64MB), but this number actually comes from the limitation on static objects, so we might want to increase that number on arm64.
Quoted 5 lines
Quoted 1 line
What happens on x86 and why would we do something different?
On x86 we just do
char __kfence_pool[KFENCE_POOL_SIZE] ...;to statically allocate the pool. On arm64 this doesn't seem to work because static memory doesn't have struct pages?
Are you using virt_to_page() directly on that statically-allocated __kfence_pool? If so you'll need to use lm_alias() if so, as is done in mm/kasan/init.c.
Anything statically allocated is part of the kernel image address range rather than the linear/direct map, and doesn't have a valid virt addr, but its linear map alias does.
If you enable CONFIG_DEBUG_VIRTUAL you should get warnings if missing lm_alias() calls.
Thanks,
Mark.
Alexander Potapenko
maintainerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
Quoted 8 lines
Quoted 3 lines
On x86 we just do
char __kfence_pool[KFENCE_POOL_SIZE] ...;to statically allocate the pool. On arm64 this doesn't seem to work because static memory doesn't have struct pages?Are you using
virt_to_page()directly on that statically-allocated__kfence_pool? If so you'll need to uselm_alias()if so, as is done inmm/kasan/init.c.
Quoted 6 lines
Anything statically allocated is part of the kernel image address range rather than the linear/direct map, and doesn't have a valid virt addr, but its linear map alias does.
If you enable
CONFIG_DEBUG_VIRTUALyou should get warnings if missinglm_alias()calls.
I just checked that on x86 CONFIG_DEBUG_VIRTUAL prints no warnings on our tests. virt_addr_valid() also returns true for addresses belonging to __kfence_pool declared in BSS.
Could this be related to x86 mapping the kernel twice?
Alexander Potapenko
maintainerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
In reply to Will DeaconWill,
Quoted 3 lines
Given that the pool is relatively small (i.e. when compared with our virtual address space), dedicating an area of virtual space sounds like it makes the most sense here. How early do you need it to be available?
How do we assign struct pages to a fixed virtual space area (I'm currently experimenting with 0xffff7f0000000000-0xffff7f0000200000)? Looks like filling page table entries (similarly to what's being done in arch/arm64/mm/kasan_init.c) is not enough.
I thought maybe vmemmap_populate() would do the job, but it didn't (virt_to_pfn() still returns invalid PFNs).
Mark Rutland
reviewerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
Quoted 8 lines · Alexander Potapenko
Will,
Quoted 3 lines
Given that the pool is relatively small (i.e. when compared with our virtual address space), dedicating an area of virtual space sounds like it makes the most sense here. How early do you need it to be available?
How do we assign struct pages to a fixed virtual space area (I'm currently experimenting with 0xffff7f0000000000-0xffff7f0000200000)?
You don't.
There should be a struct page for each of the /physical/ pages, and these can be found:
* via the physical address, using phyts_to_page() or pfn_to_page()
* via the linear/direct map, using virt_to_page()
* via the vmalloc page tables using vmalloc_to_page()
If you need virt_to_page() to work, the address has to be part of the linear/direct map.
If you need to find the struct page for something that's part of the kernel image you can use virt_to_page(lm_alias(x)).
Quoted 4 lines
Looks like filling page table entries (similarly to what's being done in
arch/arm64/mm/kasan_init.c) is not enough.
I thought maybevmemmap_populate()would do the job, but it didn't (virt_to_pfn()still returns invalid PFNs).
As above, I think lm_alias() will solve the problem here. Please see that and CONFIG_DEBUG_VIRTUAL.
Thanks,
Mark.
Alexander Potapenko
maintainerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
Mark,
Quoted 6 lines
If you need
virt_to_page()to work, the address has to be part of the linear/direct map.If you need to find the struct page for something that's part of the kernel image you can use virt_to_page(lm_alias(x)).
Quoted 7 lines
Quoted 4 lines
Looks like filling page table entries (similarly to what's being done in
arch/arm64/mm/kasan_init.c) is not enough.
I thought maybevmemmap_populate()would do the job, but it didn't (virt_to_pfn()still returns invalid PFNs).As above, I think
lm_alias()will solve the problem here. Please see that andCONFIG_DEBUG_VIRTUAL.
The approach you suggest works to some extent, but there are some caveats.
To reiterate, we are trying to allocate the pool (2Mb by default, but users may want a bigger one, up to, say, 64 Mb) in a way that:
(1) The underlying page tables support 4K granularity.
(2) is_kfence_address() (checks that __kfence_pool <= addr <= __kfence_pool + KFENCE_POOL_SIZE) does not reference memory
(3) For addresses belonging to that pool virt_addr_valid() is true (SLAB/SLUB rely on that)
On x86 we achieve (2) by making our pool a .bss array, so that its address is known statically. Aligning that array on 4K and calling set_memory_4k() ensures that (1) is also fulfilled. (3) seems to just happen automagically without any address translations.
Now, what we are seeing on arm64 is different.
My understanding (please correct me if I'm wrong) is that on arm64 only the memory range at 0xffff000000000000 has valid struct pages, and the size of that range depends on the amount of memory on the system.
This probably means we cannot just pick a fixed address for our pool in that range, unless it is very close to 0xffff000000000000.
If we allocate the pool statically in the way x86 does (assuming we somehow resolve (1)), we can apply lm_alias() to addresses returned by the KFENCE allocator, making kmalloc() always return addresses from the linear map and satisfying (3).
But in that case is_kfence_address() will also need to be updated to compare the addresses to lm_alias(__kfence_pool), and this becomes more heavyweight than just reading the address from memory.
So looks like it's still more preferable to allocate the pool dynamically on ARM64, unless there's a clever trick to allocate a fixed address in the linear map (DTS maybe?)
Thanks,
Alex
Mark Rutland
reviewerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
Quoted 16 lines · Alexander Potapenko
Mark,
Quoted 6 lines
If you need
virt_to_page()to work, the address has to be part of the linear/direct map.If you need to find the struct page for something that's part of the kernel image you can use virt_to_page(lm_alias(x)).
Quoted 7 lines
Quoted 4 lines
Looks like filling page table entries (similarly to what's being done in
arch/arm64/mm/kasan_init.c) is not enough.
I thought maybevmemmap_populate()would do the job, but it didn't (virt_to_pfn()still returns invalid PFNs).As above, I think
lm_alias()will solve the problem here. Please see that andCONFIG_DEBUG_VIRTUAL.
Quoted 7 lines
The approach you suggest works to some extent, but there are some caveats.
To reiterate, we are trying to allocate the pool (2Mb by default, but users may want a bigger one, up to, say, 64 Mb) in a way that:
(1) The underlying page tables support 4K granularity.
(2)is_kfence_address()(checks that__kfence_pool<= addr <=__kfence_pool+KFENCE_POOL_SIZE) does not reference memory
What's the underlying requirement here? Is this a performance concern, codegen/codesize, or something else?
Quoted 2 lines
(3) For addresses belonging to that pool
virt_addr_valid()is true (SLAB/SLUB rely on that)
As I hinted at before, there's a reasonable amount of code which relies on being able to round-trip convert (va->{pa,page}->va) allocations from SLUB, e.g. phys = virt_to_page(addr); ... ; phys = page_to_virt(phys). Usually this is because the phys addr is stored in some HW register, or in-memory structure shared with HW.
I'm fairly certain KFENCE will need to support this in order to be deployable in production, and arm64 is the canary in the coalmine.
I added tests for this back when tag-based KASAN broke this property. See commit:
b92a953cb7f727c4 ("lib/test_kasan.c: add roundtrip tests")
... for which IIUC the kfree_via_phys() test would be broken by KFENCE, even on x86:
| static noinline void __init kfree_via_phys(void)
| {
| char *ptr;
| size_t size = 8;
| phys_addr_t phys;
|
| pr_info("invalid-free false positive (via phys)\n");
| ptr = kmalloc(size, GFP_KERNEL);
| if (!ptr) {
| pr_err("Allocation failed\n");
| return;
| }
|
| phys = virt_to_phys(ptr);
| kfree(phys_to_virt(phys));
| }... since the code will pass the linear map alias of the KFENCE VA into kfree().
To avoid random breakage we either need to:
* Have KFENCE retain this property (which effectively requires allocation VAs to fall within the linear/direct map)
* Decide that round-trips are forbidden, and go modify that code somehow, which was deemed to be impractical in the past
... and I would strongly prefer the former as it's less liable to break any existing code.
Quoted 10 lines
On x86 we achieve (2) by making our pool a .bss array, so that its address is known statically. Aligning that array on 4K and calling
set_memory_4k()ensures that (1) is also fulfilled. (3) seems to just happen automagically without any address translations.Now, what we are seeing on arm64 is different.
My understanding (please correct me if I'm wrong) is that on arm64 only the memory range at0xffff000000000000has valid struct pages, and the size of that range depends on the amount of memory on the system.
The way virt_to_page() works is based on there being a constant (at runtime) offset between a linear map address and the corresponding physical page. That makes it easy to get the PA with a subtraction, then the PFN with a shift, then to index the vmemmap array with that to get the page. The x86 version of virt_to_page() automatically fixes up an image address to its linear map alias internally.
Quoted 2 lines
This probably means we cannot just pick a fixed address for our pool in that range, unless it is very close to
0xffff000000000000.
It would have to be part of the linear map, or we'd have to apply the same fixup as x86 does. But as above, I'm reluctant to do that as it only encourages writing fragile code. The only sensible way to detect that is to disallow virt_to_*() on image addresses, since that's the only time we can distinguish the source.
Quoted 7 lines
If we allocate the pool statically in the way x86 does (assuming we somehow resolve (1)), we can apply
lm_alias()to addresses returned by the KFENCE allocator, makingkmalloc()always return addresses from the linear map and satisfying (3).
But in that caseis_kfence_address()will also need to be updated to compare the addresses tolm_alias(__kfence_pool), and this becomes more heavyweight than just reading the address from memory.
We can calculate the lm_alias(__kfence_pool) at boot time, so it's only a read from memory in the fast-path.
Quoted 3 lines
So looks like it's still more preferable to allocate the pool dynamically on ARM64, unless there's a clever trick to allocate a fixed address in the linear map (DTS maybe?)
I'm not too worried about allocating this dynamically, but:
* The arch code needs to set up the translation tables for this, as we cannot safely change the mapping granularity live.
* As above I'm fairly certain x86 needs to use a carevout from the linear map to function correctly anyhow, so we should follow the same approach for both arm64 and x86. That might be a static carevout that we figure out the aliasing for, or something entirely dynamic.
Thanks,
Mark.
Marco Elver
maintainerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
On Thu, 1 Oct 2020 at 19:58, Mark Rutland <[email protected]> wrote:
[...]
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
If you need
virt_to_page()to work, the address has to be part of the linear/direct map.
[...]
Quoted 3 lines
What's the underlying requirement here? Is this a performance concern, codegen/codesize, or something else?
It used to be performance, since is_kfence_address() is used in the fast path. However, with some further tweaks we just did to is_kfence_address(), our benchmarks show a pointer load can be tolerated.
Quoted 12 lines
Quoted 2 lines
(3) For addresses belonging to that pool
virt_addr_valid()is true (SLAB/SLUB rely on that)As I hinted at before, there's a reasonable amount of code which relies on being able to round-trip convert (va->{pa,page}->va) allocations from SLUB, e.g. phys =
virt_to_page(addr); ... ; phys =page_to_virt(phys). Usually this is because the phys addr is stored in some HW register, or in-memory structure shared with HW.I'm fairly certain KFENCE will need to support this in order to be deployable in production, and arm64 is the canary in the coalmine.
Quoted 7 lines
I added tests for this back when tag-based KASAN broke this property. See commit:
b92a953cb7f727c4("lib/test_kasan.c: add roundtrip tests")... for which IIUC the
kfree_via_phys()test would be broken by KFENCE, even on x86:
Yeah, we're fixing that by also making x86 use a dynamically allocated pool now. The benefits we got from the static pool no longer apply, so the whole dance to make the static pool work right is no longer worth it.
Quoted 17 lines
| static noinline void __init kfree_via_phys(void)
| {
| char *ptr;
| size_t size = 8;
| phys_addr_t phys;
|
| pr_info("invalid-free false positive (via phys)\n");
| ptr = kmalloc(size, GFP_KERNEL);
| if (!ptr) {
| pr_err("Allocation failed\n");
| return;
| }
|
| phys = virt_to_phys(ptr);
| kfree(phys_to_virt(phys));
| }Quoted 7 lines
... since the code will pass the linear map alias of the KFENCE VA into
kfree().To avoid random breakage we either need to:
* Have KFENCE retain this property (which effectively requires allocation VAs to fall within the linear/direct map)
^^ Yes, this is the only realistic option.
Quoted 50 lines
* Decide that round-trips are forbidden, and go modify that code somehow, which was deemed to be impractical in the past
... and I would strongly prefer the former as it's less liable to break any existing code.
Quoted 10 lines
On x86 we achieve (2) by making our pool a .bss array, so that its address is known statically. Aligning that array on 4K and calling
set_memory_4k()ensures that (1) is also fulfilled. (3) seems to just happen automagically without any address translations.Now, what we are seeing on arm64 is different.
My understanding (please correct me if I'm wrong) is that on arm64 only the memory range at0xffff000000000000has valid struct pages, and the size of that range depends on the amount of memory on the system.The way
virt_to_page()works is based on there being a constant (at runtime) offset between a linear map address and the corresponding physical page. That makes it easy to get the PA with a subtraction, then the PFN with a shift, then to index the vmemmap array with that to get the page. The x86 version ofvirt_to_page()automatically fixes up an image address to its linear map alias internally.Quoted 2 lines
This probably means we cannot just pick a fixed address for our pool in that range, unless it is very close to
0xffff000000000000.It would have to be part of the linear map, or we'd have to apply the same fixup as x86 does. But as above, I'm reluctant to do that as it only encourages writing fragile code. The only sensible way to detect that is to disallow virt_to_*() on image addresses, since that's the only time we can distinguish the source.
Quoted 7 lines
If we allocate the pool statically in the way x86 does (assuming we somehow resolve (1)), we can apply
lm_alias()to addresses returned by the KFENCE allocator, makingkmalloc()always return addresses from the linear map and satisfying (3).
But in that caseis_kfence_address()will also need to be updated to compare the addresses tolm_alias(__kfence_pool), and this becomes more heavyweight than just reading the address from memory.We can calculate the
lm_alias(__kfence_pool)at boot time, so it's only a read from memory in the fast-path.Quoted 3 lines
So looks like it's still more preferable to allocate the pool dynamically on ARM64, unless there's a clever trick to allocate a fixed address in the linear map (DTS maybe?)
I'm not too worried about allocating this dynamically, but:
Quoted 7 lines
* The arch code needs to set up the translation tables for this, as we cannot safely change the mapping granularity live.
* As above I'm fairly certain x86 needs to use a carevout from the linear map to function correctly anyhow, so we should follow the same approach for both arm64 and x86. That might be a static carevout that we figure out the aliasing for, or something entirely dynamic.
We're going with dynamically allocating the pool (for both x86 and arm64), since any benefits we used to measure from the static pool are no longer measurable (after removing a branch from is_kfence_address()). It should hopefully simplify a lot of things, given all the caveats that you pointed out.
For arm64, the only thing left then is to fix up the case if the linear map is not forced to page granularity.
Thanks,
-- Marco
Mark Rutland
reviewerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
Quoted 6 lines · Marco Elver
On Thu, 1 Oct 2020 at 19:58, Mark Rutland <[email protected]> wrote:
[...]Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
If you need
virt_to_page()to work, the address has to be part of the linear/direct map.[...]
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
What's the underlying requirement here? Is this a performance concern, codegen/codesize, or something else?
It used to be performance, since
is_kfence_address()is used in the fast path. However, with some further tweaks we just did tois_kfence_address(), our benchmarks show a pointer load can be tolerated.
Great!
I reckon that this is something we can optimize in futue if necessary (e.g. with some form of code-patching for immediate values), but it's good to have a starting point that works everywhere!
[...]
Quoted 10 lines
Quoted 9 lines
I'm not too worried about allocating this dynamically, but:
* The arch code needs to set up the translation tables for this, as we cannot safely change the mapping granularity live.
* As above I'm fairly certain x86 needs to use a carevout from the linear map to function correctly anyhow, so we should follow the same approach for both arm64 and x86. That might be a static carevout that we figure out the aliasing for, or something entirely dynamic.
Quoted 8 lines
We're going with dynamically allocating the pool (for both x86 and arm64), since any benefits we used to measure from the static pool are no longer measurable (after removing a branch from
is_kfence_address()). It should hopefully simplify a lot of things, given all the caveats that you pointed out.For arm64, the only thing left then is to fix up the case if the linear map is not forced to page granularity.
The simplest way to do this is to modify arm64's arch_add_memory() to force the entire linear map to be mapped at page granularity when KFENCE is enabled, something like:
| diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
| index 936c4762dadff..f6eba0642a4a3 100644
| --- a/arch/arm64/mm/mmu.c
| +++ b/arch/arm64/mm/mmu.c
| @@ -1454,7 +1454,8 @@ int arch_add_memory(int nid, u64 start, u64 size,
| {
| int ret, flags = 0;
|
| - if (rodata_full || debug_pagealloc_enabled())
| + if (rodata_full || debug_pagealloc_enabled() ||
| + IS_ENABLED(CONFIG_KFENCE))
| flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS;
|
| __create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start),... and I given that RODATA_FULL_DEFAULT_ENABLED is the default, I suspect it's not worth trying to only for that for the KFENCE region unless someone complains.
Thanks,
Mark.
Marco Elver
maintainerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
Quoted 46 lines · Mark Rutland
Quoted 6 lines · Marco Elver
On Thu, 1 Oct 2020 at 19:58, Mark Rutland <[email protected]> wrote:
[...]Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
If you need
virt_to_page()to work, the address has to be part of the linear/direct map.[...]
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
What's the underlying requirement here? Is this a performance concern, codegen/codesize, or something else?
It used to be performance, since
is_kfence_address()is used in the fast path. However, with some further tweaks we just did tois_kfence_address(), our benchmarks show a pointer load can be tolerated.Great!
I reckon that this is something we can optimize in futue if necessary (e.g. with some form of code-patching for immediate values), but it's good to have a starting point that works everywhere!
[...]
Quoted 10 lines
Quoted 9 lines
I'm not too worried about allocating this dynamically, but:
* The arch code needs to set up the translation tables for this, as we cannot safely change the mapping granularity live.
* As above I'm fairly certain x86 needs to use a carevout from the linear map to function correctly anyhow, so we should follow the same approach for both arm64 and x86. That might be a static carevout that we figure out the aliasing for, or something entirely dynamic.
Quoted 8 lines
We're going with dynamically allocating the pool (for both x86 and arm64), since any benefits we used to measure from the static pool are no longer measurable (after removing a branch from
is_kfence_address()). It should hopefully simplify a lot of things, given all the caveats that you pointed out.For arm64, the only thing left then is to fix up the case if the linear map is not forced to page granularity.
The simplest way to do this is to modify arm64's
arch_add_memory()to force the entire linear map to be mapped at page granularity when KFENCE is enabled, something like:
[...]
Quoted 4 lines
... and I given that
RODATA_FULL_DEFAULT_ENABLEDis the default, I suspect it's not worth trying to only for that for the KFENCE region unless someone complains.
We've got most of this sorted now for v5 -- thank you!
The only thing we're wondering now, is if there are any corner cases with using memblock_alloc'd memory for the KFENCE pool? (We'd like to avoid page alloc's MAX_ORDER limit.) We have a version that passes tests on x86 and arm64, but checking just in case. :-)
Thanks,
-- Marco
Mark Rutland
reviewerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
Quoted 9 lines · Marco Elver
Quoted 8 lines · Mark Rutland
Quoted 7 lines · Marco Elver
Quoted 2 lines · Mark Rutland
Quoted 2 lines
Quoted 2 lines
If you need
virt_to_page()to work, the address has to be part of the linear/direct map.We're going with dynamically allocating the pool (for both x86 and arm64),
[...]
Quoted 6 lines
We've got most of this sorted now for v5 -- thank you!
The only thing we're wondering now, is if there are any corner cases with using memblock_alloc'd memory for the KFENCE pool? (We'd like to avoid page alloc's
MAX_ORDERlimit.) We have a version that passes tests on x86 and arm64, but checking just in case. :-)
AFAICT otherwise the only noticeable difference might be PageSlab(), if that's clear for KFENCE allocated pages? A few helpers appear to check that to determine how something was allocated (e.g. in the scatterlist and hwpoison code), and I suspect that needs to behave the same.
Otherwise, I *think* using memblock_alloc should be fine on arm64; I'm not entirely sure for x86 (but suspect it's similar). On arm64:
* All memory is given a struct page via memblocks_present() adding all memory memblocks. This includes memory allocated by memblock_alloc().
* All memory is mapped into the linear map via arm64's map_mem() adding all (non-nomap) memory memblocks. This includes memory allocated by memblock_alloc().
Thanks,
Mark.
Marco Elver
maintainerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
Quoted 20 lines · Mark Rutland
Quoted 9 lines · Marco Elver
Quoted 8 lines · Mark Rutland
Quoted 7 lines · Marco Elver
Quoted 2 lines · Mark Rutland
Quoted 2 lines
Quoted 2 lines
If you need
virt_to_page()to work, the address has to be part of the linear/direct map.We're going with dynamically allocating the pool (for both x86 and arm64),
[...]
Quoted 6 lines
We've got most of this sorted now for v5 -- thank you!
The only thing we're wondering now, is if there are any corner cases with using memblock_alloc'd memory for the KFENCE pool? (We'd like to avoid page alloc's
MAX_ORDERlimit.) We have a version that passes tests on x86 and arm64, but checking just in case. :-)
Quoted 4 lines
AFAICT otherwise the only noticeable difference might be
PageSlab(), if that's clear for KFENCE allocated pages? A few helpers appear to check that to determine how something was allocated (e.g. in the scatterlist and hwpoison code), and I suspect that needs to behave the same.
We had to take care of setting PageSlab before, too. We do this during kfence_init().
Quoted 9 lines
Otherwise, I *think* using
memblock_allocshould be fine on arm64; I'm not entirely sure for x86 (but suspect it's similar). On arm64:* All memory is given a struct page via
memblocks_present()adding all memory memblocks. This includes memory allocated bymemblock_alloc().* All memory is mapped into the linear map via arm64's
map_mem()adding all (non-nomap) memory memblocks. This includes memory allocated bymemblock_alloc().
Very good, thank you. We'll send v5 with these changes rebased on 5.10-rc1 (in ~2 weeks).
Thanks,
-- Marco
Marco Elver
maintainerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
In reply to Will DeaconQuoted 6 lines · Will Deacon
Quoted 5 lines · Marco Elver
Add architecture specific implementation details for KFENCE and enable KFENCE for the arm64 architecture. In particular, this implements the required interface in <
asm/kfence.h>. Currently, the arm64 version does not yet use a statically allocated memory pool, at the cost of a pointer load for eachis_kfence_address().
[...]
Quoted 9 lines
Quoted 5 lines
For ARM64, we would like to solicit feedback on what the best option is to obtain a constant address for
__kfence_pool. One option is to declare a memory range in the memory layout to be dedicated to KFENCE (like is done for KASAN), however, it is unclear if this is the best available option. We would like to avoid touching the memory layout.Given that the pool is relatively small (i.e. when compared with our virtual address space), dedicating an area of virtual space sounds like it makes the most sense here. How early do you need it to be available?
Note: we're going to send a v4 this or next week with a few other minor fixes in it. But I think we just don't want to block the entire series on figuring out what the static-pool arm64 version should do, especially if we'll have a few iterations with only this patch here changing.
So the plan will be:
1. Send v4, which could from our point-of-view be picked up for merging. Unless of course there are more comments.
2. Work out the details for the static-pool arm64 version, since it doesn't seem trivial to do the same thing as we do for x86. In preparation for that, v4 will allow the __kfence_pool's attributes to be defined entirely by <asm/kfence.h>, so that we can fiddle with sections etc.
3. Send patch switching out the simpler arm64 version here for one that places __kfence_pool at a static location.
Hopefully that plan is reasonable.
Thanks,
-- Marco
Mark Rutland
reviewerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
In reply to Marco ElverQuoted 68 lines · Marco Elver
Add architecture specific implementation details for KFENCE and enable KFENCE for the arm64 architecture. In particular, this implements the required interface in <
asm/kfence.h>. Currently, the arm64 version does not yet use a statically allocated memory pool, at the cost of a pointer load for eachis_kfence_address().Reviewed-by Dmitry Vyukov <[email protected]>Co-developed-by Alexander Potapenko <[email protected]>Signed-off-by Alexander Potapenko <[email protected]>Signed-off-by Marco Elver <[email protected]>---
For ARM64, we would like to solicit feedback on what the best option is to obtain a constant address for__kfence_pool. One option is to declare a memory range in the memory layout to be dedicated to KFENCE (like is done for KASAN), however, it is unclear if this is the best available option. We would like to avoid touching the memory layout.--- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/kfence.h | 39 +++++++++++++++++++++++++++++++++ arch/arm64/mm/fault.c | 4 ++++ 3 files changed, 44 insertions(+) create mode 100644 arch/arm64/include/asm/kfence.harch/arm64/Kconfig +1
-0@@ -132,6 +132,7 @@ config ARM64 select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN + select HAVE_ARCH_KFENCE if (!ARM64_16K_PAGES && !ARM64_64K_PAGES) select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPATarch/arm64/include/asm/kfence.h new file +27
-0@@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_KFENCE_H +#define __ASM_KFENCE_H + +#include <linux/kfence.h> +#include <linux/log2.h> +#include <linux/mm.h> + +#include <asm/cacheflush.h> + +#define KFENCE_SKIP_ARCH_FAULT_HANDLER "el1_sync" + +/* +* FIXME: Support HAVE_ARCH_KFENCE_STATIC_POOL: Use the statically allocated +* __kfence_pool, to avoid the extra pointer load for is_kfence_address(). By +* default, however, we do not have struct pages for static allocations. +*/ + +static inline bool arch_kfence_initialize_pool (void ) +{ +const unsigned int num_pages =ilog2 (roundup_pow_of_two (KFENCE_POOL_SIZE /PAGE_SIZE )); +struct page *pages =alloc_pages (GFP_KERNEL , num_pages); + +if (!pages) +return false; +
arch/arm64/include/asm/kfence.h line 37
Quoted 10 lines
arch/arm64/include/asm/kfence.h new file +10
-0+ __kfence_pool =page_address (pages); +return true; +} + +static inline bool kfence_protect_page (unsigned long addr,bool protect) +{ +set_memory_valid (addr,1 , !protect); + +return true; +}
This is only safe if the linear map is force ot page granularity. That's the default with rodata=full, but this is not always the case, so this will need some interaction with the MMU setup in arch/arm64/mm/mmu.c.
Thanks,
Mark.
Alexander Potapenko
maintainerRe: [PATCH v3 03/10] arm64, kfence: enable KFENCE for ARM64
Quoted 76 lines · Mark Rutland
Quoted 74 lines · Marco Elver
Add architecture specific implementation details for KFENCE and enable KFENCE for the arm64 architecture. In particular, this implements the required interface in <
asm/kfence.h>. Currently, the arm64 version does not yet use a statically allocated memory pool, at the cost of a pointer load for eachis_kfence_address().Reviewed-by Dmitry Vyukov <[email protected]>Co-developed-by Alexander Potapenko <[email protected]>Signed-off-by Alexander Potapenko <[email protected]>Signed-off-by Marco Elver <[email protected]>---
For ARM64, we would like to solicit feedback on what the best option is to obtain a constant address for__kfence_pool. One option is to declare a memory range in the memory layout to be dedicated to KFENCE (like is done for KASAN), however, it is unclear if this is the best available option. We would like to avoid touching the memory layout.--- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/kfence.h | 39 +++++++++++++++++++++++++++++++++ arch/arm64/mm/fault.c | 4 ++++ 3 files changed, 44 insertions(+) create mode 100644 arch/arm64/include/asm/kfence.harch/arm64/Kconfig +1
-0@@ -132,6 +132,7 @@ config ARM64 select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN + select HAVE_ARCH_KFENCE if (!ARM64_16K_PAGES && !ARM64_64K_PAGES) select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPATarch/arm64/include/asm/kfence.h new file +33
-0@@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_KFENCE_H +#define __ASM_KFENCE_H + +#include <linux/kfence.h> +#include <linux/log2.h> +#include <linux/mm.h> + +#include <asm/cacheflush.h> + +#define KFENCE_SKIP_ARCH_FAULT_HANDLER "el1_sync" + +/* +* FIXME: Support HAVE_ARCH_KFENCE_STATIC_POOL: Use the statically allocated +* __kfence_pool, to avoid the extra pointer load for is_kfence_address(). By +* default, however, we do not have struct pages for static allocations. +*/ + +static inline bool arch_kfence_initialize_pool (void ) +{ +const unsigned int num_pages =ilog2 (roundup_pow_of_two (KFENCE_POOL_SIZE /PAGE_SIZE )); +struct page *pages =alloc_pages (GFP_KERNEL , num_pages); + +if (!pages) +return false; + + __kfence_pool =page_address (pages); +return true; +} + +static inline bool kfence_protect_page (unsigned long addr,bool protect) +{
Quoted 8 lines
arch/arm64/include/asm/kfence.h line 37
Quoted 4 lines
arch/arm64/include/asm/kfence.h new file +4
-0+set_memory_valid (addr,1 , !protect); + +return true; +}This is only safe if the linear map is force ot page granularity. That's the default with rodata=full, but this is not always the case, so this will need some interaction with the MMU setup in
arch/arm64/mm/mmu.c.
On x86 we ensure this by reallocating the necessary page tables.
But looks like your suggestion is what we need for arm64 as long as we also want virt_to_page() to work for our pool.
It's still unclear to me whether a carveout you suggest can be placed at a fixed (known at link time) address, as the main point of this dance is to remove memory accesses from is_kfence_addr().
Quoted 2 lines
Thanks,
Mark.
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Marco Elver
maintainerPATCH v3 4/10 mm, kfence: insert KFENCE hooks for SLAB
In reply to Marco ElverFrom: Alexander Potapenko <[email protected]>
Inserts KFENCE hooks into the SLAB allocator.
To pass the originally requested size to KFENCE, add an argument 'orig_size' to slab_alloc*(). The additional argument is required to preserve the requested original size for kmalloc() allocations, which uses size classes (e.g. an allocation of 272 bytes will return an object of size 512). Therefore, kmem_cache::size does not represent the kmalloc-caller's requested size, and we must introduce the argument 'orig_size' to propagate the originally requested size to KFENCE.
Without the originally requested size, we would not be able to detect out-of-bounds accesses for objects placed at the end of a KFENCE object page if that object is not equal to the kmalloc-size class it was bucketed into.
When KFENCE is disabled, there is no additional overhead, since slab_alloc*() functions are __always_inline.
---
v3:
* Rewrite patch description to clarify need for 'orig_size' [reported by Christopher Lameter].
--- mm/slab.c | 46 ++++++++++++++++++++++++++++++++++------------ mm/slab_common.c | 6 +++++- 2 files changed, 39 insertions(+), 13 deletions(-)
mm/slab.c +34 -12
@@ -100,6 +100,7 @@#include <linux/seq_file.h> #include <linux/notifier.h> #include <linux/kallsyms.h> +#include <linux/kfence.h> #include <linux/cpu.h> #include <linux/sysctl.h> #include <linux/module.h>
@@ -3206,7 +3207,7 @@ static void *____cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, } static __always_inline void * -slab_alloc_node (struct kmem_cache *cachep,gfp_t flags,int nodeid, +slab_alloc_node (struct kmem_cache *cachep,gfp_t flags,int nodeid,size_t orig_size,unsigned long caller) {unsigned long save_flags;
@@ -3219,6 +3220,10 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,if (unlikely (!cachep))return NULL ; + ptr =kfence_alloc (cachep, orig_size, flags); +if (unlikely (ptr)) + goto out_hooks; +cache_alloc_debugcheck_before (cachep, flags);local_irq_save (save_flags);
@@ -3251,6 +3256,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,if (unlikely (slab_want_init_on_alloc (flags, cachep)) && ptr)memset (ptr,0 , cachep->object_size); +out_hooks:slab_post_alloc_hook (cachep, objcg, flags,1 , &ptr);return ptr; }
@@ -3288,7 +3294,7 @@ __do_cache_alloc(struct kmem_cache *cachep, gfp_t flags)#endif /* CONFIG_NUMA */ static __always_inline void * -slab_alloc (struct kmem_cache *cachep,gfp_t flags,unsigned long caller) +slab_alloc (struct kmem_cache *cachep,gfp_t flags,size_t orig_size,unsigned long caller) {unsigned long save_flags;void *objp;
@@ -3299,6 +3305,10 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller)if (unlikely (!cachep))return NULL ; + objp =kfence_alloc (cachep, orig_size, flags); +if (unlikely (objp)) + goto leave; +cache_alloc_debugcheck_before (cachep, flags);local_irq_save (save_flags); objp =__do_cache_alloc (cachep, flags);
@@ -3309,6 +3319,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller)if (unlikely (slab_want_init_on_alloc (flags, cachep)) && objp)memset (objp,0 , cachep->object_size); +leave:slab_post_alloc_hook (cachep, objcg, flags,1 , &objp);return objp; }
@@ -3414,6 +3425,11 @@ static void cache_flusharray(struct kmem_cache *cachep, struct array_cache *ac) static __always_inline void __cache_free(struct kmem_cache *cachep, void *objp,unsigned long caller) { +if (kfence_free (objp)) { +kmemleak_free_recursive (objp, cachep->flags); +return ; + } +/* Put the object into the quarantine, don't touch it for now. */ if (kasan_slab_free (cachep, objp, _RET_IP_))return ;
@@ -3479,7 +3495,7 @@ void ___cache_free(struct kmem_cache *cachep, void *objp,*/ void *kmem_cache_alloc (struct kmem_cache *cachep,gfp_t flags) { -void *ret =slab_alloc (cachep, flags, _RET_IP_); +void *ret =slab_alloc (cachep, flags, cachep->object_size, _RET_IP_);trace_kmem_cache_alloc (_RET_IP_, ret, cachep->object_size, cachep->size, flags);
@@ -3512,7 +3528,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,local_irq_disable ();for (i =0 ; i < size; i++) { -void *objp =__do_cache_alloc (s, flags); +void *objp =kfence_alloc (s, s->object_size, flags) ?:__do_cache_alloc (s, flags);if (unlikely (!objp)) goto error;
@@ -3545,7 +3561,7 @@ kmem_cache_alloc_trace(struct kmem_cache *cachep, gfp_t flags, size_t size) {void *ret; - ret =slab_alloc (cachep, flags, _RET_IP_); + ret =slab_alloc (cachep, flags, size, _RET_IP_); ret =kasan_kmalloc (cachep, ret, size, flags); trace_kmalloc(_RET_IP_, ret,
@@ -3571,7 +3587,7 @@ EXPORT_SYMBOL(kmem_cache_alloc_trace);*/ void *kmem_cache_alloc_node (struct kmem_cache *cachep,gfp_t flags,int nodeid) { -void *ret =slab_alloc_node (cachep, flags, nodeid, _RET_IP_); +void *ret =slab_alloc_node (cachep, flags, nodeid, cachep->object_size, _RET_IP_); trace_kmem_cache_alloc_node(_RET_IP_, ret, cachep->object_size, cachep->size,
@@ -3589,7 +3605,7 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep, {void *ret; - ret =slab_alloc_node (cachep, flags, nodeid, _RET_IP_); + ret =slab_alloc_node (cachep, flags, nodeid, size, _RET_IP_); ret =kasan_kmalloc (cachep, ret, size, flags); trace_kmalloc_node(_RET_IP_, ret,
@@ -3650,7 +3666,7 @@ static __always_inline void *__do_kmalloc(size_t size, gfp_t flags, cachep =kmalloc_slab (size, flags);if (unlikely (ZERO_OR_NULL_PTR (cachep)))return cachep; - ret =slab_alloc (cachep, flags, caller); + ret =slab_alloc (cachep, flags, size, caller); ret =kasan_kmalloc (cachep, ret, size, flags); trace_kmalloc(caller, ret,
@@ -4138,18 +4154,24 @@ void __check_heap_object(const void *ptr, unsigned long n, struct page *page, bool to_user) {struct kmem_cache *cachep; -unsigned int objnr; +unsigned int objnr =0 ;unsigned long offset; +bool is_kfence =is_kfence_address (ptr); ptr =kasan_reset_tag (ptr);/* Find and validate object. */ cachep = page->slab_cache; - objnr =obj_to_index (cachep, page, (void *)ptr); -BUG_ON (objnr >= cachep->num); +if (!is_kfence) { + objnr =obj_to_index (cachep, page, (void *)ptr); +BUG_ON (objnr >= cachep->num); + }/* Find offset within object. */ - offset = ptr -index_to_obj (cachep, page, objnr) -obj_offset (cachep); +if (is_kfence_address (ptr)) + offset = ptr -kfence_object_start (ptr); +else + offset = ptr -index_to_obj (cachep, page, objnr) -obj_offset (cachep);/* Allow address range falling entirely within usercopy region. */ if (offset >= cachep->useroffset &&
mm/slab_common.c +5 -1
@@ -12,6 +12,7 @@#include <linux/memory.h> #include <linux/cache.h> #include <linux/compiler.h> +#include <linux/kfence.h> #include <linux/module.h> #include <linux/cpu.h> #include <linux/uaccess.h>
@@ -448,6 +449,9 @@ static int shutdown_cache(struct kmem_cache *s)/* free asan quarantined objects */ kasan_cache_shutdown (s); +if (!kfence_shutdown_cache (s)) +return -EBUSY ; +if (__kmem_cache_shutdown (s) !=0 )return -EBUSY ;
@@ -1171,7 +1175,7 @@ size_t ksize(const void *objp)if (unlikely (ZERO_OR_NULL_PTR (objp)) || !__kasan_check_read (objp,1 ))return 0 ; - size =__ksize (objp); + size =kfence_ksize (objp) ?:__ksize (objp);/* * We assume that ksize callers could use whole allocated area, * so we need to unpoison this area.
Signature · 1 line
2.28.0.681.g6f77f65b4e-goog
Marco Elver
maintainerPATCH v3 5/10 mm, kfence: insert KFENCE hooks for SLUB
In reply to Marco ElverFrom: Alexander Potapenko <[email protected]>
Inserts KFENCE hooks into the SLUB allocator.
To pass the originally requested size to KFENCE, add an argument 'orig_size' to slab_alloc*(). The additional argument is required to preserve the requested original size for kmalloc() allocations, which uses size classes (e.g. an allocation of 272 bytes will return an object of size 512). Therefore, kmem_cache::size does not represent the kmalloc-caller's requested size, and we must introduce the argument 'orig_size' to propagate the originally requested size to KFENCE.
Without the originally requested size, we would not be able to detect out-of-bounds accesses for objects placed at the end of a KFENCE object page if that object is not equal to the kmalloc-size class it was bucketed into.
When KFENCE is disabled, there is no additional overhead, since slab_alloc*() functions are __always_inline.
---
v3:
* Rewrite patch description to clarify need for 'orig_size' [reported by Christopher Lameter].
--- mm/slub.c | 72 ++++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 53 insertions(+), 19 deletions(-)
mm/slub.c +53 -19
@@ -27,6 +27,7 @@#include <linux/ctype.h> #include <linux/debugobjects.h> #include <linux/kallsyms.h> +#include <linux/kfence.h> #include <linux/memory.h> #include <linux/math64.h> #include <linux/fault-inject.h>
@@ -1557,6 +1558,11 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s,void *old_tail = *tail ? *tail : *head;int rsize; +if (is_kfence_address (next)) { +slab_free_hook (s, next); +return true; + } +/* Head and tail of the reconstructed freelist */ *head =NULL ; *tail =NULL ;
@@ -2660,7 +2666,8 @@ static inline void *get_freelist(struct kmem_cache *s, struct page *page)* already disabled (which is the case for bulk allocation). */ static void *___slab_alloc (struct kmem_cache *s,gfp_t gfpflags,int node, -unsigned long addr,struct kmem_cache_cpu *c) +unsigned long addr,struct kmem_cache_cpu *c, +size_t orig_size) {void *freelist;struct page *page;
@@ -2763,7 +2770,8 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,* cpu changes by refetching the per cpu area pointer. */ static void *__slab_alloc (struct kmem_cache *s,gfp_t gfpflags,int node, -unsigned long addr,struct kmem_cache_cpu *c) +unsigned long addr,struct kmem_cache_cpu *c, +size_t orig_size) {void *p;unsigned long flags;
@@ -2778,7 +2786,7 @@ static void *__slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, c =this_cpu_ptr (s->cpu_slab);#endif - p =___slab_alloc (s, gfpflags, node, addr, c); + p =___slab_alloc (s, gfpflags, node, addr, c, orig_size);local_irq_restore (flags);return p; }
@@ -2805,7 +2813,7 @@ static __always_inline void maybe_wipe_obj_freeptr(struct kmem_cache *s,* Otherwise we can simply pick the next object from the lockless free list. */ static __always_inline void *slab_alloc_node(struct kmem_cache *s, -gfp_t gfpflags,int node,unsigned long addr) +gfp_t gfpflags,int node,unsigned long addr,size_t orig_size) {void *object;struct kmem_cache_cpu *c;
@@ -2816,6 +2824,11 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, s =slab_pre_alloc_hook (s, &objcg,1 , gfpflags);if (!s)return NULL ; + + object =kfence_alloc (s, orig_size, gfpflags); +if (unlikely (object)) + goto out; + redo:/* * Must read kmem_cache cpu data via this cpu ptr. Preemption is
@@ -2853,7 +2866,7 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, object = c->freelist; page = c->page;if (unlikely (!object || !node_match (page, node))) { - object =__slab_alloc (s, gfpflags, node, addr, c); + object =__slab_alloc (s, gfpflags, node, addr, c, orig_size);stat (s,ALLOC_SLOWPATH ); }else {void *next_object =get_freepointer_safe (s, object);
@@ -2889,20 +2902,21 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,if (unlikely (slab_want_init_on_alloc (gfpflags, s)) && object)memset (object,0 , s->object_size); +out:slab_post_alloc_hook (s, objcg, gfpflags,1 , &object);return object; } static __always_inline void *slab_alloc(struct kmem_cache *s, -gfp_t gfpflags,unsigned long addr) +gfp_t gfpflags,unsigned long addr,size_t orig_size) { -return slab_alloc_node (s, gfpflags,NUMA_NO_NODE , addr); +return slab_alloc_node (s, gfpflags,NUMA_NO_NODE , addr, orig_size); }void *kmem_cache_alloc (struct kmem_cache *s,gfp_t gfpflags) { -void *ret =slab_alloc (s, gfpflags, _RET_IP_); +void *ret =slab_alloc (s, gfpflags, _RET_IP_, s->object_size);trace_kmem_cache_alloc (_RET_IP_, ret, s->object_size, s->size, gfpflags);
@@ -2914,7 +2928,7 @@ EXPORT_SYMBOL(kmem_cache_alloc);#ifdef CONFIG_TRACING void *kmem_cache_alloc_trace (struct kmem_cache *s,gfp_t gfpflags,size_t size) { -void *ret =slab_alloc (s, gfpflags, _RET_IP_); +void *ret =slab_alloc (s, gfpflags, _RET_IP_, size);trace_kmalloc (_RET_IP_, ret, size, s->size, gfpflags); ret =kasan_kmalloc (s, ret, size, gfpflags);return ret;
@@ -2925,7 +2939,7 @@ EXPORT_SYMBOL(kmem_cache_alloc_trace);#ifdef CONFIG_NUMA void *kmem_cache_alloc_node (struct kmem_cache *s,gfp_t gfpflags,int node) { -void *ret =slab_alloc_node (s, gfpflags, node, _RET_IP_); +void *ret =slab_alloc_node (s, gfpflags, node, _RET_IP_, s->object_size);trace_kmem_cache_alloc_node (_RET_IP_, ret, s->object_size, s->size, gfpflags, node);
@@ -2939,7 +2953,7 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *s,gfp_t gfpflags, int node, size_t size) { - void *ret = slab_alloc_node(s, gfpflags, node, _RET_IP_); + void *ret = slab_alloc_node(s, gfpflags, node, _RET_IP_, size);trace_kmalloc_node (_RET_IP_, ret, size, s->size, gfpflags, node);
@@ -2973,6 +2987,9 @@ static void __slab_free(struct kmem_cache *s, struct page *page,stat (s,FREE_SLOWPATH ); +if (kfence_free (head)) +return ; +if (kmem_cache_debug (s) && !free_debug_processing (s, page, head, tail, cnt, addr))return ;
@@ -3216,6 +3233,13 @@ int build_detached_freelist(struct kmem_cache *s, size_t size, df->s =cache_from_obj (s, object);/* Support for memcg */ } +if (is_kfence_address (object)) { +slab_free_hook (df->s, object); +WARN_ON (!kfence_free (object)); + p[size] =NULL ;/* mark object processed */ +return size; + } +/* Start new detached freelist */ df->page = page;set_freepointer (df->s, object,NULL );
@@ -3290,8 +3314,14 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, c =this_cpu_ptr (s->cpu_slab);for (i =0 ; i < size; i++) { -void *object = c->freelist; +void *object =kfence_alloc (s, s->object_size, flags); +if (unlikely (object)) { + p[i] = object; +continue ; + } + + object = c->freelist;if (unlikely (!object)) {/* * We may have removed an object from c->freelist using
@@ -3307,7 +3337,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,* of re-populating per CPU c->freelist */ p[i] =___slab_alloc (s, flags,NUMA_NO_NODE , - _RET_IP_, c); + _RET_IP_, c, size);if (unlikely (!p[i])) goto error;
@@ -3962,7 +3992,7 @@ void *__kmalloc(size_t size, gfp_t flags)if (unlikely (ZERO_OR_NULL_PTR (s)))return s; - ret =slab_alloc (s, flags, _RET_IP_); + ret =slab_alloc (s, flags, _RET_IP_, size);trace_kmalloc (_RET_IP_, ret, size, s->size, flags);
@@ -4010,7 +4040,7 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node)if (unlikely (ZERO_OR_NULL_PTR (s)))return s; - ret =slab_alloc_node (s, flags, node, _RET_IP_); + ret =slab_alloc_node (s, flags, node, _RET_IP_, size);trace_kmalloc_node (_RET_IP_, ret, size, s->size, flags, node);
@@ -4036,6 +4066,7 @@ void __check_heap_object(const void *ptr, unsigned long n, struct page *page,struct kmem_cache *s;unsigned int offset;size_t object_size; +bool is_kfence =is_kfence_address (ptr); ptr =kasan_reset_tag (ptr);
@@ -4048,10 +4079,13 @@ void __check_heap_object(const void *ptr, unsigned long n, struct page *page, to_user, 0, n);/* Find offset within object. */ - offset = (ptr -page_address (page)) % s->size; +if (is_kfence) + offset = ptr -kfence_object_start (ptr); +else + offset = (ptr -page_address (page)) % s->size;/* Adjust for redzone and reject if within the redzone. */ -if (kmem_cache_debug_flags (s,SLAB_RED_ZONE )) { +if (!is_kfence &&kmem_cache_debug_flags (s,SLAB_RED_ZONE )) {if (offset < s->red_left_pad)usercopy_abort ("SLUB object in left red zone" , s->name, to_user, offset, n);
@@ -4460,7 +4494,7 @@ void *__kmalloc_track_caller(size_t size, gfp_t gfpflags, unsigned long caller)if (unlikely (ZERO_OR_NULL_PTR (s)))return s; - ret =slab_alloc (s, gfpflags, caller); + ret =slab_alloc (s, gfpflags, caller, size);/* Honor the call site pointer we received. */ trace_kmalloc (caller, ret, size, s->size, gfpflags);
@@ -4491,7 +4525,7 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,if (unlikely (ZERO_OR_NULL_PTR (s)))return s; - ret =slab_alloc_node (s, gfpflags, node, caller); + ret =slab_alloc_node (s, gfpflags, node, caller, size);/* Honor the call site pointer we received. */ trace_kmalloc_node (caller, ret, size, s->size, gfpflags, node);
Signature · 1 line
2.28.0.681.g6f77f65b4e-goog
Marco Elver
maintainerPATCH v3 6/10 kfence, kasan: make KFENCE compatible with KASAN
In reply to Marco ElverFrom: Alexander Potapenko <[email protected]>
We make KFENCE compatible with KASAN for testing KFENCE itself. In particular, KASAN helps to catch any potential corruptions to KFENCE state, or other corruptions that may be a result of freepointer corruptions in the main allocators.
To indicate that the combination of the two is generally discouraged, CONFIG_EXPERT=y should be set. It also gives us the nice property that KFENCE will be build-tested by allyesconfig builds.
--- lib/Kconfig.kfence | 2 +- mm/kasan/common.c | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-)
lib/Kconfig.kfence +1 -1
@@ -10,7 +10,7 @@ config HAVE_ARCH_KFENCE_STATIC_POOL menuconfig KFENCE bool "KFENCE: low-overhead sampling-based memory safety error detector" - depends on HAVE_ARCH_KFENCE && !KASAN && (SLAB || SLUB) + depends on HAVE_ARCH_KFENCE && (!KASAN || EXPERT) && (SLAB || SLUB) depends on JUMP_LABEL # To ensure performance, require jump labels select STACKTRACE help
mm/kasan/common.c +7 -0
@@ -18,6 +18,7 @@#include <linux/init.h> #include <linux/kasan.h> #include <linux/kernel.h> +#include <linux/kfence.h> #include <linux/kmemleak.h> #include <linux/linkage.h> #include <linux/memblock.h>
@@ -396,6 +397,9 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object, tagged_object = object; object =reset_tag (object); +if (is_kfence_address (object)) +return false; +if (unlikely (nearest_obj (cache,virt_to_head_page (object), object) != object)) {kasan_report_invalid_free (tagged_object, ip);
@@ -444,6 +448,9 @@ static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,if (unlikely (object ==NULL ))return NULL ; +if (is_kfence_address (object)) +return (void *)object; + redzone_start =round_up ((unsigned long )(object + size),KASAN_SHADOW_SCALE_SIZE ); redzone_end = round_up((unsigned long)object + cache->object_size,
Signature · 1 line
2.28.0.681.g6f77f65b4e-goog
Re: [PATCH v3 06/10] kfence, kasan: make KFENCE compatible with KASAN
Quoted 67 lines · Marco Elver
From: Alexander Potapenko <[email protected]>
We make KFENCE compatible with KASAN for testing KFENCE itself. In particular, KASAN helps to catch any potential corruptions to KFENCE state, or other corruptions that may be a result of freepointer corruptions in the main allocators.
To indicate that the combination of the two is generally discouraged,
CONFIG_EXPERT=yshould be set. It also gives us the nice property that KFENCE will be build-tested by allyesconfig builds.Reviewed-by Dmitry Vyukov <[email protected]>Co-developed-by Marco Elver <[email protected]>Signed-off-by Marco Elver <[email protected]>Signed-off-by Alexander Potapenko <[email protected]>--- lib/Kconfig.kfence | 2 +- mm/kasan/common.c | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-)lib/Kconfig.kfence +1
-1@@ -10,7 +10,7 @@ config HAVE_ARCH_KFENCE_STATIC_POOL menuconfig KFENCE bool "KFENCE: low-overhead sampling-based memory safety error detector" - depends on HAVE_ARCH_KFENCE && !KASAN && (SLAB || SLUB) + depends on HAVE_ARCH_KFENCE && (!KASAN || EXPERT) && (SLAB || SLUB) depends on JUMP_LABEL # To ensure performance, require jump labels select STACKTRACE helpmm/kasan/common.c +7
-1@@ -18,6 +18,7 @@#include <linux/init.h> #include <linux/kasan.h> #include <linux/kernel.h> +#include <linux/kfence.h> #include <linux/kmemleak.h> #include <linux/linkage.h> #include <linux/memblock.h> @@ -396,6 +397,9 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object, tagged_object = object; object =reset_tag (object); +if (is_kfence_address (object)) +return false; +if (unlikely (nearest_obj (cache,virt_to_head_page (object), object) != object)) {kasan_report_invalid_free (tagged_object, ip);@@ -444,6 +448,9 @@ static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,if (unlikely (object ==NULL ))return NULL ; +if (is_kfence_address (object)) +return (void *)object; + redzone_start =round_up ((unsigned long )(object + size),KASAN_SHADOW_SCALE_SIZE ); redzone_end = round_up((unsigned long)object + cache->object_size, --
Quoted 2 lines
2.28.0.681.g6f77f65b4e-goog
With KFENCE + KASAN both enabled we need to bail out in all KASAN hooks that get called from the allocator, right? Do I understand correctly that these two are the only ones that are called for KFENCE-allocated objects due to the way KFENCE is integrated into the allocator?
Alexander Potapenko
maintainerRe: [PATCH v3 06/10] kfence, kasan: make KFENCE compatible with KASAN
Quoted 62 lines · Andrey Konovalov
Quoted 60 lines · Marco Elver
From: Alexander Potapenko <[email protected]>
We make KFENCE compatible with KASAN for testing KFENCE itself. In particular, KASAN helps to catch any potential corruptions to KFENCE state, or other corruptions that may be a result of freepointer corruptions in the main allocators.
To indicate that the combination of the two is generally discouraged,
CONFIG_EXPERT=yshould be set. It also gives us the nice property that KFENCE will be build-tested by allyesconfig builds.Reviewed-by Dmitry Vyukov <[email protected]>Co-developed-by Marco Elver <[email protected]>Signed-off-by Marco Elver <[email protected]>Signed-off-by Alexander Potapenko <[email protected]>--- lib/Kconfig.kfence | 2 +- mm/kasan/common.c | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-)lib/Kconfig.kfence +1
-1@@ -10,7 +10,7 @@ config HAVE_ARCH_KFENCE_STATIC_POOL menuconfig KFENCE bool "KFENCE: low-overhead sampling-based memory safety error detector" - depends on HAVE_ARCH_KFENCE && !KASAN && (SLAB || SLUB) + depends on HAVE_ARCH_KFENCE && (!KASAN || EXPERT) && (SLAB || SLUB) depends on JUMP_LABEL # To ensure performance, require jump labels select STACKTRACE helpmm/kasan/common.c +4
-0@@ -18,6 +18,7 @@#include <linux/init.h> #include <linux/kasan.h> #include <linux/kernel.h> +#include <linux/kfence.h> #include <linux/kmemleak.h> #include <linux/linkage.h> #include <linux/memblock.h> @@ -396,6 +397,9 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object, tagged_object = object; object =reset_tag (object); +if (is_kfence_address (object)) +return false; +if (unlikely (nearest_obj (cache,virt_to_head_page (object), object) != object)) {kasan_report_invalid_free (tagged_object, ip);@@ -444,6 +448,9 @@ static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,if (unlikely (object ==NULL ))return NULL ;
Quoted 15 lines
Quoted 9 lines
mm/kasan/common.c +3
-1+if (is_kfence_address (object)) +return (void *)object; + redzone_start =round_up ((unsigned long )(object + size),KASAN_SHADOW_SCALE_SIZE ); redzone_end = round_up((unsigned long)object + cache->object_size, --2.28.0.681.g6f77f65b4e-goog
With KFENCE + KASAN both enabled we need to bail out in all KASAN hooks that get called from the allocator, right? Do I understand correctly that these two are the only ones that are called for KFENCE-allocated objects due to the way KFENCE is integrated into the allocator?
Yes, these two places were sufficient; we've checked that KFENCE and KASAN work together.
Marco Elver
maintainerPATCH v3 7/10 kfence, kmemleak: make KFENCE compatible with KMEMLEAK
In reply to Marco ElverFrom: Alexander Potapenko <[email protected]>
Add compatibility with KMEMLEAK, by making KMEMLEAK aware of the KFENCE memory pool. This allows building debug kernels with both enabled, which also helped in debugging KFENCE.
---
v2:
* Rework using delete_object_part() [suggested by Catalin Marinas].
--- mm/kmemleak.c | 6 ++++++ 1 file changed, 6 insertions(+)
mm/kmemleak.c +6 -0
@@ -97,6 +97,7 @@#include <linux/atomic.h> #include <linux/kasan.h> +#include <linux/kfence.h> #include <linux/kmemleak.h> #include <linux/memory_hotplug.h>
@@ -1948,6 +1949,11 @@ void __init kmemleak_init(void) KMEMLEAK_GREY, GFP_ATOMIC);create_object ((unsigned long )__bss_start, __bss_stop - __bss_start,KMEMLEAK_GREY ,GFP_ATOMIC ); +#if defined(CONFIG_KFENCE ) && defined(CONFIG_HAVE_ARCH_KFENCE_STATIC_POOL ) +/* KFENCE objects are located in .bss, which may confuse kmemleak. Skip them. */ +delete_object_part ((unsigned long )__kfence_pool,KFENCE_POOL_SIZE ); +#endif +/* only register .data..ro_after_init if not within .data */ if (&__start_ro_after_init < &_sdata || &__end_ro_after_init > &_edata) create_object((unsigned long)__start_ro_after_init,
Signature · 1 line
2.28.0.681.g6f77f65b4e-goog
Marco Elver
maintainerPATCH v3 8/10 kfence, lockdep: make KFENCE compatible with lockdep
In reply to Marco ElverLockdep checks that dynamic key registration is only performed on keys that are not static objects. With KFENCE, it is possible that such a dynamically allocated key is a KFENCE object which may, however, be allocated from a static memory pool (if HAVE_ARCH_KFENCE_STATIC_POOL).
Therefore, ignore KFENCE-allocated objects in static_obj().
--- kernel/locking/lockdep.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/locking/lockdep.c +8 -0
@@ -38,6 +38,7 @@#include <linux/seq_file.h> #include <linux/spinlock.h> #include <linux/kallsyms.h> +#include <linux/kfence.h> #include <linux/interrupt.h> #include <linux/stacktrace.h> #include <linux/debug_locks.h>
@@ -755,6 +756,13 @@ static int static_obj(const void *obj)if (arch_is_kernel_initmem_freed (addr))return 0 ; +/* +* KFENCE objects may be allocated from a static memory pool, but are +* not actually static objects. +*/ +if (is_kfence_address (obj)) +return 0 ; +/* * static variable? */
Signature · 1 line
2.28.0.681.g6f77f65b4e-goog
Marco Elver
maintainerPATCH v3 9/10 kfence, Documentation: add KFENCE documentation
In reply to Marco ElverAdd KFENCE documentation in dev-tools/kfence.rst, and add to index.
---
v3:
* Re-introduce reference to Documentation/dev-tools/kfence.rst.
v2:
* Many clarifications based on comments from Andrey Konovalov.
* Document CONFIG_KFENCE_SAMPLE_INTERVAL=0 usage.
* Make use-cases between KASAN and KFENCE clearer.
* Be clearer about the fact the pool is fixed size.
* Update based on reporting changes.
* Explicitly mention max supported allocation size is PAGE_SIZE.
--- Documentation/dev-tools/index.rst | 1 + Documentation/dev-tools/kfence.rst | 291 +++++++++++++++++++++++++++++ lib/Kconfig.kfence | 2 + 3 files changed, 294 insertions(+) create mode 100644 Documentation/dev-tools/kfence.rst
Documentation/dev-tools/index.rst +1 -0
@@ -22,6 +22,7 @@ whole; patches welcome! ubsan kmemleak kcsan + kfence gdb-kernel-debugging kgdb kselftest
Documentation/dev-tools/kfence.rst new file +291 -0
291 lines not on this page · raw message
lib/Kconfig.kfence +2 -0
@@ -19,6 +19,8 @@ menuconfig KFENCE to have negligible cost to permit enabling it in production environments. + See <file:Documentation/dev-tools/kfence.rst> for more details. + Note that, KFENCE is not a substitute for explicit testing with tools such as KASAN. KFENCE can detect a subset of bugs that KASAN can detect, albeit at very different performance profiles. If you can
Signature · 1 line
2.28.0.681.g6f77f65b4e-goog
Marco Elver
maintainerPATCH v3 10/10 kfence: add test suite
In reply to Marco ElverAdd KFENCE test suite, testing various error detection scenarios. Makes use of KUnit for test organization. Since KFENCE's interface to obtain error reports is via the console, the test verifies that KFENCE outputs expected reports to the console.
---
v3:
* Lower line buffer size to avoid warnings of using more than 1024 bytes stack usage [reported by kernel test robot <[email protected]>].
v2:
* Update for shortened memory corruption report.
--- lib/Kconfig.kfence | 13 + mm/kfence/Makefile | 3 + mm/kfence/kfence_test.c | 777 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 793 insertions(+) create mode 100644 mm/kfence/kfence_test.c
lib/Kconfig.kfence +13 -0
@@ -62,4 +62,17 @@ config KFENCE_STRESS_TEST_FAULTS The option is only to test KFENCE; set to 0 if you are unsure. +config KFENCE_KUNIT_TEST + tristate "KFENCE integration test suite" if !KUNIT_ALL_TESTS + default KUNIT_ALL_TESTS + depends on TRACEPOINTS && KUNIT + help + Test suite for KFENCE, testing various error detection scenarios with + various allocation types, and checking that reports are correctly + output to console. + + Say Y here if you want the test to be built into the kernel and run + during boot; say M if you want the test to build as a module; say N + if you are unsure. + endif # KFENCE
mm/kfence/Makefile +3 -0
@@ -1,3 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_KFENCE) := core.o report.o + +CFLAGS_kfence_test.o := -g -fno-omit-frame-pointer -fno-optimize-sibling-calls +obj-$(CONFIG_KFENCE_KUNIT_TEST) += kfence_test.o
mm/kfence/kfence_test.c new file +777 -0
777 lines not on this page · raw message
Signature · 1 line
2.28.0.681.g6f77f65b4e-goog
Re: [PATCH v3 10/10] kfence: add test suite
Quoted 9 lines · Marco Elver
Add KFENCE test suite, testing various error detection scenarios. Makes use of KUnit for test organization. Since KFENCE's interface to obtain error reports is via the console, the test verifies that KFENCE outputs expected reports to the console.
Reviewed-by Dmitry Vyukov <[email protected]>Co-developed-by Alexander Potapenko <[email protected]>Signed-off-by Alexander Potapenko <[email protected]>Signed-off-by Marco Elver <[email protected]>
[ . . . ]
Quoted 9 lines
mm/kfence/kfence_test.c new file +9
-0+/* Test SLAB_TYPESAFE_BY_RCU works. */ +static void test_memcache_typesafe_by_rcu (struct kunit *test) +{ +const size_t size =32 ; +struct expect_report expect = { + .type =KFENCE_ERROR_UAF , + .fn = test_memcache_typesafe_by_rcu, + }; +
mm/kfence/kfence_test.c line 586
Quoted 10 lines
mm/kfence/kfence_test.c new file +10
-0+setup_test_cache (test, size,SLAB_TYPESAFE_BY_RCU ,NULL ); +KUNIT_EXPECT_TRUE (test, test_cache);/* Want memcache. */ + + expect.addr =test_alloc (test, size,GFP_KERNEL ,ALLOCATE_ANY ); + *expect.addr =42 ; + +rcu_read_lock (); +test_free (expect.addr); +KUNIT_EXPECT_EQ (test, *expect.addr, (char )42 ); +rcu_read_unlock ();
It won't happen very often, but memory really could be freed at this point, especially in CONFIG_RCU_STRICT_GRACE_PERIOD=y kernels ...
mm/kfence/kfence_test.c line 589
Quoted 2 lines
mm/kfence/kfence_test.c new file +2
-0+/* No reports yet, memory should not have been freed on access. */ +KUNIT_EXPECT_FALSE (test,report_available ());
... so the above statement needs to go before the rcu_read_unlock().
mm/kfence/kfence_test.c line 590
Quoted 1 line
mm/kfence/kfence_test.c new file +1
-0+rcu_barrier ();/* Wait for free to happen. */
But you are quite right that the memory is not -guaranteed- to be freed until we get here.
Thanx, Paul
Marco Elver
maintainerRe: [PATCH v3 10/10] kfence: add test suite
Quoted 28 lines · Paul E. McKenney
Quoted 9 lines · Marco Elver
Add KFENCE test suite, testing various error detection scenarios. Makes use of KUnit for test organization. Since KFENCE's interface to obtain error reports is via the console, the test verifies that KFENCE outputs expected reports to the console.
Reviewed-by Dmitry Vyukov <[email protected]>Co-developed-by Alexander Potapenko <[email protected]>Signed-off-by Alexander Potapenko <[email protected]>Signed-off-by Marco Elver <[email protected]>[ . . . ]
Quoted 14 lines
mm/kfence/kfence_test.c new file +14
-0+/* Test SLAB_TYPESAFE_BY_RCU works. */ +static void test_memcache_typesafe_by_rcu (struct kunit *test) +{ +const size_t size =32 ; +struct expect_report expect = { + .type =KFENCE_ERROR_UAF , + .fn = test_memcache_typesafe_by_rcu, + }; + +setup_test_cache (test, size,SLAB_TYPESAFE_BY_RCU ,NULL ); +KUNIT_EXPECT_TRUE (test, test_cache);/* Want memcache. */ + + expect.addr =test_alloc (test, size,GFP_KERNEL ,ALLOCATE_ANY ); + *expect.addr =42 ;
Quoted 8 lines
mm/kfence/kfence_test.c line 586
Quoted 5 lines
mm/kfence/kfence_test.c new file +5
-0+ +rcu_read_lock (); +test_free (expect.addr); +KUNIT_EXPECT_EQ (test, *expect.addr, (char )42 ); +rcu_read_unlock ();It won't happen very often, but memory really could be freed at this point, especially in
CONFIG_RCU_STRICT_GRACE_PERIOD=ykernels ...
Ah, thanks for pointing it out.
Quoted 4 lines
mm/kfence/kfence_test.c line 589
Quoted 2 lines
mm/kfence/kfence_test.c new file +2
-0+/* No reports yet, memory should not have been freed on access. */ +KUNIT_EXPECT_FALSE (test,report_available ());... so the above statement needs to go before the
rcu_read_unlock().
You mean the comment (and not the KUNIT_EXPECT_FALSE that no reports were generated), correct?
Admittedly, the whole comment is a bit imprecise, so I'll reword.
Quoted 4 lines
Quoted 1 line
+
rcu_barrier(); /* Wait for free to happen. */But you are quite right that the memory is not -guaranteed- to be freed until we get here.
Right, I'll update the comment.
Thanks,
-- Marco
Re: [PATCH v3 10/10] kfence: add test suite
Quoted 43 lines · Marco Elver
Quoted 28 lines · Paul E. McKenney
Quoted 9 lines · Marco Elver
Add KFENCE test suite, testing various error detection scenarios. Makes use of KUnit for test organization. Since KFENCE's interface to obtain error reports is via the console, the test verifies that KFENCE outputs expected reports to the console.
Reviewed-by Dmitry Vyukov <[email protected]>Co-developed-by Alexander Potapenko <[email protected]>Signed-off-by Alexander Potapenko <[email protected]>Signed-off-by Marco Elver <[email protected]>[ . . . ]
Quoted 14 lines
mm/kfence/kfence_test.c new file +14
-0+/* Test SLAB_TYPESAFE_BY_RCU works. */ +static void test_memcache_typesafe_by_rcu (struct kunit *test) +{ +const size_t size =32 ; +struct expect_report expect = { + .type =KFENCE_ERROR_UAF , + .fn = test_memcache_typesafe_by_rcu, + }; + +setup_test_cache (test, size,SLAB_TYPESAFE_BY_RCU ,NULL ); +KUNIT_EXPECT_TRUE (test, test_cache);/* Want memcache. */ + + expect.addr =test_alloc (test, size,GFP_KERNEL ,ALLOCATE_ANY ); + *expect.addr =42 ;Quoted 8 lines
mm/kfence/kfence_test.c line 586
Quoted 5 lines
mm/kfence/kfence_test.c new file +5
-0+ +rcu_read_lock (); +test_free (expect.addr); +KUNIT_EXPECT_EQ (test, *expect.addr, (char )42 ); +rcu_read_unlock ();It won't happen very often, but memory really could be freed at this point, especially in
CONFIG_RCU_STRICT_GRACE_PERIOD=ykernels ...Ah, thanks for pointing it out.
Quoted 3 lines
Quoted 2 lines
mm/kfence/kfence_test.c new file +2
-0+/* No reports yet, memory should not have been freed on access. */ +KUNIT_EXPECT_FALSE (test,report_available ());
Quoted 6 lines
Quoted 1 line
... so the above statement needs to go before the
rcu_read_unlock().You mean the comment (and not the
KUNIT_EXPECT_FALSEthat no reports were generated), correct?Admittedly, the whole comment is a bit imprecise, so I'll reword.
I freely confess that I did not research exactly what might generate a report. But if this KUNIT_EXPECT_FALSE() was just verifying that the previous KUNIT_EXPECT_TRUE() did not trigger, then yes, the code is just fine as it is.
Thanx, Paul
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
+
rcu_barrier(); /* Wait for free to happen. */But you are quite right that the memory is not -guaranteed- to be freed until we get here.
Right, I'll update the comment.
Thanks,
-- Marco
Dmitry Vyukov
reviewerQuoted 6 lines · Marco Elver
This adds the Kernel Electric-Fence (KFENCE) infrastructure. KFENCE is a low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors. This series enables KFENCE for the x86 and arm64 architectures, and adds KFENCE hooks to the SLAB and SLUB allocators.
Hi Andrew,
I wanted to ask what we can expect with respect to the timeline of merging this into mm/upstream? The series got few reviews/positive feedback.
Thank you
Quoted 115 lines
KFENCE is designed to be enabled in production kernels, and has near zero performance overhead. Compared to KASAN, KFENCE trades performance for precision. The main motivation behind KFENCE's design, is that with enough total uptime KFENCE will detect bugs in code paths not typically exercised by non-production test workloads. One way to quickly achieve a large enough total uptime is when the tool is deployed across a large fleet of machines.
KFENCE objects each reside on a dedicated page, at either the left or right page boundaries. The pages to the left and right of the object page are "guard pages", whose attributes are changed to a protected state, and cause page faults on any attempted access to them. Such page faults are then intercepted by KFENCE, which handles the fault gracefully by reporting a memory access error.
Guarded allocations are set up based on a sample interval (can be set via kfence.sample_interval). After expiration of the sample interval, the next allocation through the main allocator (SLAB or SLUB) returns a guarded allocation from the KFENCE object pool. At this point, the timer is reset, and the next allocation is set up after the expiration of the interval.
To enable/disable a KFENCE allocation through the main allocator's fast-path without overhead, KFENCE relies on static branches via the static keys infrastructure. The static branch is toggled to redirect the allocation to KFENCE.
The KFENCE memory pool is of fixed size, and if the pool is exhausted no further KFENCE allocations occur. The default config is conservative with only 255 objects, resulting in a pool size of 2 MiB (with 4 KiB pages).
We have verified by running synthetic benchmarks (sysbench I/O, hackbench) that a kernel with KFENCE is performance-neutral compared to a non-KFENCE baseline kernel.
KFENCE is inspired by GWP-ASan [1], a userspace tool with similar properties. The name "KFENCE" is a homage to the Electric Fence Malloc Debugger [2].
For more details, see
Documentation/dev-tools/kfence.rstadded in the series -- also viewable here:https://raw.githubusercontent.com/google/kasan/kfence/Documentation/dev-tools/kfence.rst
[1] http://llvm.org/docs/GwpAsan.html
[2] https://linux.die.net/man/3/efencev3:
* Rewrite SLAB/SLUB patch descriptions to clarify need for 'orig_size'.
* Various smaller fixes (see details in patches).v2: https://lkml.kernel.org/r/[email protected]
* Various comment/documentation changes (see details in patches).
* Various smaller fixes (see details in patches).
* Change all reports to reference the kfence object, "kfence-#nn".
* Skip allocation/free internals stack trace.
* Rework KMEMLEAK compatibility patch.RFC/v1: https://lkml.kernel.org/r/[email protected]
Alexander Potapenko (6):
mm: add Kernel Electric-Fence infrastructure x86, kfence: enable KFENCE for x86
mm, kfence: insert KFENCE hooks for SLAB
mm, kfence: insert KFENCE hooks for SLUB kfence, kasan: make KFENCE compatible with KASAN kfence, kmemleak: make KFENCE compatible with KMEMLEAKMarco Elver (4):
arm64, kfence: enable KFENCE for ARM64 kfence, lockdep: make KFENCE compatible with lockdep kfence, Documentation: add KFENCE documentation
kfence: add test suiteDocumentation/dev-tools/index.rst | 1 + Documentation/dev-tools/kfence.rst | 291 +++++++++++ MAINTAINERS | 11 + arch/arm64/Kconfig | 1 + arch/arm64/include/asm/kfence.h | 39 ++ arch/arm64/mm/fault.c | 4 + arch/x86/Kconfig | 2 + arch/x86/include/asm/kfence.h | 60 +++ arch/x86/mm/fault.c | 4 + include/linux/kfence.h | 174 +++++++ init/main.c | 2 + kernel/locking/lockdep.c | 8 + lib/Kconfig.debug | 1 + lib/Kconfig.kfence | 78 +++ mm/Makefile | 1 + mm/kasan/common.c | 7 + mm/kfence/Makefile | 6 + mm/kfence/core.c | 733 +++++++++++++++++++++++++++ mm/kfence/kfence.h | 102 ++++ mm/kfence/kfence_test.c | 777 +++++++++++++++++++++++++++++ mm/kfence/report.c | 219 ++++++++ mm/kmemleak.c | 6 + mm/slab.c | 46 +- mm/slab_common.c | 6 +- mm/slub.c | 72 ++- 25 files changed, 2619 insertions(+), 32 deletions(-) create mode 100644 Documentation/dev-tools/kfence.rst create mode 100644 arch/arm64/include/asm/kfence.h create mode 100644 arch/x86/include/asm/kfence.h create mode 100644 include/linux/kfence.h create mode 100644 lib/Kconfig.kfence create mode 100644 mm/kfence/Makefile create mode 100644 mm/kfence/core.c create mode 100644 mm/kfence/kfence.h create mode 100644 mm/kfence/kfence_test.c create mode 100644 mm/kfence/report.c--
2.28.0.681.g6f77f65b4e-goog