Skip to contentSkip to conversation

kasan: support backing vmalloc space with real shadow memory

PATCH Superseded Daniel Axtens 3 participants 16 messages Oct 29, 2019 – Oct 31, 2019

Subsystems (12)

v10 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

v10 cover kasan: support backing vmalloc space with real shadow memory

Currently, vmalloc space is backed by the early shadow page. This means that kasan is incompatible with VMAP_STACK.

This series provides a mechanism to back vmalloc space with real, dynamically allocated memory. I have only wired up x86, because that's the only currently supported arch I can work with easily, but it's very easy to wire up other architectures, and it appears that there is some work-in-progress code to do this on arm64 and s390.

This has been discussed before in the context of VMAP_STACK:
- https://bugzilla.kernel.org/show_bug.cgi?id=202009
- https://lkml.org/lkml/2018/7/22/198
- https://lkml.org/lkml/2019/7/19/822

In terms of implementation details:

Most mappings in vmalloc space are small, requiring less than a full page of shadow space. Allocating a full shadow page per mapping would therefore be wasteful. Furthermore, to ensure that different mappings use different shadow pages, mappings would have to be aligned to KASAN_SHADOW_SCALE_SIZE * PAGE_SIZE.

Instead, share backing space across multiple mappings. Allocate a backing page when a mapping in vmalloc space uses a particular page of the shadow region. This page can be shared by other vmalloc mappings later on.

We hook in to the vmap infrastructure to lazily clean up unused shadow memory.

Daniel Axtens (5):
kasan: support backing vmalloc space with real shadow memory
kasan: add test for vmalloc
fork: support VMAP_STACK with KASAN_VMALLOC x86/kasan: support KASAN_VMALLOC
kasan debug: track pages allocated for vmalloc shadow

 Documentation/dev-tools/kasan.rst |  63 ++++++++
 arch/Kconfig                      |   9 +-
 arch/x86/Kconfig                  |   1 +
 arch/x86/mm/kasan_init_64.c       |  60 +++++++
 include/linux/kasan.h             |  31 ++++
 include/linux/moduleloader.h      |   2 +-
 include/linux/vmalloc.h           |  12 ++
 kernel/fork.c                     |   4 +
 lib/Kconfig.kasan                 |  16 ++
 lib/test_kasan.c                  |  26 +++
 mm/kasan/common.c                 | 254 ++++++++++++++++++++++++++++++
 mm/kasan/generic_report.c         |   3 +
 mm/kasan/kasan.h                  |   1 +
 mm/vmalloc.c                      |  53 ++++++-
 14 files changed, 522 insertions(+), 13 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH v10 1/5 kasan: support backing vmalloc space with real shadow memory

Hook into vmalloc and vmap, and dynamically allocate real shadow memory to back the mappings.

Most mappings in vmalloc space are small, requiring less than a full page of shadow space. Allocating a full shadow page per mapping would therefore be wasteful. Furthermore, to ensure that different mappings use different shadow pages, mappings would have to be aligned to KASAN_SHADOW_SCALE_SIZE * PAGE_SIZE.

Instead, share backing space across multiple mappings. Allocate a backing page when a mapping in vmalloc space uses a particular page of the shadow region. This page can be shared by other vmalloc mappings later on.

We hook in to the vmap infrastructure to lazily clean up unused shadow memory.

To avoid the difficulties around swapping mappings around, this code expects that the part of the shadow region that covers the vmalloc space will not be covered by the early shadow page, but will be left unmapped. This will require changes in arch-specific code.

This allows KASAN with VMAP_STACK, and may be helpful for architectures that do not have a separate module space (e.g. powerpc64, which I am currently working on). It also allows relaxing the module alignment back to PAGE_SIZE.

--

v2: let kasan_unpoison_shadow deal with ranges that do not use a
full shadow byte.

v3: relax module alignment
rename to kasan_populate_vmalloc which is a much better name deal with concurrency correctly

v4: Mark's rework
Poision pages on vfree
Handle allocation failures

v5: Per Christophe Leroy, split out test and dynamically free pages.

v6: Guard freeing page properly. Drop WARN_ON_ONCE(pte_none(*ptep)),
on reflection it's unnecessary debugging cruft with too high a false positive rate.

v7: tlb flush, thanks Mark.
explain more clearly how freeing works and is concurrency-safe.

v9: - Pull in Uladzislau Rezki's changes to better line up with the
design of the new vmalloc implementation. Thanks Vlad.
- clarify comment explaining smp_wmb() per Mark and Andrey's discussion
- tighten up the allocation of backing memory so that it only happens for vmalloc or module space allocations. Thanks Andrey Ryabinin.
- A TLB flush in the freeing path, thanks Mark Rutland.

v10: - rebase on next, pulling in Vlad's new work on splitting the
vmalloc locks. This doesn't require changes in our behaviour but does require rechecking and rewording the explanation of why our behaviour is safe.
- after much discussion of barriers, I now document where I think they are needed and why. Thanks Mark and Andrey.
- clean up some TLB flushing. We were doing it twice - once after each page and once at the end of the whole process. Only do it at the end of the whole depopulate process.
- checkpatch cleanups

---
 Documentation/dev-tools/kasan.rst |  63 ++++++++
 include/linux/kasan.h             |  31 ++++
 include/linux/moduleloader.h      |   2 +-
 include/linux/vmalloc.h           |  12 ++
 lib/Kconfig.kasan                 |  16 +++
 mm/kasan/common.c                 | 231 ++++++++++++++++++++++++++++++
 mm/kasan/generic_report.c         |   3 +
 mm/kasan/kasan.h                  |   1 +
 mm/vmalloc.c                      |  53 +++++--
 9 files changed, 403 insertions(+), 9 deletions(-)
Documentation/dev-tools/kasan.rst +63 -0
index 525296121d89..e4d66e7c50de 100644
@@ -218,3 +218,66 @@ brk handler is used to print bug reports.
 A potential expansion of this mode is a hardware tag-based mode, which would
 use hardware memory tagging support instead of compiler instrumentation and
 manual shadow memory manipulation.
+
+What memory accesses are sanitised by KASAN?
+--------------------------------------------
+
+The kernel maps memory in a number of different parts of the address
+space. This poses something of a problem for KASAN, which requires
+that all addresses accessed by instrumented code have a valid shadow
+region.
+
+The range of kernel virtual addresses is large: there is not enough
+real memory to support a real shadow region for every address that
+could be accessed by the kernel.
+
+By default
+~~~~~~~~~~
+
+By default, architectures only map real memory over the shadow region
+for the linear mapping (and potentially other small areas). For all
+other areas - such as vmalloc and vmemmap space - a single read-only
+page is mapped over the shadow area. This read-only shadow page
+declares all memory accesses as permitted.
+
+This presents a problem for modules: they do not live in the linear
+mapping, but in a dedicated module space. By hooking in to the module
+allocator, KASAN can temporarily map real shadow memory to cover
+them. This allows detection of invalid accesses to module globals, for
+example.
+
+This also creates an incompatibility with ``VMAP_STACK``: if the stack
+lives in vmalloc space, it will be shadowed by the read-only page, and
+the kernel will fault when trying to set up the shadow data for stack
+variables.
+
+CONFIG_KASAN_VMALLOC
+~~~~~~~~~~~~~~~~~~~~
+
+With ``CONFIG_KASAN_VMALLOC``, KASAN can cover vmalloc space at the
+cost of greater memory usage. Currently this is only supported on x86.
+
+This works by hooking into vmalloc and vmap, and dynamically
+allocating real shadow memory to back the mappings.
+
+Most mappings in vmalloc space are small, requiring less than a full
+page of shadow space. Allocating a full shadow page per mapping would
+therefore be wasteful. Furthermore, to ensure that different mappings
+use different shadow pages, mappings would have to be aligned to
+``KASAN_SHADOW_SCALE_SIZE * PAGE_SIZE``.
+
+Instead, we share backing space across multiple mappings. We allocate
+a backing page when a mapping in vmalloc space uses a particular page
+of the shadow region. This page can be shared by other vmalloc
+mappings later on.
+
+We hook in to the vmap infrastructure to lazily clean up unused shadow
+memory.
+
+To avoid the difficulties around swapping mappings around, we expect
+that the part of the shadow region that covers the vmalloc space will
+not be covered by the early shadow page, but will be left
+unmapped. This will require changes in arch-specific code.
+
+This allows ``VMAP_STACK`` support on x86, and can simplify support of
+architectures that do not have a fixed module region.
include/linux/kasan.h +31 -0
index cc8a03cc9674..4f404c565db1 100644
@@ -70,8 +70,18 @@ struct kasan_cache {
 	int free_meta_offset;
 };
 
+/*
+ * These functions provide a special case to support backing module
+ * allocations with real shadow memory. With KASAN vmalloc, the special
+ * case is unnecessary, as the work is handled in the generic case.
+ */
+#ifndef CONFIG_KASAN_VMALLOC
 int kasan_module_alloc(void *addr, size_t size);
 void kasan_free_shadow(const struct vm_struct *vm);
+#else
+static inline int kasan_module_alloc(void *addr, size_t size) { return 0; }
+static inline void kasan_free_shadow(const struct vm_struct *vm) {}
+#endif
 
 int kasan_add_zero_shadow(void *start, unsigned long size);
 void kasan_remove_zero_shadow(void *start, unsigned long size);
@@ -194,4 +204,25 @@ static inline void *kasan_reset_tag(const void *addr)
 
 #endif /* CONFIG_KASAN_SW_TAGS */
 
+#ifdef CONFIG_KASAN_VMALLOC
+int kasan_populate_vmalloc(unsigned long requested_size,
+			   struct vm_struct *area);
+void kasan_poison_vmalloc(void *start, unsigned long size);
+void kasan_release_vmalloc(unsigned long start, unsigned long end,
+			   unsigned long free_region_start,
+			   unsigned long free_region_end);
+#else
+static inline int kasan_populate_vmalloc(unsigned long requested_size,
+					 struct vm_struct *area)
+{
+	return 0;
+}
+
+static inline void kasan_poison_vmalloc(void *start, unsigned long size) {}
+static inline void kasan_release_vmalloc(unsigned long start,
+					 unsigned long end,
+					 unsigned long free_region_start,
+					 unsigned long free_region_end) {}
+#endif
+
 #endif /* LINUX_KASAN_H */
include/linux/moduleloader.h +1 -1
index 5229c18025e9..ca92aea8a6bd 100644
@@ -91,7 +91,7 @@ void module_arch_cleanup(struct module *mod);
 /* Any cleanup before freeing mod->module_init */
 void module_arch_freeing_init(struct module *mod);
 
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN) && !defined(CONFIG_KASAN_VMALLOC)
 #include <linux/kasan.h>
 #define MODULE_ALIGN (PAGE_SIZE << KASAN_SHADOW_SCALE_SHIFT)
 #else
include/linux/vmalloc.h +12 -0
index 4e7809408073..61c43d1a29ca 100644
@@ -22,6 +22,18 @@ struct notifier_block;		/* in notifier.h */
 #define VM_UNINITIALIZED	0x00000020	/* vm_struct is not fully initialized */
 #define VM_NO_GUARD		0x00000040      /* don't add guard page */
 #define VM_KASAN		0x00000080      /* has allocated kasan shadow memory */
+
+/*
+ * VM_KASAN is used slighly differently depending on CONFIG_KASAN_VMALLOC.
+ *
+ * If IS_ENABLED(CONFIG_KASAN_VMALLOC), VM_KASAN is set on a vm_struct after
+ * shadow memory has been mapped. It's used to handle allocation errors so that
+ * we don't try to poision shadow on free if it was never allocated.
+ *
+ * Otherwise, VM_KASAN is set for kasan_module_alloc() allocations and used to
+ * determine which allocations need the module shadow freed.
+ */
+
 /*
  * Memory with VM_FLUSH_RESET_PERMS cannot be freed in an interrupt or with
  * vfree_atomic().
lib/Kconfig.kasan +16 -0
index 6c9682ce0254..81f5464ea9e1 100644
@@ -6,6 +6,9 @@ config HAVE_ARCH_KASAN
 config HAVE_ARCH_KASAN_SW_TAGS
 	bool
 
+config	HAVE_ARCH_KASAN_VMALLOC
+	bool
+
 config CC_HAS_KASAN_GENERIC
 	def_bool $(cc-option, -fsanitize=kernel-address)
 
@@ -142,6 +145,19 @@ config KASAN_SW_TAGS_IDENTIFY
 	  (use-after-free or out-of-bounds) at the cost of increased
 	  memory consumption.
 
+config KASAN_VMALLOC
+	bool "Back mappings in vmalloc space with real shadow memory"
+	depends on KASAN && HAVE_ARCH_KASAN_VMALLOC
+	help
+	  By default, the shadow region for vmalloc space is the read-only
+	  zero page. This means that KASAN cannot detect errors involving
+	  vmalloc space.
+
+	  Enabling this option will hook in to vmap/vmalloc and back those
+	  mappings with real shadow memory allocated on demand. This allows
+	  for KASAN to detect more sorts of errors (and to support vmapped
+	  stacks), but at the cost of higher memory usage.
+
 config TEST_KASAN
 	tristate "Module for testing KASAN for bug detection"
 	depends on m && KASAN
mm/kasan/common.c +231 -0

252 lines not on this page · raw message

mm/kasan/generic_report.c +3 -0
index 36c645939bc9..2d97efd4954f 100644
@@ -86,6 +86,9 @@ static const char *get_shadow_bug_type(struct kasan_access_info *info)
 	case KASAN_ALLOCA_RIGHT:
 		bug_type = "alloca-out-of-bounds";
 		break;
+	case KASAN_VMALLOC_INVALID:
+		bug_type = "vmalloc-out-of-bounds";
+		break;
 	}
 
 	return bug_type;
mm/kasan/kasan.h +1 -0
index 35cff6bbb716..3a083274628e 100644
@@ -25,6 +25,7 @@
 #endif
 
 #define KASAN_GLOBAL_REDZONE    0xFA  /* redzone for global variable */
+#define KASAN_VMALLOC_INVALID   0xF9  /* unallocated space in vmapped page */
 
 /*
  * Stack redzone shadow values
mm/vmalloc.c +45 -8

119 lines not on this page · raw message

Signature · 1 line
2.20.1
Reviewed-by on v10 1/5
#2 lore raw
Quoted 27 lines · Daniel Axtens

Hook into vmalloc and vmap, and dynamically allocate real shadow memory to back the mappings.

Most mappings in vmalloc space are small, requiring less than a full page of shadow space. Allocating a full shadow page per mapping would therefore be wasteful. Furthermore, to ensure that different mappings use different shadow pages, mappings would have to be aligned to KASAN_SHADOW_SCALE_SIZE * PAGE_SIZE.

Instead, share backing space across multiple mappings. Allocate a backing page when a mapping in vmalloc space uses a particular page of the shadow region. This page can be shared by other vmalloc mappings later on.

We hook in to the vmap infrastructure to lazily clean up unused shadow memory.

To avoid the difficulties around swapping mappings around, this code expects that the part of the shadow region that covers the vmalloc space will not be covered by the early shadow page, but will be left unmapped. This will require changes in arch-specific code.

This allows KASAN with VMAP_STACK, and may be helpful for architectures that do not have a separate module space (e.g. powerpc64, which I am currently working on). It also allows relaxing the module alignment back to PAGE_SIZE.

Quoted 5 lines

Small nit bellow, otherwise looks fine:

Quoted 12 lines

static __always_inline bool

mm/vmalloc.c +2 -2
@@ -1196,8 +1201,8 @@ static void free_vmap_area(struct vmap_area *va)
 	 * Insert/Merge it back to the free tree/list.
 	 */
 	spin_lock(&free_vmap_area_lock);
-	merge_or_add_vmap_area(va,
-		&free_vmap_area_root, &free_vmap_area_list);
+	(void)merge_or_add_vmap_area(va, &free_vmap_area_root,
+				     &free_vmap_area_list);
 	spin_unlock(&free_vmap_area_lock);
 }
 

..

mm/vmalloc.c struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, · line 3431

Quoted 8 lines
mm/vmalloc.c +1 -2
@@ -3391,8 +3428,8 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,
 	 * and when pcpu_get_vm_areas() is success.
 	 */
 	while (area--) {
-		merge_or_add_vmap_area(vas[area],
-			&free_vmap_area_root, &free_vmap_area_list);
+		(void)merge_or_add_vmap_area(vas[area], &free_vmap_area_root,

I don't think these (void) casts are necessary.

Quoted 5 lines
mm/vmalloc.c +1 -0
+					     &free_vmap_area_list);
 		vas[area] = NULL;
 	}
 
maintainer
#3 lore raw
In reply to Daniel Axtens

Hello, Daniel

Quoted 11 lines
mm/vmalloc.c +2 -0
@@ -1294,14 +1299,19 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end)
 	spin_lock(&free_vmap_area_lock);
 	llist_for_each_entry_safe(va, n_va, valist, purge_list) {
 		unsigned long nr = (va->va_end - va->va_start) >> PAGE_SHIFT;
+		unsigned long orig_start = va->va_start;
+		unsigned long orig_end = va->va_end;
 
 		/*
 		 * Finally insert or merge lazily-freed area. It is
 		 * detached and there is no need to "unlink" it from

mm/vmalloc.c static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) · line 1315

Quoted 10 lines
mm/vmalloc.c +5 -2
 		 * anything.
 		 */
-		merge_or_add_vmap_area(va,
-			&free_vmap_area_root, &free_vmap_area_list);
+		va = merge_or_add_vmap_area(va, &free_vmap_area_root,
+					    &free_vmap_area_list);
+
+		kasan_release_vmalloc(orig_start, orig_end,
+				      va->va_start, va->va_end);
 

I have some questions here. I have not analyzed kasan_releace_vmalloc() logic in detail, sorry for that if i miss something. __purge_vmap_area_lazy() deals with big address space, so not only vmalloc addresses it frees here, basically it can be any, starting from 1 until ULONG_MAX, whereas vmalloc space spans from VMALLOC_START - VMALLOC_END:

1) Should it be checked that vmalloc only address is freed or you handle it somewhere else?

if (is_vmalloc_addr(va->va_start))

    kasan_release_vmalloc(...)

2) Have you run any bencmarking just to see how much overhead it adds? I am asking, because probably it make sense to add those figures to the backlog(commit message). For example you can run:

<snip>
sudo ./test_vmalloc.sh performance
and
sudo ./test_vmalloc.sh sequential_test_order=1
<snip>

Thanks!

--
Vlad Rezki

#4 lore raw
Quoted 29 lines · Uladzislau Rezki

Hello, Daniel

Quoted 11 lines
mm/vmalloc.c +2 -0
@@ -1294,14 +1299,19 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end)
 	spin_lock(&free_vmap_area_lock);
 	llist_for_each_entry_safe(va, n_va, valist, purge_list) {
 		unsigned long nr = (va->va_end - va->va_start) >> PAGE_SHIFT;
+		unsigned long orig_start = va->va_start;
+		unsigned long orig_end = va->va_end;
 
 		/*
 		 * Finally insert or merge lazily-freed area. It is
 		 * detached and there is no need to "unlink" it from

mm/vmalloc.c static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) · line 1315

Quoted 10 lines
mm/vmalloc.c +5 -2
 		 * anything.
 		 */
-		merge_or_add_vmap_area(va,
-			&free_vmap_area_root, &free_vmap_area_list);
+		va = merge_or_add_vmap_area(va, &free_vmap_area_root,
+					    &free_vmap_area_list);
+
+		kasan_release_vmalloc(orig_start, orig_end,
+				      va->va_start, va->va_end);
 

I have some questions here. I have not analyzed kasan_releace_vmalloc() logic in detail, sorry for that if i miss something. __purge_vmap_area_lazy() deals with big address space, so not only vmalloc addresses it frees here, basically it can be any, starting from 1 until ULONG_MAX, whereas vmalloc space spans from VMALLOC_START - VMALLOC_END:

Quoted 5 lines

1) Should it be checked that vmalloc only address is freed or you handle it somewhere else?

if (is_vmalloc_addr(va->va_start))

    kasan_release_vmalloc(...)

So in kasan_release_vmalloc we only free the region covered by the shadow of orig_start to orig_end, and possibly 1 page to either side. So it will never attempt to free an enormous area. And it will also do nothing if called for a region where there is no shadow backin installed.

Having said that, there should be a test on orig_start, and I've added that in v11 - good catch.

Quoted 9 lines

2) Have you run any bencmarking just to see how much overhead it adds? I am asking, because probably it make sense to add those figures to the backlog(commit message). For example you can run:

<snip>
sudo ./test_vmalloc.sh performance
and
sudo ./test_vmalloc.sh sequential_test_order=1
<snip>

I have now done that:

Testing with test_vmalloc.sh on an x86 VM with 2 vCPUs shows that:

- Turning on KASAN, inline instrumentation, without this feature, introuduces a 4.1x-4.2x slowdown in vmalloc operations.

- Turning this on introduces the following slowdowns over KASAN:
* ~1.76x slower single-threaded (test_vmalloc.sh performance)
* ~2.18x slower when both cpus are performing operations simultaneously (test_vmalloc.sh sequential_test_order=1)

This is unfortunate but given that this is a debug feature only, not the end of the world.

The full figures are:

Performance

No KASAN KASAN original x baseline KASAN vmalloc x baseline x KASAN

fix_size_alloc_test            1697913            14229459       8.38       22981983      13.54       1.62
full_fit_alloc_test            1841601            15152633       8.23       17902922       9.72       1.18
long_busy_list_alloc_test     17874082            58856758       3.29      103925371       5.81       1.77
random_size_alloc_test         9356047            29544085       3.16       57871338       6.19       1.96
fix_align_alloc_test           3188968            19821620       6.22       37979436      11.91       1.92
random_size_align_alloc_te     3033507            17584339       5.80       32588942      10.74       1.85
align_shift_alloc_test             325                1154       3.55           7263      22.35       6.29
pcpu_alloc_test                 231952              278181       1.20         318977       1.38       1.15
Total Cycles              235852824254        985040965542       4.18  1733258779416       7.35       1.76

Sequential, 2 cpus

No KASAN KASAN original x baseline KASAN vmalloc x baseline x KASAN

fix_size_alloc_test            2505806            17989253       7.18       39651038      15.82       2.20
full_fit_alloc_test            3579676            18829862       5.26       21142645       5.91       1.12
long_busy_list_alloc_test     21594983            74766736       3.46      140701363       6.52       1.88
random_size_alloc_test        10884695            34282077       3.15       91945108       8.45       2.68
fix_align_alloc_test           4133226            26304745       6.36       76163270      18.43       2.90
random_size_align_alloc_te     4261175            22927883       5.38       55236058      12.96       2.41
align_shift_alloc_test             948                4827       5.09           4144       4.37       0.86
pcpu_alloc_test                 371789              307654       0.83         374412       1.01       1.22
Total Cycles               99965417402        412710461642       4.13   897968646378       8.98       2.18
fix_size_alloc_test            2502718            17921542       7.16       39893515      15.94       2.23
full_fit_alloc_test            3547996            18675007       5.26       21330495       6.01       1.14
long_busy_list_alloc_test     21522579            74610739       3.47      139822907       6.50       1.87
random_size_alloc_test        10881507            34317349       3.15       91110531       8.37       2.65
fix_align_alloc_test           4119755            26180887       6.35       75818927      18.40       2.90
random_size_align_alloc_te     4297708            23058344       5.37       55969004      13.02       2.43
align_shift_alloc_test             956                5574       5.83           4591       4.80       0.82
pcpu_alloc_test                 306340              347014       1.13         571289       1.86       1.65
Total Cycles               99642832084        412084074628       4.14   896497227762       9.00       2.18

Regards,
Daniel

Quoted 4 lines

Thanks!

--
Vlad Rezki

#5 lore raw

PATCH v10 2/5 kasan: add test for vmalloc

In reply to Daniel Axtens

Test kasan vmalloc support by adding a new test to the module.

--

v5: split out per Christophe Leroy

---
 lib/test_kasan.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
lib/test_kasan.c +26 -0
index 49cc4d570a40..328d33beae36 100644
@@ -19,6 +19,7 @@
 #include <linux/string.h>
 #include <linux/uaccess.h>
 #include <linux/io.h>
+#include <linux/vmalloc.h>
 
 #include <asm/page.h>
 
@@ -748,6 +749,30 @@ static noinline void __init kmalloc_double_kzfree(void)
 	kzfree(ptr);
 }
 
+#ifdef CONFIG_KASAN_VMALLOC
+static noinline void __init vmalloc_oob(void)
+{
+	void *area;
+
+	pr_info("vmalloc out-of-bounds\n");
+
+	/*
+	 * We have to be careful not to hit the guard page.
+	 * The MMU will catch that and crash us.
+	 */
+	area = vmalloc(3000);
+	if (!area) {
+		pr_err("Allocation failed\n");
+		return;
+	}
+
+	((volatile char *)area)[3100];
+	vfree(area);
+}
+#else
+static void __init vmalloc_oob(void) {}
+#endif
+
 static int __init kmalloc_tests_init(void)
 {
 	/*
@@ -793,6 +818,7 @@ static int __init kmalloc_tests_init(void)
 	kasan_strings();
 	kasan_bitops();
 	kmalloc_double_kzfree();
+	vmalloc_oob();
 
 	kasan_restore_multi_shot(multishot);
 
Signature · 1 line
2.20.1
#7 lore raw

PATCH v10 3/5 fork: support VMAP_STACK with KASAN_VMALLOC

In reply to Daniel Axtens

Supporting VMAP_STACK with KASAN_VMALLOC is straightforward:

- clear the shadow region of vmapped stacks when swapping them in
- tweak Kconfig to allow VMAP_STACK to be turned on with KASAN

---
 arch/Kconfig  | 9 +++++----
 kernel/fork.c | 4 ++++
 2 files changed, 9 insertions(+), 4 deletions(-)
arch/Kconfig +5 -4
index 5f8a5d84dbbe..2d914990402f 100644
@@ -843,16 +843,17 @@ config HAVE_ARCH_VMAP_STACK
 config VMAP_STACK
 	default y
 	bool "Use a virtually-mapped stack"
-	depends on HAVE_ARCH_VMAP_STACK && !KASAN
+	depends on HAVE_ARCH_VMAP_STACK
+	depends on !KASAN || KASAN_VMALLOC
 	---help---
 	  Enable this if you want the use virtually-mapped kernel stacks
 	  with guard pages.  This causes kernel stack overflows to be
 	  caught immediately rather than causing difficult-to-diagnose
 	  corruption.
 
-	  This is presently incompatible with KASAN because KASAN expects
-	  the stack to map directly to the KASAN shadow map using a formula
-	  that is incorrect if the stack is in vmalloc space.
+	  To use this with KASAN, the architecture must support backing
+	  virtual mappings with real shadow memory, and KASAN_VMALLOC must
+	  be enabled.
 
 config ARCH_OPTIONAL_KERNEL_RWX
 	def_bool n
kernel/fork.c +4 -0
index 954e875e72b1..a6e5249ad74b 100644
@@ -94,6 +94,7 @@
 #include <linux/livepatch.h>
 #include <linux/thread_info.h>
 #include <linux/stackleak.h>
+#include <linux/kasan.h>
 
 #include <asm/pgtable.h>
 #include <asm/pgalloc.h>
@@ -224,6 +225,9 @@ static unsigned long *alloc_thread_stack_node(struct task_struct *tsk, int node)
 		if (!s)
 			continue;
 
+		/* Clear the KASAN shadow of the stack. */
+		kasan_unpoison_shadow(s->addr, THREAD_SIZE);
+
 		/* Clear stale pointers from reused stack. */
 		memset(s->addr, 0, THREAD_SIZE);
 
Signature · 1 line
2.20.1
Reviewed-by on v10 3/5
#8 lore raw

Re: [PATCH v10 3/5] fork: support VMAP_STACK with KASAN_VMALLOC

Quoted 8 lines · Daniel Axtens

Supporting VMAP_STACK with KASAN_VMALLOC is straightforward:

- clear the shadow region of vmapped stacks when swapping them in
- tweak Kconfig to allow VMAP_STACK to be turned on with KASAN

---

Quoted 14 lines
kernel/fork.c +1 -0
index 954e875e72b1..a6e5249ad74b 100644
@@ -94,6 +94,7 @@
 #include <linux/livepatch.h>
 #include <linux/thread_info.h>
 #include <linux/stackleak.h>
+#include <linux/kasan.h>
 
 #include <asm/pgtable.h>
 #include <asm/pgalloc.h>

kernel/fork.c static unsigned long *alloc_thread_stack_node(struct task_struct *tsk, int node) · line 230

Quoted 6 lines
kernel/fork.c +3 -0
@@ -224,6 +225,9 @@ static unsigned long *alloc_thread_stack_node(struct task_struct *tsk, int node)
 		if (!s)
 			continue;
 
+		/* Clear the KASAN shadow of the stack. */
+		kasan_unpoison_shadow(s->addr, THREAD_SIZE);
+

Just sharing the thought. We could possibly add poisoning in free_thread_stack() to catch possible usage of freed cached stack. But it might be a bad idea because cached stacks supposed to be reused very quickly. So it might just add overhead without much gain.

Quoted 4 lines
kernel/fork.c +0 -0
 		/* Clear stale pointers from reused stack. */
 		memset(s->addr, 0, THREAD_SIZE);
 
#9 lore raw

PATCH v10 4/5 x86/kasan: support KASAN_VMALLOC

In reply to Daniel Axtens

In the case where KASAN directly allocates memory to back vmalloc space, don't map the early shadow page over it.

We prepopulate pgds/p4ds for the range that would otherwise be empty. This is required to get it synced to hardware on boot, allowing the lower levels of the page tables to be filled dynamically.

---
v5: fix some checkpatch CHECK warnings. There are some that remain
around lines ending with '(': I have not changed these because it's consistent with the rest of the file and it's not easy to see how to fix it without creating an overlong line or lots of temporary variables.

v2: move from faulting in shadow pgds to prepopulating

---
 arch/x86/Kconfig            |  1 +
 arch/x86/mm/kasan_init_64.c | 60 +++++++++++++++++++++++++++++++++++++
 2 files changed, 61 insertions(+)
arch/x86/Kconfig +1 -0
index 45699e458057..d65b0fcc9bc0 100644
@@ -135,6 +135,7 @@ config X86
 	select HAVE_ARCH_JUMP_LABEL
 	select HAVE_ARCH_JUMP_LABEL_RELATIVE
 	select HAVE_ARCH_KASAN			if X86_64
+	select HAVE_ARCH_KASAN_VMALLOC		if X86_64
 	select HAVE_ARCH_KGDB
 	select HAVE_ARCH_MMAP_RND_BITS		if MMU
 	select HAVE_ARCH_MMAP_RND_COMPAT_BITS	if MMU && COMPAT
arch/x86/mm/kasan_init_64.c +60 -0

75 lines not on this page · raw message

Signature · 1 line
2.20.1
Reviewed-by on v10 4/5
#10 lore raw

Re: [PATCH v10 4/5] x86/kasan: support KASAN_VMALLOC

Quoted 12 lines · Daniel Axtens

In the case where KASAN directly allocates memory to back vmalloc space, don't map the early shadow page over it.

We prepopulate pgds/p4ds for the range that would otherwise be empty. This is required to get it synced to hardware on boot, allowing the lower levels of the page tables to be filled dynamically.

---

arch/x86/mm/kasan_init_64.c static void __init kasan_map_early_shadow(pgd_t *pgd) · line 273

Quoted 6 lines
arch/x86/mm/kasan_init_64.c +6 -0
+static void __init kasan_shallow_populate_pgds(void *start, void *end)
+{
+	unsigned long addr, next;
+	pgd_t *pgd;
+	void *p;
+	int nid = early_pfn_to_nid((unsigned long)start);

This doesn't make sense. start is not even a pfn. With linear mapping we try to identify nid to have the shadow on the same node as memory. But in this case we don't have memory or the corresponding shadow (yet), we only install pgd/p4d.
I guess we could just use NUMA_NO_NODE.

The rest looks ok, so with that fixed:

#11 lore raw

Re: [PATCH v10 4/5] x86/kasan: support KASAN_VMALLOC

Quoted 13 lines · Andrey Ryabinin
Quoted 11 lines · Daniel Axtens

In the case where KASAN directly allocates memory to back vmalloc space, don't map the early shadow page over it.

We prepopulate pgds/p4ds for the range that would otherwise be empty. This is required to get it synced to hardware on boot, allowing the lower levels of the page tables to be filled dynamically.

---

Quoted 12 lines

arch/x86/mm/kasan_init_64.c static void __init kasan_map_early_shadow(pgd_t *pgd) · line 273

Quoted 6 lines
arch/x86/mm/kasan_init_64.c +6 -0
+static void __init kasan_shallow_populate_pgds(void *start, void *end)
+{
+	unsigned long addr, next;
+	pgd_t *pgd;
+	void *p;
+	int nid = early_pfn_to_nid((unsigned long)start);

This doesn't make sense. start is not even a pfn. With linear mapping we try to identify nid to have the shadow on the same node as memory. But in this case we don't have memory or the corresponding shadow (yet), we only install pgd/p4d.
I guess we could just use NUMA_NO_NODE.

Ah wow, that's quite the clanger on my part.

There are a couple of other invocations of early_pfn_to_nid in that file that use an address directly, but at least they reference actual memory. I'll send a separate patch to fix those up.

Quoted 3 lines

The rest looks ok, so with that fixed:

Thanks heaps! I've fixed up the nit you identifed in the first patch, and I agree that the last patch probably isn't needed. I'll respin the series shortly.

Regards,
Daniel

#12 lore raw

Re: [PATCH v10 4/5] x86/kasan: support KASAN_VMALLOC

Quoted 28 lines · Daniel Axtens
Quoted 13 lines · Andrey Ryabinin
Quoted 11 lines · Daniel Axtens

In the case where KASAN directly allocates memory to back vmalloc space, don't map the early shadow page over it.

We prepopulate pgds/p4ds for the range that would otherwise be empty. This is required to get it synced to hardware on boot, allowing the lower levels of the page tables to be filled dynamically.

---

Quoted 12 lines

arch/x86/mm/kasan_init_64.c static void __init kasan_map_early_shadow(pgd_t *pgd) · line 273

Quoted 6 lines
arch/x86/mm/kasan_init_64.c +6 -0
+static void __init kasan_shallow_populate_pgds(void *start, void *end)
+{
+	unsigned long addr, next;
+	pgd_t *pgd;
+	void *p;
+	int nid = early_pfn_to_nid((unsigned long)start);

This doesn't make sense. start is not even a pfn. With linear mapping we try to identify nid to have the shadow on the same node as memory. But in this case we don't have memory or the corresponding shadow (yet), we only install pgd/p4d.
I guess we could just use NUMA_NO_NODE.

Quoted 5 lines

Ah wow, that's quite the clanger on my part.

There are a couple of other invocations of early_pfn_to_nid in that file that use an address directly, but at least they reference actual memory. I'll send a separate patch to fix those up.

I see only one incorrect, in kasan_init(): early_pfn_to_nid(__pa(_stext)) It should be wrapped with PFN_DOWN().
Other usages in map_range() seems to be correct, range->start,end is pfns.

Quoted 9 lines
Quoted 4 lines

The rest looks ok, so with that fixed:

Thanks heaps! I've fixed up the nit you identifed in the first patch, and I agree that the last patch probably isn't needed. I'll respin the series shortly.

Hold on a sec, just spotted another thing to fix.

arch/x86/mm/kasan_init_64.c void __init kasan_init(void) · line 407

Quoted 12 lines
arch/x86/mm/kasan_init_64.c +8 -0
@@ -352,9 +397,24 @@ void __init kasan_init(void)
 	shadow_cpu_entry_end = (void *)round_up(
 			(unsigned long)shadow_cpu_entry_end, PAGE_SIZE);
 
+	/*
+	 * If we're in full vmalloc mode, don't back vmalloc space with early
+	 * shadow pages. Instead, prepopulate pgds/p4ds so they are synced to
+	 * the global table and we can populate the lower levels on demand.
+	 */
+#ifdef CONFIG_KASAN_VMALLOC
+	kasan_shallow_populate_pgds(
+		kasan_mem_to_shadow((void *)PAGE_OFFSET + MAXMEM),

This should be VMALLOC_START, there is no point to allocate pgds for the hole between linear mapping and vmalloc, just waste of memory. It make sense to map early shadow for that hole, because if code dereferences address in that hole we will see the page fault on that address instead of fault on the shadow.

So something like this might work:

	kasan_populate_early_shadow(
		kasan_mem_to_shadow((void *)PAGE_OFFSET + MAXMEM),
		kasan_mem_to_shadow((void *)VMALLOC_START));

	if (IS_ENABLED(CONFIG_KASAN_VMALLOC)
		kasan_shallow_populate_pgds(kasan_mem_to_shadow(VMALLOC_START), kasan_mem_to_shadow((void *)VMALLOC_END))
	else
		kasan_populate_early_shadow(kasan_mem_to_shadow(VMALLOC_START), kasan_mem_to_shadow((void *)VMALLOC_END));

	kasan_populate_early_shadow(
		kasan_mem_to_shadow((void *)VMALLOC_END + 1),
		shadow_cpu_entry_begin);
#13 lore raw

Re: [PATCH v10 4/5] x86/kasan: support KASAN_VMALLOC

Quoted 31 lines · Andrey Ryabinin
Quoted 30 lines · Daniel Axtens
Quoted 13 lines · Andrey Ryabinin
Quoted 11 lines · Daniel Axtens

In the case where KASAN directly allocates memory to back vmalloc space, don't map the early shadow page over it.

We prepopulate pgds/p4ds for the range that would otherwise be empty. This is required to get it synced to hardware on boot, allowing the lower levels of the page tables to be filled dynamically.

---

Quoted 12 lines

arch/x86/mm/kasan_init_64.c static void __init kasan_map_early_shadow(pgd_t *pgd) · line 273

Quoted 6 lines
arch/x86/mm/kasan_init_64.c +6 -0
+static void __init kasan_shallow_populate_pgds(void *start, void *end)
+{
+	unsigned long addr, next;
+	pgd_t *pgd;
+	void *p;
+	int nid = early_pfn_to_nid((unsigned long)start);

This doesn't make sense. start is not even a pfn. With linear mapping we try to identify nid to have the shadow on the same node as memory. But in this case we don't have memory or the corresponding shadow (yet), we only install pgd/p4d.
I guess we could just use NUMA_NO_NODE.

Ah wow, that's quite the clanger on my part.

Quoted 8 lines
Quoted 3 lines

There are a couple of other invocations of early_pfn_to_nid in that file that use an address directly, but at least they reference actual memory. I'll send a separate patch to fix those up.

I see only one incorrect, in kasan_init(): early_pfn_to_nid(__pa(_stext)) It should be wrapped with PFN_DOWN().
Other usages in map_range() seems to be correct, range->start,end is pfns.

Oh, right, I didn't realise map_range was already using pfns.

Quoted 32 lines
Quoted 10 lines
Quoted 5 lines

The rest looks ok, so with that fixed:

Thanks heaps! I've fixed up the nit you identifed in the first patch, and I agree that the last patch probably isn't needed. I'll respin the series shortly.

Hold on a sec, just spotted another thing to fix.

arch/x86/mm/kasan_init_64.c void __init kasan_init(void) · line 407

Quoted 12 lines
arch/x86/mm/kasan_init_64.c +8 -0
@@ -352,9 +397,24 @@ void __init kasan_init(void)
 	shadow_cpu_entry_end = (void *)round_up(
 			(unsigned long)shadow_cpu_entry_end, PAGE_SIZE);
 
+	/*
+	 * If we're in full vmalloc mode, don't back vmalloc space with early
+	 * shadow pages. Instead, prepopulate pgds/p4ds so they are synced to
+	 * the global table and we can populate the lower levels on demand.
+	 */
+#ifdef CONFIG_KASAN_VMALLOC
+	kasan_shallow_populate_pgds(
+		kasan_mem_to_shadow((void *)PAGE_OFFSET + MAXMEM),

This should be VMALLOC_START, there is no point to allocate pgds for the hole between linear mapping and vmalloc, just waste of memory. It make sense to map early shadow for that hole, because if code dereferences address in that hole we will see the page fault on that address instead of fault on the shadow.

So something like this might work:

Quoted 12 lines
	kasan_populate_early_shadow(
		kasan_mem_to_shadow((void *)PAGE_OFFSET + MAXMEM),
		kasan_mem_to_shadow((void *)VMALLOC_START));

	if (IS_ENABLED(CONFIG_KASAN_VMALLOC)
		kasan_shallow_populate_pgds(kasan_mem_to_shadow(VMALLOC_START), kasan_mem_to_shadow((void *)VMALLOC_END))
	else
		kasan_populate_early_shadow(kasan_mem_to_shadow(VMALLOC_START), kasan_mem_to_shadow((void *)VMALLOC_END));

	kasan_populate_early_shadow(
		kasan_mem_to_shadow((void *)VMALLOC_END + 1),
		shadow_cpu_entry_begin);

Sounds good. It's getting late for me so I'll change and test that and send a respin tomorrow my time.

Regards,
Daniel

#14 lore raw

PATCH v10 5/5 kasan debug: track pages allocated for vmalloc shadow

In reply to Daniel Axtens

Provide the current number of vmalloc shadow pages in /sys/kernel/debug/kasan/vmalloc_shadow_pages.

---

v10: rebase on linux-next/master.

v8: rename kasan_vmalloc/shadow_pages -> kasan/vmalloc_shadow_pages

On v4 (no dynamic freeing), I saw the following approximate figures on my test VM:

- fresh boot: 720
- after test_vmalloc: ~14000

With v5 (lazy dynamic freeing):

- boot: ~490-500
- running modprobe test_vmalloc pushes the figures up to sometimes as high as ~14000, but they drop down to ~560 after the test ends. I'm not sure where the extra sixty pages are from, but running the test repeately doesn't cause the number to keep growing, so I don't think we're leaking.
- with vmap_stack, spawning tasks pushes the figure up to ~4200, then some clearing kicks in and drops it down to previous levels again.

---
 mm/kasan/common.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
mm/kasan/common.c +23 -0
index 6e7bc5d3fa83..a4b5c64da16f 100644
@@ -35,6 +35,7 @@
 #include <linux/vmalloc.h>
 #include <linux/bug.h>
 #include <linux/uaccess.h>
+#include <linux/debugfs.h>
 
 #include <asm/tlbflush.h>
 
@@ -750,6 +751,8 @@ core_initcall(kasan_memhotplug_init);
 #endif
 
 #ifdef CONFIG_KASAN_VMALLOC
+static u64 vmalloc_shadow_pages;
+
 static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr,
 				      void *unused)
 {
@@ -770,6 +773,7 @@ static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr,
 	if (likely(pte_none(*ptep))) {
 		set_pte_at(&init_mm, addr, ptep, pte);
 		page = 0;
+		vmalloc_shadow_pages++;
 	}
 	spin_unlock(&init_mm.page_table_lock);
 	if (page)
@@ -858,6 +862,7 @@ static int kasan_depopulate_vmalloc_pte(pte_t *ptep, unsigned long addr,
 	if (likely(!pte_none(*ptep))) {
 		pte_clear(&init_mm, addr, ptep);
 		free_page(page);
+		vmalloc_shadow_pages--;
 	}
 	spin_unlock(&init_mm.page_table_lock);
 
@@ -974,4 +979,22 @@ void kasan_release_vmalloc(unsigned long start, unsigned long end,
 				       (unsigned long)shadow_end);
 	}
 }
+
+static __init int kasan_init_debugfs(void)
+{
+	struct dentry *root;
+
+	root = debugfs_create_dir("kasan", NULL);
+	if (IS_ERR(root)) {
+		if (PTR_ERR(root) == -ENODEV)
+			return 0;
+		return PTR_ERR(root);
+	}
+
+	debugfs_create_u64("vmalloc_shadow_pages", 0444, root,
+			   &vmalloc_shadow_pages);
+
+	return 0;
+}
+late_initcall(kasan_init_debugfs);
 #endif
Signature · 1 line
2.20.1
#15 lore raw

Re: [PATCH v10 5/5] kasan debug: track pages allocated for vmalloc shadow

Quoted 3 lines · Daniel Axtens

Provide the current number of vmalloc shadow pages in /sys/kernel/debug/kasan/vmalloc_shadow_pages.

I wouldn't merge this. I don't see use-case for this, besides testing this patch set. And I think that number should be possible to extract via page_owner mechanism.