Skip to contentSkip to conversation

slab: Introduce dedicated bucket allocator

PATCH Superseded Kees Cook 3 participants 9 messages Mar 4, 2024

Subsystems (4)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Kees Cook

author
#0 lore raw

cover slab: Introduce dedicated bucket allocator

Hi,

Repeating the commit logs for patch 1 here:

Dedicated caches are available For fixed size allocations via kmem_cache_alloc(), but for dynamically sized allocations there is only the global kmalloc API's set of buckets available. This means it isn't possible to separate specific sets of dynamically sized allocations into a separate collection of caches.

This leads to a use-after-free exploitation weakness in the Linux kernel since many heap memory spraying/grooming attacks depend on using userspace-controllable dynamically sized allocations to collide with fixed size allocations that end up in same cache.

While CONFIG_RANDOM_KMALLOC_CACHES provides a probabilistic defense against these kinds of "type confusion" attacks, including for fixed same-size heap objects, we can create a complementary deterministic defense for dynamically sized allocations.

In order to isolate user-controllable sized allocations from system allocations, introduce kmem_buckets_create() and kmem_buckets_alloc(), which behave like kmem_cache_create() and like kmem_cache_alloc() for confining allocations to a dedicated set of sized caches (which have the same layout as the kmalloc caches).

This can also be used in the future once codetag allocation annotations exist to implement per-caller allocation cache isolation[0] even for dynamic allocations.

After the implemetation are 3 example patch of how this could be used for some repeat "offenders" that get used in exploits. There are more to be isolated beyond just these. Repeating the commit log for patch 2 here:

    The msg subsystem is a common target for exploiting[1][2][3][4][5][6]
    use-after-free type confusion flaws in the kernel for both read and
    write primitives. Avoid having a user-controlled size cache share the
    global kmalloc allocator by using a separate set of kmalloc buckets.

    After a fresh boot under Ubuntu 23.10, we can see the caches are already
    in use:

     # grep ^msg_msg /proc/slabinfo
     msg_msg-8k             0      0   8192    4    8 : ...
     msg_msg-4k            96    128   4096    8    8 : ...
     msg_msg-2k            64     64   2048   16    8 : ...
     msg_msg-1k            64     64   1024   16    4 : ...
     msg_msg-16          1024   1024     16  256    1 : ...
     msg_msg-8              0      0      8  512    1 : ...

    Link: https://blog.hacktivesecurity.com/index.php/2022/06/13/linux-kernel-exploit-development-1day-case-study/ [1]
    Link: https://hardenedvault.net/blog/2022-11-13-msg_msg-recon-mitigation-ved/ [2]
    Link: https://www.willsroot.io/2021/08/corctf-2021-fire-of-salvation-writeup.html [3]
    Link: https://a13xp0p0v.github.io/2021/02/09/CVE-2021-26708.html [4]
    Link: https://google.github.io/security-research/pocs/linux/cve-2021-22555/writeup.html [5]
    Link: https://zplin.me/papers/ELOISE.pdf [6]

-Kees

Kees Cook (4):
slab: Introduce dedicated bucket allocator ipc, msg: Use dedicated slab buckets for alloc_msg()
xattr: Use dedicated slab buckets for setxattr() mm/util: Use dedicated slab buckets for memdup_user()

 fs/xattr.c           | 12 ++++++++-
 include/linux/slab.h | 26 ++++++++++++++++++
 ipc/msgutil.c        | 11 +++++++-
 mm/slab_common.c     | 64 ++++++++++++++++++++++++++++++++++++++++++++
 mm/util.c            | 12 ++++++++-
 5 files changed, 122 insertions(+), 3 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH 1/4 slab: Introduce dedicated bucket allocator

Dedicated caches are available For fixed size allocations via kmem_cache_alloc(), but for dynamically sized allocations there is only the global kmalloc API's set of buckets available. This means it isn't possible to separate specific sets of dynamically sized allocations into a separate collection of caches.

This leads to a use-after-free exploitation weakness in the Linux kernel since many heap memory spraying/grooming attacks depend on using userspace-controllable dynamically sized allocations to collide with fixed size allocations that end up in same cache.

While CONFIG_RANDOM_KMALLOC_CACHES provides a probabilistic defense against these kinds of "type confusion" attacks, including for fixed same-size heap objects, we can create a complementary deterministic defense for dynamically sized allocations.

In order to isolate user-controllable sized allocations from system allocations, introduce kmem_buckets_create() and kmem_buckets_alloc(), which behave like kmem_cache_create() and like kmem_cache_alloc() for confining allocations to a dedicated set of sized caches (which have the same layout as the kmalloc caches).

This can also be used in the future once codetag allocation annotations exist to implement per-caller allocation cache isolation[1] even for dynamic allocations.

---
Cc: Vlastimil Babka <[email protected]>
Cc: Christoph Lameter <[email protected]>
Cc: Pekka Enberg <[email protected]>
Cc: David Rientjes <[email protected]>
Cc: Joonsoo Kim <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: Roman Gushchin <[email protected]>
Cc: Hyeonggon Yoo <[email protected]>
Cc: [email protected]

---
 include/linux/slab.h | 26 ++++++++++++++++++
 mm/slab_common.c     | 64 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 90 insertions(+)
include/linux/slab.h +26 -0
index b5f5ee8308d0..4a4ff84534be 100644
@@ -492,6 +492,16 @@ void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru,
 			   gfp_t gfpflags) __assume_slab_alignment __malloc;
 void kmem_cache_free(struct kmem_cache *s, void *objp);
 
+struct kmem_buckets {
+	struct kmem_cache *caches[ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL])];
+};
+
+struct kmem_buckets *
+kmem_buckets_create(const char *name, unsigned int align, slab_flags_t flags,
+		    unsigned int useroffset, unsigned int usersize,
+		    void (*ctor)(void *));
+
+
 /*
  * Bulk allocation and freeing operations. These are accelerated in an
  * allocator specific way to avoid taking locks repeatedly or building
@@ -594,6 +604,22 @@ static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags)
 	return __kmalloc(size, flags);
 }
 
+static __always_inline __alloc_size(2)
+void *kmem_buckets_alloc(struct kmem_buckets *b, size_t size, gfp_t flags)
+{
+	unsigned int index;
+
+	if (size > KMALLOC_MAX_CACHE_SIZE)
+		return kmalloc_large(size, flags);
+	if (WARN_ON_ONCE(!b))
+		return NULL;
+	index = kmalloc_index(size);
+	if (WARN_ONCE(!b->caches[index],
+		      "missing cache for size %zu (index %d)\n", size, index))
+		return kmalloc(size, flags);
+	return kmalloc_trace(b->caches[index], flags, size);
+}
+
 static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node)
 {
 	if (__builtin_constant_p(size) && size) {
mm/slab_common.c +64 -0

76 lines not on this page · raw message

Signature · 1 line
2.34.1
author
#2 lore raw

PATCH 2/4 ipc, msg: Use dedicated slab buckets for alloc_msg()

In reply to Kees Cook

The msg subsystem is a common target for exploiting[1][2][3][4][5][6] use-after-free type confusion flaws in the kernel for both read and write primitives. Avoid having a user-controlled size cache share the global kmalloc allocator by using a separate set of kmalloc buckets.

After a fresh boot under Ubuntu 23.10, we can see the caches are already in use:

 # grep ^msg_msg /proc/slabinfo
 msg_msg-8k             0      0   8192    4    8 : ...
 msg_msg-4k            96    128   4096    8    8 : ...
 msg_msg-2k            64     64   2048   16    8 : ...
 msg_msg-1k            64     64   1024   16    4 : ...
 msg_msg-16          1024   1024     16  256    1 : ...
 msg_msg-8              0      0      8  512    1 : ...

---

---
 ipc/msgutil.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
ipc/msgutil.c +10 -1
index d0a0e877cadd..36f1aa9ea1cf 100644
@@ -42,6 +42,15 @@ struct msg_msgseg {
 #define DATALEN_MSG	((size_t)PAGE_SIZE-sizeof(struct msg_msg))
 #define DATALEN_SEG	((size_t)PAGE_SIZE-sizeof(struct msg_msgseg))
 
+static struct kmem_buckets *msg_buckets __ro_after_init;
+
+static int __init init_msg_buckets(void)
+{
+	msg_buckets = kmem_buckets_create("msg_msg", 0, SLAB_ACCOUNT, 0, 0, NULL);
+
+	return 0;
+}
+subsys_initcall(init_msg_buckets);
 
 static struct msg_msg *alloc_msg(size_t len)
 {
@@ -50,7 +59,7 @@ static struct msg_msg *alloc_msg(size_t len)
 	size_t alen;
 
 	alen = min(len, DATALEN_MSG);
-	msg = kmalloc(sizeof(*msg) + alen, GFP_KERNEL_ACCOUNT);
+	msg = kmem_buckets_alloc(msg_buckets, sizeof(*msg) + alen, GFP_KERNEL);
 	if (msg == NULL)
 		return NULL;
 
Signature · 1 line
2.34.1
author
#3 lore raw

PATCH 3/4 xattr: Use dedicated slab buckets for setxattr()

In reply to Kees Cook

The setxattr() API can be used for exploiting[1][2][3] use-after-free type confusion flaws in the kernel. Avoid having a user-controlled size cache share the global kmalloc allocator by using a separate set of kmalloc buckets.

---
Cc: Christian Brauner <[email protected]>
Cc: Alexander Viro <[email protected]>
Cc: Jan Kara <[email protected]>
Cc: [email protected]

---
 fs/xattr.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
fs/xattr.c +11 -1
index 09d927603433..2b06316f1d1f 100644
@@ -821,6 +821,16 @@ SYSCALL_DEFINE4(fgetxattr, int, fd, const char __user *, name,
 	return error;
 }
 
+static struct kmem_buckets *xattr_buckets;
+static int __init init_xattr_buckets(void)
+{
+	xattr_buckets = kmem_buckets_create("xattr", 0, 0, 0,
+					    XATTR_LIST_MAX, NULL);
+
+	return 0;
+}
+subsys_initcall(init_xattr_buckets);
+
 /*
  * Extended attribute LIST operations
  */
@@ -833,7 +843,7 @@ listxattr(struct dentry *d, char __user *list, size_t size)
 	if (size) {
 		if (size > XATTR_LIST_MAX)
 			size = XATTR_LIST_MAX;
-		klist = kvmalloc(size, GFP_KERNEL);
+		klist = kmem_buckets_alloc(xattr_buckets, size, GFP_KERNEL);
 		if (!klist)
 			return -ENOMEM;
 	}
Signature · 1 line
2.34.1
#4 lore raw

Re: [PATCH 3/4] xattr: Use dedicated slab buckets for setxattr()

Quoted 40 lines · Kees Cook

The setxattr() API can be used for exploiting[1][2][3] use-after-free type confusion flaws in the kernel. Avoid having a user-controlled size cache share the global kmalloc allocator by using a separate set of kmalloc buckets.

---
Cc: Christian Brauner <[email protected]>
Cc: Alexander Viro <[email protected]>
Cc: Jan Kara <[email protected]>
Cc: [email protected]

---
 fs/xattr.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
fs/xattr.c +10 -0
index 09d927603433..2b06316f1d1f 100644
@@ -821,6 +821,16 @@ SYSCALL_DEFINE4(fgetxattr, int, fd, const char __user *, name,
 	return error;
 }
 
+static struct kmem_buckets *xattr_buckets;
+static int __init init_xattr_buckets(void)
+{
+	xattr_buckets = kmem_buckets_create("xattr", 0, 0, 0,
+					    XATTR_LIST_MAX, NULL);
+
+	return 0;
+}
+subsys_initcall(init_xattr_buckets);
+
 /*
  * Extended attribute LIST operations
  */

fs/xattr.c listxattr(struct dentry *d, char __user *list, size_t size) · line 846

Quoted 5 lines
fs/xattr.c +1 -1
@@ -833,7 +843,7 @@ listxattr(struct dentry *d, char __user *list, size_t size)
 	if (size) {
 		if (size > XATTR_LIST_MAX)
 			size = XATTR_LIST_MAX;
-		klist = kvmalloc(size, GFP_KERNEL);
+		klist = kmem_buckets_alloc(xattr_buckets, size, GFP_KERNEL);

There's a reason this uses kvmalloc() - allocations can be up to 64kB in size and it's not uncommon for large slab allocation to fail on long running machines. hence this needs to fall back to vmalloc() to ensure that large xattrs can always be read.

Essentially, you're trading a heap spraying vector that almost no-one will ever see for a far more frequent -ENOMEM denial of service that will be seen on production systems where large xattrs are used.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
author
#5 lore raw

Re: [PATCH 3/4] xattr: Use dedicated slab buckets for setxattr()

Quoted 43 lines · Dave Chinner
Quoted 42 lines · Kees Cook

The setxattr() API can be used for exploiting[1][2][3] use-after-free type confusion flaws in the kernel. Avoid having a user-controlled size cache share the global kmalloc allocator by using a separate set of kmalloc buckets.

---
Cc: Christian Brauner <[email protected]>
Cc: Alexander Viro <[email protected]>
Cc: Jan Kara <[email protected]>
Cc: [email protected]

---
 fs/xattr.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
fs/xattr.c +10 -0
index 09d927603433..2b06316f1d1f 100644
@@ -821,6 +821,16 @@ SYSCALL_DEFINE4(fgetxattr, int, fd, const char __user *, name,
 	return error;
 }
 
+static struct kmem_buckets *xattr_buckets;
+static int __init init_xattr_buckets(void)
+{
+	xattr_buckets = kmem_buckets_create("xattr", 0, 0, 0,
+					    XATTR_LIST_MAX, NULL);
+
+	return 0;
+}
+subsys_initcall(init_xattr_buckets);
+
 /*
  * Extended attribute LIST operations
  */
@@ -833,7 +843,7 @@ listxattr(struct dentry *d, char __user *list, size_t size)
 	if (size) {
 		if (size > XATTR_LIST_MAX)
Quoted 8 lines

fs/xattr.c listxattr(struct dentry *d, char __user *list, size_t size) · line 846

Quoted 3 lines
fs/xattr.c +1 -1
 			size = XATTR_LIST_MAX;
-		klist = kvmalloc(size, GFP_KERNEL);
+		klist = kmem_buckets_alloc(xattr_buckets, size, GFP_KERNEL);

There's a reason this uses kvmalloc() - allocations can be up to 64kB in size and it's not uncommon for large slab allocation to fail on long running machines. hence this needs to fall back to vmalloc() to ensure that large xattrs can always be read.

I can add a vmalloc fallback interface too. It looked like the larger xattr usage (8k-64k) was less common, but yeah, let's not remove the correct allocation fallback here. I'll fix this for v2.

Thanks!

-Kees

Signature · 1 line
Kees Cook
#6 lore raw

Re: [PATCH 3/4] xattr: Use dedicated slab buckets for setxattr()

In reply to Kees Cook
Quoted 1 line · Kees Cook

xattr: Use dedicated slab buckets for setxattr()

This patch actually changes listxattr(), not setxattr().

getxattr(), setxattr(), and listxattr() all allocate a user controlled size. Perhaps you meant to change all three? What is special about listxattr() (or setxattr() if you actually meant to change that one)?

- Eric

author
#7 lore raw

Re: [PATCH 3/4] xattr: Use dedicated slab buckets for setxattr()

Quoted 8 lines · Eric Biggers
Quoted 1 line · Kees Cook

xattr: Use dedicated slab buckets for setxattr()

This patch actually changes listxattr(), not setxattr().

getxattr(), setxattr(), and listxattr() all allocate a user controlled size. Perhaps you meant to change all three? What is special about listxattr() (or setxattr() if you actually meant to change that one)?

Whoops. Yes, I did one and stopped. :P I'll fix it up in v2.

Signature · 1 line
Kees Cook
author
#8 lore raw

PATCH 4/4 mm/util: Use dedicated slab buckets for memdup_user()

In reply to Kees Cook

The prctl() PR_SET_VMA_ANON_NAME command can be used for exploiting[1] use-after-free type confusion flaws in the kernel. This is just one path to memdup_user() which is designed for contents coming from userspace. Avoid having a user-controlled size cache share the global kmalloc allocator by using a separate set of kmalloc buckets.

After a fresh boot under Ubuntu 23.10, we can see the caches are already in use:

 # grep ^memdup /proc/slabinfo
 memdup_user-8k         4      4   8192    4    8 : ...
 memdup_user-4k         0      0   4096    8    8 : ...
 memdup_user-2k        16     16   2048   16    8 : ...
 memdup_user-1k         0      0   1024   16    4 : ...
 memdup_user-512        0      0    512   16    2 : ...
 memdup_user-256        0      0    256   16    1 : ...
 memdup_user-128        0      0    128   32    1 : ...
 memdup_user-64       256    256     64   64    1 : ...
 memdup_user-32       512    512     32  128    1 : ...
 memdup_user-16      1024   1024     16  256    1 : ...
 memdup_user-8       2048   2048      8  512    1 : ...
 memdup_user-192        0      0    192   21    1 : ...
 memdup_user-96       168    168     96   42    1 : ...

---
Cc: Andrew Morton <[email protected]>
Cc: [email protected]

---
 mm/util.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
mm/util.c +11 -1
index 5a6a9802583b..818e74d11fb6 100644
@@ -181,6 +181,16 @@ char *kmemdup_nul(const char *s, size_t len, gfp_t gfp)
 }
 EXPORT_SYMBOL(kmemdup_nul);
 
+static struct kmem_buckets *user_buckets __ro_after_init;
+
+static int __init init_user_buckets(void)
+{
+	user_buckets = kmem_buckets_create("memdup_user", 0, 0, 0, UINT_MAX, NULL);
+
+	return 0;
+}
+subsys_initcall(init_user_buckets);
+
 /**
  * memdup_user - duplicate memory region from user space
  *
@@ -194,7 +204,7 @@ void *memdup_user(const void __user *src, size_t len)
 {
 	void *p;
 
-	p = kmalloc_track_caller(len, GFP_USER | __GFP_NOWARN);
+	p = kmem_buckets_alloc(user_buckets, len, GFP_USER | __GFP_NOWARN);
 	if (!p)
 		return ERR_PTR(-ENOMEM);
 
Signature · 1 line
2.34.1