Skip to contentSkip to conversation

mm: page_alloc: fix ref bias in page_frag_alloc() for 1-byte allocs

PATCH Applied Jann Horn 3 participants 7 messages Feb 13, 2019 – Feb 14, 2019

Subsystems (4)

v1 of 2 Applied 1/1 Latest is v2 →

lore
Share & tools
lore

Jann Horn

author
#0 lore raw

PATCH 1/1 mm: page_alloc: fix ref bias in page_frag_alloc() for 1-byte allocs

The basic idea behind ->pagecnt_bias is: If we pre-allocate the maximum number of references that we might need to create in the fastpath later, the bump-allocation fastpath only has to modify the non-atomic bias value that tracks the number of extra references we hold instead of the atomic refcount. The maximum number of allocations we can serve (under the assumption that no allocation is made with size 0) is nc->size, so that's the bias used.

However, even when all memory in the allocation has been given away, a reference to the page is still held; and in the offset < 0 slowpath, the page may be reused if everyone else has dropped their references. This means that the necessary number of references is actually nc->size+1.

Luckily, from a quick grep, it looks like the only path that can call page_frag_alloc(fragsz=1) is TAP with the IFF_NAPI_FRAGS flag, which requires CAP_NET_ADMIN in the init namespace and is only intended to be used for kernel testing and fuzzing.

To test for this issue, put a WARN_ON(page_ref_count(page) == 0) in the offset < 0 path, below the virt_to_page() call, and then repeatedly call writev() on a TAP device with IFF_TAP|IFF_NO_PI|IFF_NAPI_FRAGS|IFF_NAPI, with a vector consisting of 15 elements containing 1 byte each.

---
 mm/page_alloc.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
mm/page_alloc.c +4 -4
index 35fdde041f5c..46285d28e43b 100644
@@ -4675,11 +4675,11 @@ void *page_frag_alloc(struct page_frag_cache *nc,
 		/* Even if we own the page, we do not use atomic_set().
 		 * This would break get_page_unless_zero() users.
 		 */
-		page_ref_add(page, size - 1);
+		page_ref_add(page, size);
 
 		/* reset page count bias and offset to start of new frag */
 		nc->pfmemalloc = page_is_pfmemalloc(page);
-		nc->pagecnt_bias = size;
+		nc->pagecnt_bias = size + 1;
 		nc->offset = size;
 	}
 
@@ -4695,10 +4695,10 @@ void *page_frag_alloc(struct page_frag_cache *nc,
 		size = nc->size;
 #endif
 		/* OK, page count is 0, we can safely set it */
-		set_page_count(page, size);
+		set_page_count(page, size + 1);
 
 		/* reset page count bias and offset to start of new frag */
-		nc->pagecnt_bias = size;
+		nc->pagecnt_bias = size + 1;
 		offset = size - fragsz;
 	}
 
Signature · 1 line
2.20.1.791.gb4d0f1c61a-goog
maintainer
#1 lore raw
Quoted 14 lines · On Wed, 13 Feb 2019 21:41:57 +0100 Jann Horn <[email protected]> wrote:

The basic idea behind ->pagecnt_bias is: If we pre-allocate the maximum number of references that we might need to create in the fastpath later, the bump-allocation fastpath only has to modify the non-atomic bias value that tracks the number of extra references we hold instead of the atomic refcount. The maximum number of allocations we can serve (under the assumption that no allocation is made with size 0) is nc->size, so that's the bias used.

However, even when all memory in the allocation has been given away, a reference to the page is still held; and in the offset < 0 slowpath, the page may be reused if everyone else has dropped their references. This means that the necessary number of references is actually nc->size+1.

Quoted 4 lines

Luckily, from a quick grep, it looks like the only path that can call page_frag_alloc(fragsz=1) is TAP with the IFF_NAPI_FRAGS flag, which requires CAP_NET_ADMIN in the init namespace and is only intended to be used for kernel testing and fuzzing.

For the net-naive, what is TAP? It doesn't appear to mean drivers/net/tap.c.

Quoted 25 lines

To test for this issue, put a WARN_ON(page_ref_count(page) == 0) in the offset < 0 path, below the virt_to_page() call, and then repeatedly call writev() on a TAP device with IFF_TAP|IFF_NO_PI|IFF_NAPI_FRAGS|IFF_NAPI, with a vector consisting of 15 elements containing 1 byte each.

...

mm/page_alloc.c +2 -2
@@ -4675,11 +4675,11 @@ void *page_frag_alloc(struct page_frag_cache *nc,
 		/* Even if we own the page, we do not use atomic_set().
 		 * This would break get_page_unless_zero() users.
 		 */
-		page_ref_add(page, size - 1);
+		page_ref_add(page, size);
 
 		/* reset page count bias and offset to start of new frag */
 		nc->pfmemalloc = page_is_pfmemalloc(page);
-		nc->pagecnt_bias = size;
+		nc->pagecnt_bias = size + 1;
 		nc->offset = size;
 	}
 
@@ -4695,10 +4695,10 @@ void *page_frag_alloc(struct page_frag_cache *nc,
 		size = nc->size;

mm/page_alloc.c void *page_frag_alloc(struct page_frag_cache *nc, · line 4703

Quoted 10 lines
mm/page_alloc.c +2 -2
 #endif
 		/* OK, page count is 0, we can safely set it */
-		set_page_count(page, size);
+		set_page_count(page, size + 1);
 
 		/* reset page count bias and offset to start of new frag */
-		nc->pagecnt_bias = size;
+		nc->pagecnt_bias = size + 1;
 		offset = size - fragsz;
 	}

This is probably more a davem patch than a -mm one.

author
#2 lore raw
Quoted 17 lines · Andrew Morton
Quoted 14 lines · On Wed, 13 Feb 2019 21:41:57 +0100 Jann Horn <[email protected]> wrote:

The basic idea behind ->pagecnt_bias is: If we pre-allocate the maximum number of references that we might need to create in the fastpath later, the bump-allocation fastpath only has to modify the non-atomic bias value that tracks the number of extra references we hold instead of the atomic refcount. The maximum number of allocations we can serve (under the assumption that no allocation is made with size 0) is nc->size, so that's the bias used.

However, even when all memory in the allocation has been given away, a reference to the page is still held; and in the offset < 0 slowpath, the page may be reused if everyone else has dropped their references. This means that the necessary number of references is actually nc->size+1.

Quoted 7 lines
Quoted 4 lines

Luckily, from a quick grep, it looks like the only path that can call page_frag_alloc(fragsz=1) is TAP with the IFF_NAPI_FRAGS flag, which requires CAP_NET_ADMIN in the init namespace and is only intended to be used for kernel testing and fuzzing.

For the net-naive, what is TAP? It doesn't appear to mean drivers/net/tap.c.

It's implemented in drivers/net/tun.c; the combined functionality implemented in there is called TUN/TAP. TUN refers to providing raw IP packets to the kernel, TAP refers to providing raw ethernet packets. It's documented in Documentation/networking/tuntap.txt. The code that's interesting here is tun_get_user(), which calls into tun_napi_alloc_frags() if tun_napi_frags_enabled(tfile) is true, which in turn calls into netdev_alloc_frag(), which ends up in page_frag_alloc(). This is how you can use it (except that if you were using it legitimately, you'd be writing an ethernet header, a layer 3 header, and application data instead of writing "aaaaaaaaaaaaaaa" like me):

================
#define _GNU_SOURCE
#include <stdlib.h>
#include <stdarg.h>
#include <net/if.h>
#include <linux/if.h>
#include <linux/if_tun.h>
#include <err.h>
#include <sys/types.h>
#include <fcntl.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/ioctl.h>

void systemf(const char *command, ...) {
char *full_command;
va_list ap;
va_start(ap, command);
if (vasprintf(&full_command, command, ap) == -1)

    err(1, "vasprintf");

va_end(ap);
printf("systemf: <<<%s>>>\n", full_command);
system(full_command);
}

char *devname;

int tun_alloc(char *name) {
int fd = open("/dev/net/tun", O_RDWR);
if (fd == -1)

    err(1, "open tun dev");

static struct ifreq req = { .ifr_flags =
IFF_TAP|IFF_NO_PI|IFF_NAPI_FRAGS|IFF_NAPI };
strcpy(req.ifr_name, name);
if (ioctl(fd, TUNSETIFF, &req))

    err(1, "TUNSETIFF");

devname = req.ifr_name;
printf("device name: %s\n", devname);
return fd;
}

int main(void) {
int tun_fd = tun_alloc("inject_dev%d");
systemf("ip link set %s up", devname);

  while (1) {
    struct iovec iov[15];
    for (int i=0; i<sizeof(iov)/sizeof(iov[0]); i++) {
      iov[i].iov_base = "a";
      iov[i].iov_len = 1;
    }
    writev(tun_fd, iov, sizeof(iov)/sizeof(iov[0]));

}
}
================

Quoted 30 lines
Quoted 30 lines

To test for this issue, put a WARN_ON(page_ref_count(page) == 0) in the offset < 0 path, below the virt_to_page() call, and then repeatedly call writev() on a TAP device with IFF_TAP|IFF_NO_PI|IFF_NAPI_FRAGS|IFF_NAPI, with a vector consisting of 15 elements containing 1 byte each.

...

mm/page_alloc.c +3 -3
@@ -4675,11 +4675,11 @@ void *page_frag_alloc(struct page_frag_cache *nc,
              /* Even if we own the page, we do not use atomic_set().
               * This would break get_page_unless_zero() users.
               */
-             page_ref_add(page, size - 1);
+             page_ref_add(page, size);
 
              /* reset page count bias and offset to start of new frag */
              nc->pfmemalloc = page_is_pfmemalloc(page);
-             nc->pagecnt_bias = size;
+             nc->pagecnt_bias = size + 1;
              nc->offset = size;
      }
 
@@ -4695,10 +4695,10 @@ void *page_frag_alloc(struct page_frag_cache *nc,
              size = nc->size;
 #endif
              /* OK, page count is 0, we can safely set it */
-             set_page_count(page, size);
+             set_page_count(page, size + 1);
 
Quoted 7 lines

mm/page_alloc.c void *page_frag_alloc(struct page_frag_cache *nc, · line 4703

Quoted 5 lines
mm/page_alloc.c +1 -1
              /* reset page count bias and offset to start of new frag */
-             nc->pagecnt_bias = size;
+             nc->pagecnt_bias = size + 1;
              offset = size - fragsz;
      }

This is probably more a davem patch than a -mm one.

Ah, sorry. I assumed that I just should go by which directory the patched code is in.

You did just add it to the -mm tree though, right? So I shouldn't resend it to davem?

maintainer
#3 lore raw
Quoted 7 lines · On Wed, 13 Feb 2019 22:11:58 +0100 Jann Horn <[email protected]> wrote:
Quoted 1 line

This is probably more a davem patch than a -mm one.

Ah, sorry. I assumed that I just should go by which directory the patched code is in.

You did just add it to the -mm tree though, right? So I shouldn't resend it to davem?

Yes, please send to Dave. I'll autodrop the -mm copy if/when it turns up in -next.

#4 lore raw
In reply to Jann Horn
Quoted 15 lines · Jann Horn

The basic idea behind ->pagecnt_bias is: If we pre-allocate the maximum number of references that we might need to create in the fastpath later, the bump-allocation fastpath only has to modify the non-atomic bias value that tracks the number of extra references we hold instead of the atomic refcount. The maximum number of allocations we can serve (under the assumption that no allocation is made with size 0) is nc->size, so that's the bias used.

However, even when all memory in the allocation has been given away, a reference to the page is still held; and in the offset < 0 slowpath, the page may be reused if everyone else has dropped their references. This means that the necessary number of references is actually nc->size+1.

Quoted 4 lines

Luckily, from a quick grep, it looks like the only path that can call page_frag_alloc(fragsz=1) is TAP with the IFF_NAPI_FRAGS flag, which requires CAP_NET_ADMIN in the init namespace and is only intended to be used for kernel testing and fuzzing.

Actually that has me somewhat concerned. I wouldn't be surprised if most drivers expect the netdev_alloc_frags call to at least output an SKB_DATA_ALIGN sized value.

We probably should update __netdev_alloc_frag and __napi_alloc_frag so that they will pass fragsz through SKB_DATA_ALIGN.

Quoted 31 lines

To test for this issue, put a WARN_ON(page_ref_count(page) == 0) in the offset < 0 path, below the virt_to_page() call, and then repeatedly call writev() on a TAP device with IFF_TAP|IFF_NO_PI|IFF_NAPI_FRAGS|IFF_NAPI, with a vector consisting of 15 elements containing 1 byte each.

---
 mm/page_alloc.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
mm/page_alloc.c +2 -2
index 35fdde041f5c..46285d28e43b 100644
@@ -4675,11 +4675,11 @@ void *page_frag_alloc(struct page_frag_cache *nc,
                /* Even if we own the page, we do not use atomic_set().
                 * This would break get_page_unless_zero() users.
                 */
-               page_ref_add(page, size - 1);
+               page_ref_add(page, size);
 
                /* reset page count bias and offset to start of new frag */
                nc->pfmemalloc = page_is_pfmemalloc(page);
-               nc->pagecnt_bias = size;
+               nc->pagecnt_bias = size + 1;
                nc->offset = size;
        }
 
@@ -4695,10 +4695,10 @@ void *page_frag_alloc(struct page_frag_cache *nc,
                size = nc->size;

mm/page_alloc.c void *page_frag_alloc(struct page_frag_cache *nc, · line 4703

Quoted 10 lines
mm/page_alloc.c +2 -2
 #endif
                /* OK, page count is 0, we can safely set it */
-               set_page_count(page, size);
+               set_page_count(page, size + 1);
 
                /* reset page count bias and offset to start of new frag */
-               nc->pagecnt_bias = size;
+               nc->pagecnt_bias = size + 1;
                offset = size - fragsz;
        }

If we already have to add a constant it might be better to just use PAGE_FRAG_CACHE_MAX_SIZE + 1 in all these spots where you are having to use "size + 1" instead of "size". That way we can avoid having to add a constant to a register value and then program that value. instead we can just assign the constant value right from the start.

author
#5 lore raw
Quoted 20 lines · Alexander Duyck
Quoted 14 lines · Jann Horn

The basic idea behind ->pagecnt_bias is: If we pre-allocate the maximum number of references that we might need to create in the fastpath later, the bump-allocation fastpath only has to modify the non-atomic bias value that tracks the number of extra references we hold instead of the atomic refcount. The maximum number of allocations we can serve (under the assumption that no allocation is made with size 0) is nc->size, so that's the bias used.

However, even when all memory in the allocation has been given away, a reference to the page is still held; and in the offset < 0 slowpath, the page may be reused if everyone else has dropped their references. This means that the necessary number of references is actually nc->size+1.

Quoted 4 lines

Luckily, from a quick grep, it looks like the only path that can call page_frag_alloc(fragsz=1) is TAP with the IFF_NAPI_FRAGS flag, which requires CAP_NET_ADMIN in the init namespace and is only intended to be used for kernel testing and fuzzing.

Quoted 6 lines

Actually that has me somewhat concerned. I wouldn't be surprised if most drivers expect the netdev_alloc_frags call to at least output an SKB_DATA_ALIGN sized value.

We probably should update __netdev_alloc_frag and __napi_alloc_frag so that they will pass fragsz through SKB_DATA_ALIGN.

Do you want to do a separate patch for that? I'd like to not mix logically separate changes in a single patch, and I also don't have a good understanding of the alignment concerns here.

Quoted 36 lines
Quoted 36 lines

To test for this issue, put a WARN_ON(page_ref_count(page) == 0) in the offset < 0 path, below the virt_to_page() call, and then repeatedly call writev() on a TAP device with IFF_TAP|IFF_NO_PI|IFF_NAPI_FRAGS|IFF_NAPI, with a vector consisting of 15 elements containing 1 byte each.

---
 mm/page_alloc.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
mm/page_alloc.c +3 -3
index 35fdde041f5c..46285d28e43b 100644
@@ -4675,11 +4675,11 @@ void *page_frag_alloc(struct page_frag_cache *nc,
                /* Even if we own the page, we do not use atomic_set().
                 * This would break get_page_unless_zero() users.
                 */
-               page_ref_add(page, size - 1);
+               page_ref_add(page, size);
 
                /* reset page count bias and offset to start of new frag */
                nc->pfmemalloc = page_is_pfmemalloc(page);
-               nc->pagecnt_bias = size;
+               nc->pagecnt_bias = size + 1;
                nc->offset = size;
        }
 
@@ -4695,10 +4695,10 @@ void *page_frag_alloc(struct page_frag_cache *nc,
                size = nc->size;
 #endif
                /* OK, page count is 0, we can safely set it */
-               set_page_count(page, size);
+               set_page_count(page, size + 1);
 
Quoted 11 lines

mm/page_alloc.c void *page_frag_alloc(struct page_frag_cache *nc, · line 4703

Quoted 5 lines
mm/page_alloc.c +1 -1
                /* reset page count bias and offset to start of new frag */
-               nc->pagecnt_bias = size;
+               nc->pagecnt_bias = size + 1;
                offset = size - fragsz;
        }

If we already have to add a constant it might be better to just use PAGE_FRAG_CACHE_MAX_SIZE + 1 in all these spots where you are having to use "size + 1" instead of "size". That way we can avoid having to add a constant to a register value and then program that value. instead we can just assign the constant value right from the start.

I doubt that these few instructions make a difference, but sure, I can send a v2 with that changed.

#6 lore raw
Quoted 27 lines · Jann Horn
Quoted 24 lines · Alexander Duyck
Quoted 14 lines · Jann Horn

The basic idea behind ->pagecnt_bias is: If we pre-allocate the maximum number of references that we might need to create in the fastpath later, the bump-allocation fastpath only has to modify the non-atomic bias value that tracks the number of extra references we hold instead of the atomic refcount. The maximum number of allocations we can serve (under the assumption that no allocation is made with size 0) is nc->size, so that's the bias used.

However, even when all memory in the allocation has been given away, a reference to the page is still held; and in the offset < 0 slowpath, the page may be reused if everyone else has dropped their references. This means that the necessary number of references is actually nc->size+1.

Quoted 4 lines

Luckily, from a quick grep, it looks like the only path that can call page_frag_alloc(fragsz=1) is TAP with the IFF_NAPI_FRAGS flag, which requires CAP_NET_ADMIN in the init namespace and is only intended to be used for kernel testing and fuzzing.

Actually that has me somewhat concerned. I wouldn't be surprised if most drivers expect the netdev_alloc_frags call to at least output an SKB_DATA_ALIGN sized value.

Quoted 6 lines
Quoted 2 lines

We probably should update __netdev_alloc_frag and __napi_alloc_frag so that they will pass fragsz through SKB_DATA_ALIGN.

Do you want to do a separate patch for that? I'd like to not mix logically separate changes in a single patch, and I also don't have a good understanding of the alignment concerns here.

You could just include it as a separate patch with your work. Otherwise I will get to it when I have time.

The point is the issue you pointed out will actually cause other issues if the behavior is maintained since you shouldn't be getting unaligned blocks out of the frags API anyway.

Quoted 48 lines
Quoted 36 lines
Quoted 36 lines

To test for this issue, put a WARN_ON(page_ref_count(page) == 0) in the offset < 0 path, below the virt_to_page() call, and then repeatedly call writev() on a TAP device with IFF_TAP|IFF_NO_PI|IFF_NAPI_FRAGS|IFF_NAPI, with a vector consisting of 15 elements containing 1 byte each.

---
 mm/page_alloc.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
mm/page_alloc.c +3 -3
index 35fdde041f5c..46285d28e43b 100644
@@ -4675,11 +4675,11 @@ void *page_frag_alloc(struct page_frag_cache *nc,
                /* Even if we own the page, we do not use atomic_set().
                 * This would break get_page_unless_zero() users.
                 */
-               page_ref_add(page, size - 1);
+               page_ref_add(page, size);
 
                /* reset page count bias and offset to start of new frag */
                nc->pfmemalloc = page_is_pfmemalloc(page);
-               nc->pagecnt_bias = size;
+               nc->pagecnt_bias = size + 1;
                nc->offset = size;
        }
 
@@ -4695,10 +4695,10 @@ void *page_frag_alloc(struct page_frag_cache *nc,
                size = nc->size;
 #endif
                /* OK, page count is 0, we can safely set it */
-               set_page_count(page, size);
+               set_page_count(page, size + 1);
 
Quoted 11 lines

mm/page_alloc.c void *page_frag_alloc(struct page_frag_cache *nc, · line 4703

Quoted 5 lines
mm/page_alloc.c +1 -1
                /* reset page count bias and offset to start of new frag */
-               nc->pagecnt_bias = size;
+               nc->pagecnt_bias = size + 1;
                offset = size - fragsz;
        }

If we already have to add a constant it might be better to just use PAGE_FRAG_CACHE_MAX_SIZE + 1 in all these spots where you are having to use "size + 1" instead of "size". That way we can avoid having to add a constant to a register value and then program that value. instead we can just assign the constant value right from the start.

Quoted 2 lines

I doubt that these few instructions make a difference, but sure, I can send a v2 with that changed.

You would be surprised. They all end up adding up over time.