Skip to contentSkip to conversation

page_alloc: Fix freeing non-compound pages

PATCH Applied Matthew Wilcox (Oracle) 4 participants 10 messages Sep 26, 2020 – Oct 19, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 page_alloc: Fix freeing non-compound pages

Here is a very rare race which leaks memory:

Page P0 is allocated to the page cache. Page P1 is free.

Thread A Thread B Thread C
find_get_entry():
xas_load() returns P0
Removes P0 from page cache P0 finds its buddy P1
alloc_pages(GFP_KERNEL, 1) returns P0 P0 has refcount 1
page_cache_get_speculative(P0)
P0 has refcount 2

			__free_pages(P0)
			P0 has refcount 1

put_page(P0)
P1 is not freed

Fix this by freeing all the pages in __free_pages() that won't be freed by the call to put_page(). It's usually not a good idea to split a page, but this is a very unlikely scenario.

Fixes e286781d5f2e ("mm: speculative page references")

---
v2: Add a test module. Verified it works by:
(1) loading the test module on an unmodified kernel and watching it OOM
(2) modifying __free_pages() with my v1 patch that neglected to add the PageHead test and hitting VM_BUG_ON_PAGE(PageTail(page))
(3) applying all of this patch and seeing it survive

 lib/Kconfig.debug     |  9 +++++++++
 lib/Makefile          |  1 +
 lib/test_free_pages.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 mm/page_alloc.c       |  3 +++
 4 files changed, 55 insertions(+)
 create mode 100644 lib/test_free_pages.c
lib/Kconfig.debug +9 -0
index e068c3c7189a..eed59af0e907 100644
@@ -2369,6 +2369,15 @@ config TEST_HMM
 
 	  If unsure, say N.
 
+config TEST_FREE_PAGES
+	tristate "Test freeing pages"
+	help
+	  Test that a memory leak does not occur due to a race between
+	  freeing a block of pages and a speculative page reference.
+	  Loading this module is safe if your kernel has the bug fixed.
+	  If the bug is not fixed, it will leak gigabytes of memory and
+	  probably OOM your system.
+
 config TEST_FPU
 	tristate "Test floating point operations in kernel space"
 	depends on X86 && !KCOV_INSTRUMENT_ALL
lib/Makefile +1 -0
index a4a4c6864f51..071b687b7363 100644
@@ -99,6 +99,7 @@ obj-$(CONFIG_TEST_BLACKHOLE_DEV) += test_blackhole_dev.o
 obj-$(CONFIG_TEST_MEMINIT) += test_meminit.o
 obj-$(CONFIG_TEST_LOCKUP) += test_lockup.o
 obj-$(CONFIG_TEST_HMM) += test_hmm.o
+obj-$(CONFIG_TEST_FREE_PAGES) += test_free_pages.o
 
 #
 # CFLAGS for compiling floating point code inside the kernel. x86/Makefile turns
lib/test_free_pages.c new file +42 -0

42 lines not on this page · raw message

mm/page_alloc.c +3 -0
index fab5e97dc9ca..9b259c76e285 100644
@@ -4947,6 +4947,9 @@ void __free_pages(struct page *page, unsigned int order)
 {
 	if (put_page_testzero(page))
 		free_the_page(page, order);
+	else if (!PageHead(page))
+		while (order-- > 0)
+			free_the_page(page + (1 << order), order);
 }
 EXPORT_SYMBOL(__free_pages);
 
Signature · 1 line
2.28.0
maintainer
#1 lore raw
Quoted 1 line · On Sat, 26 Sep 2020 22:39:19 +0100 "Matthew Wilcox (Oracle)"…

Here is a very rare race which leaks memory:

Not worth a cc:stable?

Quoted 11 lines

Page P0 is allocated to the page cache. Page P1 is free.

Thread A Thread B Thread C
find_get_entry():
xas_load() returns P0
Removes P0 from page cache P0 finds its buddy P1
alloc_pages(GFP_KERNEL, 1) returns P0 P0 has refcount 1
page_cache_get_speculative(P0)
P0 has refcount 2

Quoted 1 line
			__free_pages(P0)
			__free_pages(P0, 1), I assume.
Quoted 2 lines

P0 has refcount 1
put_page(P0)

but this is implicitly order 0

Quoted 1 line

P1 is not freed

huh.

Quoted 9 lines

Fix this by freeing all the pages in __free_pages() that won't be freed by the call to put_page(). It's usually not a good idea to split a page, but this is a very unlikely scenario.

...

mm/page_alloc.c void __free_pages(struct page *page, unsigned int order) · line 4952

Quoted 6 lines
mm/page_alloc.c +3 -0
@@ -4947,6 +4947,9 @@ void __free_pages(struct page *page, unsigned int order)
 {
 	if (put_page_testzero(page))
 		free_the_page(page, order);
+	else if (!PageHead(page))
+		while (order-- > 0)
+			free_the_page(page + (1 << order), order);

Well that's weird and scary looking. `page' has non-zero refcount yet we go and free random followon pages. Methinks it merits an explanatory comment?

#2 lore raw
Quoted 5 lines · Andrew Morton
Quoted 1 line · On Sat, 26 Sep 2020 22:39:19 +0100 "Matthew Wilcox (Oracle)"…

Here is a very rare race which leaks memory:

Not worth a cc:stable?

Yes, it probably should have been. I just assume the stablebot will pick up anything that has a Fixes: tag.

Quoted 13 lines
Quoted 11 lines

Page P0 is allocated to the page cache. Page P1 is free.

Thread A Thread B Thread C
find_get_entry():
xas_load() returns P0
Removes P0 from page cache P0 finds its buddy P1
alloc_pages(GFP_KERNEL, 1) returns P0 P0 has refcount 1
page_cache_get_speculative(P0)
P0 has refcount 2

Quoted 1 line
			__free_pages(P0)
Quoted 1 line
			__free_pages(P0, 1), I assume.

Good catch. That was what I meant to type.

Quoted 4 lines
Quoted 2 lines

P0 has refcount 1
put_page(P0)

but this is implicitly order 0

Right, because it's not a compound page.

Quoted 3 lines
Quoted 1 line

P1 is not freed

huh.

Yeah. Nasty, and we'll never know how often it was hit.

Quoted 6 lines
Quoted 6 lines

Fix this by freeing all the pages in __free_pages() that won't be freed by the call to put_page(). It's usually not a good idea to split a page, but this is a very unlikely scenario.

...

Quoted 13 lines

mm/page_alloc.c void __free_pages(struct page *page, unsigned int order) · line 4952

Quoted 9 lines
mm/page_alloc.c +3 -0
@@ -4947,6 +4947,9 @@ void __free_pages(struct page *page, unsigned int order)
 {
 	if (put_page_testzero(page))
 		free_the_page(page, order);
+	else if (!PageHead(page))
+		while (order-- > 0)
+			free_the_page(page + (1 << order), order);

Well that's weird and scary looking. `page' has non-zero refcount yet we go and free random followon pages. Methinks it merits an explanatory comment?

Well, poot. I lost that comment in the shuffling of patches. In a different tree, I have:

@@ -4943,10 +4943,19 @@ static inline void free_the_page(struct page *page, unsi gned int order)

                __free_pages_ok(page, order);

}

+/*
+ * If we free a non-compound allocation, another thread may have a
+ * speculative reference to the first page. It has no way of knowing
+ * about the rest of the allocation, so we have to free all but the
+ * first page here.
+ */
void __free_pages(struct page *page, unsigned int order)

 {
        if (put_page_testzero(page))
                free_the_page(page, order);

+ else if (!PageHead(page))
+ while (order-- > 0)
+ free_the_page(page + (1 << order), order);
}
EXPORT_SYMBOL(__free_pages);

Although I'm now thinking of making that comment into kernel-doc and turning it into advice to the caller rather than an internal note to other mm developers.

maintainer
#3 lore raw
Quoted 8 lines · On Tue, 29 Sep 2020 02:17:19 +0100 Matthew Wilcox <[email protected]> wrote:
Quoted 5 lines · Andrew Morton
Quoted 1 line · On Sat, 26 Sep 2020 22:39:19 +0100 "Matthew Wilcox (Oracle)"…

Here is a very rare race which leaks memory:

Not worth a cc:stable?

Yes, it probably should have been.

Have you a feeling for how often this occurs?

Quoted 2 lines

I just assume the stablebot will
pick up anything that has a Fixes: tag.

We asked them not to do that for mm/ patches. Crazy stuff was getting backported.

Quoted 19 lines
Quoted 14 lines

mm/page_alloc.c void __free_pages(struct page *page, unsigned int order) · line 4952

Quoted 10 lines
mm/page_alloc.c +3 -0
@@ -4947,6 +4947,9 @@ void __free_pages(struct page *page, unsigned int order)
 {
 	if (put_page_testzero(page))
 		free_the_page(page, order);
+	else if (!PageHead(page))
+		while (order-- > 0)
+			free_the_page(page + (1 << order), order);

Well that's weird and scary looking. `page' has non-zero refcount yet we go and free random followon pages. Methinks it merits an explanatory comment?

Well, poot. I lost that comment in the shuffling of patches. In a different tree, I have:

Review static inline void free_the_page(struct page *page, unsi · line 4948

Quoted 6 lines
(quoted diff) +2 -0
@@ -4943,10 +4943,19 @@ static inline void free_the_page(struct page *page, unsi
 gned int order)
                __free_pages_ok(page, order);
 }
 
+/*
+ * If we free a non-compound allocation, another thread may have a

"non-compound, higher-order", I suggest?

Quoted 15 lines
mm/page_alloc.c +7 -0
+ * speculative reference to the first page.  It has no way of knowing
+ * about the rest of the allocation, so we have to free all but the
+ * first page here.
+ */
 void __free_pages(struct page *page, unsigned int order)
 {
        if (put_page_testzero(page))
                free_the_page(page, order);
+       else if (!PageHead(page))
+               while (order-- > 0)
+                       free_the_page(page + (1 << order), order);
 }
 EXPORT_SYMBOL(__free_pages);
Quoted 3 lines

Although I'm now thinking of making that comment into kernel-doc and turning it into advice to the caller rather than an internal note to other mm developers.

hm. But what action could the caller take? The explanatory comment seems OK to me.

#4 lore raw
Quoted 5 lines · Andrew Morton
Quoted 3 lines · On Tue, 29 Sep 2020 02:17:19 +0100 Matthew Wilcox <[email protected]> wrote:
Quoted 2 lines · Andrew Morton

On Sat, 26 Sep 2020 22:39:19 +0100 "Matthew Wilcox (Oracle)" <[email protected]> wrote:

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Here is a very rare race which leaks memory:

Not worth a cc:stable?

Yes, it probably should have been.

Have you a feeling for how often this occurs?

I doubt it happens often. I don't think I could construct a workload to make it happen frequently. Maybe more often with a virtualised workload where a thread can be preempted between instructions.

Quoted 5 lines
Quoted 2 lines

I just assume the stablebot will
pick up anything that has a Fixes: tag.

We asked them not to do that for mm/ patches. Crazy stuff was getting backported.

That's a shame. I'll try to remember to cc them explicitly in the future.

Quoted 6 lines
Quoted 3 lines

Although I'm now thinking of making that comment into kernel-doc and turning it into advice to the caller rather than an internal note to other mm developers.

hm. But what action could the caller take? The explanatory comment seems OK to me.

Use compound pages instead of non-compound pages. Although Linus has asked that people stop using __get_free_pages(), so maybe that will be the direction we go in.

https://lore.kernel.org/lkml/CA+55aFwyxJ+TOpaJZnC5MPJ-25xbLAEu8iJP8zTYhmA3LXFF8Q@mail.gmail.com/

#5 lore raw
In reply to Andrew Morton

Excerpts from Andrew Morton's message of September 29, 2020 2:46 pm:

Quoted 6 lines · On Tue, 29 Sep 2020 02:17:19 +0100 Matthew Wilcox <[email protected]> wrote:
Quoted 5 lines · Andrew Morton
Quoted 3 lines · On Sat, 26 Sep 2020 22:39:19 +0100 "Matthew Wilcox (Oracle)"…
Quoted 2 lines

Here is a very rare race which leaks memory:

Great catch! [sorry, a bit behind with emails]

Quoted 56 lines
Quoted 4 lines
Quoted 2 lines

Not worth a cc:stable?

Yes, it probably should have been.

Have you a feeling for how often this occurs?

Quoted 2 lines

I just assume the stablebot will
pick up anything that has a Fixes: tag.

We asked them not to do that for mm/ patches. Crazy stuff was getting backported.

Quoted 19 lines
Quoted 14 lines

mm/page_alloc.c void __free_pages(struct page *page, unsigned int order) · line 4952

Quoted 10 lines
mm/page_alloc.c +3 -0
@@ -4947,6 +4947,9 @@ void __free_pages(struct page *page, unsigned int order)
 {
 	if (put_page_testzero(page))
 		free_the_page(page, order);
+	else if (!PageHead(page))
+		while (order-- > 0)
+			free_the_page(page + (1 << order), order);

Well that's weird and scary looking. `page' has non-zero refcount yet we go and free random followon pages. Methinks it merits an explanatory comment?

Well, poot. I lost that comment in the shuffling of patches. In a different tree, I have:

Review static inline void free_the_page(struct page *page, unsi · line 4948

Quoted 6 lines
(quoted diff) +2 -0
@@ -4943,10 +4943,19 @@ static inline void free_the_page(struct page *page, unsi
 gned int order)
                __free_pages_ok(page, order);
 }
 
+/*
+ * If we free a non-compound allocation, another thread may have a

"non-compound, higher-order", I suggest?

Quoted 15 lines
mm/page_alloc.c +7 -0
+ * speculative reference to the first page.  It has no way of knowing
+ * about the rest of the allocation, so we have to free all but the
+ * first page here.
+ */
 void __free_pages(struct page *page, unsigned int order)
 {
        if (put_page_testzero(page))
                free_the_page(page, order);
+       else if (!PageHead(page))
+               while (order-- > 0)
+                       free_the_page(page + (1 << order), order);
 }
 EXPORT_SYMBOL(__free_pages);
Quoted 6 lines
Quoted 3 lines

Although I'm now thinking of making that comment into kernel-doc and turning it into advice to the caller rather than an internal note to other mm developers.

hm. But what action could the caller take? The explanatory comment seems OK to me.

The version of this without the comment got merged. I didn't mind the comment...

Thanks,
Nick

#6 lore raw
In reply to Andrew Morton
Quoted 3 lines · Andrew Morton

Well that's weird and scary looking. `page' has non-zero refcount yet we go and free random followon pages. Methinks it merits an explanatory comment?

Here's some kernel-doc. Opinions?

/**
* __free_pages - Free pages allocated with alloc_pages().
* @page: The page pointer returned from alloc_pages().
* @order: The order of the allocation.
*
* This function differs from put_page() in that it can free multi-page
* allocations that were not allocated with %__GFP_COMP. This function
* does not check that the @order passed in matches that of the
* allocation, so it is possible to leak memory. Freeing more memory than
* was allocated will probably be warned about by other debugging checks. *
* It is only safe to use the page reference count to determine when
* to free an allocation if you use %__GFP_COMP (in which case, you may
* as well use put_page() to free the page). Another thread may have a
* speculative reference to the first page, but it has no way of knowing
* about the rest of the allocation, so we have to free all but the
* first page here.
*
* Context: May be called in interrupt context but not NMI context. */

#7 lore raw
Quoted 5 lines · Matthew Wilcox
Quoted 3 lines · Andrew Morton

Well that's weird and scary looking. `page' has non-zero refcount yet we go and free random followon pages. Methinks it merits an explanatory comment?

Quoted 8 lines

Here's some kernel-doc. Opinions?

/**
* __free_pages - Free pages allocated with alloc_pages().
* @page: The page pointer returned from alloc_pages().
* @order: The order of the allocation.
*
* This function differs from put_page() in that it can free multi-page

This sentence presumes existing description/prior knowledge about put_page().

Maybe

This function can free multi-page allocations that were not allocated with %__GFP_COMP, unlike put_page() that would free only the first page in such case. __free_pages() does not ...

Quoted 15 lines

* allocations that were not allocated with %__GFP_COMP. This function
* does not check that the @order passed in matches that of the
* allocation, so it is possible to leak memory. Freeing more memory than
* was allocated will probably be warned about by other debugging checks. *
* It is only safe to use the page reference count to determine when
* to free an allocation if you use %__GFP_COMP (in which case, you may
* as well use put_page() to free the page). Another thread may have a
* speculative reference to the first page, but it has no way of knowing
* about the rest of the allocation, so we have to free all but the
* first page here.
*
* Context: May be called in interrupt context but not NMI context. */

Signature · 2 lines
Sincerely yours,
Mike.
#8 lore raw
Quoted 8 lines · Mike Rapoport

This sentence presumes existing description/prior knowledge about put_page().

Maybe

This function can free multi-page allocations that were not allocated with %__GFP_COMP, unlike put_page() that would free only the first page in such case. __free_pages() does not ...

Thanks. After waking up this morning I did a more extensive rewrite:

/**
* __free_pages - Free pages allocated with alloc_pages().
* @page: The page pointer returned from alloc_pages().
* @order: The order of the allocation.
*
* This function can free multi-page allocations that are not compound
* pages. It does not check that the @order passed in matches that of
* the allocation, so it is easy to leak memory. Freeing more memory
* than was allocated will probably emit a warning.
*
* If the last reference to this page is speculative, it will be released
* by put_page() which only frees the first page of a non-compound
* allocation. To prevent the remaining pages from being leaked, we free
* the subsequent pages here. If you want to use the page's reference
* count to decide when to free the allocation, you should allocate a
* compound page, and use put_page() instead of __free_pages(). *
* Context: May be called in interrupt context or holding a normal
* spinlock, but not in NMI context or while holding a raw spinlock. */

Acked-by
#9 lore raw
Quoted 11 lines · Matthew Wilcox
Quoted 8 lines · Mike Rapoport

This sentence presumes existing description/prior knowledge about put_page().

Maybe

This function can free multi-page allocations that were not allocated with %__GFP_COMP, unlike put_page() that would free only the first page in such case. __free_pages() does not ...

Thanks. After waking up this morning I did a more extensive rewrite:

I like this one

Quoted 21 lines

/**
* __free_pages - Free pages allocated with alloc_pages().
* @page: The page pointer returned from alloc_pages().
* @order: The order of the allocation.
*
* This function can free multi-page allocations that are not compound
* pages. It does not check that the @order passed in matches that of
* the allocation, so it is easy to leak memory. Freeing more memory
* than was allocated will probably emit a warning.
*
* If the last reference to this page is speculative, it will be released
* by put_page() which only frees the first page of a non-compound
* allocation. To prevent the remaining pages from being leaked, we free
* the subsequent pages here. If you want to use the page's reference
* count to decide when to free the allocation, you should allocate a
* compound page, and use put_page() instead of __free_pages(). *
* Context: May be called in interrupt context or holding a normal
* spinlock, but not in NMI context or while holding a raw spinlock. */

Signature · 2 lines
Sincerely yours,
Mike.