page_alloc: Fix freeing non-compound pages
Subsystems (3)
PATCH v2 1/1 page_alloc: Fix freeing non-compound pages
Here is a very rare race which leaks memory:
Page P0 is allocated to the page cache. Page P1 is free.
Thread A Thread B Thread Cfind_get_entry():xas_load() returns P0
Removes P0 from page cache P0 finds its buddy P1
alloc_pages(GFP_KERNEL, 1) returns P0 P0 has refcount 1page_cache_get_speculative(P0)
P0 has refcount 2
__free_pages(P0) P0 has refcount 1
put_page(P0)
P1 is not freed
Fix this by freeing all the pages in __free_pages() that won't be freed by the call to put_page(). It's usually not a good idea to split a page, but this is a very unlikely scenario.
---
v2: Add a test module. Verified it works by:
(1) loading the test module on an unmodified kernel and watching it OOM
(2) modifying __free_pages() with my v1 patch that neglected to add the PageHead test and hitting VM_BUG_ON_PAGE(PageTail(page))
(3) applying all of this patch and seeing it survive
lib/Kconfig.debug | 9 +++++++++ lib/Makefile | 1 + lib/test_free_pages.c | 42 ++++++++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 3 +++ 4 files changed, 55 insertions(+) create mode 100644 lib/test_free_pages.c
lib/Kconfig.debug +9 -0
@@ -2369,6 +2369,15 @@ config TEST_HMM If unsure, say N. +config TEST_FREE_PAGES + tristate "Test freeing pages" + help + Test that a memory leak does not occur due to a race between + freeing a block of pages and a speculative page reference. + Loading this module is safe if your kernel has the bug fixed. + If the bug is not fixed, it will leak gigabytes of memory and + probably OOM your system. + config TEST_FPU tristate "Test floating point operations in kernel space" depends on X86 && !KCOV_INSTRUMENT_ALL
lib/Makefile +1 -0
@@ -99,6 +99,7 @@ obj-$(CONFIG_TEST_BLACKHOLE_DEV) += test_blackhole_dev.o obj-$(CONFIG_TEST_MEMINIT) += test_meminit.o obj-$(CONFIG_TEST_LOCKUP) += test_lockup.o obj-$(CONFIG_TEST_HMM) += test_hmm.o +obj-$(CONFIG_TEST_FREE_PAGES) += test_free_pages.o # # CFLAGS for compiling floating point code inside the kernel. x86/Makefile turns
lib/test_free_pages.c new file +42 -0
42 lines not on this page · raw message
mm/page_alloc.c +3 -0
@@ -4947,6 +4947,9 @@ void __free_pages(struct page *page, unsigned int order) {if (put_page_testzero (page))free_the_page (page, order); +else if (!PageHead (page)) +while (order-- >0 ) +free_the_page (page + (1 << order), order); }EXPORT_SYMBOL (__free_pages);
Signature · 1 line
2.28.0
Andrew Morton
maintainerQuoted 1 line · On Sat, 26 Sep 2020 22:39:19 +0100 "Matthew Wilcox (Oracle)"…
Here is a very rare race which leaks memory:
Not worth a cc:stable?
Quoted 11 lines
Page P0 is allocated to the page cache. Page P1 is free.
Thread A Thread B Thread C
find_get_entry():xas_load()returns P0
Removes P0 from page cache P0 finds its buddy P1
alloc_pages(GFP_KERNEL, 1)returns P0 P0 has refcount 1page_cache_get_speculative(P0)
P0 has refcount 2
Quoted 1 line
__free_pages(P0)
__free_pages(P0, 1), I assume.
Quoted 2 lines
P0 has refcount 1
put_page(P0)
but this is implicitly order 0
Quoted 1 line
P1 is not freed
huh.
Quoted 9 lines
Fix this by freeing all the pages in
__free_pages()that won't be freed by the call toput_page(). It's usually not a good idea to split a page, but this is a very unlikely scenario....
mm/page_alloc.c void __free_pages(struct page *page, unsigned int order) · line 4952
Quoted 6 lines
mm/page_alloc.c +3
-0@@ -4947,6 +4947,9 @@ void __free_pages(struct page *page, unsigned int order) {if (put_page_testzero (page))free_the_page (page, order); +else if (!PageHead (page)) +while (order-- >0 ) +free_the_page (page + (1 << order), order);
Well that's weird and scary looking. `page' has non-zero refcount yet we go and free random followon pages. Methinks it merits an explanatory comment?
Quoted 5 lines · Andrew Morton
Quoted 1 line · On Sat, 26 Sep 2020 22:39:19 +0100 "Matthew Wilcox (Oracle)"…
Here is a very rare race which leaks memory:
Not worth a cc:stable?
Yes, it probably should have been. I just assume the stablebot will pick up anything that has a Fixes: tag.
Quoted 13 lines
Quoted 11 lines
Page P0 is allocated to the page cache. Page P1 is free.
Thread A Thread B Thread C
find_get_entry():xas_load()returns P0
Removes P0 from page cache P0 finds its buddy P1
alloc_pages(GFP_KERNEL, 1)returns P0 P0 has refcount 1page_cache_get_speculative(P0)
P0 has refcount 2Quoted 1 line
__free_pages(P0)
Quoted 1 line
__free_pages(P0, 1), I assume.
Good catch. That was what I meant to type.
Quoted 4 lines
Quoted 2 lines
P0 has refcount 1
put_page(P0)but this is implicitly order 0
Right, because it's not a compound page.
Quoted 3 lines
Quoted 1 line
P1 is not freed
huh.
Yeah. Nasty, and we'll never know how often it was hit.
Quoted 6 lines
Quoted 6 lines
Fix this by freeing all the pages in
__free_pages()that won't be freed by the call toput_page(). It's usually not a good idea to split a page, but this is a very unlikely scenario....
Quoted 13 lines
mm/page_alloc.c void __free_pages(struct page *page, unsigned int order) · line 4952
Quoted 9 lines
mm/page_alloc.c +3
-0@@ -4947,6 +4947,9 @@ void __free_pages(struct page *page, unsigned int order) {if (put_page_testzero (page))free_the_page (page, order); +else if (!PageHead (page)) +while (order-- >0 ) +free_the_page (page + (1 << order), order);Well that's weird and scary looking. `page' has non-zero refcount yet we go and free random followon pages. Methinks it merits an explanatory comment?
Well, poot. I lost that comment in the shuffling of patches. In a different tree, I have:
@@ -4943,10 +4943,19 @@ static inline void free_the_page(struct page *page, unsi gned int order)
__free_pages_ok(page, order);
}
+/*
+ * If we free a non-compound allocation, another thread may have a
+ * speculative reference to the first page. It has no way of knowing
+ * about the rest of the allocation, so we have to free all but the
+ * first page here.
+ */
void __free_pages(struct page *page, unsigned int order)
{
if (put_page_testzero(page))
free_the_page(page, order);+ else if (!PageHead(page))
+ while (order-- > 0)
+ free_the_page(page + (1 << order), order);
}
EXPORT_SYMBOL(__free_pages);
Although I'm now thinking of making that comment into kernel-doc and turning it into advice to the caller rather than an internal note to other mm developers.
Andrew Morton
maintainerQuoted 8 lines · On Tue, 29 Sep 2020 02:17:19 +0100 Matthew Wilcox <[email protected]> wrote:
Quoted 5 lines · Andrew Morton
Quoted 1 line · On Sat, 26 Sep 2020 22:39:19 +0100 "Matthew Wilcox (Oracle)"…
Here is a very rare race which leaks memory:
Not worth a cc:stable?
Yes, it probably should have been.
Have you a feeling for how often this occurs?
Quoted 2 lines
I just assume the stablebot will
pick up anything that has a Fixes: tag.
We asked them not to do that for mm/ patches. Crazy stuff was getting backported.
Quoted 19 lines
Quoted 14 lines
mm/page_alloc.c void __free_pages(struct page *page, unsigned int order) · line 4952
Quoted 10 lines
mm/page_alloc.c +3
-0@@ -4947,6 +4947,9 @@ void __free_pages(struct page *page, unsigned int order) {if (put_page_testzero (page))free_the_page (page, order); +else if (!PageHead (page)) +while (order-- >0 ) +free_the_page (page + (1 << order), order);Well that's weird and scary looking. `page' has non-zero refcount yet we go and free random followon pages. Methinks it merits an explanatory comment?
Well, poot. I lost that comment in the shuffling of patches. In a different tree, I have:
Review static inline void free_the_page(struct page *page, unsi · line 4948
Quoted 6 lines
(quoted diff) +2
-0@@ -4943,10 +4943,19 @@ static inline void free_the_page(struct page *page, unsi gned int order) __free_pages_ok(page, order); } +/* + * If we free a non-compound allocation, another thread may have a
"non-compound, higher-order", I suggest?
Quoted 15 lines
mm/page_alloc.c +7
-0+* speculative reference to the first page. It has no way of knowing +* about the rest of the allocation, so we have to free all but the +* first page here. +*/ void __free_pages (struct page *page,unsigned int order) {if (put_page_testzero (page))free_the_page (page, order); +else if (!PageHead (page)) +while (order-- >0 ) +free_the_page (page + (1 << order), order); }EXPORT_SYMBOL (__free_pages);
Quoted 3 lines
Although I'm now thinking of making that comment into kernel-doc and turning it into advice to the caller rather than an internal note to other mm developers.
hm. But what action could the caller take? The explanatory comment seems OK to me.
Quoted 5 lines · Andrew Morton
Quoted 3 lines · On Tue, 29 Sep 2020 02:17:19 +0100 Matthew Wilcox <[email protected]> wrote:
Quoted 2 lines · Andrew Morton
On Sat, 26 Sep 2020 22:39:19 +0100 "Matthew Wilcox (Oracle)" <[email protected]> wrote:
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Here is a very rare race which leaks memory:
Not worth a cc:stable?
Yes, it probably should have been.
Have you a feeling for how often this occurs?
I doubt it happens often. I don't think I could construct a workload to make it happen frequently. Maybe more often with a virtualised workload where a thread can be preempted between instructions.
Quoted 5 lines
Quoted 2 lines
I just assume the stablebot will
pick up anything that has a Fixes: tag.We asked them not to do that for
mm/patches. Crazy stuff was getting backported.
That's a shame. I'll try to remember to cc them explicitly in the future.
Quoted 6 lines
Quoted 3 lines
Although I'm now thinking of making that comment into kernel-doc and turning it into advice to the caller rather than an internal note to other mm developers.
hm. But what action could the caller take? The explanatory comment seems OK to me.
Use compound pages instead of non-compound pages. Although Linus has asked that people stop using __get_free_pages(), so maybe that will be the direction we go in.
https://lore.kernel.org/lkml/CA+55aFwyxJ+TOpaJZnC5MPJ-25xbLAEu8iJP8zTYhmA3LXFF8Q@mail.gmail.com/
Excerpts from Andrew Morton's message of September 29, 2020 2:46 pm:
Quoted 6 lines · On Tue, 29 Sep 2020 02:17:19 +0100 Matthew Wilcox <[email protected]> wrote:
Quoted 5 lines · Andrew Morton
Quoted 3 lines · On Sat, 26 Sep 2020 22:39:19 +0100 "Matthew Wilcox (Oracle)"…
Quoted 2 lines
Here is a very rare race which leaks memory:
Great catch! [sorry, a bit behind with emails]
Quoted 56 lines
Quoted 4 lines
Quoted 2 lines
Not worth a cc:stable?
Yes, it probably should have been.
Have you a feeling for how often this occurs?
Quoted 2 lines
I just assume the stablebot will
pick up anything that has a Fixes: tag.We asked them not to do that for
mm/patches. Crazy stuff was getting backported.Quoted 19 lines
Quoted 14 lines
mm/page_alloc.c void __free_pages(struct page *page, unsigned int order) · line 4952
Quoted 10 lines
mm/page_alloc.c +3
-0@@ -4947,6 +4947,9 @@ void __free_pages(struct page *page, unsigned int order) {if (put_page_testzero (page))free_the_page (page, order); +else if (!PageHead (page)) +while (order-- >0 ) +free_the_page (page + (1 << order), order);Well that's weird and scary looking. `page' has non-zero refcount yet we go and free random followon pages. Methinks it merits an explanatory comment?
Well, poot. I lost that comment in the shuffling of patches. In a different tree, I have:
Review static inline void free_the_page(struct page *page, unsi · line 4948
Quoted 6 lines
(quoted diff) +2
-0@@ -4943,10 +4943,19 @@ static inline void free_the_page(struct page *page, unsi gned int order) __free_pages_ok(page, order); } +/* + * If we free a non-compound allocation, another thread may have a"non-compound, higher-order", I suggest?
Quoted 15 lines
mm/page_alloc.c +7
-0+* speculative reference to the first page. It has no way of knowing +* about the rest of the allocation, so we have to free all but the +* first page here. +*/ void __free_pages (struct page *page,unsigned int order) {if (put_page_testzero (page))free_the_page (page, order); +else if (!PageHead (page)) +while (order-- >0 ) +free_the_page (page + (1 << order), order); }EXPORT_SYMBOL (__free_pages);
Quoted 6 lines
Quoted 3 lines
Although I'm now thinking of making that comment into kernel-doc and turning it into advice to the caller rather than an internal note to other mm developers.
hm. But what action could the caller take? The explanatory comment seems OK to me.
The version of this without the comment got merged. I didn't mind the comment...
Thanks,
Nick
Quoted 3 lines · Andrew Morton
Well that's weird and scary looking. `page' has non-zero refcount yet we go and free random followon pages. Methinks it merits an explanatory comment?
Here's some kernel-doc. Opinions?
/**
* __free_pages - Free pages allocated with alloc_pages().
* @page: The page pointer returned from alloc_pages().
* @order: The order of the allocation.
*
* This function differs from put_page() in that it can free multi-page
* allocations that were not allocated with %__GFP_COMP. This function
* does not check that the @order passed in matches that of the
* allocation, so it is possible to leak memory. Freeing more memory than
* was allocated will probably be warned about by other debugging checks. *
* It is only safe to use the page reference count to determine when
* to free an allocation if you use %__GFP_COMP (in which case, you may
* as well use put_page() to free the page). Another thread may have a
* speculative reference to the first page, but it has no way of knowing
* about the rest of the allocation, so we have to free all but the
* first page here.
*
* Context: May be called in interrupt context but not NMI context. */
Quoted 5 lines · Matthew Wilcox
Quoted 3 lines · Andrew Morton
Well that's weird and scary looking. `page' has non-zero refcount yet we go and free random followon pages. Methinks it merits an explanatory comment?
Quoted 8 lines
Here's some kernel-doc. Opinions?
/**
*__free_pages- Free pages allocated withalloc_pages().
* @page: The page pointer returned fromalloc_pages().
* @order: The order of the allocation.
*
* This function differs fromput_page()in that it can free multi-page
This sentence presumes existing description/prior knowledge about put_page().
Maybe
This function can free multi-page allocations that were not allocated with %__GFP_COMP, unlike put_page() that would free only the first page in such case. __free_pages() does not ...
Quoted 15 lines
* allocations that were not allocated with %__GFP_COMP. This function
* does not check that the @order passed in matches that of the
* allocation, so it is possible to leak memory. Freeing more memory than
* was allocated will probably be warned about by other debugging checks. *
* It is only safe to use the page reference count to determine when
* to free an allocation if you use %__GFP_COMP (in which case, you may
* as well useput_page()to free the page). Another thread may have a
* speculative reference to the first page, but it has no way of knowing
* about the rest of the allocation, so we have to free all but the
* first page here.
*
* Context: May be called in interrupt context but not NMI context. */
Signature · 2 lines
Sincerely yours, Mike.
Quoted 8 lines · Mike Rapoport
This sentence presumes existing description/prior knowledge about
put_page().Maybe
This function can free multi-page allocations that were not allocated with %__GFP_COMP, unlike
put_page()that would free only the first page in such case.__free_pages()does not ...
Thanks. After waking up this morning I did a more extensive rewrite:
/**
* __free_pages - Free pages allocated with alloc_pages().
* @page: The page pointer returned from alloc_pages().
* @order: The order of the allocation.
*
* This function can free multi-page allocations that are not compound
* pages. It does not check that the @order passed in matches that of
* the allocation, so it is easy to leak memory. Freeing more memory
* than was allocated will probably emit a warning.
*
* If the last reference to this page is speculative, it will be released
* by put_page() which only frees the first page of a non-compound
* allocation. To prevent the remaining pages from being leaked, we free
* the subsequent pages here. If you want to use the page's reference
* count to decide when to free the allocation, you should allocate a
* compound page, and use put_page() instead of __free_pages(). *
* Context: May be called in interrupt context or holding a normal
* spinlock, but not in NMI context or while holding a raw spinlock. */
Mike Rapoport
Acked-byQuoted 11 lines · Matthew Wilcox
Quoted 8 lines · Mike Rapoport
This sentence presumes existing description/prior knowledge about
put_page().Maybe
This function can free multi-page allocations that were not allocated with %__GFP_COMP, unlike
put_page()that would free only the first page in such case.__free_pages()does not ...Thanks. After waking up this morning I did a more extensive rewrite:
I like this one
Quoted 21 lines
/**
*__free_pages- Free pages allocated withalloc_pages().
* @page: The page pointer returned fromalloc_pages().
* @order: The order of the allocation.
*
* This function can free multi-page allocations that are not compound
* pages. It does not check that the @order passed in matches that of
* the allocation, so it is easy to leak memory. Freeing more memory
* than was allocated will probably emit a warning.
*
* If the last reference to this page is speculative, it will be released
* byput_page()which only frees the first page of a non-compound
* allocation. To prevent the remaining pages from being leaked, we free
* the subsequent pages here. If you want to use the page's reference
* count to decide when to free the allocation, you should allocate a
* compound page, and useput_page()instead of__free_pages(). *
* Context: May be called in interrupt context or holding a normal
* spinlock, but not in NMI context or while holding a raw spinlock. */
Signature · 2 lines
Sincerely yours, Mike.