FAILED: patch "[PATCH] mm/gup: avoid an unnecessary allocation call for" failed to apply to 6.11-stable tree
Subsystems (3)
[email protected]
maintainerThe patch below does not apply to the 6.11-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.
To reproduce the conflict and resubmit, you may use the following commands:
git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-6.11.y
git checkout FETCH_HEAD
git cherry-pick -x 94efde1d15399f5c88e576923db9bcd422d217f2
# <resolve conflicts, build, test, etc.>
git commit -s
git send-email --to '<[email protected]>' --in-reply-to '2024111754-stamina-flyer-1e05@gregkh' --subject-prefix 'PATCH 6.11.y' HEAD^..
Possible dependencies:
thanks,
greg k-h
------------------ original commit in Linus's tree ------------------
From 94efde1d15399f5c88e576923db9bcd422d217f2 Mon Sep 17 00:00:00 2001
From: John Hubbard <[email protected]>
Date: Mon, 4 Nov 2024 19:29:44 -0800
Subject: [PATCH] mm/gup: avoid an unnecessary allocation call for
FOLL_LONGTERM cases
commit 53ba78de064b ("mm/gup: introduce check_and_migrate_movable_folios()") created a new constraint on the pin_user_pages*() API family: a potentially large internal allocation must now occur, for FOLL_LONGTERM cases.
A user-visible consequence has now appeared: user space can no longer pin more than 2GB of memory anymore on x86_64. That's because, on a 4KB PAGE_SIZE system, when user space tries to (indirectly, via a device driver that calls pin_user_pages()) pin 2GB, this requires an allocation of a folio pointers array of MAX_PAGE_ORDER size, which is the limit for kmalloc().
In addition to the directly visible effect described above, there is also the problem of adding an unnecessary allocation. The **pages array argument has already been allocated, and there is no need for a redundant **folios array allocation in this case.
Fix this by avoiding the new allocation entirely. This is done by referring to either the original page[i] within **pages, or to the associated folio. Thanks to David Hildenbrand for suggesting this approach and for providing the initial implementation (which I've tested and adjusted slightly) as well.
mm/gup.c +77 -39
@@ -2273,20 +2273,57 @@ struct page *get_dump_page(unsigned long addr)#endif /* CONFIG_ELF_CORE */ #ifdef CONFIG_MIGRATION + +/* +* An array of either pages or folios ("pofs"). Although it may seem tempting to +* avoid this complication, by simply interpreting a list of folios as a list of +* pages, that approach won't work in the longer term, because eventually the +* layouts of struct page and struct folio will become completely different. +* Furthermore, this pof approach avoids excessive page_folio() calls. +*/ +struct pages_or_folios { +union { +struct page **pages; +struct folio **folios; +void **entries; + }; +bool has_folios; +long nr_entries; +}; + +static struct folio *pofs_get_folio (struct pages_or_folios *pofs,long i) +{ +if (pofs->has_folios) +return pofs->folios[i]; +return page_folio (pofs->pages[i]); +} + +static void pofs_clear_entry (struct pages_or_folios *pofs,long i) +{ + pofs->entries[i] =NULL ; +} + +static void pofs_unpin (struct pages_or_folios *pofs) +{ +if (pofs->has_folios) +unpin_folios (pofs->folios, pofs->nr_entries); +else +unpin_user_pages (pofs->pages, pofs->nr_entries); +} +/* * Returns the number of collected folios. Return value is always >= 0. */ static unsigned long collect_longterm_unpinnable_folios ( -struct list_head *movable_folio_list, -unsigned long nr_folios, -struct folio **folios) +struct list_head *movable_folio_list, +struct pages_or_folios *pofs) {unsigned long i, collected =0 ;struct folio *prev_folio =NULL ;bool drain_allow = true; -for (i =0 ; i < nr_folios; i++) { -struct folio *folio = folios[i]; +for (i =0 ; i < pofs->nr_entries; i++) { +struct folio *folio =pofs_get_folio (pofs, i);if (folio == prev_folio)continue ;
@@ -2327,16 +2364,15 @@ static unsigned long collect_longterm_unpinnable_folios(* Returns -EAGAIN if all folios were successfully migrated or -errno for * failure (or partial success). */ -static int migrate_longterm_unpinnable_folios ( -struct list_head *movable_folio_list, -unsigned long nr_folios, -struct folio **folios) +static int +migrate_longterm_unpinnable_folios (struct list_head *movable_folio_list, +struct pages_or_folios *pofs) {int ret;unsigned long i; -for (i =0 ; i < nr_folios; i++) { -struct folio *folio = folios[i]; +for (i =0 ; i < pofs->nr_entries; i++) { +struct folio *folio =pofs_get_folio (pofs, i);if (folio_is_device_coherent (folio)) {/*
@@ -2344,7 +2380,7 @@ static int migrate_longterm_unpinnable_folios(* convert the pin on the source folio to a normal * reference. */ - folios[i] =NULL ; +pofs_clear_entry (pofs, i);folio_get (folio);gup_put_folio (folio,1 ,FOLL_PIN );
@@ -2363,8 +2399,8 @@ static int migrate_longterm_unpinnable_folios(* calling folio_isolate_lru() which takes a reference so the * folio won't be freed if it's migrating. */ -unpin_folio (folios[i]); - folios[i] =NULL ; +unpin_folio (folio); +pofs_clear_entry (pofs, i); }if (!list_empty (movable_folio_list)) {
@@ -2387,12 +2423,26 @@ static int migrate_longterm_unpinnable_folios(return -EAGAIN ; err: -unpin_folios (folios, nr_folios); +pofs_unpin (pofs);putback_movable_pages (movable_folio_list);return ret; } +static long +check_and_migrate_movable_pages_or_folios (struct pages_or_folios *pofs) +{ +LIST_HEAD (movable_folio_list); +unsigned long collected; + + collected =collect_longterm_unpinnable_folios (&movable_folio_list, + pofs); +if (!collected) +return 0 ; + +return migrate_longterm_unpinnable_folios (&movable_folio_list, pofs); +} +/* * Check whether all folios are *allowed* to be pinned indefinitely (long term). * Rather confusingly, all folios in the range are required to be pinned via
@@ -2417,16 +2467,13 @@ static int migrate_longterm_unpinnable_folios(static long check_and_migrate_movable_folios (unsigned long nr_folios,struct folio **folios) { -unsigned long collected; -LIST_HEAD (movable_folio_list); +struct pages_or_folios pofs = { + .folios = folios, + .has_folios = true, + .nr_entries = nr_folios, + }; - collected =collect_longterm_unpinnable_folios (&movable_folio_list, - nr_folios, folios); -if (!collected) -return 0 ; - -return migrate_longterm_unpinnable_folios (&movable_folio_list, - nr_folios, folios); +return check_and_migrate_movable_pages_or_folios (&pofs); }/*
@@ -2436,22 +2483,13 @@ static long check_and_migrate_movable_folios(unsigned long nr_folios,static long check_and_migrate_movable_pages (unsigned long nr_pages,struct page **pages) { -struct folio **folios; -long i, ret; +struct pages_or_folios pofs = { + .pages = pages, + .has_folios = false, + .nr_entries = nr_pages, + }; - folios =kmalloc_array (nr_pages,sizeof (*folios),GFP_KERNEL ); -if (!folios) { -unpin_user_pages (pages, nr_pages); -return -ENOMEM ; - } - -for (i =0 ; i < nr_pages; i++) - folios[i] =page_folio (pages[i]); - - ret =check_and_migrate_movable_folios (nr_pages, folios); - -kfree (folios); -return ret; +return check_and_migrate_movable_pages_or_folios (&pofs); }#else static long check_and_migrate_movable_pages(unsigned long nr_pages,
Quoted 6 lines · On 11/17/24 12:33 PM, [email protected] wrote:
The patch below does not apply to the 6.11-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.
Quoted 9 lines
To reproduce the conflict and resubmit, you may use the following commands:
git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-6.11.y
git checkoutFETCH_HEAD
git cherry-pick -x94efde1d15399f5c88e576923db9bcd422d217f2
# <resolve conflicts, build, test, etc.>
git commit -s
git send-email --to '<[email protected]>' --in-reply-to '2024111754-stamina-flyer-1e05@gregkh' --subject-prefix 'PATCH 6.11.y' HEAD^..
It seems that the last hunk didn't apply because it was just too far away, as far as I can tell. I've manually applied it, resulting in the same diffs as the original, and did a quick smoke test (boot and ran mm tests).
Here's the updated version for 6.11.y:
From: John Hubbard <[email protected]>
Date: Sun, 17 Nov 2024 13:08:00 -0800
Subject: [PATCH] mm/gup: avoid an unnecessary allocation call for
FOLL_LONGTERM cases
X-NVConfidentiality: public
Cc: John Hubbard <[email protected]>
commit 53ba78de064b ("mm/gup: introduce check_and_migrate_movable_folios()") created a new constraint on the pin_user_pages*() API family: a potentially large internal allocation must now occur, for FOLL_LONGTERM cases.
A user-visible consequence has now appeared: user space can no longer pin more than 2GB of memory anymore on x86_64. That's because, on a 4KB PAGE_SIZE system, when user space tries to (indirectly, via a device driver that calls pin_user_pages()) pin 2GB, this requires an allocation of a folio pointers array of MAX_PAGE_ORDER size, which is the limit for kmalloc().
In addition to the directly visible effect described above, there is also the problem of adding an unnecessary allocation. The **pages array argument has already been allocated, and there is no need for a redundant **folios array allocation in this case.
Fix this by avoiding the new allocation entirely. This is done by referring to either the original page[i] within **pages, or to the associated folio. Thanks to David Hildenbrand for suggesting this approach and for providing the initial implementation (which I've tested and adjusted slightly) as well.
[[email protected]]: tweaked the patch to apply to linux-stable/6.11.y [[email protected]: whitespace tweak, per David]
Link:
https://lkml.kernel.org/r/[email protected]
[[email protected]: bypass pofs_get_folio(), per Oscar]
Link:
https://lkml.kernel.org/r/[email protected]
--- mm/gup.c | 114 +++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 77 insertions(+), 37 deletions(-)
mm/gup.c +65 -15
@@ -2282,20 +2282,57 @@ struct page *get_dump_page(unsigned long addr)#endif /* CONFIG_ELF_CORE */ #ifdef CONFIG_MIGRATION + +/* +* An array of either pages or folios ("pofs"). Although it may seem tempting to +* avoid this complication, by simply interpreting a list of folios as a list of +* pages, that approach won't work in the longer term, because eventually the +* layouts of struct page and struct folio will become completely different. +* Furthermore, this pof approach avoids excessive page_folio() calls. +*/ +struct pages_or_folios { +union { +struct page **pages; +struct folio **folios; +void **entries; + }; +bool has_folios; +long nr_entries; +}; + +static struct folio *pofs_get_folio (struct pages_or_folios *pofs,long i) +{ +if (pofs->has_folios) +return pofs->folios[i]; +return page_folio (pofs->pages[i]); +} + +static void pofs_clear_entry (struct pages_or_folios *pofs,long i) +{ + pofs->entries[i] =NULL ; +} + +static void pofs_unpin (struct pages_or_folios *pofs) +{ +if (pofs->has_folios) +unpin_folios (pofs->folios, pofs->nr_entries); +else +unpin_user_pages (pofs->pages, pofs->nr_entries); +} +/* * Returns the number of collected folios. Return value is always >= 0. */ static unsigned long collect_longterm_unpinnable_folios ( -struct list_head *movable_folio_list, -unsigned long nr_folios, -struct folio **folios) +struct list_head *movable_folio_list, +struct pages_or_folios *pofs) {unsigned long i, collected =0 ;struct folio *prev_folio =NULL ;bool drain_allow = true; -for (i =0 ; i < nr_folios; i++) { -struct folio *folio = folios[i]; +for (i =0 ; i < pofs->nr_entries; i++) { +struct folio *folio =pofs_get_folio (pofs, i);if (folio == prev_folio)continue ;
@@ -2336,16 +2373,15 @@ static unsigned long collect_longterm_unpinnable_folios(* Returns -EAGAIN if all folios were successfully migrated or -errno for * failure (or partial success). */ -static int migrate_longterm_unpinnable_folios ( -struct list_head *movable_folio_list, -unsigned long nr_folios, -struct folio **folios) +static int +migrate_longterm_unpinnable_folios (struct list_head *movable_folio_list, +struct pages_or_folios *pofs) {int ret;unsigned long i; -for (i =0 ; i < nr_folios; i++) { -struct folio *folio = folios[i]; +for (i =0 ; i < pofs->nr_entries; i++) { +struct folio *folio =pofs_get_folio (pofs, i);if (folio_is_device_coherent (folio)) {/*
@@ -2353,7 +2389,7 @@ static int migrate_longterm_unpinnable_folios(* convert the pin on the source folio to a normal * reference. */ - folios[i] =NULL ; +pofs_clear_entry (pofs, i);folio_get (folio);gup_put_folio (folio,1 ,FOLL_PIN );
@@ -2372,8 +2408,8 @@ static int migrate_longterm_unpinnable_folios(* calling folio_isolate_lru() which takes a reference so the * folio won't be freed if it's migrating. */ -unpin_folio (folios[i]); - folios[i] =NULL ; +unpin_folio (folio); +pofs_clear_entry (pofs, i); }if (!list_empty (movable_folio_list)) {
@@ -2396,12 +2432,26 @@ static int migrate_longterm_unpinnable_folios(return -EAGAIN ; err: -unpin_folios (folios, nr_folios); +pofs_unpin (pofs);putback_movable_pages (movable_folio_list);return ret; } +static long +check_and_migrate_movable_pages_or_folios (struct pages_or_folios *pofs) +{ +LIST_HEAD (movable_folio_list); +unsigned long collected; + + collected =collect_longterm_unpinnable_folios (&movable_folio_list, + pofs); +if (!collected) +return 0 ; + +return migrate_longterm_unpinnable_folios (&movable_folio_list, pofs); +} +/* * Check whether all folios are *allowed* to be pinned indefinitely
(longterm).
* Rather confusingly, all folios in the range are required to be
pinned via
@@ -2421,16 +2471,13 @@ static int migrate_longterm_unpinnable_folios(
static long check_and_migrate_movable_folios(unsigned long nr_folios,
struct folio **folios)
{
- unsigned long collected;
- LIST_HEAD(movable_folio_list);
+ struct pages_or_folios pofs = {
+ .folios = folios,
+ .has_folios = true,
+ .nr_entries = nr_folios,
+ };
- collected = collect_longterm_unpinnable_folios(&movable_folio_list,
- nr_folios, folios);
- if (!collected)
- return 0;
-
- return migrate_longterm_unpinnable_folios(&movable_folio_list,
- nr_folios, folios);
+ return check_and_migrate_movable_pages_or_folios(&pofs);
}
/*
@@ -2442,20 +2489,13 @@ static long check_and_migrate_movable_folios(unsigned long nr_folios,
static long check_and_migrate_movable_pages(unsigned long nr_pages,
struct page **pages)
{
- struct folio **folios;
- long i, ret;
+ struct pages_or_folios pofs = {
+ .pages = pages,
+ .has_folios = false,
+ .nr_entries = nr_pages,
+ };
- folios = kmalloc_array(nr_pages, sizeof(*folios), GFP_KERNEL);
- if (!folios)
- return -ENOMEM;
-
- for (i = 0; i < nr_pages; i++)
- folios[i] = page_folio(pages[i]);
-
- ret = check_and_migrate_movable_folios(nr_pages, folios);
-
- kfree(folios);
- return ret;
+ return check_and_migrate_movable_pages_or_folios(&pofs);
}
#else
static long check_and_migrate_movable_pages(unsigned long nr_pages,
Signature · 1 line
2.47.0
Greg KH
maintainerQuoted 95 lines · John Hubbard
Quoted 6 lines · On 11/17/24 12:33 PM, [email protected] wrote:
The patch below does not apply to the 6.11-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.Quoted 9 lines
To reproduce the conflict and resubmit, you may use the following commands:
git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-6.11.y
git checkoutFETCH_HEAD
git cherry-pick -x94efde1d15399f5c88e576923db9bcd422d217f2
# <resolve conflicts, build, test, etc.>
git commit -s
git send-email --to '<[email protected]>' --in-reply-to '2024111754-stamina-flyer-1e05@gregkh' --subject-prefix 'PATCH 6.11.y' HEAD^..It seems that the last hunk didn't apply because it was just too far away, as far as I can tell. I've manually applied it, resulting in the same diffs as the original, and did a quick smoke test (boot and ran mm tests).
Here's the updated version for 6.11.y:
From: John Hubbard <[email protected]>
Date: Sun, 17 Nov 2024 13:08:00 -0800
Subject: [PATCH] mm/gup: avoid an unnecessary allocation call for
FOLL_LONGTERMcases
X-NVConfidentiality: public
Cc: John Hubbard <[email protected]>commit
53ba78de064b("mm/gup: introducecheck_and_migrate_movable_folios()") created a new constraint on the pin_user_pages*() API family: a potentially large internal allocation must now occur, forFOLL_LONGTERMcases.A user-visible consequence has now appeared: user space can no longer pin more than 2GB of memory anymore on
x86_64. That's because, on a 4KBPAGE_SIZEsystem, when user space tries to (indirectly, via a device driver that callspin_user_pages()) pin 2GB, this requires an allocation of a folio pointers array ofMAX_PAGE_ORDERsize, which is the limit forkmalloc().In addition to the directly visible effect described above, there is also the problem of adding an unnecessary allocation. The **pages array argument has already been allocated, and there is no need for a redundant **folios array allocation in this case.
Fix this by avoiding the new allocation entirely. This is done by referring to either the original page[i] within **pages, or to the associated folio. Thanks to David Hildenbrand for suggesting this approach and for providing the initial implementation (which I've tested and adjusted slightly) as well.
[[email protected]]: tweaked the patch to apply to linux-stable/6.11.y [[email protected]: whitespace tweak, per David]
Link:
https://lkml.kernel.org/r/[email protected]
[[email protected]: bypasspofs_get_folio(), per Oscar]
Link:
https://lkml.kernel.org/r/[email protected]Fixes 53ba78de064b ("mm/gup: introduce check_and_migrate_movable_folios()")Signed-off-by John Hubbard <[email protected]>Suggested-by David Hildenbrand <[email protected]>Reviewed-by Oscar Salvador <[email protected]>Cc Vivek Kasireddy <[email protected]>Cc Dave Airlie <[email protected]>Cc Gerd Hoffmann <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Jason Gunthorpe <[email protected]>Cc Peter Xu <[email protected]>Cc Arnd Bergmann <[email protected]>Cc Daniel Vetter <[email protected]>Cc Dongwon Kim <[email protected]>Cc Hugh Dickins <[email protected]>Cc Junxiao Chang <[email protected]>Cc <[email protected]>Signed-off-by Andrew Morton <[email protected]>--- mm/gup.c | 114 +++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 77 insertions(+), 37 deletions(-)mm/gup.c +3
-0@@ -2282,20 +2282,57 @@ struct page *get_dump_page(unsigned long addr)#endif /* CONFIG_ELF_CORE */ #ifdef CONFIG_MIGRATION + +/* +* An array of either pages or folios ("pofs"). Although it may seem
Quoted 8 lines
tempting to
+ * avoid this complication, by simply interpreting a list of folios as a list of
+ * pages, that approach won't work in the longer term, because eventually the
+ * layouts of struct page and struct folio will become completely different.
+ * Furthermore, this pof approach avoids excessivepage_folio()calls.
Patch is line-wrapped :(
Can you resend it in a format I can apply it in?
thanks,
greg k-h
John Hubbard
reviewerQuoted 2 lines · Greg KH
Quoted 1 line · John Hubbard
On 11/17/24 12:33 PM, [email protected] wrote:
...
Quoted 7 lines
Patch is line-wrapped :(
Can you resend it in a format I can apply it in?
thanks,
greg k-h
OK, I'm going to use git-send-email after all. Thunderbird is clearly not The Way here. Just a sec. :)
thanks,
Signature · 1 line
John Hubbard
John Hubbard
reviewerQuoted 6 lines · John Hubbard
Quoted 2 lines · Greg KH
Quoted 1 line · John Hubbard
On 11/17/24 12:33 PM, [email protected] wrote:
...
Quoted 2 lines
Patch is line-wrapped :(
Quoted 9 lines
Quoted 5 lines
Can you resend it in a format I can apply it in?
thanks,
greg k-h
OK, I'm going to use git-send-email after all. Thunderbird is clearly not The Way here. Just a sec. :)
OK, I've sent the fixed up version, here:
https://lore.kernel.org/[email protected]
thanks,
Signature · 1 line
John Hubbard
Greg KH
maintainerQuoted 13 lines · John Hubbard
Quoted 12 lines · John Hubbard
Quoted 2 lines · Greg KH
Quoted 1 line · John Hubbard
On 11/17/24 12:33 PM, [email protected] wrote:
...
Quoted 7 lines
Patch is line-wrapped :(
Can you resend it in a format I can apply it in?
thanks,
greg k-h
Quoted 7 lines
Quoted 3 lines
OK, I'm going to use git-send-email after all. Thunderbird is clearly not The Way here. Just a sec. :)
OK, I've sent the fixed up version, here:
That worked, thanks!