mm: prohibit the last subpage from reusing the entire large folio
Subsystems (2)
PATCH v2 1/1 mm: prohibit the last subpage from reusing the entire large folio
From: Barry Song <[email protected]>
In a Copy-on-Write (CoW) scenario, the last subpage will reuse the entire large folio, resulting in the waste of (nr_pages - 1) pages. This wasted memory remains allocated until it is either unmapped or memory reclamation occurs.
The following small program can serve as evidence of this behavior
main()
{
#define SIZE 1024 * 1024 * 1024UL
void *p = malloc(SIZE);
memset(p, 0x11, SIZE);
if (fork() == 0)
_exit(0);
memset(p, 0x12, SIZE);
printf("done\n");
while(1);}
For example, using a 1024KiB mTHP by:
echo always > /sys/kernel/mm/transparent_hugepage/hugepages-1024kB/enabled
(1) w/o the patch, it takes 2GiB,
Before running the test program,
/ # free -m
total used free shared buff/cache available
Mem: 5754 84 5692 0 17 5669
Swap: 0 0 0
/ # /a.out &
/ # done
After running the test program,
/ # free -m
total used free shared buff/cache available
Mem: 5754 2149 3627 0 19 3605
Swap: 0 0 0
(2) w/ the patch, it takes 1GiB only,
Before running the test program,
/ # free -m
total used free shared buff/cache available
Mem: 5754 89 5687 0 17 5664
Swap: 0 0 0
/ # /a.out &
/ # done
After running the test program,
/ # free -m
total used free shared buff/cache available Mem: 5754 1122 4655 0 17 4632
Swap: 0 0 0
This patch migrates the last subpage to a small folio and immediately returns the large folio to the system. It benefits both memory availability and anti-fragmentation.
---
-v2:
* return at the 1st beginning for a large folio according to David's comment, thanks!
-v1:
https://lore.kernel.org/linux-mm/[email protected]/
mm/memory.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
mm/memory.c +10 -0
@@ -3498,6 +3498,16 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio)static bool wp_can_reuse_anon_folio (struct folio *folio,struct vm_area_struct *vma) { +/* +* We could currently only reuse a subpage of a large folio if no +* other subpages of the large folios are still mapped. However, +* let's just consistently not reuse subpages even if we could +* reuse in that scenario, and give back a large folio a bit +* sooner. +*/ +if (folio_test_large (folio)) +return false; +/* * We have to verify under folio lock: these early checks are * just an optimization to avoid locking the folio and freeing
Signature · 1 line
2.34.1
Quoted 54 lines · Barry Song
From: Barry Song <[email protected]>
In a Copy-on-Write (CoW) scenario, the last subpage will reuse the entire large folio, resulting in the waste of (
nr_pages- 1) pages. This wasted memory remains allocated until it is either unmapped or memory reclamation occurs.The following small program can serve as evidence of this behavior
main() { #define SIZE 1024 * 1024 * 1024UL void *p = malloc(SIZE); memset(p, 0x11, SIZE); if (fork() == 0) _exit(0); memset(p, 0x12, SIZE); printf("done\n"); while(1); }For example, using a 1024KiB mTHP by:
echo always >/sys/kernel/mm/transparent_hugepage/hugepages-1024kB/enabled(1) w/o the patch, it takes 2GiB,
Before running the test program,
/ # free -m
total used free shared buff/cache available
Mem: 5754 84 5692 0 17 5669
Swap: 0 0 0/ # /a.out &
/ # doneAfter running the test program,
/ # free -m
total used free shared buff/cache available
Mem: 5754 2149 3627 0 19 3605
Swap: 0 0 0(2) w/ the patch, it takes 1GiB only,
Before running the test program,
/ # free -m
total used free shared buff/cache available
Mem: 5754 89 5687 0 17 5664
Swap: 0 0 0/ # /a.out &
/ # doneAfter running the test program,
/ # free -m
Quoted 7 lines
total used free shared buff/cache available Mem: 5754 1122 4655 0 17 4632Swap: 0 0 0
This patch migrates the last subpage to a small folio and immediately returns the large folio to the system. It benefits both memory availability and anti-fragmentation.
It might be controversial optimization, and as Ryan said, there, are likely other cases where we'd want to migrate off-of a thp if possible earlier.
But I like that it just handles large folios now in a consistent way for the time being.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 67 lines · David Hildenbrand
Quoted 66 lines · Barry Song
From: Barry Song <[email protected]>
In a Copy-on-Write (CoW) scenario, the last subpage will reuse the entire large folio, resulting in the waste of (
nr_pages- 1) pages. This wasted memory remains allocated until it is either unmapped or memory reclamation occurs.The following small program can serve as evidence of this behavior
main()
{
#define SIZE 1024 * 1024 * 1024UL
void *p = malloc(SIZE);
memset(p, 0x11, SIZE);
if (fork()== 0)
_exit(0);
memset(p, 0x12, SIZE);
printf("done\n");
while(1);
}For example, using a 1024KiB mTHP by:
echo always >/sys/kernel/mm/transparent_hugepage/hugepages-1024kB/enabled(1) w/o the patch, it takes 2GiB,
Before running the test program,
/ # free -m
total used free shared buff/cache available
Mem: 5754 84 5692 0 17 5669
Swap: 0 0 0/ # /a.out &
/ # doneAfter running the test program,
/ # free -m
total used free shared buff/cache available
Mem: 5754 2149 3627 0 19 3605
Swap: 0 0 0(2) w/ the patch, it takes 1GiB only,
Before running the test program,
/ # free -m
total used free shared buff/cache available
Mem: 5754 89 5687 0 17 5664
Swap: 0 0 0/ # /a.out &
/ # doneAfter running the test program,
/ # free -m
total used free shared buff/cache available
Mem: 5754 1122 4655 0 17 4632
Swap: 0 0 0
Quoted 6 lines
Quoted 3 lines
This patch migrates the last subpage to a small folio and immediately returns the large folio to the system. It benefits both memory availability and anti-fragmentation.
It might be controversial optimization, and as Ryan said, there, are likely other cases where we'd want to migrate off-of a thp if possible earlier.
Personally, I think there might also be cases where you want to copy/reuse the entire large folio. If you're application is using 16K THPs perhaps it's a bigger win to just treat it like a base page? I expect the cost/benefit will change as the THP size increases?
I know we have previously talked about using a khugepaged-like mechanism to re-collapse after CoW, but for the smaller sizes maybe that's just a lot more effort?
Quoted 3 lines
But I like that it just handles large folios now in a consistent way for the time being.
Yes agreed.
Quoted 3 lines
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
This patch migrates the last subpage to a small folio and immediately returns the large folio to the system. It benefits both memory availability and anti-fragmentation.
It might be controversial optimization, and as Ryan said, there, are likely other cases where we'd want to migrate off-of a thp if possible earlier.
Personally, I think there might also be cases where you want to copy/reuse the entire large folio. If you're application is using 16K THPs perhaps it's a bigger win to just treat it like a base page? I expect the cost/benefit will change as the THP size increases?
Yes, I think for small folios (i.e., 16KiB) it will be rather easy to make a decision. The larger the folio, the larger the page fault latency due to scanning, copying, modifying, which can easily turn undesirable.
At least when it comes to page reuse, I have some simple backup plans for small folios if I won't be able to make progress with my other approach. For larger folios, it won't really work/be desirable, though.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 12 lines · David Hildenbrand
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
This patch migrates the last subpage to a small folio and immediately returns the large folio to the system. It benefits both memory availability and anti-fragmentation.
It might be controversial optimization, and as Ryan said, there, are likely other cases where we'd want to migrate off-of a thp if possible earlier.
Personally, I think there might also be cases where you want to copy/reuse the entire large folio. If you're application is using 16K THPs perhaps it's a bigger win to just treat it like a base page? I expect the cost/benefit will change as the THP size increases?
Quoted 6 lines
Yes, I think for small folios (i.e., 16KiB) it will be rather easy to make a decision. The larger the folio, the larger the page fault latency due to scanning, copying, modifying, which can easily turn undesirable.
At least when it comes to page reuse, I have some simple backup plans for small folios if I won't be able to make progress with my other approach.
Do you mean "small large folios" here? i.e. order >= 1? If so, great!
For larger
Quoted 2 lines
folios, it won't really work/be desirable, though.
Quoted 13 lines · Ryan Roberts
Quoted 12 lines · David Hildenbrand
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
This patch migrates the last subpage to a small folio and immediately returns the large folio to the system. It benefits both memory availability and anti-fragmentation.
It might be controversial optimization, and as Ryan said, there, are likely other cases where we'd want to migrate off-of a thp if possible earlier.
Personally, I think there might also be cases where you want to copy/reuse the entire large folio. If you're application is using 16K THPs perhaps it's a bigger win to just treat it like a base page? I expect the cost/benefit will change as the THP size increases?
Quoted 8 lines
Quoted 6 lines
Yes, I think for small folios (i.e., 16KiB) it will be rather easy to make a decision. The larger the folio, the larger the page fault latency due to scanning, copying, modifying, which can easily turn undesirable.
At least when it comes to page reuse, I have some simple backup plans for small folios if I won't be able to make progress with my other approach.
Do you mean "small large folios" here? i.e. order >= 1? If so, great!
*smaller*, yes :)
Signature · 3 lines
Cheers, David / dhildenb