Skip to contentSkip to conversation

mm: prohibit the last subpage from reusing the entire large folio

PATCH Applied Barry Song 3 participants 6 messages Mar 8, 2024

Subsystems (2)
lore
Share & tools
lore

Barry Song

author
#0 lore raw

PATCH v2 1/1 mm: prohibit the last subpage from reusing the entire large folio

From: Barry Song <[email protected]>

In a Copy-on-Write (CoW) scenario, the last subpage will reuse the entire large folio, resulting in the waste of (nr_pages - 1) pages. This wasted memory remains allocated until it is either unmapped or memory reclamation occurs.

The following small program can serve as evidence of this behavior

main()
{

 #define SIZE 1024 * 1024 * 1024UL
         void *p = malloc(SIZE);
         memset(p, 0x11, SIZE);
         if (fork() == 0)
                 _exit(0);
         memset(p, 0x12, SIZE);
         printf("done\n");
         while(1);

}

For example, using a 1024KiB mTHP by:
echo always > /sys/kernel/mm/transparent_hugepage/hugepages-1024kB/enabled

(1) w/o the patch, it takes 2GiB,

Before running the test program,
/ # free -m
total used free shared buff/cache available
Mem: 5754 84 5692 0 17 5669
Swap: 0 0 0

/ # /a.out &
/ # done

After running the test program,
/ # free -m
total used free shared buff/cache available
Mem: 5754 2149 3627 0 19 3605
Swap: 0 0 0

(2) w/ the patch, it takes 1GiB only,

Before running the test program,
/ # free -m
total used free shared buff/cache available
Mem: 5754 89 5687 0 17 5664
Swap: 0 0 0

/ # /a.out &
/ # done

After running the test program,
/ # free -m

                total        used        free      shared  buff/cache   available
 Mem:            5754        1122        4655           0          17        4632

Swap: 0 0 0

This patch migrates the last subpage to a small folio and immediately returns the large folio to the system. It benefits both memory availability and anti-fragmentation.

Cc David Hildenbrand <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc Lance Yang <[email protected]>

---
-v2:
* return at the 1st beginning for a large folio according to David's comment, thanks!
-v1:
https://lore.kernel.org/linux-mm/[email protected]/

 mm/memory.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
mm/memory.c +10 -0
index e17669d4f72f..f2bc6dd15eb8 100644
@@ -3498,6 +3498,16 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio)
 static bool wp_can_reuse_anon_folio(struct folio *folio,
 				    struct vm_area_struct *vma)
 {
+	/*
+	 * We could currently only reuse a subpage of a large folio if no
+	 * other subpages of the large folios are still mapped. However,
+	 * let's just consistently not reuse subpages even if we could
+	 * reuse in that scenario, and give back a large folio a bit
+	 * sooner.
+	 */
+	if (folio_test_large(folio))
+		return false;
+
 	/*
 	 * We have to verify under folio lock: these early checks are
 	 * just an optimization to avoid locking the folio and freeing
Signature · 1 line
2.34.1
Acked-by on v2 1/1
#1 lore raw
Quoted 54 lines · Barry Song

From: Barry Song <[email protected]>

In a Copy-on-Write (CoW) scenario, the last subpage will reuse the entire large folio, resulting in the waste of (nr_pages - 1) pages. This wasted memory remains allocated until it is either unmapped or memory reclamation occurs.

The following small program can serve as evidence of this behavior

  main()
  {
  #define SIZE 1024 * 1024 * 1024UL
          void *p = malloc(SIZE);
          memset(p, 0x11, SIZE);
          if (fork() == 0)
                  _exit(0);
          memset(p, 0x12, SIZE);
          printf("done\n");
          while(1);
  }

For example, using a 1024KiB mTHP by:
echo always > /sys/kernel/mm/transparent_hugepage/hugepages-1024kB/enabled

(1) w/o the patch, it takes 2GiB,

Before running the test program,
/ # free -m
total used free shared buff/cache available
Mem: 5754 84 5692 0 17 5669
Swap: 0 0 0

/ # /a.out &
/ # done

After running the test program,
/ # free -m
total used free shared buff/cache available
Mem: 5754 2149 3627 0 19 3605
Swap: 0 0 0

(2) w/ the patch, it takes 1GiB only,

Before running the test program,
/ # free -m
total used free shared buff/cache available
Mem: 5754 89 5687 0 17 5664
Swap: 0 0 0

/ # /a.out &
/ # done

After running the test program,
/ # free -m

Quoted 7 lines
                 total        used        free      shared  buff/cache   available
  Mem:            5754        1122        4655           0          17        4632

Swap: 0 0 0

This patch migrates the last subpage to a small folio and immediately returns the large folio to the system. It benefits both memory availability and anti-fragmentation.

It might be controversial optimization, and as Ryan said, there, are likely other cases where we'd want to migrate off-of a thp if possible earlier.

But I like that it just handles large folios now in a consistent way for the time being.

Signature · 3 lines
Cheers,

David / dhildenb
#2 lore raw
Quoted 67 lines · David Hildenbrand
Quoted 66 lines · Barry Song

From: Barry Song <[email protected]>

In a Copy-on-Write (CoW) scenario, the last subpage will reuse the entire large folio, resulting in the waste of (nr_pages - 1) pages. This wasted memory remains allocated until it is either unmapped or memory reclamation occurs.

The following small program can serve as evidence of this behavior

  main()
  {
  #define SIZE 1024 * 1024 * 1024UL
          void *p = malloc(SIZE);
          memset(p, 0x11, SIZE);
          if (fork() == 0)
                  _exit(0);
          memset(p, 0x12, SIZE);
          printf("done\n");
          while(1);
  }

For example, using a 1024KiB mTHP by:
  echo always > /sys/kernel/mm/transparent_hugepage/hugepages-1024kB/enabled

(1) w/o the patch, it takes 2GiB,

Before running the test program,
  / # free -m
                 total        used        free      shared  buff/cache available
  Mem:            5754          84        5692           0          17 5669
  Swap:              0           0           0

  / # /a.out &
  / # done

After running the test program,
  / # free -m
                  total        used        free      shared  buff/cache available
  Mem:            5754        2149        3627           0          19 3605
  Swap:              0           0           0

(2) w/ the patch, it takes 1GiB only,

Before running the test program,
  / # free -m
                  total        used        free      shared  buff/cache available
  Mem:            5754          89        5687           0          17 5664
  Swap:              0           0           0

  / # /a.out &
  / # done

After running the test program,
  / # free -m
                 total        used        free      shared  buff/cache available
  Mem:            5754        1122        4655           0          17 4632
  Swap:              0           0           0

Quoted 6 lines
Quoted 3 lines

This patch migrates the last subpage to a small folio and immediately returns the large folio to the system. It benefits both memory availability and anti-fragmentation.

It might be controversial optimization, and as Ryan said, there, are likely other cases where we'd want to migrate off-of a thp if possible earlier.

Personally, I think there might also be cases where you want to copy/reuse the entire large folio. If you're application is using 16K THPs perhaps it's a bigger win to just treat it like a base page? I expect the cost/benefit will change as the THP size increases?

I know we have previously talked about using a khugepaged-like mechanism to re-collapse after CoW, but for the smaller sizes maybe that's just a lot more effort?

Quoted 3 lines

But I like that it just handles large folios now in a consistent way for the time being.

Yes agreed.

Quoted 3 lines
#3 lore raw
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines

This patch migrates the last subpage to a small folio and immediately returns the large folio to the system. It benefits both memory availability and anti-fragmentation.

It might be controversial optimization, and as Ryan said, there, are likely other cases where we'd want to migrate off-of a thp if possible earlier.

Personally, I think there might also be cases where you want to copy/reuse the entire large folio. If you're application is using 16K THPs perhaps it's a bigger win to just treat it like a base page? I expect the cost/benefit will change as the THP size increases?

Yes, I think for small folios (i.e., 16KiB) it will be rather easy to make a decision. The larger the folio, the larger the page fault latency due to scanning, copying, modifying, which can easily turn undesirable.

At least when it comes to page reuse, I have some simple backup plans for small folios if I won't be able to make progress with my other approach. For larger folios, it won't really work/be desirable, though.

Signature · 3 lines
Cheers,

David / dhildenb
#4 lore raw
Quoted 12 lines · David Hildenbrand
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines

This patch migrates the last subpage to a small folio and immediately returns the large folio to the system. It benefits both memory availability and anti-fragmentation.

It might be controversial optimization, and as Ryan said, there, are likely other cases where we'd want to migrate off-of a thp if possible earlier.

Personally, I think there might also be cases where you want to copy/reuse the entire large folio. If you're application is using 16K THPs perhaps it's a bigger win to just treat it like a base page? I expect the cost/benefit will change as the THP size increases?

Quoted 6 lines

Yes, I think for small folios (i.e., 16KiB) it will be rather easy to make a decision. The larger the folio, the larger the page fault latency due to scanning, copying, modifying, which can easily turn undesirable.

At least when it comes to page reuse, I have some simple backup plans for small folios if I won't be able to make progress with my other approach.

Do you mean "small large folios" here? i.e. order >= 1? If so, great!

For larger

Quoted 2 lines

folios, it won't really work/be desirable, though.

#5 lore raw
Quoted 13 lines · Ryan Roberts
Quoted 12 lines · David Hildenbrand
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines

This patch migrates the last subpage to a small folio and immediately returns the large folio to the system. It benefits both memory availability and anti-fragmentation.

It might be controversial optimization, and as Ryan said, there, are likely other cases where we'd want to migrate off-of a thp if possible earlier.

Personally, I think there might also be cases where you want to copy/reuse the entire large folio. If you're application is using 16K THPs perhaps it's a bigger win to just treat it like a base page? I expect the cost/benefit will change as the THP size increases?

Quoted 8 lines
Quoted 6 lines

Yes, I think for small folios (i.e., 16KiB) it will be rather easy to make a decision. The larger the folio, the larger the page fault latency due to scanning, copying, modifying, which can easily turn undesirable.

At least when it comes to page reuse, I have some simple backup plans for small folios if I won't be able to make progress with my other approach.

Do you mean "small large folios" here? i.e. order >= 1? If so, great!

*smaller*, yes :)

Signature · 3 lines
Cheers,

David / dhildenb