Skip to contentSkip to conversation

Hugepage program taking forever to exit

PATCH Jens Axboe 4 participants 9 messages Sep 10, 2024 – Sep 11, 2024

Subsystems (2)
lore
Share & tools
lore

Jens Axboe

author
#0 lore raw

Hi,

Investigating another issue, I wrote the following simple program that allocates and faults in 500 1GB huge pages, and then registers them with io_uring. Each step is timed:

Got 500 huge pages (each 1024MB) in 0 msec
Faulted in 500 huge pages in 38632 msec
Registered 500 pages in 867 msec

and as expected, faulting in the pages takes (by far) the longest. From the above, you'd also expect the total runtime to be around ~39 seconds. But it is not... In fact it takes 82 seconds in total for this program to have exited. Looking at why, I see:

[<0>] __wait_rcu_gp+0x12b/0x160
[<0>] synchronize_rcu_normal.part.0+0x2a/0x30
[<0>] hugetlb_vmemmap_restore_folios+0x22/0xe0
[<0>] update_and_free_pages_bulk+0x4c/0x220
[<0>] return_unused_surplus_pages+0x80/0xa0
[<0>] hugetlb_acct_memory.part.0+0x2dd/0x3b0
[<0>] hugetlb_vm_op_close+0x160/0x180
[<0>] remove_vma+0x20/0x60
[<0>] exit_mmap+0x199/0x340
[<0>] mmput+0x49/0x110
[<0>] do_exit+0x261/0x9b0
[<0>] do_group_exit+0x2c/0x80
[<0>] __x64_sys_exit_group+0x14/0x20
[<0>] x64_sys_call+0x714/0x720
[<0>] do_syscall_64+0x5b/0x160
[<0>] entry_SYSCALL_64_after_hwframe+0x4b/0x53

and yes, it does look like the program is mostly idle for most of the time while returning these huge pages. It's also telling us exactly why we're just sitting idle - RCU grace period.

The below quick change means the runtime of the program is pretty much just the time it takes to execute the parts of it, as you can see from the full output after the change:

axboe@r7525 ~> time sudo ./reg-huge
Got 500 huge pages (each 1024MB) in 0 msec
Faulted in 500 huge pages in 38632 msec
Registered 500 pages in 867 msec

________________________________________________________

Executed in   39.53 secs      fish           external
   usr time    4.88 millis  238.00 micros    4.64 millis
   sys time    0.00 millis    0.00 micros    0.00 millis

where 38632+876 == 39.51s.

Looks like this was introduced by:

commit bd225530a4c717714722c3731442b78954c765b3

Author: Yu Zhao <[email protected]>
Date:   Thu Jun 27 16:27:05 2024 -0600

mm/hugetlb_vmemmap: fix race with speculative PFN walkers

mm/hugetlb_vmemmap.c +1 -1
index 0c3f56b3578e..95f6ad8f8232 100644
@@ -517,7 +517,7 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h,
 	long ret = 0;
 
 	/* avoid writes from page_ref_add_unless() while unfolding vmemmap */
-	synchronize_rcu();
+	synchronize_rcu_expedited();
 
 	list_for_each_entry_safe(folio, t_folio, folio_list, lru) {
 		if (folio_test_hugetlb_vmemmap_optimized(folio)) {
Signature · 1 line
Jens Axboe
#1 lore raw
Quoted 15 lines · Jens Axboe

Hi,

Investigating another issue, I wrote the following simple program that allocates and faults in 500 1GB huge pages, and then registers them with io_uring. Each step is timed:

Got 500 huge pages (each 1024MB) in 0 msec
Faulted in 500 huge pages in 38632 msec
Registered 500 pages in 867 msec

and as expected, faulting in the pages takes (by far) the longest. From the above, you'd also expect the total runtime to be around ~39 seconds. But it is not... In fact it takes 82 seconds in total for this program to have exited. Looking at why, I see:

Quoted 16 lines
[<0>] __wait_rcu_gp+0x12b/0x160
[<0>] synchronize_rcu_normal.part.0+0x2a/0x30
[<0>] hugetlb_vmemmap_restore_folios+0x22/0xe0
[<0>] update_and_free_pages_bulk+0x4c/0x220
[<0>] return_unused_surplus_pages+0x80/0xa0
[<0>] hugetlb_acct_memory.part.0+0x2dd/0x3b0
[<0>] hugetlb_vm_op_close+0x160/0x180
[<0>] remove_vma+0x20/0x60
[<0>] exit_mmap+0x199/0x340
[<0>] mmput+0x49/0x110
[<0>] do_exit+0x261/0x9b0
[<0>] do_group_exit+0x2c/0x80
[<0>] __x64_sys_exit_group+0x14/0x20
[<0>] x64_sys_call+0x714/0x720
[<0>] do_syscall_64+0x5b/0x160
[<0>] entry_SYSCALL_64_after_hwframe+0x4b/0x53

Yeah, this looks wrong to me:

void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list)

{
	struct folio *folio;
	LIST_HEAD(vmemmap_pages);

	list_for_each_entry(folio, folio_list, lru) {
		int ret = hugetlb_vmemmap_split_folio(h, folio);

		/*
		 * Spliting the PMD requires allocating a page, thus lets fail
		 * early once we encounter the first OOM. No point in retrying
		 * as it can be dynamically done on remap with the memory
		 * we get back from the vmemmap deduplication.
		 */
		if (ret == -ENOMEM)
			break;
	}

	flush_tlb_all();

	/* avoid writes from page_ref_add_unless() while folding vmemmap */
	synchronize_rcu();

	list_for_each_entry(folio, folio_list, lru) {
		int ret;

		ret = __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages,
						       VMEMMAP_REMAP_NO_TLB_FLUSH);

		/*
		 * Pages to be freed may have been accumulated.  If we
		 * encounter an ENOMEM,  free what we have and try again.
		 * This can occur in the case that both spliting fails
		 * halfway and head page allocation also failed. In this
		 * case __hugetlb_vmemmap_optimize_folio() would free memory
		 * allowing more vmemmap remaps to occur.
		 */
		if (ret == -ENOMEM && !list_empty(&vmemmap_pages)) {
			flush_tlb_all();
			free_vmemmap_page_list(&vmemmap_pages);
			INIT_LIST_HEAD(&vmemmap_pages);
			__hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages,
							 VMEMMAP_REMAP_NO_TLB_FLUSH);
		}
	}

	flush_tlb_all();
	free_vmemmap_page_list(&vmemmap_pages);
}

If you don't have HVO enabled, then hugetlb_vmemmap_split_folio() does nothing. And __hugetlb_vmemmap_optimize_folio() also does nothing, leaving &vmemmap_pages empty and free_vmemmap_page_list() a nop.

So what's left is: it flushes the TLB twice and waits for RCU. What for exactly?

The same is true for hugetlb_vmemmap_optimize_folio() and the corresponding split function, which waits for RCU on every page being allocated and freed, even if the vmemmap is left alone.

Surely all those RCU waits and tlb flushes should be guarded by whether the HVO is actually enabled, no?

#2 lore raw
In reply to Jens Axboe
Quoted 51 lines · Jens Axboe

Hi,

Investigating another issue, I wrote the following simple program that allocates and faults in 500 1GB huge pages, and then registers them with io_uring. Each step is timed:

Got 500 huge pages (each 1024MB) in 0 msec
Faulted in 500 huge pages in 38632 msec
Registered 500 pages in 867 msec

and as expected, faulting in the pages takes (by far) the longest. From the above, you'd also expect the total runtime to be around ~39 seconds. But it is not... In fact it takes 82 seconds in total for this program to have exited. Looking at why, I see:

[<0>] __wait_rcu_gp+0x12b/0x160
[<0>] synchronize_rcu_normal.part.0+0x2a/0x30
[<0>] hugetlb_vmemmap_restore_folios+0x22/0xe0
[<0>] update_and_free_pages_bulk+0x4c/0x220
[<0>] return_unused_surplus_pages+0x80/0xa0
[<0>] hugetlb_acct_memory.part.0+0x2dd/0x3b0
[<0>] hugetlb_vm_op_close+0x160/0x180
[<0>] remove_vma+0x20/0x60
[<0>] exit_mmap+0x199/0x340
[<0>] mmput+0x49/0x110
[<0>] do_exit+0x261/0x9b0
[<0>] do_group_exit+0x2c/0x80
[<0>] __x64_sys_exit_group+0x14/0x20
[<0>] x64_sys_call+0x714/0x720
[<0>] do_syscall_64+0x5b/0x160
[<0>] entry_SYSCALL_64_after_hwframe+0x4b/0x53

and yes, it does look like the program is mostly idle for most of the time while returning these huge pages. It's also telling us exactly why we're just sitting idle - RCU grace period.

The below quick change means the runtime of the program is pretty much just the time it takes to execute the parts of it, as you can see from the full output after the change:

axboe@r7525 ~> time sudo ./reg-huge
Got 500 huge pages (each 1024MB) in 0 msec
Faulted in 500 huge pages in 38632 msec
Registered 500 pages in 867 msec

________________________________________________________

Executed in   39.53 secs      fish           external
   usr time    4.88 millis  238.00 micros    4.64 millis
   sys time    0.00 millis    0.00 micros    0.00 millis
Quoted 7 lines

where 38632+876 == 39.51s.

Looks like this was introduced by:

commit bd225530a4c717714722c3731442b78954c765b3

Author: Yu Zhao <[email protected]>
Date:   Thu Jun 27 16:27:05 2024 -0600

Fixes are in git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-stable

c2a967f6ab0e mm/hugetlb_vmemmap: don't synchronize_rcu() without HVO c0f398c3b2cf mm/hugetlb_vmemmap: batch HVO work when demoting

Additional improvements from mm-stable that may or may not help your test case:

e98337d11bbd mm/contig_alloc: support __GFP_COMP 463586e9ff39 mm/cma: add cma_{alloc,free}_folio() cf54f310d0d3 mm/hugetlb: use __GFP_COMP for gigantic folios

author
#3 lore raw
Quoted 62 lines · Yu Zhao
Quoted 51 lines · On Tue, Sep 10, 2024 at 12:21?PM Jens Axboe <[email protected]> wrote:

Hi,

Investigating another issue, I wrote the following simple program that allocates and faults in 500 1GB huge pages, and then registers them with io_uring. Each step is timed:

Got 500 huge pages (each 1024MB) in 0 msec
Faulted in 500 huge pages in 38632 msec
Registered 500 pages in 867 msec

and as expected, faulting in the pages takes (by far) the longest. From the above, you'd also expect the total runtime to be around ~39 seconds. But it is not... In fact it takes 82 seconds in total for this program to have exited. Looking at why, I see:

[<0>] __wait_rcu_gp+0x12b/0x160
[<0>] synchronize_rcu_normal.part.0+0x2a/0x30
[<0>] hugetlb_vmemmap_restore_folios+0x22/0xe0
[<0>] update_and_free_pages_bulk+0x4c/0x220
[<0>] return_unused_surplus_pages+0x80/0xa0
[<0>] hugetlb_acct_memory.part.0+0x2dd/0x3b0
[<0>] hugetlb_vm_op_close+0x160/0x180
[<0>] remove_vma+0x20/0x60
[<0>] exit_mmap+0x199/0x340
[<0>] mmput+0x49/0x110
[<0>] do_exit+0x261/0x9b0
[<0>] do_group_exit+0x2c/0x80
[<0>] __x64_sys_exit_group+0x14/0x20
[<0>] x64_sys_call+0x714/0x720
[<0>] do_syscall_64+0x5b/0x160
[<0>] entry_SYSCALL_64_after_hwframe+0x4b/0x53

and yes, it does look like the program is mostly idle for most of the time while returning these huge pages. It's also telling us exactly why we're just sitting idle - RCU grace period.

The below quick change means the runtime of the program is pretty much just the time it takes to execute the parts of it, as you can see from the full output after the change:

axboe@r7525 ~> time sudo ./reg-huge
Got 500 huge pages (each 1024MB) in 0 msec
Faulted in 500 huge pages in 38632 msec
Registered 500 pages in 867 msec

________________________________________________________

Executed in   39.53 secs      fish           external
   usr time    4.88 millis  238.00 micros    4.64 millis
   sys time    0.00 millis    0.00 micros    0.00 millis
Quoted 7 lines

where 38632+876 == 39.51s.

Looks like this was introduced by:

commit bd225530a4c717714722c3731442b78954c765b3

Author: Yu Zhao <[email protected]>
Date:   Thu Jun 27 16:27:05 2024 -0600

Fixes are in git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-stable

Quoted 8 lines

c2a967f6ab0e mm/hugetlb_vmemmap: don't synchronize_rcu() without HVO c0f398c3b2cf mm/hugetlb_vmemmap: batch HVO work when demoting

Additional improvements from mm-stable that may or may not help your test case:

e98337d11bbd mm/contig_alloc: support __GFP_COMP 463586e9ff39 mm/cma: add cma_{alloc,free}_folio() cf54f310d0d3 mm/hugetlb: use __GFP_COMP for gigantic folios

I didn't pick patches, just pulled all of mm-stable - yep that fixes it for me, no longer spending 60+% of the runtime during exit. I trust these are heading to stable?

Signature · 1 line
Jens Axboe
maintainer
#4 lore raw
Quoted 7 lines · On Tue, 10 Sep 2024 17:08:30 -0600 Jens Axboe <[email protected]> wrote:
Quoted 7 lines

Fixes are in git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-stable

c2a967f6ab0e mm/hugetlb_vmemmap: don't synchronize_rcu() without HVO c0f398c3b2cf mm/hugetlb_vmemmap: batch HVO work when demoting

Additional improvements from mm-stable that may or may not help your test case:

Quoted 7 lines
Quoted 3 lines

e98337d11bbd mm/contig_alloc: support __GFP_COMP 463586e9ff39 mm/cma: add cma_{alloc,free}_folio() cf54f310d0d3 mm/hugetlb: use __GFP_COMP for gigantic folios

I didn't pick patches, just pulled all of mm-stable - yep that fixes it for me, no longer spending 60+% of the runtime during exit. I trust these are heading to stable?

None of these are tagged for -stable backporting. We can later request a backport without the cc:stable tag, although narrowing down the required patches would be good.

author
#5 lore raw
Quoted 13 lines · Andrew Morton
Quoted 11 lines · On Tue, 10 Sep 2024 17:08:30 -0600 Jens Axboe <[email protected]> wrote:
Quoted 10 lines

Fixes are in git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-stable

c2a967f6ab0e mm/hugetlb_vmemmap: don't synchronize_rcu() without HVO c0f398c3b2cf mm/hugetlb_vmemmap: batch HVO work when demoting

Additional improvements from mm-stable that may or may not help your test case:

e98337d11bbd mm/contig_alloc: support __GFP_COMP 463586e9ff39 mm/cma: add cma_{alloc,free}_folio() cf54f310d0d3 mm/hugetlb: use __GFP_COMP for gigantic folios

Quoted 7 lines
Quoted 3 lines

I didn't pick patches, just pulled all of mm-stable - yep that fixes it for me, no longer spending 60+% of the runtime during exit. I trust these are heading to stable?

None of these are tagged for -stable backporting. We can later request a backport without the cc:stable tag, although narrowing down the required patches would be good.

These two:

c2a967f6ab0e mm/hugetlb_vmemmap: don't synchronize_rcu() without HVO c0f398c3b2cf mm/hugetlb_vmemmap: batch HVO work when demoting

to cover both cases.

Signature · 1 line
Jens Axboe
#6 lore raw
Quoted 19 lines · Jens Axboe
Quoted 17 lines · Andrew Morton
Quoted 14 lines · On Tue, 10 Sep 2024 17:08:30 -0600 Jens Axboe <[email protected]> wrote:
Quoted 10 lines

Fixes are in git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-stable

c2a967f6ab0e mm/hugetlb_vmemmap: don't synchronize_rcu() without HVO c0f398c3b2cf mm/hugetlb_vmemmap: batch HVO work when demoting

Additional improvements from mm-stable that may or may not help your test case:

e98337d11bbd mm/contig_alloc: support __GFP_COMP 463586e9ff39 mm/cma: add cma_{alloc,free}_folio() cf54f310d0d3 mm/hugetlb: use __GFP_COMP for gigantic folios

I didn't pick patches, just pulled all of mm-stable - yep that fixes it for me, no longer spending 60+% of the runtime during exit. I trust these are heading to stable?

Quoted 7 lines
Quoted 3 lines

None of these are tagged for -stable backporting. We can later request a backport without the cc:stable tag, although narrowing down the required patches would be good.

These two:

c0f398c3b2cf mm/hugetlb_vmemmap: batch HVO work when demoting

This has the stable tag.

Quoted 1 line

c2a967f6ab0e mm/hugetlb_vmemmap: don't synchronize_rcu() without HVO

This one doesn't -- I was hoping it could make into 6.10-rc but that didn't happen. Is it still possible to add the stable tag? If not, I'll follow up on sending the backport.

maintainer
#7 lore raw
In reply to Jens Axboe
Quoted 22 lines · On Wed, 11 Sep 2024 07:22:17 -0600 Jens Axboe <[email protected]> wrote:
Quoted 13 lines · Andrew Morton
Quoted 11 lines · On Tue, 10 Sep 2024 17:08:30 -0600 Jens Axboe <[email protected]> wrote:
Quoted 10 lines

Fixes are in git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-stable

c2a967f6ab0e mm/hugetlb_vmemmap: don't synchronize_rcu() without HVO c0f398c3b2cf mm/hugetlb_vmemmap: batch HVO work when demoting

Additional improvements from mm-stable that may or may not help your test case:

e98337d11bbd mm/contig_alloc: support __GFP_COMP 463586e9ff39 mm/cma: add cma_{alloc,free}_folio() cf54f310d0d3 mm/hugetlb: use __GFP_COMP for gigantic folios

Quoted 7 lines
Quoted 3 lines

I didn't pick patches, just pulled all of mm-stable - yep that fixes it for me, no longer spending 60+% of the runtime during exit. I trust these are heading to stable?

None of these are tagged for -stable backporting. We can later request a backport without the cc:stable tag, although narrowing down the required patches would be good.

Quoted 6 lines

These two:

c2a967f6ab0e mm/hugetlb_vmemmap: don't synchronize_rcu() without HVO c0f398c3b2cf mm/hugetlb_vmemmap: batch HVO work when demoting

to cover both cases.

OK thanks.

Yu, could you please request a -stable backport after this hits mainline?

Thanks.

#8 lore raw
Quoted 27 lines · Andrew Morton
Quoted 24 lines · On Wed, 11 Sep 2024 07:22:17 -0600 Jens Axboe <[email protected]> wrote:
Quoted 13 lines · Andrew Morton
Quoted 11 lines · On Tue, 10 Sep 2024 17:08:30 -0600 Jens Axboe <[email protected]> wrote:
Quoted 10 lines

Fixes are in git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-stable

c2a967f6ab0e mm/hugetlb_vmemmap: don't synchronize_rcu() without HVO c0f398c3b2cf mm/hugetlb_vmemmap: batch HVO work when demoting

Additional improvements from mm-stable that may or may not help your test case:

e98337d11bbd mm/contig_alloc: support __GFP_COMP 463586e9ff39 mm/cma: add cma_{alloc,free}_folio() cf54f310d0d3 mm/hugetlb: use __GFP_COMP for gigantic folios

Quoted 7 lines
Quoted 3 lines

I didn't pick patches, just pulled all of mm-stable - yep that fixes it for me, no longer spending 60+% of the runtime during exit. I trust these are heading to stable?

None of these are tagged for -stable backporting. We can later request a backport without the cc:stable tag, although narrowing down the required patches would be good.

These two:

Quoted 8 lines
Quoted 4 lines

c2a967f6ab0e mm/hugetlb_vmemmap: don't synchronize_rcu() without HVO c0f398c3b2cf mm/hugetlb_vmemmap: batch HVO work when demoting

to cover both cases.

OK thanks.

Yu, could you please request a -stable backport after this hits mainline?

Will do, thanks.