Hugepage program taking forever to exit
Subsystems (2)
Hi,
Investigating another issue, I wrote the following simple program that allocates and faults in 500 1GB huge pages, and then registers them with io_uring. Each step is timed:
Got 500 huge pages (each 1024MB) in 0 msec
Faulted in 500 huge pages in 38632 msec
Registered 500 pages in 867 msec
and as expected, faulting in the pages takes (by far) the longest. From the above, you'd also expect the total runtime to be around ~39 seconds. But it is not... In fact it takes 82 seconds in total for this program to have exited. Looking at why, I see:
[<0>] __wait_rcu_gp+0x12b/0x160 [<0>] synchronize_rcu_normal.part.0+0x2a/0x30 [<0>] hugetlb_vmemmap_restore_folios+0x22/0xe0 [<0>] update_and_free_pages_bulk+0x4c/0x220 [<0>] return_unused_surplus_pages+0x80/0xa0 [<0>] hugetlb_acct_memory.part.0+0x2dd/0x3b0 [<0>] hugetlb_vm_op_close+0x160/0x180 [<0>] remove_vma+0x20/0x60 [<0>] exit_mmap+0x199/0x340 [<0>] mmput+0x49/0x110 [<0>] do_exit+0x261/0x9b0 [<0>] do_group_exit+0x2c/0x80 [<0>] __x64_sys_exit_group+0x14/0x20 [<0>] x64_sys_call+0x714/0x720 [<0>] do_syscall_64+0x5b/0x160 [<0>] entry_SYSCALL_64_after_hwframe+0x4b/0x53
and yes, it does look like the program is mostly idle for most of the time while returning these huge pages. It's also telling us exactly why we're just sitting idle - RCU grace period.
The below quick change means the runtime of the program is pretty much just the time it takes to execute the parts of it, as you can see from the full output after the change:
axboe@r7525 ~> time sudo ./reg-huge
Got 500 huge pages (each 1024MB) in 0 msec
Faulted in 500 huge pages in 38632 msec
Registered 500 pages in 867 msec
________________________________________________________
Executed in 39.53 secs fish external usr time 4.88 millis 238.00 micros 4.64 millis sys time 0.00 millis 0.00 micros 0.00 millis
where 38632+876 == 39.51s.
Looks like this was introduced by:
commit bd225530a4c717714722c3731442b78954c765b3
Author: Yu Zhao <[email protected]> Date: Thu Jun 27 16:27:05 2024 -0600
mm/hugetlb_vmemmap: fix race with speculative PFN walkers
mm/hugetlb_vmemmap.c +1 -1
@@ -517,7 +517,7 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h,long ret =0 ;/* avoid writes from page_ref_add_unless() while unfolding vmemmap */ -synchronize_rcu (); +synchronize_rcu_expedited ();list_for_each_entry_safe (folio, t_folio, folio_list, lru) {if (folio_test_hugetlb_vmemmap_optimized (folio)) {
Signature · 1 line
Jens Axboe
Quoted 15 lines · Jens Axboe
Hi,
Investigating another issue, I wrote the following simple program that allocates and faults in 500 1GB huge pages, and then registers them with
io_uring. Each step is timed:Got 500 huge pages (each 1024MB) in 0 msec
Faulted in 500 huge pages in 38632 msec
Registered 500 pages in 867 msecand as expected, faulting in the pages takes (by far) the longest. From the above, you'd also expect the total runtime to be around ~39 seconds. But it is not... In fact it takes 82 seconds in total for this program to have exited. Looking at why, I see:
Quoted 16 lines
[<0>] __wait_rcu_gp+0x12b/0x160 [<0>] synchronize_rcu_normal.part.0+0x2a/0x30 [<0>] hugetlb_vmemmap_restore_folios+0x22/0xe0 [<0>] update_and_free_pages_bulk+0x4c/0x220 [<0>] return_unused_surplus_pages+0x80/0xa0 [<0>] hugetlb_acct_memory.part.0+0x2dd/0x3b0 [<0>] hugetlb_vm_op_close+0x160/0x180 [<0>] remove_vma+0x20/0x60 [<0>] exit_mmap+0x199/0x340 [<0>] mmput+0x49/0x110 [<0>] do_exit+0x261/0x9b0 [<0>] do_group_exit+0x2c/0x80 [<0>] __x64_sys_exit_group+0x14/0x20 [<0>] x64_sys_call+0x714/0x720 [<0>] do_syscall_64+0x5b/0x160 [<0>] entry_SYSCALL_64_after_hwframe+0x4b/0x53
Yeah, this looks wrong to me:
void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list)
{
struct folio *folio;
LIST_HEAD(vmemmap_pages);
list_for_each_entry(folio, folio_list, lru) {
int ret = hugetlb_vmemmap_split_folio(h, folio);
/*
* Spliting the PMD requires allocating a page, thus lets fail
* early once we encounter the first OOM. No point in retrying
* as it can be dynamically done on remap with the memory
* we get back from the vmemmap deduplication.
*/
if (ret == -ENOMEM)
break;
}
flush_tlb_all();
/* avoid writes from page_ref_add_unless() while folding vmemmap */
synchronize_rcu();
list_for_each_entry(folio, folio_list, lru) {
int ret;
ret = __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages,
VMEMMAP_REMAP_NO_TLB_FLUSH);
/*
* Pages to be freed may have been accumulated. If we
* encounter an ENOMEM, free what we have and try again.
* This can occur in the case that both spliting fails
* halfway and head page allocation also failed. In this
* case __hugetlb_vmemmap_optimize_folio() would free memory
* allowing more vmemmap remaps to occur.
*/
if (ret == -ENOMEM && !list_empty(&vmemmap_pages)) {
flush_tlb_all();
free_vmemmap_page_list(&vmemmap_pages);
INIT_LIST_HEAD(&vmemmap_pages);
__hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages,
VMEMMAP_REMAP_NO_TLB_FLUSH);
}
}
flush_tlb_all();
free_vmemmap_page_list(&vmemmap_pages);
}If you don't have HVO enabled, then hugetlb_vmemmap_split_folio() does nothing. And __hugetlb_vmemmap_optimize_folio() also does nothing, leaving &vmemmap_pages empty and free_vmemmap_page_list() a nop.
So what's left is: it flushes the TLB twice and waits for RCU. What for exactly?
The same is true for hugetlb_vmemmap_optimize_folio() and the corresponding split function, which waits for RCU on every page being allocated and freed, even if the vmemmap is left alone.
Surely all those RCU waits and tlb flushes should be guarded by whether the HVO is actually enabled, no?
Quoted 51 lines · Jens Axboe
Hi,
Investigating another issue, I wrote the following simple program that allocates and faults in 500 1GB huge pages, and then registers them with
io_uring. Each step is timed:Got 500 huge pages (each 1024MB) in 0 msec
Faulted in 500 huge pages in 38632 msec
Registered 500 pages in 867 msecand as expected, faulting in the pages takes (by far) the longest. From the above, you'd also expect the total runtime to be around ~39 seconds. But it is not... In fact it takes 82 seconds in total for this program to have exited. Looking at why, I see:
[<0>] __wait_rcu_gp+0x12b/0x160 [<0>] synchronize_rcu_normal.part.0+0x2a/0x30 [<0>] hugetlb_vmemmap_restore_folios+0x22/0xe0 [<0>] update_and_free_pages_bulk+0x4c/0x220 [<0>] return_unused_surplus_pages+0x80/0xa0 [<0>] hugetlb_acct_memory.part.0+0x2dd/0x3b0 [<0>] hugetlb_vm_op_close+0x160/0x180 [<0>] remove_vma+0x20/0x60 [<0>] exit_mmap+0x199/0x340 [<0>] mmput+0x49/0x110 [<0>] do_exit+0x261/0x9b0 [<0>] do_group_exit+0x2c/0x80 [<0>] __x64_sys_exit_group+0x14/0x20 [<0>] x64_sys_call+0x714/0x720 [<0>] do_syscall_64+0x5b/0x160 [<0>] entry_SYSCALL_64_after_hwframe+0x4b/0x53and yes, it does look like the program is mostly idle for most of the time while returning these huge pages. It's also telling us exactly why we're just sitting idle - RCU grace period.
The below quick change means the runtime of the program is pretty much just the time it takes to execute the parts of it, as you can see from the full output after the change:
axboe@r7525 ~> time sudo ./reg-huge
Got 500 huge pages (each 1024MB) in 0 msec
Faulted in 500 huge pages in 38632 msec
Registered 500 pages in 867 msec________________________________________________________
Executed in 39.53 secs fish external usr time 4.88 millis 238.00 micros 4.64 millis sys time 0.00 millis 0.00 micros 0.00 millis
Quoted 7 lines
where 38632+876 == 39.51s.
Looks like this was introduced by:
commit
bd225530a4c717714722c3731442b78954c765b3Author: Yu Zhao <[email protected]> Date: Thu Jun 27 16:27:05 2024 -0600
Fixes are in git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-stable
c2a967f6ab0e mm/hugetlb_vmemmap: don't synchronize_rcu() without HVO c0f398c3b2cf mm/hugetlb_vmemmap: batch HVO work when demoting
Additional improvements from mm-stable that may or may not help your test case:
e98337d11bbd mm/contig_alloc: support __GFP_COMP 463586e9ff39 mm/cma: add cma_{alloc,free}_folio() cf54f310d0d3 mm/hugetlb: use __GFP_COMP for gigantic folios
Quoted 62 lines · Yu Zhao
Quoted 51 lines · On Tue, Sep 10, 2024 at 12:21?PM Jens Axboe <[email protected]> wrote:
Hi,
Investigating another issue, I wrote the following simple program that allocates and faults in 500 1GB huge pages, and then registers them with
io_uring. Each step is timed:Got 500 huge pages (each 1024MB) in 0 msec
Faulted in 500 huge pages in 38632 msec
Registered 500 pages in 867 msecand as expected, faulting in the pages takes (by far) the longest. From the above, you'd also expect the total runtime to be around ~39 seconds. But it is not... In fact it takes 82 seconds in total for this program to have exited. Looking at why, I see:
[<0>] __wait_rcu_gp+0x12b/0x160 [<0>] synchronize_rcu_normal.part.0+0x2a/0x30 [<0>] hugetlb_vmemmap_restore_folios+0x22/0xe0 [<0>] update_and_free_pages_bulk+0x4c/0x220 [<0>] return_unused_surplus_pages+0x80/0xa0 [<0>] hugetlb_acct_memory.part.0+0x2dd/0x3b0 [<0>] hugetlb_vm_op_close+0x160/0x180 [<0>] remove_vma+0x20/0x60 [<0>] exit_mmap+0x199/0x340 [<0>] mmput+0x49/0x110 [<0>] do_exit+0x261/0x9b0 [<0>] do_group_exit+0x2c/0x80 [<0>] __x64_sys_exit_group+0x14/0x20 [<0>] x64_sys_call+0x714/0x720 [<0>] do_syscall_64+0x5b/0x160 [<0>] entry_SYSCALL_64_after_hwframe+0x4b/0x53and yes, it does look like the program is mostly idle for most of the time while returning these huge pages. It's also telling us exactly why we're just sitting idle - RCU grace period.
The below quick change means the runtime of the program is pretty much just the time it takes to execute the parts of it, as you can see from the full output after the change:
axboe@r7525 ~> time sudo ./reg-huge
Got 500 huge pages (each 1024MB) in 0 msec
Faulted in 500 huge pages in 38632 msec
Registered 500 pages in 867 msec________________________________________________________
Executed in 39.53 secs fish external usr time 4.88 millis 238.00 micros 4.64 millis sys time 0.00 millis 0.00 micros 0.00 millisQuoted 7 lines
where 38632+876 == 39.51s.
Looks like this was introduced by:
commit
bd225530a4c717714722c3731442b78954c765b3Author: Yu Zhao <[email protected]> Date: Thu Jun 27 16:27:05 2024 -0600Fixes are in
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.gitmm-stable
Quoted 8 lines
c2a967f6ab0emm/hugetlb_vmemmap: don'tsynchronize_rcu()without HVOc0f398c3b2cfmm/hugetlb_vmemmap: batch HVO work when demotingAdditional improvements from mm-stable that may or may not help your test case:
e98337d11bbdmm/contig_alloc: support__GFP_COMP463586e9ff39mm/cma: add cma_{alloc,free}_folio()cf54f310d0d3mm/hugetlb: use__GFP_COMPfor gigantic folios
I didn't pick patches, just pulled all of mm-stable - yep that fixes it for me, no longer spending 60+% of the runtime during exit. I trust these are heading to stable?
Signature · 1 line
Jens Axboe
Andrew Morton
maintainerQuoted 7 lines · On Tue, 10 Sep 2024 17:08:30 -0600 Jens Axboe <[email protected]> wrote:
Quoted 7 lines
Fixes are in
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.gitmm-stable
c2a967f6ab0emm/hugetlb_vmemmap: don'tsynchronize_rcu()without HVOc0f398c3b2cfmm/hugetlb_vmemmap: batch HVO work when demotingAdditional improvements from mm-stable that may or may not help your test case:
Quoted 7 lines
Quoted 3 lines
e98337d11bbdmm/contig_alloc: support__GFP_COMP463586e9ff39mm/cma: add cma_{alloc,free}_folio()cf54f310d0d3mm/hugetlb: use__GFP_COMPfor gigantic foliosI didn't pick patches, just pulled all of mm-stable - yep that fixes it for me, no longer spending 60+% of the runtime during exit. I trust these are heading to stable?
None of these are tagged for -stable backporting. We can later request a backport without the cc:stable tag, although narrowing down the required patches would be good.
Quoted 13 lines · Andrew Morton
Quoted 11 lines · On Tue, 10 Sep 2024 17:08:30 -0600 Jens Axboe <[email protected]> wrote:
Quoted 10 lines
Fixes are in
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.gitmm-stable
c2a967f6ab0emm/hugetlb_vmemmap: don'tsynchronize_rcu()without HVOc0f398c3b2cfmm/hugetlb_vmemmap: batch HVO work when demotingAdditional improvements from mm-stable that may or may not help your test case:
e98337d11bbdmm/contig_alloc: support__GFP_COMP463586e9ff39mm/cma: add cma_{alloc,free}_folio()cf54f310d0d3mm/hugetlb: use__GFP_COMPfor gigantic folios
Quoted 7 lines
Quoted 3 lines
I didn't pick patches, just pulled all of mm-stable - yep that fixes it for me, no longer spending 60+% of the runtime during exit. I trust these are heading to stable?
None of these are tagged for -stable backporting. We can later request a backport without the cc:stable tag, although narrowing down the required patches would be good.
These two:
c2a967f6ab0e mm/hugetlb_vmemmap: don't synchronize_rcu() without HVO c0f398c3b2cf mm/hugetlb_vmemmap: batch HVO work when demoting
to cover both cases.
Signature · 1 line
Jens Axboe
Quoted 19 lines · Jens Axboe
Quoted 17 lines · Andrew Morton
Quoted 14 lines · On Tue, 10 Sep 2024 17:08:30 -0600 Jens Axboe <[email protected]> wrote:
Quoted 10 lines
Fixes are in
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.gitmm-stable
c2a967f6ab0emm/hugetlb_vmemmap: don'tsynchronize_rcu()without HVOc0f398c3b2cfmm/hugetlb_vmemmap: batch HVO work when demotingAdditional improvements from mm-stable that may or may not help your test case:
e98337d11bbdmm/contig_alloc: support__GFP_COMP463586e9ff39mm/cma: add cma_{alloc,free}_folio()cf54f310d0d3mm/hugetlb: use__GFP_COMPfor gigantic foliosI didn't pick patches, just pulled all of mm-stable - yep that fixes it for me, no longer spending 60+% of the runtime during exit. I trust these are heading to stable?
Quoted 7 lines
Quoted 3 lines
None of these are tagged for -stable backporting. We can later request a backport without the cc:stable tag, although narrowing down the required patches would be good.
These two:
c0f398c3b2cfmm/hugetlb_vmemmap: batch HVO work when demoting
This has the stable tag.
Quoted 1 line
c2a967f6ab0emm/hugetlb_vmemmap: don'tsynchronize_rcu()without HVO
This one doesn't -- I was hoping it could make into 6.10-rc but that didn't happen. Is it still possible to add the stable tag? If not, I'll follow up on sending the backport.
Andrew Morton
maintainerQuoted 22 lines · On Wed, 11 Sep 2024 07:22:17 -0600 Jens Axboe <[email protected]> wrote:
Quoted 13 lines · Andrew Morton
Quoted 11 lines · On Tue, 10 Sep 2024 17:08:30 -0600 Jens Axboe <[email protected]> wrote:
Quoted 10 lines
Fixes are in
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.gitmm-stable
c2a967f6ab0emm/hugetlb_vmemmap: don'tsynchronize_rcu()without HVOc0f398c3b2cfmm/hugetlb_vmemmap: batch HVO work when demotingAdditional improvements from mm-stable that may or may not help your test case:
e98337d11bbdmm/contig_alloc: support__GFP_COMP463586e9ff39mm/cma: add cma_{alloc,free}_folio()cf54f310d0d3mm/hugetlb: use__GFP_COMPfor gigantic foliosQuoted 7 lines
Quoted 3 lines
I didn't pick patches, just pulled all of mm-stable - yep that fixes it for me, no longer spending 60+% of the runtime during exit. I trust these are heading to stable?
None of these are tagged for -stable backporting. We can later request a backport without the cc:stable tag, although narrowing down the required patches would be good.
Quoted 6 lines
These two:
c2a967f6ab0emm/hugetlb_vmemmap: don'tsynchronize_rcu()without HVOc0f398c3b2cfmm/hugetlb_vmemmap: batch HVO work when demotingto cover both cases.
OK thanks.
Yu, could you please request a -stable backport after this hits mainline?
Thanks.
Quoted 27 lines · Andrew Morton
Quoted 24 lines · On Wed, 11 Sep 2024 07:22:17 -0600 Jens Axboe <[email protected]> wrote:
Quoted 13 lines · Andrew Morton
Quoted 11 lines · On Tue, 10 Sep 2024 17:08:30 -0600 Jens Axboe <[email protected]> wrote:
Quoted 10 lines
Fixes are in
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.gitmm-stable
c2a967f6ab0emm/hugetlb_vmemmap: don'tsynchronize_rcu()without HVOc0f398c3b2cfmm/hugetlb_vmemmap: batch HVO work when demotingAdditional improvements from mm-stable that may or may not help your test case:
e98337d11bbdmm/contig_alloc: support__GFP_COMP463586e9ff39mm/cma: add cma_{alloc,free}_folio()cf54f310d0d3mm/hugetlb: use__GFP_COMPfor gigantic foliosQuoted 7 lines
Quoted 3 lines
I didn't pick patches, just pulled all of mm-stable - yep that fixes it for me, no longer spending 60+% of the runtime during exit. I trust these are heading to stable?
None of these are tagged for -stable backporting. We can later request a backport without the cc:stable tag, although narrowing down the required patches would be good.
These two:
Quoted 8 lines
Quoted 4 lines
c2a967f6ab0emm/hugetlb_vmemmap: don'tsynchronize_rcu()without HVOc0f398c3b2cfmm/hugetlb_vmemmap: batch HVO work when demotingto cover both cases.
OK thanks.
Yu, could you please request a -stable backport after this hits mainline?
Will do, thanks.